Add real-data validation against JPL Horizons#330
Open
matthewholman wants to merge 1 commit into
Open
Conversation
End-to-end test that runs orbitfit on real MPC astrometry (from the existing 4_random_mpc_ADES_provIDs_no_sats.csv test fixture, three real numbered asteroids) and asserts the recovered Cartesian state matches JPL Horizons' reference at the same epoch. This is the most realistic check in the layup test suite -- every other test uses synthetic observations (whether generated via predict_sequence, or from the diagnostic_scan dataset built by integrating known orbits with ASSIST). Empirical agreement on the three tested mainbelt asteroids: - 119839 (2002 CX17, 587 obs, 27-year arc): ~150 km in position - 742428 (2007 TC75, 117 obs): ~130 km in position - 609631 (2005 HE12, 109 obs): ~20 km in position All against object distances of ~2-3 AU (~3 x 10^8 km), so relative position agreement of 1e-7 to 1e-6. Velocity agreement is 1e-9 AU/day or better. Test tolerances (1e-5 AU pos, 1e-7 AU/day vel) sit comfortably above the observed accuracy so benign numerical drift won't flag false positives. Hermetic by design: the JPL reference states are committed to tests/data/jpl_reference_states.json (captured once via the public Horizons API on 2026-05-17; the file's _comment_ field documents how to regenerate). The test runs in CI without any network access. Also catches any future drift in layup's epoch-selection logic: the test asserts that the layup-chosen epoch matches the fixture's recorded epoch to 1e-9 days. If layup changes how it picks the fit epoch for a given observation set, the fixture needs regeneration. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
This branch was developed with Claude Code, on top of an initial
implementation I had started.
Adds an end-to-end validation test that runs
orbitfiton real MPCastrometry and asserts the recovered Cartesian state matches the
JPL Horizons reference at the same epoch. Every other test in the
layup suite uses synthetic observations (either generated via
predict_sequenceor from the diagnostic-scan dataset built byintegrating known orbits with ASSIST) -- this is the first
real-world ground-truth check.
What
Three numbered asteroids from the existing
4_random_mpc_ADES_provIDs_no_sats.csvtest fixture, each compared to JPL Horizons:
All three objects are mainbelt at ~2-3 AU (i.e., ~3 × 10⁸ km), so
relative position agreement is 1×10⁻⁷ to 1×10⁻⁶. Velocity agreement
is 1×10⁻⁹ AU/day or better. Test tolerances (1×10⁻⁵ AU pos,
1×10⁻⁷ AU/day vel) sit comfortably above the observed accuracy so
benign numerical drift won't flag false positives.
How
tests/data/jpl_reference_states.json-- captured once via JPLHorizons' public API on 2026-05-17 (see the
_comment_field forregeneration instructions: query
https://ssd.jpl.nasa.gov/api/horizons.apiwithCENTER='@0',REF_PLANE='FRAME',REF_SYSTEM='ICRF',VEC_TABLE='2',OUT_UNITS='AU-D'at the layup-recovered epoch).The test runs hermetically against this committed fixture -- no
network access during CI.
It also catches future drift in layup's epoch-selection logic: the
test asserts that layup's chosen epoch matches the fixture's
recorded epoch to 10⁻⁹ days. If layup changes how it picks the fit
epoch for a given observation set, the fixture needs regeneration
(the test message says so).
Independent
Doesn't depend on any of the other open BK-related PRs -- branches
off
main, touches onlytests/.Review Checklist for Source Code Changes
tests/layup/test_real_data_validation.py