A Stale Element Set Lies Quietly
Two things went wrong with our orbital data this summer. One was loud. The other produced perfectly formatted, confidently wrong answers for weeks. Here is what we changed.
Everything the planner does starts with two lines of text. A two-line element set describes a satellite’s orbit at a moment in time, and SGP4 propagates it forward from there. Feed it fresh elements and it is accurate to a kilometre or so for days. Feed it old ones and it is still perfectly happy to give you an answer.
That second property is the subject of this post. We had two failures of orbital data freshness this summer. The first was an ordinary outage and we fixed it in a day. The second was far more interesting, because nothing was down and every response was well formed. The answers were just wrong.
The loud one
In late June we moved our production stack to a new host. The move itself went fine. What did not move was the scheduled job that pulls the satellite catalogue once a day. Nobody noticed, because nothing errored. The console kept serving the elements it already had, and those elements got a day older every day.
By the time someone looked, the newest element set in the console was nineteen days old. Separately, the public tracker had gone dark, for a reason that only made sense once we traced it: its refresh routine cleared its cache before fetching a new catalogue, the fetch hit a dead fallback path left over from the migration, and it wrote back an empty catalogue. An empty catalogue on every cold start is a tracker with nothing to track.
Both were fixed the same day. The sync job was restored, the fallback was repointed at a source that still exists, and we added a freshness check to the health monitor so that a catalogue that stops updating pages someone rather than silently aging.
The lesson from that one is dull and we will only say it once. A migration checklist has to include the things that run on a timer, because they are the things that fail by not happening.
The quiet one
The second problem we only found last night, while chasing a report that a workload was infeasible on a satellite where it should have been fine.
Here is the thing about SGP4 and old elements. It does not refuse. There is no error for “these elements are three months old”. It takes the mean motion and the drag term and runs the orbit forward, and for an element set that is old enough, the modelled decay takes the satellite’s altitude down through zero. SGP4 will cheerfully return a position below the surface of the Earth.
Now think about what the planner does with that. It builds an environment from the propagated orbit. A satellite below the surface is in permanent shadow, so the eclipse fraction is one hundred percent. It never rises above the horizon of any ground station, so there are no passes. The planner looks at an environment with no sunlight and no contact and reaches the only conclusion available to it: nothing can run here.
So it returns infeasible. With reasons. Formatted beautifully. And every reason is a true statement about a satellite that does not exist.
The failure and the fix. The old path produced a plausible answer to a question about a satellite that had been propagated into the ground. The new path refuses to ask the question.
What we changed
The fix is a guard, and it runs before SGP4 does. If the element epoch is older than a threshold, or the propagated altitude at the start of the planning horizon is not physically sensible, the planner stops and says so. The response tells you the epoch it had and the limit it applied. It does not build an environment. It does not return a plan.
This felt uncomfortable at first. We like the planner to answer. But an answer built on a satellite that is not where the elements say it is is not an answer, it is a liability, and it is one that would have been very hard to catch from the outside. A customer sees infeasible, assumes the workload is too big, and shrinks it. That is the quiet failure mode we most want to avoid.
The second change is on the data side. We now serve elements through a single gateway that every service reads from, refreshed once a day from one upstream source. Once a day is a deliberate ceiling, not a limit we hit. The public catalogues are run by people who are generous with their data and understandably strict about clients that hammer them, and we have seen what happens to clients that re-query the same group twice in a couple of hours. One source, one fetch, everyone reads the copy.
The general point
If you build anything on SGP4, build in the assumption that it will never tell you the input is bad. Age is not a property it checks. Physical plausibility is not a property it checks. It is a propagator, not a validator, and the two failure modes above are what happens when you forget that.
The loud failure cost us a day. The quiet one could have cost us a customer’s trust without either of us knowing why. We would rather return no plan than a wrong one, and now the planner agrees.