Checkpointing Is Enough. We Measured It on 50 Orbital Traces.
We set out to show that training through eclipses needs a specialised optimiser. The data said otherwise. Here is the paper, and what it means if you run compute in orbit.
We expected to publish a very different paper.
Early this year we had a method for training models on compute that switches off on a known schedule. Satellites in eclipse, duty-cycled edge boxes, power-capped racks. The method anticipated the gap and repaired the damage an interruption does to momentum, the learning-rate schedule, and the normalisation statistics. Our first experiments said it recovered most of the accuracy that naive checkpoint-and-resume lost. We were pleased with it.
Then we built a proper baseline, and the advantage disappeared.
The paper that came out of that is now on arXiv, and the honest title is the finding: When Is Availability-Aware Training Worth It? The short answer is almost never, with one narrow exception, and we think that is the more useful result.
The baseline is the whole story
When a training run is interrupted, the obvious thing to do is checkpoint before the gap and restore after it. The question is what you restore and how you resume.
A lot of published comparisons, including our own early ones, resume against a baseline that restores the model weights and then keeps the learning-rate schedule running on wall-clock time. So while the machine is dark the schedule keeps decaying, and when training resumes the learning rate is lower than it should be. Over many gaps the run is quietly over-annealed. Any method that avoids this looks great by comparison.
But that baseline is a straw man. A competent practitioner restores the full optimiser state, momentum and all, and indexes the schedule on the number of steps actually taken rather than the time elapsed. Do that, and the sequence of updates after the gap is exactly the sequence uninterrupted training would have produced from the same point. The only difference is minibatch order, which is noise.
The same cosine schedule through one gap. Indexed on wall-clock time, the schedule keeps decaying while the machine is dark and resumes far too low. Indexed on steps taken, it resumes exactly where it stopped. This one difference accounts for most of what has been claimed for availability-aware methods, ours included.
OrbitTrace
To test this honestly we needed interruption schedules that look like the real thing, not coin flips. So we built OrbitTrace: fifty availability traces generated by propagating published orbital elements with SGP4 for satellites in three regimes, an ISS-like orbit, a sun-synchronous one, and a low-inclination one. Illumination comes from a shadow model, ground contacts from a twelve-station catalogue, and a workload profile turns the combination into intervals where compute is available and intervals where it is not.
The traces have a structure that random interruptions do not. Sessions are bimodal, short when the job is communications-bound and long when it is compute-bound. Gaps cluster tightly around the eclipse length, about half an hour for an ISS-like orbit. Consecutive sessions and gaps are correlated in length. That structure is exactly what an availability-aware method would exploit if exploiting it paid off, which makes OrbitTrace a fair test rather than a convenient one. The paper describes the benchmark in detail.
What we found
With a strong baseline in place, every method that preserves full optimiser state finishes within seed noise of uninterrupted training. On CIFAR-10 with a ResNet-18 the gap is essentially free. On a GPT-2 language modelling task the strong baseline matches continuous training exactly. Adding our method’s extra machinery on top does not help, and in some configurations it hurts slightly.
There are two ways a gap can still cost something. One is when the optimiser state cannot be saved across it, which is a real constraint when checkpoints compete for scarce storage or downlink and the optimiser state is larger than the model. The other is when the data distribution drifts while the machine is dark. We tested both.
Drift did not change the conclusion. Full preservation still wins.
Non-persistable state is where the one exception lives. For a larger model whose optimiser state has to be discarded, and which is interrupted by frequent short pauses, reconstructing a decayed version of the lost momentum recovers roughly a fifth of the penalty on average. It is not consistent across seeds. And it vanishes entirely for eclipse-scale gaps, because by the time half an hour has passed the decayed momentum is indistinguishable from zero.
| Can you save optimiser state? | Gaps | What to do | Is adaptation worth it? |
|---|---|---|---|
| Yes | any | checkpoint everything, resume on effective time | no |
| Yes | any, with drift | same | no |
| No | frequent and short | reconstruct a decayed moment | marginally, and noisily |
| No | eclipse-scale | plain resume | no |
What this means if you run compute in orbit
Checkpoint the full optimiser state. Index your schedule on steps taken, not on the clock. That is the entire recipe, and it is enough for every regime we could construct that resembles an orbit.
The interesting work moves somewhere else. If a gap is free as long as the checkpoint lands before it, then the valuable thing is knowing when the gap is coming. In orbit that is not a prediction problem. It is a propagation problem, and it is the same one the planner already solves to find the windows a workload can run in. The planner knows the eclipse is coming to the second, so the checkpoint can be placed just before it, every time, with no clever optimiser involved.
We would rather have published the paper we set out to write. This one is more useful. If you are building training infrastructure for intermittent compute, it tells you not to overbuild, and it tells you precisely the one corner where overbuilding might be worth it.