When Is Availability-Aware Training Worth It? A Benchmark and Empirical Study of Interruption-Resilient Optimization Under Predictable Compute Schedules
1Rota, Inc. · Submitted June 2026 · 11 pages, 3 figures, 6 tables
Abstract
Training under non-stationary but predictable compute availability (satellites under eclipse, duty-cycled edge devices, power-capped datacenters) is often framed as needing specialized, availability-aware optimizers. We test that premise. We release OrbitTrace, a benchmark of 50 physics-grounded availability traces from SGP4 propagation of live two-line element sets across three orbital regimes, and ask a falsifiable question: when an availability gap interrupts training, is a specialized resumption strategy worth it, or is competent checkpoint-and-resume enough?
Our central finding: when optimizer state can be preserved across a gap, the gap is essentially free. A strong checkpoint baseline that restores full optimizer state and indexes its learning-rate schedule in effective (active) time matches uninterrupted training to within data-ordering noise on CIFAR-10/ResNet-18, and exactly on a GPT-2/AdamW task. Advantages previously reported for availability-aware methods, including our own three-pillar method AAT, arise almost entirely from comparison against a weak baseline that indexes its schedule on wall-clock time. Against the strong baseline, reactive adaptation provides no advantage across stationary, optimizer-state-loss, and distribution-drift regimes.
We isolate one narrow regime where it helps: for large models whose optimizer state cannot be persisted across gaps and that are interrupted by frequent, short pauses, reconstructing a decayed optimizer moment recovers only ≈21% of the state-loss penalty on average (and not robustly across seeds); this vanishes for eclipse-scale gaps, where the decayed moment is indistinguishable from zero. Our contributions are a benchmark, a strong reproducible baseline protocol, and a clear characterization of when interruption-resilient optimization is worth its complexity, and when it is not.
1Introduction
A growing class of deployments trains models on compute whose availability is non-stationary, highly structured, and, unusually, known in advance. A satellite in low Earth orbit loses solar power during eclipse on a schedule derivable to the minute from its published orbital elements. Battery-powered edge nodes, solar microgrids, and power-capped datacenters follow duty cycles that are predictable hours to days ahead. In each case the training loop has access to an availability function A(t) ∈ {0, 1} over the whole training horizon.
It is tempting to treat that foreknowledge as an opportunity for a specialized optimizer: a method that anticipates and repairs the damage an interruption does to momentum, learning-rate schedules, and normalization statistics. This paper asks a deliberately falsifiable question. When training is interrupted by a predictable availability gap, does a specialized availability-aware resumption strategy beat a competent checkpoint-and-resume baseline, and if so, in which regimes?
The answer is mostly negative, and the crux is the strength of the baseline. A baseline that preserves full optimizer state across the gap and advances its learning-rate schedule in effective (active) time rather than wall-clock time1 reproduces uninterrupted training almost perfectly. Reported wins for availability-aware methods are, on inspection, wins over a weak baseline that over-anneals its learning rate during idle intervals. A competent practitioner does not build that baseline.
That does not make the question empty. A gap can be costly in two specific ways, and the paper tests both: (A) when optimizer state cannot be persisted across the gap, a realistic constraint when checkpoints compete for scarce power, storage, or downlink and the optimizer state is one to two times the model size; and (B) when the data distribution drifts across the gap so that preserved state is stale. The paper’s contributions are:
- OrbitTrace, a benchmark of 50 physics-grounded availability traces generated deterministically by SGP4 propagation of live two-line element sets across three orbital regimes and five workload patterns (§2).
- A strong baseline and evaluation protocol. Checkpoint-Strong, full-state preservation with effective-time learning-rate indexing, is shown to be equivalent to continuous training up to data-ordering noise. This, not wall-clock checkpointing, is the bar every availability-aware claim must clear (§3).
- A decision boundary for when reactive availability-aware adaptation helps, across stationary, optimizer-state-loss, and distribution-drift regimes on CIFAR-10/ResNet-18 and a GPT-2/AdamW task (§4, §5).
2The OrbitTrace benchmark
OrbitTrace is a set of 50 availability traces for reproducible evaluation of training under predictable interruption. Each trace is produced by SGP4 propagation of a published two-line element set for a satellite in one of three orbital regimes: ISS-like (420 km, 51.6°), sun-synchronous (700 km, 97.4°), and low-inclination (550 km, 28.5°). Solar illumination windows come from a cylindrical shadow model, ground-station contacts from a 12-station catalog, and a workload profile maps the combined power and communication availability to compute-active intervals. Generation is deterministic: the same element set yields the same trace.
The traces differ structurally from the random interruptions common in fault-tolerance studies. Session durations are bimodal (8-15 minutes when communications-bound, 40-60 minutes when compute-bound), gaps cluster tightly around the eclipse length (roughly 30 minutes for ISS-like orbits), and consecutive sessions and gaps are strongly length-correlated. That structure is exactly what an availability-aware method would exploit if exploitation paid off, which makes OrbitTrace a fair test of the premise rather than a straw setting.
3Why a preserved gap is free
Consider the optimizer as a state machine over parameters, momentum or moment estimates, normalization statistics, and the learning-rate schedule. If every element of that state is written before a gap and restored after it, and the schedule is indexed by the number of steps taken rather than by elapsed time, the sequence of updates after the gap is the one uninterrupted training would have produced from the same point. The only difference is the order of minibatches, which is noise. Under full-state preservation an availability gap therefore injects no bias and no excess loss. Table 1 summarizes the two baselines.
| Baseline | State across the gap | Learning-rate schedule |
|---|---|---|
| Checkpoint (wall-clock) | Full optimizer state restored | Indexed on wall-clock time; keeps decaying through the gap and resumes over-annealed |
| Checkpoint-Strong | Full optimizer state restored | Indexed on effective (active) time; resumes where it stopped |
The two ways a gap can still cost something are therefore the two ways this argument can fail: the state is not preserved (Cost A), or the preserved state is stale because the data moved (Cost B).
4Results
The paper evaluates Checkpoint-Strong against the wall-clock baseline, an elastic-training reference, a proactive foreknowledge prototype, and AAT, a reactive three-pillar method (momentum rescale, effective-time learning rate, batch-norm warm restart), on CIFAR-10/ResNet-18 and GPT-2/AdamW on WikiText-2, over three and five seeds respectively.
Stationary data, full preservation. Every method that preserves full state finishes within seed noise of continuous training. The wall-clock baseline is measurably worse, and that difference alone accounts for the previously reported advantage of availability-aware methods.
Pillar ablation. Adding momentum rescaling or batch-norm restart on top of effective-time scheduling does not help; both are within noise or slightly negative. Effective-time indexing is the whole effect.
Cost A, non-persistable state. With weights-only checkpoints under frequent short gaps, dropping optimizer state costs accuracy, and reconstructing a decayed moment claws back part of it, but every such configuration remains below full preservation. On the GPT-2 task, Checkpoint-Strong matches continuous training exactly in both gap regimes. On short gaps, reconstruction recovers about 21% of the state-loss penalty on average, with high variance across seeds. On eclipse-scale gaps the decayed moment is indistinguishable from zero and the benefit vanishes.
Cost B, distribution drift. Under a synthetic per-session input shift, full preservation still wins. The reactive pillars provide no reliable advantage.
5The decision boundary
Table 2 synthesizes the experiments into the practitioner-facing result: which strategy to use, and whether availability-aware adaptation is worth its complexity, as a function of whether optimizer state can be preserved and how long the gaps are.
| State preservable? | Gaps | Best strategy | Adaptation worth it? |
|---|---|---|---|
| Yes (full) | any | Checkpoint-Strong | No |
| Yes (full) | any, with drift | Checkpoint-Strong | No |
| No (weights only) | frequent, short | Reconstruct a decayed moment | Marginally (≈21%, noisy) |
| No (weights only) | eclipse-scale | Elastic resume | No (the decayed moment is zero) |
The actionable summary: checkpoint full optimizer state and index the schedule on effective time. Reach for availability-aware reconstruction only when optimizer state genuinely cannot be persisted (large models on constrained nodes) and the interruptions are frequent and short relative to the momentum timescale, as with thermal throttling, contention, or attitude maneuvers, rather than eclipse-scale outages.
6Limitations and what is left open
Results are over three seeds on CIFAR-10 and five on GPT-2, and the lone positive result is high-variance. The models tested are ResNet-18 and a small GPT-2; the positive Cost-A regime should grow more relevant at larger scale, where optimizer state is genuinely unpersistable, but that remains to be verified. The drift experiment uses a synthetic per-session transform rather than a true on-orbit distribution shift. Schedule foreknowledge is real, and only one evaluated method uses it; the naive version did not help, because under full preservation there is nothing to protect. The version worth testing, consolidating weights across a known boundary and allocating interruption-sensitive work into predicted-long sessions, needs a regime where the gap is costly in order to have anything to buy. The paper leaves that as an open hypothesis.
References
- Rotastellar Research. Constraint-Aware Execution Planning for Hybrid Space-Ground Compute Workloads. arXiv:2605.04052, 2026.
- Vallado, D. A., Crawford, P., Hujsak, R., and Kelso, T. S. Revisiting Spacetrack Report #3. AIAA/AAS Astrodynamics Specialist Conference, 2006.
- Montenbruck, O. and Gill, E. Satellite Orbits: Models, Methods and Applications. Springer, 2000.
- Izmailov, P., Podoprikhin, D., Garipov, T., Vetrov, D., and Wilson, A. G. Averaging Weights Leads to Wider Optima and Better Generalization. UAI, 2018.