What kept GPUs waiting: an E1 study of distributed training overhead

October 21, 2026 · Denis Voronko

Abstract

Training rounds can spend substantial time on work outside GPU computation. In Nodeox E1, we investigated that time using phase-level records from a 16-A40 ViT-B/16 experiment, a two-A40 ViT-H/14 dry run, and later functional checks on one AMD MI300X. The records distinguished slow CPU-side update encoding from upload, exposed unnecessary full-model persistence, and motivated changes to retained coordinator objects. These observations support targeted engineering conclusions. They do not establish large-model multi-GPU scaling or a controlled end-to-end speed-up.

Research question and scope

What was happening between the end of GPU computation and the start of the next training round? Our benchmarking article describes E1’s scaling and accuracy results. Here we examine the work behind those results: preparing updates, collecting and combining them, sending the new model back, and retaining or saving full-model objects.

The scope is the 16–17 October observations and the engineering checks on 29 October. Later experiments are separate. This is an exploratory study of recorded runs, rather than a comparison with standard distributed-training frameworks.

Experimental material and measurement

E1 run 6 trained ViT-B/16 on 16 NVIDIA A40s in Sweden, using either a coordinator in Germany or one in the GPUs’ datacenter. The 17 October dry run exercised ViT-H/14 on one and two A40s. The 29 October follow-up used two software hosts sharing one physical MI300X in Atlanta; it was not a two-GPU scaling test.

Both models used Oxford-IIIT Pet data. ViT-B/16 has about 86 million parameters; ViT-H/14 has about 632 million and a 2.5 GB model file. The two-A40 ViT-H probe used full fine-tuning in fp32, batch 32, q4 deltas and 20 local steps per host per round.

We inspected host and coordinator events around training, delta preparation, aggregation and synchronisation. Coordinator memory was also sampled every two seconds in the ViT-H dry run. Event intervals include the work between their named boundaries; they are not pure network-latency measurements. Hosts operate in parallel, so their component ranges cannot simply be added into a new whole-round statistic.

CPU encoding delayed an upload

In the Germany-path 16-A40 run, two hosts began uploading much later than the others. Host-log observations recorded in the session notes locate the delay before the upload: in round 4, encoding took 17.8 and 18.0 seconds on those hosts, versus 1.5–2.3 seconds on the others. Upload itself took about one second.

ViT-B/16, run 6, round 4: two hosts took 17.8 and 18.0 s to encode, versus 1.5–2.3 s on the others. Upload took about 1 s. Source: run 6 session notes.

The two slow hosts shared a physical machine. Their software could see more CPU threads than the pod’s quota, while encoding used CPU work. Oversubscription was a plausible explanation, but could not be directly tested on the rented pods. The encoding delay is observed; the underlying scheduling cause remains a hypothesis. More network bandwidth would not directly remove this local preparation time.

Coordinator placement changed one part of the exchange

At 16 A40s, synchronisation start to the last host synchronised took about 3 seconds with the coordinator beside the GPUs and about 13 seconds through Germany in rounds 2 and 3. This interval includes reception and application of the update. Moving the coordinator helped that phase; it did not turn the earlier CPU-encoding delay into a network-distance effect.

The larger model exposed more local work

ViT-H/14 ran end to end on two A40s, including final evaluation. The short run checked the execution path, not target accuracy or scaling. In the first two rounds, roughly 48 seconds of training accompanied about 60 seconds of exchange work.

Host update preparation and delivery took 17.8–22.3 seconds, including an upload of about 0.7 seconds. Aggregation took 2.9–3.4 seconds. From the recorded aggregation event to sync start was approximately 22.6–23.4 seconds. Host sync application took 9.8–14.5 seconds, although downloads took only about 0.5–0.7 seconds.

ViT-H/14 on two A40s, first two rounds. Component ranges are not additive whole-round measurements. The fallback save itself took about 14–15 s.

Saving the full fallback model accounted for about 14–15 seconds of the aggregation-to-sync interval: serialisation plus upload to storage. It was saved every round, although a host needed it only when it could not apply an update. Hashing and sync-delta preparation added other work. Calling that entire interval “model saving” would overstate the saving operation’s cost.

The coordinator also peaked at 34.4 GiB in the dry run and retained substantial memory after completion. This was another reason an end-to-end functional run could not be presented as proof of readiness to scale.

Targeted changes and follow-up checks

The first change made the full round model available on demand instead of saving it unconditionally each round. In the 29 October session, the notes report aggregation-to-sync intervals below the check’s 10-second criterion, with the round model in on-demand mode, no fallback save and successful delta application.

That was a local acceptance check on a different setup: two software hosts on one MI300X, with different CPU and storage resources from the A40 dry run. We do not turn it into a percentage improvement in complete training time.

The session also investigated retained coordinator memory. Its code investigation identified a reference cycle in the aggregation execution namespace that kept full-model objects alive until cyclic garbage collection. Clearing the namespace was followed by a lower reported peak and flatter between-round memory in a later check.

These are reported engineering observations, not repeated independent trials.

Investigation at a glance

Limitations

The records identify bottlenecks and support specific interventions under stated conditions. They do not isolate every causal factor. The rented hosts’ CPU environment could not be directly inspected, and hardware as well as software changed between the dry run and follow-up. The AMD session used one physical GPU, so it cannot demonstrate scaling across multiple physical GPUs.

The study does not compare Nodeox with standard homogeneous training tools, measure scheduler waiting, or establish customer time or cost savings. Short functional checks do not establish long-run stability. The original E1 scaling result has its own conditions, including a same-site one-GPU baseline based on one measured round; it is not a substitute for those missing comparisons.

Conclusion

The useful finding was where the exchange waited: CPU encoding before upload, model persistence on the critical path, update application, and retained model objects. Treating all of that as a network problem would have hidden several actionable costs. Phase-level measurements made targeted changes possible. A repeated larger-model test across distinct physical GPUs remains the next evidential step.

Related methodology: https://nodeox.com/blog/benchmark-mixed-gpu-training