Batched cooperative learning study
This experiment tests whether the quality-preserving shared-gradient method becomes useful training scale when both controls batch their computation efficiently. The completed results pass its quality and speed screen, with higher allocated GPU cost. It changes no network service, consensus rule, token issuance or native serving model.
The frozen plan compares one L40S against two L40S workers. Both train all 1,711,376,384 parameters of the same SmolLM2 seed for 128 updates, with a global batch of 256, shared AdamW, the same global target normalization and the same document schedule. There are 6,144 new generated training tasks and 2,048 public conversation-replay documents. Task tokens have weight eight, replay tokens weight one. Learning rate is 3e-5, with eight warmup updates. Maximum context remains 1,024.
The single-GPU control uses microbatches of sixteen; each distributed worker uses eight. Length sorting happens within each local batch. The distributed arm retains DDP gradient buffers and full-precision PowerSGD error feedback on GPU. PowerSGD uses rank 32, eight dense warmup updates and 120 compressed updates. Both arms use expandable CUDA allocation segments.
These choices follow training-only probes, whose receipts are hashed in the plan. Default allocation failed for single-worker microbatches of sixteen and distributed microbatches of eight despite reserved unused memory. Expandable segments let sixteen complete on one GPU. Twelve per distributed worker still exceeded capacity; eight completed on both. The successful short probes report median updates of 10.135 seconds on one GPU and 6.829 seconds on two workers. These measurements choose a candidate; the full-length comparison must establish its actual quality and execution cost.
A CPU test compares the new weighted batched DDP updates against the established per-document objective, including model parameters, Adam state, loss and gradient norm. Unequal sequence lengths, unequal weights concentrated on one rank, reversed assignment, prompt/padding masks and a partial final microbatch are exercised. It passes before training. Earlier paired CUDA probes establish complete numerical-state equivalence for the resident-buffer change on the original dense graph. They do not substitute for this experiment's fresh quality measurement.
All numerical sources and prepared input hashes must be committed before training. All two predetermined final candidates must be committed before any final-test scoring. The test contains 1,024 fresh generated cases, using a new seed and the same four public task families. The 128 retention probes are previously exposed. This remains one development screen, not broad assistant evaluation or evidence of model growth.
The compressed candidate must gain at least ten accuracy points over the seed, keep the approximate 99% retention upper bound below +0.02 nats, and meet the −3-point paired accuracy noninferiority margin against the stronger single-GPU control. Active and complete-process speedups must each reach 1.10. Report allocated GPU seconds, checkpoint time and transmitted bytes separately. Faster parallel execution need not reduce total GPU cost. This experiment contains no second dense distributed arm; do not present a measured dense-communication reduction for this new recipe.
The two existing temporary g6e.2xlarge hosts, in separate availability zones, are reused within the original $100 allocation bound and stop deadline. Training arms run sequentially, without competing GPU jobs or bulk artifact transfers. All failures and attempted configurations remain evidence. Final models are retained with hash-verified readback before the allocation is removed.
The implementation is study_batched_learning.py. It shares the original preparation and evaluation code, with distinct plan/input/selection commitments. Checkpoints contain model weights; complete compression-state recovery and permissionless verification remain separate work.