Adaptive shards and native GPU settlement, 2026-09-14
A 1,711,376,384-parameter model changed from two GPU owners to three while preserving every learned weight and Adam moment. Replaying the next two updates under either layout produced the same learned-state root. A fourth GPU replayed all three partitions sequentially, and an isolated native chain rejected a forged checkpoint and paid for the genuine updates. These are operated experiments on matching A10G hardware, with one infrastructure owner.
Initial learning passed its frozen gate. Both later continuation candidates failed generated-answer retention, and the larger model failed the required gain over the smaller control. No serving checkpoint was promoted.
The frozen plan, input/source commitments, learning implementation, and native verification rules define separate computation, quality and settlement claims. Numerical sources were frozen at ad7d2ca before training. The public 0.4.0 chain and its serving model were not changed.
Ownership and exact continuation
| Layout | Parameters held by each worker | Layer boundaries |
|---|---|---|
| Two workers | 838,907,904; 872,468,480 | 0, 11, 24 |
| Three workers | 503,343,104; 604,016,640; 604,016,640 | 0, 6, 15, 24 |
Each owner holds its student partition, local Adam state and a frozen reference partition. The reference supplies a KL regularizer on previously trained replay examples. No training worker constructs the full student or reference model. The first worker owns the tied embedding/output head and final norm.
Redistribution at update 32 took 54.32 seconds, including transfer, verification, durable staging and agreement on the new commit. Old learned tensors, optimizer groups and parameter ages were unchanged. The learned-state root after updates 33–34 was exactly bfd1b7eeca535ce6d82e98c674a2e2e33fde2f7a3c55fc5989b09810b8caf217 under both layouts. Full checkpoint roots differ because they also commit ownership, parent and process RNG metadata. Excluding RNG from the learned root is restricted to this dropout-free numerical profile; its updates consume no random draws.
The operated controller restarts the process group at a committed cursor. This demonstrates redistribution and continued training, rather than automatic public discovery or uninterrupted membership changes. The separate host-replacement experiment exercised physical failure recovery; that failure was not reinjected in this redistribution trial.
Complete replay and its cost
Three complete replay executions covered every partition of updates 33–34. All final parameter and Adam tensors matched. A further execution on a fourth 24 GB GPU processed the three partitions sequentially: at most 604,016,640 student parameters were resident, with a peak CUDA allocation of 13,917,486,080 bytes including the local reference, optimizer and intermediates.
That one-GPU execution took 595.12 seconds end to end. The three reported replay-and-write intervals total 138.16 seconds; they exclude initial checkpoint loading. The complete three-worker training process for this captured two-update window took 115.83 seconds. These differently provisioned paths are not an equal-hardware performance comparison. Loading, hashing and writing many tensor files materially affected the single auditor's wall time.
The closed transcript contains 3,995,785,128 unique tensor bytes, plus metadata, for just two updates. Its commitment is 49bfba54fcf2206de3223cc6c00cef7b99141b45c4ec96d14bf164361ec68274. Each send matches a receive, collective outputs follow their inputs, and actual replay checks local computation and outgoing values. Metadata closure alone is insufficient. A deliberately altered output tensor commitment, with recomputed checkpoint and transcript bindings, failed numerical replay. CPU adversarial tests also changed matching send/receive values together; neural replay rejected that self-consistent forged witness.
Verification now starts from a recent complete checkpoint and fits one shard's GPU memory. It still repeats all computation and requires substantial artifact bandwidth. Multiple audit signers were controlled by one operator; these runs do not demonstrate independent ownership or economical permissionless auditing.
A verifier without a cache needs approximately 31.38 GB of unique artifacts for this window: 20.54 GB of input weights and Adam state, 6.85 GB of frozen teacher weights, and 4.00 GB of boundary witnesses. This inventory excludes the runtime, tokenizer, prepared data and small manifests. Output weights are regenerated by replay. Keeping a verifier synchronized can reuse prior state; streaming partitions reduces GPU memory without eliminating these downloads.
Native settlement and recovery
Four local CometBFT validators ran the optional native replay-quorum profile. They used the real GPU checkpoint/transcript commitments and locally verified results from the completed numerical replay. The trial reused these results; it did not rerun the GPUs for every ledger transaction. Cached reports were checked against their exact input, output and recorded computation.
A wrong-output claim received one conflicting positive report and three negative reports. The native quorum rejected it without issuing training rewards. The genuine window then settled while the fourth validator and its application were offline. The ledger issued 2,000,000 atoms, or 2 experimental NEURO, for updates 33 and 34. Six paid audit services received 3,600,000 atoms from sponsor escrow across the rejected and accepted claims. Duplicate/stale work failed.
After the fourth validator restarted, all four agreed at height 68 on app hash 2470F111FE35789C79DE5F196AF4101CC26BDAF1723DF533568F7FE738B3A498. All four stored states satisfied supply conservation. The chain ID was neuroshard-sharded-replay-1271401c9417, with genesis SHA-256 9197ff65006f7ac0a036eb695b350ec74d6e8212f3e59745ff43919d7da36517. All temporary native processes were stopped after the trial.
The trial exposed and corrected an audit-window liveness bug: a quorum of opaque commitments does not imply a quorum for either eventual verdict. Early commitment closure now requires every bonded participant; otherwise the declared commitment deadline applies. Reveal closure still uses an actual consistent verdict quorum. The corrected path passed the real four-validator trial and regression tests.
A repeat with the hardened native sources also passed the forged/genuine, duplicate and validator-recovery checks. Its source hash was 9543af05d6c71fabf92e64ac2f98b448d9ddd1563a16a865aef7764eb409ba41. It reached matching height-68 app hash 228585FAC8B5751D21FDD97F31AE235063976D2FF34E83DB56033BC7AA3C743A.
Fresh GPU replay after an on-chain claim
A third, separate chain ran three actual audit daemons. Each observed the claim and then launched new GPU replays of all three partitions. No previous verdict supplied its result. Nine partition executions completed on the three existing GPUs, serialized per host to bound memory. All learned tensors, Adam state and complete output-manifest commitments matched. The auditors completed in 98.81, 168.81 and 232.96 seconds, including their different waits for GPUs.
With the fourth validator offline, the native chain issued 2,000,000 atoms for the two updates and paid 1,800,000 escrow atoms for three audit services. After the fourth validator rejoined, all four agreed at height 204 on app hash 022813C8AAE38DDC7A864F10366EA069781A75230FA00573F4CE4C14532A576A. Each stored state satisfied supply conservation. The chain ID was neuroshard-live-gpu-replay-6e5c64d915f0, with RPC-genesis commitment 3093347f7d46da4f3200ba25e3306dca76fcf3b18817bc40fb52f0229d418494. All temporary applications, validators and audit daemons were stopped afterward. These are separate isolated ledgers; their rewards must not be added together as a public-network balance.
Review also found a verifier gap: the frozen numerical oracle checked learned tensors but did not compare its full regenerated manifest with the output's shard commitment. The operator backend now rejects a mismatching manifest and binds recovery caches to its own source hash. Two reproducing CPU tests failed before this fix; the five controller/recovery tests passed afterward. All nine earlier archived GPU manifests were separately checked, and the fresh live backend checked every regenerated manifest before attesting. The frozen GPU numerical sources and learning recipe were unchanged.
The security assumption is native bonded voting weight, not proof that signatures represent different people. A threshold of colluding signers can approve a forgery; an explicit test retains that counterexample. This bridge enables one frozen job in a dedicated genesis. It does not yet activate future datasets, settle growth, or connect these 1.7B checkpoints to public paid inference.
Learning decision
Phase A passed its frozen learning gate. The predetermined update-128 checkpoint was committed in the selection record at 49f3633, before final evaluation. Its checkpoint root is 094138fb6e3e2a8962f8455b0bf81de3f2fbe82029222df76aa71e2e22e14d49.
| Held-out measurement | Seed | Update 128 | Paired change | One-sided 95% upper bound |
|---|---|---|---|---|
| Task response loss, 256 cases | 0.149855 | 0.141845 | −0.008010 | −0.003700 |
| Conversation retention loss, 128 cases | 0.552264 | 0.552732 | +0.000469 | +0.001097 |
| Correct generated answers, 64 cases | 47 | 49 | +2 | — |
The loss upper bound is below −0.001 and the retention upper bound is below +0.02 nats, as required. There were two correctness wins and no losses: one sorting case and one invoice-total case. The generated-answer gate requires no decrease; two wins alone do not establish a statistically significant accuracy gain (the paired one-sided sign-test value is 0.25). Arithmetic remains weak: only 1 of 16 generated invoice totals was correct.
Every evaluation rank agreed on losses and generated tokens. A separate artifact check reconstructed tokenization, target masks, decoded token IDs and answer correctness. The test covers fresh cases of four public generated task families and a new pinned public conversation range; it does not establish broad assistant improvement or unknown pretraining-data independence.
Continued learning and growth: rejected candidates
Both predetermined continuation runs completed, but neither passed the full phase-B gate. Endpoint identities were committed at 12d3518 in the phase-B selection before final evaluation. Both runs used the same 128 updates, new examples, trained replay and frozen phase-A reference.
| Candidate | Parameters | Three-worker training time | Checkpoint |
|---|---|---|---|
| Fixed depth | 1,711,376,384 | 2,699.35 s | d1901aed86149907ab6482a6c66345c85e5e4ee308eb12b215eb97c8451cb6dd |
| Two additional blocks | 1,845,602,304 | 2,845.43 s | fd1e67bb48dcbb6291ebb4c2f8255d897cb46af10f4b94e28a0f01b83902155b |
| Against phase A | Fixed-depth change / 95% upper bound | Grown change / 95% upper bound |
|---|---|---|
| New-cohort response loss, 256 cases | −0.004681 / −0.001038 | −0.005797 / −0.002146 |
| First-cohort response loss, 256 cases | −0.004834 / −0.001422 | −0.005145 / −0.001610 |
| Conversation retention loss, 128 cases | +0.001225 / +0.001798 | +0.001398 / +0.001962 |
| Correct first-cohort answers, 64 cases | 49 → 48 | 49 → 48 |
| Correct new-cohort answers, 64 cases | 46 → 45 | 46 → 45 |
Both loss and conversation-retention criteria passed. Generated correctness failed: each candidate lost the one correct invoice total in each cohort. For example, {"total":515} became {"total":525}, and {"total":445} became {"total":440}. Each final candidate answered 0/16 invoice-total cases correctly in each cohort. Lower teacher-forced response loss did not preserve those greedy answers. The criteria were not relaxed after seeing this.
The larger model's new-cohort loss was 0.001116 nats lower than the fixed-depth control, but the upper confidence bound was −0.000270, short of the required value below −0.001. Generated answers did not improve. Useful growth also failed its separate comparison. This is a same-update/data comparison; the larger model consumed more compute, not an equal-compute win.
The growth mechanism itself worked. Identity insertion added 134,225,920 parameters while preserving every old tensor and Adam moment. The larger layout held 637,569,024; 604,016,640; and 604,016,640 student parameters on the three workers, with maximum measured CUDA allocation 16,024,183,808 bytes. Before continuation, all development losses and the final identity check's 256 task losses, 128 retention losses and 64 generated token sequences were bit exact against phase A on every rank. The identity check is bound in its separate selection; it does not select another trained candidate.
Three workers supplied the declared layout and headroom. The experiment's 900M-parameter worker cap would reject a two-worker layout of the larger model; this is a profile limit, not a measurement that two physical GPUs cannot fit it.
Phase A remains the last candidate passing this experiment's learning gate. No candidate was promoted to public serving. These results establish working sharded continuation and capacity growth, but not accepted continual improvement or growth that improves the product. The next learning experiment needs a precommitted objective and fresh evaluation that test actual answer reliability; the exposed cases above cannot serve as its independent final test. The continued-learning contract is that next experiment.
Reproduction, validation and resources
The machine-readable result contains every gate decision, checkpoint identities, runtime measurements, native trial commitments, artifact hashes and final resource receipts. 395 tests passed after the manifest-verifier fix. Wheel and source-distribution builds, content checks and metadata checks passed; no package was released.
Thirteen checkpoint/transcript inventories were preserved with complete SHA-256-verified read-back. The evidence archive contains frozen inputs, selection records, numerical sources, operator-verifier sources, complete evaluations, raw logs, public native-chain summaries and checkpoint receipts. Signing keys, credentials, provisioning-account metadata and Git history are excluded. The result record identifies the immutable archive and its checksum. Large weights, optimizer states and witnesses are separate content-addressed S3 objects referenced by the inventories. This is operator-access storage, not a public availability network or an anonymously downloadable model release.
All four temporary GPU instances, their root volumes and the experiment's security group were confirmed deleted. Both permanent CPU network hosts remained running. GPU compute through confirmed termination was estimated at $12.19. Charging all measured outbound traffic conservatively at $0.02/GB and adding $10 for other costs gives a $43.39 planning estimate, below the $100 study budget. This is not an AWS invoice; retained S3 storage continues to accrue.