Accepted-model provider service trial
The frozen contract measures the accepted answering graph through native provider discovery, certificate-pinned streaming, concurrent customers, full numerical verification and replacement of failed owners. It changes no model weights and opens no new quality final. The operator owns the seven disposable hosts; this cannot satisfy the independent administration requirement in TODO item 4.
The graph has nine logical owners: three backbone partitions and six accepted expert tails, including the auxiliary selection/interpretation weights needed by the complete answering system. Two provider replicas and the numerical referee use the same seven physical hosts in separate processes. No physical host needs the full backbone. Replays execute anew, but they are not independently operated auditors merely because three native audit keys sign their results.
The targets are two concurrent customers, six ordinary requests and two injected losses, with three complete numerical replays per response. Warm generation p95 must be at most 90 seconds, first visible output at most 45 seconds, ordinary settlement at most 1,200 seconds and recovered settlement at most 1,500 seconds. All six ordinary requests count, including structured responses with no early token. The separate 600-second cold limit includes both paid warmups and their settlement. Six observations establish a bounded demonstration, not a population latency guarantee. Drafts remain unverified until native settlement.
Timing follows the frozen driver: first visible output and complete generation start at native reservation submission, after quoting and audit funding. Settlement times include that funding phase. Cold readiness starts when owned model restoration begins, after cloud/node/provider bootstrap, and includes both paid warmups and their settlement. Cloud provisioning and registration are additional startup time; all of their resources are included in the cost record.
The absolute allocation deadline is two hours. The final correction's $160 combined ceiling includes the prior failed setups, all compute and replays, disks, network traffic, public IPv4 and 90-day artifact retention. Price estimates are pre-tax upper estimates, not an AWS invoice or a NEURO/USD exchange rate. The experiment uses finite operator sponsorship; it does not demonstrate independent customer demand. The finite learning/service sponsorship also accounts for the three learning allocations, both existing controllers and deduplicated public retention. Its $1,000 aggregate cap does not rewrite any earlier experiment's cap or establish a perpetual serving subsidy.
Retention accounting
The serving trial's costs.json prices its final-lineage catalog and model files, with a separate evidence allowance. That inventory is not the entire history of training checkpoints. A separate S3 inventory at 17:09:27 UTC on September 19 counts 11,291 public research objects, 591,804,881,417 bytes, all in Standard storage. At the listed S3 rate of $0.023/GB-month, using decimal GB conservatively, 90 days costs approximately $40.83. The complete inventory passed public hash readback. This whole-pool estimate includes earlier failed research and therefore exceeds the accepted model's own retention requirement.
All 1,960 distinct objects checked from the latest continuation's initial catalog, training/comparison publication receipts, source/evidence archives and current serving inventory are present at the expected sizes. The listing checks presence and size; the original full-readback receipts establish content verification. Retention charges must be deduplicated when combining the pool estimate with a trial estimate. This is a storage snapshot, not a permanent funding guarantee; requests, delivery and new future objects cost separately.
First allocation: rejected during setup
Source 03fc78accc3a7dd0dc352c962bd8d7aeed9c6004 restored the declared model files and registered 18 providers across five availability zones. All nine numerical referees then refused initialization: the deployment preparation used the earlier baseline executor profile instead of resolving the final accepted graph's exact executor_root. Their numerical-profile commitments differed. No request was answered, no claim settled and no token was issued.
Application replay reproduces 329 headers and 54 accepted transactions, matching all seven saved full-node states. All seven disposable instances, volumes and the security group were retired. Protected instances retained their prior states. The compute upper estimate is $1.60; the complete conservative estimate is $17.23, including 90-day retained data and deliberately overcounted network traffic. It is preserved as a failed setup, not a serving result.
Public evidence (SHA-256 1e219e8957cf080e6ec659bc4005896b4ed1196bc8082c96f8a57ee0883d2da0) contains 60 allowlisted files, including the mismatched inputs, numerical errors, ledger export/replay, ownership-limited restoration and cost/retirement records. Keys and private native backups are excluded. Use the matching source archive to reproduce its application replay with scripts/replay_provider_preflight.py.
The corrected preparation resolves the executor by its committed content hash and checks graph/profile/source consistency locally before allocation or model download. Its focused regression includes an earlier-cohort numerical profile whose enclosing graph correctly hashes that wrong profile: even this internally rehashed mismatch is rejected before any transfer. The corrected allocation retains the original requests, generation, faults, replay count and performance thresholds. Its outcome must be recorded before counting serving evidence.
Second allocation: reference loaded, controller stopped
Source c359ad98543f74f402c3b2692371e7dde6188ddd passed the corrected profile checks. All nine numerical reference owners loaded their committed partitions across seven hosts in five availability zones. The controller then stopped before funding a customer request: it attempted to open customer wallets before creating them. No answer, inference claim, payment or issuance occurred.
Application replay reproduces 414 headers and 54 accepted transactions, matching all seven saved states. All disposable instances, volumes and the security group were retired, with protected instances unchanged. The compute upper estimate is $1.71; the standalone complete estimate is $17.33. Both failed allocations total $34.56 by the same conservative accounting. The second costs.json also includes a $30 prior-allocation reserve; that reserve is not another charge to add to the first allocation. The next frozen run reserves $45 for both prior attempts within the unchanged $100 aggregate ceiling.
Public evidence contains 61 allowlisted files, including the nine ready reports, exact inputs, failure, ledger replay, cost and retirement records. Its matching source archive reproduces application replay. Both archives passed full public hash readback.
The controller now creates and validates customer identities before allocation. A local signed-native-transition check also caught and repaired a separate SQLite thread-affinity error: the audit thread opens connections to the same durable signer journals. Those checks require three replay results before voting and reject an invalid result without any vote or payment. Their numerical reports are explicit fixtures; actual accepted-model replay is still required. The third allocation uses committed source 7d8c2a09b60739b2b027359f147a7dc72a5854f4, unchanged accepted weights and the original request and performance gates.
Third allocation: slow disk exposed a startup timeout
That allocation stopped before requests when small reference owners timed out waiting for a backbone owner. A process stack places the backbone in checkpoint SHA-256 reads, and a five-second I/O sample measures about 18 MB/s. Available memory remained above 11 GiB. Gloo's underlying group had a 300-second timeout; the 1,200-second monitored_barrier argument did not extend a non-root owner's underlying send timeout. This was an initialization failure, not a numerical disagreement or a completed serving result.
Public evidence and matching source preserve the error, stack, I/O sample, ledger and costs. Replay reproduces 590 headers and 54 transactions with all seven states matching, no inference claim and zero issuance. All seven disposable hosts, volumes and the security group were retired. The standalone upper estimate is $17.72; the three failed setups total $52.28, including conservatively repeated retention allowances.
The next deployment uses the AMI-managed local NVMe for model caches and keeps wallets and native stores on EBS. It requires the separate mounted filesystem before dependency or model downloads, and validates every model byte as before. AWS documents this instance-store mount. The reference's startup/control group now has the same 1,200-second timeout as its loading barrier; inference subgroups retain 300 seconds. A real two-process Gloo probe with scaled deadlines reproduces the old failure and passes with an aligned timeout. The outer 600-second cold-readiness gate, request bounds and two-hour retirement deadline stay fixed. The combined $100 trial ceiling now reserves $65 for prior setups; it is not an additional $65 expenditure.
Fourth allocation: stale policy commitment
Source a9803bddae7abc06b243dc5e81f16aa47178b0b2 passed all seven separate instance-store checks and registered 18 providers. Asset preparation then rejected the reused policy before loading model tensors. The executor source check had passed, but the learned policy additionally committed the earlier scripts/run_native_expert_service.py hash. That script's timeout had changed. The controller did not retain the remote command's stderr; the evidence records the separate CPU reproduction of the policy rejection and both script hashes.
Evidence and matching source preserve the failed setup. Replay matches 191 headers, 54 transactions and all seven states. No inference claim, payment or issuance occurred. All disposable hosts, volumes and the security group were retired. This attempt's standalone upper estimate is $11.30, bringing the four setups to $63.58.
4a2630fc1b9b5acb3e72be87280e495e38f5427d checks the full content-addressed serving policy, including learned/planned script commitments, before allocation. Its six controller tests include an internally hash-consistent graph with a stale script commitment. It also preserves bounded restoration errors. Regenerated metadata passes executor, complete-policy and native-genesis checks, with no weight or answering-rule change. The new public policy 381cf3a38ffd71f10279f8d0d2573c02ae2f886980771ca2270abd11586b9d7b passed full hash readback. The next allocation reserves $70 for prior attempts inside the unchanged $100 aggregate ceiling; its outcome follows below.
Fifth allocation: ordinary serving passes, recovery fails
Source 4a2630fc1b9b5acb3e72be87280e495e38f5427d restored the nine reference owners in 100.53 seconds. Cold readiness, including both paid warmups and their complete verification/settlement, took 311.84 seconds. All six ordinary requests settled after three actual numerical replays each, with zero issuance.
| Request | First visible output | Generated response | Settlement |
|---|---|---|---|
| Client package | 23.78 s | 26.86 s | 97.21 s |
| Retained directory | 31.10 s | 34.51 s | 160.19 s |
| General explanation | 23.49 s | 38.58 s | 169.88 s |
| Constrained arithmetic | 29.57 s | 31.48 s | 105.61 s |
| Multi-turn recall | 26.82 s | 28.67 s | 100.97 s |
| Combined question | 44.87 s | 45.27 s | 168.64 s |
The answers include neuroshard-ai, Fenn Varden's city Sofia, 43, and violet kettle. The combined response includes both retained facts; the general response explains Rayleigh scattering in two sentences. Generation clocks start at reservation; settlement clocks additionally include customer funding and audit acceptance. The six-sample first-visible p95 is narrowly inside 45 seconds. Both warmups also settled; all 24 complete replays matched. Every completed request has one provider payment and exact unused execution/audit refunds.
Coordinator loss did not recover. The replacement registered a fresh key and endpoint and restored 7.19 GB of its committed files, but all assigned owners then persisted execution errors. A five-process CPU reproduction identifies incompatible startup collectives: fresh owners declare loaded graph commitments, whereas surviving owners with cached weights sent a different assignment binding. The run was stopped through its normal evidence/retirement path once this became irrecoverable. The backbone-loss case was not started. This is a failed complete trial with a passing ordinary-serving subset, not a recovery pass.
Application replay matches 2,124 headers, 251 accepted transactions and all seven saved states, including the eight settled claims. The unfinished coordinator request remains in the preserved stopped chain; its escrow is not reported as a payment or completed refund. All seven instances, volumes and the security group were retired at 18:19 UTC, with protected hosts unchanged. The standalone conservative cost is $22.62, including $4.73 compute, 82.88 GB of measured aggregate interface traffic priced conservatively at $12.43, disk provisioning, public IPv4 and 90-day retention. The five allocations total $86.20 under this accounting. costs.json also reports $92.62 when using its frozen $70 reserve for prior attempts; that reserve must not be added again.
Public evidence and matching source passed full public hash readback. The 160 allowlisted evidence files include replies, token-time events, all numerical replays, payments/refunds, native replay, fresh restoration, failure diagnosis, client onboarding, costs and retirement. Private keys, native backups and client/node homes are excluded.
The corrective implementation uses the same epoch-bound graph handshake for fresh and cached owners. The complete answering policy has a similar collective initialization requirement, so owners negotiate service-cache readiness and reconstruct policy wrappers together on a partial miss, retaining model caches. Real HTTPS regressions reproduce the old failure and check fresh coordinator and backbone owners alongside cached survivors against fixed-group execution. Twelve focused tests pass. The next allocation explicitly raises the cumulative ceiling to $125, reserving $95 for earlier attempts; the fifth run retains its original $100 ceiling and failed result. Questions, weights, replay count, latency thresholds and recovery requirements remain unchanged.
A separate read-only onboarding probe installed the exact frozen source in a clean client environment, joined a new non-voting node through the public peer address, matched block 2,095 and obtained a complete hosted-chat quote. PyTorch, Transformers and NumPy were absent from that client environment. The first probe failed because its script parsed human-readable wallet creation output as JSON; the corrected probe passed and stopped its temporary node. No GPU, vote or paid request was added. This still represents one administrator, not independent public participation.
The provider guide and chat guide describe the implemented interfaces. The trial endpoints are disposable; they are not a permanent public service or a replacement for the public 0.4.0 network.
Sixth allocation: both recoveries pass; combined-output latency fails
Frozen source 8ac29c7bbde61434c1a5cfb23881b78227dab23a settled both warmups and all six ordinary requests, with three complete replays per reply and zero issuance. Cold readiness was 302.79 seconds. The combined question first became visible at 46.79 seconds, missing the unchanged 45-second limit; its complete generated reply arrived at 47.18 seconds. It remains a failed complete performance trial despite both passing recovery measurements.
The fresh coordinator successfully restored only its committed files while surviving owners retained their model caches. Three full replays matched, and the recovered request settled exactly once in 624.12 seconds, below the 1,500-second limit. Replacing a killed backbone owner then settled in 644.87 seconds, also after three complete matching replays. Each replacement uses a fresh key, endpoint and empty model cache; surviving owners keep their resident weights. In total, ten requests settled once after thirty real complete replays, with no inference issuance.
| Ordinary request | First visible output | Complete generation | Settled, including funding |
|---|---|---|---|
| Client package | 26.34 s | 29.99 s | 100.38 s |
| Retained directory | 32.93 s | 36.38 s | 161.91 s |
| General explanation | 23.90 s | 39.18 s | 170.12 s |
| Constrained arithmetic | 30.41 s | 32.11 s | 105.75 s |
| Multi-turn recall | 29.39 s | 31.26 s | 106.21 s |
| Combined question | 46.79 s — failed | 47.18 s | 175.65 s |
Application replay reproduces 2,534 headers, 294 accepted transactions and all seven saved states, with ten settled claims and zero issuance. Every disposable instance, volume and the security group was retired at 19:06 UTC; protected hosts retained their previous states. The standalone conservative cost is $24.86, including $5.43 compute and 92.63 GB of measured aggregate interface traffic. The six attempts total $111.06. The sixth frozen cost gate also passes at $119.86 including its $95 prior-attempt reserve, below $125.
Public evidence and matching source passed full public hash readback. The 159 allowlisted files include every reply, stream observation, full numerical replay, payment/refund, replacement receipt, application replay, cost and retirement record. No private keys or node backups are included.
The correction emits a completed visible answer part while the next part runs. It does not change routing, generated tokens or final rendering, expose planner or composer inputs, or count question-only previews as progress. A later invalid answer retracts the draft. All 82 focused checks pass, including real provider HTTPS execution and complete-answer replay. The next freeze keeps every request, model, latency threshold and fault unchanged, reserves $120 for earlier attempts inside a $160 cumulative cap, and forbids allocation before both preceding recoveries pass, resources are retired and complete prior costs fit that reserve. Those preconditions passed; source 95a19b87026bce91a1bc567da3c0e1988db3b011 began its corrective allocation at 19:07 UTC with a 21:07 UTC hard deadline.
Seventh allocation: complete serving and recovery pass
Source 95a19b87026bce91a1bc567da3c0e1988db3b011 passes the complete frozen contract. Model/reference restoration took 101.92 seconds; cold readiness, including both paid warmups and full settlement, took 307.51 seconds against 600. Two concurrent customers used the committed native provider reservations.
| Ordinary request | First visible output | Complete generation | Settled, including funding |
|---|---|---|---|
| Client package | 26.41 s | 29.37 s | 101.77 s |
| Retained directory | 35.77 s | 39.86 s | 163.98 s |
| General explanation | 23.60 s | 38.45 s | 169.38 s |
| Constrained arithmetic | 29.79 s | 31.68 s | 103.45 s |
| Multi-turn recall | 28.99 s | 31.04 s | 102.61 s |
| Combined question | 44.63 s | 49.32 s | 169.59 s |
All six meet the unchanged 45-second first-visible and 90-second generation gates; combined-answer visibility remains a narrow pass. Its first visible content includes the completed Sofia answer while the protocol answer is still being generated. The final reply includes both answers. The other replies include neuroshard-ai, 43, violet kettle, and an explanation of Rayleigh scattering. The general explanation reaches its 64-token bound mid-sentence; passing timing is not a new broad-quality result. Every full reply and draft is available in the evidence. Six observations do not establish a population p95.
A killed coordinator recovered with a fresh provider key and endpoint, settling once in 622.34 seconds. A separately killed required backbone owner recovered and settled once in 641.97 seconds. Both are below the unchanged 1,500-second limit. Surviving owners retained their cached model state. These are process-loss faults and replacement on surviving physical hosts; they are not an availability soak or a new physical-host-loss demonstration. The final collector includes the backbone replacement's start and result records, but its separate asset-download receipt was not captured before retirement. The preceding trial preserves both replacement asset receipts. We do not infer a missing measurement.
All ten requests, including warmups and recoveries, received three actual complete numerical replays. Exact final text, every neural prompt/output token record and metered neural work match the sixth trial for all ten requests. No model weights, routing rules or quality data changed. Each request settled once; inference issued zero NEURO. Across the ten requests, native accounting records 245 execution atoms paid and 9,275 refunded; 9,360 provider atoms paid; and 73,500,000 audit atoms paid with 3,734,500,000 refunded. Transaction fees are separate. These are research balances, with no assumed cash exchange rate.
Application replay reproduces 2,519 headers and 294 accepted transactions, matching all seven saved full-node states. Its final state root is 93857d3476a24695ca27e4c5f0ba7089d69623ce5457316e962540b68ca98bfb. This replays application transitions against recorded states; it does not independently validate CometBFT commit signatures or prove separate ownership.
All seven disposable instances, their volumes and the security group were retired at 19:53 UTC. The three protected instances retained their previous states. This run's standalone conservative cost is $24.81, including $5.37 compute and 92.70 GB of aggregate interface traffic. Its frozen cumulative gate passes at $144.81 against $160, including the $120 prior-attempt reserve. The sum of all seven standalone estimates is $135.86; this sum repeats their retention allowances and is therefore not the deduplicated whole-project cost. Complete finite-sponsorship accounting reconciles the learning and serving work, controllers and entire public retention pool at $792.54 against $1,000.
Public evidence and matching source passed full public SHA-256 readback. The 162 allowlisted files include the frozen inputs, all stream/answer observations, thirty replays, settlement, faults, application ledger, costs and resource retirement. They exclude private keys, credentials and native backups. The source's scripts/replay_provider_preflight.py --home EVIDENCE/trial reproduces the application replay from the included public ledger export.
This closes the operated serving/recovery experiment. Item 5 meets its bounded finite-funding criterion. Items 4 and 6 retain their public-operation requirements: independent administration, admission/selection abuse resistance, a declared availability soak and a persistent public deployment. The trial ran one declared A10G numerical profile under one administrator; its endpoints are now retired. The public 0.4.0 chain and PyPI package have not been upgraded by this trial.