Architecture Overview

NeuroShard's system architecture consists of three main layers: Model, Network, and Economics.

High-Level Architecture

Core Components

DynamicNeuroLLM

The neural network model, distributed across nodes.

Key Features:

• Transformer architecture with RMSNorm, RoPE, GQA, SwiGLU
• Dynamic depth and width based on network capacity
• Sharded across nodes via Layer Pool
• Checkpoint-based persistence

class DynamicNeuroLLM:
    architecture: ModelArchitecture  # Dynamic config
    my_layers: Dict[int, nn.Module]  # Only layers we hold
    embedding: nn.Embedding          # If Driver
    lm_head: nn.Linear              # If Validator

Layer Pool

Manages layer distribution across the network.

Key Features:

• DHT-based layer registry
• Automatic layer assignment based on memory
• Redundancy (MIN_REPLICAS = 2 per layer)
• Heartbeat-based liveness detection

class DynamicLayerPool:
    layer_assignments: Dict[int, List[LayerAssignment]]
    current_architecture: ModelArchitecture
    node_capacities: Dict[str, float]

Training Coordinator

Manages distributed training via DiLoCo protocol.

Key Features:

• Inner loop: 500 local training steps
• Outer loop: Pseudo-gradient synchronization
• Nesterov momentum optimizer
• Robust gradient aggregation

class DiLoCoTrainer:
    inner_steps: int = 500
    outer_optimizer: OuterOptimizer
    initial_weights: Dict[str, Tensor]

P2P Manager

Handles peer-to-peer communication.

Key Features:

• Tracker-based bootstrap
• DHT for peer discovery
• gRPC for direct communication
• NAT traversal

class P2PManager:
    tracker_url: str
    dht: DHTProtocol
    grpc_server: NeuroShardService

Swarm Router

Intelligent routing for fault tolerance.

Key Features:

• Multipath routing with failover
• Capacity-weighted peer selection
• Heartbeat-based liveness
• 200ms failover timeout

class SwarmRouter:
    layer_peers: Dict[int, List[PeerInfo]]
    failover_timeout: float = 0.2

NEURO Ledger

Token economics and accounting.

Key Features:

• PoNW proof verification
• Reward calculation
• Stake management
• Transaction history
• Fee burn mechanism

class NEUROLedger:
    db_path: str  # SQLite
    crypto: NodeCrypto  # ECDSA
    inference_market: InferenceMarket

Data Flow

Inference Flow

Training Flow

Technology Stack

Component	Technology
Model	PyTorch
API	FastAPI
RPC	gRPC + protobuf
Database	SQLite
Crypto	ECDSA (secp256k1)
P2P	Custom DHT
UI	Web Dashboard (HTML/JS)

Scaling Properties

Horizontal Scaling

• More nodes → more layers → larger model
• Linear scaling of compute capacity
• Automatic load distribution

Architecture Scaling

Nodes	Memory	Architecture	Params
10	40GB	16L × 1024H	350M
100	800GB	32L × 3072H	9.2B
1000	8TB	64L × 7168H	123B

Next Steps

• NeuroLLM Model — Model architecture
• Dynamic Scaling — How scaling works
• DiLoCo Protocol — Training protocol
• P2P Network — Network layer