blueprints / frontier-llm@2026-11
The Frontier LLM
A composite blueprint of the open-weight frontier: hybrid or compressed attention, ultra-sparse MoE, widened residual streams, FP4-native training and serving.
Select a component to read how it works. Numbers mark open community proposals.
Every component
- Byte-level BPE tokenizer, embedding & LM head
Lossless byte-level BPE with 128K–262K entries, feeding an untied input embedding and output head.
- Native multimodal early fusion
Images, video and audio become embeddings in the same token stream, in the data from early pretraining.
- Engram conditional memory (hashed n-gram lookup)
Huge n-gram embedding tables read by O(1) hashing and gated into the residual stream: a second sparsity axis.
- Multi-token prediction (MTP) & speculative drafting
Extra sequential heads predict t+2…t+k in training; at inference they draft tokens for self-speculation.
- Manifold-Constrained Hyper-Connections (mHC)
Widen the residual into n = 4 parallel streams mixed by a Sinkhorn-projected doubly-stochastic matrix.
- RMSNorm (pre-norm placement)
Scale-only normalization before every sublayer; cheaper than LayerNorm and the universal default.
- QK-Norm
Per-head RMSNorm on queries and keys before the dot product: caps logit growth, enables high LRs and Muon.
- Partial RoPE → NoPE (position encoding)
Rotate only a slice of each head (e.g. 64 dims), or drop RoPE entirely when other layers carry position.
- Hybrid linear attention (Gated DeltaNet / Kimi Delta Attention)
A fixed-size state updated by a gated delta rule in ~3 of every 4 layers; full attention in the rest.
- Multi-head Latent Attention (MLA)
Cache one low-rank latent per token (512 + 64 dims) instead of per-head K/V; heads are absorbed at decode.
- Sparse & compressed attention (DSA → CSA / HCA)
A cheap FP8/FP4 'lightning indexer' picks top-k tokens or compressed blocks; softmax attends only to those.
- Sliding-window / global interleaving
Most layers see only the last 128–1,024 tokens; every Nth layer (or a compressed branch) sees everything.
- Gated attention output & learnable sinks
A per-head sigmoid gate on the attention output, or a learnable sink logit, lets heads 'do nothing' cleanly.
- Sigmoid-family router with auxiliary-loss-free balancing
Independent per-expert affinities (sigmoid or √softplus); a bias steers top-k selection to balance load.
- Fine-grained MoE with shared experts (DeepSeekMoE / LatentMoE)
Hundreds of small SwiGLU experts (top-6 to 22 active) plus 1–2 always-on shared experts; ~2–4% params/token.
- Muon optimizer (MuonClip, per-head Muon)
Orthogonalized momentum updates for matrix weights via Newton–Schulz; AdamW kept for embeddings, head, norms.
- FP8 training & FP4 quantization-aware training
Block-scaled FP8 GEMMs for pretraining; MXFP4/NVFP4 expert weights via QAT or native 4-bit pretraining.
- Staged long-context extension (YaRN → native 1M)
Pretrain short, then extend in stages (4K → 16K → 64K → 1M) with RoPE rescaling or position-free layers.
- RL with verifiable rewards & on-policy distillation
GRPO-family RL on verifiable tasks and agent environments, then specialists merged by on-policy distillation.
- KV-cache compression & tiered caching
Bytes per token is the 2026 design metric: low-bit KV, cross-layer sharing and SSD-tiered prefix caches.