Skip to content

Frontier models, as diffs

Each model is drawn against the November 2026 LLM blueprint: which components it keeps and how it configures them, what it does differently, and what it leaves out. Every profile cites the model's own report, config or code.

Compare two models side by side

DeepSeek, September 2026

DeepSeek-V4.1-Flash

552B + 196B Engram total, 16B (8B in prefill) active, 1M tokens

Causal encoder-decoder MoE: decoder reads encoder-projected KV, CSA2 shares KV across layers, FP4 KV at 890 B/token, Engram, DSpark.

Of 27 blueprint components: 23 same, 3 different, 1 not used.

Thinking Machines Lab, July 2026

Inkling

975B total, 41B active, 1M tokens (1,048,576)

5:1 sliding-window/global GQA with a learned relative-position bias instead of RoPE, short causal convs, encoder-free image and audio.

Of 27 blueprint components: 18 same, 5 different, 3 not used, 1 not disclosed.

Moonshot AI, July 2026

Kimi K3

2.8T total, 104B active, 1M tokens (1,048,576)

93 layers of 3:1 KDA / gated MLA with no positional encoding, Block Attention Residuals and a 16-of-896 latent MoE in MXFP4.

Of 27 blueprint components: 19 same, 5 different, 3 not used.

NVIDIA, June 2026

Nemotron 3 Ultra

550B total, 55B active, 1M tokens

Mamba-2 does most of the token mixing, 12 NoPE GQA layers anchor it, and LatentMoE routes 22 of 512 experts in a 2,048-dim latent.

Of 27 blueprint components: 16 same, 4 different, 7 not used.

DeepSeek, April 2026

DeepSeek-V4-Pro

1.6T total, 49B active, 1M tokens

Attention over a compressed sequence: CSA (4→1, top-1,024) and HCA (128→1) layers with a 128-token window, 4-stream mHC, Muon.

Of 27 blueprint components: 22 same, 1 different, 3 not used, 1 not disclosed.

Google DeepMind, April 2026

Gemma 4 31B

30.7B, 256K tokens

Dense 5:1 local/global stack: 1,024-token windows, global layers with K = V and p-RoPE, sandwich norms, tied 262K vocabulary.

Of 27 blueprint components: 10 same, 7 different, 6 not used, 4 not disclosed. Adds 2.

MiniMax, March 2026

MiniMax-M2.7

229.9B total, 9.8B active, 204,800 tokens (M2 to M2.5: 196,608)

Full GQA attention in all 62 layers after hybrid-attention ablations; 256-expert top-8 sigmoid MoE with 9.8B active; 3 MTP modules.

Of 27 blueprint components: 14 same, 4 different, 7 not used, 2 not disclosed.

Z.ai (Zhipu AI), February 2026

GLM-5

744B total, 40B active, 202,752 tokens (GLM-5.2: 1,048,576)

MLA with DeepSeek Sparse Attention in every layer, 256-expert MoE, one MTP layer reused for 3 steps; GLM-5.2 adds IndexShare and 1M context.

Of 27 blueprint components: 19 same, 3 different, 5 not used.

Alibaba Qwen, February 2026

Qwen3.5-397B-A17B

397B total, 17B active, 262K tokens (1.01M with YaRN)

15 × (3 Gated DeltaNet + 1 gated GQA) layers over 512 small experts: only 15 of 60 layers keep a KV cache.

Of 27 blueprint components: 19 same, 3 different, 3 not used, 2 not disclosed.

OpenAI, August 2025

gpt-oss-120b

117B total, 5.1B active, 131K tokens

A deliberately plain MoE: alternating 128-token banded and dense GQA layers, attention sinks, MXFP4 experts.

Of 27 blueprint components: 12 same, 4 different, 9 not used, 2 not disclosed.

Component by component

A dot means the model uses the blueprint's design (hover for its configuration). Named cells are where it does something else. A dash means it doesn't use the component at all; a question mark means its authors haven't said.

Blueprint componentDeepSeek-V4.1-FlashSeptember 2026InklingJuly 2026Kimi K3July 2026Nemotron 3 UltraJune 2026DeepSeek-V4-ProApril 2026Gemma 4 31BApril 2026MiniMax-M2.7March 2026GLM-5February 2026Qwen3.5-397B-A17BFebruary 2026gpt-oss-120bAugust 2025
Input & embedding
Byte-level BPE tokenizerSentencePiece BPE with byte fallback
Vision / audio encoder—————
Token embedding
Engram n-gram memory—————————
Decoder block
Residual streamsSingle pre-norm residual streamBlock Attention Residuals (AttnRes)Single residual stream, one sublayer per layerSingle residual stream with per-layer scalarsSingle residual streamSingle residual streamSingle residual streamSingle residual stream
RMSNorm
QK-NormQK-Clip and normalized latents (no per-head QK-Norm)—Layer-wide QK RMSNormRMSNorm on MLA latents (no per-head QK-Norm)—
Partial RoPE / NoPELearned relative-position bias (no RoPE)RoPE by layer type: full RoPE local, p-RoPE globalFull RoPE with YaRN scaling
Gated DeltaNet / KDA——Mamba-2 selective state-space layers—————
Latent attention (MLA)CED decoder attention (CSA2 over encoder-projected KV)Global GQA with log-length scalingGlobal GQA attention without positional encodingHeavily Compressed Attention (HCA)Global GQA with keys reused as valuesFull grouped-query attention in all 62 layersGated grouped-query attentionDense grouped-query attention
Sparse / compressed attnCSA2 encoder layers (2× compression, Full + Reuse)———————
Sliding-window attention—————
Output gate + sinkShort causal convolutions (SConv)————
RMSNorm
MoE routerSigmoid router with shared-expert sinkSigmoid router with Quantile BalancingDense GeGLU feed-forwardSoftmax top-k routerSoftmax top-k router
Routed expertsStable LatentMoE routed experts (SiTU-GLU)LatentMoE routed experts (ReLU², 2,048-dim latent)—
Shared expert———
Output heads
Final RMSNorm
LM headTied LM head with logit soft-capping
MTP module(s)DSpark drafter (trained after pretraining)MTP layer turned EAGLE-3 drafter (not in open weights)Autoregressive MTP drafter (4 layers, reads trunk KV)—
Training & inference toolbox
Muon optimizer—????
FP8 → FP4 trainingFP8 block-scaled checkpointINT4 QAT in SFT, FP8 rollouts
Long-context extension?
RL with verifiable rewards?
On-policy distillation?????
Speculative decoding?—
KV-cache compression—
Adds beyond the blueprintNoneNoneNoneNoneNoneNoneNoneNoneNone