Skip to content
archdiff

models / gemma-4

Google DeepMind

Gemma 4 31B

Dense 5:1 local/global stack: 1,024-token windows, global layers with K = V and p-RoPE, sandwich norms, tied 262K vocabulary.

Compare with another model
Parameters
30.7B
Context
256K tokens
Depth
60 blocks (50 local, 10 global)
Weights
Open, Apache-2.0
Released
April 2026

--- frontier-llm@2026-11+++ gemma-4@2026-04+2~7−6

The November 2026 LLM blueprint, marked with Gemma 4 31B's differences: 10 components the same, 7 done differently, 6 not used, 2 added. Select any component.

Text · code · tool outputText · code · tool outputImages · video · audioImages · video · audioSentencePiece BPE with byte fallback — Input & outputSentencePiece BPE with byte fallbackInput & outputVision / audio encoder — native ViT or encoder-free patchesVision / audio encodernative ViT or encoder-free patchesToken embedding — media embeds spliced in placeToken embeddingmedia embeds spliced in placeEngram n-gram memory — hashed 2–4-gram lookup (V4.1)Engram n-gram memoryhashed 2–4-gram lookup (V4.1)Single residual stream with per-layer scalars — Residual & normSingle residual stream with per-layer…Residual & normRMSNorm — pre-normRMSNormpre-normQK-Norm — per-head RMSNorm on q, kQK-Normper-head RMSNorm on q, kRoPE by layer type: full RoPE local, p-RoPE global — AttentionRoPE by layer type: full RoPE local, p…AttentionGated DeltaNet / KDA — linear · ~3 of every 4 layersGated DeltaNet / KDAlinear · ~3 of every 4 layersGlobal GQA with keys reused as values — AttentionGlobal GQA with keys reused…AttentionSparse / compressed attn — DSA indexer · CSA / HCASparse / compressed attnDSA indexer · CSA / HCASliding-window attention — local · 128–1,024-token windowSliding-window attentionlocal · 128–1,024-token windowOutput gate + sink — σ(x·W_g) ⊙ o · learnable sinkOutput gate + sinkσ(x·W_g) ⊙ o · learnable sinkPost-attention RMSNorm (sandwich norm) — Residual & normPost-attention RMSNorm (sandwich norm)Residual & norm⊕ residual⊕residualRMSNorm — pre-normRMSNormpre-normDense GeGLU feed-forward — Mixture of expertsDense GeGLU feed-forwardMixture of expertsPost-feed-forward RMSNorm (sandwich norm) — Residual & normPost-feed-forward RMSNorm (sandwich no…Residual & normRouted experts — top-6–22 of 256–896 · SwiGLURouted expertstop-6–22 of 256–896 · SwiGLUShared expert — 1–2 · always activeShared expert1–2 · always active⊕ residual⊕residualFinal RMSNormFinal RMSNormTied LM head with logit soft-capping — Input & outputTied LM head with logit soft-cappingInput & outputAutoregressive MTP drafter (4 layers, reads trunk KV) — Input & outputAutoregressive MTP drafter (4 layers,…Input & outputNext-token logitsNext-token logitsDraft tokensDraft tokensMuon optimizer — Newton–Schulz · QK-Clip · per-headMuon optimizerNewton–Schulz · QK-Clip · per-headFP8 → FP4 training — block-scaled FP8 · MXFP4/NVFP4 QATFP8 → FP4 trainingblock-scaled FP8 · MXFP4/NVFP4 QATLong-context extension — staged 4K → 1M · YaRNLong-context extensionstaged 4K → 1M · YaRNRL with verifiable rewards — GRPO family · async agentic RLRL with verifiable rewardsGRPO family · async agentic RLOn-policy distillation — reverse KL to specialist teachersOn-policy distillationreverse KL to specialist teachersSpeculative decoding — MTP self-drafting · DSparkSpeculative decodingMTP self-drafting · DSparkKV-cache compression — FP4 KV · cross-layer reuse · SSDKV-cache compressionFP4 KV · cross-layer reuse · SSD
AttentionMixture of expertsResidual & normInput & outputTraining & inferenceData flowResidual / skipAddedReplacedNot usedNot disclosed

Other models