Skip to content
archdiff

Every frontier model is a diff.

ArchDiff keeps one reference blueprint of how state-of-the-art LLMs and agents are built, current as of November 2026, with every component explained at research depth. Each real model, and each improvement a member proposes, is shown as a diff against it.

Read the LLM blueprintRead the agent blueprint
Open DeepSeek-V4.1-Flash on the diagram

blueprints/

The base everything is compared with

Two composite blueprints of the November 2026 frontier. Select any component for how it works, why it won, its trade-offs and its sources.

The Frontier LLM

A composite blueprint of the open-weight frontier: hybrid or compressed attention, ultra-sparse MoE, widened residual streams, FP4-native training and serving.

20 components, 2026-11

Text · code · tool outputText · code · tool outputImages · video · audioImages · video · audioByte-level BPE tokenizer — 128K–262K vocabByte-level BPE tokenizer128K–262K vocabVision / audio encoder — native ViT or encoder-free patchesVision / audio encodernative ViT or encoder-free patchesToken embedding — media embeds spliced in placeToken embeddingmedia embeds spliced in placeEngram n-gram memory — hashed 2–4-gram lookup (V4.1)Engram n-gram memoryhashed 2–4-gram lookup (V4.1)Residual streams — mHC: n = 4 mixed streamsResidual streamsmHC: n = 4 mixed streamsRMSNorm — pre-normRMSNormpre-normQK-Norm — per-head RMSNorm on q, kQK-Normper-head RMSNorm on q, kPartial RoPE / NoPE — rotate 64 dims, or nonePartial RoPE / NoPErotate 64 dims, or noneGated DeltaNet / KDA — linear · ~3 of every 4 layersGated DeltaNet / KDAlinear · ~3 of every 4 layersLatent attention (MLA) — global · GQA in many modelsLatent attention (MLA)global · GQA in many modelsSparse / compressed attn — DSA indexer · CSA / HCASparse / compressed attnDSA indexer · CSA / HCASliding-window attention — local · 128–1,024-token windowSliding-window attentionlocal · 128–1,024-token windowOutput gate + sink — σ(x·W_g) ⊙ o · learnable sinkOutput gate + sinkσ(x·W_g) ⊙ o · learnable sink⊕ residual⊕residualRMSNorm — pre-normRMSNormpre-normMoE router — σ / √softplus · bias balancingMoE routerσ / √softplus · bias balancingRouted experts — top-6–22 of 256–896 · SwiGLURouted expertstop-6–22 of 256–896 · SwiGLUShared expert — 1–2 · always activeShared expert1–2 · always active⊕ residual⊕residualFinal RMSNormFinal RMSNormLM head — untied · vocab logitsLM headuntied · vocab logitsMTP module(s) — predict t+2 … t+kMTP module(s)predict t+2 … t+kNext-token logitsNext-token logitsDraft tokensDraft tokensMuon optimizer — Newton–Schulz · QK-Clip · per-headMuon optimizerNewton–Schulz · QK-Clip · per-headFP8 → FP4 training — block-scaled FP8 · MXFP4/NVFP4 QATFP8 → FP4 trainingblock-scaled FP8 · MXFP4/NVFP4 QATLong-context extension — staged 4K → 1M · YaRNLong-context extensionstaged 4K → 1M · YaRNRL with verifiable rewards — GRPO family · async agentic RLRL with verifiable rewardsGRPO family · async agentic RLOn-policy distillation — reverse KL to specialist teachersOn-policy distillationreverse KL to specialist teachersSpeculative decoding — MTP self-drafting · DSparkSpeculative decodingMTP self-drafting · DSparkKV-cache compression — FP4 KV · cross-layer reuse · SSDKV-cache compressionFP4 KV · cross-layer reuse · SSD
Open the LLM blueprint

The Frontier Agent

The long-horizon, tool-using agent harness you would build in November 2026

18 components, 2026-11

Task / user intent — goal · repo · constraintsTask / user intentgoal · repo · constraintsOrchestrator loop — think → act → observeOrchestrator loopthink → act → observeContext assembly — prompt · tool names · AGENTS.mdContext assemblyprompt · tool names · AGENTS.mdSkills — progressive disclosureSkillsprogressive disclosureMemory hierarchy — MEMORY.md index · notes · searchMemory hierarchyMEMORY.md index · notes · searchReasoning model — interleaved thinking · effortReasoning modelinterleaved thinking · effortTool calls / action select — parallel · deferred · as codeTool calls / action selectparallel · deferred · as codeMCP servers — stateless 2026-07-28 specMCP serversstateless 2026-07-28 specSandboxed shell & code — OS sandbox / VM · egress proxySandboxed shell & codeOS sandbox / VM · egress proxyBrowser & computer use — screenshot · zoom · mouse · keysBrowser & computer usescreenshot · zoom · mouse · keysSub-agents — fresh context · return summarySub-agentsfresh context · return summaryObserve & verify — tool results · tests · evaluatorObserve & verifytool results · tests · evaluatorVerified result — PR · report · answerVerified resultPR · report · answerPermissions & auto-approval — tiers · classifier · escalatePermissions & auto-approvaltiers · classifier · escalatePrompt-injection defense — rule of two · probes · no credsPrompt-injection defenserule of two · probes · no credsCompaction & context edits — summarize · clear old tool resultsCompaction & context editssummarize · clear old tool resultsPrompt caching — stable prefix · append-onlyPrompt cachingstable prefix · append-onlyPlanning & task lists — plan mode · tasks · progress filesPlanning & task listsplan mode · tasks · progress filesDurable sessions — event log · resume · checkpointsDurable sessionsevent log · resume · checkpointsTracing & evals — OTel spans · transcripts · pass^kTracing & evalsOTel spans · transcripts · pass^k
Open the agent blueprint

models/

What each real model changes

Every profiled model drawn on the blueprint: what it keeps, what it does differently and what it leaves out, cited to its own report, config or code.

Compare all 10 models

  • DeepSeek, September 2026

    DeepSeek-V4.1-Flash

    552B + 196B Engram total, 16B (8B in prefill) active

    Causal encoder-decoder MoE: decoder reads encoder-projected KV, CSA2 shares KV across layers, FP4 KV at 890 B/token, Engram, DSpark.

    +0~3−1

  • Thinking Machines Lab, July 2026

    Inkling

    975B total, 41B active

    5:1 sliding-window/global GQA with a learned relative-position bias instead of RoPE, short causal convs, encoder-free image and audio.

    +0~5−3

  • Moonshot AI, July 2026

    Kimi K3

    2.8T total, 104B active

    93 layers of 3:1 KDA / gated MLA with no positional encoding, Block Attention Residuals and a 16-of-896 latent MoE in MXFP4.

    +0~5−3

  • NVIDIA, June 2026

    Nemotron 3 Ultra

    550B total, 55B active

    Mamba-2 does most of the token mixing, 12 NoPE GQA layers anchor it, and LatentMoE routes 22 of 512 experts in a 2,048-dim latent.

    +0~4−7

proposals/

Propose your own diff

Think a component should be different? Change it on the diagram and make the case. Good diffs get merged.

All proposals

  1. Open a proposal

    Pick any component and replace it, add something after or alongside it, or remove it. Say what it fixes, what it costs and which papers or results back it.

  2. Members vote

    The diagram shows your change in place, so everyone sees exactly what would be different.

  3. An AI reviewer checks it

    Once it has enough votes, the reviewer opens the cited sources and checks every claim against them.

  4. Approved diffs are merged

    The change becomes a commit in the blueprint's history, and any commit can be reverted.

Start a proposal

Leading proposals

Loading proposals…

The frontier moves every month. So does this.

One email a month: what changed in the blueprints, newly profiled models and the proposals that were merged. Read past changes