Nemotron 3 Ultra is not built from attention + FFN blocks. It is a stack of 108 single-sublayer layers, each x ← x + F(RMSNorm(x)) with F one of: a Mamba-2 state-space mixer (48 layers), a LatentMoE layer (48), or a grouped-query attention layer (12). Mamba-2 → MoE pairs make up most of the stack, and every few pairs an attention layer sits between a Mamba-2 layer and the next MoE. The attention layers are plain: 64 query heads sharing 2 KV heads, no positional encoding, no QK-norm, no gate or sink. Order and most local context come from the Mamba-2 layers, so the whole KV cache is 12 layers × 2 KV heads, or 6 KiB per token in FP8.
The MoE layers use LatentMoE. The router and one shared expert work at the full width d = 8,192, but the token is projected down to a 2,048-dim latent before the routed experts: 512 non-gated squared-ReLU experts, 22 active per token. Shrinking expert weights and all-to-all traffic 4× is what pays for the large expert count and top-k. One weight-shared MTP block (an attention layer plus a MoE layer) is applied recursively for speculative decoding.
Training follows Nemotron 3 Super's recipe at larger scale: 20T tokens pretrained mostly in NVFP4 with AdamW, then a 33B-token phase trained directly at 1M tokens. Post-training is SFT, one unified asynchronous-GRPO RLVR stage, and two rounds of multi-teacher on-policy distillation from more than ten domain teachers. The released NVFP4 checkpoint is a separate post-training quantization at 5.03 bits per element.