Thinking Machines' first open-weight model keeps a conventional skeleton (pre-norm blocks, GQA with QK-norm, a DeepSeek-V3-style MoE with 256 routed experts, top-6 and 2 shared experts) and changes three things around it. The first is position: there is no RoPE. Each attention layer adds a learned relative-position bias, a per-head mix of 16 learned bias-versus-distance profiles weighted by the query token. It spans the full 512-token window in the 55 sliding-window layers and 1,024 tokens in the 11 global layers. Beyond that, the global layers see no positional term, and past 128K positions they scale their logits logarithmically.
The second is short causal convolutions (kernel 4, with a skip) in four places per block: on keys and values right after projection, and on the attention and MoE outputs before they rejoin the residual stream. The third is in the router, which scores the shared experts too. Their weights are normalized jointly with the 6 selected routed experts, so the always-on shared experts act as a sink for routing mass instead of being added ungated.
Inputs are natively multimodal without a pretrained encoder: 40 × 40-pixel patches go through a 4-layer hierarchical MLP, and audio enters as discretized mel frames. The model was trained on 45T tokens of text, images, audio and video with Muon plus Adam, then post-trained mostly with asynchronous RL (over 30M rollouts). Weights ship under Apache-2.0 in BF16 and NVFP4, with 8 MTP drafter layers in a separate file.