A language model running in this tab.

Run the model

Comparison with PyTorch

The test replays a fixed prompt through the C++ engine, including its KV cache, one token at a time. These saved results compare the final logits with PyTorch. They check the tested prompt and weights; they do not establish that every input will match.

SmolLM2-135M · 49,152 logits

tadiwa-nano-359K · 117 logits

Max |diff| is the largest logit difference; RMS describes the overall difference. Argmax checks whether both engines select the same next token. Quantisation can change the logits even when that token agrees.

Grounding and refusals

Grounded mode retrieves passages from this site before generation. A LoRA fine-tune teaches the model to answer from those sources or decline when they do not contain an answer. These saved evaluations compare the base model with that fine-tune on held-out pages.

Correct refusals are measured on unanswerable questions; incorrect refusals on answerable ones. Source overlap measures generated five-word sequences found verbatim in the retrieved passages. It misses faithful paraphrases and can reward quotations that do not answer the question. It is not a factual-accuracy score.

Kernel tests

10 of 10 matching their golden fixtures · tolerance 2e-4

• layernormpassNormalise a row to zero mean and unit variance, then scale and shift. Runs twice per transformer block.
• softmaxpassTurn scores into a distribution. Must subtract the row max first, or real logits overflow to NaN.
• matmulpassWhere essentially all the time goes. Correct first, then loop order, blocking, and SIMD, each measured.
• gelupassThe activation inside the MLP. Tanh approximation, which is what GPT-2 was trained against.
• matmul_i8passWeight-only int8: the codes stay packed and the scale is applied once per output. This is what lets a 135M model sit in 129 MB instead of 513 MB.
• attentionpassScaled dot-product with a causal mask, applied before the softmax. The mask order is the classic bug.
• rmsnormpassLayerNorm without the mean subtraction. Cheaper, and what every model since Llama actually uses.
• silupassThe SwiGLU gate, x·σ(x), applied to one of the two MLP branches and not the other.
• ropepassRotary position embedding: position enters as a rotation of query and key pairs. The pairing convention is (i, i + d/2), and getting it wrong is exact at position 0 and worse every token after.
• residual_addpassThe residual connection. Trivial, and the natural first one to do.

NumPy produces the reference fixtures. The softmax fixture includes an extreme value to test numerical stability: a version without max-subtraction overflows and fails that case.

Limits of this demo