Lab / Inference engine
A language model running in this tab.
A transformer forward pass I wrote by hand in C++ and compiled to WebAssembly. No ML framework, no BLAS, no server. Ask the box below something and the answer is computed on your machine.
- 135M
- parameters
- 30 × 576
- layers × dim
- 78 KB wasm
- engine
- 0
- network calls after load
The weights are not mine. HuggingFaceTB/SmolLM2-135M-Instruct was trained by Hugging Face. I fine-tuned it to answer from retrieved sources or decline, quantised it, and wrote the code that runs it. The distinction matters: the engine is the artefact, the model is a payload. The second model in the selector is the opposite trade: 359K parameters I did train, on this site’s own prose, and it is much worse. The third is neither: Kimi K3 is 2.8 trillion parameters in somebody else’s datacentre, reached over an API, and it is there as the control. Same question, same retrieved passages, same system prompt, so the difference between its answer and the 135M model’s is the model rather than the plumbing.
Run it
Does it match the model it came from?
The only claim worth making about an engine is that it computes what the reference computes. A fixed prompt is replayed through the C++, KV cache and all, one token at a time, exactly as generation runs, and the final logits are diffed against PyTorch’s.
SmolLM2-135M · 49,152 logits
| Weights | Max |diff| | RMS | Argmax | Verdict |
|---|---|---|---|---|
| fp32 | 5.25e-5 | 1.57e-5 | match | pass |
| int8 | 1.34e+0 | 2.06e-1 | match | pass |
fp32 agrees to 5.2e-5 across thirty layers, which is accumulated float noise and not a bug. int8 diverges by 1.34 in the worst logit and still picks the same token. That gap is the quantisation itself, stated as a measurement rather than waved at. This number is what caught the one bug that mattered: a silently-defaulted RoPE θ left the engine exact at position zero, still selecting the right token, and wrong by 6 × 10⁻¹ by the fifth. Nothing but a logit diff would have found it.
tadiwa-nano-359K · 117 logits
| Weights | Max |diff| | RMS | Argmax | Verdict |
|---|---|---|---|---|
| fp32 | 7.63e-6 | 2.74e-6 | match | pass |
| int8 | 2.41e-1 | 8.87e-2 | match | pass |
Making it stop making things up
Asked cold what a KV cache is, this model said it “stores the current and voltage values from previous cycles”. At 135M parameters there is nowhere to keep the fact, so it is not asked to recall one: retrieval puts the passages in the prompt and the job drops to reading them. Chat mode above is the control.
That fixed the answers and not the refusals. Handed four sources that did not contain the answer, the base model answered anyway, zero refusals out of fourteen. Declining is a short, learnable habit rather than a fact to store, so that part is a LoRA: 4.9M trainable parameters, 3.5% of the model, on 122 examples from this corpus.
| Model | Refuses when it should | Refuses when it shouldn't | Grounding |
|---|---|---|---|
| SmolLM2 stock | 0% | 0% | 6% |
| + grounded LoRA | 100% | 7% | 94% |
The middle column is what keeps the first one honest. A model trained hard enough on refusals refuses everything, scores 100% on the left, and is useless. Both are always reported, and the training set is held under 40% refusals rather than over half. “Grounding” is the share of generated five-word sequences appearing verbatim in the sources: an approximation, not a proof. It scores a faithful paraphrase low, and it scores an answer drawn from the right project’s wrong sectionas a success. That happened, and the number cannot see it. Held out by page, so the eval pages contributed nothing to training.
Training answers are quoted straight from the corpus, not written by a bigger model. Supervision from something smarter would teach this one to sound smarter while inventing more.
The kernels underneath
10 of 10 matching their golden fixtures · tolerance 2e-4
- layernormokNormalise a row to zero mean and unit variance, then scale and shift. Runs twice per transformer block.
- softmaxokTurn scores into a distribution. Must subtract the row max first, or real logits overflow to NaN.
- matmulokWhere essentially all the time goes. Correct first, then loop order, blocking, and SIMD, each measured.
- geluokThe activation inside the MLP. Tanh approximation, which is what GPT-2 was trained against.
- matmul_i8okWeight-only int8: the codes stay packed and the scale is applied once per output. This is what lets a 135M model sit in 129 MB instead of 513 MB.
- attentionokScaled dot-product with a causal mask, applied before the softmax. The mask order is the classic bug.
- rmsnormokLayerNorm without the mean subtraction. Cheaper, and what every model since Llama actually uses.
- siluokThe SwiGLU gate, x·σ(x), applied to one of the two MLP branches and not the other.
- ropeokRotary position embedding: position enters as a rotation of query and key pairs. The pairing convention is (i, i + d/2), and getting it wrong is exact at position 0 and worse every token after.
- residual_addokThe residual connection. Trivial, and the natural first one to do.
The fixtures are data produced by a numpy oracle, not a second implementation. Otherwise the test only proves the two copies agree. One of them was too weak and said so: the softmax fixture was written to catch a missing max-subtraction, and when a deliberately broken softmax was dropped in, it passed. The sampled maximum simply never reached the float32 expf ceiling. It pins an extreme value now, and the broken version fails with inf.
What this is not
It is not fast. The matmul is a cache-aware triple loop with SIMD left to the compiler. A token takes about 23 ms natively and longer in WebAssembly. Blocking, hand-written SIMD and a fused dequantise-and-multiply are the interesting part, and I have not done them.
It is not a good model. 135M parameters gets facts wrong confidently and the demo does not hide it. It is here because it is the largest thing that fits down a browser connection. The one I trained is worse and measurably so: 359K parameters on 64 KB of prose, training loss 1.19 against validation 1.94, which is memorisation. It stays in the selector because the engine and the kernels are identical and only the weights differ.
What it is is the whole path from a weights file to a token, checked against a reference: the format, the loader, the tokeniser, the KV cache, grouped-query attention, the int8 kernel and the sampler. Loading it costs 129 MB, once, and only if you ask.