LLM Inference Engine
GPT-2 inference in C++ with KV caching, SIMD kernels, and INT8/INT4 quantisation.
The forward pass
I wrote the code that turns model weights and input tokens into a prediction: tokenisation, embeddings, transformer blocks, attention, and sampling. The project started with GPT-2 inference in C++.
Correctness and optimisation
I check logits against a reference implementation before changing the kernels. A correct next token can hide numerical errors earlier in the forward pass, so I compare the values as well as the selected token.
The optimisation work includes a KV cache, cache-aware matrix multiplication, SIMD, multithreading, and INT8/INT4 quantisation. Each changes the tradeoff between memory use, execution time, and numerical error. Benchmark results need the model, precision, and hardware alongside them to be comparable.
Run the browser implementation
The inference lab runs a C++ implementation compiled to WebAssembly. It includes SmolLM2 and a small model trained on this site’s prose, plus reference-comparison results when the build includes them. The lab reports generation timing on your device; it is a separate workload from the GPT-2 implementation described here.