CUDA Kernels, Chasing cuBLAS
A matrix-multiply kernel optimised rung by rung toward cuBLAS, ending with an honest account of why the last 15% is hard.
What’s hard about it
Every rung of the optimisation ladder requires reading a profiler correctly rather than guessing: deciding whether occupancy, memory throughput, or compute utilisation is the binding constraint, and predicting the gain before implementing the fix. Benchmarks also lie in specific ways here, measuring the first launch, or never checking for kernel errors and timing a kernel that did nothing.
The ladder
Naive kernel → global-memory coalescing → shared-memory tiling → register blocking → vectorised loads and double buffering. Every rung benchmarked as a percentage of cuBLAS in a GFLOPS table.
The target is 70–85%. Then the best section of the writeup: precisely why the remaining gap exists: occupancy against register pressure, instruction scheduling, and the microarchitectural tuning NVIDIA does not publish. Documented understanding of a gap signals more depth than any victory claim, and a suspiciously large one signals the opposite.
Phase two
FlashAttention's forward pass from the paper (fused, memory-hierarchy-aware attention with online softmax rescaling) integrated as a PyTorch custom op and benchmarked end to end against the built-in scaled dot-product attention.