LSM Storage Engine
A crash-safe key-value store that real redis-cli connects to. The hard part is compaction that never loses a key.
What’s hard about it
Compaction is where this class of project dies. Merging sorted files while concurrent writes land, tombstones expire, and a crash can arrive between any two syscalls means bugs corrupt data silently and surface hours later. The mitigation is property-based testing from the first commit (random operation sequences checked against a plain HashMap model) plus a manifest log so a crash mid-compaction is replayable rather than fatal.
The plan
A storage engine, built in four milestones, because it composes the four things every systems interview actually probes: data structures on disk, OS interfaces, concurrency, and performance measurement.
- A RESP-speaking server. Enough of the Redis wire protocol that the real
redis-cliconnects and issuesGET/SET/DEL. The moment an unmodified client talks to your binary is the moment the project becomes demonstrable. - Persistence and crash recovery. Write-ahead log, a configurable fsync
policy, and a test that
kill -9s the server mid-write-storm and verifies every acknowledged write survived. That test runs in CI. - The LSM tree. Memtable, sorted immutable SSTables with sparse indexes, bloom filters, and background compaction.
- Concurrency. Many clients, readers that don't block on compaction, and a flamegraph identifying where the time actually goes.
Why this one
Almost every backend I use sits on a storage engine I couldn't explain past a block diagram. The interesting problems (the durability-versus-throughput cliff, compaction without corruption, reads across immutable layers) only become real when you have to make them work.