Storage Engine Internals
This section walks through every layer of the storage engine, from the moment a write arrives to how it eventually lands on S3 and gets read back. Each page builds on the previous one.
| Layer | What it does |
|---|---|
| Composite Key | The encoding that underpins every data structure |
| WAL | Segment-based durability with group commit |
| Memtable | In-memory skip list with arena allocator |
| SSTable | Immutable sorted files on S3 |
| Manifest | Single source of truth for live SSTables, CAS coordination |
| Flush Pipeline | Frozen memtable → SSTable → S3 |
| Compaction | Leveled merge across L0–L3 |
| Cache | Three-tier W-TinyLFU with continuity tracking |
| S3 Storage Layer | Object layout, leases, large values, GC |
| Data Flow | End-to-end write, read, and recovery paths |