Fast Writes Need a Durability Contract
Fast writes look great on a benchmark chart. They also hide the key reliability question: what work finished before the system acknowledged success?
Shayon Mukherjee's post, Every fast write moves work somewhere else, is a useful reminder for operators. A write can return after memory, a local SSD flush, an object storage PUT, a durable network volume, or a replicated WAL majority. Each choice changes latency and failure tolerance.
What Success Means
When an application writes to a database, the client sees one response. Under the hood, that response may mean very different things:
- bytes reached process memory or the kernel page cache
fdatasync()finished on one local SSD- a network block volume acknowledged its own copies
- object storage accepted an immutable file
- a quorum of database replicas persisted the WAL record
Those are not interchangeable success points. Returning after page cache is fast, but a machine loss can erase the acknowledged write. Returning after local fdatasync() survives process and kernel crashes, but not loss of the host or disk. Waiting for object storage or a replicated WAL costs more latency, but moves durability outside one machine.
Why SRE Teams Should Care
The danger is unlabeled performance. A chart that says "1 ms write latency" is not enough. On-call engineers need to know what the write survives and what background work must catch up later.
This shows up in object-storage-backed databases, log-structured systems, event pipelines, and queue-heavy services. A local WAL can make the client path fast while uploads happen later. That creates a recovery point objective tied to the oldest unuploaded record.
The same pattern applies to batching. One flush can cover many client writes, which improves throughput, but the earliest request waits longer and a failed flush fails the whole batch.
Operational Checks
Before trusting a fast write path, document the contract:
- What exact event triggers client success?
- Which failures can still lose an acknowledged write?
- How much unreplicated or unuploaded data can accumulate?
- Where is backpressure applied when the slower path falls behind?
- Are retries idempotent when success reached storage but the reply was lost?
That last point matters during incidents. A timeout does not always mean failure. The write may have committed while the response disappeared. Use operation IDs or idempotency keys so clients can retry safely and receive the recorded result.
Conclusion
Fast writes are not bad. Hidden write contracts are bad. If a system moves durability, cleanup, replication, or upload work out of the client path, SRE teams need metrics for the place that work moved to.
Akmatori helps SRE teams investigate incidents, automate operational workflows, and keep critical services reachable with global TCP and UDP load balancing. Built on Gcore infrastructure, Akmatori gives operators a clearer control plane when fast systems fail in slow, stateful ways.