On August 12, 2026, Tailscale engineer Alex Chan published a long account of a hunt that lasted half a year: the SQLite databases under the company’s control plane were corrupted 19 times in six months, with no reproducible steps — and the culprit turned out to be a data race latent in the SQLite source code for at least 16 years. The control plane is the set of services that coordinates every node on a Tailscale network, so database integrity there is not an academic concern. The SQLite documentation now dedicates a section to the bug, called the “WAL reset bug.” The Hacker News discussion reached 1,223 points; engineering communities never resist a deep-sea-creature story like this one, and this one earns the attention.
A Data Race Latent for 16 Years
The bug now lives in the official SQLite documentation: a rare race between a checkpoint and a write transaction. For the unfamiliar, SQLite in write-ahead-logging mode periodically checkpoints — copying committed pages out of the WAL file into the main database. When a write lands at just the wrong moment during that copy, the checkpoint mistakenly believes certain pages were already transferred — when they were not. The result is committed data vanishing silently and a corrupted database file. The SQLite team estimates the defect existed for “at least 16 years”; the trigger conditions are so narrow that their own tests had to add deliberate code to provoke it, which is also why it survived in the wild for so long without being pinned down.
Nineteen Corruptions, Production Forensics
Each shard of Tailscale’s control plane runs a single-writer SQLite database, so one corrupted file can degrade a whole shard of the coordination layer. Nineteen corruptions in six months, no pattern, and every conventional test passing. Their answer was forensics in production: deploy telemetry to capture evidence, use a transaction-replay pipeline to prove that committed data was disappearing without errors, and then find the tell in the metrics — checkpoints were copying more pages out of the WAL than actually existed in the file. The replay pipeline mattered because it turned an intermittent mystery into a testable claim: if committed data could vanish without an error, the bug was below the application layer. Mitigations rolled out in parallel: hard-stop the service the moment corruption is detected, watch backups with an automated PRAGMA integrity_check, and drill the recovery procedure down to under an hour.
Working With Upstream: From Tracing Shim to Fix
The turning point came from a professional support contract. The SQLite team built a VFS tracing shim called tmstmpvfs, and the logs from the next incident finally captured the race in the act. The fix itself is restrained: one added check in the checkpoint function that detects the WAL being reset by another thread. There was a detour along the way — the fix first shipped in 3.52.0, but that release was pulled because of a second bug, a changed text-to-float rounding behavior that broke expression indexes and produced false corruption reports; 3.51.3, carrying only the WAL-reset fix, was the clean one. A release found to introduce new corruption reports being withdrawn, rather than left out with a warning, is its own small statement about how the project treats releases. Tailscale also reduced its timestamp precision to integer seconds as a safeguard, and SQLite later added self-healing indexes in 3.53.0.
Three Lessons for Engineering Teams
First, vanishingly rare races only show up under real traffic: for a bug you cannot reproduce, observability is the only lever, and Tailscale’s transaction-replay pipeline proves that production can be used for forensics, not just monitoring. Second, design for database corruption as a certainty — integrity checks, clean backups, and a rehearsed recovery runbook that keeps mean time to recovery at the hour level; the alternative is discovering your restore procedure during an outage. Third, a formal engagement with upstream maintainers beats patching on your own: the VFS shim, the root-cause analysis, and the fix landing upstream were all channels the contract bought — and even a 16-year-old bug moves when the people who own the code are on the case.
Sources
- Tracking down the 16-year-old WAL-reset SQLite bug — Tailscale
- The WAL Reset Bug — SQLite Documentation
- Hacker News discussion
AI-assisted summary compiled from the sources above, reviewed by a human before publishing.