Summary: A long post from Tailscale engineer Alex Chan, published on August 12, 2026, laid out a chase that took half a year to close: 19 times in six months, the SQLite databases beneath the company’s control plane had gone corrupt, no reproduction steps existed, and what they finally caught was a data race that had been sitting in the SQLite source code for at least 16 years. Should a write arrive at precisely the wrong instant during that copy, the checkpoint comes away believing certain pages had already made the trip — they had not.
A long post from Tailscale engineer Alex Chan, published on August 12, 2026, laid out a chase that took half a year to close: 19 times in six months, the SQLite databases beneath the company’s control plane had gone corrupt, no reproduction steps existed, and what they finally caught was a data race that had been sitting in the SQLite source code for at least 16 years. The control plane — the collection of services that keeps every node on a Tailscale network coordinated — is no place where database integrity counts as an academic worry. A section of the SQLite documentation now carries the bug’s name: the “WAL reset bug.” Over on Hacker News the thread climbed to 1,223 points; a deep-sea-creature story of this kind is something engineering communities simply will not scroll past, and this one has earned the attention.
A Data Race Latent for 16 Years
The defect has a home in the official SQLite documentation now: a rare race pitting a checkpoint against a write transaction. For readers who do not live in the engine’s internals: under write-ahead logging, SQLite checkpoints on a regular cadence, copying committed pages out of the WAL file and into the main database. Should a write arrive at precisely the wrong instant during that copy, the checkpoint comes away believing certain pages had already made the trip — they had not. What follows is committed data disappearing without a sound and a corrupted database file. “At least 16 years” is the SQLite team’s estimate for how long the defect has existed; the conditions that trigger it are so narrow that even their own test suite needed purpose-written code to provoke a hit, which is also why it survived in the wild for so long without being pinned down.
Nineteen Corruptions, Production Forensics
Every shard of Tailscale’s control plane runs its own single-writer SQLite database, which means one corrupted file can degrade an entire slice of the coordination layer. Nineteen corruptions over six months, no discernible pattern, and not one conventional test failing. So they carried forensics into production: telemetry deployed to gather evidence, a transaction-replay pipeline built to demonstrate that committed data was disappearing without any error, and then the tell found in the metrics — checkpoints were pulling more pages out of the WAL than the file actually held. The replay pipeline was the piece that mattered, because it turned an intermittent mystery into a testable claim: if committed data could vanish with no error raised, the fault had to sit beneath the application layer. Mitigations went out in parallel: stop the service hard the moment corruption is detected, keep an automated PRAGMA integrity_check watching the backups, and rehearse the recovery procedure until it fits inside an hour.
Working With Upstream: From Tracing Shim to Fix
A professional support contract is what broke the case open. The SQLite team wrote a VFS tracing shim named tmstmpvfs, and when the next incident arrived, its logs caught the race in the act. The fix itself shows restraint: a single check added inside the checkpoint function, watching for the WAL being reset by another thread. One detour came along with it — the fix first shipped in 3.52.0, only for that release to be pulled over a second defect, a changed text-to-float rounding behavior that broke expression indexes and produced false corruption reports; 3.51.3, carrying the WAL-reset fix and nothing else, is the clean one. That a release found to be generating fresh corruption reports gets withdrawn, rather than left out with a warning attached, is a small statement in its own right about how the project treats its releases. Tailscale cut its own timestamp precision to integer seconds as a safeguard, and SQLite later added self-healing indexes in 3.53.0.
Three Lessons for Engineering Teams
Lesson one: a race this vanishingly rare shows itself only under real traffic, so when a bug will not reproduce, observability is the one lever you hold — and Tailscale’s transaction-replay pipeline is proof that production can serve forensics, not merely monitoring. Lesson two: treat database corruption as a certainty and design for it — integrity checks, clean backups, and a recovery runbook you have actually rehearsed, holding mean time to recovery in the hour range; skip that work and you will be reading your restore procedure for the first time in the middle of an outage. Lesson three: a formal relationship with upstream maintainers costs less than patching on your own: the VFS shim, the root-cause analysis, the fix reaching mainline — each of those channels came with the contract, and even a 16-year-old bug starts moving once the people who own the code are on the case.