Summary: OpenAI explained on September 11, 2026 how Habitat, its Python-based online storage service, grew to more than 70 million requests per second and supported products used by over 1 billion people weekly. The service accepted latency and resource overhead as deliberate debt so product teams could ship faster, then reduced tail latency by limiting per-process concurrency, instrumenting event-loop delays, and fixing configuration and connection-pooling imbalances. The lesson is to measure scheduling delay and tail behavior rather than assume throughput alone shows service health.
One request from a ChatGPT user can split into hundreds of database calls, and the one the user actually notices is the slowest of them. That framing opens OpenAI’s September 11, 2026 write-up on scaling online storage, and it is also why Habitat, the company’s online storage platform, makes a useful case study for anyone who runs a service layer in front of a database.
What Habitat is, in practice
The first version showed up in mid-2024: a small client-side Python library talking to Azure Cosmos DB. Engineers working on product features were spared from thinking about schema lookup, routing, authorization, encryption, serialization, request shaping, and connection pooling. Teams adopted it on their own, with no central mandate pushing it.
By the middle of 2025 the library had run out of room. Take a change such as moving critical datasets onto regionally distributed Cosmos DB accounts: it meant writing new routing logic, putting that logic behind a feature flag, rolling it out across dozens of services, adding shadowing, fixing bugs, and rolling out again. OpenAI recounts a rollout of exactly that kind which stretched over days and ended in an outage anyway, after a team rolled back to a buggy client. The answer was to move storage logic into a service of its own, giving the company a single point of control for deployments, observability, and security policy enforcement.
Habitat today takes on more than 70 million requests per second. It supports products used by over 1 billion people each week, reaches almost 40 geographic regions, and serves more than 500 petabytes of data.
The Python bet, and the bill attached to it
OpenAI kept the service in Python with the overhead in full view. Measured against running the same code as a local library, a Python service introduces network latency and drives up CPU and memory cost, and the team expected a rewrite somewhere down the line. They booked it as deliberate technical debt in exchange for unblocking product developers, and they wagered that Codex and GPT would be capable enough to make the eventual migration tractable. By their account, that wager paid off.
What deserves a builder’s attention is what had to be handled in the meantime. Asyncio supplies concurrency for I/O-bound work, but it supplies no CPU parallelism. On top of proxying, Habitat does routing, compression, encryption, checksumming, health checking, shadowing, and hedging. With that much CPU-bound work in the path, asyncio scheduling delay can end up dominating tail latency: the traces showed downstream storage answering quickly while the request sat waiting for a coroutine to be rescheduled so it could parse the response.
Their mitigation cuts against the instinct to maximize throughput per process. Each process handles only a small number of concurrent requests, and capacity comes from scaling out worker processes instead. The team also watches the event loop directly, scheduling background tasks and recording how far actual execution time drifts from the expected time. Under high utilization that jitter can reach hundreds of milliseconds, and in edge cases run to several seconds.
Two concrete bugs, and fixes worth copying
The first involved feature flag configuration. Statsig was polled once a minute with no jitter, and the config file carried every production rule from every service. Since each pod runs up to eight Python processes, every pod would periodically stall all of its workers to parse one large JSON file. The remedy had three parts: a smaller targeted config, a longer refresh interval, and jitter on background tasks.
The second was connection pooling working against load balancing. With client-side pooling, a busy client process could get by with only a handful of server connections and push all of its traffic onto a handful of processes. Utilization swung widely, and tail processes carried 5-10x the concurrent requests of the average. If a proxy or gateway layer sits in your stack, that asymmetry is worth measuring before you touch any other knob. The same instinct appears in prefix-aware routing for LLM latency, where a request’s landing spot matters as much as how fast the backend is.
What to take from this
Habitat’s story is not an argument for Python at this scale. OpenAI states plainly that the inefficiencies would not hold up at 100x and that a rewrite was nearly certain. It is an argument for knowing which resource you are genuinely short on. They traded per-request efficiency for speed of product iteration, then paid to win tail latency back through process-level tuning, event loop instrumentation, and a few small operational fixes.
For anyone running a service layer in front of a datastore, the practical next step is to instrument scheduling delay, not just CPU and memory. How Habitat’s multi-tenancy reliability and read-path optimization work is not specified in the supplied material; OpenAI says a second post will cover them.