Summary: Cohere published (October 9, 2026) a breakdown of when shared, consumption-based inference beats dedicated capacity for embedding and reranking workloads. The core argument: requests per minute is nearly useless as a sizing metric, because request shape and traffic pattern move the economic break-even from about 4 requests per minute to tens of thousands on the same model. If you run retrieval or RAG infrastructure, this changes how you should estimate before you provision.
RPM Is the Wrong First Number
Most teams comparing shared and dedicated inference start with two numbers: price per token and requests per minute. Cohere’s argument, based on their analysis, is that neither tells you much on its own. What matters is how much work each request carries and how steadily that work arrives.
Their example is concrete. A catalog embedding job at 30 requests per minute sounds tiny — until you notice each request holds 100 texts at roughly 200 tokens each, about 600K tokens per minute of sustained processing. Meanwhile thousands of short query embeddings per minute can be lighter in total compute. Same RPM-adjacent region, opposite infrastructure stories.
One Model, Two Very Different Jobs
The e-commerce search case Cohere walks through splits an embedding workload into two modes. Indexing sends large batches — the shopper isn’t waiting, so throughput and price-performance win. Live queries are one short string with a spinner on screen, so responsiveness wins even at worse price-performance. Infrastructure tuned for one mode won’t serve the other well.
Reranking adds a third variable: retrieval depth. Each query-document pair is scored independently, so a 20-token query plus 50 candidates at 200 tokens each is roughly 11,000 tokens per search. Bump candidates from 20 to 100 and token consumption jumps from about 4,400 to about 22,000 per search — invisible to the user, very visible on your bill and your capacity plan. Retrieval depth is an infrastructure decision, not just a quality knob.
The Break-Even Moves With the Workload
Using representative consumption pricing and a single dedicated NVIDIA A10 GPU billed hourly, with traffic sustained 24/7 and no negotiated discounts, Cohere’s illustrative crossovers are striking:
- Query embedding (~8 tokens per request): break-even in the tens of thousands of requests per minute
- Catalog embedding (100 texts × 200 tokens, ~20K tokens): around 20 requests per minute
- Long-document embedding (~100K tokens): around 4 requests per minute
- Reranking (1 query + 50 docs): around 29 requests per minute
Same model, same price — only the request shape changes, and the crossover spans four orders of magnitude. Their catalog example at 30 RPM is already past its own ~20 RPM break-even, so dedicated capacity is likely cheaper despite the modest rate. Cohere also cautions that these figures are workload-specific illustrations, not universal thresholds; partial-day traffic, a different instance type, or negotiated pricing all shift the crossover. Note also that embedding is billed per token while reranking is billed per search unit, so those break-evens aren’t points on one scale.
Traffic Pattern and Latency Close the Loop
Two apps with identical peaks can have very different economics. A global search service with steady all-day flow keeps dedicated capacity busy. A batch reporting job with spikes in narrow windows pays for idle capacity if provisioned — consumption-based serving fits better there. The real question Cohere poses isn’t “how high is peak traffic” but “how consistently can the infrastructure be utilized.”
Latency complicates throughput math too. Batch indexing can run at maximum throughput; search queries can’t, because capacity has to be sized to sustainable throughput at your target latency, with headroom beyond peak demand. Expanding rerank candidate sets can even raise documents-per-second while lowering searches-per-second served.
The practical takeaway: before choosing, answer what a typical request contains, whether it’s document or query embedding, how many documents get reranked, whether traffic is steady or bursty, and what latency you actually require. It’s the same discipline I covered in the Cloudflare piece on deferring hardware spend — decide from utilization over time, not from a single peak number. The cheap next step is logging your real request sizes and traffic shape for a week; that data will tell you which side of the crossover you’re on before you commit to an instance.
Source: Cohere’s shared vs. dedicated inference analysis and the earlier post What Cloudflare’s Data Pipeline Teaches About Deferring Hardware Spend.