Summary: OpenRouter published a reranking tutorial on November 13, 2026, with a 274-question benchmark showing that a dedicated cross-encoder reranker pushed top-chunk accuracy from 81% to 97% for roughly $0.10 per 1,000 queries. The setup keeps your existing vector search and adds one rerank call before the LLM. For anyone shipping RAG, it’s a cheap, measurable fix for the most common retrieval failure: the right document exists but never reaches the model.
Why vector search misses the right chunk
The failure mode is structural, not a bug. An embedding model compresses an entire chunk into a single vector before it ever sees the user’s question. In a 130-word chunk about bucket limits, the plan name is one or two words out of the whole block — so the vectors for Starter, Growth, Scale, and Enterprise plans cluster almost on top of each other. In OpenRouter’s full test, the question “How many buckets can I create in a single Fernbank account on the Growth plan?” got the Starter, Enterprise, and Scale chunks ranked first, second, and third. The Growth answer sat at fifth. Any pipeline that sends only its top three candidates to the LLM never shows the model the correct document.
A reranker changes the scoring itself. Instead of judging chunks in isolation, a cross-encoder reads the user’s question alongside each candidate chunk and re-scores the pair. Running a cross-encoder across thousands of documents is too slow for real-time traffic, so vector search stays as the fast first stage, narrowing the field to a shortlist of about 30 candidates before the reranker sorts it. You don’t replace your embedding search; you widen its recall and add one call.
What the benchmark actually measured
OpenRouter built a controlled corpus for a fictional storage service called Fernbank: 274 chunks spanning docs, error codes, SDK guides, changelogs, and deliberate noise — including unanswered forum posts meant to trap naive matches. Chunks were generated with anthropic/claude-sonnet-5.5 and verified by script; openai/gpt-6.1-sol wrote the questions. Because every question maps to exactly one correct chunk, retrieval could be scored without judgment calls.
The headline numbers:
- google/gemini-embedding-2 alone put the right chunk first 81% of the time; reranking the top 30 with voyageai/rerank-3-lite pushed precision@1 to 97%.
- voyageai/rerank-3-lite and voyageai/rerank-3 were both the most accurate and the cheapest of the 11 rerankers tested, at $0.10 and $0.26 per 1,000 queries.
- An LLM used as the reranker (openai/gpt-6-luna) hit 99.6% precision@1, but took 4.0 seconds per call against 1.3 seconds for the dedicated rerank model.
- The budget pairing — voyageai/voyage-4-lite plus voyageai/rerank-3-lite — reached 96.7% precision@1, beating the pricier Gemini embedding alone at 81.0%.
The end-to-end answer test is the part worth staring at. In an October 8, 2026 run over all 274 questions, gemini-embedding-2 retrieving the top 30 and reranking down to 3 produced 99.3% correct answers with the right chunk in context 100% of the time — versus 97.1% and 98.2% for the top 5 with no rerank. Fewer chunks, better answers. Most incorrect answers in the no-rerank setups traced back to missing context: in nine out of ten cases where the right chunk wasn’t in the top three, the model simply said the passages didn’t contain the answer. The app told users it didn’t know something its own documentation could answer.
The two details that quietly cost points
Set input_type on every embedding call. It tells the model whether it’s encoding a search query or a document, and some models treat them differently. Leaving it out dropped precision@1 for Gemini from 81.0% to 74.1%, and for voyage-4-lite from 64.6% to 52.9%. That’s 7 to 12 points from one omitted parameter.
Know what top_n does: it controls how many results the reranker returns, not how many it scores. The model evaluates every candidate you send, so your candidate count drives both cost and latency — pick it deliberately rather than by default.
One more quiet gotcha from the tests: cohere/rerank-v3.5 reads only the first 4,096 tokens of a document and gives no error. If your chunks run long, results will degrade silently.
Whether it actually helped your answers
OpenRouter’s own advice on measurement matches what I’d tell a team: score retrieval before and after, then score the final answers. Start with precision@1 — the percentage of questions where the first retrieved chunk holds the correct answer — and fifty real questions from your logs, each labeled with its answer chunk, is enough for a baseline. Their guide to turning production traffic into a golden evaluation dataset covers how to build that set.
Don’t lean on recall@5 alone. In their test, Gemini hit 98.2% recall@5 but only 81.0% precision@1 — the right information was usually somewhere in the results, just not at the top. Recall numbers flatter a pipeline that’s feeding the LLM five mediocre chunks; precision tells you whether three good ones would do.
The trade-offs, stated plainly
The cost math in OpenRouter’s setup was nearly neutral: sending 3 chunks instead of 5 cut LLM answering cost from $0.22 to $0.16 per 1,000 questions, the rerank call added $0.10, so reranking cost a net $0.04 more per 1,000 questions. Your numbers will differ with your models, document sizes, and query volume — measure the whole pipeline, retrieval through final answer, before assuming savings or costs.
Relevance also isn’t correctness. In their worked example, an unanswered forum post asking the same question ranked second after reranking, at 0.793 — the reranker rightly saw it as relevant, even though it answers nothing. Filter out content that mentions questions without resolving them, and send the LLM a few strong candidates rather than one.
The biggest caveat: this is a small synthetic corpus for a fictional product, so the exact gains — 81% to 97%, 97.1% to 99.3% — won’t transfer to your documents. Use them as evidence the mechanism works, not as a forecast. Worth noting too: the excerpt of the tutorial I’m working from cuts off partway through, so its later sections on hybrid search, better embedding models, and failure modes aren’t reflected here.
The practical next step is small: pull fifty labeled questions from your logs, run precision@1 on your current pipeline, add a rerank call over your top 30 with voyageai/rerank-3-lite or an equivalent, and diff the two. If your top-1 accuracy jumps the way the benchmark suggests, keep it; if your chunks were already landing first, you’ve spent an afternoon to learn you don’t need it.