Summary: OpenRouter launched Visual Image Benchmarks on August 21, 2026 to compare the 39 image models then available across seven challenge families, with results sortable by price and generation time. The benchmark is intended to build a shortlist for capabilities such as text, counting, spatial relations, editing, and consistency, not to replace tests on a product’s own prompts. It is static and does not cover style, photorealism, domain knowledge, reliability, or rate limits.
The Problem: Choosing an Image Model Feels Arbitrary
Reach for a text model and there is an enormous library of LLM benchmarks to consult. Reach for an image model and that library is not there. In its August 21, 2026 announcement, OpenRouter notes that vendor output samples tend to be “curated eye candy,” and that LLM-as-a-judge evals still miss details a person would catch in an instant. Arena scores are worth something, but what they measure is which output people prefer — not what a model can actually do.
For anyone building a product, that gap bites. If a feature depends on dependable text rendering, an accurate object count, or more than one language inside a single image, sample images can lead you to the wrong conclusion. What you need is a way to compare models on the capabilities your use case actually rests on.
What OpenRouter’s Image Benchmarks Offer
On August 21, 2026, OpenRouter launched Visual Image Benchmarks so you can size up every image model they carry (39 as of August ‘26) without doing the legwork yourself. A set of deliberately challenging prompts is meant to separate models by capability, and every result lands in a grid you can sort by price as well as by generation time.
So you get side-by-side outputs for the same prompt, orderable by cost or speed, and the difference in how models handle a challenge shows up at a glance. The prompts themselves are written to be judged by eye. Take the wine glass: can a model follow the instruction and fill it all the way to the rim? Each entry in the grid pairs a model’s output with its price and generation time, which makes it easy to see who nails the task and who misses.
The Seven Challenge Families
Prompts are grouped into seven families, each one aimed at a particular capability boundary:
- Improbable scenes. A wine glass filled level with the rim, umbrellas that are closed. Both appear in training data only in their ordinary form, so a model has to reach past memorized patterns.
- Counting. Three fingers, specific numbers of cards and dice. The question is whether a model can reproduce a quantity accurately.
- Text. One long exact string on a poster, plus several languages in the same frame. Anyone whose product needs legible text in images should care about this one.
- Spatial relations. Occlusion and mirror reflections. This probes how well a model handles physical space and the way objects interact.
- Negation. A zebra with no stripes, a Times Square with no advertising. Can the model follow a negative instruction?
- Editing. Minimal diffs, object removal, person removal, all starting from a reference image. Core to any image editing workflow.
- Consistency. Holding a product or four reference subjects steady across a new scene. It matters for brand consistency and for character consistency.
Every prompt is built so the result can be judged visually in an instant. The wine glass challenge, for instance, shows whether a model obeys the instruction to fill the glass to the rim—a small test that says a lot.
How to Use the Benchmarks in Your Workflow
For product builders, the payoff is a shortlist you can assemble fast. A workable sequence:
- Browse the benchmark grids. Open the image benchmarks page and see how models do across the seven families. Sort by price or generation time to narrow things down to what your budget and latency can live with.
- Identify models that excel in your relevant categories. Text rendering in your product? Start with the “Text” family. Need consistent characters? Look at the “Consistency” family.
- Test your own prompts. With a few models shortlisted, run your own prompts through the image generation API or Chat and watch how they do on your content.
Screening with benchmarks first and testing custom prompts second is what keeps you out of the trap of trusting curated samples alone.
Limitations and Trade-offs
These benchmarks are a real step forward, and they still have limits. The prompts are hard by design, but they cannot cover every use case — style mimicry, photorealism, and domain-specific knowledge, for instance, go untested. The benchmarks are also static: they capture model capabilities as of August 2026. Models ship updates often, so a model that struggles today may do better tomorrow.
A second trade-off: the focus is visual output quality, not API reliability, rate limits, or cost per image. Those still belong in your final decision.
OpenRouter says evaluating video and audio models is just as hard, and it plans to extend the tool across modalities while keeping it current as new image models arrive. Expect it to keep changing.
The Takeaway: Use Benchmarks as a Starting Point
For product builders, OpenRouter’s image benchmarks are a useful new instrument: a transparent, side-by-side view of what models can do across seven challenge families, sortable by price and generation time. That makes it faster to cut the field down to the models likely to fit your needs.
Keep one thing in mind, though — a benchmark is a starting point. It builds your shortlist; the final call should rest on testing with your own content, particularly when brand text, specific quantities, or complex layouts are involved. Treat the benchmarks as a decision aid rather than the only answer. Put benchmark signal together with your own testing, and you are better equipped to land on the right image model for your product.