On April 7, 2026, the Allen Institute for Artificial Intelligence (Ai2) released WildDet3D, an open model for monocular 3D detection. Feed it an ordinary RGB photo and it returns 3D bounding boxes for the objects in frame — position, size, and orientation in metric coordinates. No depth sensor required, no per-camera fine-tuning. Model weights, training code, dataset, and the technical report are all public.
Monocular 3D detection is a foundational need for robotics, AR, and spatial computing, yet it has lived in the world of specialist models: switch datasets or cameras and you retrain from scratch. WildDet3D aims to make it a general-purpose primitive — open vocabulary, multiple prompt types, camera-agnostic — so that 3D detection works the way 2D detection already does: off the shelf.
From One Photo to 3D Boxes
WildDet3D has three components. First, a 2D detector built on the SAM3 vision backbone that accepts three prompt types: text queries, point clicks, and existing 2D boxes. Second, a geometry backend — a frozen DINOv2 encoder paired with a trainable depth decoder, with spherical harmonic encodings of camera ray directions replacing the usual camera-calibration branch. Third, a 3D detection head that fuses 2D detections with depth features via cross-attention to emit the final 3D boxes.
Training cost is also unusually low: the full model needs 12 epochs, where prior monocular 3D detectors typically need 80 to 120. For teams planning to fine-tune on their own data, that is a direct compute and labor difference.
Three Prompts, Optional Depth
Interaction is prompt-centric. You can name a category in text (open vocabulary, not a fixed list), click a location in the image, or hand over an existing 2D box; in every case the output is lifted into 3D. Lifting boxes one by one also gives the model zero-shot 3D tracking essentially for free.
Depth is optional. If you have sparse depth from LiDAR, a time-of-flight sensor, or stereo matching, the model will consume it for higher accuracy; without it, the model still runs, just at lower scores. This “use depth when present, compute it when not” flexibility matters most on edge hardware and consumer cameras, and it traces back to an architectural choice: because the depth decoder encodes camera ray directions rather than leaning on a separate calibration branch, the model tolerates camera swaps that would quietly break intrinsics-tuned pipelines.
WildDet3D-Data: A Million-Image Dataset
The companion dataset, WildDet3D-Data, is half the story: more than 1 million images carrying 3.7 million verified 3D annotations across 13,000-plus categories, with over 100,000 images annotated by humans. Source imagery comes from COCO, LVIS, Objects365, and V3Det; candidate annotations were generated with five different 3D estimation methods, then filtered by vision-language models and human reviewers.
Expensive annotation is why 3D data stays scarce. The pipeline here — multi-method estimation plus verification filtering — is a pragmatic way to mass-produce “accurate enough” labels, and it is a methodology other 3D tasks could borrow directly.
The Scores: Big Zero-Shot Gains
On Omni3D, WildDet3D scores 34.2 AP with text prompts, 5.8 above 3D-MOOD, and 36.4 AP with predicted boxes (oracle setting), 2.0 above DetAny3D. Adding sparse depth lifts those to 41.6 and 45.8.
Zero-shot transfer is where the gaps get dramatic. On Argoverse 2 it reaches 40.3 ODS against a prior best of 23.8; on ScanNet, 48.9 ODS — 17.4 higher than the previous method. On Stereo4D, it moves from 7.5 AP without depth to 27.7 AP with real stereo depth. On the team’s own WildDet3D-Bench (700-plus categories), text prompting scores 22.6 AP against a 2.3 baseline, and rare categories show a 47.4-versus-2.4 spread.
Read those zero-shot numbers as the deployment story, not just leaderboard positioning. Argoverse 2 is autonomous-driving imagery and ScanNet is indoor RGB-D — two very different camera regimes — and the model clears both without seeing their training splits. That is the difference between a model you can point at a new robot or a new drone fleet on day one and one you cannot.
What Developers Can Do With It
Everything ships through public channels: a model collection and dataset on Hugging Face, inference code on GitHub (updated April 21 with full training code), a live demo on HF Spaces, and an iOS app for testing against a phone camera. For robotics, AR, or spatial-computing teams, this is a starting point you can wire into a prototype and then fine-tune on your own scenes — and with a 12-epoch training bill, “fine-tune our own version” stops being a project.
Sources
- WildDet3D: Open Monocular 3D Detection — Ai2
- WildDet3D-Data — Hugging Face
- allenai/WildDet3D — GitHub
AI-assisted summary compiled from the sources above, reviewed by a human before publishing.