Summary: Ai2 released WildDet3D on April 7, 2026 as an open monocular 3D detector that turns ordinary RGB photos into metric 3D bounding boxes without depth sensors or camera-specific tuning. It ships with more than 1 million images and 3.7 million verified 3D annotations, and reports 34.2 AP with text prompts on Omni3D.
WildDet3D is the open model the Allen Institute for Artificial Intelligence (Ai2) put out on April 7, 2026, built for monocular 3D detection. Give it any ordinary RGB photo and it returns 3D bounding boxes for the objects in frame — each carrying position, size, and orientation in metric coordinates. The setup needs neither a depth sensor nor camera-specific fine-tuning. Weights, training code, the dataset, and the technical report have all been made public.
Robotics, AR, and spatial computing all lean on monocular 3D detection, yet specialist models have long owned the task: change the dataset or the camera and training starts over. The wager behind WildDet3D is that this should be a general-purpose building block — open vocabulary, several prompt types, indifferent to the camera — so 3D detection can be picked up the way 2D detection already is: ready to use.
One Photo In, 3D Boxes Out
Three pieces make up the model. Leading is a 2D detector on the SAM3 vision backbone, which takes three kinds of prompts: text queries, clicked points, and 2D boxes you already have. Next is the geometry backend, a DINOv2 encoder kept frozen and joined to a trainable depth decoder; the decoder encodes camera ray directions with spherical harmonics, so the customary camera-calibration branch can be dropped. A 3D detection head comes last, running cross-attention to merge the 2D detections with depth features and produce the final 3D boxes.
The training bill is unusually light as well: the whole model needs 12 epochs, while earlier monocular 3D detectors generally demand 80 to 120. Teams intending to fine-tune on their own data will feel that gap directly, in compute and labor.
Three Ways to Prompt, Depth When You Have It
Prompts drive the interaction. Spell out the category in text — the vocabulary is open rather than a fixed list — or click a spot in the image, or pass along a 2D box you already hold; every route ends with the result lifted into 3D. Because each 2D box is lifted one at a time, zero-shot 3D tracking falls out of the design.
Depth is your call. Sparse depth from LiDAR, a time-of-flight sensor, or stereo matching can be fed in and buys extra accuracy; leave it out and the model still runs, only with weaker numbers. That “use depth when present, compute it when not” quality matters most on edge hardware and consumer cameras, and it follows from an architectural decision: the depth decoder encodes camera ray directions rather than relying on a separate calibration branch, so the model shrugs off camera swaps that would quietly break pipelines tuned to fixed intrinsics.
WildDet3D-Data: A Dataset of a Million-Plus Images
The dataset arriving alongside it matters just as much. WildDet3D-Data holds more than 1 million images carrying 3.7 million verified 3D annotations across 13,000-plus categories, and humans annotated over 100,000 of those images themselves. The imagery is drawn from COCO, LVIS, Objects365, and V3Det; candidates were produced by five different 3D estimation methods, then filtered by vision-language models together with human reviewers.
Scarce 3D data traces straight back to the price of annotation. This pipeline — estimation from multiple methods plus verification filtering — is a pragmatic route to mass-producing “accurate enough” labels, and other 3D tasks can adopt the methodology as-is.
The Numbers: Zero-Shot Margins That Stand Out
With text prompts on the Omni3D test set, WildDet3D posts 34.2 AP, 5.8 clear of 3D-MOOD; with predicted boxes under the oracle setting it records 36.4 AP, 2.0 ahead of DetAny3D. Supply sparse depth and both figures climb, to 41.6 and 45.8.
Zero-shot transfer is where the gaps turn dramatic. Argoverse 2 returns 40.3 ODS with the prior best at only 23.8; ScanNet yields 48.9 ODS, 17.4 beyond the earlier method. On Stereo4D the model goes from 7.5 AP with no depth to 27.7 AP once real stereo depth is supplied. On WildDet3D-Bench, the team’s own benchmark spanning 700-plus categories, text prompting lands 22.6 AP against a 2.3 baseline, and on rare categories the spread widens to 47.4 versus 2.4.
Read those zero-shot results as a deployment story rather than leaderboard trivia. Argoverse 2’s imagery comes from autonomous driving and ScanNet’s from indoor RGB-D — two camera regimes far apart — yet the model clears both without ever seeing their training splits. That is what separates a model you can point at a new robot or a fresh drone fleet on day one from one you cannot.
How Developers Can Put It to Work
Everything travels through public channels: Hugging Face hosts the model collection and the dataset, GitHub carries the inference code (an April 21 update added the full training code), HF Spaces runs a demo you can try on the spot, and an iOS app lets you test against a phone camera. Teams in robotics, AR, or spatial computing can treat this as a starting point — wire it into a prototype, then fine-tune on your own scene data — and at a 12-epoch training bill, “fine-tune our own version” stops being a project.