The constraint that shapes everything
Assistive mobility platforms have a hard problem that most AI deployments can sidestep: the model has to run on the device, in real time, on battery power. A child darting across a sidewalk or a curb appearing suddenly does not wait for a round trip to a cloud endpoint.
According to Meta’s July 27, 2026 post on the RAMMP project, the Human Engineering Research Laboratories (HERL) at the University of Pittsburgh is leading an ARPA-H-supported initiative with up to $41.5 million in funding. The Robotic Assistive Mobility and Manipulation Platform (RAMMP) blends robotics, AI, and user-centered design, with ATDev as an engineering partner. Meta’s open source vision models, DINO and Segment Anything Model (SAM), are part of the perception stack.
The numbers behind the urgency: an estimated 5.5 million wheelchair users in the United States, and over 100,000 wheelchair-related injuries treated in US emergency departments each year, often from trips and falls.
What actually runs on the robot
DINOv3 acts as a compact visual brain, a general-purpose foundation that task-specific lightweight modules can layer on for detection or movement tracking. SAM handles segmentation, identifying and outlining objects with minimal prompting. The RAMMP perception system is built on RF-DETR, a lightweight detection model fine-tuned with DINOv2 embeddings. Training data gets auto-labeled using SAM, which lets the team generate annotations across the angles, heights, backgrounds, and lighting conditions assistive devices actually encounter.
The engineering tradeoffs are concrete. Teams optimize for edge devices by reducing memory footprint, using lower precision where appropriate, and deploying in formats suited to real-world conditions. Running at practical resolutions with efficient batching keeps both models fast on battery-powered hardware. The post is explicit that this sometimes means trading boundary precision or feature detail for the speed and stability users need.
Sivashankar Sivakanthan, Chief of Staff to the RAMMP project, frames the standard: performance is not measured by benchmark accuracy alone, but by whether a system operates reliably in the unpredictability of everyday life.
Natural language as the interface layer
One of the more interesting design moves is using the user’s immediate surroundings as context. Instead of building complex interfaces, the system lets users query the robot’s image sensors with natural language combined with image data. That reduces cognitive load and context switching for something as simple as picking up a cup.
The first prototype already uses DINO-based tools to detect automatic door buttons, cups, and curbs for navigation assistance. Engineers are now working on voice and touch input so users can select and interact with specific objects. That introduces a new robustness problem: ensuring predictable behavior across varied user prompts and inputs, on top of the existing challenges of accuracy and temporal coherence.
If you are building agentic tools that mix language and perception, this is the same class of problem covered in what a hosted sandboxed shell changes for agentic tools — the model output is only half the system; the interface and validation layer determines whether it is usable.
Where this goes next
The RAMMP consortium includes Kinova Robotics, LUCI Mobility, ATDev, and academic partners at Carnegie Mellon, Cornell, Northeastern, and Purdue. HERL leads with biomedical engineering and user-centered research; ATDev handles making the research function in real devices. The participatory approach engages wheelchair users, clinicians, and advocacy groups throughout design.
Future work focuses on strengthening temporal consistency, robustness across diverse real-world conditions, and tighter integration with decision-making and control systems. The team plans to advance integration of next-generation perception models including SAM 3.1 and DINOv3.
For builders, the practical takeaway is not the model names. It is the pattern: pick a general-purpose vision foundation, fine-tune a lightweight detector on top, auto-label with a segmentation model, then aggressively optimize for the hardware you actually have. The post does not specify exact latency numbers or power budgets, so treat the precision-versus-stability tradeoff as a design principle rather than a measured threshold. The harder work is deciding what accuracy you can give up before the system stops being trustworthy to the person relying on it.
Sources
AI-assisted summary compiled from the sources above, reviewed by a human before publishing.