Self-Evaluated Robot Mapping System Struggles With Outdoor Region Recognition

TL;DR

A team at Brigham Young University built a mapping system for outdoor robots, called Terra 3DSG, that combines 3D maps with AI-generated labels for objects and places. They tested it themselves, not an outside lab, across five real outdoor sites. The robot found the object it was looking for most of the time, but its routes were often far from the shortest path, and it struggled to correctly answer questions about named regions like “near the dock.” The maps themselves stayed small even over long treks. Because the same team built and graded the system, no outside group has confirmed these numbers.

What happened

Researchers Chad R. Samuelson, Gabriel R. Slade, and Joshua G. Mangelson of Brigham Young University’s FROST Lab tested how 3D scene graphs — hierarchical maps that combine 3D geometry from lidar or cameras with AI-generated labels for objects and places — behave outdoors, using their own system, Terra 3DSG, as the test case across five real-world robotic datasets: River Park, Nunns Park, Marina Part 1, Marina Part 2, and Rock Canyon Campground.

The system labels the environment with open-set vision-language models — AI models that can name objects they were never explicitly trained to recognize, rather than picking from a fixed list. Across all five datasets, the researchers found that the numerical representations these models produce for each point (called embeddings) frequently split into multiple clusters or drifted into outliers, with an outlier ratio above 0.1 for around 30% of points.

When the robot used the scene graph to find and retrieve named objects, it succeeded 69.1% of the time under a relaxed definition of success and 54.8% under a strict one, with failures driven mainly by the robot being sent toward locations it physically could not reach and by routes roughly 66% less efficient than the optimal path. “Navigation failure occurs when the planned path directs the robot to go to a location that is not-traversable for our ground robot to navigate to,” the authors wrote.

Asking the system about named regions of an environment, rather than individual objects, proved harder still: the average F1 score — a standard accuracy measure balancing false positives against missed detections — came out to around 0.359. “The inherent ambiguity and homogeneity of regions in complex outdoor scenes are likely to present a significant challenge for region detection and querying,” the authors wrote.

Despite these failure rates, the maps stayed compact: scene graph files stayed under 600 MB even for robot trajectories spanning multiple kilometers.

The study was explicit about what it did not test. The ground robot was a manually driven wheelchair-based platform rather than a fully autonomous one, it used GPS instead of the scene graph itself for localization, and it did not evaluate moving objects or seasonal change; long-term map stability was checked across only four repeated visits to a single location. The paper has been accepted for publication in IEEE Transactions on Field Robotics, and the work was funded by the Office of Naval Research and the NSF Industry-University Cooperative Research Center CAAMS.

What this means (and what it does not)

The results are a rare, detailed account of how one outdoor scene-graph system behaves once it leaves the lab: it finds requested objects most of the time, but its region-level reasoning and path planning both show clear room for improvement. The compact map size is a real engineering result on its own — a scene graph under 600 MB for a multi-kilometer trajectory matters regardless of the accuracy numbers, since memory limits are a practical constraint for any robot on a long deployment.

What doesn’t follow is that these specific numbers describe outdoor scene-graph systems in general, or even Terra 3DSG under different conditions. Samuelson, Slade, Mangelson and BYU’s FROST Lab built Terra 3DSG and then ran this evaluation of it themselves; every quantitative claim in the paper comes from that one group, even though the framing is self-critical rather than promotional. The Office of Naval Research and the NSF’s CAAMS center, which funded the work, and IEEE Transactions on Field Robotics, which accepted it for publication, each have their own interest in the underlying research program looking productive and field-relevant. None of that makes the numbers wrong, but it means no outside party has checked them.

It also doesn’t show Terra 3DSG is ready for autonomous outdoor use: the robot was driven by a person, not driving itself, and it located itself with GPS rather than with the scene graph it was building. The study says nothing about how these failure modes would play out in a robot that has to navigate and localize entirely on its own. And the claim that Terra “matches” or “exceeds” other outdoor scene-graph systems on some tasks comes from the same authors’ earlier paper introducing Terra, not from this field evaluation, which included no head-to-head test against competing systems.

The struggles this paper documents don’t appear to be unique to Terra 3DSG, either. A separate, unaffiliated survey of the wider field — co-authored by researchers including Luca Carlone and Nathan Hughes — describes 3D scene graph research overall as split across incompatible approaches: “the field remains fragmented: different communities adopt distinct formulations, construction pipelines, and evaluation protocols, making it difficult to compare methods,” and names robust real-world deployment as an open challenge for the field as a whole.

What we still do not know

• No independent lab has re-run or replicated the specific numbers in this report — the roughly 30% embedding-outlier rate, the 69.1%/54.8% navigation success rates, the 66% path-inefficiency figure, or the 0.359 region F1 score. All of them come solely from the team that built the system being evaluated.
• The paper does not describe weather conditions — rain, fog, lighting, seasonal change — across its five test sites, so whether these failure rates hold up, or worsen, in different weather is untested.
• Because the robot was driven manually and localized with GPS, the study does not show how these same failure modes would play out in a robot navigating and localizing itself using only the scene graph.
• The field report does not benchmark Terra 3DSG against other outdoor scene-graph systems under the same conditions; the only comparative performance claim comes from the team’s own earlier paper introducing Terra.
• The available material reports only an aggregate statistic — 30% of points flagged as outliers — with no breakdown of which specific object categories the open-set recognition succeeded or failed on.
• Long-term map stability was checked across only four repeated visits to one location, leaving open how the system holds up over longer stretches of time or many more repeat visits.