refactor(perception): replace OSR tracking with Dan localization

[ruthwikdasyam]

Contribution path

• Linked issue or discussion: None; this is an internal perception/manipulation refactor submitted as a draft for review.

Problem

The xArm object scene registration path maintained a second object-tracking and persistence stack beside Dan perception. Pick-and-place consumed stable OSR UUIDs and then made another perception RPC for the point cloud, which allowed a pick to use geometry that differed from the scan result. The real xArm stack also lacked a published camera mount edge needed to resolve camera observations into the world frame.

Solution

• Replace OSR detection, promotion, and ObjectDB lookup APIs with one warm DanDetector over bounded RGB-D, camera-info, and split-TF history.
• Expose one batched localize_objects(prompts) RPC that returns request-scoped Localization results and rejects ambiguous, stale, or frame-inconsistent geometry.
• Make pick-and-place return dense integer selections from each scan and cache each selection's complete localization, including the exact point cloud used for grasp generation.
• Remove OSR-only stable identity, legacy grasp orchestration, and ObjectDB-backed manipulation obstacle plumbing while preserving explicit collision APIs, voxel maps, floor obstacles, and unrelated WorldBelief/ObjectDB consumers.
• Publish the calibrated real xArm link7 -> camera_link mount transform repeatedly and configure explicit real and simulated optical frames.
• Reconcile the headless xArm room simulation, agent prompt, and documentation with Dan localization and integer selections.

Key tradeoffs:

• Object selections intentionally expire when a new scan replaces the request snapshot; they are not persistent identities.
• Dan's EdgeTAM segmenter requires CUDA or MPS. Real camera timing and mount calibration still need on-robot validation.

How to Test

Run the headless xArm room stack on a CUDA/MPS-capable host:

MUJOCO_GL=egl LIBGL_ALWAYS_SOFTWARE=true MESA_LOADER_DRIVER_OVERRIDE=llvmpipe dimos --viewer none run xarm-room-sim

In dimos shell:

from dimos.robot.manipulators.xarm.blueprints.simulation import XARM_ROOM_PROMPTS
app.PickAndPlaceModule.scan_objects(XARM_ROOM_PROMPTS)

Validation completed:

• 123 focused manipulation, OSR, grasp, monitor, and xArm blueprint tests passed.
• Strict mypy passed across 1,034 source files.
• Ruff check and format passed.
• CI-mode blueprint registry validation passed.
• Diff-scoped pre-commit passed.
• pytest-fast: 4,725 passed and 15 skipped. Two unrelated relay static-file tests still fail with existing HTTP 404 behavior.

AI assistance

OpenCode with GPT-5.6-sol was used extensively for codebase analysis, implementation, conflict resolution, tests, and PR preparation. The author is responsible for reviewing the resulting changes.

Checklist

• I have read and approved the CLA.

[codecov]

❌ 1 Tests Failed:

Tests completed	Failed	Passed	Skipped
4541	1	4540	176

View the top 1 failed test(s) by shortest run time

dimos.robot.manipulators.xarm.blueprints.test_perception::test_real_and_simulation_use_their_optical_frames

Stack Traces | 6.96s run time

def test_real_and_simulation_use_their_optical_frames() -> None:
        assert _osr_config(xarm_perception).optical_frame == "camera_color_optical_frame"
>       assert _osr_config(xarm_perception_sim).optical_frame == "wrist_camera_color_optical_frame"


.../xarm/blueprints/test_perception.py:47: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
.../xarm/blueprints/test_perception.py:36: in _osr_config
    parsed = BlueprintConfigParser(blueprint).parse(environ={})
        atom       = BlueprintAtom(kwargs={'target_frame': 'world', 'optical_frame': 'wrist_camera_color_optical_frame'}, module=<class 'di...ame='tf', type=<class 'dimos.msgs.tf2_msgs.TFMessage.TFMessage'>, direction='in')), module_refs=(), instance_name=None)
        blueprint  = <[RuntimeError("Failed to pull LFS file .../dimos/data/.lfs/xarm7.tar.gz after 3 attempts: Command.../xarm7.tar.gz', '--exclude=']' returned non-zero exit status 1.") raised in repr()] Blueprint object at 0xff2274910fb0>
.../coordination/blueprint_config/parser.py:168: in parse
    key: plain(value)
        cli_tokens = ()
        config_path = None
        environ    = {}
        global_overrides = None
        overrides  = None
        schema     = <[RuntimeError("Failed to pull LFS file .../dimos/data/.lfs/xarm7.tar.gz after 3 attempts: Command...rm7.tar.gz', '--exclude=']' returned non-zero exit status 1.") raised in repr()] ParserSchema object at 0xff22559cbe80>
        self       = <dimos.core.coordination.blueprint_config.parser.BlueprintConfigParser object at 0xff22d92ff500>
.../coordination/blueprint_config/values.py:45: in plain
    if isinstance(value, BaseModel):
        value      = <[RuntimeError("Failed to pull LFS file .../dimos/data/.lfs/xarm7.tar.gz after 3 attempts: Command...fs/xarm7.tar.gz', '--exclude=']' returned non-zero exit status 1.") raised in repr()] LfsPath object at 0xff22e4a928d0>
dimos/utils/data.py:370: in __getattribute__
    resolved = object.__getattribute__(self, "_ensure_downloaded")()
        name       = '__class__'
        self       = <[RuntimeError("Failed to pull LFS file .../dimos/data/.lfs/xarm7.tar.gz after 3 attempts: Command...fs/xarm7.tar.gz', '--exclude=']' returned non-zero exit status 1.") raised in repr()] LfsPath object at 0xff22e4a928d0>
dimos/utils/data.py:353: in _ensure_downloaded
    cache = get_data(filename)
        cache      = None
        filename   = 'xarm7/scene.xml'
        self       = <[RuntimeError("Failed to pull LFS file .../dimos/data/.lfs/xarm7.tar.gz after 3 attempts: Command...fs/xarm7.tar.gz', '--exclude=']' returned non-zero exit status 1.") raised in repr()] LfsPath object at 0xff22e4a928d0>
dimos/utils/data.py:310: in get_data
    archive_path = _decompress_archive(_pull_lfs_archive(archive_name))
        archive_name = 'xarm7'
        data_dir   = PosixPath('.../dimos/dimos/data')
        file_path  = PosixPath('.../dimos/dimos/data/xarm7/scene.xml')
        name       = 'xarm7/scene.xml'
        nested_path = PosixPath('scene.xml')
        path_parts = ('xarm7', 'scene.xml')
dimos/utils/data.py:254: in _pull_lfs_archive
    _lfs_pull(file_path, repo_root)
        file_path  = PosixPath('.../dimos/data/.lfs/xarm7.tar.gz')
        filename   = 'xarm7'
        repo_root  = PosixPath('.../work/dimos/dimos')
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

file_path = PosixPath('.../dimos/data/.lfs/xarm7.tar.gz')
repo_root = PosixPath('.../work/dimos/dimos')

    def _lfs_pull(file_path: Path, repo_root: Path, *, retries: int = 2) -> None:
        relative_path = file_path.relative_to(repo_root)
    
        env = os.environ.copy()
        env["GIT_LFS_FORCE_PROGRESS"] = "1"
    
        last_err: subprocess.CalledProcessError | None = None
        for attempt in range(1, retries + 2):  # retries + 1 total attempts
            try:
                subprocess.run(
                    # --exclude= overrides lfs.fetchexclude from .lfsconfig, which
                    # otherwise silently skips data/.lfs/* even when --include matches.
                    ["git", "lfs", "pull", "--include", str(relative_path), "--exclude="],
                    cwd=repo_root,
                    check=True,
                    env=env,
                )
                return
            except subprocess.CalledProcessError as e:
                last_err = e
                if attempt <= retries:
                    time.sleep(attempt)  # 1s, 2s backoff
    
>       raise RuntimeError(
            f"Failed to pull LFS file {file_path} after {retries + 1} attempts: {last_err}"
        )
E       RuntimeError: Failed to pull LFS file .../dimos/data/.lfs/xarm7.tar.gz after 3 attempts: Command '['git', 'lfs', 'pull', '--include', 'data/.lfs/xarm7.tar.gz', '--exclude=']' returned non-zero exit status 1.

attempt    = 3
env        = {'ACCEPT_EULA': 'Y', 'ACTIONS_ID_TOKEN_REQUEST_TOKEN': 'eyJhbGciOiJSUzI1NiIsImtpZCI6IjM4ODI2YjE3LTZhMzAtNWY5Yi1iMTY5LT...-version=2.0', 'ACTIONS_ORCHESTRATION_ID': '59f85c6c-405a-4eb6-9613-6d3cf2f2ade1.tests.ubuntu-24_04-arm_3_14_fal', ...}
file_path  = PosixPath('.../dimos/data/.lfs/xarm7.tar.gz')
last_err   = CalledProcessError(1, ['git', 'lfs', 'pull', '--include', 'data/.lfs/xarm7.tar.gz', '--exclude='])
relative_path = PosixPath('data/.lfs/xarm7.tar.gz')
repo_root  = PosixPath('.../work/dimos/dimos')
retries    = 2

dimos/utils/data.py:224: RuntimeError

To view more test analytics, go to the Test Analytics Dashboard
_{📋 Got 3 mins? Take this short survey to help us improve Test Analytics.}