feat(manip): draw the ranked grasp proposals in viser

[mustafab0]

A learned provider returns up to a hundred grasps in a scored order, and until now the only way to see them was get_grasp_candidates, an RPC that answers after the fact. That is no help while the arm is choosing between them, and no help at all when the question is whether the generator is pointing anywhere sane.

PickAndPlaceModule publishes its proposals on a stream as it makes them. ManipulationModule subscribes and hands them to the viewer through the world monitor, the same path obstacles and previews already take — the grasp pipeline is a sibling module here and owns no visualizer of its own. VisualizationSpec grows one hook for it; meshcat implements it as a no-op, because it draws the planning world and these are not planning geometry.

The scene draws each proposal as a pose glyph — approach axis, closing axis across it — coloured best-green through worst-orange, with the top three thicker and labelled with their score, capped at twenty because a hundred glyphs bury the ranking they exist to show. It is deliberately a glyph and not a gripper: a jaw-shaped drawing sized by anything other than the configured sweep volume would be a claim about the hardware that nothing checks, and the branch this stack re-derives from had exactly that — a to_rerun on GraspCandidateArray with its own hardcoded _JAW_HALF_WIDTH and _FINGER_LENGTH, duplicating the real gripper config. That is not ported. Neither is the ~800-line backend-neutral layers framework the branch grew around it, whose single consumer was this overlay; this is ~250 lines in the package's existing idiom instead.

One test binds the drawn arrays against the real viser.SceneApi signature and asserts their shapes, because a mocked server accepts any array and the first version of this shipped colours as (N, 3) where viser wants (N, 2, 3) — which surfaced only as a swallowed Grasp proposal visualization failed in a live run.

Verified in the grasp sim: proposals reach the scene and render with no errors in the log, and the panel checkbox toggles them.

Last of nine in the xArm grasping re-landing stack.

[codecov]

❌ 1 Tests Failed:

Tests completed	Failed	Passed	Skipped
4757	1	4756	41

View the top 1 failed test(s) by shortest run time

dimos.utils.test_shm::test_create_or_attach_splits_owner_and_reader

Stack Traces | 0.922s run time

name = 'dimos_test_7879191eac0d'
slow_ftruncate = <function slow_ftruncate.<locals>.delayed at 0x7f78cfd95f80>

    def test_create_or_attach_splits_owner_and_reader(name, slow_ftruncate):
        """Concurrent symmetric callers: exactly one owns, the other waits and attaches."""
        out: list[tuple[bool, int]] = []
        lock = threading.Lock()
        barrier = threading.Barrier(2)
    
        def racer() -> None:
            barrier.wait()
            shm, owner = create_or_attach_shm(name, SIZE, timeout=5.0)
            with lock:
                out.append((owner, shm.size))
            shm.close()
    
        threads = [threading.Thread(target=racer, daemon=True) for _ in range(2)]
        for t in threads:
            t.start()
        for t in threads:
            t.join(timeout=8)
    
        assert len(out) == 2, f"a racer did not finish: {out}"
        assert sorted(owner for owner, _ in out) == [False, True]
>       assert {size for _, size in out} == {SIZE}
E       assert {0, 65536} == {65536}
E         
E         Extra items in the left set:
E         #x1B[0m#x1B[94m0#x1B[39;49;00m#x1B[90m#x1B[39;49;00m
E         
E         Full diff:
E         #x1B[0m#x1B[90m #x1B[39;49;00m {#x1B[90m#x1B[39;49;00m
E         #x1B[92m+     0,#x1B[39;49;00m#x1B[90m#x1B[39;49;00m
E         #x1B[90m #x1B[39;49;00m     65536,#x1B[90m#x1B[39;49;00m
E         #x1B[90m #x1B[39;49;00m }#x1B[90m#x1B[39;49;00m

barrier    = <threading.Barrier at 0x7f78dd324f50: waiters=0/2>
lock       = <unlocked _thread.lock object at 0x7f7945c24b80>
name       = 'dimos_test_7879191eac0d'
out        = [(True, 65536), (False, 0)]
racer      = <function test_create_or_attach_splits_owner_and_reader.<locals>.racer at 0x7f78cfd96020>
slow_ftruncate = <function slow_ftruncate.<locals>.delayed at 0x7f78cfd95f80>
t          = <Thread(Thread-2606 (racer), stopped daemon 140156511536832)>
threads    = [<Thread(Thread-2605 (racer), stopped daemon 140153691514560)>, <Thread(Thread-2606 (racer), stopped daemon 140156511536832)>]

dimos/utils/test_shm.py:148: AssertionError

To view more test analytics, go to the Test Analytics Dashboard
_{📋 Got 3 mins? Take this short survey to help us improve Test Analytics.}

[greptile-apps]

Greptile Summary

This change forwards ranked grasp proposals into the manipulation visualizer and renders labeled pose overlays. One correctness issue remains: a proposal is broadcast before its coordinate frame is checked, so a rejected proposal from another frame can still be rendered at incorrect coordinates. Publish proposals only after validating that they use the planning frame.

T-Rex validation blocked

The focused reproduction could not run because the required Python tool at /home/user/repo/.venv/bin/python has a missing interpreter target. The fallback Python installation lacks the dimos package, so it could not exercise the pick-to-visualizer flow.

Confidence Score: 4/5

Not ready to merge until rejected grasp proposals can no longer reach the visualizer.

The picker publishes proposals before frame validation, and the downstream forwarding and rendering paths do not transform or reject those coordinates. The focused runtime check could not execute because the available Python environments cannot run this repository.

Files Needing Attention: dimos/manipulation/pick_and_place_module.py needs publication moved after frame validation; dimos/manipulation/manipulation_module.py and dimos/manipulation/visualization/viser/scene.py receive and render the invalid publication.

T-Rex Logs

What T-Rex did

• A focused reproduction was authored to invoke the picker, manipulation receiver, monitor callback, and Viser scene with camera-framed proposals against a world planning frame.
• The reproduction could not start due to environment issues: the virtual-environment interpreter path could not be started (exit 127), and the fallback interpreter reported a missing 'dimos' module (exit 1).
• Code-path validation shows that candidates are stored before header.frame_id checks, and that the visualization renders candidate glyphs and labels without consulting header.frame_id, indicating the environment issue rather than the code path in question.

_{Ran code and verified through T-Rex}

Comments Outside Diff (1)

1. dimos/manipulation/pick_and_place_module.py, line 140-146 (link)

   Rejected frames remain visible

   pick_object publishes generated proposals before confirming that their frame_id matches planning_frame. A generator returning poses in another frame therefore sends those untransformed coordinates to the visualizer even though the pick is rejected with GRASP_FRAME_MISMATCH, leaving incorrectly positioned grasp markers visible. Validate the frame before publishing the proposal array.

_{Reviews (1): Last reviewed commit: "feat(manip): draw the ranked grasp propo..." | Re-trigger Greptile}