feat(manipulation): GraspGenX provider for the xArm grasp sim - #3842
Open
mustafab0 wants to merge 13 commits into
Open
feat(manipulation): GraspGenX provider for the xArm grasp sim#3842mustafab0 wants to merge 13 commits into
mustafab0 wants to merge 13 commits into
Conversation
Adds xarm-grasp-sim-graspgenx, a blueprint variant that swaps HeuristicGraspModule for GraspGenXModule over the shared room scene, and renames xarm-room-sim to xarm-grasp-sim to match data/xarm_grasp_sim. Also fixes two pre-existing defects the learned provider exposed: the grasp scene inherited the data/xarm7 pedestal base_pose, putting the planning model 12cm above the arm MuJoCo simulates, and pick_object only ever tried the top-ranked candidate.
mustafab0
requested review from
Dreamsorcerer,
aclauer,
leshy,
paul-nechifor and
spomichter
as code owners
August 31, 2026 22:25
Codecov Report❌ Patch coverage is @@ Coverage Diff @@
## main #3842 +/- ##
==========================================
- Coverage 77.58% 77.40% -0.19%
==========================================
Files 1315 1320 +5
Lines 124697 125376 +679
Branches 10867 10970 +103
==========================================
+ Hits 96748 97048 +300
- Misses 24816 25188 +372
- Partials 3133 3140 +7
Flags with carried forward coverage won't be shown. Click here to find out more.
... and 3 files with indirect coverage changes 🚀 New features to boost your workflow:
|
Contributor
Greptile SummaryThis change adds heuristic and GraspGenX xArm grasp-simulation blueprints, aligns the simulation planning model with the scene origin, and retries planning with later ranked grasps. It removes the existing Confidence Score: 4/5Not safe to merge until the renamed xArm simulation command remains compatible with existing invocations or has a supported migration path. The command removal was confirmed by comparing the parent registry and CLI behavior with the current revision: the former name was registered before this change and is rejected afterward. No security consequence was observed. Files Needing Attention:
 T-Rex LogsWhat T-Rex did
Reviews (1): Last reviewed commit: "feat(manipulation): GraspGenX provider f..." | Re-trigger Greptile |
![greptile-apps[bot]](https://avatars.githubusercontent.com/in/867647?s=60&v=4){kind=small}
| "unitree-go2-webrtc-keyboard-teleop": "dimos.robot.unitree.go2.blueprints.basic.unitree_go2_webrtc_keyboard_teleop:unitree_go2_webrtc_keyboard_teleop", | ||
| "unitree-go2-webrtc-rage-keyboard-teleop": "dimos.robot.unitree.go2.blueprints.basic.unitree_go2_webrtc_rage_keyboard_teleop:unitree_go2_webrtc_rage_keyboard_teleop", | ||
| "unity-sim": "dimos.simulation.unity.blueprint:unity_sim", | ||
| "xarm-grasp-sim": "dimos.robot.manipulators.xarm.blueprints.simulation:xarm_grasp_sim", |
Contributor
There was a problem hiding this comment.
xarm-room-sim was a registered, documented command before this change, but its registry key has been removed. Running dimos run xarm-room-sim --help now exits with Unknown blueprint or module: xarm-room-sim; the fuzzy-match suggestion does not preserve scripts or automation. Keep the old key as an alias for xarm_grasp_sim (with a deprecation notice if appropriate) so existing invocations continue to work.
Artifacts
- Executed shell script that creates a parent worktree and compares registry membership plus the real CLI lookup on the parent and HEAD; it captures the precise commands and output.
Before registry and CLI output
- Parent-commit capture showing that `xarm-room-sim` is registered and maps to `xarm_room_sim`; the later blueprint import is blocked by the minimal environment lacking the optional `ultralytics` dependency.
- HEAD capture showing that `xarm-room-sim` is absent from the registry and the real `dimos run xarm-room-sim --help` command exits 1 with an unknown-blueprint error.
Wires the wrist camera through PointCloudSelfFilter and RayTracingVoxelMap into ManipulationModule.voxel_map, so the planner avoids mapped geometry rather than only registered objects. Three fixes were needed to make the chain carry data: the sim's pointcloud output is off by default, the self filter needs TF for every collision link and drops a whole cloud when one is missing, and the two edges whose stream names differ are remapped. Because the target object is itself mapped, a collision-checked plan into it can only be rejected. The pregrasp-to-grasp leg and the retreat now use move_linear with collision checking off, which is what that primitive is for.
…era mount The self filter looked up transforms with a 20ms tolerance against robot tf published at 10Hz, so a ~92ms publish period could not be bracketed and 61% of clouds were dropped, each with a warning pair. One full period admits them all: measured 0 drops over 20s. Adds xarm-grasp-sim-agent and xarm-grasp-sim-graspgenx-agent. Both need n_workers=6 -- the scene already carries a detector, a segmenter and a voxel mapper, and packing those with the agent into one worker makes the detector's lazy transformers import fail. Publishes the xArm's eye-in-hand camera mount edge, which resolves the TODO in the hardware perception blueprint: RealSenseCamera publishes only its own subtree, so camera_link had no parent and the existing hand-eye calibration constant was dead code. Also documents how to actually pick, which the walkthrough never showed.
… viewer on The grasp sim shipped with visualization backend "none", so there was no viewer at all. Turning it on exposed the second half: #3714 added the OCTREE obstacle type and its planner support but no rendering for it, in viser or anywhere else, so the scene's dispatch raised "unsupported obstacle type" and drew a grey proxy box 78 times instead of the map. Occupied cells now draw as a point cloud sized to the cell edge. One box per cell would be tens of thousands of scene nodes; square points at the cell edge read as the same grid for one node.
Collapses xarm-grasp-sim, xarm-grasp-sim-graspgenx and xarm-perception into xarm-grasp and xarm-grasp-graspgenx. The two differ only in which GraspGenSpec provider they compose; everything else is shared, including the arm-versus-sim split, which is decided at import time the way mujoco_if_sim already does it. Hardware is the default and --simulation swaps the hardware adapter, the base pose, the camera, the detector backends and the home pose. That also gives the hardware path a coordinator, which xarm-perception never had -- it could plan but had nothing to drive the arm with -- plus the voxel map, which is now always composed rather than sim-only.
dimos-module, the ray tracing mapper and the mls planner all moved to validator 0.21; the realsense crate stayed on 0.20. Both landed in its dependency graph, so its Config derived Validate from 0.20 while Module::Config wanted 0.21 and the build failed on an unsatisfied trait bound. This blocks any blueprint composing RealSenseCamera, which is now every hardware grasping run.
XArmAdapter.activate() unconditionally drove the arm to a hardcoded pose -- _XARM7_INITIAL_JOINTS_DEG, carrying a "TODO change this once we have 7dof arm setup" -- and deactivate() did it again on the way out. Bringing a blueprint up should not command the arm anywhere. ManipulationModule already adopts wherever the arm is as the "init" preset, from the first joint state it receives, so the forced move was both redundant and destructive. The pose is now an explicit initial_positions argument, matching the mock and a750 adapters, and no motion happens without one. Also drops the 45 degree pitch from the hardware base pose. It came from the old xarm_perception blueprint, but the arm is mounted at identity, and a wrong base pose puts every planned pose wrong the same way the 12cm pedestal did.
grasp.py inherited two omissions from the sim blueprint it was built from. It never composed RerunBridgeModule, so there was no rerun window on either path. And RealSenseCamera defaults enable_pointcloud to False, exactly as MujocoSimModule does, so on hardware the camera published no cloud and the voxel chain silently had nothing to map -- the sim path had already been fixed for this, the hardware path had not. Verified against the attached D435i: with the cloud enabled the camera publishes 97k points in camera_color_optical_frame, which resolves through the wrist mount edge into the planning frame.
reset was removed in #3447 but FAULT stayed, and planning only runs from IDLE or COMPLETED, so a faulted module accepted nothing further. Both agent prompts still instruct the model to call reset, describing it as available as a skill and an RPC, so an agent that faulted was told to call a tool that did not exist. reset cancels an active trajectory rather than refusing one. Refusing and asking the caller to cancel first is a dead end for anyone holding only the skill surface, because cancel was never exposed there; it is now.
… threshold await_gripper_settle used settle_tolerance for two different questions: how little the jaws moved between polls, and how near the commanded target they stopped. A gripper driven to its mechanical stop always halts short of the extreme -- the real xArm rests at 0.988 of a nominal 850 count -- so opening an already-open gripper moved nothing, could never "arrive", and timed out with "gripper did not settle within 3.00s". Arrival is now a separate tolerance. The open path passes open_tolerance, which is the band that already decides whether where the jaws stopped counts as open. That also removes the sim blueprint's settle_tolerance override, which was the same bug seen from the other end: those jaws rest at 0.995 and the override was chasing the gap rather than naming it.
…orld floor_z installed a 0.6x1.2x0.2m box centred at (0.7, 0, -0.12): not a floor but one bench, pinned to one spot, leaving everything nearer than x=0.4 unguarded. What the camera sees defines the obstacles now.
Ports the grasp overlay from manip/grasp-sprint-july-26-sim-perception onto the rerun bridge that main actually has. GraspCandidateArray renders as gripper wireframes in the TCP frame, coloured best-green through worst-orange so the ranking reads at a glance, and labelled with the score. PickAndPlaceModule publishes the proposals on a stream as it makes them. The get_grasp_candidates RPC only answers after the fact, which is no help while the arm is choosing between them.
Brings the grasp overlay from manip/grasp-sprint-july-26-sim-perception: the backend-neutral layer types, the viser layer manager and scene rendering, and the grasp layers themselves -- object cloud, ranked candidate wireframes, and the attempted grasp with its pre-grasp and approach segment. Two adaptations were forced. gripper_wireframe_geometry moves out of demo_graspgenx into dimos/manipulation/grasping, so a live viewer can draw the sweep-volume wireframe without pulling matplotlib. And PickAndPlaceModule is a sibling module here rather than a ManipulationModule subclass, so it has no _world_monitor to reach the visualizer through; the layer travels over a set_visualization_layer rpc instead. The wireframe is built from the same sweep volume the generator is configured with, so a drawn grasp cannot diverge from the commanded one.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
There are now two xArm grasping blueprints,
xarm-graspandxarm-grasp-graspgenx, differing only in whichGraspGenSpecprovider they compose. Both run on the real arm by default and swap to the MuJoCo room scene with--simulation, which is decided at import time the waymujoco_if_simalready does it — the hardware adapter, base pose, camera, detector backends and home pose all follow from that one flag. They replacexarm-grasp-sim,xarm-grasp-sim-graspgenxandxarm-perception.That also fixes something
xarm-perceptionnever had: a coordinator. It could plan but had nothing to drive the arm, so hardware neededdimos run xarm-perception coordinator-xarm7to do anything at all.Also wires the wrist camera into a live voxel map the planner treats as an obstacle, using the modules #3714 added:
PointCloudSelfFilterdrops the arm's own returns and emits a clear mask,RayTracingVoxelMapaccumulates world-frame cells, andManipulationModule.voxel_maprebuilds them as one octree obstacle. Three things had to change for that chain to carry data — the sim'spointcloudoutput is off by default, the self filter needs a capture-time transform for every collision link and drops the whole cloud when one is missing (onlylink7andlink_tcpwere published), and the two edges whose stream names differ are remapped.Once the map exists the target object is itself mapped geometry, so a collision-checked plan into it can only be rejected. The pregrasp-to-grasp leg and the retreat now use
move_linearwith collision checking off, which is what that primitive is for; only the approach to the pregrasp pose stays a checked plan.Three pre-existing defects surfaced while validating and are fixed here. The grasp scene reused
make_xarm7_sim_robot_config(), which hardcodes thedata/xarm712cm pedestalbase_pose, butdata/xarm_grasp_sim/xarm7.xmlboltslink_baseto the world origin — the planning model sat 12cm above the arm MuJoCo simulates and every commanded pose executed 12cm low.pick_objectonly ever tried the top-ranked candidate, which is invisible with a single-candidate heuristic but throws away 99 usable poses from a learned provider. And these jaws asymptote to 0.995 and never reach 1.0, so opening an already-open gripper moved nothing and passed or failed the stock 0.005 settle tolerance by luck.Box setup is
uv sync --extra graspgenxfor the checkpoints, which download once from Hugging Face and cache under~/.cache/huggingface, plus the Rust binary for the native mapper:curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | sh -s -- -ythencargo build --release -p dimos-voxel-ray-tracing.Verified on the room scene with the voxel map live.
propose_graspsreturns 100 learned candidates from the real 680-point cloud with scores up to 0.96 in ~0.9s on the GPU,get_obstacles()reportsmapping/voxel-map, and both providers complete a pick — GraspGenX on rank 2 at score 0.93, which is the ranked fallback earning its keep.Adds
xarm-grasp-agentandxarm-grasp-graspgenx-agent, which put anMcpClientagent over the same scene so the skills are driven in natural language. Both setn_workers=6: the scene already carries a detector, a segmenter and a voxel mapper, and packing those with the agent into one worker makes the detector's lazytransformersimport fail withCould not import module 'Owlv2ForObjectDetection'.Two self-filter and hardware fixes came out of running it. The filter looked up transforms with a 20ms tolerance against robot TF published at 10Hz, so a ~92ms publish period could not be bracketed and 61% of clouds were dropped, each with a warning pair — the logs were unreadable. One full period admits them all, measured at 0 drops over 20s. Separately, the hardware blueprint's TF tree was broken by a live TODO:
RealSenseCamerapublishes only its own subtree, socamera_linkhad no parent andXARM_PERCEPTION_CAMERA_TRANSFORM, the hand-eye calibration, was dead code. That edge is now published offlink7. The hardware path is still untested — I have no arm here, and the parent link is inferred from the fact thattf_extra_links=["link7"]already existed.The capability doc also never showed how to pick, only how to scan and inspect obstacles. It now covers
pick_object,place_at, inspecting candidates without moving, and the agentic variants.The voxel map is composed unconditionally rather than being sim-only, so hardware plans against it too. That needs the Rust binary on the robot box.
The grasp sim also shipped with
visualization={"backend": "none"}, so it had no viewer at all; it is now on viser athttp://127.0.0.1:8095. That exposed a second gap: #3714 added theOCTREEobstacle type and its planner support but no rendering for it in any backend, so the viser scene raisedunsupported obstacle typeand drew a grey proxy box 78 times instead of the map. Occupied cells now draw as a point cloud sized to the cell edge — one box per cell would be tens of thousands of scene nodes.Two things worth a look before merge. Planning against the octree is slow: a pick takes 45-51s versus 3.5s without the map, which is the checked plan to the pregrasp pose against a dense voxel grid and probably wants tuning of
XARM_GRASP_VOXEL_SIZEormax_range. And although close-time verification passes, the gripper reads nearly shut after the pick returns (0.015 and 0.0003 across the two runs), so the object looks like it is being lost during the retreat rather than carried.