ArmY — Grasp-Pose Selection

To grasp an object, what an arm needs is a pose in space — where to put the jaws, from which direction, and how far to open them.

Producing those is the open-source GPD library's job: it samples the point cloud and returns a ranked set of candidate gripper poses — a great many on a cluttered scene, plenty of them unusable, driving the gripper through the surface the item rests on or sitting on the ragged fringe of the cloud.

What I added around it

• A reachability box — the stock all sits inside it, and candidates falling outside that conservative box in the arm’s base frame are dropped before anything else is considered; this is what removes most of them
• Ranking — what survives is ordered by the classifier’s own score, with distance to a preferred point as the tie-break

I made three pieces that were never built for each other behave as one system: an open-source grasp generator, a depth camera on the wrist, and the arm's own control interface.

The point-cloud bridge is where that is settled. GPD is written for a camera on a fixed mount, watching the scene from outside while the arm moves through it. Ours is the other arrangement — the depth camera is bolted to the wrist, directly above the jaws — so there is no fixed camera-to-base extrinsic to write down, and the frame the cloud arrives in is a different one after every joint move.

That breaks an assumption GPD never states, because with a fixed camera it never has to. It crops with a plain box of metres — −0.05 to 0.36 m out from the base, ±0.30 m across, −0.05 to 0.40 m up — which describes the space above the table only if the cloud is in the arm's base frame. Ours arrives wrist-relative, so the box cuts a volume out of a different wrong place after every joint move. Every cloud is therefore pulled through the live TF tree into base_link first, downsampled on the way. Get it wrong and nothing errors: the arm reaches confidently for a place that is not there.

We could have run GPD first and transformed afterwards, throwing away whatever landed outside the crop box — moving the pose GPD hands back is far cheaper than moving the cloud, a handful of poses against a few hundred thousand points. We still transform the whole cloud into base_link first. The crop box takes effect before the data reaches GPD, so on an untransformed cloud GPD generates a great many candidates outside the box, and those can score high enough to crowd out the ones actually on the item. That is what we hit in testing.

Around it sits the node that takes the chosen candidate to the team's IK and MoveIt 2 service, and a PyQt5 panel keeping hand-eye calibration, exposure and grasp offsets adjustable while the arm runs — so an experiment costs a slider instead of a rebuild.

Verification I owned

The perception and grasp requirements were mine to test. What passed:

• Object detection — 15/15 across the baseline and cluttered sets, against distractors and partial occlusion
• Grasp sequence automation — detection through to extraction; the operator only clicks the item they want

And what did not

Position accuracy is the one that matters, and the one that failed: the estimate had to stay within ± 1.0 cm for 95% of samples, and did so for 7 of 10. Pick cycle time landed at 10–60 s against a < 45 s budget.

15/15 detection is not 15/15 grasping: the system almost always sees the item, and still does not always know where it is well enough to grasp it cleanly. The limit is the depth camera — a grasp pose is only as accurate as the surface it was fitted to, and the camera's own precision leaves this cloud ragged at its edges.

ArmY was a five-person team. The mechanical design, the motion-planning line and the electrical work were teammates’. Mine was the grasp pipeline: the grasp-selection node, the point-cloud bridge, the operator interface, and the verification above.

The grasp-candidate generator is not mine either — it is the open-source GPD library, sitting in the tree unchanged. What I wrote is everything around it that makes it run on this arm.

And what the project did not get to: shelf-aware obstacle avoidance never reached a working state. The picks here are in open scenes and on a rack, not reaching past neighbouring stock inside a confined bay — the grasp pipeline is the part that works.

Sponsor: Prof. Solmaz Kia. Advisors: Prof. Mark Walter and Prof. David Copp. Submitted as the UC Irvine degree capstone for MAE 151A/B.