Geometric Constraint as General Interface for Robot Manipulation

Weiliang Tang , Jia-Hui Pan,Yun-Hui Liu, Masayoshi Tomizuka,

Li Erran Li, Chi-Wing Fu^*, Mingyu Ding

Paper Code

Framework Overview

Given language instruction, our framework:

(1). Task decomposition and flow control generation.
(2). Geometric constraints generation for each stage.
(3). Parse geometry.
(4). Translate geometric constraints into cost functions
(5). Solve for the trajectory with optimization to minimize the overall costs
(6). The robot executes according to predicted gripper's poses and trajectory.

Geometry Parser

Given language description of the geometric part, the module:

(1). Use GroundingDINO to locate object.
(2). Apply Segment Anything Model to generate all part-level masks.
(3). Ask VLM model to select the mask corresponding to the object part.
(4). Ask VLM model to generate post-processing function to process the object part mask.

MetaWorld Environment

Task: Put the basketball onto the red hoop

Task: Press the handle

Task: Press the button

Task: Press the button from the side

Task: Place the white cube into the blue bin

Task: Place the cube onto the shelf.

Task: Grasp handle and place ring into stick.

OmniGibson Environment

Task: Put pen into pen-holder

Task: Cut carrot with the knife

Task: Open the fridge

Task: Open the fridge a little bit

Task: Type "Hi"

RealWorld Deployment

We deploy our method on FANUC Mate 200iD robot, with a RealSense D435 camera.

Task: Pour water from cup to pan

Task: Pour water from cup to pan, and put it to wooden tray

Task: Pour macaroni to pan

Task: Press start button on the induction cooktop

Task: Put banana into the wooden tray

Task: Stir bowl with spoon

Task: Turn nut

Task: Open microwave

Task: Close microwave

Task: Put pan onto induction cooktop

BibTex

                
@article{Tang2025GeoConst,
title={{Geometric Constraint as General Interface for Robot Manipulation}},
author={Weiliang Tang , Jia-Hui Pan,Yun-Hui Liu, Masayoshi Tomizuka, and Li Erran Li, Chi-Wing Fu*, Mingyu Ding},
journal={arXiv preprint arXiv:2501.09783},
year={2025},
}