Task-Oriented Grasp Prediction with Visual-Language Inputs

Chao Tang1; Dehao Huang1; Lingxiao Meng2; Weiyu Liu3; Hong Zhang1

10.1109/IROS55552.2023.10342268

2023

IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS)

2153-0858

978-1-6654-9191-4

2023 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS)

4881-4888

1-5 Oct. 2023

Detroit, MI, USA

345 E 47TH ST, NEW YORK, NY 10017 USA

IEEE

To perform household tasks, assistive robots receive commands in the form of user language instructions for tool manipulation. The initial stage involves selecting the intended tool (i.e., object grounding) and grasping it in a task-oriented manner (i.e., task grounding). Nevertheless, prior researches on visual-language grasping (VLG) focus on object grounding, while disregarding the fine-grained impact of tasks on object grasping. Task-incompatible grasping of a tool will inevitably limit the success of subsequent manipulation steps. Motivated by this problem, this paper proposes GraspCLIP, which addresses the challenge of task grounding in addition to object grounding to enable task-oriented grasp prediction with visual-language inputs. Evaluation on a custom dataset demonstrates that GraspCLIP achieves superior performance over established baselines with object grounding only. The effectiveness of the proposed method is further validated on an assistive robotic arm for grasping previously unseen kitchen tools given the task specification. Our presentation video is available at: https://www.youtube.com/watch?v=e1wfYQPeAXU.

Grounding Grasping Manipulators Assistive robots 6-DOF Task analysis Intelligent robots

第一

英语

EI ; CPCI-S

Shenzhen Key Laboratory of Robotics and Computer Vision[ZDSYS20220330160557001]

Computer Science ; Robotics

Computer Science, Artificial Intelligence ; Computer Science, Information Systems ; Computer Science, Theory & Methods ; Robotics

WOS:001133658803106

20240315412045

Robotics

Computer Programming Languages:723.1.1 ; Robotics:731.5

IEEE

1.Shenzhen Key Laboratory of Robotics and Computer Vision, Southern University of Science and Technology, Shenzhen, China
2.Department of Electronic and Electrical Engineering, Southern University of Science and Technology, Shenzhen, China
3.Stanford University, United States

Chao Tang,Dehao Huang,Lingxiao Meng,et al. Task-Oriented Grasp Prediction with Visual-Language Inputs[C]. 345 E 47TH ST, NEW YORK, NY 10017 USA:IEEE,2023:4881-4888.