Kinova Gemini: Interactive Robot Grasping with Visual Reasoning and Conversational AI

Hanxiao Chen1; Jiankun Wang2; Max Q.-H. Meng3

10.1109/ROBIO55434.2022.10011896

2022

2022 IEEE International Conference on Robotics and Biomimetics

978-1-6654-8110-6

2022 IEEE International Conference on Robotics and Biomimetics (ROBIO)

129-134

5-9 Dec. 2022

Jinghong, China

To facilitate recent advances in robotics and AI for delicate collaboration between humans and machines, we propose the Kinova Gemini, an original robotic system that integrates conversational AI dialogue and visual reasoning to make the Kinova Gen3 lite robot help people retrieve objects or complete perception-based pick-and-place tasks. When a person walks up to Kinova Gen3 lite, our Kinova Gemini is able to fulfill the user's requests in three different applications: (1) It can start a natural dialogue with people to interact and assist humans to retrieve objects and hand them to the user one by one. (2) It detects diverse objects with YOLO v3 and recognize color attributes of the item to ask people if they want to grasp it via the dialogue or enable the user to choose which specific one is required. (3) It applies YOLO v3 to recognize multiple objects and let you choose two items for perception-based pick-and-place tasks such as “Put the banana into the bowl” with visual reasoning and conversational interaction.

Visualization Service Robots Collaboration Grasping Color Object Detection Cognition

通讯

英语

EI

IEEE

正式出版

1.Harbin Institute of Technology
2.Department of Electronic and Electrical Engineering, Shenzhen Key Laboratory of Robotics Perception and Intelligence, Southern University of Science and Technology, Shenzhen, China
3.Shenzhen Research Institute, The Chinese University of Hong Kong in Shenzhen, Shenzhen, China

Hanxiao Chen,Jiankun Wang,Max Q.-H. Meng. Kinova Gemini: Interactive Robot Grasping with Visual Reasoning and Conversational AI[C],2022:129-134.