Reconstructing Soft Robotic Touch via In-Finger Vision

Guo, Ning1; Han, Xudong1; Zhong, Shuqiao2; Zhou, Zhiyuan2; Lin, Jian2; Wan, Fang3; Song, Chaoyang4

2024-07-01

10.1002/aisy.202400022

ADVANCED INTELLIGENT SYSTEMS   影响因子和分区

2640-4567

["Incorporating authentic tactile interactions into virtual environments presents a notable challenge for the emerging development of soft robotic metamaterials. In this study, a vision-based approach is introduced to learning proprioceptive interactions by simultaneously reconstructing the shape and touch of a soft robotic metamaterial (SRM) during physical engagements. The SRM design is optimized to the size of a finger with enhanced adaptability in 3D interactions while incorporating a see-through viewing field inside, which can be visually captured by a miniature camera underneath to provide a rich set of image features for touch digitization. Employing constrained geometric optimization, the proprioceptive process with aggregated multi-handles is modeled. This approach facilitates real-time, precise, and realistic estimations of the finger's mesh deformation within a virtual environment. Herein, a data-driven learning model is also proposed to estimate touch positions, achieving reliable results with impressive R2 scores of 0.9681, 0.9415, and 0.9541 along the x, y, and z axes. Furthermore, the robust performance of the proposed methods in touch-based human-cybernetic interfaces and human-robot collaborative grasping is demonstrated. In this study, the door is opened to future applications in touch-based digital twin interactions through vision-based soft proprioception.","In this study, a vision-based approach is introduced to learning proprioceptive interactions by reconstructing the shape and touch of a soft robotic metamaterial during physical engagements. It enables real-time, precise estimations of mesh deformation and touch positions in a virtual environment, showcasing the potential for touch-based digital twin interactions.image (c) 2024 WILEY-VCH GmbH"]

inverse problems shape reconstructions soft robotics tactile sensings

SCI ; EI

英语

第一 ; 通讯

National Natural Science Foundation of China[62206119] ; Shenzhen Long-Term Support for Higher Education at SUSTech[20231115141649002] ; SUSTech Virtual Teaching Lab for Machine Intelligence Design and Learning[Y01331838] ; Science, Technology, and Innovation Commission of Shenzhen Municipality["JCYJ20220818100417038","JSGG20220831110002004"]

Automation & Control Systems ; Computer Science ; Robotics

Automation & Control Systems ; Computer Science, Artificial Intelligence ; Robotics

WOS:001268838200001

WILEY

Web of Science

1.Southern Univ Sci & Technol, SUSTech Inst Robot, Shenzhen, Peoples R China
2.Southern Univ Sci & Technol, Ocean Sci & Engn, Shenzhen, Peoples R China
3.Southern Univ Sci & Technol, Sch Design, Shenzhen, Peoples R China
4.Southern Univ Sci & Technol, Mech & Energy Engn, Shenzhen, Peoples R China

Guo, Ning,Han, Xudong,Zhong, Shuqiao,et al. Reconstructing Soft Robotic Touch via In-Finger Vision[J]. ADVANCED INTELLIGENT SYSTEMS,2024.

Guo, Ning.,Han, Xudong.,Zhong, Shuqiao.,Zhou, Zhiyuan.,Lin, Jian.,...&Song, Chaoyang.(2024).Reconstructing Soft Robotic Touch via In-Finger Vision.ADVANCED INTELLIGENT SYSTEMS.

Guo, Ning,et al."Reconstructing Soft Robotic Touch via In-Finger Vision".ADVANCED INTELLIGENT SYSTEMS (2024).