ADAPT: Action-aware Driving Caption Transformer

Bu Jin1; Xinyu Liu2; Yupeng Zheng1; Pengfei Li3; Hao Zhao2; Tong Zhang4; Yuhang Zheng2; Guyue Zhou2; Jingjing Liu2

10.1109/ICRA48891.2023.10160326

2023-05-29

IEEE International Conference on Robotics and Automation (ICRA)

1050-4729

2577-087X

979-8-3503-2366-5

2023 IEEE International Conference on Robotics and Automation (ICRA)

2023-May

7554-7561

29 May-2 June 2023

London, United Kingdom

345 E 47TH ST, NEW YORK, NY 10017 USA

IEEE

End-to-end autonomous driving has great potential in the transportation industry. However, the lack of transparency and interpretability of the automatic decision-making process hinders its industrial adoption in practice. There have been some early attempts to use attention maps or cost volume for better model explainability which is difficult for ordinary passengers to understand. To bridge the gap, we propose an end-to-end transformer-based architecture, ADAPT (Action-aware Driving cAPtion Transformer), which provides user-friendly natural language narrations and reasoning for each decision making step of autonomous vehicular control and action. ADAPT jointly trains both the driving caption task and the vehicular control prediction task, through a shared video representation. Experiments on BDD-X (Berkeley DeepDrive eXplanation) dataset demonstrate state-of-the-art performance of the ADAPT framework on both automatic metrics and human evaluation. To illustrate the feasibility of the proposed framework in real-world applications, we build a novel deployable system that takes raw car videos as input and outputs the action narrations and reasoning in real time. The code, models and data are available at https://github.com/jxbbb/ADAPT.

Measurement Training Adaptation models Transportation industry Decision making Streaming media Transformers

其他

英语

CPCI-S ; EI

Automation & Control Systems ; Computer Science ; Engineering ; Robotics

Automation & Control Systems ; Computer Science, Artificial Intelligence ; Engineering, Electrical & Electronic ; Robotics

WOS:001048371100078

20233514632988

Autonomous vehicles

Highway Transportation:432 ; Robot Applications:731.6 ; Management:912.2

IEEE

1.Chinese Academy of Sciences, Institute of Automation, China
2.Institute for AI Industry Research (AIR), Tsinghua University, China
3.Department of Computer Science and Technology, Tsinghua University, China
4.Southern University of Science and Technology, China

Bu Jin,Xinyu Liu,Yupeng Zheng,et al. ADAPT: Action-aware Driving Caption Transformer[C]. 345 E 47TH ST, NEW YORK, NY 10017 USA:IEEE,2023:7554-7561.