On the Comparability and Optimal Aggressiveness of the Adversarial Scenario-Based Safety Testing of Robots

Weng, Bowen1; Castillo, Guillermo A.1; Zhang, Wei2,3; Hereid, Ayonga4

2023-05-01

10.1109/TRO.2023.3267020

IEEE TRANSACTIONS ON ROBOTICS   影响因子和分区

1552-3098

1941-0468

This article studies the class of scenario-based safety testing algorithms in the black-box safety testing configuration. For algorithms sharing the same state-action set coverage with different sampling distributions, it is commonly believed that prioritizing the exploration of high-risk states and actions leads to a better sampling efficiency. Our proposal disputes the above intuition by introducing an impossibility theorem that provably shows that all the safety testing algorithms of the aforementioned difference perform equally well with the same expected sampling efficiency. Moreover, for testing algorithms covering different sets of states and actions, the sampling efficiency criterion is no longer applicable as different algorithms do not necessarily converge to the same termination condition. We then propose a testing aggressiveness definition based on the almost safe set concept along with an unbiased and efficient algorithm that compares the aggressiveness between testing algorithms. Empirical observations from the safety testing of bipedal locomotion controllers and vehicle decision-making modules are also presented to support the proposed theoretical implications and methodologies.

Testing Safety Costs Behavioral sciences Vehicle dynamics Heuristic algorithms Closed box Adversarial testing scenario intelligent transportation systems legged robots robot safety

SCI ; EI

英语

通讯

National Natural Science Foundation of China["62073159","62003155"] ; Shenzhen Science and Technology Program[JCYJ20200109141601708] ; Science, Technology and Innovation Commission of Shenzhen Municipality[ZDSYS20200811143601004] ; National Science Foundation[FRR-2144156]

Robotics

Robotics

WOS:000986552000001

IEEE-INST ELECTRICAL ELECTRONICS ENGINEERS INC

20232114131174

Anthropomorphic robots ; Behavioral research ; Decision making ; Efficiency ; Heuristic algorithms ; Intelligent robots ; Intelligent systems ; Well testing

Ergonomics and Human Factors Engineering:461.4 ; Computer Programming:723.1 ; Artificial Intelligence:723.4 ; Robotics:731.5 ; Robot Applications:731.6 ; Management:912.2 ; Production Engineering:913.1 ; Accidents and Accident Prevention:914.1 ; Social Sciences:971

Web of Science

1.Ohio State Univ, Dept Elect & Comp Engn, Columbus, OH 43210 USA
2.Southern Univ Sci & Technol, Shenzhen Key Lab Biomimet Robot & Intelligent Syst, Shenzhen 518055, Peoples R China
3.Southern Univ Sci & Technol, Dept Mech & Energy Engn, Shenzhen 518055, Peoples R China
4.Ohio State Univ, Dept Mech & Aerosp Engn, Columbus, OH 43210 USA

Weng, Bowen,Castillo, Guillermo A.,Zhang, Wei,et al. On the Comparability and Optimal Aggressiveness of the Adversarial Scenario-Based Safety Testing of Robots[J]. IEEE TRANSACTIONS ON ROBOTICS,2023,PP(99):1-20.

Weng, Bowen,Castillo, Guillermo A.,Zhang, Wei,&Hereid, Ayonga.(2023).On the Comparability and Optimal Aggressiveness of the Adversarial Scenario-Based Safety Testing of Robots.IEEE TRANSACTIONS ON ROBOTICS,PP(99),1-20.

Weng, Bowen,et al."On the Comparability and Optimal Aggressiveness of the Adversarial Scenario-Based Safety Testing of Robots".IEEE TRANSACTIONS ON ROBOTICS PP.99(2023):1-20.