Automated Repair of Programs from Large Language Models

Fan, Zhiyu1; Gao, Xiang2; Mirchev, Martin1; Roychoudhury, Abhik1; Tan, Shin Hwei3

10.1109/ICSE48619.2023.00128

2023

45th IEEE/ACM International Conference on Software Engineering (ICSE)

0270-5257

978-1-6654-5702-6

2023 IEEE/ACM 45TH INTERNATIONAL CONFERENCE ON SOFTWARE ENGINEERING, ICSE

1469-1481

MAY 14-20, 2023

null,Melbourne,AUSTRALIA

10662 LOS VAQUEROS CIRCLE, PO BOX 3014, LOS ALAMITOS, CA 90720-1264 USA

IEEE COMPUTER SOC

Large language models such as Codex, have shown the capability to produce code for many programming tasks. However, the success rate of existing models is low, especially for complex programming tasks. One of the reasons is that language models lack awareness of program semantics, resulting in incorrect programs, or even programs which do not compile. In this paper, we systematically study whether automated program repair (APR) techniques can fix the incorrect solutions produced by language models in LeetCode contests. The goal is to study whether APR techniques can enhance reliability in the code produced by large language models. Our study revealed that: (1) automatically generated code shares common programming mistakes with human-crafted solutions, indicating APR techniques may have potential to fix auto-generated code; (2) given bug location information provided by a statistical fault localization approach, the newly released Codex edit mode, which supports editing code, is similar to or better than existing Java repair tools TBar and Recoder in fixing incorrect solutions. By analyzing the experimental results generated by these tools, we provide several suggestions: (1) enhancing APR tools to surpass limitations in patch space (e.g., introducing more flexible fault localization) is desirable; (2) as large language models can derive more fix patterns by training on more data, future APR tools could shift focus from adding more fix patterns to synthesis/semantics based approaches, (3) combination of language models with APR to curate patch ingredients, is worth studying.

Large Language Model Program Repair

其他

英语

CPCI-S ; EI

Singapore Ministry of Education (MoE) Tier 3 grant "Automated Program Repair"[MOE-MOET32021-0001] ; National Natural Science Foundation of China["61902170","62202026","62141209"]

Computer Science

Computer Science, Software Engineering ; Computer Science, Theory & Methods

WOS:001032629800119

20233914775287

Automation ; Computational linguistics

Computer Theory, Includes Formal Logic, Automata Theory, Switching Theory, Programming Theory:721.1 ; Automatic Control Principles and Applications:731

Web of Science

1.Natl Univ Singapore, Singapore, Singapore
2.Beihang Univ, Beijing, Peoples R China
3.Southern Univ Sci & Technol, Shenzhen, Peoples R China

Fan, Zhiyu,Gao, Xiang,Mirchev, Martin,et al. Automated Repair of Programs from Large Language Models[C]. 10662 LOS VAQUEROS CIRCLE, PO BOX 3014, LOS ALAMITOS, CA 90720-1264 USA:IEEE COMPUTER SOC,2023:1469-1481.