Enhancing Sharpness-Aware Minimization by Learning Perturbation Radius

Wang, Xuehao1; Jiang, Weisen1,2; Fu, Shuai1; Zhang, Yu1

10.1007/978-3-031-70344-7_22

2024

Joint European Conference on Machine Learning and Knowledge Discovery in Databases (ECML PKDD)

2945-9133

1611-3349

978-3-031-70343-0

MACHINE LEARNING AND KNOWLEDGE DISCOVERY IN DATABASES: RESEARCH TRACK, PT II, ECML PKDD 2024

14942

SEP 09-13, 2024

null,Vilnius,LITHUANIA

GEWERBESTRASSE 11, CHAM, CH-6330, SWITZERLAND

SPRINGER INTERNATIONAL PUBLISHING AG

Sharpness-aware minimization (SAM) is to improve model generalization by searching for flat minima in the loss landscape. The SAM update consists of one step for computing the perturbation and the other for computing the update gradient. Within the two steps, the choice of the perturbation radius is crucial to the performance of SAM, but finding an appropriate perturbation radius is challenging. In this paper, we propose a bilevel optimization framework called LEarning the perTurbation radiuS (LETS) to learn the perturbation radius for sharpness-aware minimization algorithms. Specifically, in the proposed LETS method, the upper-level problem aims at seeking a good perturbation radius by minimizing the squared generalization gap between the training and validation losses, while the lower-level problem is the SAM optimization problem. Moreover, the LETS method can be combined with any variant of SAM. Experimental results on various architectures and benchmark datasets in computer vision and natural language processing demonstrate the effectiveness of the proposed LETS method in improving the performance of SAM.

Sharpness-Aware Minimization Bilevel Optimization Hyperparameter Optimization

第一 ; 通讯

英语

CPCI-S

NSFC[62136005] ; NSFC general grant[62076118] ; Shenzhen fundamental research program[JCYJ20210324105000003]

Computer Science

Computer Science, Artificial Intelligence ; Computer Science, Theory & Methods

WOS:001308375100022

Web of Science

1.Southern Univ Sci & Technol, Dept Comp Sci & Engn, Shenzhen, Peoples R China
2.Hong Kong Univ Sci & Technol, Hong Kong, Peoples R China

Wang, Xuehao,Jiang, Weisen,Fu, Shuai,et al. Enhancing Sharpness-Aware Minimization by Learning Perturbation Radius[C]. GEWERBESTRASSE 11, CHAM, CH-6330, SWITZERLAND:SPRINGER INTERNATIONAL PUBLISHING AG,2024.