A generic first-order algorithmic framework for bi-level programming beyond lower-level singleton

Liu，Risheng1; Mu1，Pan2; Yuan，Xiaoming3; Zeng，Shangzhi4; Zhang，Jin1

2020

37th International Conference on Machine Learning, ICML 2020

PartF168147-9

6261-6271

In recent years, a variety of gradient-based bilevel optimization methods have been developed for learning tasks. However, theoretical guarantees of these existing approaches often heavily rely on the simplification that for each fixed upperlevel variable, the lower-level solution must be a singleton (a.k.a., Lower-Level Singleton, LLS). In this work, by formulating bi-level models from the optimistic viewpoint and aggregating hierarchical objective information, we establish Bi-level Descent Aggregation (BDA), a flexible and modularized algorithmic framework for bilevel programming. Theoretically, we derive a new methodology to prove the convergence of BDA without the LLS condition. Furthermore, we improve the convergence properties of conventional first-order bi-level schemes (under the LLS simplification) based on our proof recipe. Extensive experiments justify our theoretical results and demonstrate the superiority of the proposed BDA for different tasks, including hyper-parameter optimization and meta learning.

其他

英语

EI

20211910314216

2-s2.0-85105227421

Scopus

1.DUT-RU International School of Information Science and Engineering,Dalian University of Technology,
2.Key Laboratory for Ubiquitous Network and Service Software of Liaoning Province,
3.Department of Mathematics,University of Hong Kong,Hong Kong
4.SUSTech International Center for Mathematics and Department of Mathematics,Southern University of Science and Technology,

Liu，Risheng,Mu1，Pan,Yuan，Xiaoming,et al. A generic first-order algorithmic framework for bi-level programming beyond lower-level singleton[C],2020:6261-6271.