On performance estimation in automatic algorithm configuration

Liu，Shengcai1; Tang，Ke2; Lei，Yunwen2; Yao，Xin2

2020

AAAI 2020 - 34th AAAI Conference on Artificial Intelligence

2384-2391

Over the last decade, research on automated parameter tuning, often referred to as automatic algorithm configuration (AAC), has made significant progress. Although the usefulness of such tools has been widely recognized in real world applications, the theoretical foundations of AAC are still very weak. This paper addresses this gap by studying the performance estimation problem in AAC. More specifically, this paper first proves the universal best performance estimator in a practical setting, and then establishes theoretical bounds on the estimation error, i.e., the difference between the training performance and the true performance for a parameter configuration, considering finite and infinite configuration spaces respectively. These findings were verified in extensive experiments conducted on four algorithm configuration scenarios involving different problem domains. Moreover, insights for enhancing existing AAC methods are also identified.

通讯

英语

2-s2.0-85095337214

Scopus

1.School of Computer Science and Technology,University of Science and Technology of China,Hefei,230027,China
2.Department of Computer Science and Engineering,Southern University of Science and Technology,Shenzhen,518055,China

Liu，Shengcai,Tang，Ke,Lei，Yunwen,et al. On performance estimation in automatic algorithm configuration[C],2020:2384-2391.