Best Arm Identification with Retroactively Increased Sampling Budget for More Resource-Efficient HPO

verfasst von

Jasmin Brandt, Marcel Wever, Viktor Bengs, Eyke Hüllermeier

Abstract

Hyperparameter optimization (HPO) is indispensable for achieving optimal performance in machine learning tasks. A popular class of methods in this regard is based on Successive Halving (SHA), which casts HPO into a pure-exploration multi-armed bandit problem under finite sampling budget constraints. This is accomplished by considering hyperparameter configurations as arms and rewards as the negative validation losses. While enjoying theoretical guarantees as well as working well in practice, SHA comes, however, with several hyperparameters itself, one of which is the maximum budget that can be allocated to evaluate a single arm (hyperparameter configuration). Although there are already solutions to this meta hyperparameter optimization problem, such as the doubling trick or asynchronous extensions of SHA, these are either practically inefficient or lack theoretical guarantees. In this paper, we propose incremental SHA (iSHA), a synchronous extension of SHA, allowing to increase the maximum budget a posteriori while still enjoying theoretical guarantees. Our empirical analysis of HPO problems corroborates our theoretical findings and shows that iSHA is more resource-efficient than existing SHA-based approaches.

Externe Organisation(en)

Universität Paderborn
Ludwig-Maximilians-Universität München (LMU)
Munich Center for Machine Learning (MCML)

Typ

Paper

Seiten

3742-3750

Publikationsdatum

08.2024

Publikationsstatus

Veröffentlicht

Peer-reviewed

Ja

Elektronische Version(en)

https://doi.org/10.24963/ijcai.2024/414 (Zugang: Geschlossen)