SynthACticBench: A Capability-Based Synthetic Benchmark for Algorithm Configuration

verfasst von

Valentin Margraf, Anna Lappe, Marcel Dominik Wever, Carolin Benjamins, Eyke Hüllermeier, Marius Lindauer

Abstract

Algorithm configuration deals with the automatic optimization of an algorithm's parameters to maximize its performance on a distribution of problem instances, such as Boolean satisfiability or the traveling salesperson problem. While significant progress has been made in developing optimizers for algorithm configuration -- so-called algorithm configurators -- their evaluation remains computationally expensive and often relies on real-world scenarios with hard-to-control characteristics. This makes it challenging to analyze their strengths and weaknesses systematically. To address this, we introduce SynthACticBench, a synthetic benchmark specifically designed to isolate and investigate key properties of algorithm configuration problems. Our benchmark distinguishes between properties related to the configuration space and those associated with the objective function. We define a configurator's ability to handle a particular property as its capability -- for example, the capability to manage hierarchical configuration spaces. Using SynthACticBench, we evaluate two state-of-the-art algorithm configurators, SMAC and irace, examining their complementary capabilities and analyzing their performances across diverse benchmark functions. By providing a controlled, scalable, and capability-based evaluation environment, SynthACticBench facilitates a more targeted analysis of algorithm configurators, helping to advance research in the field. The benchmark is available at: github.com/annaelisalappe/SynthACticBench.

Organisationseinheit(en)

Fachgebiet Maschinelles Lernen
Forschungszentrum L3S

Externe Organisation(en)

Ludwig-Maximilians-Universität München (LMU)
Munich Center for Machine Learning (MCML)
Universität Paderborn
Universidad de la Sabana

Typ

Aufsatz in Konferenzband

Publikationsdatum

06.2025

Publikationsstatus

Elektronisch veröffentlicht (E-Pub)

Peer-reviewed

Ja