Dynamic Algorithm Configuration (DAC) aims at dynamically adjusting the hyperparameters of a target algorithm to improve performance in a data-driven manner [Biedenkapp et al., 2020]. This allows to not only statically tune hyperparameters, but adjust them while the learning takes place. In contrast to previous methods, this allows us to deeply go into the algorithms and open up new potentials to further improvements of performance. Theoretical and empirical results demonstrated the advantages of dynamically controlling hyperparameters, e.g. in the domains of deep learning, evolutionary algorithms and AI planning [Daniel et al., 2016; Vermetten et al., 2019; Doerr & Doerr, 2020; Shala et al. 2020; Speck et al., 2021]. However, there are several remaining challenges and opportunities to improve DAC w.r.t. building trust into DAC by considering the various aspects of trustworthy AI (transparency, explainability, fairness, robustness, and privacy) as defined in the EC Guidelines for Trustworthy AI and the Assessment List for Trustworthy AI. These include: (i) Typically AI models are overfitted to the selected data instances for training, which means that their selected hyperparameters can not generalize the learned knowledge on new data instances that were not involved in the training process. (ii) There is an inherent bias presented in the learning process that originates from the quality of data used in the training process. (iii) There are no quantitative measures to estimate the level of trust/confidence when using a developed AI model on new unseen data, which will indicate to which level generalization of the learned knowledge is possible. To overcome the aforementioned challenges, the goals of our project are: 1. To investigate different meta-feature representations for data instances for DAC learning tasks presented in the DACBench library, which will allow us to perform complementary analysis between them and perform a landscape coverage analysis of the problem/feature space, leading to a thorough and fair comparison of DAC methods. The utility of meta-feature representations will also be investigated by transforming them with matrix factorization and deep learning techniques. 2. To develop methodologies that will allow us automatically to select more representative data instances that will uniformly cover the landscape space of the data instances using their meta-feature representation, which can be used for benchmarking studies to produce reproducible and transferable results. 3. To define quantitative indicators that will measure the level of diversity between the data instances used for training DAC and instances used for testing by using their meta-feature representation. This will provide researchers the level of trust of applying the selected hyperparameters on new unseen data instances. 4. To determine computational time and energy variables that will be measured to estimate the greener level (i.e. encouraging a reduction in resources spent) of performing DAC experiments only on the selected representative data instances.
Lead at LUHAI: Prof. Lindauer
Funding Program: DAAD Slovenia
Project Period: January 2023 to December 2024
