Pitfalls and Best Practices in Algorithm Configuration

Good parameter settings are crucial to achieve high performance in many areas of artificial intelligence (AI), such as satisfiability solving, AI planning, scheduling, and machine learning (in particular deep learning). Automated algorithm configuration methods have recently received much attention in the AI community since they replace tedious, irreproducible and error-prone manual parameter tuning and can lead to new state-of-the-art performance. However, practical applications of algorithm configuration are prone to several (often subtle) pitfalls in the experimental design that can render the procedure ineffective. We identify several common issues and propose best practices for avoiding them, including a tool called GenericWrapper4AC for preventing the many possible problems in measuring the performance of the algorithm being optimized by executing it in a standardized, controlled manner.

Although the usability of AC systems improved over the years (e.g., SpySMAC (Falkner et al., 2015)), we still often observe fundamental issues in the design and execution of experiments with algorithm configuration methods by both experts and new users.The goals of this work are therefore to: • highlight the many pitfalls we have encountered in AC experiments (run by ourselves and others); • present best practices to avoid most of these pitfalls; and • propose a unified interface between an AC system and the algorithm it optimizes (the so-called target algorithm) that directly implements best practices related to properly measuring the target algorithm's performance with different parameter settings.
Specifically, after a short overview of algorithm configuration (Section 2.1), we present an extensively tested package to provide an interface between AC systems and target algorithms that aims to improve the reliability, reproducibility and robustness of AC experiments (Section 2.2).Afterwards, we describe common pitfalls in using AC systems and recommendations on how to avoid them.We first discuss pitfalls concerning the interface between AC systems and target algorithms (Section 3), followed by pitfalls regarding overtuning (Section 4).We note that, while we observed the described pitfalls in the context of AC, most of them also apply to the experimental evaluation of algorithms in general.Throughout, we illustrate pitfalls by AC experiments on propositional satisfiability solvers (Biere et al., 2009) as a prototypical AC example, but insights directly transfer to other AC problems.We end this note by providing further general recommendations for effective configuration in Section 5.

Background: Algorithm Configuration
The algorithm configuration problem can briefly be described as follows: given an algorithm A to be optimized (the so-called target algorithm) with parameter configuration space Θ, a set of instances Π, and a cost metric c : Θ × Π → R, find a configuration θ ∈ Θ that minimizes the cost metric c across the instances in Π: θ * ∈ arg min θ∈Θ π∈Π c(θ, π). (1) A concrete example for this algorithm configuration problem would be to find a parameter setting θ ∈ Θ of a solver A for the propositional satisfiability problem (SAT) (such as glucose (Audemard & Simon, 2009) or lingeling (Biere, 2013)) on a set of CNF instances Π (e.g., hardware or software verification instances) that minimizes A's average running time c.Another example would be to find a hyperparameter setting for a machine learning algorithm that minimizes its error c on a given dataset (Snoek et al., 2012;Feurer et al., 2015); in this latter example, error could either be measured via k-fold inner crossvalidation (giving rise to k instances for algorithm configuration), or a single validation set (in which case there is just a single instance for algorithm configuration).
The general workflow of a sequential algorithm configuration procedure (short: configurator ) is shown in Figure 1.In each step, the configurator picks a configuration θ ∈ Θ and an instance π ∈ Π, triggers a run of algorithm A with configuration θ on instance π with a maximal running time cutoff κ, and measures the resulting cost c(θ, π).(As detailed in Section 2.2, this step is usually mediated by a target-algorithm specific wrapper.)The configurator uses this collected data about the target algorithm's performance to find a well-performing configuration, typically operating as an anytime procedure until its configuration budget is exhausted (e.g., a maximal number of target algorithm calls or a time budget); when terminated, it returns its best found configuration so far.

Approaches for solving the AC problem
There are several quite different configurators for solving this AC problem.ParamILS (Hutter et al., 2009) uses local search in the configuration space, employing a racing strategy to decide which of two configurations performs better without running both of them on all instances.irace (López-Ibáñez et al., 2016) uses iterative races via F-race (Birattari et al., 2002) on a set of picked configurations to determine the best one.SMAC (Hutter et al., 2011) and its distributed version dSMAC (Hutter et al., 2012)) use probabilistic models of algorithm performance, so-called empirical performance models (Hutter et al., 2014), to guide the search for good configurations by means of an extension of Bayesian Optimization (Brochu et al., 2010).GGA (Ansótegui et al., 2009) represents parameters as genes and uses a genetic algorithm with a competitive and a non-competitive gender; its newest version GGA++ (Ansótegui et al., 2015) also uses an empirical performance model for guidance.For a more detailed description of these algorithms, we refer the interested reader to the original papers or to the report of the Configurable SAT Solver Challenge (Hutter et al., 2017).
If the cost metric c is running time, several configurators use an adaptive capping strategy (Hutter et al., 2009) to terminate slow algorithm runs prematurely to save time.For example, if the maximal cutoff time used at test time is κ max = 5000 seconds and the best configuration known so far solves each instance in 10 seconds, we can save dramatically by cutting off slow algorithm runs after κ > 10 seconds instead of running all the way to κ max .Since κ is adapted dynamically, each target algorithm run can be issued with a different one.

The Role of the Target Algorithm Wrapper
As depicted in Figure 1, configurators execute the target algorithm with configurations θ ∈ Θ on instances π ∈ Π and measure the resulting cost c(θ, π).To be generally applicable, configurators specify an interface through which they evaluate the cost c(θ, π) of arbitrary algorithms to be optimized.
For a new algorithm A, users need to implement this interface to actually execute A with the desired configuration θ on the desired instance π and measure the desired cost metric c(θ, π) (e.g., running time or solution cost).In order to avoid having to change the algorithm to be optimized, this interface is usually implemented by a wrapper.1In the simplest case, the input to the wrapper is just a parameter configuration, but it can also include an instance, a random seed and computational resource limits, such as a running time cutoff κ.Given these inputs, the wrapper returns the cost function to be optimized (e.g., the running time an algorithm required to solve a problem instance).
From an implementation point of view, the wrapper has to provide a communication layer between the configurator and the target algorithm.It needs to 1. parse the input arguments provided by the configurator and call the target algorithm with them; 2. limit the run's computational resources (i.e., running time and memory consumption); and 3. parse the target algorithm's output and compute the cost function being optimized.
It turns out that there are many possible pitfalls when writing such a wrapper, and we will demonstrate several of these in Section 3. To simplify the task of writing a robust and reliable wrapper that automatically avoids all of the pitfalls described in that section, we provide a Python package called GenericWrapper4AC 2 .This wrapper supports the input format of the most popular configurators ParamILS , GGA, irace and SMAC and automatically limits the computational resources of the target algorithm run with the runsolver tool developed by Roussel (2011).The user only needs to implement two target algorithm-specific functions: 1. translate the provided input arguments to a call of the target algorithm; 2. parse the target algorithm's output to extract its cost.
We provide example implementations of these functions for SAT solvers, other combinatorial problem solvers, machine learning algorithms, and arbitrary black box functions as example configuration scenarios that ship with the newest SMAC release.Appendix A provides additional details about our generic wrapper, and an example wrapper for a SAT solver.

Pitfalls and Best Practices Concerning Algorithm Execution
In this and the next section, we describe common pitfalls in algorithm configuration and illustrate their consequences on existing benchmarks from the algorithm configuration library AClib (Hutter et al., 2014) 3 .Based on the insights we acquired in thousands of algorithm configuration experiments over the years, we propose best practices to avoid these pitfalls.Throughout, we will use the state-of-the-art configurator SMAC (Hutter et al., 2011) as an example, typically optimizing penalized average running time (PAR-10), which counts timeouts at κ max as 10 • κ max .Where not specified otherwise, we ran all experiments using our GenericWrapper4AC wrapper on the University of Freiburg's META cluster, each of whose nodes shares 64 GB of RAM among two Intel Xeon E5-2650v2 8-core CPUs with 20 MB L3 cache and runs Ubuntu 14.04 LTS 64 bit. 4itfall 1: Trusting Your Target Algorithm Many state-of-the-art algorithms have been exhaustively benchmarked and tested with their default parameter configuration.However, since the configuration space of many algorithms is very large, we frequently observed hidden bugs triggered only by rarely-used combinations of parameter values.For example, Hutter et al. (2010a) reported finding bugs in mixed integer programming solvers and Manthey and Lindauer (2016) bugs in SAT solvers.Due to the size of the associated configuration spaces (e.g., 214 parameters and a discretized space of 10 86 configurations in the state-of-the-art SAT solver Riss (Manthey, 2014)), exhaustive checks are infeasible in practice.
Over the years, the types of bugs we have experienced even in commercial solvers (that are the result of dozens of person-years of development time) include: • Segmentation faults, Null pointer exceptions, and other unsuccessful algorithm terminations; • Wrong results (e.g., claiming a satisfiable SAT instance to be unsatisfiable); • Not respecting a specified running time cutoff that is passed as an input; • Not respecting a specified memory limit that is passed as an input; • Rounding down running time cutoffs to the next integer (even if that integer is zero); and • Returning faulty running time measurements (even negative ones!) Effects The various issues above have a multitude of negative effects, from obvious to subtle.If the algorithm run does not respect its resource limits this can lead to congested compute nodes (see Pitfall 3) and to configurator runs that are stuck waiting for an endless algorithm run to finish.Wrongly reported running times (e.g., close to negative infinity in one example) can lead to endless configuration runs when trusted.Rounding down cutoff times can let configurators miss the best configuration (e.g., when they use adaptive capping to cap running times at the best observed running time for an instance -if that running time is below one second then each new configuration will fail on the instance due to using a cutoff of zero seconds).Algorithm crashes can be fairly benign when they are noticed and counted with the highest possible cost, but they can be catastrophic when not recognized as crashes: e.g., when blindly minimizing an algorithm's running time the configurator will typically simply find a configuration that crashes quickly.While this can be exploited to quickly find bugs (Hutter et al., 2010a;Manthey & Lindauer, 2016), obtaining faulty configurations is typically the worst possible result of using algorithm configuration in practice.Bugs that  Difference in test set performance as judged when trusting the target algorithm (red) and using external solution checking (blue).We plot the penalized average running time (PAR-10) scores of Glucose v2.1 on the industrial instances from the SAT Challenge 2012, as a function of time spent for configuration, when the configuration process trusted Glucose v2.1 to be correct.We ran 12 SMAC runs and at each time step show the median and quartiles of their incumbents' scores.The red curve computes these scores trusting the solutions Glucose returns, while the blue curve penalizes faulty configurations with the worst value of 3000 (where faulty configurations are those that yield at least one wrong result on the test instances; such configurations would, e.g., be disqualified in the SAT competition).lead to wrong results tend to be discovered by configurators when optimizing for running time, since (at least for N P-hard problems) we found that such bugs often allow algorithms to find shortcuts and thus shorten running times.Therefore, blindly minimizing running time without solution checking often yields faulty configurations.

Detailed Example
In 2012, we used algorithm configuration to minimize the running time of the state-of-the-art solver glucose (Audemard & Simon, 2009).We quickly found a parameter configuration that appeared to yield new state-of-the-art performance on the industrial instances of the SAT Challenge 20125 ; however, checking this configuration with the authors of Glucose revealed that it led to a bug which made Glucose falsely report some satisfiable instances as unsatisfiable.6 Figure 2 plots the performance of the best Glucose v2.1 configurations that SMAC found over time when trusting Glucose's correctness at configuration time; we contrast the low running time we thought we obtained (when trusting Glucose's outputs) to the dramatically worse true result actually obtained (when using solution checking).This reveals that the parameter configurations we believed to perform well triggered bugs in Glucose.
Best Practice Most of the issues above can be avoided by wrapping target algorithm runs with a reliable piece of code that limits their resources and checks whether they yield correct results.Cast differently, the job of this wrapper is to actually measure the cost function c(θ, π) of interest, which should intuitively heavily penalize any sort of crashes or bugs that lead to wrong results.Our generic wrapper is one readily-available tool for this purpose.
If enough computational time is available, we recommend to first run systems such as SpyBug (Manthey & Lindauer, 2016) to find bugs in the configuration space, and to either fix them or to exclude the faulty part of the configuration space from consideration.Regardless of whether this is done or not, since it is infeasible to perfectly check the entire configuration space, we always recommend to check the returned solution of the target algorithms during the configuration process.For example, for SAT instances our example instantiations of the generic wrapper exploit the standard SAT checker tool routinely used in the SAT competitions to verify the correctness of runs.For solvers that output unsatisfiability proofs, there are also effective tools for checking these proofs (Heule et al., 2014).

Pitfall 2: Not Terminating Target Algorithm Runs Properly
Given the undecidability of the halting problem, target algorithm runs need to be limited by some kind of running time cutoff κ max to prevent poor configurations from running forever.In many AI communities, it is a common practice to set a running time cutoff as part of the cost metric and measure the number of timeouts with that cutoff (e.g., κ max = 5000 seconds in the SAT race series).In algorithm configuration, the ability to prematurely cut off unsuccessful runs also enables adaptive capping (see Section 2).Therefore, it is essential that target algorithm runs respect their cutoff.
Effects Consequences of target algorithm runs not respecting their cutoffs can include: 1.If the target algorithm always uses the maximal cutoff κ max and ignores an adapted cutoff κ < κ max , the configuration process is slowed down since the benefits of adaptive capping are given up; 2. If the target algorithm completely ignores the cutoff, the configuration process may stall since the configurator waits for a slow target algorithm to terminate (which, in the worst case, may never happen); 3. If a wrapper is used that fails to terminate the actual algorithm run but nevertheless returns the control flow to the configurator after the cutoff time κ then the slow runs executed by the configurator will continue to run in parallel and overload the machine, messing up the cost computation (e.g., wallclock time).
Example The latter (quite subtle) issue actually happened in a recent publication that compared GGA++ and SMAC , in which a wrapper bug caused SMAC to perform poorly (Ansótegui et al., 2015).The authors wrote a wrapper for SMAC that tried to terminate its target algorithm runs (here: Glucose or Lingeling) after the specified cutoff time κ We show PAR-10 test set performance for optimizing Cryptominisat with SMAC on Circuit Fuzz , when using a correct and a broken wrapper during configuration, respectively.We show median test performance (measured using a correct wrapper) with quartiles across 80 runs of SMAC .Not terminating target algorithm runs properly eventually slowed down the machine, affecting running time measurements.by sending a KILL signal, but since it ran the target algorithm through a shell (using subprocess.Popen(cmd, shell=True) in Python) the KILL signal only terminated the shell process but not the actual target algorithm (which continued uninterrupted until successful, sometimes for days).When attempting to reproduce the paper's experiments with the original wrapper kindly provided by the authors, over time more and more target algorithms were spawned without being terminated, causing our 16-core machine to slow down and eventually become unreachable.This issue demonstrates that SMAC heavily relies on a robust wrapper that automatically terminates its target algorithm runs properly. 7his downside of SMAC is alleviated by our new tool GenericWrapper4AC.
To illustrate this issue in isolation, we compared SMAC using the standard version of GenericWrapper4AC and a broken version that returns the control flow to the configurator when the running time cutoff is reached, without terminating the target algorithm run process.Figure 3 shows the performance achieved when SMAC is run with either wrapper to configure Cryptominisat (Soos, 2014) for penalized average running time (PAR-10) to solve Circuit Fuzz instances (Brummayer et al., 2012) as used in the CSSC 2014 (Hutter et al., 2017).We executed 80 SMAC runs for each wrapper, with 16 independent parallel runs each on five 16-core machines.Both SMAC versions performed equally well until too many target algorithm processes remained on the machines and prevented SMAC from progressing further.Only on one of the five machines that ran SMAC with the broken wrapper the runs terminated after the specified wallclock-limit of 2 days; after an additional day, three of the remaining machines were still frozen caused by overload and the fourth could not be reached at all.
Best Practice To avoid this pitfall, we recommend to use the GenericWrapper4AC or some other well-tested piece of code to reliably control and terminate target algorithm runs.

Pitfall 3: Slow File System
Related to Pitfall 2, another way to ruin running time measurements by slowing down a machine is to overload the used file system.Each target algorithm run typically has to read the given problem instance and writes some log files, and thus executing many algorithm configuration runs in parallel can stress the file system.
Example 1 Over the years, we have experienced file system issues on a variety of clusters with shared file systems when target algorithm runs were allowed to write to the shared network file system.When executing hundreds (or on one cluster, even thousands) of algorithm configuration runs in parallel, this stressed the file system to the point where the system became very slow for all users and we measured 100-fold overheads in individual target algorithm evaluations.Writing target algorithm outputs to the local file system fixed these issues.
Example 2 Distributing configuration runs across multiple nodes in a compute cluster (e.g., in GGA, irace, or dSMAC can be error-prone if the configurators communicate over the file system.In particular, we experienced issues with several shared network file systems with asynchronous I/O; e.g., on one compute node a file was written, but that file was not immediately accessible (or still empty) on other compute nodes.Often a second read access resolved the problem, but this solution can be brittle; a change of parallelization strategy may in that case yield more robust results.
Example 3 Even when writing target algorithm output to the local file system, we once experienced 200-fold overheads in target algorithm runs (invocations of sub-second target algorithm runs hanging for minutes) due to a subtle combination of issues when performing hundreds of algorithm configuration experiments in parallel.On the Orcinus cluster (part of Compute Canada's Westgrid cluster), which uses a Lustre file system, we had made our algorithm configuration benchmarks read-only to prevent accidental corruption.While that first seemed like a good idea, it disallowed our Python wrapper to create .pycbytecode files and forced it to recompile at every invocation, which in turn triggered a stats call (similar to ls on the Linux command line) for each run.Stats calls are known to be slow on the Lustre file system, and executing them for each sub-second target algorithm run on hundreds of compute nodes lead to extreme file system slowdowns.After testing many other possible reasons for the slowdowns, removing the read-only condition immediately fixed all issues.
Best Practice Issues with shared file systems on compute clusters can have subtle reasons and sometimes require close investigation (as in our Example 3).Nevertheless, most issues can be avoided by using the faster local file system (typically /tmp/, or even better, a temporary job-specific subdirectory thereof8 ) for all temporary files, and by measuring CPU time instead of wallclock time (at least for sequential algorithms).Both best practices are implemented in the GenericWrapper4AC package.

Pitfall 4: Comparing Configurators using Different Wrappers
The required functionalities of the target algorithm wrapper differ slightly for different configurators.For example, SMAC and ParamILS trust the wrapper to terminate target algorithms, but GGA sends a KILL signal on its own.Therefore, sometimes configurators are compared by using different wrappers.However, if this is not done properly, it can lead to a biased comparison between configurators.
Effect Using different wrappers for different configurators can lead to different behaviors of the target algorithm and hence, to different returned performance values for the same input.If the configurators receive different performance measurements, they will optimize different objective functions and their runs become incomparable.
Example During the early development of SMAC (before any publication), we used the same wrappers for ParamILS and SMAC but an absolute path to the problem instance for one and a relative path for the other.Even this tiny difference lead to reproducible differences of running time measurements of up to 20% when optimizing an algorithm implemented in UBCSAT 1.1.0(Tompkins & Hoos, 2005).The reason was that that version of UBCSAT stored its callstring in its heap space such that the number of characters in the instance name affected data locality and therefore the number of cache misses and the running time (whereas the number of search steps stayed the same).9This subtle issue demonstrates the importance of using exactly the same wrapper for all configurators being compared.
Best Practice We recommend to use a single wrapper (e.g., based on GenericWrapper4AC) when comparing configurators against each other, in order to guarantee that all configurators optimize the same objective.Maintaining and testing a single wrapper also requires less effort than doing so for multiple wrappers.For studies comparing configurators, it is also paramount to use tried-and-tested publicly available benchmark scenarios (lowering the riks of typos, etc); our algorithm configuration benchmark library AClib (Hutter et al., 2014) provides a very broad collection of such benchmarks.

Pitfalls and Best Practices Concerning Over-Tuning
A common issue in applying algorithm configuration is the over-tuning effect (Birattari, 2004;Hutter et al., 2007).Over-tuning is very related to the concept of over-fitting in machine learning and denotes the phenomenon of finding parameter configurations that yield strong performance for the training task but do not generalize to test tasks.To safeguard against over-tuning effects, we recommend to always evaluate generalization performance (typically, using a set of benchmark instances disjoint from the benchmarks used for training).In the following, we discuss three pitfalls related to over-tuning.

Pitfall 5: Over-tuning to Random Seeds
Many algorithms are randomized (e.g., SAT solvers or AI planners).However, in many communities, the random seeds of these algorithms are fixed to simulate a deterministic behavior and to ensure reproducibility of benchmark results.
Effect Ignoring the stochasticity of an algorithm in algorithm configuration by fixing the random seed can lead to over-tuning effects to this seed, i.e., finding a configuration that yields good performance with this fixed random seed (or set of seeds) but poor performance when used with other random seeds.The extreme case is not to only fix the random seed, but to tune the random seed, which can lead to an even stronger over-tuning effect.
Example To illustrate over-tuning to a random seed in its purest form, independent of a difference between training and test instances, we optimized the parameters of the local-search SAT solver Saps (Hutter et al., 2002) on a single instance, the only difference between training and test being the set of random seeds used.We used different settings of SMAC to handle random seeds: SMAC (1) used only one run with a fixed seed to evaluate a configuration; SMAC (10) and SMAC (100) evaluated each configuration with a fixed set of 10 or 100 random seeds, respectively; and standard SMAC handled the random seeds itself (using a larger number of seeds to evaluate the best configurations).
As a cost metric, we minimized the number of local search steps (the solver's so-called runlength) since this is perfectly reproducible.For the parameter configurations recommended at each step of each SMAC run, we measured SMAC 's training cost (as the mean across the respective sets of seeds discussed above) as well as its test cost (the mean runlength across 1000 fixed random seeds that were disjoint from the sets of seeds used for configuration).
Figure 4 shows median costs across 10 SMAC runs, contrasting training cost (left) and test cost (right).On training, SMAC (1) quickly improved and achieved the best training cost on its one random seed, but its performance does not generalize to the test seeds.SMAC (10) and SMAC (100) were slower but generalized better, and standard SMAC was both fast and generalized best by adaptively handling the number of seeds to run for each configuration.
Best Practice For randomized algorithms, we recommend to tune parameter configurations across different random seeds-most configurators will take care of the required number of random seeds if the corresponding options are used.If a configuration's performance does not even generalize well to new random seeds, we expect it to also not generalize well to new instances.Furthermore, the number of available instances is often restricted, but there are infinitely many random seeds which can be easily sampled.Likewise, when there are only few test instances, at validation time we recommend to perform multiple runs with different random seeds for each test instance.

Pitfall 6: Over-tuning to Training Instances
The most common over-tuning effect is over-tuning to the set of training instances, i.e., finding configurations that perform well on training instances but not on new unseen instances.This can happen if the training instances are not representative for the test instances; in particular this is often an issue if the training instance set is too small or the instances are not homogeneous (Hutter et al., 2010b;Schneider & Hoos, 2012), i.e., if there exists no single configuration with strong performance for all instances.
Effect In practice, over-tuned configurations are of little value, because users are not interested in configurations that only perform well on a small finite set of instances but that also perform well on new instances with similar characteristics.Phrasing this more generally, research insights should also generalize to experiments with similar characteristics.
Example To illustrate this problem, we studied training and test performance of various configurations for three exemplary benchmarks (see Figure 5): Clasp on N-Rooks We studied running time of the solver Clasp (Gebser et al., 2012) on N-Rooks instances (Manthey & Steinke, 2014), a benchmark from the Configurable SAT Solver Challenge (CSSC 2014;Hutter et al. (2017)).In this case, the running times on the training and test set were almost perfectly linearly correlated, with a Spearman correlation coefficient of 0.99, i.e., the ranking of the configurations on both sets is nearly identical; this is also visualized in Figure 5a.This is a very good case for applying algorithm configuration, and, correspondingly, in the CSSC 2014 algorithm configuration yielded large improvements for this benchmark.
Lingeling on mixed SAT We reconstructed a benchmark from Ansótegui et al. (2015) in which they optimized Lingeling (Biere, 2014) on a mixed set of industrial SAT instances.Instead of randomly splitting the data into train and test instances, they first created a training set by removing hard instances (i.e., not solved within the cutoff time by reference solvers) and used these remaining hard instances as test instances.Figure 5b shows that SMAC improved the running time of Lingeling on the training set but that these improvements did not generalize to the test instances.In fact, the optimized configurations (blue dots) are weakly correlated with a Spearman correlation coefficient of 0.15.The benchmark's heterogeneity and the corresponding mismatch between training and test set make this benchmark poorly suited for applying algorithm configuration and call for portfolio approaches (Xu et al., 2008;Malitsky et al., 2012;Lindauer et al., 2015) that can handle heterogeneous instances.
Clasp on LABS Figure 5c shows another benchmark from the CSSC: configuration of Clasp on SAT-encoded low autocorrelation binary sequence (LABS) benchmarks (Mugrauer & Balint, 2013).This illustrates a rare worst case for algorithm configuration, in which performance even degrades on the training set, which is possible due to SMAC 's (and any other configurator's) racing approach: the configurator already changes the incumbent before all training instances have been evaluated, and if a subset is not representative of the full set this may lead to performance degradation on the full set.Although visually not apparent from Figure 5c, for this benchmark, there was also no clear correlation between the running time on training and test instances for good configurations (with a correlation coefficient of 0.42 on the 20% best-performing randomly sampled configurations).
While we have occasionally observed such strong heterogeneity on instances with very heterogeneous sources, it was very surprising to observe this in a case where all instances stemmed from the same instance family.We therefore analyzed this benchmark further (Hutter et al., 2017), showing that twice as many SMAC runs with a fivefold larger configuration budget managed to improve training performance slightly.However, that improvement on the training set did not generalize to the test set due to the benchmark's heterogeneity.Again, for such heterogeneous benchmarks we recommend the usage of portfolio approaches.
Best Practice Over-tuning is often not easy to avoid before running algorithm configuration experiments, since the effect can only be observed afterwards (for example by scatter plots such as in Figure 5).Nevertheless, the following strategies minimize the chance of over-tuning (see also Section 5): 1.The training instances should be representative of the test instances; 2. The training set should be relatively large (typically hundreds to thousands of instances) to increase the chance of being representative; 3. The instance sets should stem from a similar application, use context, etc., increasing the likelihood that they have similar structures which can be exploited with similar solution strategies.

Pitfall 7: Over-tuning to a Particular Machine Type
In the age of cloud computing and large compute clusters, an obvious idea is to use these remotely-accessible compute resources to benchmark algorithms and configure them.However in the end, these remote machines are not always the production systems the algorithms are used on in the end.Geschwender et al. (2014) showed in a preliminary study that it is possible in principle to configure algorithms in the cloud and for the found configurations to perform well on another machine.Unfortunately, this does not hold for all kind of algorithms -for example, the performance of solvers for SAT (Aigner et al., 2013) and mixed integer programming (Lodi & Tramontani, 2014) can depend strongly on the used machine type (including hardware, operating system and installed software libraries) Effect If different parameter configurations of an algorithm perform well on the machine used for configuration than on the production machine, we might choose configurations that perform poorly on our production system.
Example An example for such machine-dependent algorithms are SAT solvers that are often highly optimized against cache misses (Aigner et al., 2013).To study the effect of different machines, we optimized three SAT solvers from the configurable SAT solver challenge (Hutter et al., 2017), namely Minisat-HACK-999ED (Oh, 2014), Clasp (Gebser et al., 2012) and Lingeling (Biere, 2014) on Circuit Fuzz instances (Brummayer et al., 2012).As different machine types, we used AWS m4.4xlarge instances with 2.4-GHz Intel Xeon E5-2676 v3 CPUs with 30MB level-3 cache and the META-cluster at the University of Freiburg with 2.6GHz Intel Xeon E5-2650 v2 CPUs with 20MB level-3 cache.On both systems, we ran Ubuntu 14.04 64bit and allowed for a memory limit of 3GB for each solver run.The binaries were statically compiled such that they are not linked against different libraries on the different systems.We configured the solvers with 12 independent runs and validated the cost on test instances on the same system used for the configuration process.
Table 1 lists the ranking and the PAR10 scores of the solvers on each machine (showing the test cost of the configuration performing best on training); we note that the PAR10 scores are only comparable on the same system.The ranking between Clasp and Lingeling is always the same, with Lingeling performing worst on both systems.However, Minisat-HACK-999ED is ranked differently on both machines, i.e., best on AWS, but second on the Meta-cluster.If the AWS cloud would be our benchmark environment we would decide for Minisat-HACK-999ED based on the AWS results, but this would not be the best choice on the Meta cluster.We verified this by also validating the configurations from AWS on the Meta-cluster; in that setting the configured Minisat-HACK-999ED performed even worse than Lingeling and Clasp.
Best Practice We note that this pitfall exists only for some machine-sensitive algorithms.Therefore, we recommend to investigate whether an algorithm at hand has machinedependent performance, for example, by validating the performance of various configurations on both the system used for configuration and the production system.

Further Recommendations for Effective Configuration
In the following, we describe recommendations for users of algorithm configuration systems to obtain parameter configurations that will perform better in production.Some of these recommendations are rules of thumb, since the involved factors for a successful configuration can be very complex and can change across configuration scenarios.For general empirical algorithmics, Hoos (2017) recommends further best practices, including design, reports and analysis of computational experiments.
Training and Test Sets As discussed before, we strongly recommend to split the available instances into a training and a test set to obtain an unbiased estimate of generalization performance from the test set.To obtain trivial parallelization of randomized configuration procedures, we recommend to run n independent configuration runs and use the training set to select the best of the n resulting configurations (Hutter et al., 2012).Only that single chosen configuration should be evaluated on the test set; we explicitly note that we cannot select the configuration that performs best on the test set, because that would amount to peeking at our test data and render performance estimates on the test set biased.
Representative Instances and Running Time Cutoff Intuitively, instances for which every parameter configuration times out do not help the configurator to make progress.One strategy can be to remove these from the training set.However, this comes with the risk to bias the training set towards easy instances and should be used with caution.Generally, we therefore recommend to use training instances for the configuration process that are representative of the ones to be solved later.Using training instances from a range of hardness can also often help yield configurations that generalize (Hoos et al., 2013).If feasible, we recommend to select instances and running time cutoffs such that roughly 75% or more of the training instances used during configuration can be solved by the initial parameter configuration within the cutoff.We emphasize that -while the configuration protocol may in principle choose to subsample the training instances in arbitrary ways -the test set should never be touched and not pre-evaluated to ensure an unbiased cost estimate of the optimized configurations in the end (see Pitfall 6).To select a good training instance set, Bayless et al. (2014) proposed a way to quantify whether an instance set is a good proxy for another instance set.Furthermore, Styles and Hoos (2015) proposed a splitting strategy of the instances for better scaling to hard instances.They split the instances into a training, validation and test set to use easy instances during configuration for fast progress and select a configuration on the harder validation set such that the configuration will perform well on the hard test set.
Homogeneous vs Heterogenous Instance Sets Sometimes configurators are used to obtain well-performing and robust configurations on a heterogeneous instance set.However, we know from algorithm selection (Rice, 1976;Kotthoff, 2014) that often no single configuration exists that performs well for all instances in a heterogeneous set, but a portfolio of configurations is required to obtain good performance (Xu et al., 2011;Kadioglu et al., 2010).Furthermore, the task of algorithm configuration becomes a lot harder if all instances can be solved best with very different configurations.Therefore, we recommend to use algorithm configuration mainly on homogeneous instance sets (Schneider & Hoos, 2012), i.e., instances with similar characteristics where we can expect to find one configuration that performs well on all instances.Furthermore, the size of the used instance set should be adjusted accordingly to the homogeneity of the instance set: on homogeneous instance sets, 50 instances might suffice for good generalization performance to new instances, but on fairly heterogeneous instance sets, we recommend to use at least 300 or, if possible, more than 1000 instances to obtain a robust parameter configuration.
Appropriate Configuration Settings To use configurators, the user has to set the available configuration budget for the configurator.If the configuration budget is too small, the configurator might make little or no progress within in.In contrast, if the configuration budget is too large, we waste a lot of time and computational resources because the configurator might converge long before the budget is used up.A good rule of thumb in our experience is to use a budget that equals at least the expected running time of the default configuration on 200 to 1000 instances.In practice, an effective configuration budget strongly depends on several factors, including heterogeneity of the instance set (more heterogeneous instance sets require a larger configuration budget) or size of the configura-tion space (larger configuration spaces require more time to search effectively (Hutter et al., 2017)).
Efficient Use of Parallel Resources Some configurators (such as GGA, irace and dSMAC ) can make use of parallel resources, while others (such as ParamILS and SMAC ) benefit from executing several independent parallel runs10 (and using the result from the one with the best training set performance; see, e.g., Hutter et al. (2012)).In the special case of GGA, using more parallel resources can actually improve the adaptive capping mechanism.Given k cores, we therefore recommend to execute one GGA run with k cores, but k independent ParamILS or SMAC runs with one core each.While this protocol was not used in early works 11 , it has been used in more recent evaluations (Ansótegui et al., 2015;Hutter et al., 2017).
Reasonable Configuration Space Another challenge in using algorithm configuration systems is to find the best configuration space.The user has to decide which parameters to optimize and which ranges to allow.The optimal set of parameters is often not clear and in case of doubt, we recommend to add more parameters to the configuration space and to use generous value ranges.With SMAC , we have successfully optimized configuration spaces with hundreds of parameters (Thornton et al., 2013).However, we note that unreasonably large configuration spaces are hard to configure and require substantially larger configuration budgets.For example, the state-of-the-art SAT solver Lingeling (Biere, 2013) has more than 300 parameters and most of them have a value range between 0 and 32bit maxint, but most of these parameters are either not really relevant for optimizing Lingeling's running time or the relevant value ranges are much smaller.Even though Lingeling can already substantially benefit from configuration we expect that with a more carefully designed configuration space even better results can be obtained.Therefore, we recommend to avoid including such parameters and to use smaller value ranges if corresponding expert knowledge is available.
Which Parameters to Tune Parameters should never be part of the configuration space if they change the semantics of the problem to be solved; e.g., do not tune the allowed memory or parameters that control whether a run is counted as successful (such as the allowed optimality gap in an optimization setting).Furthermore, to obtain an unbiased estimate of a configuration's performance across seeds one should not include the seed (or parameters with a similar effect) as a tunable parameter.
Running Time Metrics A common cost metric in algorithm configuration is running time.However, obtaining clean running time measurements can be tricky because benchmark machines can be influenced by other running processes or I/O load on a shared file system (see Pitfall 3).Therefore, we recommend to measure CPU time instead of wallclock time.However, CPU time can also sometimes be brittle; e.g., its resolution can be insufficient for very short target algorithm runs, such as milliseconds.One solution for this case is to measure elementary operations, such as search steps of a local search algorithm or MEMS (number of memory accesses (Knuth, 2011)); however, it has to be ensured that such proxy metrics correlate well with running time.
Comparing Configurators on Existing, Open-Source Benchmarks Although algorithm configuration has been well established for over a decade, nearly every new paper on this topic uses a new set of benchmarks to compare different configurators.This makes it harder to assess progress in the field, and every new benchmark could again suffer from one of the pitfalls described above.Therefore, we recommend to use existing and open-source algorithm configuration benchmarks that are already well tested and can be freely used by the community.The only existing library of such benchmarks we are aware of is the algorithm configuration library AClib (Hutter et al., 2014), which comprises 326 benchmarks in Version 1.2 and allows users to pick benchmarks from different domains (e.g., mixed integer programming, AI Planning, SAT, machine learning) and with different characteristics (e.g., small or large configuration spaces).

A Details on GenericWrapper4AC
Listing 1 shows an example for how to extend the GenericWrapper4AC to wrap the wellknown SAT Solver MiniSAT (Eén & Sörensson, 2004).Since the output format is standardized in the SAT community, we already provide a domain-specific generic wrapper, called SatWrapper, which can parse and verify the SAT solver's output using standard tools from the annual SAT competitions.Therefore, SAT solver users only need to implement one method, which constructs a command line call string for their SAT solver from the provided input arguments (parameter settings, instance, cutoff time, seed).In the example shown, the command line call of MiniSAT consists of passing the random seed (Line 4), adding all parameters in the format parameter=value (Lines 5 and 6), and appending the CNF instance name at the end (Line 7).Importantly, all aspects of handling cutoff times, measuring running times, etc, is already taken care of to avoid the pitfalls discussed in Section 3.For users of algorithm configuration outside SAT solving, Listing 2 shows an example for how to write a function process results to parse algorithm outputs.Let us assume the target algorithm only prints the target cost to be minimized (similar to the format of irace (López-Ibáñez et al., 2016)).Reading the output of the provided file pointer fp, the function builds and returns a dictionary which includes the cost value (called quality) and a status, which is either SUCCESS if the target algorithm printed only a single number or CRASHED otherwise.Other states can be TIMEOUT for using more than the time cutoff κ or ABORT to signal the configurator to abort the AC experiment because of major issues.Furthermore, the exit code of the target algorithm run is also provided (but not used in our example).Another possible functionality that is not shown here is to implement a method to verify the target algorithm's returned solution.• Calling the target algorithm and limiting its resource limits using the runsolver tool (Roussel, 2011) • Measuring the CPU time of the target algorithm run (using runsolver) • Returning the cost of the target algorithm run to the configurator The GenericWrapper4AC is available at GitHub13 and can be easily installed via python setup.pyinstall (including the runsolver).

Figure 1 :
Figure 1: Workflow of Algorithm Configuration

Figure 2 :
Figure2: Difference in test set performance as judged when trusting the target algorithm (red) and using external solution checking (blue).We plot the penalized average running time (PAR-10) scores of Glucose v2.1 on the industrial instances from the SAT Challenge 2012, as a function of time spent for configuration, when the configuration process trusted Glucose v2.1 to be correct.We ran 12 SMAC runs and at each time step show the median and quartiles of their incumbents' scores.The red curve computes these scores trusting the solutions Glucose returns, while the blue curve penalizes faulty configurations with the worst value of 3000 (where faulty configurations are those that yield at least one wrong result on the test instances; such configurations would, e.g., be disqualified in the SAT competition).
Figure3: Effect of a broken wrapper that does not terminate target algorithm runs properly.We show PAR-10 test set performance for optimizing Cryptominisat with SMAC on Circuit Fuzz , when using a correct and a broken wrapper during configuration, respectively.We show median test performance (measured using a correct wrapper) with quartiles across 80 runs of SMAC .Not terminating target algorithm runs properly eventually slowed down the machine, affecting running time measurements.

Figure 4 :
Figure 4: Optimizing Saps with SMAC on one "quasigroup with holes" instance (QWH) and different numbers of random seeds.The left plot shows the estimated median training cost and the right plot the test cost of Saps over time.

( a )
Figure 5: Comparing training and test performance of different configurations to study whether these performances on both sets are correlated.Red dots indicate randomly sampled configurations, the black cross marks the performance of the default configuration of the solver, and blue dots correspond to incumbent configurations of 16 SMAC runs.
t c o m m a n d l i n e a r g s ( s e l f , r u n a r g s , c o n f i g ) : 4 cmd = " m i n i s a t −rnd−s e e d=%d" %( r u n a r g s [ " s e e d " ] ) 5f o r name , v a l u e in c o n f i g .i t e m s ( ) : 6 cmd += " %s=%s " %(name , v a l u e ) 7 cmd += " %s " %( r u n a r g s [ " i n s t a n c e " ] ) 8 return cmdListing 1: Example GenericWrapper for SAT Solver MiniSAT, building on domain-specific SatWrapper t c o m m a n d l i n e a r g s ( s e l f , r u n a r g s , c o n f i g o c e s s r e s u l t s ( s e l f , fp , e x i t c o d e ) : 7 try : 8 resu ltMa p = { ' s t a t u s ' : 'SUCCESS ' , 9 ' q u a l i t y ' : f l o a t ( f p .r e a d ( )) } 10 except V a l u e E rr o r : 11 res ultMa p = { ' s t a t u s ' : 'CRASHED ' } 12 13 return res ultM ap Listing 2: Example GenericWrapper from scratch Except these two target algorithm-specific functions, the GenericWrapper4AC handles everything else, including • Parsing the input format; native interfaces to ParamILS , ROAR and SMAC are supported right now, GGA(++) and irace with slight modifications (see AClib2 12 for examples).

Table 1 :
Three SAT solvers from the configurable SAT solver challenge on Circuit Fuzz instances on three different hardware systems.