This book presents a solution to the challenging issue of optimizing expensive-to-evaluate industrial problems such as the hyperparameter tuning of machine learning models. The approach combines two well-established concepts, Surrogate-Based Optimization (SBO) and parallelization, to efficiently search for optimal parameter setups with as few function evaluations as possible.
Through in-depth analysis, the need for parallel SBO solvers is emphasized, and it is demonstrated that they outperform model-free algorithms in scenarios with a low evaluation budget. The SBO approach helps practitioners save significant amounts of time and resources in hyperparameter tuning as well as other optimization projects. As a highlight, a novel framework for objectively comparing the efficiency of parallel SBO algorithms is introduced, enabling practitioners to evaluate and select the most effective approach for their specific use case.
Based on practical examples, decision support is delivered, detailing which parts of industrial optimization projects can be parallelized and how to prioritize which parts to parallelize first. By following the framework, practitioners can make informed decisions about how to allocate resources and optimize their models efficiently.
Author(s): Frederik Rehbach
Series: Studies in Computational Intelligence, 1099
Publisher: Springer
Year: 2023
Language: English
Pages: 122
City: Cham
Acknowledgments
Contents
Symbols
1 Introduction
1.1 Motivation
1.2 Research Questions
1.3 Outline
1.4 Publications
References
2 Background
2.1 Surrogate-Based Optimization
2.2 Evolutionary Algorithms
2.3 A Taxonomy for Parallel SBO
2.3.1 Parallel Objective Function Evaluation (L1)
2.3.2 Parallel Model Building (L2)
2.3.3 Parallel Evaluation Proposals (L3)
2.3.4 Multi-algorithm Approaches (L4)
2.3.5 Recommendations for Practitioners
2.4 Parallel SBO—A Literature Review
References
3 Methods/Contributions
3.1 Benchmarking Parallel SBO Algorithms
3.1.1 A Framework for Parallel SBO Algorithms
3.1.2 Conclusions
3.2 Test Problems
3.2.1 Simulation Based Functions
3.2.2 Experiments
3.2.3 Results
3.2.4 Conclusions
3.3 Why Not Other Parallel Algorithms?
3.3.1 A Hybrid SBO Baseline
3.3.2 Experiments
3.3.3 Results
3.3.4 Conclusions
3.4 Parallelization as Hyper-Parameter Selection
3.4.1 Multi-config SBO
3.4.2 Experiments
3.4.3 Results
3.4.4 Conclusions
3.5 Exploration Versus Exploitation
3.5.1 Experiments
3.5.2 Results
3.5.3 Conclusions
3.6 Multi-local Expected Improvement
3.6.1 Batched Multi-local Expected Improvement
3.6.2 Experiments
3.6.3 Results
3.6.4 Conclusions
3.7 Adaptive Parameter Selection
3.7.1 Benchmark Based Algorithm Configuration
3.7.2 Experiments
3.7.3 Results
3.7.4 Conclusions
References
4 Application
4.1 Electrostatic Precipitators
4.1.1 Problem Description
4.1.2 Methods
4.1.3 Results
4.1.4 Conclusions
4.2 Application Case Studies
References
5 Final Evaluation
5.1 Define
5.2 Analyze
5.3 Enhance
5.4 Final Evaluation
References
Appendix Glossary
