Data-Driven Evolutionary Optimization

Foreword
Preface
Contents
Acronyms
Symbols
1 Introduction to Optimization
1.1 Definition of Optimization
1.1.1 Mathematical Formulation
1.1.2 Convex Optimization
1.1.3 Quasi-convex Function
1.1.4 Global and Local Optima
1.2 Types of Optimization Problems
1.2.1 Continuous Versus Discrete Optimization
1.2.2 Unconstrained Versus Constrained Optimization
1.2.3 Single Versus Multi-objective Optimization
1.2.4 Deterministic Versus Stochastic Optimization
1.2.5 Black-Box and Data-Driven Optimization
1.3 Multi-objective Optimization
1.3.1 Mathematical Formulation
1.3.2 Pareto Optimality
1.3.3 Preference Modeling
1.3.4 Preference Articulation
1.4 Handling Uncertainty in Optimization
1.4.1 Noise in Evaluations
1.4.2 Robust Optimization
1.4.3 Multi-scenario Optimization
1.4.4 Dynamic Optimization
1.4.5 Robust Optimization Over Time
1.5 Comparison of Optimization Algorithms
1.5.1 Algorithmic Efficiency
1.5.2 Performance Indicators
1.5.3 Reliability Assessment
1.5.4 Statistical Tests
1.5.5 Benchmark Problems
1.6 Summary
References
2 Classical Optimization Algorithms
2.1 Unconstrained Optimization
2.1.1 The Gradient Based Method
2.1.2 Newton's Method
2.1.3 Quasi-Newton Method
2.2 Constrained Optimization
2.2.1 Penalty and Barriers
2.2.2 Lagrangian Multipliers
2.3 Derivative-Free Search Methods
2.3.1 Line Search and Pattern Search
2.3.2 Nelder-Mead Simplex Method
2.3.3 Model-Based Derivative-Free Search Methods
2.4 Deterministic Global Optimization
2.4.1 Lipschitzian-Based Methods
2.4.2 DIRECT
2.5 Summary
References
3 Evolutionary and Swarm Optimization
3.1 Introduction
3.2 Genetic Algorithms
3.2.1 Definitions
3.2.2 Representation
3.2.3 Crossover and Mutation
3.2.4 Environmental Selection
3.3 Real-Coded Genetic Algorithms
3.3.1 Real-Valued Representation
3.3.2 Blended Crossover
3.3.3 Simulated Binary Crossover and Polynomial Mutation
3.4 Evolution Strategies
3.4.1 (1+1)-ES
3.4.2 Evolution Strategies with One Global Step Size
3.4.3 Evolution Strategies with Individual Step Sizes
3.4.4 Reproduction and Environmental Selection
3.4.5 Covariance Matrix Adaptation Evolution Strategy
3.5 Genetic Programming
3.5.1 Tree-Based Genetic Programming
3.5.2 Initialization
3.5.3 Crossover and Mutation
3.6 Ant Colony Optimization
3.6.1 Overall Framework
3.6.2 Extensions
3.7 Differential Evolution
3.7.1 Initialization
3.7.2 Differential Mutation
3.7.3 Differential Crossover
3.7.4 Environmental Selection
3.8 Particle Swarm Optimization
3.8.1 Canonical Particle Swarm Optimization
3.8.2 Competitive Swarm Optimizer
3.8.3 Social Learning Particle Swarm Optimizer
3.9 Memetic Algorithms
3.9.1 Basic Concepts
3.9.2 Lamarckian Versus Baldwinian Approaches
3.9.3 Multi-objective Memetic Algorithms
3.9.4 Baldwin Effect Versus Hiding Effect
3.10 Estimation of Distribution Algorithms
3.10.1 A Simple EDA
3.10.2 EDAs for Discrete Optimization
3.10.3 EDAs for Continuous Optimization
3.10.4 Multi-objective EDAs
3.11 Parameter Adaptation and Algorithm Selection
3.11.1 Automated Parameter Tuning
3.11.2 Hyper-heuristics
3.11.3 Fitness Landscape Analysis
3.11.4 Automated Recommendation Systems
3.12 Summary
References
4 Introduction to Machine Learning
4.1 Machine Learning Problems
4.1.1 Clustering
4.1.2 Dimension Reduction
4.1.3 Regression
4.1.4 Classification
4.2 Machine Learning Models
4.2.1 Polynomials
4.2.2 Multi-layer Perceptrons
4.2.3 Radial-Basis-Function Networks
4.2.4 Support Vector Machines
4.2.5 Gaussian Processes
4.2.6 Decision Trees
4.2.7 Fuzzy Rule Systems
4.2.8 Ensembles
4.3 Learning Algorithms
4.3.1 Supervised Learning
4.3.2 Unsupervised Learning
4.3.3 Reinforcement Learning
4.3.4 Advanced Learning Algorithms
4.4 Multi-objective Machine Learning
4.4.1 Single- and Multi-objective Learning
4.4.2 Multi-objective Clustering, Feature Selection and Extraction
4.4.3 Multi-objective Ensemble Generation
4.5 Deep Learning Models
4.5.1 Convolutional Neural Networks
4.5.2 Long Short-Term Memory Networks
4.5.3 Autoassociative Neural Networks and Autoencoder
4.5.4 Generative Adversarial Networks
4.6 Synergies Between Evolution and Learning
4.6.1 Evolutionary Learning
4.6.2 Learning for Evolutionary Optimization
4.7 Summary
References
5 Data-Driven Surrogate-Assisted Evolutionary Optimization
5.1 Introduction
5.2 Offline and Online Data-Driven Optimization
5.2.1 Offline Data-Driven Optimization
5.2.2 Online Data-Driven Optimization
5.3 Online Surrogate Management Methods
5.3.1 Population-Based Model Management
5.3.2 Generation-Based Model Management
5.3.3 Individual-Based Model Management
5.3.4 Trust Region Method for Memetic Algorithms
5.4 Bayesian Model Management
5.4.1 Acquisition Functions
5.4.2 Evolutionary Bayesian Optimization
5.4.3 Bayesian Evolutionary Optimization
5.5 Bayesian Constrained Optimization
5.5.1 Acquisition Function for Constrained Optimization
5.5.2 Two-Stage Acquisition Functions
5.6 Surrogate-Assisted Robust Optimization
5.6.1 Bi-objective Formulation of Robust Optimization
5.6.2 Surrogate Construction
5.7 Performance Indicators for Surrogates
5.7.1 Accuracy
5.7.2 Selection-based Performance Indicator
5.7.3 Rank Correlation
5.7.4 Fitness Correlation
5.8 Summary
References
6 Multi-surrogate-Assisted Single-objective Optimization
6.1 Introduction
6.2 Local and Global Surrogates Assisted Optimization
6.2.1 Ensemble Surrogate Model
6.2.2 Multi-surrogate for Single-objective Memetic Optimization
6.2.3 Multi-surrogate for Multi-objective Memetic Optimization
6.2.4 Trust Region Method Assisted Local Search
6.2.5 Experimental Results
6.3 Two-Layer Surrogate-Assisted Particle Swarm Optimization
6.3.1 Global Surrogate Model
6.3.2 Local Surrogate Model
6.3.3 Fitness Estimation
6.3.4 Surrogate Management
6.3.5 Experimental Results and Discussions
6.4 Committee Surrogate Assisted Particle Swarm Optimization
6.4.1 Committee of Surrogate Models
6.4.2 Infill Sampling Criteria
6.4.3 Overall Framework
6.4.4 Experimental Results on Benchmark Problems
6.5 Hierarchical Surrogate-Assisted Multi-scenario Optimization
6.5.1 Multi-scenario Airfoil Optimization
6.5.2 Hierarchical Surrogates for Multi-scenario Optimization
6.6 Adaptive Surrogate Selection
6.6.1 Basic Idea
6.6.2 Probabilistic Model for Surrogate Selection
6.7 Summary
References
7 Surrogate-Assisted Multi-objective Evolutionary Optimization
7.1 Evolutionary Multi-objective Optimization
7.1.1 Hypothesis and Methodologies
7.1.2 Decomposition Approaches
7.1.3 Dominance Based Approaches
7.1.4 Performance Indicator Based Approaches
7.2 Gaussian Process Assisted Randomized Weighted Aggregation
7.2.1 Challenges for Surrogate-Assisted Multi-objective Optimization
7.2.2 Efficient Global Optimization Algorithm
7.2.3 Extension to Multi-objective Optimization
7.3 Gaussian Process Assisted Decomposition-Based Multi-objective Optimization
7.3.1 MOEA/D
7.3.2 Main Framework
7.3.3 Local Surrogate Models
7.3.4 Surrogate Management
7.3.5 Discussions
7.4 High-Dimensional Multi-objective Bayesian Optimization
7.4.1 Main Challenges
7.4.2 Heterogeneous Ensemble Construction
7.4.3 Pareto Approach to Multi-objective Bayesian Optimization
7.4.4 Overall Framework
7.5 Summary
References
8 Surrogate-Assisted Many-Objective Evolutionary Optimization
8.1 New Challenges in Many-Objective Optimization
8.1.1 Introduction
8.1.2 Diversity Versus Preferences
8.1.3 Search for Knee Solutions
8.1.4 Solving Problems with Irregular Pareto Fronts
8.2 Evolutionary Many-Objective Optimization Algorithms
8.2.1 Reference Vector Guided Many-Objective Optimization
8.2.2 A Knee-Driven Many-Objective Optimization Algorithm
8.2.3 A Two-Archive Algorithm for Many-Objective Optimization
8.2.4 Corner Sort for Many-Objective Optimization
8.3 Gaussian Process Assisted Reference Vector Guided Many-Objective Optimization
8.3.1 Surrogate Management
8.3.2 Archive Maintenance
8.4 Classification Surrogate Assisted Many-Objective Optimization
8.4.1 Main Framework
8.4.2 Radial Projection Based Selection
8.4.3 Reference Set Based Dominance Relationship Prediction
8.4.4 Surrogate Management
8.4.5 Surrogate-Assisted Environmental Selection
8.5 Dropout Neural Network Assisted Many-Objective Optimization
8.5.1 AR-MOEA
8.5.2 Efficient Deep Dropout Neural Networks
8.5.3 Model Management
8.5.4 Overall Framework of EDN-ARMOEA
8.5.5 Operational Optimization in Crude Oil Distillation Units
8.6 Summary
References
9 Knowledge Transfer in Data-Driven Evolutionary Optimization
9.1 Introduction
9.2 Co-Training for Surrogate-Assisted Interactive Optimization
9.2.1 Overall Framework
9.2.2 Surrogate for Interval Prediction
9.2.3 Fitness Estimation
9.2.4 An Improved CSSL
9.2.5 Surrogate Management
9.3 Semi-Supervised Learning Assisted Particle Swarm Optimization
9.3.1 Algorithm Framework
9.3.2 Social Learning Particle Swarm Optimization
9.3.3 Surrogate Management Strategy
9.3.4 Selection of Unlabeled Data
9.3.5 Experimental Results and Discussions
9.4 Knowledge Transfer between Problems in Multi-objective Optimization
9.4.1 Domain Adaptation for Transfer Learning
9.4.2 Knowledge Transfer from Cheap to Expensive Problems
9.4.3 CE-BDA for Data Augmentation
9.4.4 Evolutionary Multi-Objective Bayesian Optimization
9.5 Knowledge Transfer between Objectives in Multi-objective Optimization
9.5.1 Motivation
9.5.2 Parameter Based Transfer Learning
9.5.3 Overall Framework
9.6 Data-Driven Multi-fidelity Transfer Optimization
9.6.1 Transfer Learning for Bi-Fidelity Optimization
9.6.2 Transfer Stacking
9.6.3 Surrogate-Assisted Bi-Fidelity Evolutionary Optimization
9.6.4 Experimental Results
9.7 Surrogate-Assisted Multitasking Multi-Scenario Optimization
9.7.1 Multi-Scenario Minimax Optimization
9.7.2 Surrogate-Assisted Minimax Multifactorial Evolutionary Optimization
9.7.3 Experiment Results
9.8 Summary
References
10 Surrogate-Assisted High-Dimensional Evolutionary Optimization
10.1 Surrogate-Assisted Cooperative Optimization for High-Dimensional Optimization
10.1.1 RBF-Assisted SL-PSO
10.1.2 FES-Assisted PSO
10.1.3 Archive Update
10.1.4 Experimental Results and Discussions
10.2 A Multi-objective Infill Criterion for High-Dimensional Optimization
10.2.1 Main Framework
10.2.2 Multiobjective Infill Criterion
10.2.3 Experimental Results and Discussions
10.3 Multi-surrogate Multi-tasking Optimization of Expensive problems
10.3.1 Multi-factorial Evolutionary Algorithms
10.3.2 Main Framework
10.3.3 Global and Local Surrogates
10.3.4 Multi-tasking Optimization Based on Global and Local Surrogates
10.3.5 Experimental Results and Discussions
10.4 Surrogate-Assisted Large Optimization with Random Feature Selection
10.4.1 Main Framework
10.4.2 Sub-problem Formation and Optimization
10.4.3 Global Best Position Update
10.4.4 Experimental Results and Discussions
10.5 Summary
References
11 Offline Big or Small Data-Driven Optimization and Applications
11.1 Adaptive Clustering for Offline Big-Data Driven Optimization …
11.1.1 Problem Formulation
11.1.2 Adaptive Clustering for Offline Data-Driven Optimization
11.1.3 Empirical Results
11.1.4 Discussions
11.2 Small Data-Driven Multi-objective Magnesium Furnace Optimization
11.2.1 Model Management Based on a Global Surrogate
11.2.2 Empirical Verification on Benchmark Problems
11.2.3 Optimization of Fused Magnesium Furnaces
11.3 Selective Ensemble for Offline Airfoil Optimization
11.3.1 Problem Formulation
11.3.2 Selective Ensemble for Offline Data-Driven Optimization
11.3.3 Comparative Results
11.4 Knowledge Transfer in Offline Data-Driven Beneficiation Process …
11.4.1 Introduction
11.4.2 Knowledge Transfer by Multi-surrogate Optimization
11.4.3 Reference Vector Based Final Solution Selection
11.4.4 Optimization of Beneficiation Process
11.5 Transfer Learning for Offline Data-Driven Dynamic Optimization
11.5.1 Dynamic Data-Driven Optimization
11.5.2 Data Stream Ensemble for Incremental Learning
11.5.3 Ensemble Based Transfer Optimization
11.5.4 Support Vector Domain Description for Final Solution Selection
11.5.5 Empirical Results
11.6 Summary
References
12 Surrogate-Assisted Evolutionary Neural Architecture Search
12.1 Challenges in Neural Architecture Search
12.1.1 Architecture Representation
12.1.2 Search Strategies
12.1.3 Performance Evaluation
12.2 Bayesian Optimization for Neural Architecture Search
12.2.1 Architecture Encoding
12.2.2 Kernel Functions
12.2.3 Discussions
12.3 Random Forest Assisted Neural Architecture Search
12.3.1 Block-Based Architecture Representation
12.3.2 Offline Data Generation
12.3.3 Random Forest Construction
12.3.4 Search Methodology
12.3.5 Experimental Results
12.4 Summary
References
Index