This book systematically narrates the fundamentals, methods, and recent advances of evolutionary deep neural architecture search chapter by chapter. This will provide the target readers with sufficient details learning from scratch. In particular, the method parts are devoted to the architecture search of unsupervised and supervised deep neural networks. The people, who would like to use deep neural networks but have no/limited expertise in manually designing the optimal deep architectures, will be the main audience. This may include the researchers who focus on developing novel evolutionary deep architecture search methods for general tasks, the students who would like to study the knowledge related to evolutionary deep neural architecture search and perform related research in the future, and the practitioners from the fields of computer vision, natural language processing, and others where the deep neural networks have been successfully and largely used in their respective fields.
Author(s): Yanan Sun, Gary G. Yen, Mengjie Zhang
Series: Studies in Computational Intelligence, 1070
Publisher: Springer
Year: 2022
Language: English
Pages: 334
City: Cham
Preface
References
Contents
Acronyms
Part I Fundamentals and Backgrounds
1 Evolutionary Computation
1.1 Genetic Algorithms (GAs)
1.2 Particle Swarm Optimization (PSO)
1.3 Differential Evolution (DE)
1.4 Genetic Programming (GP)
1.5 Chapter Summary
References
2 Deep Neural Networks
2.1 Deep Belief Networks
2.2 Stacked Auto-Encoders
2.2.1 Sparse Auto-Encoders
2.2.2 Weight Decay Auto-Encoders
2.2.3 Denoising Auto-Encoders (DAEs)
2.2.4 Contractive Auto-Encoders
2.2.5 Convolutional Auto-Encoders (CAEs)
2.2.6 Variational Auto-Encoders (VAEs)
2.3 Convolutional Neural Networks (CNNs)
2.3.1 CNN Skeleton
2.3.2 Convolution
2.3.3 Pooling
2.3.4 Reflect Padding
2.3.5 Batch Normalization (BN)
2.3.6 ResNet Blocks (RBs) and DenseNet Blocks (DBs)
2.4 Benchmarks for Deep Neural Networks
2.5 Chapter Summary
References
Part II Evolutionary Deep Neural Architecture Search for Unsupervised DNNs
3 Architecture Design for Stacked AEs and DBNs
3.1 Introduction
3.2 Related Work and Motivations
3.2.1 Unsupervised Deep Learning
3.2.2 Evolutionary Algorithms for Evolving Neural Networks
3.3 Algorithm Details
3.3.1 Framework of EUDNN
3.3.2 Evolving Connection Weights and Activation Functions
3.3.3 Fine-Tuning Connection Weights
3.3.4 Discussion
3.4 Experimental Design
3.4.1 Performance Metric
3.4.2 Peer Competitors
3.4.3 Parameter Settings
3.5 Experimental Results and Analysis
3.5.1 Performance of EUDNN
3.5.2 Analysis on Pre-training of EUDNN
3.5.3 Analysis on Fine-Tuning of EUDNN
3.5.4 Representation Visualizations
3.6 Chapter Summary
References
4 Architecture Design for Convolutional Auto-Encoders
4.1 Introduction
4.2 Motivation of FCAE
4.3 Algorithm Details
4.3.1 Algorithm Overview
4.3.2 Encoding Strategy
4.3.3 Particle Initialization
4.3.4 Fitness Evaluation
4.3.5 Velocity Calculation and Position Update
4.3.6 Deep Training on Global Best Particle
4.4 Experimental Design
4.4.1 Peer Competitors
4.4.2 Parameter Settings
4.5 Experimental Results and Analysis
4.5.1 Overview Performance
4.5.2 Evolution Trajectory of PSOAO
4.5.3 Performance on Different Numbers of Training Examples
4.5.4 Investigation on x-Reference Velocity Calculation
4.6 Chapter Summary
References
5 Architecture Design for Variational Auto-Encoders
5.1 Introduction
5.2 Algorithm Details
5.2.1 Algorithm Overview
5.2.2 Strategy of Gene Encoding
5.2.3 Initialization of Population
5.2.4 Evaluation
5.2.5 Crossover Operator and Mutation Operator
5.2.6 Environmental Selection
5.3 Experimental Design
5.3.1 Parameter Setting
5.3.2 Peer Competitors
5.3.3 Performance Evaluation
5.4 Experimental Results and Analysis
5.4.1 Overall Performance
5.4.2 Evolution Trajectory of EvoVAE
5.4.3 Running Time
5.4.4 The Obtained Architecture
5.4.5 Ablation Experiments
5.5 Chapter Summary
References
Part III Evolutionary Deep Neural Architecture Search for Supervised DNNs
6 Architecture Design for Plain CNNs
6.1 Introduction
6.2 Algorithm Details
6.2.1 Algorithm Overview
6.2.2 Strategy of Gene Encoding
6.2.3 Initialization of Population
6.2.4 Evaluation of Fitness
6.2.5 Slack Binary Tournament Selection
6.2.6 Offspring Generation
6.2.7 Environmental Selection
6.2.8 Select and Decode Best Individual
6.3 Experimental Design
6.3.1 Peer Competitors
6.3.2 Parameter Settings
6.4 Experimental Results and Discussion
6.4.1 Overall Results
6.4.2 Performance of Weight Initialization
6.4.3 Discussion
6.5 Chapter Summary
References
7 Architecture Design for RBs and DBs Based CNNs
7.1 Introduction
7.2 Algorithm Details
7.2.1 Algorithm Overview
7.2.2 Population Initialization
7.2.3 Fitness Evaluation
7.2.4 Offspring Generation
7.2.5 Environmental Selection
7.3 Experimental Design
7.3.1 Peer Competitors
7.3.2 Parameter Settings
7.4 Experimental Results and Analysis
7.4.1 Performance Overview
7.4.2 Evolution Trajectory
7.4.3 Designed CNN Architectures
7.5 Chapter Summary
References
8 Architecture Design for Skip-Connection Based CNNs
8.1 Introduction
8.2 Algorithm Details
8.2.1 Algorithm Overview
8.2.2 Population Initialization
8.2.3 Fitness Evaluation
8.2.4 Offspring Generating
8.2.5 Environmental Selection
8.3 Experimental Design
8.3.1 Peer Competitors
8.3.2 Parameter Settings
8.4 Experimental Results and Analysis
8.4.1 Overall Results
8.4.2 Transferable Performance on ImageNet
8.4.3 Performance of Crossover Operator
8.4.4 Performance of Acceleration Components
8.4.5 Evolution Trajectory
8.5 Chapter Summary
References
9 Hybrid GA and PSO for Architecture Design
9.1 Introduction
9.2 Algorithm Details
9.2.1 Overall Structure of the System
9.2.2 The Evolved CNN Architecture-DynamicNet
9.2.3 HGAPSO Encoding Strategy
9.2.4 HGAPSO Search
9.2.5 HGAPSO Fitness Evaluations
9.3 Experimental Studies
9.3.1 Parameter Settings
9.3.2 State-of-the-Art Methods Versus HGAPSO
9.3.3 Evolved CNN Architecture
9.3.4 One-Run Result on CIFAR-10 Dataset
9.4 Chapter Summary
References
10 Internet Protocol Based Architecture Design
10.1 Introduction
10.2 Algorithm Details
10.2.1 Algorithm Overview
10.2.2 Encoding Strategy of Particle
10.2.3 Initialization of Population
10.2.4 Evaluation of Fitness
10.2.5 Update Particle with Velocity Clamping
10.2.6 Selection and Decoding of Best Individual
10.3 Experimental Design
10.3.1 Peer Competitors
10.3.2 Parameter Settings
10.4 Experimental Results and Analysis
10.4.1 Overall Performance
10.4.2 Evolved CNN Architectures
10.4.3 Trajectory Visualization
10.5 Chapter Summary
References
11 Differential Evolution for Architecture Design
11.1 Introduction
11.2 Algorithm Details
11.2.1 DECNN Algorithm Overview
11.2.2 Adjusted IP-Based Encoding Strategy
11.2.3 Population Initialization
11.2.4 Fitness Evaluation
11.2.5 DECNN DE Mutation and Crossover
11.2.6 DECNN Second Crossover
11.3 Experimental Design
11.3.1 State-of-the-Art Competitors
11.3.2 Parameter Settings
11.4 Experimental Results and Analysis
11.4.1 DECNN Versus State-of-the-Art Methods
11.4.2 DECNN Versus IPPSO
11.4.3 Evolved CNN Architecture
11.5 Chapter Summary
References
12 Architecture Design for Analyzing Hyperspectral Images
12.1 Introduction
12.2 Algorithm Details
12.2.1 Algorithm Overview
12.2.2 Gene Encoding Strategy
12.2.3 Offspring Generation
12.2.4 Environmental Selection
12.3 Experimental Design
12.3.1 Benchmark Dataset
12.3.2 Peer Competitors
12.3.3 Parameter Settings
12.3.4 Training Details
12.4 Experimental Results and Analysis
12.4.1 Overall Results
12.4.2 Comparisons with Artificial-CNN
12.5 Chapter Summary
References
Part IV Recent Advances in Evolutionary Deep Neural Architecture Search
13 Encoding Space Based on Directed Acyclic Graphs
13.1 Introduction
13.2 Algorithm Details
13.2.1 Encoding Strategy Overview
13.2.2 Representation and Decoding Details
13.2.3 Initialization Algorithm Overview
13.2.4 Initialization Algorithm Details
13.3 Experimental Studies
13.3.1 Overview
13.3.2 Parameter Settings
13.3.3 Experimental Results
13.4 Chapter Summary
References
14 End-to-End Performance Predictors
14.1 Introduction
14.2 Related Work
14.3 Algorithm Details
14.3.1 Encoding
14.3.2 Training of the Random Forest
14.3.3 Performance Prediction
14.3.4 Strength and Weakness of E2EPP
14.4 Experimental Design
14.4.1 Peer Competitors
14.4.2 Parameter Settings
14.5 Experimental Results
14.5.1 Overall Results
14.5.2 Efficiency of E2EPP
14.5.3 Effectiveness of E2EPP
14.5.4 Comparison to Radial Basis Network
14.6 Chapter Summary
References
15 Deep Neural Architecture Pruning
15.1 Introduction
15.2 Background
15.3 Algorithm Details
15.3.1 Genetic Representation of an Individual
15.3.2 Population Initialization
15.3.3 Individual Evaluation
15.3.4 Selection of Knee and Boundary
15.3.5 Offspring Generation
15.3.6 Fine Tuning
15.4 Experimental Design
15.4.1 Chosen CNN Architectures for Pruning
15.4.2 Algorithm Parameters
15.5 Experimental Results and Discussion
15.5.1 Experimental Results
15.5.2 Result Discussion
15.6 Chapter Summary
References
16 Deep Neural Architecture Compression
16.1 Introduction
16.2 Related Work and Motivation
16.2.1 Convolutional Neural Network Compression
16.2.2 Evolutionary Algorithms and MMD
16.3 KGEA for Compressing DNNs
16.3.1 Convolutional Filter Pruning
16.3.2 Multi-objevtive Modeling for CNN Compression
16.3.3 KGEA
16.3.4 Encoding Scheme and Genetic Operators
16.3.5 Discussion
16.4 Experimental Studies
16.4.1 Experimental Settings
16.4.2 Experiments on Fully Convolutional LeNet
16.4.3 Experiments on VGG-19
16.5 Chapter Summary
References
17 Distribution Training Framework for Architecture Design
17.1 Introduction
17.2 Distributed Deep Learning
17.3 The Distributed Framework
17.3.1 Motivation
17.3.2 Framework Overview
17.3.3 Definition of the Data Packet
17.3.4 Server Node
17.3.5 Computing Node
17.4 Experimental Studies
17.4.1 Evolutionary Pelee
17.4.2 Speedup Analysis
17.4.3 Inconsistent Performance Node Analysis
17.4.4 Communication Analysis
17.4.5 Efficiency Analysis
17.5 Chapter Summary
References
Appendix Book Conclusions
