Engineering Design Optimization

Contents
Preface
Acknowledgements
1 Introduction
1.1 Design Optimization Process
1.2 Optimization Problem Formulation
1.2.1 Design Variables
1.2.2 Objective Function
1.2.3 Constraints
1.2.4 Optimization Problem Statement
1.3 Optimization Problem Classification
1.3.1 Smoothness
1.3.2 Linearity
1.3.3 Multimodality and Convexity
1.3.4 Deterministic versus Stochastic
1.4 Optimization Algorithms
1.4.1 Order of Information
1.4.2 Local versus Global Search
1.4.3 Mathematical versus Heuristic
1.4.4 Function Evaluation
1.4.5 Stochasticity
1.4.6 Time Dependence
1.5 Selecting an Optimization Approach
1.6 Notation
1.7 Summary
Problems
2 A Short History of Optimization
2.1 The First Problems: Optimizing Length and Area
2.2 Optimization Revolution: Derivatives and Calculus
2.3 The Birth of Optimization Algorithms
2.4 The Last Decades
2.5 Toward a Diverse Future
2.6 Summary
3 Numerical Models and Solvers
3.1 Model Development for Analysis versus Optimization
3.2 Modeling Process and Types of Errors
3.3 Numerical Models as Residual Equations
3.4 Discretization of Differential Equations
3.5 Numerical Errors
3.5.1 Roundoff Errors
3.5.2 Truncation Errors
3.5.3 Iterative Solver Tolerance Error
3.5.4 Programming Errors
3.6 Overview of Solvers
3.7 Rate of Convergence
3.8 Newton-Based Solvers
3.9 Models and the Optimization Problem
3.10 Summary
Problems
4 Unconstrained Gradient-Based Optimization
4.1 Fundamentals
4.1.1 Derivatives and Gradients
4.1.2 Curvature and Hessians
4.1.3 Taylor Series
4.1.4 Optimality Conditions
4.2 Two Overall Approaches to Finding an Optimum
4.3 Line Search
4.3.1 Sufficient Decrease and Backtracking
4.3.2 Strong Wolfe Conditions
4.3.3 Interpolation for Pinpointing
4.4 Search Direction
4.4.1 Steepest Descent
4.4.2 Conjugate Gradient
4.4.3 Newton's Method
4.4.4 Quasi-Newton Methods
4.4.5 Limited-Memory Quasi-Newton Methods
4.5 Trust-Region Methods
4.5.1 Quadratic Model with Spherical Trust Region
4.5.2 Trust-Region Sizing Strategy
4.5.3 Comparison with Line Search Methods
4.6 Summary
Problems
5 Constrained Gradient-Based Optimization
5.1 Constrained Problem Formulation
5.2 Understanding n-Dimensional Space
5.3 Optimality Conditions
5.3.1 Equality Constraints
5.3.2 Inequality Constraints
5.3.3 Meaning of the Lagrange Multipliers
5.3.4 Post-Optimality Sensitivities
5.4 Penalty Methods
5.4.1 Exterior Penalty Methods
5.4.2 Interior Penalty Methods
5.5 Sequential Quadratic Programming
5.5.1 Equality Constrained SQP
5.5.2 Inequality Constraints
5.5.3 Merit Functions and Filters
5.5.4 Quasi-Newton SQP
5.5.5 Algorithm Overview
5.6 Interior-Point Methods
5.6.1 Modifications to the Basic Algorithm
5.6.2 SQP Comparisons and Examples
5.7 Constraint Aggregation
5.8 Summary
Problems
6 Computing Derivatives
6.1 Derivatives, Gradients, and Jacobians
6.2 Overview of Methods for Computing Derivatives
6.3 Symbolic Differentiation
6.4 Finite Differences
6.4.1 Finite-Difference Formulas
6.4.2 The Step-Size Dilemma
6.4.3 Practical Implementation
6.5 Complex Step
6.5.1 Theory
6.5.2 Complex-Step Implementation
6.6 Algorithmic Differentiation
6.6.1 Variables and Functions as Lines of Code
6.6.2 Forward-Mode AD
6.6.3 Reverse-Mode AD
6.6.4 Forward Mode or Reverse Mode?
6.6.5 AD Implementation
6.6.6 AD Shortcuts for Matrix Operations
6.7 Implicit Analytic Methods—Direct and Adjoint
6.7.1 Residuals and Functions
6.7.2 Direct and Adjoint Derivative Equations
6.7.3 Direct or Adjoint?
6.7.4 Adjoint Method with AD Partial Derivatives
6.8 Sparse Jacobians and Graph Coloring
6.9 Unified Derivatives Equation
6.9.1 UDE Derivation
6.9.2 UDE for Mixed Implicit and Explicit Components
6.9.3 Recovering AD
6.10 Summary
Problems
7 Gradient-Free Optimization
7.1 When to Use Gradient-Free Algorithms
7.2 Classification of Gradient-Free Algorithms
7.3 Nelder–Mead Algorithm
7.4 Generalized Pattern Search
7.5 DIRECT Algorithm
7.6 Genetic Algorithms
7.6.1 Binary-Encoded Genetic Algorithms
7.6.2 Real-Encoded Genetic Algorithms
7.6.3 Constraint Handling
7.6.4 Convergence
7.7 Particle Swarm Optimization
7.8 Summary
Problems
8 Discrete Optimization
8.1 Binary, Integer, and Discrete Variables
8.2 Avoiding Discrete Variables
8.3 Branch and Bound
8.3.1 Binary Variables
8.3.2 Integer Variables
8.4 Greedy Algorithms
8.5 Dynamic Programming
8.6 Simulated Annealing
8.7 Binary Genetic Algorithms
8.8 Summary
Problems
9 Multiobjective Optimization
9.1 Multiple Objectives
9.2 Pareto Optimality
9.3 Solution Methods
9.3.1 Weighted Sum
9.3.2 Epsilon-Constraint Method
9.3.3 Normal Boundary Intersection
9.3.4 Evolutionary Algorithms
9.4 Summary
Problems
10 Surrogate-Based Optimization
10.1 When to Use a Surrogate Model
10.2 Sampling
10.2.1 Latin Hypercube Sampling
10.2.2 Low-Discrepancy Sequences
10.3 Constructing a Surrogate
10.3.1 Linear Least Squares Regression
10.3.2 Maximum Likelihood Interpretation
10.3.3 Nonlinear Least Squares Regression
10.3.4 Cross Validation
10.3.5 Common Basis Functions
10.4 Kriging
10.5 Deep Neural Networks
10.6 Optimization and Infill
10.6.1 Exploitation
10.6.2 Efficient Global Optimization
10.7 Summary
Problems
11 Convex Optimization
11.1 Introduction
11.2 Linear Programming
11.3 Quadratic Programming
11.4 Second-Order Cone Programming
11.5 Disciplined Convex Optimization
11.6 Geometric Programming
11.7 Summary
Problems
12 Optimization Under Uncertainty
12.1 Robust Design
12.2 Reliable Design
12.3 Forward Propagation
12.3.1 First-Order Perturbation Method
12.3.2 Direct Quadrature
12.3.3 Monte Carlo Simulation
12.3.4 Polynomial Chaos
12.4 Summary
Problems
13 Multidisciplinary Design Optimization
13.1 The Need for MDO
13.2 Coupled Models
13.2.1 Components
13.2.2 Models and Coupling Variables
13.2.3 Residual and Functional Forms
13.2.4 Coupled System Structure
13.2.5 Solving Coupled Numerical Models
13.2.6 Hierarchical Solvers for Coupled Systems
13.3 Coupled Derivatives Computation
13.3.1 Finite Differences
13.3.2 Complex Step and AD
13.3.3 Implicit Analytic Methods
13.4 Monolithic MDO Architectures
13.4.1 Multidisciplinary Feasible
13.4.2 Individual Discipline Feasible
13.4.3 Simultaneous Analysis and Design
13.5 Distributed MDO Architectures
13.5.1 Collaborative Optimization
13.5.2 Analytical Target Cascading
13.5.3 Bilevel Integrated System Synthesis
13.5.4 Asymmetric Subspace Optimization
13.5.5 Other Distributed Architectures
13.6 Summary
Problems
A Mathematics Background
A.1 Taylor Series Expansion
A.2 Chain Rule, Total Derivatives, and Differentials
A.3 Matrix Multiplication
A.3.1 Vector-Vector Products
A.3.2 Matrix-Vector Products
A.3.3 Quadratic Form (Vector-Matrix-Vector Product)
A.4 Four Fundamental Subspaces in Linear Algebra
A.5 Vector and Matrix Norms
A.6 Matrix Types
A.7 Matrix Derivatives
A.8 Eigenvalues and Eigenvectors
A.9 Random Variables
B Linear Solvers
B.1 Systems of Linear Equations
B.2 Conditioning
B.3 Direct Methods
B.4 Iterative Methods
B.4.1 Jacobi, Gauss–Seidel, and SOR
B.4.2 Conjugate Gradient Method
B.4.3 Krylov Subspace Methods
C Quasi-Newton Methods
C.1 Broyden's Method
C.2 Additional Quasi-Newton Approximations
C.2.1 Davidon–Fletcher–Powell Update
C.2.2 BFGS
C.2.3 Symmetric Rank 1 Update
C.2.4 Unification of SR1, DFP, and BFGS
C.3 Sherman–Morrison–Woodbury Formula
D Test Problems
D.1 Unconstrained Problems
D.1.1 Slanted Quadratic Function
D.1.2 Rosenbrock Function
D.1.3 Bean Function
D.1.4 Jones Function
D.1.5 Hartmann Function
D.1.6 Aircraft Wing Design
D.1.7 Brachistochrone Problem
D.1.8 Spring System
D.2 Constrained Problems
D.2.1 Barnes Problem
D.2.2 Ten-Bar Truss
Bibliography
Index