Convex Optimization for Machine Learning

This book covers an introduction to convex optimization, one of the powerful and tractable optimization problems that can be efficiently solved on a computer. The goal of the book is to help develop a sense of what convex optimization is, and how it can be used in a widening array of practical contexts with a particular emphasis on Machine Learning. The first part of the book covers core concepts of convex sets, convex functions, and related basic definitions that serve understanding convex optimization and its corresponding models. The second part deals with one very useful theory, called duality, which enables us to: (1) gain algorithmic insights; and (2) obtain an approximate solution to non-convex optimization problems which are often difficult to solve. The last part focuses on modern applications in Machine Learning and Deep Learning. A defining feature of this book is that it succinctly relates the "story" of how convex optimization plays a role, via historical examples and trending Machine Learning applications. Another key feature is that it includes programming implementation of a variety of Machine Learning algorithms inspired by optimization fundamentals, together with a brief tutorial of the used programming tools. The implementation is based on Python, CVXPY, and TensorFlow. Since one of the key tools in convex optimization is an algorithm that requires computation on a computer, it is crucial to know how to implement algorithms using software tools. We employ Python as a major programming platform. To solve traditional convex optimization problems such as linear program, least squares, and semi-definite program, we utilize an easy-to-use and high-level language, CVXPY, running in Python. To implement Machine Learning and Deep Learning algorithms, we employ TensorFlow, one of the most popular Deep Learning frameworks. TensorFlow provides numerous powerful built-in functions that ease performing many important procedures in deep learning. One of the key benefits of TensorFlow is that it is fully integrated with Keras, the most high-level library with a focus on enabling fast user experimentation. Keras allows us to go from idea to implementation with very few steps. This book does not follow a traditional textbook-style organization, but is streamlined via a series of lecture notes that are intimately related, centered around coherent themes and concepts. It serves as a textbook mainly for a senior-level undergraduate course, yet is also suitable for a first-year graduate course. Readers benefit from having a good background in linear algebra, some exposure to probability, and basic familiarity with Python.