Entropy Randomization in Machine Learning presents a new approach to machine learning―entropy randomization―to obtain optimal solutions under uncertainty (uncertain data and models of the objects under study). Randomized machine-learning procedures involve models with random parameters and maximum entropy estimates of the probability density functions of the model parameters under balance conditions with measured data. Optimality conditions are derived in the form of nonlinear equations with integral components. A new numerical random search method is developed for solving these equations in a probabilistic sense. Along with the theoretical foundations of randomized machine learning, Entropy Randomization in Machine Learning considers several applications to binary classification, modelling the dynamics of the Earth’s population, predicting seasonal electric load fluctuations of power supply systems, and forecasting the thermokarst lakes area in Western Siberia.
Features
• A systematic presentation of the randomized machine-learning problem: from data processing, through structuring randomized models and algorithmic procedure, to the solution of applications-relevant problems in different fields
• Provides new numerical methods for random global optimization and computation of multidimensional integrals
• A universal algorithm for randomized machine learning
This book will appeal to undergraduates and postgraduates specializing in artificial intelligence and machine learning, researchers and engineers involved in the development of applied machine learning systems, and researchers of forecasting problems in various fields.
Author(s): Yuri S. Popkov, Alexey Yu. Popkov, Yuri A. Dubnov
Series: Chapman & Hall/CRC Machine Learning & Pattern Recognition
Publisher: CRC Press/Chapman & Hall
Year: 2022
Language: English
Pages: 404
City: Boca Raton
Cover
Half Title
Series Page
Title Page
Copyright Page
Contents
Preface
CHAPTER 1: GENERAL CONCEPT OF MACHINE LEARNING
1.1. TRANSFORMATION OF KNOWLEDGE INTO DECISIONS
1.2. STRUCTURE OF MACHINE LEARNING PROCEDURE
1.3. MAIN CONCEPTS OF MACHINE LEARNING PROCEDURE
1.4. PRINCIPLES OF RANDOMIZED MACHINE LEARNING PROCEDURE
CHAPTER 2: DATA SOURCES AND MODELS
2.1. ANALOG SOURCE OF DATA
2.1.1. Deterministic functions
2.1.2. Random functions
2.2. DIGITAL SOURCE OF DATA
2.2.1. Amplitude and time quantization
2.2.2. Audio data
2.2.3. Graphical data
2.2.4. Text data
2.2.5. Government statistical data
2.3. RESTORATION METHODS FOR MISSING DATA
2.3.1. Interpolation
2.3.2. Auxiliary dynamic models
2.3.3. Spatial entropy decomposition
2.3.4. Randomized restoration method for missing data
CHAPTER 3: DIMENSION REDUCTION METHODS
3.1. REVIEW OF DIMENSION REDUCTION METHODS
3.1.1. Singular decomposition method for data matrix
3.1.2. Principal component analysis
3.1.3. Random projection method
3.1.4. Direct and inverse projection
3.2. ENTROPY OPTIMIZATION OF SEQUENTIAL PROCEDURE
3.2.1. Optimality conditions and algorithm
3.2.2. Approximation of information cross-entropy functional
3.3. ENTROPY OPTIMIZATION OF PARALLEL PROCEDURE
3.3.1. Definition and structure
3.3.2. Optimality conditions and algorithm
3.4. ENTROPY REDUCTION UNDER MATRIX NORM AND INFORMATION CAPACITY CONSTRAINTS
3.5. ESTIMATING EFFICIENCY OF DIMENSION REDUCTION FOR LINEAR MODEL LEARNING
3.5.1. Linear model
3.5.2. Comparing ?- and ?-problems of RML
3.6. ESTIMATING EFFICIENCY OF EDR FOR BINARY CLASSIFICATION PROBLEMS
3.6.1. Linear classifier
3.6.2. Scheme of computational experiment
3.6.3. Results of experiment
3.7. ENTROPY METHODS FOR RANDOM PROJECTION
3.7.1. Statements of Entropy Randomized Projection Problems
3.7.2. Algorithms for Entropy Randomized Projection
3.7.3. Implementation of random projectors and their numerical characteristics
3.7.4. Random projector matrices with given values of elements
3.7.5. Choice of appropriate projector matrix from Q (3.138)
CHAPTER 4: RANDOMIZED PARAMETRIC MODELS
4.1. DEFINITION, CHARACTERISTICS AND CLASSIFICATION
4.2. “SINGLE INPUT–ENSEMBLE OUTPUT” RANDOMIZED PARAMETRIC MODEL
4.2.1. Static models
4.2.2. Functional description of dynamic models
4.2.3. Linear dynamic models
4.2.4. Nonlinear dynamic models with power nonlinearities
4.2.5. Nonlinear dynamic models with polynomial nonlinearities
4.2.6. Randomized neural networks
4.3. “(SINGLE INPUT, FEEDBACK)–ENSEMBLE OUTPUT” DYNAMIC RANDOMIZED PARAMETRIC MODEL
4.3.1. Definition and structure
4.3.2. Linear dynamic models
4.3.3. Nonlinear dynamic models with power nonlinearities
4.3.4. Nonlinear dynamic models with polynomial nonlinearities
4.4. PROBABILISTIC CHARACTERISTICS OF RANDOMIZED PARAMETERS AND ENSEMBLES
CHAPTER 5: ENTROPY-ROBUST ESTIMATION PROCEDURES
5.1. STRUCTURE OF ENTROPY-ROBUST ESTIMATION
5.2. ENTROPY-ROBUST ESTIMATION ALGORITHMS FOR PROBABILITY DENSITY FUNCTIONS
5.2.1. Estimation algorithms for RPM-Orig model
5.2.2. Estimation algorithms for RPM-Rel model
5.2.3. Estimation algorithms for RPM-F model with measurement errors of input and output
5.3. OPTIMALITY CONDITIONS FOR LYAPUNOV-TYPE PROBLEMS
5.4. OPTIMALITY CONDITIONS AND STRUCTURE OF ENTROPY-OPTIMAL PROBABILITY DENSITY FUNCTIONS
5.4.1. Randomized models of the RPM-Orig class with output errors
5.4.2. Randomized models of the RPM-Rel class with output errors
5.4.3. Randomized models of the RPM-Orig class with input and output errors
5.5. EQUATIONS FOR LAGRANGE MULTIPLIERS
CHAPTER 6: ENTROPY-ROBUST ESTIMATION METHODS
6.1. ENTROPY-ROBUST ESTIMATION ALGORITHMS FOR PROBABILITIES OF BELONGING
6.1.1. ML algorithm for RPM-QuaRand model with normalized probabilities of belonging
6.1.2. ML algorithm for RPM-QuaRand model with interval-type probabilities of belonging
6.2. FUNCTIONAL DESCRIPTION OF DYNAMIC RPM-QUARAND MODELS
6.3. OPTIMALITY CONDITIONS AND STRUCTURE OF ENTROPY-OPTIMAL PROBABILITIES OF BELONGING
CHAPTER 7: COMPUTATIONAL METHODS OF RANDOMIZED MACHINE LEARNING
7.1. CLASSES OF BALANCE EQUATIONS IN RML AND ML PROCEDURES
7.2. MONTE CARLO PACKET ITERATIONS FOR GLOBAL OPTIMIZATION
7.2.1. Canonical form of global optimization problem
7.2.2. Idea of method and concept of solution
7.2.3. Probabilistic characteristics of random sequences ℱ and ?
7.2.4. Convergence of GFS algorithm
7.2.5. Study of decrements sequence ?
7.2.6. Admissible set ?(z) of general form
7.2.7. Logical structure of GFS algorithm
7.2.8. Experimental study of GFS algorithm
7.3. ON CALCULATION OF MULTIDIMENSIONAL INTEGRALS USING MONTE CARLO METHOD
7.4. MULTIPLICATIVE ALGORITHMS WITH ?-ACTIVE VARIABLES
CHAPTER 8: GENERATION METHODS
8.1. A SURVEY OF GENERATION METHODS FOR RANDOM OBJECTS
8.1.1. Random variables
8.1.2. Random vectors
8.2. DIRECT GENERATION METHOD FOR RANDOM VECTORS WITH GIVEN PROBABILITY DENSITY FUNCTION
8.2.1. Unit cube ?
8.2.2. Compact set ?
8.3. APPROXIMATION OF GIVEN PROBABILITY DENSITY FUNCTION
CHAPTER 9: INFORMATION TECHNOLOGIES OF RANDOMIZED MACHINE LEARNING
9.1. ARCHITECTURE OF MODERN COMPUTER SYSTEMS
9.2. UNIVERSAL MULTITHREADED ARCHITECTURE
9.3. INFORMATION TECHNOLOGIES OF RANDOMIZED MACHINE LEARNING
9.4. IMPLEMENTATION OF PACKET ITERATIONS
CHAPTER 10: ENTROPY CLASSIFICATION
10.1. STANDARD CLASSIFICATION METHODS
10.1.1. Decision Tree
10.1.2. ?-Nearest Neighbor
10.1.3. Naive Bayes
10.1.4. Linear classifiers
10.2. COMPOSITION ALGORITHMS
10.2.1. Bagging
10.2.2. Stacking
10.2.3. Boosting
10.3. ENTROPY CLASSIFICATION
10.3.1. Problem formulation
10.3.2. Learning stage
10.3.3. Testing stage
10.3.4. Numerical examples
CHAPTER 11: PROBLEMS OF DYNAMIC REGRESSION
11.1. RESTORATION OF DYNAMIC RELATIONSHIPS IN APPLICATIONS
11.2. RANDOMIZED MODEL OF WORLD POPULATION DYNAMICS
11.2.1. RML procedure for learning of World Population Dynamics Model
11.2.2. Numerical results
11.3. RANDOMIZED FORECASTING OF DAILY ELECTRICAL LOAD IN POWER SYSTEM
11.3.1. Electrical Load Model
11.3.2. Learning dataset
11.3.3. Entropy-optimal probability density functions of parameters and noises
11.3.4. Results of model learning
11.3.5. Model testing
11.3.6. Randomized prediction of ?-daily load
11.4. ENTROPY RANDOMIZED MODELLING AND FORECASTING OF THERMOKARST LAKE AREA EVOLUTION IN WESTERN SIBERIA
11.4.1. Thermokarst lakes and climate change
11.4.2. Thermokarst lakes of Western Siberia, tools and problems of their study
11.4.3. Structures of randomized models of thermokarst lakes state
11.4.4. Data on the state of thermokarst lakes
11.4.5. Entropy-Randomized machine learning of ????
11.4.5.1. Formation of data sets to train ????
11.4.5.2. ERML algorithm
11.4.6. Testing procedure, data and accuracy estimates
11.4.7. Randomized forecasting of thermokarst lakes area evolution
11.4.8. Results of training, testing and forecasting the temporal evolution of thermokarst lakes area in Western Siberia
11.4.8.1. Randomized training (1973–1997)
11.4.8.2. Testing (1998–2007)
11.4.8.3. Randomized forecasting (2008–2023)
APPENDIX A: MAXIMUM ENTROPY ESTIMATE (MEE) AND ITS ASYMPTOTIC EFFICIENCY
A.1. STATEMENT OF MAXIMUM ENTROPY ESTIMATION PROBLEM
A.2. EXISTENCE OF IMPLICIT FUNCTION ?(˜Y(?),X(?))
A.3. ASYMPTOTIC EFFICIENCY OF MAXIMUM ENTROPY ESTIMATES
APPENDIX B: APPROXIMATE ESTIMATION OF LDR
B.1. ORDER ?
B.2. PARAMETERS RANGES
Bibliography
Index
