Introduction to Neural Networks for C#

Introduction
Chapter 1: Overview of Neural Networks
Chapter 2: Matrix Operations
Chapter 3: Using a Hopfield Neural Network
Chapter 4: How a Machine Learns
Chapter 5: Feedforward Backpropagation Networks
Chapter 6: Understanding Genetic Algorithms
Chapter 7: Understanding Simulated Annealing
Chapter 8: Pruning Neural Networks
Chapter 9: Predictive Neural Networks
Chapter 10: Application to the Financial Markets
Chapter 11: Understanding the Self-Organizing Map
Chapter 12: OCR with the Self-Organizing Map
Chapter 13: Bot Programming and Neural Networks
Chapter 14: The Future of Neural Networks
Appendix A: Downloading Examples
Appendix B: Mathematical Background
Appendix C: Common Threshold Functions
Appendix D: Executing Examples
Glossary
Introduction
A Historical Perspective on Neural Networks
Chapter 1: Overview of Neural Networks
Solving Problems with Neural Networks
Problems Commonly Solved With Neural Networks
Using a Simple Neural Network
Chapter Summary
Vocabulary
Questions for Review
Chapter 2: Matrix Operations
The Weight Matrix
Matrix Classes
Constructing a Matrix
Matrix Operations
Bipolar Operations
Chapter Summary
Vocabulary
Questions for Review
Chapter 3: Using a Hopfield Neural Network
The Hopfield Neural Network
Recalling Patterns
Creating a C# Hopfield Neural Network
Simple Hopfield Example
Visualizing the Weight Matrix
Hopfield Pattern Recognition Application
Chapter Summary
Vocabulary
Questions for Review
Chapter 4: How a Machine Learns
Learning Methods
Error Calculation
Training Algorithms
Chapter Summary
Vocabulary
Questions for Review
Chapter 5: Feedforward Backpropagation Networks
A Feedforward Neural Network
Solving the XOR Problem
Activation Functions
The Number of Hidden Layers
Examining the Feedforward Process
Examining the Backpropagation Process
Chapter Summary
Vocabulary
Questions for Review
Chapter 6: Understanding Genetic Algorithms
Genetic Algorithms
Understanding Genetic Algorithms
How Genetic Algorithms Work
Implementation of a Generic Genetic Algorithm
The Traveling Salesman Problem
Implementing the Traveling Salesman Problem
XOR Operator
Tic-Tac-Toe
Chapter Summary
Vocabulary
Questions for Review
Chapter 7: Understanding Simulated Annealing
Simulated Annealing Background
Understanding Simulated Annealing
Simulated Annealing and the Traveling Salesman Problem
Implementing Simulated Annealing
Simulated Annealing for the Traveling Salesman Problem
Simulated Annealing for Neural Networks
Chapter Summary
Vocabulary
Questions for Review
Chapter 8: Pruning Neural Networks
Understanding Pruning
Pruning Algorithms
Implementing Pruning
Chapter Summary
Vocabulary
Questions for Review
Chapter 9: Predictive Neural Networks
How to Predict with a Neural Network
Predicting the Sine Wave
Chapter Summary
Vocabulary
Questions for Review
Chapter 10: Application to the Financial Markets
Collecting Data for the S&P 500 Neural Network
Running the S&P 500 Prediction Program
Creating the Actual S&P 500 Data
Training the S&P 500 Network
Attempting to Predict the S&P 500
Chapter Summary
Vocabulary
Questions for Review
Chapter 11: Understanding the Self-Organizing Map
Introducing the Self-Organizing Map
Implementing the Self-Organizing Map
The SOM Implementation Class
The SOM Training Class
Using the Self-organizing Map
Chapter Summary
Vocabulary
Questions for Review
Chapter 12: OCR with the Self-Organizing Map
The OCR Application
Implementing the OCR Program
Downsampling the Image
Using the Self-Organizing Map
Beyond This Example
Chapter Summary
Vocabulary
Questions for Review
Chapter 13: Bot Programming and Neural Networks
A Simple Bot
Introducing the Neural Bot
Gathering Training Data for the Neural Bot
Training the Neural Bot
Querying the Neural Bot
Chapter Summary
Vocabulary
Questions for Review
Chapter 14: The Future of Neural Networks
Neural Networks Today
A Fixed Wing Neural Network
Quantum Computing
Reusable Neural Network Frameworks
Chapter Summary
Vocabulary
Appendix A: Downloading Examples
Appendix B: Mathematical Background
Matrix Operations
Sigma Notation
Derivatives and Integrals
Appendix C: Common Threshold Functions
Linear Threshold Function
Sigmoidal Threshold Function
Hyperbolic Tangent Threshold Function
Appendix D: Executing Examples
Using Visual Studio
Glossary
Figure 1.1: A neuron cell.
Figure 1.2: A digital signal.
Figure 1.3: Sound recorder showing an analog file.
Figure 1.4: Activation levels of a neuron.
Figure 1.5: Different Traffic Lights
Figure 1.6: Artificial neuron.
Figure 1.7: A neural network that recognizes the AND logical operation.
Figure 1.8: A neural network that recognizes the OR logical operation.
Figure 1.9: A neural network that recognizes the XOR logical operation.
Figure 2.1: A two neuron layer connected to a three neuron layer.
Figure 3.1: A Hopfield neural network with 12 connections.
Figure 3.2: A Hopfield Application
Figure 3.3: A pattern recognition Hopfield application.
Figure 4.1: Unsupervised training.
Figure 4.2: Supervised training.
Figure 5.1: A typical feedforward neural network (single hidden layer).
Figure 5.2: The Sigmoid function.
Figure 5.3: The hyperbolic tangent function.
Figure 5.4: The linear activation function.
Figure 6.1: Mating two chromosomes.
Figure 6.2: The traveling salesman program.
Figure 6.3: The game of tic-tac-toe.
Figure 7.1: Overview of the simulated annealing process.
Figure 8.1: Flowchart of the incremental pruning algorithm.
Figure 8.2: Flowchart of the selective pruning algorithm.
Figure 8.3: The incremental pruning example.
Figure 8.4: The selective pruning example.
Figure 9.1: The sine wave.
Figure 10.1: The S&P 500 stock index.
Figure 10.2: US prime interest rate.
Figure 10.3: Global and Local Minima
Figure 11.1: A self-organizing map.
Figure 11.2: Training a self-organizing map.
Figure 12.1: The OCR application.
Figure 13.1: Local time in St. Louis, MO.
Figure 14.1: An ornithopter.
Figure C.1: Graph of the Linear Threshold Function
Figure C.2: Graph of the Sigmoidal Threshold Function
Figure C.3: Graph of the hyperbolic tangent threshold function.
Figure D.1: Visual Studio Solution File
Figure D.2: Neural Network Examples in Visual Studio
Figure D.3: Selecting the Startup Project
Figure D.4: Locate the Data Files
Listing 3.1: The Hopfield Neural Network (HopfieldNetwork.cs)
Listing 3.2: Simple Console Example (ConsoleHopfield.cs)
Listing 3.3: Visual Hopfield Application (HopfieldApp.cs)
Listing 3.4: Hopfield Pattern Recognition (HopfieldPattern.cs)
Listing 4.1: The ErrorCalculation Class (ErrorCalculation.cs)
Listing 4.2: Using Hebb's Rule (Hebb.cs)
Listing 4.3: Using the Delta Rule (Delta.cs)
Listing 5.1: The XOR Problem (XOR.cs)
Listing 5.2: The Sigmoid Activation Function Class (ActivationSigmoid.cs)
Listing 5.3: The Hyperbolic Tangent Function Class (ActivationTANH.cs)
Listing 5.4: The Linear Activation Function (ActivationLinear.cs)
Listing 5.5: The Train Interface (Train.cs)
Listing 6.1: The MateWorker Class (MateWorker.cs)
Listing 6.2: XOR with a Genetic Algorithm (GeneticXOR.cs)
Listing 7.1: Simulated Annealing and the XOR Operator (AnnealXOR.cs)
Listing 9.1: Training the Sine Wave Predictor
Listing 9.2: Predicting the Sine Wave
Listing 9.3: Actual Sine Wave Data (ActualData.cs)
Listing 10.1: S&P 500 Historical Data (sp500.csv)
Listing 10.2: Prime Interest Rate Historical Data
Listing 10.3: Training the SP500 Neural Network
Listing 10.4: Predicting the SP500 Neural Network
Listing 10.5: Storing Actual S&P 500 Data (SP500Actual.cs)
Listing 10.6: Financial Samples (FinancialSample.cs)
Listing 10.7: Try to Predict the S&P 500 (PredictSP500.cs)
Listing 10.8: Hybrid Training Output
Listing 11.1: The SOM Training Example (TestSOMForm.cs)
Listing 12.1: Downsampling Images (DownSample.cs)
Listing 13.1: A Simple Bot (SimpleBot.cs)
Listing 13.2: HTML Data Encountered by the Bot
Listing 13.3: Configuring the Neural Bot (Config.cs)
Listing 13.4: Famous People
Listing 13.5: Gathering Training Data (GatherForTrain.cs)
Listing 13.6: Collection Worker (CollectionWorker.cs)
Listing 13.7: Parsing Websites (Text.cs)
Listing 13.8: Training the Bot (TrainBot.cs)
Listing 13.9: Managing Histograms (WordHistogram.cs)
Listing 13.10: Histogram Elements (HistogramElement.cs)
Listing 13.11: Traning Set Management (TrainingSet.cs)
Listing 13.12: WhenBornBot Class (WhenBornBot.cs)
Table 1.1: The AND Logical Operation
Table 1.2: The OR Logical Operation
Table 1.3: The XOR Logical Operation
Table 2.1: Matrix Classes
Table 2.2: The BiPolarUtil Class
Table 2.3: Methods of the Matrix Class
Table 2.4: Methods of the Matrix Class
Table 2.5: The MatrixMath Class
Table 3.1: Connections in a Hopfield Neural Network
Table 3.2: Weights Used to Recall 0101 and 1010
Table 3.3: Summary of HopfieldNetwork Properties
Table 3.4: Summary of HopfieldNetwork Methods
Table 4.1: Using Hebb’s Rule
Table 5.1: Determining the Number of Hidden Layers
Table 6.1: Common Uses for Genetic Algorithms
Table 6.2: Number of Steps to Solve TSP with a Conventional Program
Table 6.3: Classes Used for the GA Version of the Traveling Salesman
Table 7.1: Simulated Annealing Inputs
Table 8.1: Variables Used for the Prune Process
Table 9.1: Sample Training Sets for a Predictive Neural Network
Table 9.2: Sine Wave Training Data
Table 11.1: Sample Inputs to a Self-Organizing Map
Table 11.2: Connection Weights in the Sample Self-Organizing Map
Table 11.3: Classes Used to Implement the Self-organizing Map
Table 11.4: TestSOM Constants
Table 12.1: Classes for the OCR Application
Table 13.1: When Born Neural Bot Modes
Table 13.2: Configuring the Neural Bot
Equation 2.1: A Weight Matrix
Equation 2.2: A Threshold and Weight Matrix
Equation 2.3: An Empty Matrix
Equation 2.4: An Initialized Matrix
Equation 2.5: A Row Matrix/Vector
Equation 2.6: A Column Matrix/Vector
Equation 2.7: Matrix Addition
Equation 2.8: Matrix Division by a Scalar
Equation 2.9: Dot Product
Equation 2.10: Matrix Multiplication
Equation 2.11: Non-Commutative Matrix Multiplication
Equation 2.12: Identity Matrix
Equation 2.13: Multiply by an Identity Matrix
Equation 2.14: Matrix Multiplication by a Scalar
Equation 2.15: Matrix Subtraction
Equation 2.16: Matrix Transpose
Equation 2.17: Calculate Vector Length
Equation 2.18: Boolean to Bipolar
Equation 2.19: Bipolar to Boolean
Equation 2.20: A Bipolar Matrix
Equation 3.1: A Blank Matrix
Equation 3.2: Binary to Bipolar
Equation 3.3: Bipolar to Binary
Equation 3.4: Input Matrix
Equation 3.5: Inverse Matrix
Equation 3.6: Resulting Matrix
Equation 3.7: Contribution Matrix
Equation 4.1: Root Mean Square Error (RMS)
Equation 4.2: RMS for a Neural Network
Equation 4.3: Hebb’s Rule
Equation 4.4: The Delta Rule
Equation 5.1: The Sigmoid Function.
Equation 5.2: The TANH function
Equation 5.3: A Linear Function
Equation 5.4: Feedforward Calculations
Equation 5.5: An Input Matrix
Equation 7.1: Scaling the Temperature
Equation 11.1: Multiplicative Normalization
Equation 11.2: Z-Axis Normalization
Equation 11.3: Synthetic Input
Equation 11.4: Calculating the SOM Output
Equation 11.5: Adjusting the SOM Weights (Additive)
Equation 11.6: Adjusting the SOM Weight (Subtractive)
Equation B.1: A typical matrix.
Equation B.2: Sum the Numbers Between 1 and 10
Equation B.3: Sum the Values Between 1 and 10
Equation B.4: Taking the Derivative of x2
Equation B.5: Calculating the Integral of the Sigmoid Function
Equation C.1: The Linear Threshold Function
Equation C.2: The Sigmoidal Threshold Function
Equation C.3: The Integral of the Sigmoidal Threshold Function
Equation C.4: The Hyperbolic Tangent Threshold Function
Equation C.5: The Integral of the Hyperbolic Tangent Threshold Function