NEURAL NETWORKS A Comprehensive Foundation

Transcription

1 NEURAL NETWORKS A Comprehensive Foundation Second Edition Simon Haykin McMaster University Hamilton, Ontario, Canada Prentice Hall Prentice Hall Upper Saddle River; New Jersey 07458

2 Preface xii Acknowledgments xv Abbreviations and Symbols xvii 1 Introduction 1.1 What Is a Neural Network? Human Brain Models of a Neuron Neural Networks Viewed as Directed Graphs Feedback Network Architectures Knowledge Representation Artificial Intelligence and Neural Networks Historical Notes 38 Notes and References 45 Problems 45 2 Learning Processes Introduction Error-Correction Learning Memory-Based Learning Hebbian Learning Competitive Learning Boltzmann Learning 60

3 vi Contents 2.7 Credit Assignment Problem Learning with a Teacher Learning without a Teacher Learning Tasks Memory Adaptation Statistical Nature of the Learning Process Statistical Learning Theory Probably Approximately Correct Model of Learning Summary and Discussion 105 Notes and References 106 Problems Single Layer Perceptrons Introduction Adaptive Filtering Problem Unconstrained Optimization Techniques Linear Least-Squares Filters Least-Mean-Square Algorithm Learning Curves Learning Rate Annealing Techniques Perceptron Perceptron Convergence Theorem Relation Between the Perceptron and Bayes Classifier for a Gaussian Environment Summary and Discussion 148 Notes and References 150 Problems Multilayer Perceptrons Introduction Some Preliminaries Back-Propagation Algorithm Summary of the Back-Propagation Algorithm XOR Problem Heuristics for Making the Back-Propagation Algorithm Perform Better Output Representation and Decision Rule Computer Experiment Feature Detection Back-Propagation and Differentiation Hessian Matrix Generalization 205

4 vii 4.13 Approximations of Functions Cross-Validation Network Pruning Techniques Virtues and Limitations of Back-Propagation Learning Accelerated Convergence of Back-Propagation Learning Supervised Learning Viewed as an Optimization Problem Convolutional Networks Summary and Discussion 247 Notes and References 248 Problems Radial-Basis Function Networks Introduction Cover's Theorem on the Separability of Patterns Interpolation Problem Supervised Learning as an Ill-Posed Hypersurface Reconstruction Problem Regularization Theory Regularization Networks Generalized Radial-Basis Function Networks XOR Problem (Revisited) Estimation of the Regularization Parameter Approximation Properties of RBF Networks Comparison of RBF Networks and Multilayer Perceptrons Kernel Regression and Its Relation to RBF Networks Learning Strategies Computer Experiment Summary and Discussion 308 Notes and References 308 Problems Support Vector Machines Introduction Optimal Hyperplane for Linearly Separable Patterns Optimal Hyperplane for Nonseparable Patterns How to Build a Support Vector Machine for Pattern Recognition Example: XOR Problem (Revisited) Computer Experiment e-insensitive Loss Function Support Vector Machines for Nonlinear Regression Summary and Discussion 343 Notes and References 347 Problems 348

5 viii Contents 7 Committee Machines Introduction Ensemble Averaging Computer Experiment I Boosting Computer Experiment II Associative Gaussian Mixture Model Hierarchical Mixture of Experts Model Model Selection Using a Standard Decision Tree A Priori and a Posteriori Probabilities Maximum Likelihood Estimation Learning Strategies for the HME Model EM Algorithm Application of the EM Algorithm to the HME Model Summary and Discussion 386 Notes and References 387 Problems Principal Components Analysis Introduction Some Intuitive Principles of Self-Organization Principal Components Analysis Hebbian-Based Maximum Eigenfilter Hebbian-Based Principal Components Analysis Computer Experiment: Image Coding Adaptive Principal Components Analysis Using Lateral Inhibition Two Classes of PCA Algorithms Batch and Adaptive Methods of Computation Kernel-Based Principal Components Analysis Summary And Discussion 437 Notes And References 439 Problems Self-Organizing Maps Introduction Two Basic Feature-Mapping Models Self-Organizing Map Summary of the SOM Algorithm Properties of the Feature Map Computer Simulations Learning Vector Quantization Computer Experiment: Adaptive Pattern Classification 468

6 ix 9.9 Hierarchical Vector Quantization Contextual Maps Summary and Discussion 476 Notes and References 477 Problems Information-Theoretic Models Introduction Entropy Maximum Entropy Principle Mutual Information Kullback-Leibler Divergence Mutual Information as an Objective Function To Be Optimized Maximum Mutual Information Principle Infomax and Redundancy Reduction Spatially Coherent Features Spatially Incoherent Features Independent Components Analysis Computer Experiment Maximum Likelihood Estimation Maximum Entropy Method Summary and Discussion 533 Notes and References 535 Problems Stochastic Machines And Their Approximates Rooted In Statistical Mechanics Introduction Statistical Mechanics Markov Chains Metropolis Algorithm Simulated Annealing Gibbs Sampling Boltzmann Machine Sigmoid Belief Networks Helmholtz Machine Mean-Field Theory Deterministic Boltzmann Machine Deterministic Sigmoid Belief Networks Deterministic Annealing Summary and Discussion 592 Notes and References 594 Problems 597

7 12 Neurodynamic Programming Introduction Markovian Decision Processes Bellman's Optimality Criterion Policy Iteration Value Iteration Neurodynamic Programming Approximate Policy Iteration Q-Leaming Computer Experiment Summary and Discussion 629 Notes and References 631 Problems Temporal Processing Using Feedforward Networks Introduction Short-term Memory Structures Network Architectures for Temporal Processing Focused Time Lagged Feedforward Networks Computer Experiment Universal Myopic Mapping Theorem Spatio-Temporal Models of a Neuron Distributed Time Lagged Feedforward Networks Temporal Back-Propagation Algorithm Summary and Discussion 659 Notes and References 660 Problems Neurodynamics Introduction Dynamical Systems Stability of Equilibrium States Attractors Neurodynamical Models Manipulation of Attractors as a Recurrent Network Paradigm Hopfield Models Computer Experiment I Cohen-Grossberg Theorem Brain-State-in-a-Box Model Computer Experiment II Strange Attractors and Chaos Dynamic Reconstruction of a Chaotic Process 714

8 xi Computer Experiment III Summary and Discussion 722 Notes and References 725 Problems Dynamically Driven Recurrent Networks Introduction Recurrent Network Architectures State-Space Model Nonlinear Autoregressive with Exogenous Inputs Model Computational Power of Recurrent Networks Learning Algorithms Back-Propagation Through Time Real-Time Recurrent Learning Kalman Filters Decoupled Extended Kalman Filters Computer Experiment Vanishing Gradients in Recurrent Networks System Identification Model-Reference Adaptive Control Summary and Discussion 782 Notes and References 783 Problems 785 Epilogue 790 Bibliography 796 Index 837