NEURAL NETWORK FUNDAMENTALS WITH GRAPHS, ALGORITHMS, AND APPLICATIONS

Transcription

1 NEURAL NETWORK FUNDAMENTALS WITH GRAPHS, ALGORITHMS, AND APPLICATIONS N. K. Bose HRB-Systems Professor of Electrical Engineering The Pennsylvania State University, University Park P. Liang Associate Professor of Electrical Engineering University of California, Riverside McGraw-Hill, Inc. New York St. Louis San Francisco Auckland Bogota Caracas Lisbon London Madrid Mexico City Milan Montreal New Delhi San Juan Singapore Sydney Tokyo Toronto

2 CONTENTS List of Figures List of Tables Preface List of Acronyms Glossary of Notations xiv xix xxi xxix xxxii I Fundamentals 1 Basics of Neuroscience and Artificial Neuron Models The Brain as a Neural Network Basic Properties of Neurons 9-1:2.1 Structure of a Neuron Dendritic Tree Action Potential and Its Propagation Synapses Connection Patterns between Neurons An Example: A Motion Detection Neuron Neuron Models McCulloch-Pitts Model Neuron Models with Continuous Transfer Characteristics Other Neuron Models Conclusions and Suggestions 28 Problems 31 2 Graphs Terminology and Preliminaries Special Types of Graphs Directed Graphs 41 ix

3 X CONTENTS 2.4 Matrix Representation of Graphs Adjacency Matrix Interconnection Matrix Topological Invariants Euler and Schlaefli Invariants Genus Thickness Some Other Topological Invariants Voronoi Diagrams and Deiaunay Tessellation Conclusions and Suggestions 55 Problems 57 3 Algorithms Computational Complexity: P- and NP-Complete Problems Shortest-Path and Max-Flow Min-Cut Problems Dijkstra's Shortest-Path Algorithm Max-Flow Min-Cut Algorithm Interconnection and Routing Algorithms Problem Formulation Minimal Spanning Tree (MST) Algorithms Minimal Fermat Tree (MFT) Problem Traveling Salesperson (TS) Problem Steiner Minimal Tree (SMT) Placement and Partitioning Placement Partitioning Parallel Computation Associative Memory The Linear Associator: Solution by Hebbian Rule The Linear Associator: Solution by Generalized Inverse Implementation of Associative Memory Conclusions 106 Problems 108 II Feedforward Networks 4 Perceptrons and the LMS Algorithm Rosenblatt's Perceptron Definitions Linear Separability of Training Patterns Perceptron Learning Algorithms Derivation of the Perceptron Algorithm as Gradient Descent The Perceptron Convergence Theorem The Widrow-Hoff LMS Algorithm Order of a Predicate and a Perceptron Conclusions and Suggestions 147 Problems 148

4 CONTENTS XI 5 Multilayer Networks Exact and Approximate Representation Using Feedforward Networks Exact Representation: Kolmogorov's Theorem and Its Consequences Approximate Representations Fixed Multilayer Feedforward Network Training by Backpropagation Implementation Considerations for Backpropagation Variants of BPA Temporal Signal Recognition and Prediction Structural Training of Multilayer Feedforward Networks Algorithm for Design Based on VoD Robustness and Size Issues Unsupervised and Reinforcement Learning Principal Component Analysis Networks Self-Organization in a Perceptual Network Reinforcement Learning The Probabilistic Neural Network Conclusions and Suggestions 209 Problems Complexity of Learning Using Feedforward Networks Learnability in ANN The Problem of Loading Using an Appropriate Network to Get Around Intractability Generalizability of Learning VC Dimension and Generalization 232 \ "6.2.2 Sufficient Conditions for Valid Generalization ' in Feedforward Networks Necessary Conditions for Valid Generalization in Feedforward Networks Discussions and Ways to Improve Generalization Space Complexity of Feedforward Networks Order of a Function and the Complexity of a Network High Connectivity in Analog Neural Computations Summary and Discussion 250 Problems Adaptive-Structure Networks Growth Algorithms The Upstart Algorithm Learning by Divide and Conquer Other Growth Algorithms 265

5 XH CONTENTS 7.2 Networks with Nonlinear Synapses and Nonlinear Synaptic Contacts Quasi-Polynomial Synapses and Product Synaptic Contacts Generalization of Learning and Hardware Considerations Conclusions and Suggestions 278 Problems 281 III Recurrent Networks 8 Symmetric and Asymmetric Recurrent Networks Symmetric Hopfield Networks and Associative Memory Convergence Proofs Computation in a Network and Minimum Cuts in a Graph Capacity and Spurious Memory Correlated Patterns Hopfield Networks with Variations in the Connection Weights Bidirectional Associative Memory Symmetric Networks with Analog Units Analog Hopfield Networks Convergence Proof Relation between Stable States of Discrete and Analog Hopfield Networks Cellular Neural Networks Seeking the Global Minimum: Simulated Annealing Simulated Annealing in Optimization Stochastic Networks: Applying Simulated Annealing to Hopfield Networks A Learning Algorithm for the Boltzmann Machine Learning the Underlying Structure of an Environment The Learning Procedure Mean Field Theory and the Deterministic Boltzmann Machine Asymmetric Recurrent Networks Phase Transition from Stationary to Chaotic Spatial and Temporal Patterns Learning in Asymmetric Networks: Recurrent Backpropagation Summary and Discussion 340 Problems Competitive Learning and Self-Organizing Networks Unsupervised Competitive Learning Two Phases of Competitive Learning 346

6 CONTENTS XUl Using a Competitive Learning Network for Associative Memory Adaptive Resonant Networks The ART1 Clustering Algorithm The ART1 Network Self-Organizing Feature Maps The Kohonen Map Analysis of Kohonen Maps Adaptive and Learning Vector Quantization Two-Dimensional Topographic Maps A Multilayer Self-Organizing Feature Map Hybrid Learning Counterpropagation Network Regularizing Networks and Radial Basis Functions Summary and Discussion Problems IV Applications of Neural Networks 10 Neural Network Approaches to Solving Hard Problems The Traveling Salesperson Problem Multitarget Tracking Time Series Prediction Talking Network and Phonetic Typewriter Speech Generation Speech Recognition Autonomous Vehicle Navigation Handwritten Digit Recognition Image Compression by a Multilayer Feedforward { - - Structure Trained through Backpropagation 430 I 10.8 Character Retrieval Using the Discrete Hopfield Network Visual Processing Networks Conclusion and Discussion 443 References 447 Appendix A Basis of Gradient-Based Optimization Methods 463 A.I The Gradient Descent Method 464 A.2 Newton's Method 467 A.3 The Conjugate Gradient Method 468 A.4 Constrained Optimization 469 Bibliography 470 Index 471