Neural Networks Kohonen Self-Organizing Maps

Transcription

1 Neural Networks Kohonen Self-Organizing Maps Mohamed Krini Christian-Albrechts-Universität zu Kiel Faculty of Engineering Institute of Electrical and Information Engineering Digital Signal Processing and System Theory

2 Contents of the Lecture Entire Semester Introduction Pre-Processing and Feature Extraction Threshold Logic Units - Single Perceptrons Multilayer Perceptrons Training Multilayer Perceptrons Radial Basis Function Networks Learning Vector Quantization Kohonen Self-Organizing Maps Hopfield and Recurrent Networks Slide VIII-2

3 Contents of this Part Kohonen Self-Organizing Maps Introduction Definitions and Properties Neighborhood of Output Neurons Learning Rule Topology Functions Examples Applications Literature Slide VIII-3

4 Introduction Structure of a Self-Organizing Map: Output neurons and their neighborhood relationships The map shows the output neurons and the neighborhood relationships. These neighborhood relationships are called topology. Connections Each input neuron is connected to all output neurons. Input neurons Slide VIII-4

5 Definitions and Properties (1/2) Definitions: A self-organizing map (SOM) or Kohonen feature map is a two layer network without hidden neurons: All input neurons are connected to all output neurons: The distance between the input and the weight vectors is used as input function, similar to RBFN. The activation function is a radial function, i.e. a monotonously decreasing function: The identity is used as output function for each output neuron. Slide VIII-5

6 Definitions and Properties (2/2) Properties: The neurons are interconnected by neighborhood relationships. These relationships are called topology which is described by a distance function: The training of a SOM is highly influenced by its topology. A SOM always activates the neuron with the least distance to an input pattern (principle of winner-takes-all is often used). In absence of neighborhood relationships the SOM operates as vector quantization of the input space. Self-Organizing Maps are networks trained without a teacher. Slide VIII-6

7 Neighborhood of Output Neurons One-Dimensional Topology: Two-Dimensional Topology: Two example topologies of a self-organizing map. Black lines indicate neighbors of a neuron. Gray lines indicate regions assigned to neurons. Topologies with more dimensions would also be possible. They are often not employed due to visualization complexity. Slide VIII-7

8 Learning Rule (1/2) Training Procedure: 1. The network starts with random neuron centers 2. An input vector is selected from the input space. 3. The distance is determined for every neuron. The winner neuron with the maximum activation is selected. 4. The reference vectors are updated using a neighborhood function and the update rule: Neighborhood radius 5. The training is continued with step 2 as long as the maximum number of iterations is not reached. Slide VIII-8

9 Learning Rule (2/2) Training Parameters: The function, also called topology function, represents the neighborhood relationships between the neurons. Note that the function is defined on the grid and not on the input space. The function can be any unimodal function that reaches its maximum at. Time-varying learning rate: Time-varying neighborhood radius: Slide VIII-9

10 Topology Functions (1/2) Common Topology Function: A common topology function is the Gaussian function: with represents the neuron positions on the grid, not on the input space. denotes the winner neuron position. Other functions, such as the cone function or the Mexican hat function, can also be used. Slide VIII-10

11 Topology Functions (2/2) Different Functions: Gaussian Triangular Mexican Hat Rectangle Slide VIII-11

12 Examples Two Dimension Topology (1/3) Unfolding of a Two-Dimensional Self-Organizing Map: 10x10 output neurons were used for the two-dimensional topology. Training patterns randomly distributed within the interval are utilized. Random initialization of the reference vectors. Nearest neighbor neurons are connected by a straight line. Learning rate Neighborhood radius Gaussian topology function is used. Slide VIII-12

13 Examples Two Dimension Topology (2/3) Unfolding of a Two-Dimensional Self-Organizing Map (continued): Initial state Slide VIII-13

14 Examples Two Dimension Topology (3/3) Activation of the Output Neurons: (Input pattern:, Gaussian activation function) Slide VIII-14

15 Examples One Dimension Topology Unfolding of a One-Dimensional Self Organizing Map: Initial state Slide VIII-15

16 Examples Failure of the Training Unfolding of a Two-Dimensional Map Inappropriate Initialization: If the initialization is not chosen appropriately, e.g. if the learning rate or the neighborhood radius parameters are chosen too low, the training of a selforganizing network may fail. Examples on the left demonstrate the maps at iteration 1000, 10000, and The maps are not correctly unfolded. Slide VIII-16

17 Examples Dimension Reduction 2D Self Organizing Map in a 3D Input Space: Self-organizing maps were trained with random points of a rotation parabola (upper graphs) and of a cubic function (lower graphs). Three input neurons are used. Initial state State after iterations A map with 10x10 output neurons is utilized. Reference vectors of adjacent output neurons are connected by a straight line. Original and image space have different dimensions. Self-organizing maps can be used for dimensionality reduction. Slide VIII-17

18 Applications Speech Recognition Phoneme Map in Finnish: a a a ah h ae ae o a a h r ae l o o a r r r g o o r r r m m Part of (hexagonal) feature map Phonemes are extracted from the input signal using the FFT, logarithm, averaging and normalization. During learning, each neuron begins to respond strongly to a specific phoneme. A specific neuron is most active after the training. Some nodes and their phoneme labels are shown on the left. The map works very well in Finnish (phonetic language). Slide VIII-18

19 Applications Dimension Reduction (1/3) dove hen duck goose owl hawk eagle fox wolf dog cat tiger lion horse zebra small medium big legs legs hair hooves mane feathers hunt run fly swim cow Animal Names and Attributes: Each column is a description of an animal, based on the presence (=1) or absence (=0) of some of the 13 different attributes given on the left. Each animal was encoded by a 29- dim data vector: a 13-dimensional vector for the attributes and a 16- dimensional vector for the animal name (vector in which only the element corresponding to the animal has a non-zero value). Slide VIII-19

20 Applications Dimension Reduction (2/3) Trained Network: duck * * horse * * * cow * * * * * * zebra * * * * * * * * * * * * * * tiger goose * * * * * wolf * * * * * * hawk * * * * * * * * owl * * * * * * lion dove * * * * * dog * * * * * * * * * * * * * * * * eagle * * * * * * hen * * * * * fox * * cat A Kohonen Network of 10 x 10 x 29 was utilized. The network was trained for 2000 epochs. The map shows the cells giving the strongest response when only the animal name is presented as input (the dots indicate neurons with weaker responses). Slide VIII-20

21 Applications Dimension Reduction (3/3) Contextual Map: duck duck horse horse zebra zebra cow cow cow cow duck duck horse zebra zebra zebra cow cow tiger tiger goose goose goose zebra zebra zebra wolf wolf tiger tiger goose goose hawk hawk hawk wolf wolf wolf tiger tiger goose owl hawk hawk hawk wolf wolf wolf lion lion dove owl owl hawk hawk dog dog dog lion lion dove dove owl owl owl dog dog dog dog lion dove dove eagle eagle eagle dog dog dog dog cat hen hen Eagle eagle eagle fox fox fox cat cat hen hen eagle eagle eagle fox fox fox cat cat This map shows the result of a simulated electrode penetration mapping". Each cell has been labeled with the animal name that is its best stimulus, i.e., elicits the strongest response for that cell. The result is a contextual map. Similar animals are adjacent. Slide VIII-21

22 Literature Further details can be found in: S. Haykin: Neural Networks and Learning Machines (Chapter 9), Prentice-Hall, 3rd edition, C. Borgelt, F. Klawonn, R. Kruse, D. Nauck: Neuro-Fuzzy-Systeme (Chapter 6), Vieweg Verlag, Wiesbaden, 2003 (in German). R. Rojas: Neural Networks A Systematic Introduction (Chapter 15), Springer, Berlin, Germany, H. Ritter, T. Kohonen: Self-Organizing Semantic Maps, Biol. Cybern. 61, , Springer, D. Kriesel: A Brief Introduction to Neural Networks (chapter 10), Slide VIII-22