Pit Pattern Classification with Support Vector Machines and Neural Networ

Transcription

1 Pit Pattern Classification with s and s February 1, 2007 Pit Pattern Classification with s and Neural Networ

2 Introduction Pit Pattern Classification with s and Neural Networ

3 Objectives Investigation of support vector machine and neural network for classification of pit pattern (256 x 256 RGB images) with previous feature extraction (co-occurrence histogram) and feature selection (principle component analysis) for a 2-class and a 6-class problem. Pit Pattern Classification with s and Neural Networ

4 Classification Process Pit Pattern Classification with s and Neural Networ

5 Co-occurrence histogram Considers dependencies between adjacent pixels. Co-occurrence distance (between two samples) can be varied. Orientation (vertical,horizontal,...) is not fixed. Pit Pattern Classification with s and Neural Networ

6 Principle Components Analysis (PCA) PCA aims to provided a better representation with lower dimension. Compaction of information. Process Create a new mean-adjusted data matrix X. Calculate a m m covariance matrix Σ from the mean-adjusted data X. Compute n significant eigenvectors W from the covariance matrix Σ. Perform dimensionality reduction: Y = W T X. n is a tradeoff between compression and quality. Pit Pattern Classification with s and Neural Networ

7 PCA Example Data points : x = ( ) ; y = ( ) ( ) ( ) X = = W = x = W T x = ( ) y = W T ỹ = ( ) Pit Pattern Classification with s and Neural Networ

8 Basic Concept Inputs: x 1,..., x j [0, 1] Synapses: w 1,..., w j R Neuron: net = w j x j Output: y = f (net θ) Bias value: θ Pit Pattern Classification with s and Neural Networ

9 Multi Layer Network Layer weights are adjusted during learning based on some input/output patterns. Learning typically starts at the output layer and moves toward the input layer (back-propagation). Pit Pattern Classification with s and Neural Networ

10 Mathematical Model Activation of the hidden layer net j = i w ji x i Output of the hidden layer ( y j = f (net j ) = f Activation of the output layer i w ji x i ) net k = j w kj f (net j ) Net output ( y k = f (net k ) = f j w kj f ( i w ji x i )) = f ( j w kj y j ) Pit Pattern Classification with s and Neural Networ

11 Basic Concept A support vector machine is a learning algorithm which attempts to separate patterns by a hyperplane defined through: normal vector w offset parameter b. Pit Pattern Classification with s and Neural Networ

12 What is an Optimal Hyperplane? Pit Pattern Classification with s and Neural Networ

13 Separation with maximal Margin Support vectors are all points lying on the margin closest to the hyper plane. Pit Pattern Classification with s and Neural Networ

14 Kernel Trick Nonlinear and complex separation in the 2-dimensional input space. Easier and often linear separation in higher dimensional feature spaces. Pit Pattern Classification with s and Neural Networ

15 Kernel Examples Linear kernel k(x, x ) = x T x = x, x Polynomial kernel of degree d k(x, x ) = (γ + x, x + coef 0) d Radial basis kernel (RBF) k(x, x ) = exp( γ x x 2 ) MLP or Sigmoid kernel k(x, x ) = tanh(γ x, x + coef 0) Pit Pattern Classification with s and Neural Networ

16 Classification Principle Pit Pattern Classification with s and Neural Networ

17 Mathematical Model Dual optimization problem max W (α) = m α R m i=1 α i 1 2 m i,j=1 α i y i α j y j k(x i, x j ) subject to α i 0, for all i = 1,..., m, and m i,j=1 α i y i = 0 Decision function ( m f (x) = sgn = sgn i=1 ( m i=1 ) α i y i φ(x), φ(x i ) ) + b ) α i y i k(x, x i ) + b Pit Pattern Classification with s and Neural Networ

18 Multi-Class Approaches Decomposition into several binary classification tasks. Pit Pattern Classification with s and Neural Networ

19 SVM Optimization Pit Pattern Classification with s and Neural Networ

20 PCA Dependency Pit Pattern Classification with s and Neural Networ

21 Co-occurrence Distance Dependency Pit Pattern Classification with s and Neural Networ

22 PCA Dependency for 6 Classes Pit Pattern Classification with s and Neural Networ

23 Additional Investigations Color-histogram (3-dimensional). Too high data dimension. Low classification results due to high data compression. Vertical co-occurrence histogram. 3-5% lower classification results compared with horizontal histogram. Combination of horizontal and vertical co-occurrence histogram. Lower classification results as with horizontal histogram. Pit Pattern Classification with s and Neural Networ

24 Problems High data dimension. Data scaling. Time intensive parameter optimization. SVM accepts invalid input. Too less training samples Pit Pattern Classification with s and Neural Networ

25 Summary SVM provides 10% better results than the NN. SVM parameter have to be optimized carefully. Better classification with PCA due to higher compaction of information. Low impact co-occurrence distance on classification accuracy. Pit Pattern Classification with s and Neural Networ

26 Outlook Evaluation of SVM with a higher amount of pit patterns. Consideration of other feature extraction and selection strategies. Investigation of other neural network topologies. Pit Pattern Classification with s and Neural Networ

27 Bibliography R.O. Duda, P.E. Hart and D.G. Stork. Pattern Classification, 2nd ed. New York: John Wiley & Sons, C.C. Chang and C.J. Lin. LIBSVM: a library for support vector machines. URL: cjlin/libsvm, [Feb. 24, 2006]. N. Cristianini and J. Shawe-Taylor. An Introduction to Support Vector Machines and other kernel-based learning methods. Cambridge: Cambridge University Press,2000. B. Schoelkopf and A.J. Smola. Learning with kernels. Cambridge,MA: MIT Press, J.E. Jackson A User s Guide to Principal Components. John Wiley & Sons Inc, P. Chang and J. Krumm. Object recognition with color cooccurance histograms. In Proceediungs of CVPR 99, Pit Pattern Classification with s and Neural Networ

28 Questions? Thank you for your attention. Pit Pattern Classification with s and Neural Networ