Part 3. Nonnegative Matrix Factorization K-means and Spectral Clustering. PCA & Matrix Factorization for Learning, ICML 2005 Tutorial, Chris Ding 100

Transcription

1 Part 3. Nonnegative Matrix actorization K-means and Spectral Clustering PCA & Matrix actorization for Learning ICML 5 utorial Chris Ding 1

2 Nonnegative Matrix actorization (NM Data Matrix: n points in p-dim: X ( x x L x 1 n x i is an image document webpage etc Decomposition (low-rank approximation X Nonnegative Matrices X ij ij ij ( f f L f 1 k ( g g L g 1 k PCA & Matrix actorization for Learning ICML 5 utorial Chris Ding 11

3 Some historical notes Earlier work by statistics people P. Paatero (1994 Environmetrices Lee and Seung (1999 Parts of whole (no cancellation A multiplicative update algorithm PCA & Matrix actorization for Learning ICML 5 utorial Chris Ding 1

4 M 8... Pixel vector PCA & Matrix actorization for Learning ICML 5 utorial Chris Ding 13

5 Parts-based perspective X ( x x L x 1 n ( f f L f 1 k ( g g L g 1 k X PCA & Matrix actorization for Learning ICML 5 utorial Chris Ding 14

6 Sparsify to get parts-based picture X f f L f ( 1 k Li et al ; oyer 3 PCA & Matrix actorization for Learning ICML 5 utorial Chris Ding 15

7 heorem. NM kernel K-means clustering NM produces holistic modeling of the data heoretical results and experiments verification (Ding e Simon 5 PCA & Matrix actorization for Learning ICML 5 utorial Chris Ding 16

8 Our Results: NM Data Clustering PCA & Matrix actorization for Learning ICML 5 utorial Chris Ding 17

9 Our Results: NM Data Clustering PCA & Matrix actorization for Learning ICML 5 utorial Chris Ding 18

10 heorem: K-means NM Reformulate K-means and Kernel K-means max I Show equivalence r( W min I W PCA & Matrix actorization for Learning ICML 5 utorial Chris Ding 19

11 NM K-means arg max r( I W arg min [-r( W I ] arg min [ W -r( W ] + I arg min I W arg min W PCA & Matrix actorization for Learning ICML 5 utorial Chris Ding 11

12 Normalized Cut: Normalized Cut J Spectral Clustering NM Ncut < k l> k l k k 1 ( D W h1 hk ( D W hk + + h1 Dh1 hk Dhk n 1/ 1/ Unsigned cluster indicators: yk D (L1 Re-write: ~ J Ncut ( y1 L yk y1 ( I W y1 ~ r( Y ( I W Y k } L1 L + L+ ~ Optimize : maxr( Y W Y subject to Y s( C kc d h L l min I ~ W y yk ( I k ~ 1/ 1/ (u et al 1 PCA & Matrix actorization for Learning ICML 5 utorial Chris Ding s( CkC d l W s( C / D C d Y h k Y D WD ~ W k k I

13 Advantages of NM over standard K-means Soft clustering PCA & Matrix actorization for Learning ICML 5 utorial Chris Ding 11

14 Experiments on Internet Newsgroups N: comp.graphics N9: rec.motorcycles N1: rec.sport.baseball N15: sci.space N18: talk.politics.mideast cosine similarity 1 articles from each group. 1 words f.idf weight. Cosine similarity Accuracy of clustering results K-means W PCA & Matrix actorization for Learning ICML 5 utorial Chris Ding 113

15 Summary for Symmetric NM K-means Kernel K-means Spectral clustering max I r( W Equivalence to min I W PCA & Matrix actorization for Learning ICML 5 utorial Chris Ding 114

16 Nonsymmetric NM K-means Kernel K-means Spectral clustering max I r( W Equivalence to min I W PCA & Matrix actorization for Learning ICML 5 utorial Chris Ding 115

17 Non-symmetric NM Rectangular Data Matrix Bipartite raph Information Retrieval: word-to-document DNA gene expressions Image pixels Supermarket transaction data PCA & Matrix actorization for Learning ICML 5 utorial Chris Ding 116

18 PCA & Matrix actorization for Learning ICML 5 utorial Chris Ding 117 K-means Clustering of Bipartite raphs r( ( 1 B C R B s J k j j j C R Kmeans j j Simultaneous clustering of rows and columns cut f f f ( g g g ( row indicators column indicators

19 PCA & Matrix actorization for Learning ICML 5 utorial Chris Ding 118 NM K-means clustering arg maxr( B I I arg min r( B I I arg min r( B B I I + arg min I B I

20 Solving NM with non-negative least square J X ix solve for ; ix solve for J g~ 1 x i g~ i M 1 g~ n n i Iterate converge to a local minima PCA & Matrix actorization for Learning ICML 5 utorial Chris Ding 119

21 Solving NM with multiplicative updating J X ix solve for ; ix solve for Lee & Seung ( propose ( X ( jk jk ( X ( jk jk PCA & Matrix actorization for Learning ICML 5 utorial Chris Ding 1

22 Symmetric NM J W Constraint Optimization. KK 1 st condition Complementarity slackness condition radient decent ε J ( 4W + 4 J (1 β + ( W β ( PCA & Matrix actorization for Learning ICML 5 utorial Chris Ding 11 ε 4(

23 Summary NM is a new low-rank approximation he holistic picture (vs. parts-based NM is equivalent to spectral clustering Main advantage: soft clustering PCA & Matrix actorization for Learning ICML 5 utorial Chris Ding 1