Dimensionality Reduction and Visualization

Transcription

1 MTTTS16 Dimensionality Reduction and Visualization Spring 2016 Jaakko Peltonen Lecture 7: Nonlinear dimensionality reduction, part 2

2 Two measures of faithfulness - precision and recall

3 Faithfully? Good Precision: Points that are close in the reduced space are close in the original space Good Recall: Points that are close in the original space are close in the reduced space R d R 2 3

4 Faithfully? Good Precision: Points that are close in the reduced space are close in the original space Good Recall: Points that are close in the original space are close in the reduced space R d R 2 4

5 Faithfully? Good Precision: Points that are close in the reduced space are close in the original space Good Recall: Points that are close in the original space are close in the reduced space In general, impossible to get both

6 May be impossible to get both good recall and good precision from Venna, Peltonen, Nybo, Aidos, and Kaski, Information retrieval perspective to nonlinear dimensionality reduction for data visualization,journal of Machine Learning Research,

7 May be impossible to get both good recall and good precision Squashed flat sphere surface yields good recall but bad precision from Venna, Peltonen, Nybo, Aidos, and Kaski, Information retrieval perspective to nonlinear dimensionality reduction for data visualization,journal of Machine Learning Research,

8 May be impossible to get both good recall and good precision Cut-open sphere surface yields good precision but less good recall from Venna, Peltonen, Nybo, Aidos, and Kaski, Information retrieval perspective to nonlinear dimensionality reduction for data visualization,journal of Machine Learning Research,

9 MDS Performance of MDS MDS is tries to preserve the large distances at the expense of small ones, hence, it can collapse some small distances on the expense of preserving large distances A projection is trustworthy (precision) if k closest neighbors of a sample on the projection are also close by in the original space. A projection preserves the original neighborhoods (recall) if all k closest neighbors of a sample in the original space are also close by in the projection. Precision-recall curve, as a function of the neighborhood size k for an s- curve data set. k not shown - each k Precision yields some (precision, recall) value pair onto the curve. 20 closest original-space neighbours are considered true matches. Curves close to top right are better. Recall Figures are from Venna et al., Information retrieval perspective to nonlinear dimensionality reduction for data visualization, JMLR 2010.

10 More methods: Self-Organizing Map, Laplacian Eigenmap, Curvilinear Component Analysis

11 Self-Organizing Maps

12 Self-Organizing Maps today: about Google Scholar results for Kohonen. 12

13 Self-Organizing Maps 13

14 Vector Quantization 14

15 Vector Quantization 15

16 Vector Quantization 16

17 Vector Quantization minimizes the distortion E = 1 NV NX kx i v(x i )k 2 i=1 with N the number of points (samples), V the number of centroids (units), x the samples, v the centroids 17

18 Vector Quantization One of the 1000 algorithms: - random initialization among the samples - iterations v(x(t)) = v(x(t)) + (t)(x(t) v(x(t))) with v(x(t)) the closest centroid of x(t) and Robbins-Monro conditions Z 1 Z 1 0 (t)dt = 1 0 (t) 2 dt < 1 18

19 Vector Quantization Matlab examples 19

20 Self-Organizing Maps like stretching a sheet of plastic by pulling on certain points on it 20

21 SOM Structure of the map (definition) 3 5 Map in the data space 4

22 SOM Structure of the map (definition) (1,1) (1,2) (1,1) Map in the data space (1,2) (2,1) (2,2) (2,1) (2,2) (3,2) (3,1) (3,2) (3,1)

23 SOM

24 SOM Random initialization or PCA initialization Iterations Example: 24

25 SOM Matlab Examples 25

26 SOM And then? What about visualization? (1,1) (1,2) (2,2) (2,1) (3,2) 26 (3,1)

27 SOM And then? What about visualization? (1,1) (1,2) (2,2) (2,1) (3,2) 27 (3,1)

28 SOM And then? What about visualization? (1,1) (1,2) (1,1) (1,2) (2,1) (2,2) (2,1) (2,2) (3,2) (3,1) (3,2) 28 (3,1)

29 SOM And then? What about visualization? (1,1) (1,2) (1,1) (1,2) (2,1) (2,2) (2,1) (2,2) (3,2) (3,1) (3,2) 29 (3,1)

30 SOM Wealth Data Example: Excel + Matlab 30

31 SOM 11 United States Canada Brazil India China Russian Federation Japan 10 Germany Australia Indonesia 9 France United Kingdom Italy Chile Saudi Arabia 8 Spain South Africa Mexico Nigeria Iran, Islamic Rep. Venezuela, RB Turkey Poland Greece 7 Thailand Pakistan Korea, Rep. Norway Kuwait United Arab Emirates 6 Czech Republic Romania New Zealand Bangladesh Sudan Netherlands Ukraine Ecuador Argentina Colombia Algeria Malaysia Mongolia Bulgaria 5 Hungary Sweden Guatemala Peru Philippines Vietnam Egypt, Arab Rep. Oman Uzbekistan Ireland 4 Portugal Morocco Denmark Austria Kenya Ethiopia Honduras Finland Congo, Dem. Rep. Angola El Salvador Croatia 3 Israel Slovak Republic Uganda Tunisia Sri Lanka Belarus Dominican Republic Nepal Côte d Ivoire Ghana Madagascar Papua New Guinea Zambia Cameroon Bolivia Syrian Arab Republic Azerbaijan Congo, Rep. Gabon Georgia Kyrgyz Republic Costa Rica Niger Uruguay Senegal Zimbabwe Guyana Latvia 2 Jordan Albania Mozambique Chad Lao PDR Lithuania Burkina Faso Panama Nicaragua Mali Benin Central African Republic Cape Verde Mauritius TogoSt. Vincent and the Grenadines Haiti Burundi Malawi Botswana Bahrain Brunei Darussalam Singapore Macedonia, FYR 1 Armenia Mauritania Bhutan Belgium Jamaica 2.5 Liberia 3 Guinea Fiji Vanuatu St. Kitts and Nevis Sierra Leone Rwanda Trinidad and Tobago Luxembourg Switzerland Moldova Tonga Tajikistan Namibia Lesotho Seychelles SwazilandComoros Belize Iceland Maldives Malta Dominica Gambia, The Grenada Guinea Bissau 31

32 SOM Values of the prototype vectors shown on the SOM grid. Left: one of the feature values. Right: many of the feature values. From Juha Vesanto s doctoral thesis,

33 Laplacian eigenmap Manifolds Laplacian eigenmap is a spectral method, like PCA. construct k-nearest neighbors graph. Assign W ij=1, if i and j are neighbors, otherwise assign Wij=0. Define diagonal matrix D, Dii= jwij, and graph Laplacian, L=D-W. The embedding of data points is given by the eigenvectors of L, corresponding to the d smallest nonzero eigenvalues. 33

34 Laplacian eigenmap Manifolds N=6 octave:20> L L = k= octave:21> [V,D] = eig(l) V = D =

35 Laplacian eigenmap Manifolds >> L L = >> [V,D]=eig(L) V = D =

36 Laplacian eigenmap Manifolds >> L L = >> [V,D]=eig(L) V = D = Y X Y X 36

37 Manifolds Curvilinear component analysis (CCA) Demartines, Hérault, Curvilinear component analysis (CCA) is like (absolute) MDS, except that only short distances are taken into account. More formally, the cost function reads r = i<j (d(x i, x j ) d(y i, y j )) 2 F (d(y i, y j ), y) where F(d,λy) equals unity, if d<λy, and zero otherwise; and d denotes the Euclidean distance of points in the original space (x) and in the projection (y), respectively. (Actually, F(d,λy), could be any monotonically decreasing function in d.) 37

38 Manifolds Curvilinear component analysis (CCA) CCA performs generally well in terms of precision; it appears to be quite robust. Notice outliers at right: they are result of small neighborhood Original 3 2 CCA !0.5!1!1!1.5! !1 0 1!3!6!5!4!3!2! Figures by Jarkko Venna 38

39 Manifolds Laplacian eigenmap Original !0.5 1 Eigenmap a CCA 0 0!6!0.5 b 39!5!4!3!2!1 0 1

40 Methods that directly use precision and recall? We interpreted some methods by precision and recall. Most methods don t optimize these, they use various cost functions. The Neighbor Retrieval Visualizer (Venna & Kaski 2007; Venna, Peltonen, Nybo, Aidos, Kaski 2010) directly maximizes a user-specified tradeoff of precision & recall NeRV, result for max recall NeRV, result for max precision Stochastic Neighbor Embedding (Hinton&Roweis 2002) can also be interpreted like this: maximizes recall Details later on the Neighbor Embedding lectures

41 Next lecture: general mathematical framework for nonlinear dimensionality reduction more methods: Isomap, Locally Linear Embedding, Maximum Variance Unfolding Supervised nonlinear dimensionality reduction