Kernel Methods for General Pattern Analysis

Transcription

1 Kernel Methods for General Pattern Analyss Nello Crstann Unversty of Calforna, Davs

2 Overvew Kernel Methods are a new class of pattern analyss algorthms whch can operate on very general types of data and can detect very general types of relatons. Correlaton, factor, cluster and dscrmnant analyss are just some of the types of pattern analyss tasks that can be performed on data as dverse as sequences, tet, mages, graphs and of course vectors. The method provdes also a natural way to merge and ntegrate dfferent types of data.

3 Overvew Kernel methods offer a modular framework. In a frst step, a dataset s processed nto a kernel matr. Data can be of varous types, and also heterogeneous types. In a second step, a varety of kernel algorthms can be used to analyze the data, usng only the nformaton contaned n the kernel matr

4 Overvew Computatonally most kernel-based learnng algorthms reduce to optmzng conve cost functons or to computng generalzed egenvectors of large matrces. Kernel desgn s based on varous methods. For dscrete data (eg sequences) often use methods lke dynamc programmng, branch and bound, dscrete contnuous optmzaton Combnaton s very effcent but stll computatonally challengng, for our ambtons

5 Overvew The fleble combnaton of approprate kernel desgn and relevant kernel algorthms has gven rse to a powerful and coherent class of methods, whose computatonal and statstcal propertes are well understood, Increasngly used n applcatons as dverse as bosequences and mcroarray data analyss, tet mnng, machne vson and handwrtng recognton.

6 Overvew General ntroducton to kernel methods Dscusson of specfc kernel-based algorthms Dscusson of specfc kernels for strngs Few eamples based on tet and sequence data Dscusson of how to combne heterogeneous data types (maybe) Focus on egen- algorthms from classc multvarate stats, combned wth kernels

7 The Basc Idea: φ( ) Kernel Methods work by: 1-embeddng data n a vector space 2-lookng for (lnear) relatons n such space If map chosen sutably, comple relatons can be smplfed, and easly detected

8 Man Idea / two observatons 1- Much of the geometry of the data n the embeddng space (relatve postons) s contaned n all parwse nner products* We can work n that space by specfyng an nner product functon between ponts n t (rather than ther coordnates) <1,1> <2,1>. <1,2> <1,n>. <2,2> <2,n> 2- In many cases, nner product n the embeddng space very cheap to compute. <n,1>... <n,2>. <n,n> * Inner products matr

9 Algorthms Algorthms that can be used wth nner product nformaton: Rdge Regresson Fsher Dscrmnant Prncpal Components Analyss Canoncal Correlaton Analyss Spectral Clusterng Support Vector Machnes Lots more

10 Notaton For now let us assume we have a vector space X where the data lves, and where we can defne nner products And we also have a sample S of ponts from that space o b o o z o, = w w o o z, + b= 0

11 Dual Representaton We consder lnear functons lke ths: they could be the result of Fsher Dscrmnant Analys, Rdge regresson, PCA, etc f ( ) = w' + b We consder re-wrtng them as a functon of the data ponts (see perceptron as eample) (we have a sample S of data ponts) w = α f ( ) = w' + b = α S ' + b

12 More generally = + = + = + = = = S j S j S span j S j j S span S S f w w f ' 0 ' ' ' ) ( ' ' ) ( ) ( ) ( α α α α α α α The lnear functon f() can be wrtten n ths form Wthout changng ts behavor on the sample See Wahba s Representer theorem for theory behnd Not crucal for ths talk for all algorthms we consder ths s always vald

13 w = αy Dual Representaton f () = w, + b= α y, + b Ths representaton of the lnear functon only needs nner products between data ponts (not ther coordnates!) JUST REWRITTEN THE SAME THING from #parameters w = dmenson to #parameters α = sample sze! If we want to work n the embeddng space φ( ) just need to know ths: K ( 1, 2) = φ( 1), φ( 2) Pardon my notaton

14 Kernels K( 1, 2) = φ ( 1), φ ( 2) Kernels are functons that return nner products between the mages of data ponts n some space. By replacng nner products wth kernels n lnear algorthms, we obtan very fleble representatons Choosng K s equvalent to choosng Φ (the embeddng map) Kernels can often be computed effcently even for very hgh dmensonal spaces see eample

15 Classc Eample Polynomal Kernel z = ( 1, 2); = ( z1, z2); 2 z, = ( z 1 1+ z 2 2) 2 = = z + z + 2zz = = (,, 212),( z, z, 2z1z2) = 1 2 = φ( ), φ( z) 2 1 2

16 Can Learn Non-Lnear Separatons f ( ) = αk(, ) By combnng a smple lnear dscrmnant algorthm wth ths smple Kernel, we can learn nonlnear separatons (effcently).

17 More Important than Nonlnearty Kernels can be defned on general types of data (as long as few condtons are satsfed see later) Many classcal algorthms can naturally work wth general, non-vectoral, data-types! We can represent general types of relatons on general types of data Kernels est to embed sequences, trees, graphs, general structures Semantc Kernels for tet, Kernels based on probablstc graphcal models etc

18 The Kernel Matr Gven a sample of data, one can defne the kernel matr <1,1> <2,1>. <n,1>. <1,2> <1,n>... <2,2>. <n,2> <2,n>. <n,n> Mercer Theorem: The kernel matr s Symmetrc Postve Defnte Any symmetrc postve defnte matr can be regarded as a kernel matr, that s as an nner product Matr n some space

19 The Pont More sophstcated algorthms* and kernels** est, than lnear dscrmnant and polynomal kernels The dea s the same: modular systems, a general purpose learnng module, and a problem specfc kernel functon Learnng Module Kernel Functon f ( ) = αk(, )

20 Algorthms (to gve you an dea) Computaton 2.5 Rdge regresson 30 Computaton 5.14 Regularsed Fsher dscrmnant 131 Computaton 5.15 Regularsed kernel Fsher dscrmnant 133 Computaton 6.3 Mamsng varance 141 Computaton 6.18 Mamsng covarance 154 Computaton 6.30 Canoncal correlaton analyss 163 Computaton 6.32 Kernel CCA 165 Computaton 6.34 Regularsed CCA 169 Computaton 6.35 Kernel regularsed CCA 169 Computaton 7.1 Smallest enclosng hypersphere 193 Computaton 7.7 Soft mnmal hypersphere 199 Computaton 7.10 nu-soft mnmal hypersphere 202 Computaton 7.19 Hard margn SVM 209 Computaton norm soft margn SVM 216 Computaton norm soft margn SVM 223 Computaton 7.40 Rdge regresson optmsaton 229 Computaton 7.43 Quadratc e-nsenstve SVR 231 Computaton 7.46 Lnear e-nsenstve SVR 233 Computaton 7.50 nu-svr 235 Computaton 8.8 Soft rankng 254 Computaton 8.17 Cluster qualty 261 Computaton 8.19 Cluster optmsaton strategy 265 Computaton 8.25 Multclass clusterng 272 Computaton 8.27 Relaed multclass clusterng 273 Computaton 8.30 Vsualsaton qualty 277 Algorthm 5.1 Normalsaton 110 Algorthm 5.3 Centerng data 113 Algorthm 5.4 Smple novelty detecton 116 Algorthm 5.6 Parzen based classfer 118 Algorthm 5.12 Cholesky decomposton or dual Gram Schmdt 126 Algorthm 5.13 Standardsng data 128 Algorthm 5.16 Kernel Fsher dscrmnant 134 Algorthm 6.6 Prmal PCA 143 Algorthm 6.13 Kernel PCA 148 Algorthm 6.16 Whtenng 152 Algorthm 6.31 Prmal CCA 164 Algorthm 6.36 Kernel CCA 171 Algorthm 6.39 Prncpal components regresson 175 Algorthm 6.42 PLS feature etracton 179 Algorthm 6.45 Prmal PLS 182 Algorthm 6.48 Kernel PLS 187 Algorthm 7.2 Samllest hypersphere enclosng data 194 Algorthm 7.8 Soft hypersphere mnmsaton 201 Algorthm 7.11 nu-soft mnmal hypersphere 204 Algorthm 7.21 Hard margn SVM 211 Algorthm 7.26 Alternatve hard margn SVM 214 Algorthm norm soft margn SVM 218 Algorthm 7.32 nu-svm 221 Algorthm norm soft margn SVM 225 Algorthm 7.41 Kernel rdge regresson 229 Algorthm norm SVR 232 Algorthm norm SVR 234 Algorthm 7.51 nu-support vector regresson 236 Algorthm 7.52 Kernel perceptron 237 Algorthm 7.59 Kernel adatron 242 Algorthm 7.61 On-lne SVR 244 Algorthm 8.9 nu-rankng 254 Algorthm 8.14 On-lne rankng 257 Algorthm 8.22 Kernel k-means 269 Algorthm 8.29 MDS for kernel-embedded data 276 Algorthm 8.33 Data vsualsaton 280

21 Kernels (to gve you an dea) Defnton 9.1 Polynomal kernel 286 Computaton 9.6 All-subsets kernel 289 Computaton 9.8 Gaussan kernel 290 Computaton 9.12 ANOVA kernel 293 Computaton 9.18 Alternatve recurson for ANOVA kernel 296 Computaton 9.24 General graph kernels 301 Defnton 9.33 Eponental dfuson kernel 307 Defnton 9.34 von Neumann dfuson kernel 307 Computaton 9.35 Evaluatng dfuson kernels 308 Computaton 9.46 Evaluatng randomsed kernels 315 Defnton 9.37 Intersecton kernel 309 Defnton 9.38 Unon-complement kernel 310 Remark 9.40 Agreement kernel 310 Secton 9.6 Kernels on real numbers 311 Remark 9.42 Splne kernels 313 Defnton 9.43 Derved subsets kernel 313 Defnton 10.5 Vector space kernel 325 Computaton 10.8 Latent semantc kernels 332 Defnton 11.7 The p-spectrum kernel 342 Computaton The p-spectrum recurson 343 Remark Blended spectrum kernel 344 Computaton All-subsequences kernel 347 Computaton Fed length subsequences kernel 352 Computaton Nave recurson for gap-weghted subsequences kernel 358 Computaton Gap-weghted subsequences kernel 360 Computaton Tre-based strng kernels 367 Algorthm 9.14 ANOVA kernel 294 Algorthm 9.25 Smple graph kernels 302 Algorthm All non-contguous subsequences kernel 350 Algorthm Fed length subsequences kernel 352 Algorthm Gap-weghted subsequences kernel 361 Algorthm Character weghtng strng kernel 364 Algorthm Soft matchng strng kernel 365 Algorthm Gap number weghtng strng kernel 366 Algorthm Tre-based p-spectrum kernel 368 Algorthm Tre-based msmatch kernel 371 Algorthm Tre-based restrcted gap-weghted kernel 374 Algorthm Co-rooted subtree kernel 380 Algorthm All-subtree kernel 383 Algorthm 12.8 Fed length HMM kernel 401 Algorthm Par HMM kernel 407 Algorthm Hdden tree model kernel 411 Algorthm Fed length Markov model Fsher kernel 427

22 Eamples of Kernels Smple eamples of kernels are: Kz (, ) = z, d Kz (, ) z = e 2 / 2σ

23 Obtanng Informaton about the Embeddng What do we know about the geometry of the data sample embedded n the feature space? We have only the kernel matr Eg we know all parwse dstances We can know dstance of ponts from centre of mass of a set, and ths can also lead to fsherdscrmnant type of classfers etc etc etc ), ( 2 ), ( ), ( ) ( ), ( 2 ) ( ), ( ) ( ), ( ) ( ) ( 2 2 z K z z K K z z z z + = = + = φ φ φ φ φ φ φ φ

24 Etc etc etc We wll see that we can also know the prncpal aes, we can perform regresson and dscrmnant analyss, as well as factor and cluster analyss All n the kernel-nduced space wthout need to use data coordnates eplctly

25 Fleblty of KMs Ths s a hyperplane! (n some space)

26 Smallest Sphere Quck eample of applcaton: fnd smallest sphere contanng all the data n the embeddng space (a QP problem) Smlarly: fnd small sphere that contans a gven fracton of the ponts (Ta and Dun)

27 Eample 100 QP 100 QP

28 Eample 100 QP 100 QP(lnear)

29 Effect of Kernels 100 QP 100 QP(RBF)

30 Generalzed Egenproblems Many multvarate statstcs algorthms based on generalzed egenproblems can be kernelzed.. PCA CCA FDA Spectral Clusterng (See webste for free code )

31 Generalzed Egenproblems A=λB Wth A, B symmetrc, B nvertble Many classcal multvarate statstcs problems can be reduced to ths But here matrces have same sze as sample

32 Fsher Lnear Dscrmnant In FDA the parameters are chosen so as to optmze a crteron functonal that depends on the dstance between the means of the two populatons beng separated, and on ther varances. More precsely, consder projectng all the multvarate (tranng) data onto a generc drecton w, and then separately observng the mean and the varance of the projectons of the two classes.

33 Mamal separaton If we denote µ + and µ - the means of the projectons of the two classes on the drecton w, and s + and s - ther varances, the cost functon assocated to the drecton 2 w s then defned as ( µ + µ ) C( w) =. FDA selects the drecton w that mamzes ths measure of separaton. s s 2

34 C( w) = ( µ µ ) s s 2 2 Ths mathematcal problem can be rewrtten n the followng way. Defne the scatter matr of the -th class as (where by we denote the mean of a set of vectors) and defne the total wthnclass scatter matr as. In ths way one can wrte the denomnator of the crteron as and smlarly, the numerator can be wrtten as: where S B s the between-class scatter matr. Re-formulaton s s = S = ( class( ) 2 T T ( µ µ ) = w ( )( ) w = w T S w C( w) = S W S B 1 w' S w' S B W = S + S = w w 1 1 B w T S W w )( T ( 1 1)(1 1) ) T

35 Fsher Dscrmnant The drecton that mamzes the separaton measure gven above s the same that mamzes the cost functon: w' S Bw C( w) = w' S w Ths rato (Raylegh quotent) s related to generalzed egenproblems W

36 Another egenproblem In ths way the crteron functon becomes (an epresson known as Reylegh quotent) and ts mamzaton amounts to solvng the generalzed egenproblem. C w' S w w B ( w) = S w = λs w w' SW B w

37 PCA

38 Spectral Clusterng

39 CCA

40 Prncpal Components Analyss Egenvectors of the data n the embeddng space can be used to detect drectons of mamum varance We can project data onto prncpal components by solvng a (dual) egen-problem We wll use ths later for vsualzaton of the embeddng space: projectng data onto a 2-dm plane

41 Kernel PCA Prmal PCA: Dual PCA: Cv v m C T = = = λ 1 0 α λα α α λ φ α λ φ φ φ K m K K m v m C T = = = = = = 2 ) ( ) ( ) ( 1 0 ) ( Cv v The egenvectors can be wrtten as: w=σα Problem: fnd the alphas Schoelkopf et al

42 Dscusson Lke normal PCA, also kernel PCA has the property that the most nformaton (varance) s contaned n the frst prncpal components (projectons on egenvectors) Etc etc We wll use t just to vsualze a slce of the embeddng space, later

43 Kernel CCA Smlarly can be done for CCA (*) (*) was done here n Berkeley! (Bach and Jordan)

44 CCA n pctures Englsh corpus French corpus

45 CCA n pctures

46 CCA n pctures

47 Kernels for Structured Data We wll show how to use these algorthms on nonvector data Kernel functons do not need to be defned over vectors As long as a kernel functon s guaranteed to always gve rse to a symmetrc postve defnte matr, t s far game As an eample, we consder kernels over strngs, to embed tet documents n a space spanned by all k-mers from a gven alphabet

48 logarthm borhythm algorthm rhythm bology competng computaton bocomputng computng

49 Strng Kernels A frst generaton of strng kernels was based on dynamc programmng deas, and had a cost quadratc n the length of the sequences More recent mplementatons, based on structures smlar to Tres and Suff Trees, can acheve lnear complety n the total length of the corpus

50 Suff Tree Kernels For each document a feature vector s constructed ndeed by all possble length-k strngs (k-mer) of the gven alphabet; the value of these entres s equal to the number of tmes ths substrng occurs n the gven tet. how many??? The kernel between two tets s then computed n the usual way, as the nner product of ther correspondng feature vectors. Ths can be done n a hghly effectve way by usng a recursve relaton and the approprate data structures (chrstna lesle)

51 kernel The kernels we use here are the 2- mer kernel, the 4-mer kernel (and gappy 4-mers and wldcard 4-mers, these count also partal matches, n dfferent ways)

52 Features and Embeddng The feature space s that of all k-grams The embeddngs: Eact match (count how many tmes each k-gram appears) Wldcard (count how many tmes, allowng some msmatches, parameters can be used to tune the cost of msmatches) Gappy kernel (allows gaps, parameters tune cost of gaps) Cost of computaton (depends on tolerance, lnear n length of sequence)

53 Puttng t all together We can mplctly/vrtually embed sequence data nto a hgh dmensonal space spanned by all k- grams, and look for lnear relatons n that space We can do factor analyss, correlaton analyss, clusterng, dscrmnaton, regresson and many other thngs ON STRINGS! Let s try wth some tet data (fun eample, not real applcaton)

54 Suff Trees ST are a magc data structure: constructed n lnear tme (*) can then be used as oracles to answer a seres of queres n constant tme (*): Fll an entry of Kernel Matr (for p-spectrum kernels and smlar) Poston, frequency, of words (also wth msmatches at an addtonal cost) (*) n length of the corpus

55 Suff Trees A suff tree of a strng s s a compacted tre of all suffes of s Powerful data structure, many applcatons Typcal way to answer queres: depth frst traversals Constructon: Ukkonen s algorthm s lnear (not smple, can gve you a smple quadratc one)

56 Swss Consttuton The artcles of the Swss consttuton, whch s avalable n 4 languages: Englsh, French, German and Italan. The artcles are dvded nto several parts, called 'Ttles' (n the Englsh translaton). All data can be found onlne at tml.

57 Swss Consttuton A few artcles were omtted n ths case study (some because they do not have an eact equvalent n the dfferent languages, 2 others because they are consderably dfferent n length than the bulk of the artcles), leavng a total 195 artcles per language (few hundreds symbols long) The tets are processed by removng punctuaton and by stemmng. (ths software also onlne)

58 Eamples We used varous sequence kernels on those artcles, as well as classc bag of words kernels We epermented wth dfferent choces of parameters We appled varous algorthms no attempts to optmze performance just demonstratons of feasblty

59 Eamples Frst let s see how kernel matrces look lke (here the data set contans all 4 versons of the consttuton, see just as a set of sequences) (all eperments by Tjl de Be, KU Leuven)

60 Kernel Matr - Bag of Words Languages: Englsh French German Italan

61 Kernel Matrces 2-mer kernel 4-mer kernel

62 Eamples Here we removed the dagonal n order to mprove vsblty Dfferent colors are largely an artfact of matlab Notce that some structure appears n the kernel matr, but not always dstngushable We used several dfferent kernel functons

63 Eamples We played both wth the possblty of dstngushng languages based on countng and comparng k-mers, and wth the possblty of dstngushng topcs (parts of the consttuton known as ttles) We tred frst PCA and CCA

64 Prncpal Components Analyss 2-PCA on the total sample. 2mer kernel. Colored later accordng to language. Only german stands out, for ths smple kernel

65 Prncpal Components Analyss 2-PCA on the total sample. 4mer kernel. Colored later accordng to language. Better kernel, Better embeddng

66 Prncpal Components Analyss 2-PCA on the total sample. Bag of Words kernel. As epected, words are better features than 4-grams Compare german wth talan

67 Semantc Informaton - englsh: dstngushng Ttles (chapters) PCA projecton of englsh artcles Usng BOW. Colored later Accordng to Chapter (topc)

68 Mult-way kcca One can defne Canoncal Correlaton Analyss among more than 2 spaces (*) Much more supervsed choce of projecton We have chosen the regularzaton parameters to emphasze dfference from randomzed data Projected, then colored accordng to chapter (*) Bach and Jordan

69 Projectng on frst 2 englsh components of 4- way kcca on BOW Projecton on 2 CCA components Colored by chapter CCA performed on all 4 languages

70 Other eamples I have done smlar work last year on canadan parlament data, sgnfcant semantc relatons were etracted across french and englsh We then tred varous methods of dscrmnant and cluster analyss wth these same kernels and data see all of them onlne (fsher vs svm) (k-means vs spectral clusterng)

71 Summary of classfcaton results Englsh vs French Englsh vs German Englsh vs Italan SVM 3.3%±0.2.12% ± % ±0.2 FDA 5.0% ±0.2 0 ±0 1.2% ±0.1 Nose free classfcaton error rates, averaged over 100 randomzatons wth balanced 80/20 splts. The 2-mer kernel s used. NOTE: when supervsed learnng s used on all dmensons, task s easy

72 Other kernels for structured data Kernels can be defned based on Probablstc Graphcal Models Margnalzaton kernels and Fsher kernels allow us to nsert sgnfcant doman knowledge n the embeddng

73 Scalng thngs up We processed 800 sequences, of 300 symbols I would lke to do sequences of 3000 symbols. Can we?

74 Statstcal Aspects All these methods requre very careful regularzaton Our models nvolve use of Rademacher Complety and other unform convergence arguments Have papers on reducng problems to egenproblems Have papers on statstcal stablty and generalzaton power of egenproblems

75 Combnng Heterogeneous Data An mportant problem s to combne dfferent data sources, or dfferent representatons of the same data (eg: n bonformatcs: combne sequence nformaton wth gene epresson nformaton and proten-nteracton )

76 Conclusons Kernel methods a fun set of methods for fndng varous types of patterns on structured data Allow us to apply well known set of statstcal procedures to a whole new set of problems Potental to scale up to massve sample szes, for the frst tme n pattern recognton Can use Graphcal Models as kernels easly

77 Conclusons If you want to fnd out more, here webstes and books SUPPORT-VECTOR.NET (crstann and shawe-taylor) Fnally: New book on Kernel methods for pattern analyss CUP Press 2004 KERNEL-METHODS.NET