Decision Trees and Random Forests. Reference: Leo Breiman,

Transcription

1 Decision Trees and Random Forests Reference: Leo Breiman, 1. Decision trees Example (Guerts, Fillet, et al., Bioinformatics 2005): Patients to be classified: normal vs. diseased

2 Decision trees Classification of biomarker data: large number of values (e.g., microarray or mass spectrometry analysis of biological sample)

3 Decision trees Mass spectrometry parameters or gene expression parameters (around 15k values) Given new patient with biomarker data, is s/he normal or ill?

4 Decision trees Needed: selection of relevant variables from many 8 x 3 3 3œ" Number 8 of known examples in H œ ÖÐ ßC Ñ is small (characteristic of data mining problems) Assume we have for each biological sample a feature vector x, and will classify it: diseased: Cœ"à normal: Cœ "Þ Goal: find function 0ÐxÑ C which predicts C from x.

5 Decision trees How to estimate error of 0ÐxÑ and avoid over-fitting the small dataset H? Use cross-validation to test predictor part of the sample HÞ 0ÐxÑ in an unexamined For biological sample, feature vector x œðbßáßb ". Ñ consists of features ( or biomarkers or attributes) B3 œ E3 describing the biological sample from which x is obtained.

6 The decision tree approach Decision tree approach to finding predictor 0ÐxÑ œ C on data set H: based Š form a tree whose nodes are attributes B3 œ E3 in x Š decide which attributes E3 to look at first in predicting C from x find those with highest information gain - place these at top of tree Š then use recursion to form sub-trees based on attributes not used in the higher nodes:

7 The decision tree approach Advantages: interpretable, easy to use, scalable, robust

8 Decision tree example Example 1 (Moore): UCI data repository ( MPG ratings of cars: Goal: predict MPG rating of a car from a set of attributes E 3

9 Decision tree example Examples (each row is attribute set for a sample car): R. Quinlan

10 Decision tree example Simple assessment of information gain: how much does a particular attribute E 3 help to classify a car with respect to MPG?

11 Decision tree example Begin the decision tree: start with most informative

12 Decision tree example criterion, cylinders:

13 Decision tree example Recursion: build next level of tree. Initially have: Now build sub-trees: split each set of cylinder numbers into

14 Decision tree example further groups- Resulting next level:

15 Final tree: Decision tree example

16 Decision tree example

17 Points: Decision tree example è Don't split node if all records have same value (e.g. cylinders œ'ñ è Don't split node if can't have more than 1 child (e.g. acceleration œ medium)

18 Pseudocode: Program Tree(Input, Output) If all output values are the same, then return leaf (terminal) node which predicts the unique output If input values are balanced in a leaf node (e.g. 1 good, 1 bad in acceleration) then return leaf predicting majority of outputs on same level (e.g. bad in this case) Else find attribute with highest information gain E 3 If attribute at current node has values E 7 3

19 then Return internal (non-leaf) node with 7 children Build child 3 by calling Tree(NewIn, NewOut), where NewIn œ values in dataset consistent with value E 3 and all values above this node

20 Another decision tree: prediction of wealth from census data (Moore):

21 Prediction of age from census:

22 Prediction of gender from census: A. Moore

23 2. Important point: always cross-validate It is important to test your model on new data (test data) which are different from the data used to train the model (training data). This is cross-validation. Cross-validation error 2% is good; 40% is poor.

24 3. Background: mass spectroscopy What does a mass spectrometer do? 1. It measures masses of molecules better than any other technique. 2. It can give information about chemical structures of molecules.

25 Mass spectroscopy How does it work? 1. Takes unknown molecule Mß adds 3 protons to it giving it charge 3(forming MH Ñ 2. Accelerates ion MH 3 in known electric field I. 3. Measures time of flight along a known distance H. 4. Time X of flight is inversely proportional to electric charge 3 and proportional to mass 7 of ion. 3

26 Thus Mass spectroscopy Xº3Î7 So mass spectrometer measures ratio of charge 3 (also known as D) and 7, i.e., 3Î7 œ DÎ7Þ With a large number of molecules in a biosample, this gives a spectrum of DÎ7 values, which allows identification of molecules in sample (here IgG œ immunoglobin G)

27 Mass spectroscopy

28 Mass spectroscopy What are the measurements good for? To identify, verify, and quantify: metabolites, proteins, oligonucleotides, drug candidates, peptides, synthetic organic chemicals, polymers Applications of Mass Spectrometry Biomolecule characterization Pharmaceutical analysis Proteins and peptides Oligonucleotides

29 Mass spectroscopy How does a mass spectrometer work?

30 Mass spectroscopy [Source: Sandler Mass Spectroscopy]

31 Two types of ionization: Mass spectroscopy 1. Electrospray ionization (ESI):

32 Mass spectroscopy SMS

33 3 Mass spectroscopy [above MH denotes molecule with 3 protons (H ) attached]

34 2. MALDI: Mass spectroscopy SMS

35 Mass spectroscopy Mass analyzers separate ions based on their mass-tocharge ratio (m/z) Operate under high vacuum Measure mass-to-charge ratio of ions (m/z)

36 Components: Mass spectroscopy 1. Quadrupole Mass Analyzer (filter) Uses a combination of RF and DC voltages to operate as a mass filter before masses are accelerated. Has four parallel metal rods. Lets one mass pass through at a time. Can scan through all masses or only allow one fixed mass.

37 Mass spectroscopy

38 Mass spectroscopy 2. Time-of-flight (TOF) Mass Analyzer Accelerates ions with electric field, detects them, analyzes flight time.

39 Mass spectroscopy

40 Mass spectroscopy Ions are formed in pulses. The drift region is field free. Measures the time for ions to reach the detector. Small ions reach the detector before large ones. Ion trap mass analyzer:

41 Mass spectroscopy

42 Mass spectroscopy 3. Detector: Ions are detected with a microchannel plate:

43 Mass spectroscopy Mass spectrum shows the results: ESI Spectrum of Trypsinogen (MW 23983)

44 Mass spectroscopy

45 Mass spectroscopy 4. Dimensional reduction (G. Izmirlian): Sometimes we perform a dimension reduction by reducing mass spectrum information of human subject 3 to store only peaks:

46 Mass spectroscopy

47 Mass spectroscopy Then have (compressed) peak information in feature vector x œðbßáßb ". Ñ, >2 with B5 œ location of 5 mass spectrum peak (above a fixed threshold). Compressed or not, outcome value to feature vector subject 3 is C œ "Þ 3 x 3 for

48 5. Random forest example Example (Guerts, et al.): Normal/sick dichotomy for RA and for IBD (above - Geurts, et al.): we now build a forest of decision trees based on differing attributes in the nodes:

49 Random forest application For example: Could use mass spectroscopy data to determine disease state

50 Random forest application Mass Spec segregates protein and other molecules through spectrum of 7ÎD ratios ( 7 œ mass; D œ charge).

51 Random forest application Geurts, et al.

52 Random Forests: Advantages: accurate, easy to use (Breiman software), fast, robust Disadvantages: difficult to interpret More generally: How to combine results of different predictors (e.g. decision trees)? Random forests are examples of ensemble methods, which combine predictions of weak classifiers :Ð Ñ. 3 x

53 Ensemble methods: observations 1. Boosting: As seen earlier, take linear combination of predictions :Ð 3 xñ by classifiers 3(assume these are decision trees) 0ÐxÑ œ " + 3: 3 ÐxÑ, (1) if tree predicts illness where :Ð Ñœ " 3 3 x œ, " otherwise >2 and predict Cœ" if 0ÐxÑ! and Cœ " if 0ÐxÑ!Þ 3

54 Ensemble methods: observations 2. Bagging: Take a vote: majority rules (equivalent in this case to setting + 3 œ " for all 3 in (1) above). Example of a Bagging algorithm is random forest, where a forest of decision trees takes a vote.

55 General features of a random forest: If original feature vector x has. features E ßáßE,. ". Each tree uses a random selection of 7 È. features 7 ÖE 3 4œ" chosen from all features E", E# ß á, E. ; 4 the associated feature space is different for each tree and denoted by Jß"Ÿ5ŸOœ 5 # trees. (Often O œ # trees is large; e.g., O œ &!!ÑÞ For each split in a tree node based on a given variable choose the variable from information content. E 3

56 Information content in a node To compute information content of a node: Assume input set to node is W: then information content of node R is

57 Information content in a node MÐRÑ œ lwl LÐWÑ lw l LÐW Ñ lw l LÐW Ñß P P V V where lwl œ input sample size; lwpßvl œ size of leftß right subclasses of W LÐWÑ œ Shannon entropy of W œ ": log : 3œ " 3 # 3 with : œ TÐG s lwñ œ proportion of class G in sample W

58 Information content in a node [later we will use Gini index, another criterion] Thus LÐWÑ œ "variablity" or "lack of full information" in the probabilities : 3 forming sample W input into current node R. MÐRÑ œ "information from node R" Þ For each variable E3, average over all nodes R in all trees involving this variable to find average information content LavÐE3Ñof E3.

59 Information content in a node (a) Rank all variables E 3 according to information content (b) For each fixed 8" 8 use only the first 8" variables. Select which minimizes prediction error. 8 "

60 Information content in a node Geurts, et al.

61 Application to: Random forests: application ì early diagnosis of Rehumatoid arthritis ì rapid diagnosis of inflammatory bowel diseases (IBD) 3 patient groups (University Hospital of Liege):

62 Random forests: application RA IBD Disease patients Negative controls Inflammatory controls Total Mass spectra obtained by SELDI-TOF mass spectrometry on chip arrays:

63 Random forests: application ì Hydrophobic (H4) ì weak cation-exchange (CM10) ì strong cation-exchange (Q10)

64 Random forests: application Feature vectors: x J consists of about 15,000 values in each case. Effective dimension reduction method: Discretize horizontally and vertically to go from 15,000 to 300 variables

65 Random forests: application

66 Random forests: application Sensitivity and specificity:

67 Random forests: application Accuracy measures: DT=Decision tree; RF=random forest; 5NN = 5-nearest neighbors; Note on sensitivity and specificity: use confusion matrix Actual Condition Test outcome True False Positive TP FP Negative FN TN

68 Sensitivity œ Random forests: application XT XT œ XT JR Total positives Specificity œ XR XR JT œ XR Total negatives Positive predictive value œ XT XT JT œ XT Total predicted positives

69 Random forests: application Variable ranking on the IBD dataset: 10 most important variables in spectrum:

70 Random forests: application RF-based (tree ensemble) - based variable ranking vs. variable ranking by individual variable : values:

71 Random forests: application RA IBD

72 6. RF software: Spider: html Leo Breiman: software.htm WEKA machine learning software