Big Data Analytics Clustering and Classification

Transcription

1 E6893 Big Data Analytics Lecture 4: Big Data Analytics Clustering and Classification Ching-Yung Lin, Ph.D. Adjunct Professor, Dept. of Electrical Engineering and Computer Science IBM Distinguished Researcher and Chief Scientist, Graph Computing September 29th,

2 Review Key Components of Mahout 2

3 Machine Learning example: using SVM to recognize a Toyota Camry Non-ML Rule 1.Symbol has something like bull s head Rule 2.Big black portion in front of car. Rule 3...???? ML Support Vector Machine Feature Space Positive SVs Negative SVs CY Lin, Columbia University

4 Machine Learning example: using SVM to recognize a Toyota Camry ML Support Vector Machine Positive SVs PCamry > 0.95 Feature Space Negative SVs CY Lin, Columbia University

5 Clustering 5

6 Clustering on feature plane 6

7 Clustering example 7

8 Steps on clustering 8

9 K-mean clustering 9

10 Making initial cluster centers 10

11 Parameters to Mahout k-mean clustering algorithm 11

12 HelloWorld clustering scenario 12

13 HelloWorld Clustering scenario - II 13

14 HelloWorld Clustering scenario - III 14

15 HelloWorld clustering scenario result 15

16 Testing difference distance measures 16

17 Manhattan and Cosine distances 17

18 Tanimoto distance and weighted distance 18

19 Results comparison 19

20 Data preparation in Mahout vectors 20

21 vectorization example 0: weight 1: color 2: size 21

22 Mahout codes to create vectors of the apple example 22

23 Mahout codes to create vectors of the apple example II 23

24 Vectorization of text Vector Space Model: Term Frequency (TF) Stop Words: Stemming: 24

25 Most Popular Stemming algorithms 25

26 Term Frequency Inverse Document Frequency (TF-IDF) The value of word is reduced more if it is used frequently across all the documents in the dataset. or 26

27 n-gram It was the best of time. it was the worst of times. ==> bigram Mahout provides a log-likelihood test to reduce the dimensions of n-grams 27

28 Examples using a news corpus Reuters dataset: 22 files, each one has 1000 documents except the last one. reuters21578/ Extraction code: 28

29 Mahout dictionary-based vectorizer 29

30 Mahout dictionary-based vectorizer II 30

31 Mahout dictionary-based vectorizer III 31

32 Outputs & Steps 1. Tokenization using Lucene StandardAnalyzer 2. n-gram generation step 3. converts the tokenized documents into vectors using TF 4. count DF and then create TF-IDF 32

33 A practical setting of flags 33

34 normalization Some documents may pop up showing they are similar to all the other documents because it is large. ==> Normalization can help. 34

35 Clustering methods provided by Mahout 35

36 K-mean clustering 36

37 Hadoop k-mean clustering jobs 37

38 K-mean clustering running as MapReduce job 38

39 Hadoop k-mean clustering code 39

40 The output 40

41 Canopy clustering to estimate the number of clusters Tell what size clusters to look for. The algorithm will find the number of clusters that have approximately that size. The algorithm uses two distance thresholds. This method prevents all points close to an already existing canopy from being the center of a new canopy. 41

42 Running canopy clustering Created less than 50 centroids. 42

43 News clustering code 43

44 News clustering example > finding related articles 44

45 News clustering code II 45

46 News clustering code III 46

47 Other clustering algorithms Hierarchical clustering 47

48 Different clustering approaches 48

49 When to use Mahout for classification? 49

50 The advantage of using Mahout for classification 50

51 Classification definition 51

52 How does a classification system work? 52

53 Key terminology for classification 53

54 Input and Output of a classification model 54

55 Four types of values for predictor variables 55

56 Sample data that illustrates all four value types 56

57 Supervised vs. Unsupervised Learning 57

58 Work flow in a typical classification project 58

59 Classification Example 1 Color-Fill 59 Position looks promising, especially the x-axis ==> predictor variable. Shape seems to be irrelevant. Target variable is color-fill label.

60 Target leak A target leak is a bug that involves unintentionally providing data about the target variable in the section of the predictor variables. Don t confused with intentionally including the target variable in the record of a training example. Target leaks can seriously affect the accuracy of the classification system. 60

61 Classification Example 2 Color-Fill (another feature) 61

62 Mahout classification algorithms Mahout classification algorithms include: Naive Bayesian Complementary Naive Bayesian Stochastic Gradient Descent (SDG) Random Forest 62

63 Comparing two types of Mahout Scalable algorithms 63

64 Step-by-step simple classification example 1.The data and the challenge 2.Training a model to find color-fill: preliminary thinking 3.Choosing a learning algorithm to train the model 4.Improving performance of the classifier 64

65 Classification Example 3 65

66 What may be a good predictor? 66

67 Choose algorithm via Mahout 67

68 Stochastic Gradient Descent (SGD) 68

69 Characteristic of SGD 69

70 Support Vector Machine (SVM) maximize boundary distances; remembering support vectors 70 nonlinear kernels

71 Naive Bayes Training set: Classifier using Gaussian distribution assumptions: Test Set: 71 ==> female

72 Random Forest Random forest uses a modified tree learning algorithm that selects, at each candidate split in the learning process, a random subset of the features. 72

73 Choosing a learning algorithm to train the model One low overhead classification method is the stochastic gradient descent (SGD) algorithm for logistic regression. This algorithm is sequential, but it s fast. 73

74 The donut.csv data file in Example 3 74

75 Build a model using Mahout 75

76 Trainlogistic program 76

77 Evaluate the model AUC (0 ~ 1): 1 perfect 0 perfectly wrong 0.5 random confusion matrix 77

78 Questions? 78