Text Mining with R. Rob Zinkov. October 19th, Rob Zinkov () Text Mining with R October 19th, / 38

Transcription

1 Text Mining with R Rob Zinkov October 19th, 2010 Rob Zinkov () Text Mining with R October 19th, / 38

2 Outline 1 Introduction 2 Readability 3 Summarization 4 Topic Modeling 5 Sentiment Analysis 6 Entity Extraction 7 Demo Rob Zinkov () Text Mining with R October 19th, / 38

3 What is Text Mining? Introduction Text mining is any process or program that: Raw human written text Structured information Rob Zinkov () Text Mining with R October 19th, / 38

4 R is very good for this Introduction Rob Zinkov () Text Mining with R October 19th, / 38

5 Introduction Themes R is a great glue language CRAN already has a lots of packages that work well together Rob Zinkov () Text Mining with R October 19th, / 38

6 Introduction Caveats I use outside libraries more than necessary Many of these algorithms could be written completely in R There are nicer ways to integrate these libraries Text mining is a vast field that can t be covered in 40 minutes Rob Zinkov () Text Mining with R October 19th, / 38

7 Readability Readability Rob Zinkov () Text Mining with R October 19th, / 38

8 Readability Readability gives us an idea of the difficulty of the document It also gives a rough measure of the quality Rob Zinkov () Text Mining with R October 19th, / 38

9 Readability Flesch-Kincaid readability test Readability can be roughly measured with ( w ) ( y ) s w where w = total words y = total syllables s = total sentences Rob Zinkov () Text Mining with R October 19th, / 38

10 Readability Score Notes easily understandable by an average 11-year-old student easily understandable by 13- to 15-year-old students best understood by university graduates Rob Zinkov () Text Mining with R October 19th, / 38

11 Readability Flesch-Kincaid Reading Age (0.39 ASL) + (11.8 ASW ) where ASL = average sentence length where ASW = average syllables per word Rob Zinkov () Text Mining with R October 19th, / 38

12 Readability Rob Zinkov () Text Mining with R October 19th, / 38

13 Readability system(paste("java -jar CmdFlesh.jar","test_review.txt")) Rob Zinkov () Text Mining with R October 19th, / 38

14 Readability Demo! Rob Zinkov () Text Mining with R October 19th, / 38

15 Notes Readability This algorithm isn t hard to implement. Quick trick. Count vowel clusters in words to estimate syllables Rob Zinkov () Text Mining with R October 19th, / 38

16 Summarization Summarization is about a succient distinct distillation of the relevant content in a document. Rob Zinkov () Text Mining with R October 19th, / 38

17 Summarization Rob Zinkov () Text Mining with R October 19th, / 38

18 Summarization Most approaches involve selecting out the most relevant sentences The simplest technique is to just look for sentences with popular terms Rob Zinkov () Text Mining with R October 19th, / 38

19 Summarization I use libots, as an example but there is much better work Rob Zinkov () Text Mining with R October 19th, / 38

20 Summarization Demo! Rob Zinkov () Text Mining with R October 19th, / 38

21 Topic Modeling Topic Modeling Topic Modeling is a way to group and categorize documents Usually unsupervised approach Rob Zinkov () Text Mining with R October 19th, / 38

22 Topic Modeling Topic Modeling - continued CRAN includes a package for topicmodeling This package using LDA and CTM Rob Zinkov () Text Mining with R October 19th, / 38

23 Topic Modeling LDA - Latent Dirchilet Allocation θ k j D[α] φ w k D[β] z ij θ k j x ij φ w zij Rob Zinkov () Text Mining with R October 19th, / 38

24 Topic Modeling n jkw = #{i : x ij = w, z ij = k} Rob Zinkov () Text Mining with R October 19th, / 38

25 Topic Modeling CTM - Coorelated Topic Models Rob Zinkov () Text Mining with R October 19th, / 38

26 Topic Modeling CTM - Coorelated Topic Models θ k j log(n(µ, Σ)) φ w k D[β] z ij θ k j x ij φ w zij Rob Zinkov () Text Mining with R October 19th, / 38

27 Topic Modeling Demo! Rob Zinkov () Text Mining with R October 19th, / 38

28 Sentiment Analysis Rob Zinkov () Text Mining with R October 19th, / 38

29 Sentiment Analysis Sentiment analysis is about gauging mood based on the text. Rob Zinkov () Text Mining with R October 19th, / 38

30 Sentiment Analysis Opinion corpus available at: Wiebe s corpora Sentiwordnet: Rob Zinkov () Text Mining with R October 19th, / 38

31 Sentiment Analysis For more sophistication Best solved using a Conditional Random Field This area is still new No R libraries Entity Extraction needed for more fine-grained sentiment Rob Zinkov () Text Mining with R October 19th, / 38

32 Sentiment Analysis Demo! Rob Zinkov () Text Mining with R October 19th, / 38

33 Entity Extraction Named Entity Recognition The purpose of NER is to extract out and label phrases in a sentence Bill Clinton arrived at the United Nations Building in Manhattan. Rob Zinkov () Text Mining with R October 19th, / 38

34 Entity Extraction Challenges People and locations may be referred to in ambiguous ways. Entity may never have been seen before Entity may be referred to with pronouns Wikipedia and Capitalization heuristics aren t good enough Rob Zinkov () Text Mining with R October 19th, / 38

35 Entity Extraction I use the Illinois Named Entity Extractor view/4 Rob Zinkov () Text Mining with R October 19th, / 38

36 Entity Extraction Demo! Rob Zinkov () Text Mining with R October 19th, / 38

37 Conclusions Demo There are lots of interesting things you can do with text mining R is very good at integrating all of them. Rob Zinkov () Text Mining with R October 19th, / 38

38 Demo Questions? Rob Zinkov () Text Mining with R October 19th, / 38