Text Analysis for Big Data. Magnus Sahlgren

Transcription

1 Text Analysis for Big Data Magnus Sahlgren

2

3

4 Data Size Style (editorial vs social) Language (there are other languages than English out there!)

5 Data Size Style (editorial vs social) Language (there are other languages than English out there!)

6 Data Size Style (editorial vs social) Language (there are other languages than English out there!)

7 Data Size Style (editorial vs social) Language (there are other languages than English out there!)

8 Data Size Style (editorial vs social) Language (there are other languages than English out there!)

9 Data Size Style (editorial vs social) Language (there are other languages than English out there!)

10 Data Size Style (editorial vs social) Language (there are other languages than English out there!)

11

12 Technologies Knowledge-based (use resources like Wikipedia) Supervised machine learning (use annotated data) Unsupervised machine learning (use unstructured data)

13 Technologies Knowledge-based (use resources like Wikipedia) Supervised machine learning (use annotated data) Unsupervised machine learning (use unstructured data)

14 Technologies Knowledge-based (use resources like Wikipedia) Supervised machine learning (use annotated data) Unsupervised machine learning (use unstructured data)

15 Technologies Knowledge-based (use resources like Wikipedia) Supervised machine learning (use annotated data) Unsupervised machine learning (use unstructured data)

16 Technologies Knowledge-based (use resources like Wikipedia) Supervised machine learning (use annotated data) Unsupervised machine learning (use unstructured data)

17 Technologies Knowledge-based (use resources like Wikipedia) Supervised machine learning (use annotated data) Unsupervised machine learning (use unstructured data)

18 Technologies Knowledge-based (use resources like Wikipedia) Supervised machine learning (use annotated data) Unsupervised machine learning (use unstructured data)

19 Semantic memories

20 Semantic memories (systems that learn language by reading large amounts of text)

21 Semantic memories (systems that learn language by reading large amounts of text)

22

23 Insights Identify and extract items (e.g. entities and events) Find relations (e.g. synonyms and associations) Compress and refine the information (e.g. summarization and topic detection) Measure things (e.g. attitudes and opinions)

24 Insights Identify and extract items (e.g. entities and events) Find relations (e.g. synonyms and associations) Compress and refine the information (e.g. summarization and topic detection) Measure things (e.g. attitudes and opinions)

25 Insights Identify and extract items (e.g. entities and events) Find relations (e.g. synonyms and associations) Compress and refine the information (e.g. summarization and topic detection) Measure things (e.g. attitudes and opinions)

26 Insights Identify and extract items (e.g. entities and events) Find relations (e.g. synonyms and associations) Compress and refine the information (e.g. summarization and topic detection) Measure things (e.g. attitudes and opinions)

27 Insights Identify and extract items (e.g. entities and events) Find relations (e.g. synonyms and associations) Compress and refine the information (e.g. summarization and topic detection) Measure things (e.g. attitudes and opinions)

28 Find relations

29 Find relations

30 Find relations (lexicon.gavagai.se)

31 Insights Identify and extract items (e.g. entities and events) Find relations (e.g. synonyms and associations) Compress and refine the information (e.g. summarization and topic detection) Measure things (e.g. attitudes and opinions)

32 Insights Identify and extract items (e.g. entities and events) Find relations (e.g. synonyms and associations) Compress and refine the information (e.g. summarization and topic detection) Measure things (e.g. attitudes and opinions)

33 Insights Identify and extract items (e.g. entities and events) Find relations (e.g. synonyms and associations) Compress and refine the information (e.g. summarization and topic detection) Measure things (e.g. attitudes and opinions)

34 Compress and refine Summarization and topic detection

35 Compress and refine (monitor.gavagai.se) Summarization and topic detection

36 Compress and refine (monitor.gavagai.se) Summarization and topic detection

37 Compress and refine (monitor.gavagai.se) Summarization and topic detection

38 Insights Identify and extract items (e.g. entities and events) Find relations (e.g. synonyms and associations) Compress and refine the information (e.g. summarization and topic detection) Measure things (e.g. attitudes and opinions)

39 Insights Identify and extract items (e.g. entities and events) Find relations (e.g. synonyms and associations) Compress and refine the information (e.g. summarization and topic detection) Measure things (e.g. attitudes and opinions)

40 Insights Identify and extract items (e.g. entities and events) Find relations (e.g. synonyms and associations) Compress and refine the information (e.g. summarization and topic detection) Measure things (e.g. attitudes and opinions)

41 Measure Sentiment analysis

42 Measure Sentiment analysis Positivity vs negativity wrt the global economy in English online media

43 Measure Sentiment analysis Worry wrt the global economy in English online media

44 Measure Sentiment analysis Negativity towards China in English online media

45 Measure Sentiment analysis Attitude towards Sweden in Russian online media

46 Measure Predict

47 Measure Predict Rönnqvist & Sarlin (2015): Detect & Describe: deep learning of bank stress in the news