Introduction to Big Data Science

Size: px

Start display at page:

Download "Introduction to Big Data Science"

Christal Richard
8 years ago
Views:

1 Introduction to Big Data Science 13 th Period Project: Situation Awareness and Statistical Analysis On Big Data Big Data Science 1

2 Contents What is Situation Awareness (SA)? 3 Levels for SA Role of Data Mining and Reasoning in SA Extracting Information from Big Data Entire Scenario of SA on Facebook Data Big Data Science 2

3 Awareness The goal of computational awareness: to realize awareness in computing machines Awareness is the ability to perceive, to feel, or to be conscious of events, objects or sensory patterns. Big Data Science 3

the ability to perceive, to feel, or to be conscious

4 Situation Awareness Situation awareness is the perception of environmental elements with respect to time and/or space, the comprehension of their meaning, and the projection of their status in the near future after some variable has changed. (Mica Endsley, Wikipedia). Big Data Science 4

comprehension of their meaning, and the projection of their status in

5 JDL: Data Fusion Levels A. Steinberg, et al., Rethinking the JDL Data Fusion Levels Big Data Science 5

6 Sources of SA Information M.R. Endsley, Theoretical Underpinnings of SA: A Critical Review Big Data Science 8

7 Mechanisms and Processes in SA M.R. Endsley, Theoretical Underpinnings of SA: A Critical Review Big Data Science 9

8 Provenance Endsley s Model Semantic Analysis thematic Spatio-Temporal trust Relate Situation Entities Identify Situation Entities Collect Relevant Data M. Kokar, et al., Ontology-based Situation Awareness* (Modified Figure by A. Sheth) Big Data Science 12

Situation Entities Collect Relevant Data M. Kokar, et al.

9 Three layers for situation awareness Big Data Science 13

10 A novel architecture for active situation awareness Image processing and pattern recognition, data mining, signal processing in computer technology can be applied to perception layer to recognize low level objects and data patterns. Situation awareness is inferring some conclusion from observation in the perception layer. Ontologybased rules are usually used for comprehension. The top layer is for projection, which anticipates future events and their implications. Big Data Science 14

Situation awareness is inferring some conclusion from observation in the perception layer.

11 A novel architecture for active situation awareness Projection recommendtoparticipate TheEvent(Building, Event) needreplyto (ITM) checkhisevent (ITM) Comprehension (Situation) givehottopic (ITM,ATopicHisBlog) hasevent (Building, Event) israre(event) saycelebration (ITM, myblog) Perception Stand (People, Longline) isat (People, Building) Wrote (ITM, myblog) needreplyto (ITM) World Facebook Twitter Google Web Data Service Big Data Science 15

hasevent (Building, Event) israre(event) saycelebration (ITM, myblog) Perception Stand (People, Longline) isat

Document Processing Classification (TF-IDF) Event Information Extraction

12 Perceptions by mining SNS data Active Situation Awareness Ontology for Comprehension at Upper Layer Latent Query for SA (Time, Space, Theme) Document Processing Classification (TF-IDF) Event Information Extraction Perception Information Documents Twitter Facebook Data SNS, Web Data Services Big Data Science 16

13 Perception by mining SNS data Select data set to extract information to be used in comprehension layer. The information can be modeled by Web APIs to provide facts to rule engine. For example, we have analyzed the Facebook user s sentences by data mining technique to catch use s intension or changes in mind. There are various data and information set for each layer. Big Data Science 17

For example, we have analyzed the Facebook user s sentences by data mining technique to catch

14 Ontology for Comprehension of the information Big Data Science 18

15 Comprehension of the information by inference of ontology and rule %% Cafeteria Event Inference %% Rules %%longlinestand(human) :- stand(human), long(human). mayhaveevent(place) :- longlinestand(human), areat(human, Place). hasevent(place,event) :- mayhaveevent(place), foundevent(place, Event). recommendtoparticipatetheevent(place, Event) :- hasevent(place,event), israre(event). %% Facts longlinestand(students). areat(students, cafeteria). foundevent(cafeteria, sinsobamatsuri). israre(sobamatsuri). Big Data Science 19

hasevent(place,event) :- mayhaveevent(place), foundevent(place, Event).

16 ASA System Architecture on SNS Smart Phone Inference Engine Facts RESTful Services for Perception FaceBook Service Rules Mapping Ontologies Domain Ontologies Twiter Service Web Data Service Big Data Science 20

FaceBook Service Rules Mapping Ontologies Domain

17 Scenarios Scenario I A student in our university bought a lunch box because he saw a long waiting line in the university cafeteria. But he didn t know it was the waiting line for new soba festival in the cafeteria. If he got the information about the new soba festival from his smart phone when he was near to the cafeteria, he would have chosen the soba. Scenario II, III When I was in my office, a student came in. When I shake my smart phone, the phone tells me the followings about the student based on information on the Facebook: (Example) - The Opponent's Name: Leo Saito - He has interest to me - Saito has Events (Part Time Job, Date) - Saito has changed his topic from food to research Big Data Science 21

If he got the information about the new soba festival from his smart phone when he was near to the cafeteria, he would have chosen the soba.

18 Mining SNS Data (By TF-IDF for Perception layer) Function: Category_calculate{//calculate category of a writing Input: word // set of words that are split Output: category //category of words set Data = learning data set for i = 1 to n {// n = number of word in words set calculate IDF i = log 2 (number of all document in Data / number of word i containing document in Data )} //IDF i = IDF value of word i for i = 1 to n {// n = number of word in words set for j = 1 to m { // m = number of data of Data set calculate TF ij =(frequency of word i in Data j / number of all word i in Data j ) calculate TFIDF ij = Tf ij * IDF i }} for j = 1 to m { // m = number of data of Data set calculate Sum_of_TFIDF j = sumof TFIDF 1j, TFIDF 2j,TFIDF nj if Max_Sum_of_TFIDF < Sum_of_TFIDF j { category=category of Data j }} return category } Function: determine the difference between the two categories{ Input: writing1, writing2 //writing is document set Output: true or false //If accordance -> true, Else -> false for i = i to n {//n = number of document in writing 1 Category_calculate(writing1 i ) } category_of_writing1 = most common category of document in writing1 for i = j to m {//m = number of document in writing 2 Category_calculate(writing2 j ) } category_of_writing2 = most common category of document in writing2 if category_of_writing1 = category_of_writing2 return false else return true } Big Data Science 22

$) calculate TFIDF ij = Tf ij * IDF i }} for j = 1 to m { // m = number of data of Data set calculate Sum_of_TFIDF j = sumof TFIDF 1j, TFIDF 2j,TFIDF nj if Max_Sum_of_TFIDF < Sum_of_TFIDF j {$

19 Ontology for SA (Example 2) Big Data Science 23

20 Rules for SA (Example 2) 1) ITM wantsmyreply(itm) :- wrote(itm, myblog) and thereis(questionmark,hiswriting). enjoyme(itm) :- wrotenumbermorethan(itm, myblog, threshold). givehottopic(itm,atopichisblog) :- wrote(itm, ATopicHisBlog) and therearerepliesmorethan(atopichisblog, threshold). givegoodevaluation(itm, ATopicHisBlog) :- wrote(itm, ATopicHisBlog) and therearegoodrepliesmorethan(atopichisblog, threshold). saycelebration(itm, myblog) :- wrote(itm, myblog) and thereis(celebration, myblog). havenewevent(itm) :- wrote(itm, hiseventblog). * Example of Upper Level Factor or Situation needreplyto(itm) :- wantsmyreply(itm) and saycelebration(itm, myblog) adn enjoyme(itm). checkhisevent(itm) :- havenewevent(itm) and givehottopic(itm, ATopicHistBlog). 2) MC wantsmyreply(mc) :- wrote(mc, myblog) and thereis(questionmark,hiswriting). enjoyme(mc) :- wrotenumbermorethan(mc, myblog, threshold). givehottopic(mc,atopichisblog) :- wrote(mc, ATopicHisBlog) and therearerepliesmorethan(atopichisblog, threshold). givegoodevaluation(mc, ATopicHisBlog) :- wrote(mc, ATopicHisBlog) and therearegoodrepliesmorethan(atopichisblog, threshold). saycelebration(mc, myblog) :- wrote(mc, myblog) and thereis(celebration, myblog). havenewevent(mc) :- wrote(mc, hiseventblog). 3) IL hasnewevent(il) :- wrotesomeblogforevent(il) --> * large complex task * haschangedmind(il) :- wrotedifferentcontextinblog(il) --> * large complex task * Big Data Science 24

givegoodevaluation(itm, ATopicHisBlog) :- wrote(itm, ATopicHisBlog) and therearegoodrepliesmorethan(atopichisblog, threshold).

21 Running Example of Projection by ASA Demonstration Big Data Science 25

Data Mining Yelp Data - Predicting rating stars from review text

Data Mining Yelp Data - Predicting rating stars from review text Rakesh Chada Stony Brook University rchada@cs.stonybrook.edu Chetan Naik Stony Brook University cnaik@cs.stonybrook.edu ABSTRACT The majority