De la Business Intelligence aux Big Data. Marie- Aude AUFAURE Head of the Business Intelligence team Ecole Centrale Paris. 22/01/14 Séminaire Big Data

Transcription

1 De la Business Intelligence aux Big Data Marie- Aude AUFAURE Head of the Business Intelligence team Ecole Centrale Paris 22/01/14 Séminaire Big Data 1

2 Agenda EvoluHon of Business Intelligence SemanHc Technologies for Big Data Social networks Big Data opportunihes : use cases 22/01/14 Séminaire Big Data 2

3 EvoluHon of Business Intelligence Classical Business Intelligence Seman4c Business Intelligence Real- 4me Business Intelligence StaHc report Visual analyhcs Real- Hme analyhcs Output User InteracHon Store Ad- hoc queries AnalyHcs C Flexible queries / SPARQL C Databases/ Triplestores Knowledge enrichment ConHnuous queries/ Business rules Real Hme visual- analyhcs Retro- achon Gathering InformaHon Data Warehouse ETL/Batch processing Triple Sore SemanHc ETL/Batch processing SemanHc ETL stream processing Load shedding Data sources databases Structured/unstructured data sensors Data streams StaHc data 22/01/14 Séminaire Big Data 3

4 EvoluHon of Business Intelligence Classical Business Intelligence Seman4c Business Intelligence Real- 4me Business Intelligence StaHc report Visual analyhcs Real- Hme analyhcs Output User InteracHon Store Ad- hoc queries AnalyHcs C Flexible queries / SPARQL C Databases/ Triplestores Knowledge enrichment ConHnuous queries/ Business rules Real Hme visual- analyhcs Retro- achon Gathering InformaHon Data Warehouse ETL/Batch processing Triple Sore SemanHc ETL/Batch processing SemanHc ETL stream processing Load shedding Data sources databases Structured/unstructured data sensors Data streams StaHc data 22/01/14 Séminaire Big Data 4

5 And now? Big Data A fundamental shi[ in the way companies conduct business and interact with customers Open Data /Linked Data Connected objects 22/01/14 Séminaire Big Data 5

6 Data sources TradiHonal data sources (transachons, behavior models, etc.) + new ones (social media, mobile informahon) Extract content and consumer senhment from social stream to get consumer insights, product feedback and market trends Use GPS to deliver targeted real- Hme messaging based on consumer s locahon Structured and unstructured data must be properly formabed, integrated and cleansed to extract achonable knowledge in real- Hme Using real- Hme processing can help for personalizing a customer s on- line website visit, enhancing her overall experience Monitoring of transachons in real- Hme has benefits for security 22/01/14 Séminaire Big Data 6

7 BIG DATA: TECHNOLOGICAL CHALLENGES Data infrastructure tools and plagorms : data centers, cloud infrastructures, nosql databases, in- memory databases, Hadoop/Map Reduce Ecosphere New generahon of front- end tools for BI and analyhc systems: data visualizahon and visual analyhcs, self- service BI, Mobile BI Data processing : supercomputers, distributed or massively parallel- compuhng 22/01/14 Séminaire Big Data 7

8 EvoluHon of Business Intelligence Classical Business Intelligence Seman4c Business Intelligence Real- 4me Business Intelligence StaHc report Visual analyhcs Real- Hme analyhcs Output User InteracHon Store Ad- hoc queries AnalyHcs C Flexible queries / SPARQL C Databases/ Triplestores ( Knowledge enrichment ConHnuous queries/ Business rules Real Hme visual- analyhcs Retro- achon Gathering InformaHon Data sources Data Warehouse ETL Batch processing databases Triple Sore SemanHc ETL Batch processing Structured/unstructured data SemanHcETL stream processing Load shedding sensors Data stream StaHc data 22/01/14 Séminaire Big Data 8

9 22/01/14 Séminaire Big Data 9

10 SemanHc Technologies for Big Data 22/01/14 Séminaire Big Data 10

11 Linked Data / Web of Data Linked Data is a set of principles that allows publishing, querying and consump4on of RDF data, distributed across different servers Not necessarily free / open data ExponenHal growth - > a Big Data approach: enriching Big Data with metadata & semanhcs, interlinking Big Data sets 22/01/14 Séminaire Big Data 11

12 SemanHc Data AggregaHon and Linking for Big Data Transforming unstructured content into a structured format for later analysis is a major challenge. The value of data explodes when it can be linked with other data, thus data integrahon is a major creator of value Data aggregahon from various sources can establish the veracity SemanHc technologies are a way of addressing variety 22/01/14 Séminaire Big Data 12

13 SemanHc Data AggregaHng and Linking for Big Data Linked Open Data KNOWLEDGE LAYER Ontologies SemanHc aggregahon SemanHc Enrichment and disambiguahon Linking data Linked Open Data Structured Non-structured Social Media DATA LAYER Documents Web pages Database 22/01/14 Sensor data Séminaire Big Textual Data content 13

14 Value of SemanHc Technologies SemanHc Technologies provide opportunihes for reducing the cost and complexity of data integrahon Common metadata layer Powerful soluhons to find and explore informahon SemanHc Technologies are a good fit for Big Data s Variety Velocity and Volume: challenging issues for SemanHc Technologies Linked Data will grow into Big Linked Data, but Big Data will also benefit from evolving into Linked Big Data 22/01/14 Séminaire Big Data 14

15 Social Networks 22/01/14 Séminaire Big Data 15

16 Benefits for enterprises A technology for internal communicahon, informahon sharing and collaborahon A technology for informahon communicahon towards clients Vote for the best product, Understand the clients needs A technology for watching the gossip E- reputahon, opinion mining A technology for creahng collechve intelligence CollaboraHve common knowledge Wikis and blogs associated to social networks 22/01/14 Séminaire Big Data 16

17 Benefits for public administrahons Public administrahons need social networks: As enterprises: To analyze internal networks (projects, organizahon ) To analyze external networks (suppliers, clients, partners ) As an interface for cihzens: To be well- understood by cihzens (who does what) To understand cihzens (who says what) Scenarios examples: Need to look over the organizahonal structure (employees, departments, transversal projects) and idenhfy costs Need for cihzens to understand the impact of public polihcs (offered services, available resources for each district of the city, which projects are the most relevant, cihzens complains) Opinion analysis from external social networks (Twiber for example) 22/01/14 Séminaire Big Data 17

18 SenHment Analysis Opinion mining Find out what other people think. Is it possible? Recommender systems (avoid recommending items that received a lot of negative feedback). Information Filtering Business Intelligence (why aren t consumers buying my laptop?). Question answering (what did you want to say?) Clarification of politicians positions! edemocracy and so on 22/01/14 Séminaire Big Data 18

19 DetecHng feature senhment in user- generated reviews It is not possible to summarize everything with a unique vote polarity detect local polarities expressed about the salient features of a considered domain. Combine NLP+statistics: 1) We identify the most characterizing aspects of one domain (hotels, restaurant, products) by analyzing the domain corpus and extracting the most frequent terms (eventually structuring them as a vocabulary and/or ontology) 2) We formalize the content of each review as a dependency tree among its terms and retrieve (if they exist) the features discussed within it. Then, by using the tree, we aim at discovering all the other terms that vehiculate some polarity linguistically connected to them. 22/01/14 Séminaire Big Data 19

20 English Corpus R E Raw text Dep. Graph G Dep. Graph G φi,1 φ i,2 ranking feature4 V φ i, n feature1 feature3 feature2 F Feature Set Feature Extractor POS- tagging τ Linguis4c Parser term Phrase Structure synset pos. polar neg. polar SensiWordNet Synset Polarity computa4on Subset of features F i in G feature1 Polarity for feature1 Sen4ment Computa4on synset1 synset2 Synsets in G, carrying some senhment, referred to a feature in F i

21 Studying InformaHon Influence and ReputaHon in Social Networks Availability of data on the Social Web allows us to ask new questions about social behavior How does information spread on networks? How far and how fast does information flow? What is the structure of the network? Who are the influential users and communities? How does network structure affect the flow of information? What is the collective behavior of users? How does individual behavior affect collective behavior? 22/01/14 Séminaire Big Data 21

22 InformaHon flow n n n Users are the generators of information. Wide range of domains (daily chats, politics, sports, celebrity gossips, etc). Influence depends on the domain? A B A B C D E C D E politics sports A fundamental challenge is to trace the spread of the information (also termed as contamination in literature) in the network along with the cause of its spread. 22/01/14 Séminaire Big Data 22

23 EsHmaHng informahon influence The goal of our work was to provide an unsupervised technique for estimating influence of users of social network, in a particular domain (or set of domains). What does it mean influence? The influence is intended as the capacity of a user to make the posting activity of the others similar to his/her own one. Why domain-based influence? Because an author can be very authorative in some domain and not in others (Is Obama an influential user when he talks about sport?) 22/01/14 Séminaire Big Data 23

24 Approach Following this intuition, instead of only taking into account the structural connections among users (i.e. the following/follower relationship in Twitter), we aim at discovering the nature of their relationship (the domain on which the relationship is based) based on their exchange of information about a particular domain. 3-Steps algorithm: n Domain Classification of Tweets n Creation of a Domain-based information Exchange Graph n Estimation of the influence of each for each considered domain Is TwiKer just a mirror of mass senhment or is it also able to influence opinion? 22/01/14 Séminaire Big Data 24

25 Big Data opportunihes Use cases 22/01/14 Séminaire Big Data 25

26 Big Data opportunihes Source: Big Data opportunihes survey, Unisphere / SAP, May /01/14 Séminaire Big Data 26

27 OpportuniHes: big data use cases 360 view of the customer E- reputa4on IntegraHon of data from social networks, CRM, transachonal data, etc. Example: T- Mobile, telecom operator - > ReducHon of the customer leave of 50% in a quarter SenHment analysis, proachve monitoring of social networks Example: Nestlé, food group- > Gain of 4 places in the ReputaHon InsHtute s Index due to an interachon 24/7 Op4misa4on PredicHve analysis for anomalies detechon, processes ophmizahon using sensors and operahonal data Example: Union Pacific Railroad, reduce train derailments, increase train shipment, carbon emission reduchon Public security Monitoring social networks, integrahon of spahal data and sensors Example: Serious Request > monitoring of crowd movements with Twiber and sensors, localizahon of public force, integrahon with GIS

28 Fraud detechon With Big Data, addihonal criteria may be considered: e.g. how ooen a user typically accesses an account from a mobile device or PC, how quickly the user types in a username and password New data sources: mobile data, social data o credit card fraud detechon: customer locahon can be inferred from social media data (and even from their contacts data e.g. coworkers) o insurance fraud detechon: mine social networks for fraud suspicions, e.g. people who claim sick leave or disability benefits while engaging in incompahble leisure / sport / travel achvihes 22/01/14 Séminaire Big Data 28

29 Big Data for the oil & gas industry More upstream data (structured / unstructured / real- Hme) TradiHonal data sources (e.g. equipment sensor data, maintenance records) can now be analyzed on long ranges Insights can be gained from new data sources, e.g. geophysical data (seismic in 2D, 3D, & 4D, weather, soil, ocean currents, ice flows), image & video data, market data, social media data, but also produchon / transport costs Big Data analyhcs is the next challenge facing the development of the digital oilfield: seismic image processing (4-24 TB) intelligent modeling and visualizahon of the Earth s structure faster explorahon well achvity characterizahon etc. 22/01/14 Séminaire Big Data 29

30 Transports (Union Pacific Road) AutomaHc rescheduling system relying on event streaming Sensors, RFID, GPS, unplanned maintenance, accidents, weather forecast, etc. Displays real- Hme informahon on: Trains speeds and locahons, fuel consumphon, problems and repair status, etc. Trains derailments being reduced using predichve analysis from thermometers, acoushc and visual sensors Dangerous condihons known in advance (40 M noise paberns evaluated every day and alerts given for any anomaly) 22/01/14 Séminaire Big Data 30

31 Conclusion : challenges SemanHc InformaHon aggregahon Pabern extrachon from streams and cross- analysis InformaHon extrachon from Linked Open Data: concepts and relahons linked to the streams paberns Opinion aggregahon from social media and web Social aspects for collaborahon InformaHon aggregahon: too much data to assimilate but not enough knowledge to act Distributed and real- Hme processing Design of real- Hme and distributed algorithms for stream processing and informahon aggregahon Storage and indexahon of a knowledge base IntegraHon of business processes with aggregated informahon DistribuHon and parallelizahon of data mining algorithms visual analyhcs and user modeling Dynamic user model Novel visualizahons for very large datasets 22/01/14 Séminaire Big Data 31

32 Big Data Volume, Variety, Velocity, Veracity, Virality Integrate, aggregate, analyze, visualize large data sets, whatever is their type, provenance, speed of their flow Value Big Linked Data Linked Big Data Seman4c Technologies Living Lab Linked & Big Data Academic Chair Linked Data Web of data, SemanHc Web - - A set of principles and good prachces allowing to link, publish and search for web data Structure and semanhcally enrich RDF data, with a very high scalability - > Big Linked Data Our Value proposi4on SemanHc aggregahon from textual and non textual streams Manage semanhc heterogeneity, real- Hme and distributed processing Ensure data quality and veracity Visual analyhcs

33 QUESTIONS? 22/01/14 Séminaire Big Data 33