Applications for Big Data Analytics

Transcription

1

2

3 Smarter Healthcare Applications for Big Data Analytics Multi-channel sales Finance Log Analysis Homeland Security Traffic Control Telecom Search Quality Manufacturing Trading Analytics Fraud and Risk Retail: Churn, NBO

4 Big Data Definition No single standard definition Big Data is data whose scale, diversity, and complexity require new architecture, techniques, algorithms, and analytics to manage it and extract value and hidden knowledge from it

5 Big Data: 3V s

6 Volume (Scale) Data Volume 44x increase from From 0.8 zettabytes to 35zb Data volume is increasing exponentially Exponential increase in collected/generated data

7 12+ TBs of tweet data every day data every day 30 billion RFID tags today (1.3B in 2005) 4.6 billion camera phones world wide? TBs of 100s of millions of GPS enabled devices sold annually 25+ TBs of log data every day 76 million smart meters in M by billion people on the Web by end 2011

8 Variety (Complexity) Relational Data (Tables/Transaction/Legacy Data) Text Data (Web) Semi-structured Data (XML) Graph Data Social Network, Semantic Web (RDF), Streaming Data You can only scan the data once A single application can be generating/collecting many types of data Big Public Data (online, weather, finance, etc) To extract knowledge all these types of data need to linked together

9 A Single View to the Customer Social Media Banking Finance Gaming Customer Our Known History Entertain Purchas e

10 Velocity (Speed) Data is begin generated fast and need to be processed fast Online Data Analytics Late decisions missing opportunities Examples E-Promotions: Based on your current location, your purchase history, what you like send promotions right now for store next to you Healthcare monitoring: sensors monitoring your activities and body any abnormal measurements require immediate reaction

11 Real-time/Fast Data Mobile devices (tracking all objects all the time) Social media and networks (all of us are generating data) Scientific instruments (collecting all sorts of data) Sensor technology and networks (measuring all kinds of data) The progress and innovation is no longer hindered by the ability to collect data But, by the ability to manage, analyze, summarize, visualize, and discover knowledge from the collected data in a timely manner and in a scalable fashion 11

12 Some Make it 4V s

13 The Model Has Changed The Model of Generating/Consuming Data has Changed Old Model: Few companies are generating data, all others are consuming data New Model: all of us are generating data, and all of us are consuming data

14 Big Data Solutions Map/Reduce Model Apache Hadoop HDFS MapReduce Hbase PIG Hive Nosql Databases Key/Value Column-oriented Document-oriented Graph Database XML Data

15

16 So many NoSQL options More than just the Elephant in the room Over 120+ types of NoSQL databases

17 CAP CA CP AP Highly-available consistency Enforced consistency Eventual consistency

18 Flavors of NoSQL

19 Graph Database Use for data with a lot of many-to-many relationships recursive self-joins when your primary objective is quickly finding connections, patterns and relationships between the objects within lots of data Examples: Neo4J, FreeBase (Google)

20 Column Database Wide, sparse column sets Schema-light Examples: Cassandra HBase BigTable GAE HR DS

21 Document Database (Mongo DB) Use for data that is document-oriented (collection of JSON documents) w/semi structured data Encodings include XML, YAML, JSON & BSON binary forms PDF, Microsoft Office documents -- Word, Excel ) Examples: MongoDB, CouchDB

22 Key / Value Database Schema-less State (Persistent or Volatile) Examples AWS Dynamo DB Project Voldemort

23 So which type of NoSQL? Back to CAP CA = SQL/RDBMS SQL Sever / SQL Azure Oracle MySQL CP = nosql/column Hadoop Big Table HBase MemCacheDB AP = nosql/document or key/value DynamoDB CouchDB Cassandra Voldemort

24

25 THINK