Big Data: Opportunities & Challenges, Myths & Truths 資 料 來 源 : 台 大 廖 世 偉 教 授 課 程 資 料

Transcription

1 Big Data: Opportunities & Challenges, Myths & Truths 資 料 來 源 : 台 大 廖 世 偉 教 授 課 程 資 料

2 美 國 13 歲 學 生 用 Big Data 找 出 霸 淩 熱 點 Puri 架 設 網 站 Bullyvention, 藉 由 分 析 Twitter 上 找 出 提 到 跟 霸 凌 相 關 的 詞, 搭 配 地 理 位 置 呈 現 在 地 圖

3 大 數 據 直 播 中 國 春 運 流 量 百 度 公 司 利 用 LBS 定 位 數 據 開 發 的 百 度 遷 移

4 Google Flu Trend Ginsberg, J., M. H. Mohebbi, R. S. Patel, and L. Brammer Detecting influenza epidemics using search engine query data, Nature Vol. 457 (19). It uses aggregated Google search data to estimate current flu activity around the world in near real-time.

5 Big Data (1/3): How Big is Big? A Google VP says at least 10TB in terms of data rate. Petabytes is preferable Please call your work Small Data, NOT Big Data. Luckily, Google agrees that their work is Extreme Data.

6 Big Data (2/3): Is Big Data a real discipline standing on its own? Some heavyweights said Big Data is not new. Database and data mining have been around for more than 30 years. Rebuttal: Quantitative and Qualitative Big data already disrupted the field of data models and relational database and demanded new ways of building systems. In the case of data mining above, see the new book on Professor Ullman s website at Stanford: Mining of Massive Datasets

7 Big Data (3/3): Is Big Data just a small part of Cloud Computing? Some heavyweights said Big Data is just a small part of Cloud Computing. Don t make a big deal out of it. Rebuttal: They are probably just different perspectives: Cloud focuses more on elastic computing and warehouse computing. Big Data focuses more on enterprise cloud and possible-time (Best case: real-time) analytics. Big Data is a big deal: Many impossible business model possible now.

8 Top Technology Trends for 2014 from IEEE Computer Society journals 1. Emergence of the Mobile Cloud 2. From Internet of Things to Web of Things 3. From Big Data to Extreme Data Simpler analytics tools needed to leverage the data deluge. 4. The Revolution Will Be 3D 5. Supporting New Learning Styles 6. Next-generation mobile networks 7. Balancing Identity and Privacy 8. Smart and Connected Healthcare 9. E-Government 10. Scientific Cloud Computing Key to solving grand challenges, pursuing breakthroughs

9 Big Data Everywhere Big Data in various professions: Big Data by itself doesn t mean much E.g., Google indexed World Wide Web pages Which demanded the creation of Map-Reduce and NoSQL. Big Data is not an isolated discipline: Big Data is not exploding suddenly in itself. Always ask which domain first. In many domains, data increases quantitative changes and qualitative changes New business applications, in turn solidifying the Big Data research.

10 Big Data Everywhere Applications Everywhere:Domain knowledge, Business models Analytics:Ad-hoc analytics, statistics, machine learning Systems:Tools, Infrastructure

11 Summary: Big Data is Not in Vacuum When people claim they are doing Big Data, always ask them these 3 questions: Where are your data coming from? How do you process the data? How do you monetize the data?

12 Big Data from CSO s perspective Top-line vs. Bottom-line ( 開 源 vs. 節 流 ) CSO (Chief Strategy Officer) vs. CIO (Chief Information Officer) First, here we talk about Big Data from CSO s perspective IBM says: Big Data = Many impossible business model possible now ( 無 限 商 機 )

13 Big Data from CIO s perspective CIO = Chief Information Officer To many CIO, Big Data means: Breaking the bondage of IOE. No more expensive hardare from IBM: PC is enough No more expensive software from Oracle: opensource, NoSQL NO more expensive storage form EMC: No RAID. Just common hard drives

14 Big Data helps both top-line and bottom-line! It helps in a way that both CSO and CIO from everywhere can understand! No wonder Big Data sees meteoric rise When even layman everywhere talks about Big Data, there is bubble. Back to Basics

15 Back to Basics: What is Data? Texts, Records, Statistics... Intelligence: Usable Analysis Information: Readable Data: Visible Storage: Storable

16 Data is growing exponentially

17 Data Business Retailer Transportation Healthcare Education Manufacturing Contents/ Labels Agriculture Advertisement Telco Finance Power Grid

18 Data Business with Examples: Biology Virus DNA sequencing Human Genome Project Government NSA: Internet Surveillance Marketing Market survey and analysis Stock indexes Epidemiology Flu prediction

19 Big Data s Value in Fast Follower CPFR Collaborative Planning Forecast Replenishment Walmart is well-known in this business intelligence Fast Follower 資 料 一 大, 資 料 本 身 的 特 性 自 己 陳 現 出 來 一 斤 的 data 大 於 一 兩 的 演 算 法 Peter Norvig on Why I Quit Academia and Joined Google

20 Myths About Big Data Big Data is Only About Massive Data Volume Volume is just an element of Big Data Big Data is all-powerful Can get All Of The Data Big Data Yields Certainty Can answer WHY

21 4 Vs Big Data is Only About Massive Data Volume Volume : The start point of Big Data, but the least important of 3 elements Variety : Traditional data management processes can t cope with the heterogeneity of big data Velocity : Data is generated in real time, with demands for usable information to be served up immediately Veracity : Data in real world is with noise, how to make data to be trusted for the organization to make crucial decision?

22 4Vs of Big Data

23

24 Some Make it 4V's

25 Variety:NoSQL means No SQL NoSQL means not only SQL Big Data stores offer domain-specific access and query techniques in addition to SQL or SQL-like interfaces Example :key-value stores, document-oriented databases, graph databases, big table structures NoSQL technology to fit both the business problem and data type and the many categories of NoSQL technologies offer plenty of choice

26 NoSQL DEFINITION Next Generation Databases mostly addressing some of the points: being non-relational, distributed, opensource and horizontally scalable. Characteristics Schema-less, schema-free Scalability Eventual consistency (not ACID) Huge amount of data Simple API Theorem ACID vs CAP

27 Emerging Technologies for Managing Big Data Storage Computing Graph Database Analytics & Business knowledge

28 Data is growing exponentially Even faster than Moore s law! Moreover, does Moore s law still work? Disk seek? Network?

29 Problems when data is BIG How to store? How to retrieve? How to process?

30 How to store? It s not likely to store it on a single machine Facebook generates TBs of data every day 60hr. of video is uploaded to Youtube every minute Distributed File System Google File System Hadoop Distributed File System(HDFS)

31 How to retrieve? You may want to use a database system to organize data MySQL, PostgreSQL. Unfortunately, they don t scale well to this level... One naive reason is that they usually run on only 1 machine.

32 Data Warehousing BigTable / HBase Cassandra Dynamo Hive/Shark

33 How to process? Finally, you can get the data we need efficiently from the monster-like data set. And it s time to do something cool now ;) Learning, mining, retrieval But you ll soon face some trouble... Data can t fit in memory / disk on a single machine Not powerful enough with a single machine

34 Computing MapReduce Massive Parallel Processing Spark: in-memory computing

35 Graph Why graph? Graphs abstract application-specific algorithms into generic problems represented as interactions using vertices and edges Algorithms vary in their computational requirements

36 Graph Pregel: Paper from Google Giraph HAMA

37 END