Big Data Technology ดร.ช ชาต หฤไชยะศ กด. Choochart Haruechaiyasak, Ph.D.

Transcription

1 Big Data Technology ดร.ช ชาต หฤไชยะศ กด Choochart Haruechaiyasak, Ph.D. Speech and Audio Technology Laboratory (SPT) National Electronics and Computer Technology Center (NECTEC) National Science and Technology Development Agency (NSTDA)

2 Presentation outline Big Data Technology Overview, definition, motivation and properties BI vs. Data science Applications Big Data Tools Hadoop NoSQL MongoDB

3 What is Big Data?

4

5 Big Data: Motivation Source:

6 Structured vs. Unstructured Data Source:

7 Structured vs. Unstructured Data Source:

8 Big Data vs. Traditional Data Source:

9 Big Data vs. Traditional Data Source:

10 Big Data vs. Traditional Data Source:

11 Big Data vs. Traditional Data Source:

12 7 Key Drivers for the Big Data Market Source:

13

14

15 3Vs of Big Data

16 3Vs of Big Data

17

18 4Vs of Big Data

19 4Vs of Big Data

20 4Vs of Big Data

21 4Vs of Big Data

22 4Vs of Big Data

23 Where does Big Data come from? Source:

24 Gartner s Hype Cycle for Big Data chart

25 Big Data Business Model Maturity Chart Source:

26 Big Data Business Model Maturity Chart Source:

27 Big Data Business Model Maturity Chart Source: Bill Schmarzo, Big Data: Understanding How Data Powers Big Business

28 Difference between BI and Data Science Source: Bill Schmarzo, CTO, EMC Consulting, Enterprise Information Management & Analytics,

29 Difference between BI and Data Science Source: Bill Schmarzo, CTO, EMC Consulting, Enterprise Information Management & Analytics,

30 Difference between BI and Data Science Source: Bill Schmarzo, CTO, EMC Consulting, Enterprise Information Management & Analytics,

31 Data Scientist Source: Source:

32 Source:

33 Source:

34

35

36

37 What is Hadoop? Source: The content from this section is summarized from

38 Data is growing fast Three reasons why we are generating data faster than ever: (1) Processes are increasingly automated; (2) Systems are increasingly interconnected; (3) People are increasingly living online.

39 Data and system evolution

40 Real-time data analytics The continuous challenge in Web 2.0 is how to improve site relevance, performance, understand user behavior, and predictive insight to influence decisions. Industries - Travel, Retail, Financial Services, Digital Media, Search etc. that are consumer oriented are all facing similar real-time information dynamics.

41 What is Hadoop? Hadoop is a scalable fault-tolerant distributed system for data storage and processing (Apache license). Core Hadoop has two main systems: (1) Hadoop Distributed File System (HDFS): self-healing high-bandwidth clustered storage. (2) MapReduce: distributed faulttolerant resource management and scheduling coupled with a scalable data programming abstraction.

42 Hadoop: MapReduce

43 Hadoop's benefts Flexibility Store any data (structured or not), Run any analysis. Scalability Start at 1TB/3-nodes grow to PB/1000s of nodes. Economics Cost per TB at a fraction of traditional options.

44 Traditional DBMS Traditional relational databases and data warehouse products excel at OLAP and OLTP workloads over structured data. These form the underpinnings of most IT applications. Use relational databases when dealing with (1) Interactive OLAP Analytics; (2) Multistep ACID Transactions (3) 100% SQL Compliance. It is becoming increasingly more diffcult for classic techniques to support the wide range of use cases and workloads that power the next wave of digital business

45 Hadoop's approach Hadoop is designed to solve a different problem: the fast, reliable analysis of both structured, unstructured and complex data. Hadoop and related software are designed for 3V s: (1) Volume Commodity hardware and open source software lowers cost and increases capacity; (2) Velocity Data ingest speed aided by append-only and schema-on-read design; (3) Variety Multiple tools to structure, process, and access data.

46 Scenarios for Using Hadoop When a user types a query, it isn t practical to exhaustively scan millions of items. Instead it makes sense to create an index and use it to rank items and fnd the best matches. Hadoop provides a distributed indexing capability. Hadoop runs on a collection/cluster of commodity, shared-nothing x86 servers. You can add or remove servers in a Hadoop cluster (sizes from 50, 100 to even nodes) at will; the system detects and compensates for hardware or system problems on any server. Hadoop is self-healing and fault tolerant. It can deliver data and can run large-scale, high-performance processing batch jobs in spite of system changes or failures.

47 Scenarios for Using Hadoop (cont'd) 1) Hadoop as an ETL and Filtering Platform One of the biggest challenges with high volume data sources is extracting valuable signal from lot of noise. Hadoop platforms can read in the raw data, apply appropriate flters and logic, and output a structured summary or refned data set. This output (e.g., hourly index refreshes) can be further analyzed or serve as an input to a more traditional analytic environment like SAS. Typically a small % of a raw data feed is required for any business problem.

48 Scenarios for Using Hadoop (cont'd) 2) Hadoop as an exploration engine Once the data is in the MapReduce cluster, using tools to analyze data where it sits makes sense. As the refned output is in a Hadoop cluster, new data can be added to the existing pile without having to reindex all over again. In other words, new data can be added to existing data summaries. Once the data is distilled, it can be loaded into corporate systems so users have wider access to it.

49 Scenarios for Using Hadoop (cont'd) 3) Hadoop as an Archive Historical data is usually archived by tape or disk to secondary storage or sent offsite. When this data is needed for analysis, it s painful and costly to retrieve it and load it back up. With cheap storage in a distributed cluster, lot s of data can be kept active for continuous analysis. Hadoop is effcient it allows better utilization of hardware by allowing the generation of different index types in one cluster.

50 The Hadoop Stack Cloudera s Distribution for Hadoop (CDH)

51

52

53

54 Hadoop's Case Study:

55

56 Inverted index Source: searchkit_basics/searchkit_basics.html

57 Source: This slide is from Stanford course: CS 276 / LING 286: Information Retrieval and Web Search,

58 Source: This slide is from Stanford course: CS 276 / LING 286: Information Retrieval and Web Search,

59 Source: This slide is from Stanford course: CS 276 / LING 286: Information Retrieval and Web Search,

60 Source: This slide is from Stanford course: CS 276 / LING 286: Information Retrieval and Web Search,

61 Source: This slide is from Stanford course: CS 276 / LING 286: Information Retrieval and Web Search,

62 Source: This slide is from Stanford course: CS 276 / LING 286: Information Retrieval and Web Search,

63 Source: This slide is from Stanford course: CS 276 / LING 286: Information Retrieval and Web Search,

64 Source: This slide is from Stanford course: CS 276 / LING 286: Information Retrieval and Web Search,

65 Source: This slide is from Stanford course: CS 276 / LING 286: Information Retrieval and Web Search,

66 Source: This slide is from Stanford course: CS 276 / LING 286: Information Retrieval and Web Search,

67 Source: This slide is from Stanford course: CS 276 / LING 286: Information Retrieval and Web Search,

68 Source: This slide is from Stanford course: CS 276 / LING 286: Information Retrieval and Web Search,

69 Source: This slide is from Stanford course: CS 276 / LING 286: Information Retrieval and Web Search,

70 Hadoop: MapReduce

71 MapReduce: Google Distributed Indexing

72 Source: This slide is from Stanford course: CS 276 / LING 286: Information Retrieval and Web Search,

73 Handling Big Data with MongoDB

74 What is NoSQL?

75 RDBMS VS. NoSQL Database transaction properties: Atomic: Everything in a transaction succeeds or the entire transaction is rolled back. Consistent: A transaction cannot leave the database in an inconsistent state. Isolated: Transactions cannot interfere with each other. Durable: Completed transactions persist, even when servers restart etc. BASE Basically Available Soft-state services with Eventual-consistency.

76 Why is NoSQL becoming popular?

77 RDMBS VS NoSQL: an example

78 RDMBS VS NoSQL: an example

79 Source:

80

81

82

83

84

85

86 Google Trends

87

88

89 MongoDB: Key idea

90

91

92 MongoDB: Auto-sharding

93 RDBMS vs. MongoDB

94

95

96

97

98 Example 1. Create a Java Project 2. Get Mongo Java Driver

99 Example

100 Example

101 Example

102 Example

103 Example

104

105 Thank You Q&A