Big Data Pipeline and Analytics Platform

Transcription

1 Big Data Pipeline and Analytics Platform Using NetflixOSS and Other Open Source Software Sudhir Tonse Danny Yuan

2 Netflix is a log generating company that also happens to stream movies - Adrian Cockroft photo credit:

3 Data Is the most important asset at Netflix

4 If all the data is easily available to all teams, it can be leveraged in new and exciting ways

5 ~1000 Device Types ~500 Apps/Web Services ~100 Billion Events/Day! 3.2M messages per second at peak time! 3GB per second at peak time Dashboard

6 Type of Events User Interface Events Search Event ( Matrix using PS3 ) Star Ra>ng Event (HoC : 5 stars, Xbox, US, )! Infrastructural Events RPC Call (API - > Billing Service, /bill/.., 200, ) Log Errors (NPE, Movie is null,, )! Other Events!

7 Making Sense of Billions of Events

8 + Druid ElasticSearch

9

10 A Humble Beginning

11

12

13

14 Evolution Scale!

15 Application Application Application Application Application Application Application Application Application Application

16 We Want to Process App Data in Hadoop

17

18

19

20

21

22 Our Hadoop Ecosystem

23 @NetflixOSS Big Data Tools

24 Hadoop as a Service

25 Pig Scripting on Steroids

26 Pig Married to Clojure

27 S3MPER S3mper is a library that provides an additional layer of consistency checking on top of Amazon's S3 index through use of a consistent, secondary index. S3mper is a library that provides an additional layer of consistency checking on top of Amazon's S3 index through use of a consistent, secondary index.

28 Efficient ETL with Cassandra Cassandra

29 Offline Analysis

30 Evolution Speed!

31 We Want to Aggregate, Index, and Query Data in Real Time

32 Interactive Exploration

33 Let s walk through some use cases

34 * client activity event /name = moviestarts

35 Pipeline Challenges App owners: send and forget Data scientists: validation, ETL, batch processing DevOps: stream processing, targeted search

36 Message Routing

37

38 We Want to Consume Data Selectively in Different Ways

39

40 Message broker! High-throughput! Persistent and replicated

41 There Is More

42 Intelligent Alerts

43 Intelligent Alerts

44 Guided Debugging in the Right Context

45 Guided Debugging in the Right Context

46 Guided Debugging in the Right Context

47 What We Need Ad-hoc query with different dimensions! Quick aggregations and Top-N queries! Time series with flexible filters! Quick access to raw data using boolean queries

48 Druid Rapid exploration of high dimensional data! Fast ingestion and querying! Time series

49 Real-time indexing of event streams! Killer feature: boolean search! Great UI: Kibana

50 The Old Pipeline

51 The New Pipeline

52

53 There Is More

54 It s Not All About Counters and Time Series

55

56 Status:200 RequestId Parent Id Node Id Service Name Status a Edge Service a Gateway a Service A a74e 456 abc Service B 200

57 Distributed Tracing

58 Distributed Tracing

59 Distributed Tracing

60 A System that Supports All These

61 A Data Pipeline To Glue Them All

62 Make It Simple

63 Message Producing Simple and Uniform API messagebus.publish(event)

64 Consumption Is Simple Too consumer.observe().subscribe(new Subscriber<>() public void onnext(ackable<incomingmessage> ackable) { process(ackable.getentity(myeventtype.class)); ackable.ack(); } });! consumer.pause(); consumer.resume()

65 RxJava Functional reactive programming model! Powerful streaming API! Separation of logic and threading model

66 Design Decisions Top Priority: app stability and throughput Asynchronous operations Aggressive buffering Drops messages if necessary

67 Anything Can Fail

68 Cloud Resiliency

69 Fault Tolerance Features Write and forward with auto-reattached EBS (Amazon s Elastic Block Storage) disk-backed queue: big-queue Customized scaling down

70

71 There s More to Do Contribute Join us :-)

72 Summary + Druid ElasticSearch

73 You can build your own web-scale data pipeline using open source components

74 Thank You! Sudhir Tonse Danny Yuan