the missing log collector Treasure Data, Inc. Muga Nishizawa

Transcription

1 the missing log collector Treasure Data, Inc. Muga Nishizawa

2 Muga Nishizawa Chief Software Architect, Treasure Data

3 Treasure Data Overview Founded to deliver big data analytics in days not months without specialist IT resources for one-tenth the cost of other alternatives Service based subscription business model World class open source team Founded world s largest Hadoop User Group Developed Fluentd and MessagePack Contributed to Memcached, Hibernate, etc. Treasure Data is in production 60+ customers incl. Fortune 500 companies 400+ billion records stored Processing 40,000 messages per second 3

4 Fluentd = syslogd + many

5 Fluentd = Plugins syslogd + JSON many

6 In short > Open sourced log collector written in Ruby > Using rubygems ecosystem for plugins It s like syslogd, but uses JSON for log messages

7 Make log collection easy using Fluentd

8 Reporting & Monitoring

9 Collect Store Process Visualize Reporting & Monitoring

10 easier & shorter time Collect Store Process Visualize Hadoop / Hive MongoDB Treasure Data Reporting & Monitoring Excel Tableau R

11 How to shorten here? easier & shorter time Collect Store Process Visualize Hadoop / Hive MongoDB Treasure Data Excel Tableau R

12 How to shorten here? easier & shorter time Collect Store Process Visualize Hadoop / Hive MongoDB Treasure Data Excel Tableau R

13 Before Fluentd Server1 Server2 Server3 Application Application Application Log Fluent Server High Latency! must wait for a day...

14 After Fluentd Server1 Application Server2 Application Server3 Application Fluentd Fluentd Fluentd In streaming! Fluentd Fluentd

15 Many Users

16 Many Meetups

17 Growth by Community

18 Why did we develop Fluentd?

19 Treasure Data Service Architecture Apache App App RDBMS td-agent Treasure Data columnar data warehouse Other data sources MAPREDUCE JOBS User td-command BI apps HIVE, PIG (to be supported) JDBC, REST Query API Query Processing Cluster

20 Treasure Data Service Architecture Open Sourced Apache App App RDBMS td-agent Treasure Data columnar data warehouse Other data sources MAPREDUCE JOBS User td-command BI apps HIVE, PIG (to be supported) JDBC, REST Query API Query Processing Cluster

21 Example Use Case MySQL to TD hundreds of app servers Rails app Rails app writes logs to text files Nightly INSERT MySQL MySQL Daily/Hourly Batch Google Spreadsheet Rails app writes logs to text files MySQL MySQL writes logs to text files Limited scalability Fixed schema Not realtime Unexpected INSERT latency Feedback rankings KPI visualization

22 Example Use Case MySQL to TD hundreds of app servers Rails app td-agent sends event logs Daily/Hourly Batch Google Spreadsheet Rails app td-agent Treasure Data sends event logs MySQL Rails app td-agent sends event logs Logs are available after several mins. Unlimited scalability Flexible schema Realtime Less performance impact Feedback rankings KPI visualization

23 td-agent > Open sourced distribution package of fluentd > ETL part of Treasure Data > Including useful components > ruby, jemalloc, fluentd > 3rd party gems: td, mongo, webhdfs, etc... td plugin is for TD >

24 How Fluentd works?

25 Fluentd = Plugins syslogd + JSON many

26 Access logs Apache App logs Frontend Backend System logs syslogd Databases filter / buffer / routing Alerting Nagios Analysis MongoDB MySQL Hadoop Archiving Amazon S3

27 Access logs Apache App logs Frontend Backend System logs syslogd Databases filter / buffer / routing Alerting Nagios Analysis MongoDB MySQL Hadoop Archiving Amazon S3

28 Access logs Apache App logs Frontend Backend System logs syslogd Databases filter / buffer / routing Alerting Nagios Analysis MongoDB MySQL Hadoop Archiving Amazon S3

29 Access logs Apache Input Plugins Alerting Output Plugins Nagios App logs Frontend Backend System logs syslogd Databases Buffer Plugins filter / buffer / routing (Filter Plugins) Analysis MongoDB MySQL Hadoop Archiving Amazon S3

30 Architecture Pluggable Pluggable Pluggable Input Buffer Output > Forward > HTTP > File tail > dstat >... > Memory > File > Forward > File > Amazon S3 > MongoDB >...

31 Architecture Pluggable Pluggable Pluggable Input Buffer Output > Forward > HTTP > File tail > dstat >... > Memory > File 117 plugins! > Forward > File > Amazon S3 > MongoDB >... Contributions by Community

32 Input Plugins log Output Plugins time tag JSON :33:51 myapp.buylog { user : me, path : /buyitem, price : 150, referer : /landing } record

33 Event structure(log message) Time > second unit > from data source or adding parsed time Tag Record > JSON format > MessagePack internally > non-unstructured > for message routing

34 in_tail: reads file and parses lines apache in_tail fluentd access.log read a log file custom regexp custom parser in Ruby

35 out_mongo: writes buffered chunks apache in_tail fluentd access.log buffer

36 failure handling & retrying apache in_tail fluentd access.log buffer retry automatically exponential retry wait persistent on a file

37 out_s3 apache in_tail fluentd access.log buffer Amazon S3 slice files based on time /01/access.log.gz /02/access.log.gz /03/access.log.gz... retry automatically exponential retry wait persistent on a file

38 out_hdfs custom text formater apache in_tail fluentd access.log buffer HDFS slice files based on time /01/access.log.gz /02/access.log.gz /03/access.log.gz... retry automatically exponential retry wait persistent on a file

39 routing / copying apache in_tail fluentd Hadoop access.log buffer Amazon S3 routing based on tags copy to multiple storages

40 Client libraries > Ruby > Java > Perl > PHP > Python > D > Scala >... Application Fluentd Time:Tag:Record # Ruby Fluent.open( myapp ) Fluent.event( login, { user => 38}) #=> :56:01 myapp.login { user :38}

41 # logs from a file <source> type tail path /var/log/httpd.log format apache2 tag web.access </source> # logs from client libraries <source> type forward port </source> # store logs to MongoDB and S3 <match **> type copy <match> type mongo host mongo.example.com capped capped_size 200m </match> <match> type s3 path archive/ </match> </match> Fluentd

42 out_forward automatic fail-over load balancing apache in_tail fluentd fluentd fluentd fluentd access.log buffer slice files based on time /01/access.log.gz /02/access.log.gz /03/access.log.gz... retry automatically exponential retry wait persistent on a file

43 forwarding Fluentd fluentd fluentd fluentd fluentd fluentd fluentd send / ack fluentd

44 Fluentd - plugin distribution platform $ fluent-gem search -rd fluent-plugin $ fluent-gem install fluent-plugin-mongo

45 Use cases

46 Cookpad hundreds of app servers Rails app td-agent sends event logs Daily/Hourly Batch Google Spreadsheet Rails app td-agent Treasure Data sends event logs MySQL Rails app td-agent sends event logs Logs are available after several mins. Unlimited scalability Flexible schema Realtime Less performance impact Feedback rankings KPI visualization Over 100 RoR servers (2012/2/4)

47 NHN Japan Web Servers Fluentd Cluster Archive Storage (scribed) STREAM Fluentd Watchers Notifications (IRC) Graph Tools 16 nodes 120,000+ lines/sec 400Mbps at peak 1.5+ TB/day (raw) webhdfs Hadoop Cluster CDH4 (HDFS, YARN) hive server Huahin Manager BATCH Shib SCHEDULED BATCH ShibUI

48 Treasure Data Frontend Job Queue Worker Hadoop Hadoop Applications push metrics to Fluentd (via local Fluentd) Fluentd Fluentd sums up data minutes (partial aggregation) Treasure Data for historical analysis Librato Metrics for realtime analysis

49 Key to Fluentd s growth is...

50 Fluentd = syslogd + Plugins JSON many + Community

51 the missing log collector Treasure Data, Inc. Muga Nishizawa