the missing log collector Treasure Data, Inc. Muga Nishizawa

the missing log collector Treasure Data, Inc. Muga Nishizawa

Muga Nishizawa (@muga_nishizawa) Chief Software Architect, Treasure Data

Treasure Data Overview Founded to deliver big data analytics in days not months without specialist IT resources for one-tenth the cost of other alternatives Service based subscription business model World class open source team Founded world s largest Hadoop User Group Developed Fluentd and MessagePack Contributed to Memcached, Hibernate, etc. Treasure Data is in production 60+ customers incl. Fortune 500 companies 400+ billion records stored Processing 40,000 messages per second 3

Fluentd = syslogd + many

Fluentd = Plugins syslogd + JSON many

In short > Open sourced log collector written in Ruby > Using rubygems ecosystem for plugins It s like syslogd, but uses JSON for log messages

Make log collection easy using Fluentd

Reporting & Monitoring

Collect Store Process Visualize Reporting & Monitoring

easier & shorter time Collect Store Process Visualize Hadoop / Hive MongoDB Treasure Data Reporting & Monitoring Excel Tableau R

How to shorten here? easier & shorter time Collect Store Process Visualize Hadoop / Hive MongoDB Treasure Data Excel Tableau R

How to shorten here? easier & shorter time Collect Store Process Visualize Hadoop / Hive MongoDB Treasure Data Excel Tableau R

Before Fluentd Server1 Server2 Server3 Application Application Application Log Fluent Server High Latency! must wait for a day...

After Fluentd Server1 Application Server2 Application Server3 Application Fluentd Fluentd Fluentd In streaming! Fluentd Fluentd

Many Users

Many Meetups

Growth by Community

Why did we develop Fluentd?

Treasure Data Service Architecture Apache App App RDBMS td-agent Treasure Data columnar data warehouse Other data sources MAPREDUCE JOBS User td-command BI apps HIVE, PIG (to be supported) JDBC, REST Query API Query Processing Cluster

Treasure Data Service Architecture Open Sourced Apache App App RDBMS td-agent Treasure Data columnar data warehouse Other data sources MAPREDUCE JOBS User td-command BI apps HIVE, PIG (to be supported) JDBC, REST Query API Query Processing Cluster

Example Use Case MySQL to TD hundreds of app servers Rails app Rails app writes logs to text files Nightly INSERT MySQL MySQL Daily/Hourly Batch Google Spreadsheet Rails app writes logs to text files MySQL MySQL writes logs to text files Limited scalability Fixed schema Not realtime Unexpected INSERT latency Feedback rankings KPI visualization

Example Use Case MySQL to TD hundreds of app servers Rails app td-agent sends event logs Daily/Hourly Batch Google Spreadsheet Rails app td-agent Treasure Data sends event logs MySQL Rails app td-agent sends event logs Logs are available after several mins. Unlimited scalability Flexible schema Realtime Less performance impact Feedback rankings KPI visualization

td-agent > Open sourced distribution package of fluentd > ETL part of Treasure Data > Including useful components > ruby, jemalloc, fluentd > 3rd party gems: td, mongo, webhdfs, etc... td plugin is for TD > http://packages.treasure-data.com/

How Fluentd works?

Fluentd = Plugins syslogd + JSON many

Access logs Apache App logs Frontend Backend System logs syslogd Databases filter / buffer / routing Alerting Nagios Analysis MongoDB MySQL Hadoop Archiving Amazon S3

Access logs Apache App logs Frontend Backend System logs syslogd Databases filter / buffer / routing Alerting Nagios Analysis MongoDB MySQL Hadoop Archiving Amazon S3

Access logs Apache App logs Frontend Backend System logs syslogd Databases filter / buffer / routing Alerting Nagios Analysis MongoDB MySQL Hadoop Archiving Amazon S3

Access logs Apache Input Plugins Alerting Output Plugins Nagios App logs Frontend Backend System logs syslogd Databases Buffer Plugins filter / buffer / routing (Filter Plugins) Analysis MongoDB MySQL Hadoop Archiving Amazon S3

Architecture Pluggable Pluggable Pluggable Input Buffer Output > Forward > HTTP > File tail > dstat >... > Memory > File > Forward > File > Amazon S3 > MongoDB >...

Architecture Pluggable Pluggable Pluggable Input Buffer Output > Forward > HTTP > File tail > dstat >... > Memory > File 117 plugins! > Forward > File > Amazon S3 > MongoDB >... Contributions by Community

Input Plugins log Output Plugins time tag JSON 2012-02-04 01:33:51 myapp.buylog { user : me, path : /buyitem, price : 150, referer : /landing } record

Event structure(log message) Time > second unit > from data source or adding parsed time Tag Record > JSON format > MessagePack internally > non-unstructured > for message routing

in_tail: reads file and parses lines apache in_tail fluentd access.log read a log file custom regexp custom parser in Ruby

out_mongo: writes buffered chunks apache in_tail fluentd access.log buffer

failure handling & retrying apache in_tail fluentd access.log buffer retry automatically exponential retry wait persistent on a file

out_s3 apache in_tail fluentd access.log buffer Amazon S3 slice files based on time 2013-01-01/01/access.log.gz 2013-01-01/02/access.log.gz 2013-01-01/03/access.log.gz... retry automatically exponential retry wait persistent on a file

out_hdfs custom text formater apache in_tail fluentd access.log buffer HDFS slice files based on time 2013-01-01/01/access.log.gz 2013-01-01/02/access.log.gz 2013-01-01/03/access.log.gz... retry automatically exponential retry wait persistent on a file

routing / copying apache in_tail fluentd Hadoop access.log buffer Amazon S3 routing based on tags copy to multiple storages

Client libraries > Ruby > Java > Perl > PHP > Python > D > Scala >... Application Fluentd Time:Tag:Record # Ruby Fluent.open( myapp ) Fluent.event( login, { user => 38}) #=> 2012-12-11 07:56:01 myapp.login { user :38}

# logs from a file <source> type tail path /var/log/httpd.log format apache2 tag web.access </source> # logs from client libraries <source> type forward port 24224 </source> # store logs to MongoDB and S3 <match **> type copy <match> type mongo host mongo.example.com capped capped_size 200m </match> <match> type s3 path archive/ </match> </match> Fluentd

out_forward automatic fail-over load balancing apache in_tail fluentd fluentd fluentd fluentd access.log buffer slice files based on time 2013-01-01/01/access.log.gz 2013-01-01/02/access.log.gz 2013-01-01/03/access.log.gz... retry automatically exponential retry wait persistent on a file

forwarding Fluentd fluentd fluentd fluentd fluentd fluentd fluentd send / ack fluentd

Fluentd - plugin distribution platform $ fluent-gem search -rd fluent-plugin $ fluent-gem install fluent-plugin-mongo

Use cases

Cookpad hundreds of app servers Rails app td-agent sends event logs Daily/Hourly Batch Google Spreadsheet Rails app td-agent Treasure Data sends event logs MySQL Rails app td-agent sends event logs Logs are available after several mins. Unlimited scalability Flexible schema Realtime Less performance impact Feedback rankings KPI visualization Over 100 RoR servers (2012/2/4)

NHN Japan Web Servers Fluentd Cluster Archive Storage (scribed) STREAM Fluentd Watchers Notifications (IRC) Graph Tools 16 nodes 120,000+ lines/sec 400Mbps at peak 1.5+ TB/day (raw) webhdfs Hadoop Cluster CDH4 (HDFS, YARN) hive server Huahin Manager BATCH Shib SCHEDULED BATCH ShibUI by @tagomoris

Treasure Data Frontend Job Queue Worker Hadoop Hadoop Applications push metrics to Fluentd (via local Fluentd) Fluentd Fluentd sums up data minutes (partial aggregation) Treasure Data for historical analysis Librato Metrics for realtime analysis

Key to Fluentd s growth is...

Fluentd = syslogd + Plugins JSON many + Community

the missing log collector Treasure Data, Inc. Muga Nishizawa