Exploratory AnalyAcs for Shared- service Hadoop Clusters

Transcription

1 Copyright 2014 Splunk Inc. Exploratory AnalyAcs for Shared- service Hadoop Clusters Sagi Zelnick Principal Architect, Yahoo

2 Disclaimer During the course of this presentaaon, we may make forward- looking statements regarding future events or the expected performance of the company. We cauaon you that such statements reflect our current expectaaons and esamates based on factors currently known to us and that actual events or results could differ materially. For important factors that may cause actual results to differ from those contained in our forward- looking statements, please review our filings with the SEC. The forward- looking statements made in the this presentaaon are being made as of the Ame and date of its live presentaaon. If reviewed arer its live presentaaon, this presentaaon may not contain current or accurate informaaon. We do not assume any obligaaon to update any forward- looking statements we may make. In addiaon, any informaaon about our roadmap outlines our general product direcaon and is subject to change at any Ame without noace. It is for informaaonal purposes only, and shall not be incorporated into any contract or other commitment. Splunk undertakes no obligaaon either to develop the features or funcaonality described or to include any such feature or funcaonality in a future release. 2

3 Overview! Yahoo: 8+ years of innovaaon! Yahoo: organizaaon- wide investment for next 3+ years! Yahoo providing Hunk as a self- service to explore, analyze & visualize data in HDFS Hunk allows for visually browsing very complex tables (250+ fields) Rapid prototyping for new jobs with almost instant results for searches, without having to wait for the enare job/query to finish Cuts down on the development cycles by faster interacaon with results Built- in graphs/charts makes for a powerful soluaon for many situaaons

4 History of Hadoop Yahoo

5 Over 600PB of Hadoop Storage (Over Half an Exabyte)! Very large clusters used by many groups across the enterprise! More than 35,000 individual datanodes! Hadoop is provided as a service! MulAple cluster types such as research, dev, sandbox and producaon! Services such as HBase, Hive, Oozie, etc! Users are free to run jobs, but have resource constraints! Maintained by the Grid OperaAons Group

6 Integrated AnalyAcs Plajorm for Diverse Data Stores Full- featured, Integrated Product Fast Insights for Everyone Explore Analyze Visualize Dashboard s Share Works with What You Have Today Hadoop Client Libraries Hadoop Clusters Streaming Resource Libraries NoSQL and Other Data Stores

7 Improving OperaAonal Visibility with Hunk! We pointed Hunk at many operaaonal logs and event data we already had on the grid! This includes system metrics, HDFS ops, JVM stats and YARN metrics! Created instrumentaaon to measure usage per user and job! Analyzed terabytes of NameNode audit logs! Job history leveraged for visualizing usage/growth and historical views! Custom events for HBase staasacs

8 Tracking Hadoop Performance & Metrics in Hunk Use Case Customer Benefits System metrics from 35k nodes Historical insights of resources Grid Ops / Grid Customers All Grid Customers IdenAfy slow tasks/nodes when debugging Track organic growth Job performance All Grid Customers Improved job SLAs HBase metrics Job logs in near real- Ame All Grid Customers All Grid Customers / Ops Track region/rs/table metrics Search for errors directly from the YARN logs

9 Measuring NameNode Performance Pre & Post Upgrades! Historical visualizaaons of all operaaons! Search data in Hunk from billions of NameNode events! Measure JVM and memory usage! Insights into operaaonal performance

10 10,086 events (5/15/14 1:00: AM to 5/22/14 1:36: AM) Visualization VisualizaAon Using Hunk num_blockreports num_copybl...perations num_heartbeats num_readbl...perations num_readme...perations num_replac...operations num_writeb...operations num_blockchecksumop 600,000, ,000, ,000,000 Fri May Sun May 18 Tue May 20 _time _time num_bl ockrep orts num_copy BlockOpera tions num_ HeartB eats num_read BlockOpera tions num_readme tadataoperati ons num_replac eblockoperat ions num_write BlockOpera tions num_blo ckchecks umop :

11 Visualization Sample TroubleshooAng in Hunk of 750 Million Events num_blockreports num_copybl...perations num_heartbeats num_readbl...perations num_readme...perations num_replac...operations num_writeb...operations num_blockchecksumop 1,000,000, ,000, ,000, ,000,000 12:00 PM Tue May :00 AM Wed May 21 _time 12:00 PM _time num_bl ockrep orts num_copy BlockOpera tions num_ HeartB eats num_read BlockOpera tions num_readme tadataoperati ons num_replac eblockoperat ions num_write BlockOpera tions num_blo ckchecks umop :15:

12 Big Picture Plus Granular Details

13 Analyzing NameNode RPC Calls (TroubleshooAng)! Who is making what RPC call (open, liststatus, create, etc.)! How oren are they making these RPC calls! From which IP/host are they coming from! Search and visualize historical data from billions of events! Prevent NameNode abuse/misuse

14 Visualizing 834 Million Discrete Events

15 ConAnued

16 Queue Insights (Capacity & Provisioning)! Each Hadoop job runs in a specific queue! We track every aspect of the YARN framework! Immediate queue performance and configuraaon profiling via job history server! Historical views and trends that enable beper capacity management! Improved queue ualizaaon and allocaaon management

17 Search Splunk New Search Visualizing Queues index="jobsummary_logs_all_red" cluster="dilithium*" eval total_slot_seconds=(mapslotseconds + reduceslotsec onds) eval gb_hours=((total_slot_seconds * 0.5) / 3600) eval gb_hours=round(gb_hours) timechart span=6h sum (gb_hours) as gb_hours by queue Last 7 days 1,175,726 events (5/20/14 8:00: PM to 5/27/14 8:26: PM) Visualization 600, , ,000 Wed May Thu May 22 Fri May 23 Sat May 24 Sun May 25 Mon May 26 _time _time OTH ER apg_dai lyhigh_ p3 apg_dail ymedium _p5 apg_hou rlyhigh_ p1 apg_ho urlylow_ p4 apg_hourl ymedium _p2 apg _p7 curveb all_larg e curveb all_me d sling shot sling stone :

18 Self- Service Job Reports! Each job is unique and so are the map and reduce elements! How to start analyzing jobs?! Historical job performance and profiling enables in- depth performance tuning! Long terms historical views and trending of growth

19 index="jobsummary_logs_all_blue" cluster="*" user="gmon" eval total_slot_seconds=(mapslotseconds + reduceslotseconds) eval gb_hours=((total_slot_seconds * 0.5) / 3600) eval gb_hours=round(gb_hours,2) eval runtime=(finishtime-submittime)/1000 stats sum(gb_hours) as gb-hours avg(runtime) as run_mins by cluster user queue jobname jobid status eval run_mins=round(run_mins/60,2) sort -gb-hours Yesterday 4,871 events (5/26/14 12:00: AM to 5/27/14 12:00: AM) Statistics (4,871) clu ster us er que ue jobname jobid status gb-ho urs run_ mins cob alt g m on grid eng PigLatin:findRemoteHDFSFromAudits.pig job_ _ SUCCE EDED cob alt g m on grid eng PigLatin:findRemoteHDFSFromAudits.pig job_ _ SUCCE EDED cob alt g m on grid eng PigLatin:findRemoteHDFSFromAudits.pig job_ _ SUCCE EDED cob alt g m on grid ops distcp: job_ _ SUCCE EDED

20

21

22

23 ...It s Not Just Logs We re Looking At More data to tap into with the metastore / Hive sources! Using the metastore we can setup virtual indexes to any table(s) in Hive, without the need to define the schema up- front! Visualize very complex tables (250+ fields)! Rapid prototyping for new jobs with almost instant results for searches, without having to wait for the enare job/query to finish! Built- in aggregates and graphs/charts! Accelerates development workflow by providing faster interacaon with data

24

25 THANK YOU