Scalable Network Measurement Analysis with Hadoop. Taghrid Samak and Daniel Gunter Advanced Computing for Sciences, LBNL

Transcription

1 Scalable Network Measurement Analysis with Hadoop Taghrid Samak and Daniel Gunter Advanced Computing for Sciences, LBNL

2 Outline Motivation Hadoop overview Approach doing the right thing, Avro what worked, Pig Results Summary JointTechs, Summer

3 Hadoop overview Hadoop Ecosystem ZOO KEEPER Coordination PIG Data Flow HIVE Batch SQL Hadoop MapReduce Job Scheduling & Raw data processing HBASE Read Time Quering HDFS Hadoop Distributed File System Unstructured Storage SQOOP Data Import AVRO Serialization Framework JointTechs, Summer

4 For analysts Hadoop supports Line-oriented format Large text files Uniform structure, or known format Other cases write your own Java classes; record reader, file splitter, JointTechs, Summer

5 Internet2 owamp logs JointTechs, Summer

6 Owamp logs Binary files with header Line format Index: int, Seqno: int SendIP: bytes, RecvIP: bytes sendts: double senderr: float, RecvErr: float Delay: float

7 Challenges Hadoop support for binary file format 100GB of owamp test results, small files of 300K files Solutions Avro, the right way Preprocessing binary files to.csv, the easy way Whole file reader Streaming JointTechs, Summer

8 Avro, first attempt Avro is a data serialization system. Avro provides: Rich data structures. A compact, fast, binary data format. A container file, to store persistent data. Remote procedure call (RPC). Simple integration with dynamic languages. JointTechs, Summer

9 Avro, owamp schema { type : record, name : ow_record, fields : [ { name : index, type : int }, { name : seqno, type : int }, { name : sndip, type : int }, { name : sndport, type : int }, } { name : rcvip, type : bytes }, { name : rcvport, type : int }, { name : sndts, type : double }, { name : snderr, type : float }, { name : rcverr, type : float }, { name : delay, type : float }] JointTechs, Summer

10 Pig, the working example Apache Pig is a platform for analyzing large data sets that consists of a high-level language for expressing data analysis programs, coupled with infrastructure for evaluating these programs. Pig's language layer currently consists of a textual language called Pig Latin, which has the following key properties: Ease of programming. Optimization opportunities. Extensibility. JointTechs, Summer

11 Pig Latin Script structure LOAD data FILTER Performed as early as possible GROUP Keys of map-reduce GENERATE Aggregations, and evaluations STORE JointTechs, Summer

12 Queries performed with Pig Count number of negative delay values an indication of clock synchronization problem Basic delay statistics min, max, mean, and variance Delay histogram similar to owstats output Different time scales JointTechs, Summer

13 Time series Original key (groups): <src_domain> <dst_domain> <src_ip> <dst_ip> Value to aggregate: mean delay Time is added as part of the key by masking to a certain precision Example , , , JointTechs, Summer

14 Count negative delay values LOAD dataset Filter by negative numbers Project only relevant values, masking time at the same step Group by key, <src_domain> <dst_domain> <src_ip> <dst_ip> Aggregate mean delay JointTechs, Summer

15 Delay histogram (owstats) Demo ready! JointTechs, Summer

16 Count negative delay values JointTechs, Summer

17 Count negative delay values SALT negative counts JointTechs, Summer

18 Delay statistics LOAD dataset Filter by negative numbers Project only relevant values, masking time at the same step, generating delay*delay Group by key, <src_domain> <dst_domain> <src_ip> <dst_ip> Aggregate statistics JointTechs, Summer

19 Variance Variance formula Knowing statistics

20 Mean delay at different time scales JointTechs, Summer

21 Mean delay at different time scales JointTechs, Summer

22 Mean delay at different time scales JointTechs, Summer

23 Outlier detection JointTechs, Summer

24 Outlier detection JointTechs, Summer

25 Performance 14 machines Fastest: 12-core, Intel(R) Core(TM) i7 CPU JointTechs, Summer

26 Performance JointTechs, Summer

27 Summary Trade-offs - Hadoop configuration and tweaking - data preprocessing - structured formats - support for advanced calculation + reliability and fault-tolerance + usability + speed Hadoop requires time & effort to set up, but it's worth it. JointTechs, Summer

28 Future directions More Pig Latin options; UDF Other tools; HBase RHIPE Usability study JointTechs, Summer

29 Thank you!