Big Data Operations: Basis for Benchmarking Big Data Systems

Transcription

1 Big Data Operations: Basis for Benchmarking Big Data Systems Justin Zhan North Carolina State A&U University, Greensboro Arcot Rajasekar Reagan Moore Shu Huang Yufeng Xin University of North Carolina at Chapel Hill

2 Agenda Big Data Characteristics Big Data Benchmarking (BDB) Operations as a basis for BDB

3 An Analogy

4 Data Systems - Today

5 Data Systems - Tomorrow

6 Big Data EveryWhere! Lot of data collected and analyzed Web data, e-commerce Scientific projects Commercial/Financial transactions Social Network data Medical & Health Information

7 How much data? Google processes 20 PB a day (2008) Wayback Machine has 3 PB (3/2009) Growing at 100 TB/month Facebook has 2.5 PB of user data (4/2009) Growing at 30 TB/day ebay has 6.5 PB of user data(5/2009) Growing at 50 TB/day CERN s Large Hydron Collider (LHC) generates 15 PB/year

8 Characteristics of Big Data Five Vs - Volume Exponential Increase in Size & Count Increasing number of items being juggled Velocity Speed at which Data is Created, Processed, Disbursed and Used Speed of tems being juggled Variety Multi-dimensionality, arrangement, format, etc. Disparate Types of items being juggled Veracity Integrity & Fidelity Value Dodging rotten tomatoes and catching the hilts of knives Worth Amount of money collected for the juggling act

9 Paradigm Shift Compute Intensive to Data Intensive Large Action on Small Amounts of Data to Small Actions on Large Numbers of Data Move Data to Processing Site (Supercomputer Model) to Move Process to Data Site (Map-Reduce Model) Function Chaining to Service Chaining Model-based Science to Data-based Science (Data Mining, Knowledge Discovery)

10 Benchmarking for Big Data Query Domain Data Loading Data Aggregation Map Reduce Selection Join Startup, Execution, Cleanup metrics Failure and Recovery Data Layout Tunability Ease of Use Data Types Handled Scalability Cost and ROI Time to First Byte Time to Launch Traditional Benchmarks: ECperf, RUBiS, SPEC jappserver, SPEC JBB, Stock-Online and TPC-W

11 Motivation for Data Operation based BDB In Big Data, performance of analytics is only one part of the equation Data is distributed and globally, in some case Identifying operational characteristics provides additional insight about end-to-end performance-base analytics One cannot assume that data is close to computational platform effectiveness of moving data/computation across the network needs to measured One cannot assume that all computation can be done at one site sharding/partitioning effectiveness need to be measured. Replication of data can be used for load-balancing, parallel computation and recovery Robustness in synchronization need to be examined Because of large sizes of data involved, failure recovery becomes an important aspect of computation Graceful degradations and self-healing capabilities need to be considered Not easy to find one-size fits all set of benchmarks So, metrics at finer granularities are needed for BDB

12 Some Questions Our Premise for BDB Can we enumerate fundamental data operations that can form the basis for benchmarking large-scale data systems? Can we prioritize such a list for importance in benchmarking? What types of data are needed to perform benchmarking for these operations?

13 Four Broad Areas for BenchMarking based on Data Operations Ingestion, Access & Discovery Compare capabilities for ingesting data, discovering and accessing large-scale data Data & Compute Movement Compare movement of data and computational capabilities across multi-scale networks User Interactions, Security Enforcement & Verification Compare capabilities for integrity and security maintenance as well as authorization Since Big Data lives in wide area, security becomes paramount Analysis Compare the types of supported computational paradigms (we don t consider this in our paper)

14 Ingestion, Access & Discovery Data and Metadata Ingestion Integrity Checking Stream Ingestion/Dissemination dropped packets Catalog & Collection Abstracts Automatic Redirection self-healing Placement Strategies Automatic Sharding Partition Tolerance Load Balancing Fault Tolerance Availability Consistency across Distribution Transactions Full, Partial, Eventual, Recoverable Catalog Discovery, Object Discovery Naming Support Ontology-informed discovery Federation of Catalogs

15 Data & Compute Movement Support for types of Data Movement Replication Full availability, Load balancing, Redundancy, Immediate Consistency Backup Recovery, Periodic Consistency Caching Temporary Copy, Surge protection Archiving Permanent copy of settled data Network-aware data movement Lambda switching, Dark Fiber Leasing, VPN reservations Bulk Data Movement, Parallel Data Movement, Lazy Data Movement capabilities Content Delivery Services, Network caching Multi-protocol support automatic tuning Support for Compute Movement Auto Sharding of computations Scatter Gather Models Service Orchestration Message Oriented Paradigm support Capacity Tuning

16 User Interactions, Security Enforcement & Verification Identification & Authentication Community Services, PKI, Challenge-Response Tickets Multi-level security TCSEC Biometrics, RFID Internal identity, temporary identity, avatars, Authorization Access Control MAC, DAC, RBAC, CBAC, HBAC Policy-based Security Access vs Denial Levels of Access Coarse to fine grain Capability levels Security Perimeters Host-level, Site-level, Grid level, Federation level Logical vs. Physical perimeters Trust relationships across administrative domains

17 Benchmarking for Big Data Too many different types of Big Data Systems NoSQL: Big Table, Dynamo, Cassandra, Hbase, SQL: RDB, NewSQL, Science DB, Types: Column-oriented, Row-oriented, Types: Key-Value, Document, Graph, Types: Textual, Semi-structure, Structured, RDF,. Not easy to benchmark based on common queries on databases Necessary to define fundamental operations Define Metrics on these operations Benchmark for these metrics This approach is also extensible & can define classes of Big Data systems

18 Conclusion Our attempt is to start a dialog on these types of benchmarking metrics Not new, but better focused for Big Data systems Is necessary because of their diversity We have defined some fundamental operational classes (and internal operations) Our method is informed by the types of problems we face in data grids an how to quantify them (irods) Future Work: Define a system for performing such benchmarks