NextGen Infrastructure for Big DATA Analytics.
So What is Big Data? Data that exceeds the processing capacity of conven4onal database systems. The data is too big, moves too fast, or doesn t fit the structures of your database architectures. To gain value from this data, you must choose an alterna4ve way to process it Ed Dumbill, program chair for the O Reilly Strata Conference
Big Data Characteristics Variety: Manage the complexity of data in many different structures, ranging from relational, to logs, to raw text. Structured (logs, business transactions). Semi-structured and unstructured Velocity: Streaming data and large volume data movement. Moves at very high rates Volume: Scale from Terabytes to Petabytes (1K TBs) to Zetabytes (1B TBs). Valuable for mining patterns, trends and relationships Source : IBM Research Extracting Business Insights from large volume, variety and velocity of data, beyond what was previously possible!!
Big Data Product Metrics Choices
Big Data Enriches the Information Management Ecosystem Active Archive Cost Optimization Master Data Enrichment via Life Events, Hobbies, Roles, +++ Establishing Information as a Service Audit MapReduce Jobs and tasks OLTP Optimization (SAP, checkout, +++) Managing a Governance Initiative Who Ran What, Where, and When?
The Infrastructure The internet has spawned an EXPLOSION in data growth in the form of data sets, called Big Data, which are so large they are difficult to store, manage and analyze using tradi4onal DB and Storage architecture. Not only this new data heavily unstructured, voluminous and streams rapidly and difficult to harness, But also if we look into scale of Volume Only from The 3 V of the Big Data around Volume, Variety and Velocity then someone should imagine about the massive infrastructure requirement around this calcula4on - :.Jet Engine produces ~10PB of data every 30 minutes of flight 4me..Google processes ~20 PB of data per day.if one Exabyte s worth of data has to be placed on to DVD and stored in thin jewellary boxes and subsequently loaded in to Boeing 747 aircra[, it would take 13,513 planes to transport this one Exabyte. To capitalize on the Big Data trend, a new breed of Big Data technologies such as Hadoop, HIVE,PIG,AVRO and NoSQL have emerged which are leveraging new parallelized processing, commodity hardware to capture and analyze these new data sets and provide a price/performance that is 10 4mes beber than exis4ng Database/Data warehousing/business Intelligence Systems. Her, we will understand With higher end systems, there is a lot of data coming from all of the business processes, from managing inventories to analyzing the data for trends for future products. So in these systems, there are a lot of different applica4ons a lot of different usages of the same massive amount of data and how all of these pieces go together with evolving need for new infrastructure around Storage, Networking, Virtualiza4on and Cloud.
Key Technologies Infrastructure Required for Big Data Cloud Infrastructure Virtualization Networking Storage In Memory Database Tiered Storage Software De-Duplication Data Protection
Cloud Infrastructure
Virtualization Infrastructure: Workload Consolidation / TCO Savings
Big Data Infrastructure - Map 20-40 Nodes/Racks 16 cores 48 G RAM 6 12 * 2 TB disk 1-2 GigE to node Reduce Easy to use, developer writes few func4ons Moves compute to data Schedules work on HDFC node with data Scans through the data
Big Data Infrastructure - HDFS Immutable file structure Read, write, synch No Random writes Storage server used for computa>on Move computa>on to data Fault tolerant and easy management Built in redundancy, Tolerates disk & Node failure, Auto managing addi>on/removal of disks, One operator/8k nodes Not a SAN but high bandwidth network access to data via Ethernet Used typically to solve problems not feasible with tradi>onal systems: with Large storage capacity > 100PB raw Large I/O computa>onal BW > 4k node/cluster, scale by adding commodity HW, MR Cluster
Oracle Big Data System
EMC Big Picture
Storage Infrastructure Storage Efficiency Service Efficiency Virtualization Mapping P>V,VM Management Performance In-Memory DB, Auto -Tiering-SSD/HDD Costs Reduction Thin Provisioning De -Duplication Availability RAID/Auto-Discover HA, Snapshots,CDP, Cloning, DRS Security Encryption/DLP Storage -as -a service Service Catalogs by Workload etc. Policy Infrastructure Service Level Attributes Service Measurements Performance Analytics IOPS/Response Time, Bandwidth Automation Unified SAN/NAS Protocols Auto learning workload forensics Provisioning to match workloads Assured auto Recovery.
Problem: seeks are expensive CPU & transfer speed, RAM & disk size double every 18-24 months Seek time nearly constant (~5%/year) Time to read entire drive is growing scalable computing must go at transfer rate Example: Updating a terabyte DB, given: 10MB/s transfer, 10ms/seek, 100B/ entry (10Billion entries), 10kB/page (1Billion pages) Updating 1% of entries (100Million) takes: 1000 days with random B-Tree updates 100 days with batched B-Tree updates 1 day with sort & merge To process 100TB datasets on 1 node: scanning @ 50MB/s = 23 days on 1000 node cluster: scanning @ 50MB/s = 33 min MTBF = 1 day Need framework for distribution efficient, reliable, easy to use
New Data and Management Economics
Storage Infrastructure Big Data Targets Value Potential of Using Big Data by Data Intensive Verticals
Key Take ways Big data creating paradigm shift in IT industry o Leverage the opportunity to optimize your computing infrastructure after making a due diligence in selection of vendors/products, industry testing and interoperability. Optimize Big data analytics for query response time vs # of Users o Improving query response time for a given number of users (IOPs) or Serving for a given query response time. Select Automated Storage Management Software o Data Forensics and Tiered Placement o Every workload has unique I/O access signature. o Historical performance data for a LUN can identify performance skews. Optimize infrastructure to meet the need of Applications/ SLAs