Dealing with Big Data: Planning for and Surviving the Petabyte Age

Transcription

1 Cogizat White paper Deaig with Big Data: Paig for ad Survivig the Petabyte Age Executive Summary 1 The cocept of big data is gaiig attetio across idustries ad the gobe, thaks to the growth i socia media (Twitter, Facebook, bogs, etc.) ad the exposio of rich cotet from other iformatio sources (activity ogs from the Web, proximity ad wireess sources, etc.). The desire to create actioabe isights from the evericreasig voumes of ustructured ad structured data sets is forcig eterprises to rethik their approaches to big data, particuary as traditioa approaches have proved difficut if eve possibe to appy to structured data sets. Whie data voume proiferates, the kowedge it creates has ot kept pace. For exampe, the sheer compexity of how to store ad idex arge data stores, as we as the iformatio modes required to access them, have made it difficut for orgaisatios to covert captured data ito isight. The media appears obsessed with how today s eadig compaies are deaig with big data, a pheomeo kow as ivig i the Petabyte Age. However, coverage ofte focuses o the techoogy aspects of big data, eavig such cocers as usabiity argey utouched. For years, the acceeratig data deuge has aso received sigificat attetio from idustry pudits ad researchers. What is ew is the threshod that has bee crossed as the osaught cotiues to acceerate i terms of voume, 2 compexity ad formats. A traditioa approach to hadig big data is to 3 repace SQL with toos ike MapReduce. However, the sheer voume of data cotaied i a data set that is routiey aaysed does ot sove the more pressig issue that peope have difficuty focusig o massive amouts of tabes, fies, Web sites ad data marts, a of which are cadidates for aaysis. It s ot a about just data warehouses, aymore. Usabiity is a factor that wi overshadow the techica characteristics of big data aaysis for at east the ext five years. This paper focuses specificay o the roadmap orgaisatios must create ad foow to survive the Petabyte Age. Big Data = Big Chaeges The Petabyte Age is creatig a mutitude of chaeges for orgaisatios. The acceeratig Copyright 2012, Cogizat. A rights reserved. White Paper 2012

2 deuge of data is probematic to a, for withi the massive array of data sources icudig data warehouses, data marts, portas, Web sites, socia media, fies ad more is the iformatio required to make the smartest strategic busiess decisios. May eterprises are facig the diemma that the systems ad processes devised specificay to itegrate a t h i s i fo r m a t i o a c k t h e re q u i re d resposiveess to pace the iformatio ito a eaty orgaised warehouse i time to be used at the curret speed of busiess. The heighteed use of Exce ad other desktop toos to itegrate eeded iformatio i the most expediet way possibe oy adds compexity to the probem eterprises face. There are a umber of factors at the heart of big data that make aaysis difficut. For exampe, the timig of data, the compexity of data, the compexity of the sythesised eterprise warehouse ad the idetificatio of the most appropriate data are equa if ot arger chaeges tha deaig with arge data sets, themseves. The icreased compexity of the data avaiabe for geeratig isights is a direct cosequece of the foowig: The highy commuicative ad itegrated goba ecoomy. Eterprises across idustry are icreasigy seekig more grauar isight ito market forces that utimatey shape their success ad faiure. Data geerated by the aways-o ecoomy is the impetus, i may cases, for the kee iterest i impemetig so-caed isight faciitatio appiaces o mobie devices smartphoes, ipads, Adroid ad other tabets throughout the eterprise. The eighteed cosumer. Give the exposio i socia media ad smart devices, may cosumers have more iformatio at their figertips tha ever before (ofte more so tha busiesses) ad are becomig icreasigy sophisticated i how they gather ad appy such iformatio. The goba reguatory cimate. New reguatory madates coverig fiacia trasactios, corporate heath, food-bore iesses, eergy usage ad avaiabiity ad other issues require busiesses to store icreasigy greater voumes of iformatio for much oger time frames. As a resut of these factors, eterprises wordwide have bee rapidy icreasig the amout of data housed for aaysis to compete i the Petabyte Age. May have respoded by embracig ew techoogies to hep maage the sheer voume of data. However, these ew toys aso itroduce data usabiity issues that wi ot be soved by ew techoogy but rather wi require some rethikig of the busiess cosequeces of big data. Amog these chaeges: Big data is ot oy tabuar; it aso icudes documets, e-mais, pictures, videos, soud bites, socia media extracts, ogs ad other forms of iformatio that is difficut to fit ito the icey orgaised word of traditioa database tabes (rows ad coums). Compaies that tacke big data as a techoogy- oy iitiative wi oy sove a sige dimesio of the big data madate. There are sheer voumetric issues, such as biios of rows of data, that eed to be soved. Whie tried-ad-true techoogies (partitioig) ad ewer techoogies (MapReduce, etc.) permit orgaisatios to segmet data ito more maageabe chuks, such a approach does ot dea with the issue that rarey used iformatio is coggig the pathway to ecessary i fo r m a t i o. Tra d i t i o a i fe cyc e maagemet techoogies wi aeviate may of the voumetric issues, but they wi do itte to sove the o-techica issues associated with voumetrics.

3 Tabuatig the Iformatio Lifecyce Types of Iformatio Retaied the Logest How Much Iformatio is Retaied Deveopmet Records Emai Fiacia Records Database Archive Govermet Records orgaisatio Records Customer Records Source Fies 4% 4% 5% 6% 11% 18% 19% 25% >1 PB >500 TB >100 TB >25 TB >10 TB >5 TB >1 TB <1 TB 4% 4% 7% 7% 14% 18% 21% 25% 0% 5% 10% 15% 20% 25% 0% 5% 10% 15% 20% 25% Source: The 100 Year Archive Task Force, SNIA Data Maagemet Forum Figure 1 As a resut of mergers ad acquisitios, goba sourcig, data types ad other issues, the sheer umber of tabes ad objects avaiabe for access has mushroomed. This icrease i the voume of objects has made access schemes for big data overy compex, ad it has made fidig eeded data aki to fidig a eede i a haystack. 4 The iformatio ifecyce maagemet cosideratios of data have ot received the attetio they deserve. Iformatio ifecyce maagemet shoud ot be imited to partitioig schemes ad the physica ocatio of data. (Much attetio is beig give to coud ad virtuaised storage, which presumes a process has bee devised for ratioaisig the fact that aways-o iformatio made avaiabe i the coud is worthy of this heighteed avaiabiity.) Iformatio ifecyce maagemet is the process that traditioay stratifies the physica ayout for techica performace. I the Petabyte Age, where the amout of iformatio avaiabe for aaysis is icreasig at a acceeratig rate, the iformatio ifecyce maagemet process shoud aso esure a heighteed focus o iformatio that matters. This stratificatio shoud categorise iformatio ito the foowig groups: Iformatio directy reated to the creatio, extractio or capture of vaue Supportig iformatio that coud be referred to whe devisig a strategy to create, extract or capture vaue. Iformatio required for the operatios of the eterprise but ot ecessariy reated to the creatio, extractio or capture of vaue. Iformatio required for reguatory activities but ot ecessariy reated to the creatio, extractio or capture of vaue. Historica supportig iformatio. Historica iformatio that was oce aiged with vaue, reguatory or other purposes but is ow kept because it might be usefu at some future date. Much of the compexity of iformatio made avaiabe for derivig isight is a compex weave of mutipe versios of the truth, data orgaised for differet purposes, data of differet vitages ad simiar data obtaied from differet sources. This is a pheomeo that may orgaisatioa stakehoders describe as a back box of iformatio ito which they have o isight ito its ieage. This adds deay to the use of isight gaied from such iformatio, a deveopmet caused by the obvious ecessity of vaidatig iformatio prior to usig it for aythig out of the ordiary. Much of the data avaiabe for aaysis resuts from the covetioa wisdom that wiig orgaisatios are data pack rats ad that iformatio that arrives o their doorsteps teds to stay as a permaet artifact of the busiess. Iterestigy, accordig to the Year Archive Task Force, source fies are the most frequety idetified source of data retaied as part of the 100-Year Archive.

4 May Sources of Data Porta Taxoomy Parter Data Data Warehouse Dimesios Other Iformatio Spreadsheets ad Loca Data Persoa orgaisatio? Iforma Network Operatioa System Keys Figure 2 A sizeabe amout of operatioa iformatio is ot housed i officia systems admiistered by eterprise IT groups but istead is stored o desktops throughout the orgaisatio. Much of these oca data stores were created with good itetios; peope resposibe for busiess fuctios had o other way of gaiig cotro over the iformatio they eeded. As a resut, these desktop-based systems have created a differet form of big data a weave of Microsoft Access, Exce ad other desktop toobased sources that are just as critica i ruig eterprises. The cotets of these idividuay sma to medium-size sources of iformatio coectivey add up to a sizeabe umber of sources. Oe arge eterprise was foud to have cose to 1,000 operatioa metrics maaged i desktop appicatios (i.e., Exce) which is ot a ucommo situatio. These sources ever make it to the productio e v i ro m e t a d d ow stream d a ta warehouses. Much of the big data housed i orgaisatios resuts from reguatory requiremets that ecessitate storig arge amouts of historica data. This arge voume of historica data hides the data required for isight. Whie it is importat to retai this iformatio, the ecessity to house it with the same priority as iformatio used for derivig isight is both expesive ad uecessary. iformatio required for vaue extractio, 6 origiatio ad capture. This partitioig process shoud resut i a process that tiers iformatio aog the traditioa dimesios of a busiess iformatio mode. Such a mode ehaces the focus of iformatio that supports the extractio, origiatio ad capture of vaue. The Roadmap to Maagig Big Data Big data wi be soved through a combiatio of ehacemets to the peope, process a dtechoogy stregths of a eterprise. Peope-based iitiatives that wi impact big data programs empoyed at compaies icude the foowig: Maagig the iformatio ifecyce empoyed at the orgaisatios. For good reaso (garig privacy ad security cocers, amog them), orgaisatios have paced sigificat focus o iformatio goverace. The madate for determiig which data deserves focus shoud be part of the overa goverace charter. Esurig a sufficiet ski set to tacke the issues itroduced as a cosequece of big data. The Case for Horizota Partitioig Horizota partitioig is the process defied as segmetig data i such a way that prioritises

5 Deveopig a series of metrics to maage the effectiveess of the big data programme. This icudes: Average, miimum ad maximum time required to tur data ito isight Average, miimum ad maximum time required to itegrate ew iformatio sources Average, miimum ad maximum time required to itegrate existig iformatio sources Time required for the maagemet process Percetage of peope associated with the program participatig i the maagemet process Vaue achieved from the programme 7 Accordig to McKisey, the activities of peope steerig big data wi icude: Esurig that big data is more accessibe ad timey Measurig the vaue achieved by uockig bigdata Embracig experimetatio through data to expose variabiity ad raise performace Eabig the customisatio of popuatios through segmetatio Faciitatig the use of automated agorithms to repace ad support huma decisio makig, thereby improvig decisios, miimisig risks ad uearthig vauabe isights that woud otherwise remai hidde. Faciitatig iovatio programmes that use ew busiess modes, products ad services. Process-based iitiatives that wi impact big dat a programmes are best eabed as augmetatios of a compay s goverace activities. These augmetatios icude: Esurig sufficiet focus o iformatio that wi drive vaue withi the orgaisatio. These processes are best empoyed as ikages betwee corporate strategy ad iformatio ifecyce maagemet programmes. It is importat to ote that iformatio ifecyce maagemet is defied i may orgaisatios as a programme to maage hierarchies ad busiess cotiuity. For the purposes of this paper, this defiitio is exteded to icude promotio ad demotio of data items as aiged with the busiess iformatio mode (how iformatio is used to support eterprise s t ra te g i e s a d ta c t i c s) of t h e orgaisatio. The process used to gover big data ad its iformatio ifecyce shoud cotiuay refie ad prioritise the beefits, costraits, priorities ad risks associated with the timey pubicatio ad use of reevat, focused, actioabe ad trustworthy iformatio pubished uder big data iitiatives. Covertig Big Data Ito Vaue Acquired & Created Kowedge Capabiities Chaes Risks Ivestors Heard Iferece Extracted Vaue Captured Trasactio Vaue Chai Reevat Actioabe Trustworthy Data Isight Actio Vaue Captured Vaue Customers Reguatory Disruptios Leared Iferece Iovatio Origiated Vaue Vaue Stream Markets Expected Outcomes Figure 3

6 Esurig the metrics that drive proper Esurig that toos itroduced to avigate adhesio ad use of big data are deveoped. This shoud cover the foowig topics: big data are usabe by the iteded audiece without upsettig sef-service paradigms that have sowy gaied tractio durig the past severa years. Goverig big data Big data ifecyce Esurig that the architecture ad supportig etwork, techoogy & software ifrastructures are capabe of supportig big data. Big data use ad adoptio Big data pubicatio metrics Big Data Gettig Bigger TB ebay: 6.5 PB, 2009 Googe: 1 PB of ew data every 3 days, Size of the Largest Data Warehouse i the Witer Top Te Survey CAGR = 173% Moore s Law Growth Rate 200 It is safe to state that if history is ay predictio of the future, the sheer voume of data that orgaisatios wi eed to dea with wi outstrip our coective imagiatios for how much data wi be avaiabe for geeratig isights. Oy eight years ago, a 300 to 400 terabyte data warehouse was cosidered a outier. Today, muti-petabyte warehouses are easiy foud. Faiure to take actio to maage the usabiity of iformatio pourig ito the eterprise is (ad wi be) a competitive disadvatage (see Figure 4) Actua Recommedatios Projected Source: Witer Corp. Figure 4 Techoogy-based iitiatives that wi impact big data programs empoyed at compaies icude: Esurig that the speciaised skis required to admiister ad use the fruits of the big data iitiative are preset. Some of these icude the databases, the otoogies used to avigate big data ad the MapReduce cocepts that augmet or fuy repace SQL access techiques. Big data is a reaity i most eterprises. However, compaies that tacke big data as merey a techoogy imperative wi sove a ess importat dimesio of their big data chaeges. Big data is much more tha a extesio of the techoogies used i pa r t i t i o i g st ra te g i es e m p oye d a t eterprises. Storage Defiitios Terabyte Wi fit 200,000 photos or MP3 sogs o a sige 1 terabyte hard drive. Petabyte Wi fit o 16 Backbaze storage pods racked i two data ceter cabiets. Exabyte Wi fit i 2,000 cabiets ad fi a four story data ceter that takes up a city bock. Zettabyte Wi fit i 1,000 data ceters or about 20% of Mahatta, New York. Yottabyte Wi fit i the states of Deaware ad Rhode Isad with a miio data ceters. Figure 5

7 Compaies have proved that they are pack rats. They eed to house arge amouts of history for advaced aaytics, ad reguatory pressures ifuece them to just store everythig. The reduced cost of storage has aowed compaies to tur their data warehousig eviromets ito data dumps, which has added both a compexity to the modes (makig it difficut for kowedge workers to avigate the data eeded for aaysis) ad added a aaytic paque, which makes fidig the required iformatio for aaysis more aki to fidig a eede i a haystack. I may orgaisatios, iformatio ifecyce iitiatives are madated that too ofte focus o the optima partitioig ad archivig of the eterprise (i.e., vertica partitioig). Largey a techoogy focus, this thread of the iformatio ifecyce overooks data that is o oger aiged with the strategies, tactics ad itetios of the orgaisatio. The scope ad breadth of iformatio housed by eterprises i the Petabyte Age madates that data be stratified accordig to its usefuess i the orgaisatioa vaue creatio (i.e., horizota partitioig). I today s orgaisatios, the oy cross-fuctioa bodies with the abiity to perform this horizota partitioig arevirtua orgaisatios empoyed to gover the eterprise iformatio asset. It was oy a few years ago that a data warehouse requirig a terabyte of storage was the exceptio. As we embrace the Petabyte Age, compaies are eterig a era where they wi eed to be capabe of hadig ad aaysig much arger popuatios of iformatio. Regardess of the processes put i pace, ever-icreasig voumes of structured ad ustructured data wi oy proiferate, chaegig compaies to quicky ad effectivey covert raw data ito isight i ways that stad the test of time. Foototes Big data refers to data sets that grow so arge that they become awkward to work with usig ohad database maagemet toos. Difficuties icude capture, storage, search, sharig, aaytics ad visuaisig. The Toxic Terabyte (IBM Research Labs, Juy 2006) provides a thorough aaysis of how compaies had to get their houses i order to dea with a terabyte of data. If this work were rewritte today, it woud be caed The Probematic Petabyte ad i five years, most probaby, The Exhaustive Exabyte. MapReduce is a Googe-ispired framework specificay devised for processig arge amouts of data optimised across a grid of computig power. Iformatio ifecyce maagemet is a process used to improve the usefuess of data by movig esser used data ito segmets. Iformatio ifecyce maagemet is most commoy iterested i movig data from aways-eeded partitios to rarey eeded partitios ad, fiay, ito archives. 5 6 SNIA Data Maagemet Forum s 100 Year Archive Task Force, Horizota partitioig is a term created by the author. It describes the appicatio of geeray accepted techiques of gaiig performace by segmetig data ito partitios (vertica partitioig) to segmetig groups of data by the ikeihood of it achievig orgaisatioa vaue. 7 Big Data, The Next Frotier for Iovatio, Competitio ad Productivity, McKisey & Compay, May 2011.

8 About the Author Mark Abaa is Director of Cogizat s North America Busiess Iteigece ad Data Warehousig Cosutig Practice. This practice provides soutio architecture, iformatio goverace, iformatio strategy ad programme goverace services to compaies across idustries ad supports Cogizat s busiess iteigece ad data warehouse deivery capabiities. A graduate of Syracuse Uiversity, Mark has hed seior thought eadership, advaced techica ad trusted advisory roes for orgaisatios focused o the discipies of iformatio maagemet for over 20 years. He ca be reached at Mark.Abaa@cogizat.com. EIM at Cogizat Eterprise Iformatio Maagemet withi Cogizat has more tha 10,220+* BI cosutats across the gobe. Cogizat's award-wiig BI practice is at the forefrot of parterig eadig compaies aroud the word i architectig pragmatic, busiess-focused, eterprise-wide BI soutios. The practice has bee recogised for its roe i eabig BI exceece through prestigious idustry awards, icudig three Computerword BI Perspectives Best Practices Awards, the DM Review Iovative Soutio Award, two TDWI Awards, the Cogos Performace Leadership Award, the Cogos Exceece Award, three Iformatica Iovatio Awards, the Iformatica Soutio Parter of the Year Award, the Teradata Goba Parter of the year Award, two Teradata EPIC Awards ad CIO 100 Award. For more iformatio o Cogizat's EIM soutios, cotact us at iquiry@cogizat.com or visit our website at * As of February, 2012 Word Headquarters 500 Frak W. Burr Bvd. Teaeck, NJ USA Phoe: Fax: To Free: Emai: iquiry@cogizat.com Europea Headquarters 1 Kigdom Street Paddigto Cetra Lodo W26BD UK Phoe: +44 (0) Fax: +44 (0) Emai: ifouk@cogizat.com Idia Operatios Headquarters #5/535, Od Mahabaipuram Road Okkiyam Pettai, Thoraipakkam Cheai, Idia Phoe: +91 (0) Fax: +91 (0) Emai: iquiryidia@cogizat.com Copyright 2012, Cogizat. A rights reserved. No part of this documet may be reproduced, stored i a retrieva system, trasmitted i ay form or by ay meas, eectroic, mechaica, photocopyig, recordig, or otherwise, without the express writte permissio from Cogizat. The iformatio cotaied herei is subject to chage without otice. A other trademarks metioed herei are the property of their respective owers.