!!!!!!!!!! Big!Data!Revealed!! An!Information!Difference!Research!Study!! December!2013!!!!!!!!!!!! Sponsored!by!!!!!!!!!!!!!

Transcription

1 BigDataRevealed AnInformationDifferenceResearchStudy December2013 Sponsoredby

2 TABLEOFCONTENTS Big$Data$Revealed$ 2$ EXECUTIVE$SUMMARY$...$4 BACKGROUND$TO$THE$SURVEY$...$6 THE$APPROACH$...$6 ABOUT$THE$RESPONDENTS$...$7 ADOPTION$OF$BIG$DATA$...$8 LEADERSHIP$AND$FOCUS$...$10 KEY$BENEFITS$AND$SUCCESS$FACTORS$...$12 BIG$DATA$DOMAINS$...$15 COSTS$AND$SIZE$OF$BIG$DATA$...$16 TECHNOLOGIES$DEPLOYED$...$18 ADDITIONAL$FEEDBACK$...$19 CONCLUSIONS$...$21 ENTERPRISES...21 VENDORS...22 ABOUT$THE$INFORMATION$DIFFERENCE$...$23 QUESTIONNAIRE$...$24 Copyright 2013TheInformationDifferenceCompanyLtd.AllRightsReserved.

3 Big$Data$Revealed$ 3$ LISTOFFIGURES Figure'1' 'Respondents'by'Company'Revenue'...'7 Figure'2' 'Respondents'by'Job'Function'...'7 Figure'3' 'Respondents'by'Industry'Sector'...'8 Figure'4' 'Status'of'Big'Data'in'Organizations'...'9 Figure'5' 'Reasons'for'Implementing'Big'Data'...'9 Figure'6' 'Primary'Focus'for'Big'Data'in'Business'...'10 Figure'7' 'Business'areas'leading'Big'Data'Initiatives'...'11 Figure'8' 'Responsible'for'Big'Data'in'the'Organization'...'11 Figure'9' 'Benefits'from'Big'Data'Initiatives'...'12 Figure'10' 'The'Main'Roadblocks'for'Big'Data'Initiatives'...'13 Figure'11' 'Positioning'of'Big'Data'and'Data'Management'Initiatives'...'13 Figure'12' 'Data'Management'Initiatives'in'Organizations'...'14 Figure'13' 'Critical'for'Success'with'Big'Data'...'14 Figure'14' 'Is'Big'Data'focused'on'Data'from'Web'Applications'...'15 Figure'15' 'Key'Big'Data'Domains'...'15 Figure'16' 'Focus'on'Structured'or'Unstructured'Big'Data'...'16 Figure'17' 'Volumes'of'Big'Data'Handled'...'17 Figure'18' 'Expenditure'on'Big'Data'to'Date'...'17 Figure'19' 'Planned'Expenditure'on'Big'Data'next'Year'...'18 Figure'20' 'Software'Solutions'Deployed'(or'Planned)'...'18 Copyright 2013TheInformationDifferenceCompanyLtd.AllRightsReserved.

4 EXECUTIVESUMMARY Big$Data$Revealed$ 4$ TherehasbeenagreatdealofrecentinterestintheindustryabouthowtodealwithsoQcalledBig Data,whichisdefinedasfollows: Definition:Big$Data$is$the$term$applied$to$data$sets$whose$size$is$beyond$the$ability$of$ commonly$used$software$tools$to$capture,$manage,$and$process$the$data$within$a$tolerable$ elapsed$time.$(source:$wikipedia). AlthoughBigDatahasrecentlyreceivedwidepressattention,thereisstillsurprisinglylittleconcrete informationavailableregardingthestateofbigdatainitiativesinbusiness.forexample,howmany companieshaveactuallyimplementedbigdatatechnologies,andinwhatareas;howmuchmoney andeffortareorganizationsinvestinginit;whichpartsofthebusinessaredrivingbigdata investments;whatbenefitsaretheyseeing;andwhichdatavolumesarebeinghandled?this survey,whichwassponsoredbyhelpitandteradata,aimstoaddressthesequestions. Themainfindingsfromthesurvey,basedon178responses,aresummarizedbelow: 30%currentlyhavenoplanstoimplementBigData,27%currentlylivewiththeirBigData initiative,andafurther5%areabouttogolive.thiscompareswellwiththefindingsfromour previoussurveyof24%.24%nowplantogolivenextyear. The27%liveimplementationssplitsoutregionallyasfollows:NorthAmerica(includingCanada) 6%,Europe18%,andtherestoftheworld3%.Thisresultfliesinthefaceofconventional wisdomthatsuggeststhatnorthamericaiswellaheadinbigdata. 24%wanttobeabletoanalyzeverylargedatasets.Theredoesnotappeartobeadominant themeastothereasonsforimplementingbigdataprojects. Thefocusisonimprovingthequalityofdecisionmaking(39%),followedbymeetingcustomer needs(apoorsecondat19%). ITisclearlyinthedrivingseat(33%)withregardstoBigDataprojects. TheCIOisgenerallyresponsibleforandleadingBigDatainitiatives(31%). 13%haveidentifiednewbusinessopportunitieswiththeirBigDatainitiatives.Thisleadsusto concludethatitisstillearlydayswhenitcomestobigdata. Themainroadblocktoimplementationisthatitisdifficulttopresentabusinesscase. Encouragingly,45%reportedthattheirBigDatainitiativeisorwillbeapartoftheirwiderdata managementinitiativessuchasmdm. Managementunderstandingandawareness(42%),leadershipattheCxOlevel(18%)anda soundandwellqdocumentedbusinesscase(9%)areallcitedascriticaltosuccessful implementationofbigdatainitiatives.thissuggeststhereiscurrentlyalackofsufficientbuyqin andcommitmentfromthebusinessside. BigDataisnotmainlyfocusedonanalyzingdatacollectedfromwebQbasedapplications(52%). Customerdataisthemostsignificantdomainformostorganizations,withproductdatasecond andfinancialdatathird.soorganizationsarenotjustinterestedinanalyzingtwitterfeeds. 40%claimedthatitistheirintentiontouseHadooptechnologiesasthestagingareaforalltheir datainfuture. Themajorityofthedataisintherange1to10terabytes.5%reporteddatavolumesinthe petabyterange,withjust4%inthe200terabyteto1petabyterange.itwouldseemthatbig Dataisnotnecessarilythatlarge. 8%havespentover$5millionontheirBigDatainitiativewhile42%havespentlessthan$1 million,25%lessthan$100,000.thisfurthersuggeststhatbigdataisatanearlystageinmost companies. Copyright 2013TheInformationDifferenceCompanyLtd.AllRightsReserved.

5 Big$Data$Revealed$ 5$ 11%plantospendmorethan$5millionwhile32%plantospendlessthan$1millioninthe comingyear.13%willspendlessthan$100,000. ThemostpopularthreetechnologiesdeployedareMicrosoft,Oracle,andTeradata,butthereis nodominanttechnologyreportedbythesurveyrespondents. Respondentfeedbackindicatesthatthereisadealofscepticismabouttheneedforbusiness case,viability,andpracticalimplementationofbigdata. Copyright 2013TheInformationDifferenceCompanyLtd.AllRightsReserved.

6 BACKGROUNDTOTHESURVEY Big$Data$Revealed$ 6$ TherehasbeenagreatdealofrecentinterestintheindustryabouthowtodealwithsoQcalledBig Data,whichisdefinedasfollows: Definition:Big$Data$is$the$term$applied$to$data$sets$whose$size$is$beyond$the$ability$of$ commonly$used$software$tools$to$capture,$manage,$and$process$the$data$within$a$tolerable$ elapsed$time.$(source:$wikipedia). AlthoughBigDatahasrecentlyreceivedwidepressattention,thereisstillsurprisinglylittleconcrete informationavailableregardingthestateofbigdatainitiativesinbusiness.webelievethetimeis ripetosurveythisareaindepth.inparticular,wewanttoputhardnumbersaroundsomequestions including: HowmanycompanieshaveactuallyimplementedBigDatatechnologies,andinwhatareas? Howmuchmoneyandeffortareorganizationsinvestinginit? WhichpartsofthebusinessaredrivingBigDatainvestments? Whatbenefitsaretheyseeing? Whichdatavolumesarebeinghandled? THEAPPROACH TheInformationDifferencesurvey, Big%Data%Revealed,wasconductedovertheInternetduring theperiodoctobertonovember2013.theparticipantswereselectedby invitations originatingdirectlyfromtheinformationdifference.participationwasalsopossibleviaalinkfrom TheInformationDifferenceLtd.websiteandvialinksonLinkedIn. ThesurveywastargetedatseniorbusinessandITleadersworldwide,drawnfromlarger organizations(withrevenuesgreaterthanus$1billionannually). Theparticipantswereprovidedwiththefollowinginformationpriortocompletingthesurvey: There$has$been$a$great$deal$of$recent$interest$in$the$industry$about$how$to$deal$with$soBcalled$Big$ Data,$which$is$defined$as$follows:$ Big$Data$is$the$term$applied$to$data$sets$whose$size$is$beyond$the$ ability$of$commonly$used$software$tools$to$capture,$manage,$and$process$the$data$within$a$tolerable$ elapsed$time.$$(source:$wikipedia) $$ $ Despite$the$plethora$of$press$attention,$there$is$still$surprisingly$little$concrete$information$available$ regarding$the$state$of$big$data$initiatives$in$business.$$at$the$information$difference,$we$consider$the$ time$ripe$to$survey$this$area.$$ $ All$information$provided$will$be$used$in$aggregate$form$only$and$will$be$kept$strictly$confidential.$$ The$survey$has$only$20$questions$on$the$topic$and$should$not$take$more$than$10$minutes$to$complete.$ In$return$for$a$fully$completed$survey,$you$will$receive$a$free$summary$of$the$analysis$of$the$survey$ results.$$additionally,$your$name$will$be$entered$in$a$prize$draw$and$the$first$five$winners$will$receive$ a$free$vendor$profile$(worth$$495)$of$their$choice.$$we$will$also$make$a$$2$contribution$to$the$red$ Cross$for$each$fully$completed$survey.$ ThequestionnaireisappendedinthesectionheadedQuestionnaire. Copyright 2013TheInformationDifferenceCompanyLtd.AllRightsReserved.

7 ABOUTTHERESPONDENTS Big$Data$Revealed$ 7$ Atotalof178respondentsworldwidetookthesurvey.34%werefromNorthAmerica(including Canada),43%fromEuropeandtheremainder(22%)fromtherestoftheworld. 52%oftherespondentswerefromlargerorganizationswithannualrevenuesinexcessofUS$1 billion.some21%werefromorganizationswhoseannualrevenuelastyearwasgreaterthanus$10 billion.48%werefromcompanieswithannualrevenuesbelowus$1billion.thisrepresentsa balancedmixofbothlargerandsmallorganizations.thedetailedbreakdownisshownasfigure1. Figure$1$ $Respondents$by$Company$Revenue$ Some42%oftherespondentsweredrawnfromabusinessbackgroundwiththemajorityhavingan ITrole(58%).ThislikelyreflectsthecurrentfocusofBigDatainthemediatowardstheIT community.22%hadjobtitlesatthedirectorleveloraboveand27%hadthetitleofenterprise Architect.ThedetailsaresetoutinFigure2. Figure$2$ $Respondents$by$Job$Function$ Thelargestlevelofparticipationwasfromthebanking,insurance,andfinancialservicesindustry (22%),perhaps,supportingtheviewthatthefinancialsectorisseekingnewbusinessopportunities tohelpitemergefromthefinancialcrisis.thefinancialsectorwasthelargestverticalofthe businessrespondentswithretailandpharmaceuticals/biotech/healthcarebothcominginat9%. Incontrasttoourprevioussurvey 1 theprofessionalservicesandcomputingsectors(32%)wereonly representedatacombinedlevelof10%.thisearliersurveyshowedthatthemainfocusandinterest 1 DoesBigDatameanBigMDM?,InformationDifferenceResearchReport,November2012 Copyright 2013TheInformationDifferenceCompanyLtd.AllRightsReserved.

8 Big$Data$Revealed$ 8$ appearedtocomefromtheitsector.thepresentresultswouldindicateashiftofinteresttothe businessintheformofthefinancialsector. OthersincludedEQCommerceandRecruitment(7%).Theremainderrepresentsawiderangeof industrysectors.thefullresultsareshowninfigure3. Figure$3$ $Respondents$by$Industry$Sector$ Theanalysisoftheresultsfromthesurveyispresentedbelow.Thequestionsreferredtointhetext areindicatedas[qn]andaresetoutinfullintheappendixheaded Questionnaire. Apartfromtheexceptionsmentionedinthetext,analysisoftheresultsfromthesurveyforregional dependencies forexample,comparisonsbetweeneuropeandnorthamerica didnotyieldany statisticallysignificantdifferencesortrends. ADOPTIONOFBIGDATA Theburningquestionwas:HowmanyorganizationsareactuallyalreadyupandrunningwithliveBig Dataimplementations?Weaskedrespondents[Q1]tosharewithuswhethertheyhadlive initiativesorwereplanningtoimplementbigdatawithinthecomingyear.theresultsare summarizedinfigure4. Some30%toldusthattheyhavecurrentlynoplanstoimplementBigDataintheirorganizations. Surprisingly,27%reportedthattheyarecurrentlylivewiththeirBigDatainitiative,andafurther5% areabouttogolive.thisisverysimilartothefindingsfromourprevioussurvey(24%)andsuggests thattherehasbeenrelativelylittlerealprogressintermsoftheproportionofcompanies implementingitsince2012.afurther24%plantogolivenextyear.takentogether,giventhislevel ofinterest,itwouldseemthatbigdataisbeginningtobetakenseriouslybyorganizations. Copyright 2013TheInformationDifferenceCompanyLtd.AllRightsReserved.

9 Big$Data$Revealed$ 9$ Lookingingreaterdetailatthe27%ofliveimplementations,wefindthatthissplitsoutregionallyas follows:northamerica(includingcanada)6%,europe18%andtherestoftheworld3%. Figure$4$ $Status$of$Big$Data$in$Organizations$ ThisresultfliesinthefaceofconventionalwisdomthatsuggeststhatNorthAmericaiswellahead. Certainly,anecdotalevidencehasbeenthattheUSAhasbeenfaraheadofEuropeintermsofBig DatatakeQup,especiallyasthingslikewebmarketinganalyticsandsocialmediawerereallythe pioneersofhadoop.giventhat34%ofrespondentsweredrawnfromnorthamerica,compared with43%fromeurope,itcannotreasonablybeconcludedthatthisdifferencecanbewhollyascribed tothedifferenceinsamplesizes.ifweturnourattentiontothefutureplans,thenhereagainwe seeaslightlyhigherleveloftakeupineurope(althoughmuchlessmarkedthanforlive implementations).ofatotalof42%eitherabouttogoliveorplanned,13%werefromnorth America,16%fromEurope,and12%fromtherestoftheworld.MightitbethatEuropehasmore interestinthesubjectthaniswidelyperceived,andthatvendorswoulddowelltofocusontheir Europeancustomerbase? WethenaskedrespondentstotelluswhytheywereconsideringorhadinitiatedtheirBigData initiative[q2].theresultsaresetoutinfigure5. Figure$5$ $Reasons$for$Implementing$Big$Data$ Copyright 2013TheInformationDifferenceCompanyLtd.AllRightsReserved.

10 Big$Data$Revealed$ 10$ Some24%areclearthattheywanttobeabletoanalyzeverylargedatasets.Whilethisclearly makessensethereisawiderangeofbusinessdemandsforbigdata.theredoesnotappeartobea dominanttheme.interestingly Other isthesecondlargestcategory(16%)andincludes requirementssuchas: TodealwithCustomerRequestsforResearch Tocopewithalargeincreaseincustomers WewanttoanalyzegeoQdataaspartofourriskassessment AnalyzinggenomicandotherpatientQrelateddata Tostoremedicalclaim837data. Afurther37%dividedroughlyequallybetweentheabilitytobeabletoanalyzeweblogs(13%), financialtransactions(12%),andclickqstreamdata(12%).sothebroadfocusisondealingwithvery largedatasetsingeneral,withawiderangeofspecificdrivers. LEADERSHIPANDFOCUS SowhatisthefocusforBigDatainitiativesinmostorganizationsandwhoisleadingthese initiatives?wefirstaskedabouttheprimaryfocusforbigdatainitiativesintheirorganizations[q3]. TheresultsareshowninFigure6. Figure$6$ $Primary$Focus$for$Big$Data$in$Business$ Interestingly,themainfocusisonimprovingthequalityofdecisionmaking(39%),withmeeting customerneedsapoorsecond(19%).itisintriguingthatcostreductionappearstohavebeen relegatedtoalowerpriority(8%)and,rathersurprisingly,managementofriskcomesinatamere 5%.Other(whichaccountsfor13%)included:enhancesales;exploringall(customers,fraud,risk, newbusinessopportunities);identifying"best"prospectsformarketingcampaigns;increasingthe customerexperience;etc. WenextaskedrespondentswhatareasoftheirbusinesswereleadingtheBigDatainitiative[Q4]? TheresultsareshowninFigure7. ItisperhapssurprisingthatITissoclearlyinthedrivingseat(33%)giventhefactthatcostreduction (seeabove)onlycameinat8%.maybethissuggeststhatitandbusinessaregenerallystartingto workclosertogether.almostaquarter(24%)oftheinitiativesarebeingledbycustomerservices, whichsquaresupwiththeprevioushighfocuson meetingcustomerneeds (19%). Copyright 2013TheInformationDifferenceCompanyLtd.AllRightsReserved.

11 Big$Data$Revealed$ 11$ Figure$7$ $Business$areas$leading$Big$Data$Initiatives$ Otherareasrated16%included:Drilling,ExploringviaIT,FraudandFinance. WhointhebusinessisactuallyresponsibleforBigDataintheorganizationandforleadingthe initiatives[q10]?theresponsesfromtherespondentsaresetoutinfigure8. Figure$8$ $Responsible$for$Big$Data$in$the$Organization$ Inlightofthepreviousresult,itisperhapsunsurprisingthattheCIOisinthedrivingseatwhenit comestoresponsibilityforandtheleadingofbigdatainitiatives(31%).aseniorbusinessmanager isdrivinginlessthanaquarteroforganizations(22%).thissuggeststhatmanyoftheinitiativesare originatingfromtheitsideofthebusinessandthat,inturn,suggeststhattheremaybeissueswith definingaclearbusinesscase. Other(12%)included:ITManager,LeadR&Dmanager, Nooneisnamedortherearemany leaders, Responsibilityisnotdefinedatthetimebeing,VPInformationSystems,chiefmarketing officer,and nosingleperson. Copyright 2013TheInformationDifferenceCompanyLtd.AllRightsReserved.

12 KEYBENEFITSANDSUCCESSFACTORS Big$Data$Revealed$ 12$ Whatkeybenefitsareorganizationsseekingandwhatroadblocksdotheyidentifyhampering progress?howdoestheirbigdatainitiativerelatetootherinitiativessuchasmasterdata management(mdm),dataquality,anddatagovernance? Weaskedrespondents,iftheyalreadyhadaBigDatainitiatelive,whatbenefitstheyhadrealizedto date[q14]?theirresponsesaresummarizednfigure9. Figure$9$ $Benefits$from$Big$Data$Initiatives$ Whilesome(13%)claimedtohaveidentifiednewbusinessopportunities,othersreportedimproved operationalefficiency(11%),whileoverhalfoftherespondentsdidn'tknowatthistime.this stronglysuggeststhatdespitetheapparentnumberof live implementations,therewasasyetno resoundingwinnerforbenefits.thisleadsustoconcludethatitstillearlydays. Whatarethemainroadblocksencounteredbyorganizations[Q9]?Figure10showstheresults. Respondentswereaskedtoselectuptothreeoptions,sotheresultsarenotadditive.Themain conclusionappearstobethatitisdifficulttopresentabusinesscase.perhapsweshouldreferback tofigures7and8aboveandviewthisresultinthelightofthestrongleadershipbyit(33%)inmany oftheinitiatives.thereisastrongsuggestionthatlackofclearleadershipfromthebusinessside maybetherootcauseofthedifficultyinpreparingaconvincingbusinesscase.thisisreinforced whenweconsiderthatthesecondmostselectedoptionisthatthereisalackofclearunderstanding ofthebusinessobjectives.onecouldbeforgivenforconcludingthatitwouldbegoodtoapplythe principleof morehastelessspeed tomanyoftheinitiatives.oneisforcedtotheconclusionthat inmanycases,initiativeshavebeenstartedwithoutclearbusinessbuyqinandbusinessgoals.isthe currentmediahypearoundbigdatacausingorganizationstojumptooquicklyandillqpreparedly intotheseinitiatives? IsitpossiblethatorganizationsarejumpingintoBigDatainitiativestooearlyandbeforetheyhave paidadequateattentiontotheirdataqualityanddatagovernance? Copyright 2013TheInformationDifferenceCompanyLtd.AllRightsReserved.

13 Big$Data$Revealed$ 13$ Figure$10$ $The$Main$Roadblocks$for$Big$Data$Initiatives$ WeaskedrespondentstosharewithushowtheircurrentorproposedBigDatainitiativewas positionedwithregardtotheirotherdatamanagementinitiatives[q16].theresultsaresetoutin Figure11. Figure$11$ $Positioning$of$Big$Data$and$Data$Management$Initiatives$ Veryencouragingly,45%reportedthattheirBigDatainitiativeisorwillbeapartoftheirwiderdata managementinitiativessuchasmdm.however,thisresultneedstobeinterpretedagainstthe Copyright 2013TheInformationDifferenceCompanyLtd.AllRightsReserved.

14 Big$Data$Revealed$ 14$ backgroundofwhichdatamanagementinitiativestheyhavealreadyimplemented.weasked respondentstotellusabouttheirdatamanagementlandscape[q17].theresultsaresetoutin Figure12. Figure$12$ $Data$Management$Initiatives$in$Organizations$ Itisverypositivethatmorethanhalfofalltheorganizationshaveinfactimplementeddata managementacrosstheboard.surprisingly,54%toldustheyalreadyhavedatagovernance,which isencouragingsincethisbecamecommonlyavailablemuchlaterthanmasterdatamanagement (MDM)anddataquality.Clearly,wearehalfwaythereandthereisstillalongwaytogoinadopting fulldatamanagement.itisveryencouragingthatsomanyorganizationsseebigdataasintegralto theirotherdatamanagementinitiatives. WhatdoorganizationsbelieveiscriticaltosuccesswithBigDatainitiatives[Q18]?Theresponses aresummarizedinfigure13. Figure$13$ $Critical$for$Success$with$Big$Data$ Copyright 2013TheInformationDifferenceCompanyLtd.AllRightsReserved.

15 Big$Data$Revealed$ 15$ TheresponsesherearereallyallfocusedaroundsecuringbusinessbuyQin.Management understandingandawareness(42%),leadershipatthecxolevel(18%),andasoundandwellq documentedbusinesscase(9%)allunderlinethefactthatthereiscurrentlyalackofsufficientbuyq inandcommitmentfromthebusinessside.thehighlevelofleadershipandinvolvementfromit reportedfurthersupportsthisconclusion.wewouldsuggestthatthebusinesswouldlargelyappear tobe sittingonthefence atthemomentasfarasbigdataisconcerned. BIGDATADOMAINS SowhatdoorganizationsbelieveisBigDataandwhichdatadomainsareofimportanceforthem? WefirstaskedwhetherBigDatawasmainlyfocusedonanalyzingdatacollectedfromwebQbased applications[q5].theresponsesweremixedwithaheftynofrom52%,butwithafurther36% reportingthatitismostlyaboutwebqbasedapplicationdata.some9%consideredbigdatatobe exclusivelyaboutwebqbasedapplicationdata.theresultsareshowninfigure14. Figure$14$ $Is$Big$Data$focused$on$Data$from$Web$Applications$ Morethananything,theseresultsagainsupportthefactthatthereareclearlymultipledriversfor BigData.WethenaskedrespondentstotelluswhichdatadomainswerekeytotheirBigData initiatives[q6].theresultsareshowninfigure15. Figure$15$ $Key$Big$Data$Domains$ Copyright 2013TheInformationDifferenceCompanyLtd.AllRightsReserved.

16 Big$Data$Revealed$ 16$ Respondentswereaskedtoselectalldomainsthatappliedfortheirbusinesssotheresultshereare notadditive.customerdataclearlyisthemostsignificantdomainformostorganizationswith productdatasecondandfinancialdatathird interestingly,bothafterunstructureddata,whichis clearlygainingininterest.theoveralltrendisverymuchinlinewithwhatwehavefoundreported formdmdatadomainsinprevioussurveys. 2 Thereferencetounstructureddatainthepreviousquestionisfurthersupportedbyresponsesto ournextquestionrelatingtowhethertheirbigdatainitiativewasfocusedonstructuredor unstructureddata[q7].theresponsesaresummarizedasfigure16. Figure$16$ $Focus$on$Structured$or$Unstructured$Big$Data$ AscanbeseenfromFigure16,theresponsewasaresounding both.soclearlyorganizationsare notjustinterestedinanalyzingtwitterfeeds. TherehasbeenincreasinginterestinthemediaintheuseofHadoopQtypetechnologiesasthe stagingarea foralldatapriortopassingitontootherdatastoressuchasdatawarehousesor OLAPcubes.WeaskedourrespondentstosharewithustheirviewsonthisuseofHadoop[Q8].A resounding40%claimedthatitisindeedtheirintentiontousehadooptechnologiesasthestaging areaforalltheirdatainfuture(priortoloadingitintodatawarehousesorolapcubes).lessthana thirddonotplantodoso,andaboutoneqthirddoesn tknowatthisstage.thisisakeyresultforthe conventionaletlvendors,whoneedeithertoadapttheirtechnologiesorembracehadoopinthe nearfuture.otherwise,wesuspecttheywillfaceincreasingdifficulties. COSTSANDSIZEOFBIGDATA SohowbigisBigDataandwhatareorganizationsspendinginthisarea?Wefirstlyasked respondentstotellusaboutthemaximumvolumeofdatatheyarecurrentlyhandling(orplanning tohandle)intheirbigdatainitiative[q11].thefeedbackfromtherespondentsissetoutinfigure 17. Fromtheresultsreported,itisclearthatthemajorityofthedataisintherange1to10terabytes. Only5%arereportingdatavolumesinthepetabyterange,withonly4%inthe200terabyteto1 petabyterange.perhapsweshouldbereferringto quitebigdata ratherthanbigdata.itis surprisingthatthemostpopularresultliesinthe1to10terabyterange,suggestingthatcomplexity ratherthansheersizemaybethekeydriver. WethenturnedourattentiontowhatorganizationswerespendingorplanningtospendontheirBig Datainitiatives. 2 MDMMarketResearchSurvey,InformationDifferenceResearchReport,July2012 Copyright 2013TheInformationDifferenceCompanyLtd.AllRightsReserved.

17 Big$Data$Revealed$ 17$ Figure$17$ $Volumes$of$Big$Data$Handled$ WhatisBigDatacosting?WeaskedorganizationswhattheyhavespenttodateontheirBigData initiatives[q13].theexpenditurerangeissetoutinfigure18. Figure$18$ $Expenditure$on$Big$Data$to$Date$ WealsoaskedwhattheyplantospendonBigDatainthecomingyear[Q12]?Theexpenditure rangeissetoutinfigure19.onlysome8%havespentover$5million,while42%havespentless than$1million,25%lessthan$100,000.thissuggeststhatitisstillveryearlyformostbigdata initiativeswithrelativelymodestoutlaystodate.roughlyathirdeitherdon'tknoworhavespent nothing. Turningtotheirplannedexpenditureoverthenexttwelvemonths,11%plantospendmorethan$5 millionwhile32%plantospendlessthan$1million.13%willspendlessthan$100,000.whilethis suggestsaslightincreaseinplannedexpenditureoverthecomingtwelvemonths,itisnotamajor commitment.thisleadsustoconcludethatbigdataisstillintheearlydaysofbeingadopted,with manyorganizationsperhapstestingthetemperatureofthewaterwithrelativelylimited expenditure. ItseemsfairtoconcludethatBigDatahassomewaytogobeforeitreacheseventhefirststagesof maturity. Copyright 2013TheInformationDifferenceCompanyLtd.AllRightsReserved.

18 Big$Data$Revealed$ 18$ Figure$19$ $Planned$Expenditure$on$Big$Data$next$Year$ TECHNOLOGIESDEPLOYED WhattechnologiesarebeingdeployedbyorganizationsfortheirBigDatainitiatives?Weasked respondentstoindicatewhichofthecurrentrangeofsoftwaretoolsforbigdatatheyeitherhad deployedorplannedtouse[q15].theresultsarepresentedasfigure20. Figure$20$ $Software$Solutions$Deployed$(or$Planned)$ Copyright 2013TheInformationDifferenceCompanyLtd.AllRightsReserved.

19 Big$Data$Revealed$ 19$ Therespondentswereaskedtoselectallthatapplied,sotheresultsarenotadditiveanddonot representmarketshare. ThemostpopularthreeappeartobeMicrosoft,Oracle,andTeradata.Mostoftheremainderhave beendeployedorplannedbyasmallernumberoforganizations.itisinterestingtonotethatkey vendorssuchashortonworks,mapr,andjaspersoftdonotfigurehighonthislist.perhapsthisis becauseinthefirstinstance,organizationshaveturnedtotheircurrentsoftwaresuppliersto providebigdatasolutions.weseethisasalikelyfuturetrendwithbusinessexpectingtheirhouse vendor,beitsap,ibm,microsoft,informatica,ororacle,toprovideabigdatasolution. Othersoftwaresolutionsmentionedincluded:AWSRedshift,Cassandra,Causata,Infobright, QlikView,andSAS. ADDITIONALFEEDBACK Finally,weaskedrespondentstoprovideanyadditionalviewsorcommentsthattheyhadinregard totheroleofbigdataintheirorganization[q19].theirextensivefeedbackincluded: Asofnowit(ourBigDatainitiative)isbeingusedasahugearchivalsystemforalltransaction dataforfurtheranalysis. BigDataisnotonlyabouthavingaHadoopcluster BigDataseemstobebuzzwordonlyforus needstobemoreconcrete. Bigdataisandwillbemoreimportant. DatagovernanceandgoodqualityandaccuratedataiskeyfordecisionQmakingprocesses. Idon'tthinkBigDatawillhelporganizationsintheanalyticsarea. Itisallaquestionoffurthertrialanderror ItwillhaveakeyroleinourInformationmanagement. It'sasolutionlookingforaprobleminmanycases. Itisacrucialpartoffinancialreportingandcontrolling. Lotsofmovingpartswithevolvingusecases Sponsorshipneeded. Theabilitytoseparatethe BigNoise from BigData willbecrucialforthesuccess. Toosoontoknowwhattheproperapproachis. Weneedeasyinstructions. Wearestillintheexperimentalstage,learningthetechnologies. Wewilldevelopourowntools. BigDataisaparadigmshiftinITthinkingandbusinessneedstobecontinuouslyinvolvedas implementationpartner. OrganizationswithimmatureBI/DGwilleventuallymakeahuge,costlyerroneousdecisionthat noonecouldhavepredicted alataleb'sblackswanwarning. WenothaveaBigDatainitiativeinternaltotheorganization,butarecreatingofferingsforour customers. Wesimplyhaven tidentifiedausecaseyet itisn'tneeded,andourcurrentprocessessuffice. Difficulttofindagoodbusinessusecaseforitandthenbeabletodemonstratethevalueina costqeffectiveway. ThekeychallengewillbetointegrateBigDatawithexistinginternaldatasources.Sodata integrationanddataqualityarecriticalenablers. Copyright 2013TheInformationDifferenceCompanyLtd.AllRightsReserved.

20 Big$Data$Revealed$ 20$ OurBigDatausecaseexists,butitisanevolutionarystepandrequirescommitmentfromend customerstoimplement. Understandtheopportunities,havealongtermstrategy,futureprooftheorganization,gain executivesupport,createbusinesscaseandjustification,startwithlowhangingfruitlike operationalefficiencies. Asabusinessuser,questionsregardingdatasizearecompletelyirrelevant.I'minterestedin capabilities thetechchoicesarenotofinterest WearetryingtoworkwithourmarketingareaandMedicalInformaticsareatofindan appropriatebusinesscaseforbigdata.wearegoingtolookatweblogsandanalyzelargedata setstodosomehealthcarepredictivemodeling. Thekeygoalispredictiveanalyticsappliedtomachinedata,maintenanceeventdataand operationaldatainordertounderstandandpredictlargefleetperformanceandoptimizethe maintenanceregime. Atthismoment,thereareveryfewpeoplewhocanreallyanalyzetheBigDataparadigm withoutbeingcarriedawaybythehype.itiscrucialtoknowhowit'sgoingtohelpyour particularbusinesscaseinsteadoftryingtobeonthelatestbandwagon. Beforeyoustart,youneedtounderstandhowyouwillmanageBigData,andthescalabilityof yoursolution. BigdatainahealthcaresettingismaybeofadifferentproportionthanBigDataasit'sdefinedin theworldoftelco,socialmedia,etc.,butit'sreallybig. BigDataisanextensionofourBI/DataWarehousingprogram.BigDataextendsexistingoptions byopeningupadepthofdata.bigdatatoolsalsoopenthedoorfornewtypesofanalytical techniques. WeareexploringBigData,andtryingtounderstandhowitmaybeappliedandwhatbenefitsit wouldprovideforourbusiness. BDisnotmature,yetsomebonafidesuccessesaredemonstrated adifficultcombinationto sell.further,westillhavewaystogowithclassicrdbmsdata:dq,dg,etc. Itisclearfromtherespondentfeedbackabovethatthereisadealofskepticismabouttheneedfor, businesscase,viabilityandpracticalimplementationofbigdata. Copyright 2013TheInformationDifferenceCompanyLtd.AllRightsReserved.

21 Big$Data$Revealed$ 21$ CONCLUSIONS Keyconclusionsandrecommendationsresultingfromthesurveyanalysisaresummarizedbelow. Thesehavebeensplitintotwogroups:thoseofdirectrelevancetoenterprisesandorganizations consideringorintheprocessofimplementing(orwhohavealreadyimplemented)bigdata initiatives,andthoserelatingtothesoftwarevendorsandsystemsintegrators(sis). Enterprises$ 30%toldusthattheycurrentlyhavenoplanstoimplementBigData.27%reportedthatthey arecurrentlylivewiththeirbigdatainitiative,andafurther5%areabouttogolive.this compareswellwiththefindingsfromourprevioussurvey(24%),andsuggeststhattherehas beenrelativelylittleprogresssince2012.afurther24%plantogolivenextyear. The27%liveimplementationssplitoutregionallyasfollows:NorthAmerica(includingCanada) 6%,Europe18%,andtherestoftheworld3%.Thisresultfliesinthefaceofconventional wisdomthatsuggeststhatnorthamericaiswellahead. 24%wantstobeabletoanalyzeverylargedatasets.Clearly,thereisawiderangeofbusiness demandsforbigdata.theredoesnotappeartobeadominanttheme. Themainfocusisonimprovingthequalityofdecisionmaking(39%),withmeetingcustomer needsapoorsecond(19%). ITisclearlyinthedrivingseat(33%).Almostaquarter(24%)oftheinitiativesarebeingledby CustomerServices. GiventhepreviousresultitisperhapsunsurprisingthattheCIOisinthedrivingseatas responsibleforandleadingbigdatainitiatives(31%). Some13%claimedtohaveidentifiednewbusinessopportunitieswiththeirBigDatainitiatives. Othersreportedimprovedoperationalefficiency(11%)whileoverhalfoftherespondentsdidn't knowatthistime.thisstronglysuggeststhatdespitetheapparentnumberof live implementations,therewasasyetnoresoundingwinnerforbenefits.thisleadsustoconclude thatitstillearlydays. Themainroadblocktoimplementationappearstobethatitisdifficulttopresentabusiness case. Encouragingly,45%reportedthattheirBigDatainitiativeisorwillbeapartoftheirwiderdata managementinitiativessuchasmdm.morethanhalfofalltheorganizationshaveinfact implementeddatamanagementacrosstheboard(mdm,dataqualityanddatagovernance). 54%toldusthattheyalreadyhavedatagovernance. Managementunderstandingandawareness(42%),leadershipattheCxOlevel(18%)anda soundandwellqdocumentedbusinesscase(9%)areallcitedascriticaltosuccessful implementationofbigdatainitiatives.thissuggeststhereiscurrentlyalackofsufficientbuyin andcommitmentfromthebusinessside. BigDataisnotmainlyfocusedonanalyzingdatacollectedfromwebQbasedapplications. Customerdataisthemostsignificantdomainformostorganizationswithproductdatasecond andfinancialdatathird,bothafterunstructureddatawhichisclearlygainingininterest.so clearlyorganizationsarenotjustinterestedinanalyzingtwitterfeeds. 40%claimedthatitistheirintentiontouseHadooptechnologiesasthestagingareaforalltheir datainfuture(priortoloadingitintodatawarehousesorolapcubes),sothepossibleimpacton enterpriseetlstrategyneedstobeconsidered. Copyright 2013TheInformationDifferenceCompanyLtd.AllRightsReserved.