Big Data Adoption An Iterative Approach to Harness the Power of Big Data

Transcription

1 White Paper Big Data Adoptio A Iterative Approach to Haress the Power of Big Data The emergece of every ew techology brigs its ow share of Wow! ad Aha! factors. While these factors add value to the techology developmet process, they ca also lead to beefits ad limitatios of the techology remaiig uevaluated before adoptio. If these factors are ot examied properly, they ca subsequetly derail the etire program ad sed it back to the drawig board. While Big Data appears to be a techology-drive movemet, its strategic importace requires special focus ad attetio durig its adoptio, because Techology to support Big Data is evolvig rapidly ad will take some time to mature. Effective utilizatio of Big Data requires a chage i midset regardig the way it is used. While other techologies help solve problems like streamliig the ivetory maagemet process or providig a olie system that eables the order shipmet trackig o realtime basis, Big Data helps fid the problems that require attetio; for example, the combiatio of factors that causes defects i the maufacturig process, or the factors cotributig to the sales differetial betwee two stores.

2 While the three Vs (i.e., Volume, Velocity ad Variety) have bee regularly used to defie Big Data, the use of techologies that support Big Data are ot cofied oly to the way the three Vs are defied. Their usage has a much broader scope whe correctly aalyzed from the orgaizatio s perspective. Much like other evolvig techologies, it may be useful to first coduct small-scale projects i Big Data, for a better uderstadig of the techology ad the busiess areas that may beefit from it. While these projects (more like proofs-of-cocept) are good as a startig poit, the maistream adoptio of Big Data requires a structured framework. This is because the solutio space i this area comprises ot oly large, isolated ad varied data sets, but also a rapidly evolvig techology ladscape. Tryig to get all the requiremets defied at oe go ad selectig the techology of choice at the outset ca potetially derail the etire program ad leave the orgaizatio with a dead ivestmet. The paper addresses this eed by presetig a systematic ad focused approach to be followed while adoptig Big Data. 2

3 About the Authors Ajay Parashar With a Master s degree i Busiess Admiistratio ad a Bachelor s degree i Idustrial ad Productio Egieerig, Ajay Parashar is a Solutio Architect ad the Big Data Lead for the TCS HiTech Idustry Solutio Uit s Domai group. Durig the last year, Ajay has bee ivolved i busiess developmet projects aroud Big Data, ad has overall experiece of 12 years i Database Admiistratio, Project ad Program Maagemet ad Delivery Maagemet. 3

4 Table of Cotets 1 Itroductio 5 2 Adoptio Framework 5 Data Discovery 7 Aalytics Discovery 7 Tools ad Techology Discovery 8 Ifrastructure Discovery 9 Implemetatio 10 3 Coclusio 12 4

5 Itroductio Big Data, the ew buzz word i the IT space, has captured the imagiatio of may orgaizatios, makig its way to the strategy desk i the relatively short time it has bee i existece. The web space, the aalysts ad aybody-who-matters are talkig about the immese beefits that the haressig of Big Data ca brig to a orgaizatio or to society at large, by tappig ito iformatio that has so far lai hidde. With rewards i billios of dollars, Big Data is a highly attractive ivestmet. While there are beefits to be gaied, like usig market buzz to reduce the time take to chage product features or the product mix, eablig targeted promotio ad promptig shorter aalysis cycle time, the picture aroud Big Data adoptio is still fuzzy as is usually the case with ay ew emergig techology. Coceptually, Big Data is defied by the size of data, but is this classificatio aloe sufficiet to aswer the questios that most orgaizatios ask? For istace, questios like: What data is to be aalyzed? Which techology is to be used? What is the quatum of ivestmet to be made? What should be the eterprise adoptio approach towards Big Data? Aswers to these questios are ot easy to come by, if we rely o the established system developmet approach as prescribed by the Waterfall ad Agile models. The very premise of these models havig a set of requiremets does ot exist i the case of Big Data. Some may argue that the requiremet is to haress Big Data, which is ot etirely wrog; however, this is similar to startig a project with the stated requiremet of poverty eradicatio a requiremet too broad ad sweepig to be easily fulfilled. It is importat to start with a clea slate while approachig the idea of Big Data, cotiuously buildig o it without discardig aythig created i the process. Adoptio Framework A key factor i the success of ay ew program is the way it is approached from the iceptio phase itself. Ay Big Data program that requires the itegratio of data with strategic plaig is goig to be critical ad will carry heavy pealties i case of failure. The right framework to eable the adoptio of Big Data aalytics withi the orgaizatio must be adopted. The critical compoets of this framework iclude: Data discovery Aalytics discovery Tools ad techology discovery\ Ifrastructure discovery Implemetatio 5

6 Data Discovery Structured Data Semi-Structured Data U-Structured Data Aalytics Discovery Data correlatio Algorithm Frequecy Tools ad Techology Discovery Ope Source v/s Proprietary Packaged Implemetatio Data Iterfaces Ifrastructure Discovery Capacity Plaig Commodity H/W v/s Appliace Virtualizatio Discovery Implemetatio Orchestratig Data, tools ad ifrastructure Figure 1 : Big Data Adoptio Framework Ulike a covetioal aalytics solutio that starts with a problem statemet ad the fids differet data sources to address the problem, Big Data aalytics must be studied from a completely differet perspective. For istace, the problem a orgaizatio is tryig to solve is ukow, because had it bee kow, the relevat solutio or aswer would already have bee i place ad we would ot be lookig at Big Data for a solutio. For example, uderstadig cosumer setimets is as good a problem statemet as someoe askig for the cout of galaxies i the uiverse! The value of Big Data aalytics lies i brigig together differet data sources i a cost-efficiet maer ad providig a eviromet that eables questioig the itegrated data set. Comig back to the imagied questio o cosumer setimets, Big Data helps ot just i uderstadig the setimets but also to develop a deeper uderstadig of the factors causig a particular setimet that is, the busiess problem to be solved. 6

7 Data Discovery For ay orgaizatio lookig to Big Data for a solutio, the first ad most importat step is to defie the visible sources of data. Eterprise data warehouses, Olie Trasactio Processig (OLTP) systems, the Customer Relatioship Maagemet (CRM) system, the Eterprise Resource Plaig (ERP) platform etc. are the usual sources of data. However, the selectio should ot be limited to these aloe. There may be other data sources that are very operatioal i ature ad may ot be see as importat from the Busiess Itelligece perspective. These iclude salespersos daily logs (if there are ay), machie operators logs or maual logs maitaied by security persos. Brigig them o a commo platform ad itegratig them with other data sources may reveal iformatio that was hitherto ukow. For example, itegratig sales ad CRM data with web applicatio logs may help uderstad the decisio matrix followed by a customer before makig the purchase decisio. Data discovery is the most crucial part of the etire exercise, as it defies the subsequet solutio desig ad developmet. Creatig a Big Data eviromet is a ever-evolvig process, with the additio ad itegratio of each ew data source further stregtheig the existig aalytics operatios ad brigig to light ew problem areas to be researched. Aalytics Discovery Idetifyig data sets ad puttig them together is oe part of the Big Data adoptio roadmap. The real challege is to mie the data for iformatio of busiess importace. The foudatio of this process is derived from the existig busiess itelligece dashboards ad reports. You must idetify the vital dashboards ad start askig the Big Data repository questios i order to obtai isights ito the what ad why part of the dashboards. For example, examie the regioal sales data, ad try to dig for iformatio that first addresses the what of the sales differece; as a ext step, uderstad the why part of the differece. This is a evolutioary process that moves from vailla dashboard reportig to complex decisio-support predictive aalytics, while stregtheig attribute idetificatio ad establishig correlatios betwee them. The aalytics discovery process will: Create a elaborate aalytics ladscape touchig upo the critical busiess fuctios that play a importat role i the top-lie ad bottom-lie of the orgaizatio, Establish the relatioship betwee differet data sources; this will help uderstad the effect of their mutual iteractio, ad Create aalytics modelig that meshes with the orgaizatio s goals ad objectives, evolvig cotiuously with the idetificatio ad additio of ewer data elemets. 7

8 Tools ad Techology Discovery Havig started with the idetificatio of the data set ad defiig the aalytics requiremets from the data set, the ext step is choosig the right mix of techology ad tools. This is crucial for the success of ay Big Data program, for the followig reasos: a) No two busiess problems are the same; each requires a differet treatmet. For example, the aalysis of structured data might be more efficiet ad less complex i a fie-tued Relatioal Database Maagemet System (RDBMS) drive setup as compared to a setup drive by NoSQL databases or ope source commuity-provided parallel processig egies such as Apache Hadoop. b) Each idetified data set may eed to be hadled differetly, especially durig the iitial stages, i order to clear the oise ad make it presetable to the aalytics egie. c) The core techology drivig Big Data aalytics (such as Apache Hadoop) ad other aalytic ecosystem tools are still evolvig. I the curret techology ladscape that addresses the aalytics requiremets of Big Data, there are multiple products, solutios or frameworks. Names such as Hadoop, MapReduce, R, Mahout, SAS, Flume, Lucee etc. are frequetly referred to i discussios about Big Data. However, all these compoets address specific areas: for example, Lucee is associated with search fuctios, Mahout with machie learig to automate routie system decisio-makig ad Hadoop with parallel processig of large amouts of data. Pickig a tool is, therefore, depedet o the outcome of the earlier two steps, i.e., Data Discovery ad Aalytics Discovery. For example, the techology requiremets for aalyzig web log data o a real-time basis are completely differet from what is required to aalyze ustructured data to access the setimets of customers. There is o sigle techology solutio that fits all aalytics requiremets. Ivestig upfrot i oe techology or a tool might ot be coducive to evolvig a robust Big Data aalytics platform. A focused ad detailed evaluatio of tools ad techologies is required, based o the outcome of the earlier two phases. Aother poit to cosider durig techology evaluatio is whether to opt for a packaged solutio from a established OEM, or a custom-built solutio from ope source products. While most of the techologies drivig the Big Data movemet have their origi i the ope source commuity, these techologies are beig rapidly adopted by established OEMs such as EMC, IBM, Teradata ad Oracle. These are also beig budled i the form of Big Data appliaces. The guidig pillars i this decisio makig process should be the velocity of the Big Data adoptio program, the maturity of the aalytics model, the busiess value realizatio ad the appetite for capital expediture. This is oe of the most importat phases i ay Big Data adoptio program as this will have a direct budgetary impact. This is also a complex phase sice the techologies ad solutios are still evolvig ad are far from mature ulike, say, a data warehouse program or a web developmet program. Uder the circumstaces, the ideal approach will be to take small steps with a techology compoet ad validate its busiess worthiess. 8

9 Ifrastructure Discovery With orgaizatios today reelig uder the operatioal overheads of maitaiig multiple data ceters, idetifyig ad defiig ifrastructural requiremets for a Big Data eco-system is importat, to avoid the additio of further complexity to the IT ladscape ad additioal capital ad operatioal expeses. The ifrastructural decisios to be take ca be grouped ito three distict categories: a) Server Hardware: The decisio o server hardware is depedet o the followig factors: a. Data characteristics icludig its source, privacy ad security aspects b. Nature of the aalytics operatios, i.e., regular aalysis to support areas such as risk assessmet of a customer for isurace policy issuace, i-store product placemet decisio-makig, setimet aalysis from social chaels, periodic aalysis to support strategic decisio-makig such as customer segmetatio while fializig the product lauch program, ew brach/store opeig decisio-makig etc. c. Existig data ceter capacity, maturity of the server virtualizatio program ad opeess to cloud eviromet b) Storage Ifrastructure: Dealig with large quatities of data utilized maily i a read-oly maer will require closer examiatio of the storage solutio chose to support the Big Data platform. The choice betwee Network Attached Storage (NAS), Storage Area Network (SAN), Redudat Array of Idepedet Disks (RAID) or O-server storage requires oe to scrutiize the beefits of each ad compare them agaist the performace impact. By its very ature, the core techology drivig Big Data relies o parallel processig of data spread across multiple machies. The reductio i I/O bottleecks as a result of local processig (each server processig data stored withi it) plays a very importat role i esurig the promised performace beefits are delivered. Our earlier paper titled Scale-out Storage Systems provides isights about storage ifrastructure decisio-makig. c) Networkig Setup: A Big Data aalytics eviromet will ivolve aalytics computatio o large quatities of data spread across multiple systems ad accessed i a radom maer. The decisio to have a separate etwork for the Big Data eviromet istead of leveragig the existig eterprisewide etwork should be weighed agaist the busiess risk associated with eterprise applicatio performace degradatio due to etwork badwidth utilizatio by Big Data aalytics operatios. 9

10 Lift Chart 100% 90% 80% Cloud IAAS Cum % of Respose 70% 60% 50% 40% 30% 20% 10% 0% Cum % of Customers Cum % of Customers Cum % of Respose Supplier Social Aalytics Vedor Retail Brach Office - Potetial Market Services Wholesale Maufacturig BTBL GSL OL Fiace Trasport & Commuicatios ERP Eterprise DWH CRM OLTP Logs Big Data Aalytics Eviromet A key poit to cosider ad evaluate here is whether to utilize Ifrastructure as a Service provided by cloud vedors such as AWS ad Microsoft Azure. A cloud-based eviromet may be ideal to aalyze social data geerated from Facebook, Twitter ad weblogs, itegratig the resultat aalytics with the ihouse Big Data eviromet data elemets comig from systems such as ERP ad CRM. However, this must be aalyzed as per the aalytics operatios surroudig social data ad the level of expected correlatio with i-house data elemets. Implemetatio Figure 2: Represetative Big Data Eviromet The Big Data aalytics adoptio program is a iterative program, with each iteratio addressig a specific busiess area ad slowly evolvig ito a comprehesive platform that caters to the aalytics requiremets of the orgaizatio. The iterative adoptio framework described above is supported by a iterative implemetatio pla as depicted below. 10

11 Developmet/ result verificatio at differet levels Develop ad Verify Solutio desig at differet levels Solutio desig Implemetatio Solutio Collaborative, facilitated workshops for settig the expectatios Refie solutio Data Idetificatio Team Data Project Stat-up Implemeted Solutio Team composed of customer represetatives, egieers, computer scietists, statisticias, domai experts ad program maagers Idetificatio of data/related logistics Figure 3: Implemetatio Methodology The iterative ature of the solutio implemetatio helps i a umber of ways ad is strogly recommeded for ay Big Data adoptio program. Big Data programs usually begi with a clea slate approach, where the outcome is ot kow ad is difficult to defie. The selectio of the aalytics model ad tools/techiques is based o certai assumptios made by examiig the idetified data elemets. Oce implemeted, the outcome of result iterpretatio may shed ew light o these assumptios ad require corrective actio. The eed to aalyze ad itegrate ewer data elemets ad data sources will chage over a period of time, accordig to the strategic directio of the busiess. Havig a iterative approach allows for this seamless itegratio. The framework or the adoptio approach calls for a cotiuous improvemet exercise carried out i a iterative maer, with each iteratio critically examiig the achievemets of the past iteratio ad weighig them agaist the busiess beefits extracted from the exercise. New data sources added durig a iteratio ca potetially reder a earlier developed aalytics model useless. Without refiemet, a aalytics model will ted to lose its sigificace i the log ru. 11

12 Coclusio I coclusio, a Big Data adoptio program should be viewed as a holistic program that is drive iteratively over a period of time by critically examiig the assumptios made i earlier iteratios, ad weighig the busiess beefits derived from the results as well as the level of maturity/stability achieved with respect to broader busiess strategic objectives. The differet phases, i.e., data discovery, aalytics discovery, tools ad techology discovery, ifrastructure discovery ad implemetatio, whe carried out o a cotiuous basis, will help establish a robust Big Data aalytics eviromet that supports the strategic decisio-makig process by augmetig it with data-cetric aalysis ad predictios. Data Discovery Implemetatio Aalytics Discovery Ifrastructure Discovery Tools ad Techology Discovery Figure 4: Iterative Adoptio Approach Refereces Big Data: The ext frotier for iovatio, competitio, ad productivity, McKisey Global Istitute, May 2011, accessed Feb 11, Big Data ad Advaced Aalytics: Success Stories From the Frot Lies, Forbes, Dec 2012, accessed Feb 11, 2012, Briig Big Data to the eterprise, IBM, accessed Feb 11, 2012, Aalytics: The real-world use of big data, IBM Istitute of Busiess Value, accessed Feb 11, 2012, Beyod the Hype of Big Data, CIO.com, October 2011, accessed Feb 11,

13 About TCS HiTech Idustry Solutio Uit TCS' HiTech idustry Solutios Uit provides optimal, customized, ad comprehesive solutios across varied High Tech idustry segmets: Computer Platform ad Services Compaies, Software Firms, Electroics ad Semicoductor Compaies, ad Professioal Services Firms (Legal, HR, Tax & Accoutig ad Cosultig & Advisory/Aalyst firms). Buildig o its vast experiece i egieerig, busiess process trasformatio, iovatio ad IT solutios, TCS offers a comprehesive portfolio of services that maximize growth, maage risk, ad reduce costs. The TCS HiTech Idustry Solutio Uit parters with High Tech eterprises to provide ed-to-ed solutios which help realize operatioal excellece, iovatio ad greater profitability. For more iformatio, visit us at Cotact For feedback o this article ad more iformatio, please cotact us at: HiTech.Marketig@tcs.com Subscribe to TCS White Papers TCS.com RSS: Feedburer: About Tata Cosultacy Services (TCS) Tata Cosultacy Services is a IT services, cosultig ad busiess solutios orgaizatio that delivers real results to global busiess, esurig a level of certaity o other firm ca match. TCS offers a cosultig-led, itegrated portfolio of IT ad IT-eabled ifrastructure, egieerig TM ad assurace services. This is delivered through its uique Global Network Delivery Model, recogized as the bechmark of excellece i software developmet. A part of the Tata Group, Idia s largest idustrial coglomerate, TCS has a global footprit ad is listed o the Natioal Stock Exchage ad Bombay Stock Exchage i Idia. For more iformatio, visit us at IT Services Busiess Solutios Outsourcig All cotet / iformatio preset here is the exclusive property of Tata Cosultacy Services Limited (TCS). The cotet / iformatio cotaied here is correct at the time of publishig. No material from here may be copied, modified, reproduced, republished, uploaded, trasmitted, posted or distributed i ay form without prior writte permissio from TCS. Uauthorized use of the cotet / iformatio appearig here may violate copyright, trademark ad other applicable laws, ad could result i crimial or civil pealties. Copyright 2013 Tata Cosultacy Services Limited TCS Desig Services I M I 03 I 13