Make Your Dive into Big Data Rewarding with an Effective Quality Assurance and Testing Strategy

Transcription

1 A Poit of View Make Your Dive ito Big Data Rewardig with a Effective Quality Assurace ad Testig Strategy Data has always bee a itegral part of busiess. Techology trasformatio has further reiforced this importace. The advet of Social Media, Mobility, ad the Iteret of Thigs has ot oly blurred the lies betwee olie ad offlie busiess, but also resulted i the availability of a huge amout of ustructured data related to cosumer behavior ad iterests. Busiesses eed to deal with this data to idetify demad sigals, ad create the right marketig strategy, product mix, ad pricig models ad also compute their Retur o Ivestmet (RoI) from marketig ad advertisig. There is more data tha ever before - for busiesses to facilitate itelliget isights, differetiate, stay ahead of competitio, ad succeed. Ad this is barely a fractio of what Big Data promises to uravel. With big isights, Big Data could ideed mea big busiess for your compay. However, idustry aalysts are echoig growig cocers o data quality. This paper presets a Quality Assurace (QA) solutio to the data quality challege. It attempts a deep dive a exploratio of how a robust ad effective QA strategy ca secure you big returs from Big Data. What s the Big Deal? People have take to social media to discuss what they are doig, where they are doig, as they are doig it. Social media discussios o cosumers experieces with products ad brads have ow become commoplace. With this rapid expasio of social media ad mobility, busiesses have access to large amouts of ustructured data o cosumer iterests, behavior, ad actios. But that is ot really the big deal. The big deal is that with this Big Data, your busiess ca idetify demad sigals, track supply chai, ad meet the right demad, at the right time, with the right price. Big Challeges ad Needs Aalysis Util ow, Moore s Law o doublig of computig power every 18 moths was cosidered a breakthrough. But with over a billio social media posts every two days, Big Data is chagig defiitios ad bechmarks for breakthrough techologies. The eed to stream ad collect real time data from varied data sources ad i differet formats results i a expoetial icrease i volume, resemblig a data tsuami. Further, there is also the eed to cotiuously liste to the stream ad sift out irrelevat data. There is o poit i respodig to a customer tweet after a week, or havig dirty, oisy data affectig busiess decisios. The Midset Chage: Big Data ad Data Warehousig Traditioally, compaies based their busiess strategy o structured data from eterprise applicatios ad relatioal databases. Give Big Data s Variety challege, relatioal databases that store structured iformatio are ill-suited to storig, aalyzig, ad processig Big Data. While data-drive plaig has bee recogized as the best practice i busiess, Big Data is chagig some age-old otios of maagemet. It is ofte cofused with data warehousig. Here are some key differeces. Parameter Data Warehousig Big Data Size Gigabytes Petabytes or Zetabytes Volume Liear Expoetial Format Structured Ucostraied Applicatio Aalyze Existig Busiess Create New Busiess Processig Batch Mode Real Time Streamig Table 1: Data Warehousig ad Big Data

2 While data warehousig provides aswers to busiess problems, Big Data gives itelliget isights o the questios that matter. A iterestig example is a large retail busiess that aalyzed buyig patters for baby products from its sales database. The busiess listeed to real time social media posts, idetified pregat wome customers, ad rolled out persoalized commuicatio with year-log special discouts o baby products ot oly esurig cross sales, but also customer delight ad loyalty. While data warehousig would have typically stopped at sales aalytics, Big Data helped take this a step ahead, providig busiess isight ad itelligece, ad esurig the success of this cross-sales strategy, ultimately resultig i sales trasformatio. From a assurace perspective, we too eed to go the extra mile to assure Big Data quality. While we have well-defied, time-tested assurace ad testig strategies for data warehousig, these traditioal approaches, whe directly applied to Big Data could result i compellig isights that are outright wrog! Big Data testig requires test strategies for structured, semi-structured ad ustructured data. Other pre-requisites iclude statistical tests o data sets, a optimal ad scalable test eviromet, ad the competecy to work with orelatioal databases. We eed a efficiet assurace framework for Big Data. The Big Ecosystem Big Data comes from various sources such as databases, eterprise systems, social media, global positioig systems (GPS), radars, radio frequecy idetificatio devices (RFIDs) or satellites. The data is processed through a multi-ode MapReduce processig layer, which resides i a file system istead of a structured database. The processed data is the loaded oto a aalytical egie, which trasforms it to a readable format for busiess reportig. Exteral Data RDBS File System Data Ad hoc Reportig Structure Data ETL Data Warehouse Aalytical Radar MR Process GPS Ustructure Data ETL Egie Social Media Data File System Data Dashboard Reportig Figure 1: The Big Data Ecosystem 2

3 The Four Pillar Framework As is clear from the ecosystem represetatio, Big Data processig ivolves itroducig ew steps to the lifecycle that icrease the complexity of the process. This complexity itroduces four types of iaccuracies i the Big Data processig cycle: 1. Wrog data sources durig requiremets plaig, resultig i data mismatch ad wrog iputs for processig 2. Data iaccuracies resultig from the multi-mode MapReduce process that operates across file systems, requirig adequate cofiguratio testig 3. Codig defects detected after comparig results of the trasformatio ad busiess logic 4. Process ad data gaps i the visualizatio layer These iaccuracies ca impact the quality of Big Data aalytics, resultig i iaccurate isights ad wrog busiess decisios. A efficiet testig strategy, comprisig adequate quality checks ad cotrols is required to mitigate such risks. If left to ad-hoc implemetatio, some of these cotrols may be missed. I additio, such implemetatios may appear complex ad dautig. It is therefore importat to maage these complexities ad address the challeges with a robust quality assurace framework that is easy to implemet, without compromisig o the cotrol rigor. Here is oe based o four pillars that adequately supports quality assurace teams, addresses specific challeges, ad reduces the complexity of Big Data processig to make it simple, ot dautig: 1. People: Right assurace team with the right skills, ad well-defied roles ad resposibilities 2. Process: Proper plaig, ivolvemet of assurace at the outset, with the right assurace cotrols, data validatio, ad other checks ad balaces 3. Ifrastructure: Established ad scalable test eviromet ad iterface support 4. Automatio: Right-sized automatio scripts ad testig tools The First Assurace Pillar: People Like ay IT implemetatio, i Big Data too, quality is the resposibility of all stakeholders ad ot just QA teams. Effective implemetatio requires focus o data quality, which i tur, calls for additioal quality assurace roles i the four phases of the Software Developmet Lifecycle (SDLC): SDLC Phase Roles Resposibilities Requiremets, Plaig Data Steward Tester Gather data validatio ad fuctioal requiremets Defie Service Level Agreemets (SLAs) Idetify Key Performace Idicators (KPIs) Desig ad Strategy Data Aalyst Testers Documet system requiremets Defie data validatio ad testig strategy Develop test scearios ad test cases Developmet ad Itegratio Automatio Testers Create test data ad automatio scripts Execute test automatio scripts Productio Support Data Scietist Moitor SLAs ad KPIs Clea iput data ad aalyze outliers Table 2: Roles ad resposibilities of people across the SDLC 3

4 The Secod Assurace Pillar: Process Big Data processig ivolves specific activities across four phases. The complexity of these activities itroduces some iaccuracies i the process output. It is importat to address these iaccuracies early i the cycle, through a effective assurace ad testig strategy simply because the cost of fixig defects i productio is high. The secod pillar of the framework a process based approach - helps idetify iaccuracies to improve the overall effectiveess of the project. Phase Iaccuracies Process Cotrol ad Check Deployed Sourcig Processig Aalyzig Visualizig Failure to itroduce the right data sources results i icomplete or mismatched data for reportig Multi-mode MapReduce process across file systems ca result i cofiguratio iaccuracies Codig defects result i misiterpretatio ad wrog implemetatio of data Process gaps result i compromised data security ad accuracy Rigorous testig of source data with detailed data compariso ad iput file cout Emphasis o failover testig ad real time processig of data from the 'Sourcig' phase Rigorous code testig ad data recociliatio betwee trasformatio ad busiess logic Focus o security compliace ad testig, ad authorizatio rules at reportig level Table 3: The process checks eeded across phases The Third Assurace Pillar: Ifrastructure ad the Role of the Cloud The success of Big Data testig depeds o executig the right test scearios, which i tur, demads the right testig eviromet. Further, testig teams have limited budget ad tight schedules. The right (ad scalable) testig eviromet eeds to be provisioed quickly withi budgetary ad schedule costraits. Cloud based ifrastructure plays a big role here. A combiatio of public, private ad hybrid cloud ifrastructure ca adequately address the computig, storage, software, security ad scalability requiremets with zero or miimal upfrot ifrastructure cost - a big boo for testig teams with limited budget. Cloud Model Software as a Service (SaaS) Platform as a Service (PaaS) Ifrastructure as a Service (IaaS) Beefits Provisios software without upfrot licese cost Provisios the right eviromet without startig from scratch Provisios optimal storage for volumious data Table 4: Cloud models ad their beefits The Fourth Assurace Pillar: Automatio Automated testig is the best way to icrease effectiveess, efficiecy, ad coverage i Big Data testig. Real-time listeig demads that testig teams be agile. The fourth pillar proposes that automatio tests be coducted i parallel with developmet. Automatio scripts ad tools for data profilig, data validatio, test data geeratio, test data base-liig, predictive aalysis ad modelig esure the correctess of the implemetatio ad drive sigificat cost savigs. Spread across all types of testig (uit, system, ad fuctioal), automatio tools further stregthe the fourth pillar. Preseted below is a quick deploymet referece: 4

5 Tool Purpose Proposed Beefits Data Profilig Data Geeratio Aalytics Data Validatio Idetify iaccuracies i source data ad create test data subsets for scearios Create sythetic (productio-like) test data to speed up testig Perform predictive aalytics ad modelig to create proper test data baselie Esure correctess with validatio tests ad recociliatio of data sets Reduced Cost of Data Quality with reductio i rework due to improved data quality Improved Productivity with lower test data geeratio effort Quality Upscale due to reductio i data defects Process Efficiecy with sigificat improvemet i test optimizatio Table 5: Tools ad how they help Big Beefits The four pillars, whe workig i uiso, offer the potetial to drive huge cost ad effort savigs, ad also wi user trust. Some of the features iclude: Clear defiitio of roles ad resposibilities Iclusio of testig roles at the outset Regular moitorig by data scietist Icreased collaboratio betwee teams Testig strategy defied early i the process Early availability of test cases ad test scripts Cloud based, scalable ifrastructure Beefits Improved productivity Defect prevetio ad early flaggig of defects Reduced cost of data quality Zero or miimal upfrot ivestmet Quick creatio of testig eviromet Agility of testig teams Scalable storage ad computig ifrastructure Faster turaroud time Use of automatio ad testig tools The Big Takeaway The recommeded approach is iovative ad effective to cruch large volumes of Big Data. It ca help determie cosumer buyig patters ad icrease sales with Also Buy suggestios, aalyze crime treds to predict (ad prevet) crime, associate climate ad crop health data for better crop cultivatio, or eve predict the outbreak of epidemic through meas that are faster tha hospital admissio records. Imagie the disaster that could result from basig all these decisios o iaccurate or flawed data. For the best results from Big Data, leverage the framework ad its four assurace pillars. With effective QA, you ca use Big Data isights to pre-empt competitio, explore ew markets, predict cosumer behavior, drive marketig ad improve brad value - importat factors to busiess bottom-lie ad agility. About the Authors Tom Edwards, Seior Vice Presidet, Techology, Nielse With over 18 years i Software Developmet ad Quality Assurace, Tom has vast exposure i Busiess Itelligece Testig methodologies. Raji Sacha, Busiess Relatioship Maager, Tata Cosultacy Services Ltd. Raji has spet 14 years i Busiess Itelligece ad Quality Assurace, implemetig strategic ad complex IT trasformatio programs across domais. 5

6 About TCS' Assurace Services Uit With oe of the most comprehesive portfolios of idepedet test capabilities o offer, TCS addresses both busiess ad quality challeges for its global cliets. We empower orgaizatios across domais to optimize overheads, realize first mover advatage ad improve customer satisfactio. TCS offers assurace services across the testig value cycle, icludig test cosultig ad advisory, test services implemetatio, ad maaged services for test eviromet ad test data maagemet. We cotiually redefie testig ad QA paradigms to help our cliets stay ahead of the curve. Our library of domai-based reusable busiess fuctios ad prove egagemet model fouded o the twi pillars of product ad process quality eable us to deliver certaity to our cliets. Over 28,000 testig cosultats, strategic alliaces ad parterships with key product vedors, more tha 60 dedicated test ceters of excellece ad our iovatio labs power our tailor-made solutios, testig assets ad accelerators. With specialized test eviromets ad labs, TCS drives the delivery of assurace i a o-disruptive, agile, ad automated maer, makig the etire developmet lifecycle more efficiet. Cotact For more iformatio about TCS Assurace Services Uit, Visit: global.assurace@tcs.com Blog: #ThikAssurace About Tata Cosultacy Services Ltd (TCS) Tata Cosultacy Services is a IT services, cosultig ad busiess solutios orgaizatio that delivers real results to global busiess, esurig a level of certaity o other firm ca match. TCS offers a cosultig-led, itegrated portfolio of IT ad IT-eabled ifrastructure, egieerig TM ad assurace services. This is delivered through its uique Global Network Delivery Model, recogized as the bechmark of excellece i software developmet. A part of the Tata Group, Idia s largest idustrial coglomerate, TCS has a global footprit ad is listed o the Natioal Stock Exchage ad Bombay Stock Exchage i Idia. For more iformatio, visit us at IT Services Busiess Solutios Cosultig All cotet / iformatio preset here is the exclusive property of Tata Cosultacy Services Limited (TCS). The cotet / iformatio cotaied here is correct at the time of publishig. No material from here may be copied, modified, reproduced, republished, uploaded, trasmitted, posted or distributed i ay form without prior writte permissio from TCS. Uauthorized use of the cotet / iformatio appearig here may violate copyright, trademark ad other applicable laws, ad could result i crimial or civil pealties. Copyright 2015 Tata Cosultacy Services Limited TCS Desig Services I M I 11 I 15