Creating an Agile BI Environment

Transcription

1 White Paper Creatig a Agile BI Eviromet The rapidly chagig IT ecoomy has iflueced the Busiess Itelligece (BI) systems to look at iovative ways to be equally fast ad flexible. There is a eed to be more ituitive ad quick i implemetatio so as to adapt to the chagig eviromet. Oe of the ways by which orgaizatios ca achieve these goals is by usig Agile based BI developmet models. There are may compoets i a successful BI solutio which iclude data itegratio, aalytics, data quality, metadata maagemet, eterprise data warehouse, dashboards ad so o. Each of these compoets are critical for a orgaizatio, ad stakeholders are ready to ivest i these. The oly issue is how quickly we ca provide these solutios ad how flexible these solutios are with the chagig demads. Traditioally, we have bee usig the waterfall SDLC model for BI implemetatios which ecourages gettig requiremets clarity i the iitial phases of the projects ad havig distict deliverables for each phase. Though this approach has bee successful i the past, the BI practitioers recogize that busiess requiremets are ot static ad we must be able to effectively mould the deliverables based o chagig requiremets. Hece, we caot cotiue with the Waterfall project maagemet approach that is either fast or flexible. Applyig the cocepts of agile developmet to BI is the ituitive way forward. The aim of this paper is to provide a backgroud o agile project maagemet & developmet techiques, ad suggest some guidelies ad best practices which ca help i successful Agile BI implemetatios.

2 About the Author Bhawa Rehai Bhawa Rehai has seve years of experiece i the IT idustry. She has worked o may projects i various verticals such as HiTech, Bakig, Fiace, Airlies, Healthcare, ad so o. She is curretly workig as part of Cetre of Excellece Group for the HiTech ad Professioal Services segmet ad has bee ivolved i providig techical solutios for HiTech ad PS customers. Bhawa holds a Masters of Techology (M.Tech.) degree i Iformatio Techology from IIITM, Gwalior 2

3 Table of Cotets 1. Itroductio 4 2. Backgroud 4 Itroductio to Agile Software Developmet 4 Agile vs. Waterfall 5 Value Additio of Agile i BI implemetatios 7 Metrics 8 Refactorig 10 Gradual Movemet towards Agile 10 Team Structure 11 Factory Model Agile Beefits Choosig betwee Agile ad Waterfall Agile BI@ TCS 13 BIDS BI Rapid Delivery Framework 13 BIDS BI Competecy Cetre 14 TCS Tools Coclusio 14 3

4 Itroductio The recessio ad ecoomic turmoil i the past few years have forced the developers ad sposors to recosider the developmet ad delivery of BI solutios. Low cost ad quicker deploymet models are the eed of the busiess hour. Hece, we eed to thik beyod the traditioal software developmet approaches ad look for solutios which brig i more capabilities ad provide faster deploymets. This ca certaily be achieved usig the Agile approach for BI implemetatios. This paper attempts to address few questios which will help i Creatig a Agile BI Eviromet. The documet gives isight o the followig questios: What is Agile? Why Agile for BI? How is it implemeted? Whe to move to Agile? How is the Agile performace measured? The paper gives a backgroud o Agile ad specifically how Agile helps i BI implemetatio. It explais various agile methodologies applicable to BI. It also suggests the best practices that ca be followed i a Agile BI implemetatio. This paper also touches upo TCS's offerigs that aid Agile developmet of BI solutios. Backgroud Itroductio to Agile Software Developmet Agile is a iterative software developmet methodology which ecourages icremetal ad cotiual developmet of the product. The requiremets ad desig of the software evolve through close collaboratio betwee the sposors ad the developmet teams. I this methodology, the requiremets are broke i to small user stories which should be idepedet, egotiable, valuable, estimable, small ad testable. Iteratios or Sprits are short time frames that typically last from oe to four weeks. I each iteratio, the team works through a full software developmet cycle icludig plaig, requiremets aalysis, desig, codig, uit testig, ad acceptace testig. I acceptace testig, a workig product is demostrated to stakeholders. This miimises the overall risk ad allows the project to adapt to chages quickly. Documetatio is produced as ad whe required. A Sprit should add some valuable features that are bug free ad ready to be deployed to Productio. Productio release may be doe oce a group of features are added to the Product ad ca be used by the Sposors. Agile developmet use commuicatio as a importat tool. All the Agile meetigs are face-to-face where the mai goal is to collaborate ad take decisios as a team. This face-to-face commuicatio 4

5 exposes ay issues as they rise ad icreases cofidece i the sposor commuity. Agile also ecourages thought partership amogst the customers ad vedors. Agile developmet emphasises workig software as the primary measure of progress. This, combied with the preferece for face-to-face commuicatio, produces less writte documetatio tha other methods. The agile method ecourages stakeholders to prioritise wats with other iteratio outcomes based exclusively o busiess value perceived at the begiig of the iteratio. The most widely used methodologies based o the agile philosophy are Scrum ad Extreme Programmig (XP). XP: XP is a highly collaborative methodology that cocetrates o Product developmet rather tha maagerial aspect. A user or a represetative is geerally a part of the team, so that he or she ca add details to requiremets as the software is beig built. This eables the users ad developers to evolve the requiremets to defie the ed product. Whe the product has eough features to satisfy users, the team termiates iteratio ad releases the software. If users decide that eough user stories have bee delivered, the team ca choose to termiate the project before all of the origially plaed user stories have bee implemeted. Scrum: Scrum for software developmet came out of the rapid prototypig commuity because prototypers wated a methodology that would support a eviromet i which the requiremets were ot oly icomplete at the start, but also could chage rapidly durig developmet. Ulike XP, Scrum methodology icludes both maagerial ad developmet processes. Scrum projects revolve aroud a Product Backlog of the pedig work. It cosists of a prioritised list of stories which the team picks oe by oe. Each story is the desiged, developed, tested ad released to the users. Whe eough of the backlog has bee implemeted so that the ed users believe the release is worth puttig ito productio, maagemet closes developmet. Agile vs. Waterfall Differece betwee Waterfall ad Agile ca be appreciated by lookig at the uderlyig policy for each of these: The maifesto for waterfall software developmet is [5]: Software developmet ca be equated to ay other egieerig task. We believe software developmet projects ca be effectively maaged by: Uderstadig ad writig specificatios that defie how the software will look ad what it will do Performig i-depth aalysis ad desig work before estimatig developmet costs Esurig software developers follow the specificatios Testig the software after implemetatio to make sure it works as specified Deliverig the fiished result to the user. That is, if the specificatio is of sufficiet detail, the the software will be writte such that it will satisfy the customer, will be withi budget, ad will be delivered o time. 5

6 I compariso to Waterfall, below is the maifesto for Agile [4]: We are ucoverig better ways of developig software by doig it ad helpig others do it. Through this work we have come to value: Idividuals ad Iteractios over processes ad tools Workig software over comprehesive documetatio Customer Collaboratio over cotract egotiatio Respodig to chage over followig a pla That is, while there is value i items o the right, we value the items o the left more Few key differeces betwee the two approaches are listed below: Deliverables Flexibility Visibility Desig Risk Waterfall Distict phases ad documeted deliverables pre defied for each phase. Requiremets are froze i first phase of requiremet gatherig, rest of the phases use these requiremets as base. Ay chages after that have to go through a process of CR Oce the requiremets are fialised, the users are ot ivolved i the developmet cycle. They are ot aware of what is beig developed, till it is exposed for UAT after full iteratio of 90 days Data model ad ETL jobs are desiged keepig i mid all the requiremets Loger cycle ad less visibility to customer may result i ed product which do ot meet user requiremets Agile Iteratios rather tha phases, typically the oly deliverable after each istace is code Requiremets ca keep chagig withi a iteratio based o user iputs Users ad developers are part of the same team. Users are ivolved i each phase Framework is defied before startig iteratios; same model is revised i each iteratio. If required, refactorig is used (explaied i Sectio 2.5) Users is egaged i early stage ad lesser risk of project failure Table 1: Agile vs Waterfall - Beefits 6

7 Value Additio of Agile i BI implemetatios Agile (or ay kid of iterative style of developmet) works well i the BI area due to the followig reasos: A lot of BI projects are about addig ew data, or ew reports, to a existig data warehouse. These require few ehacemets/chages i the existig data warehouses ad create reports that derive that from data warehouse. This fits i well with the cocept of Agile A typical BI implemetatio cosists of various modules such as Data Modellig, Data Quality, Data Extractio ad Reportig. Agile methodology ca be used for developig each module separately ad the itegratig each module to make a complete BI solutio. Reportig part of BI requires lot of busiess iteractio ad Agile helps to brig i busiess users ad developer as a team, which help to accelerate the processes. The users ca see the reports, commet o them, ad evolve the requiremets, for example, i 2 week iteratios, users ca work with the developmet team ad chage the look ad feel of reports i reports demo sessios. So by the ed of sprit, reports delivered are the oes that really work for them. Agile is testig drive approach,user is actively ivolved i the desig ad implemeatio process, hece may reports ad data bugs are foud i the early phases as compared to the traditioal approach. A typical Agile BI Cycle is illustrated i Figure 1 below. Requiremet Requiremet Backlog Test Desig Release 1 Implemet Requiremet Sprit 1 Test Desig Implemet Requiremet Sprit 2 Test Desig Implemet Sprit N Figure 1: Traditioal Agile Developmet Approach 7

8 As illustrated i Figure 1, Requiremet Backlog is received from users. These requiremets are divided ito sprits based o their priority ad complexity. Oe sprit is a full life cycle of uderstadig the requiremet, aalysis, desig, build ad user testig. A Sprit lasts for 1-2 weeks. User Demostratio (typically called Show ad Tell) is coducted after every sprit to get the user feedback. As per the requiremets, after every few sprits, a productio roll out is carried out to deploy chages to productio. This is called Release. Best Practices for Agile BI Implemetatios Testig Automatio A release geerally cosists of ew added reports alog with few chages i ETL(Extract Trasform ad Load) ad database layer. As Agile sprits are of short duratio i.e. 1-2 Weeks, testig automatio should be doe to reduce the testig cycle. It is recommeded that a regressio test suite is developed cotaiig a set of test cases which cover the core fuctioal areas. This test suite is automated for database ad ETL testig. For example, SQL scripts ca be used for record cout match betwee source ad destiatio of key ETL loads. Metrics Key to Agile estimatios is Story Poits Story poits for each user story are calculated based o the complexity of requiremet, desig ad curret uderstadig of the solutio. It gives a estimate o how much time a user story is goig to take for developmet. Based o the user stories grouped together i a sprit, the total time required for sprit is calculated as give below: Expected Time required for a sprit = Sum of Story poits for each user story i a sprit * Average time required to implemet oe poit This approach is differet from effort estimatio doe for the Waterfall approach. The mai advatages of usig this agile estimatio techique are:the above metioed measuremet techique is idepedet of factors such as team member s expertise or umber of members i the team. It just sigifies the complexity of a user story compared to other user stories i the same sprit. So while calculatig the story poits of a user story, other factors eed ot be cosidered. Steady state: After few sprits, the team is able to easily judge how may story poits ca be delivered i a sprit. The estimatio of story poits ad sprit effort the becomes easier. User satisfactio: After reachig the steady state, deliveries become more certai, eve the sprit velocity (umber of story poits delivered per sprit) icrease. This helps i gaiig user cofidece ad project deliverables are show at short ad regular itervals. Followig are some of the metrics which ca be used to measure the performace of Agile BI implemetatio: 8

9 Scrum velocity: Velocity is a measure of how much work is gettig doe o the project i oe scrum. This is a importat metric which drives release plaig ad schedule updates. I project maagemet terms, velocity is the amout of work that a team ca complete i a specified period of time. I Agile, it ca be measured as the umber of story poits completed. Figure 2: Sample Scrum Velocity Chart Sprit Bur Dow: The performace metrics i Figure 2 measures the total absolute amout of effort i perso hours or perso days left i the sprit. The bur dow ca be calculated eve daily, to uderstad the status of how much work is pedig i the sprit ad if ay chage of actio pla is required. Sprit bur dow calculatio ca be part of daily Scrum calls. Burout = New estimate (tasks backlog) + Estimate (ew issues) Sprit bur dow chart calculates the amout of work to be doe over the sprits. So, for example, if you have 15 sprits, ad 200 tasks, your chart will have 200 alog the vertical axis, ad 15 alog the horizotal. After each sprit, mark the umber of tasks remaiig alog the vertical axis. Durig the first few sprits, it may be slow, but it will show progress towards the goal. Release Bur Dow: The metric i Figure 3 idicates the bur dow across the sprits. This helps to appreciate how the performace has improved/chaged with every sprit. 9

10 Figure 3: Release Bur Dow Chart Refactorig I Agile BI implemetatios, the desig of ETL or data model is chaged i every sprit to take care of user stories of that sprit. Sice all the requiremets are ot aalyzed at the begiig of the desig, there will be scearios where at later stages we realize that the data model should be modified ad correspodig ETL layer should be chaged based o the requiremets added i the curret sprit or to make the data model more adaptable or scalable. That is, the desig alterative chose i a sprit may ot be the best approach i log term but meets the user eeds. To address this issue, a refactorig process is recommeded to icorporate the chage i data model or ETL, while keepig the reports uaffected. Refactorig meas makig small chages to the iteral structure of a applicatio without chagig its exteral behavior. It may ot chage ay existig fuctioality from the user s perspective but helps i improvig the system behavior ad performace i log term. I terms of effort, refactorig ca be cosidered as user stories ad will be implemeted as part of sprits. The oly differece beig that it will ot have ay user impact associated with it. The key factor which eeds to be cosidered while refactorig is esurig that the curret fuctioality is ot impacted. To esure this: Perform a rigorous regressio testig exercise. The test suite should cover all the key fuctioality ad metrics. Perform comprehesive impact aalysis to esure o surprises later. Maitaiig the metadata across ETL ad Reportig will be helpful for this Defie Chage Maagemet process, usig cofiguratio maagemet ad versio cotrol tools. Audit techiques will be helpful to cotrol the chages. Gradual Movemet towards Agile If a orgaizatio is curretly usig the Waterfall model of implemetatio ad plas to shift to Agile, a gradual movemet should be plaed from oe methodology to the other. To start with, Agile ca be itroduced for project which meets the Agile requiremets. For example, out of the total requiremets, if 10

11 70% of the requiremets are very clear to the developmet team ad 30 % are vague ad eve the user is ot sure of what is required, cotiue usig the Waterfall/Spiral model for 70% of the stable requiremets, ad use the sprit approach for the rest 30%. Aother factor which govers the movemet to Agile model is the maturity of data warehouse. If the eterprise data warehouse already exists ad requiremets are more towards addig reports rather tha makig chages i warehouse data model, the movig to Agile is a comparatively smoother process. Team Structure The ature of Agile itself promotes oral commuicatio ad discourages spedig time o writte commuicatio ad documetatio. Typically, a team size should cosist of a group of 6-8 people. Small teams help i better commuicatio ad team work. If the umber of user stories to be covered i a sprit is large ad a larger team is required, it is suggested to split the team ito multiple groups, each team caterig to separate fuctioal area ad preferably hadlig requiremets for separate set of stakeholders. Withi a small Agile BI team, there should be a mix of techical skills ad domai kowledge, the team should have primary owers for each of the below areas: User requiremets Source data ad applicatios Datawarehouse data model Data trasformatios ad ETL routies Reportig applicatios I order to satisfy these eeds the team comprises of: Ed user, Busiess aalyst, ETL desiger/developer, Report desiger/developer, Database admiistrator, ad BI architect. Agile implemetatio ca be practiced i a osite offshore model as well.i this case regular scrum calls play a very sigificat role for effective ad upfrot commuicatio. Factory Model Sice Agile approach works typically for a team of 5-8 people (Sectio 2.7 above), it is best to split the project ito small teams, each workig towards a busiess area so that oe busiess user is part of oe team. I this case coordiatio ad prioritizatio across teams is vital ad ca be hadled usig factory model/competecy cetre. Role of this team is to gover the process, people ad data i a BI implemetatio across various agile modules. Followig are the beefits of this model: Stadardized delivery model Scalability to support a icrease i umber of projects to be executed. Use of robust processes that would aid the executio. These are refied over a period of time Reduced cost by idetifyig ad usig reusable compoets Extremely flexible staffig model i terms of ramp-up ad ramp-dow capabilities as well as resource sharig betwee projects i order to maage crests ad trough situatios i projects. 11

12 Agile Beefits Agile implemetatio is beeficial to both IT ad busiess users. For IT Prompt feedback esures miimum last miute chages or updates ad ivolves miimum risk. Customer collaboratio helps developers uderstad the ed to ed system ad deliver what is really required by the customers. Cotiuous flow of value to customers, hece better satisfactio idex. Daily scrum calls brig eergy to the team. Lesser documetatio ivolved eables user to directly judge the ed product. Iterative model helps to reduce complexity ad improve productivity Better resource utilizatio. For Busiess/Customer Agile methodology gives users the flexibility to chage requiremets withi a developmet cycle, which brigs i flexibility to the implemetatio Users are able to start usig the ed product quickly as compared to waterfall due to shorter developmet cycles. This leads to better ROI. Agile projects are cheaper as they reduce the cost of rework by esurig customer satisfactio ad avoid last miute chages. User acceptace testig is a early phase i the agile methodology as compared to o agile developmet process. Higher probability of o time ad defect free deliveries Choosig betwee Agile ad Waterfall Differet busiess scearios demad differet methodologies. Followig table provides a compariso of Agile ad Waterfall agaist differet criteria. Criteria Type of Cotract Requiremets Type of Solutio Team Size Agile Suitable for model based o T&M ad focus o reducig time to market More tha 30% chage expected i requiremets Solutio that requires more chages at reportig. Maiteace projects Team size betwee 5-9 people Waterfall Suitable for Fixed Bid projects. For example, for developig caed reports or solutios withi fixed time frame. Less chage expected i requiremets Buildig Data warehouse from scratch Team size more tha 10 12

13 Criteria Developers Customer Agile Multi skilled, motivated team Egage with developers ad get ivolved i day to day activities. Waterfall Not so skilled, juior developers Remote customer, Does ot egage i developmet, Iterested i the fial product. Agile BI@ TCS BIDS BI Rapid Delivery Framework After successfully usig Agile methodology i various projects, TCS has ow developed a framework which assists orgaizatios i choosig the right methodology for BI implemetatios. The framework icludes the followig: Table 2: Agile vs. Waterfall- Selectio Criteria Assessmet of Agile readiess: This help to assess if a orgaizatio is suitable for movig to agile implemetatio. This is doe o the basis of the followig three key aspects: Project: Key factors for cosiderig a project for Agile implemetatio are: the ature of requiremets, depedecy o third party, ad project duratio. People: Availability of expert developers is a key factor for the orgaizatio to start Agile implemetatio. Techology: Sice Agile ivolves a lot of refactorig to the existig code, oe of the key factors of cosideratio is the amout of legacy code beig used curretly. Also the developmet platform/toolset beig used is importat. Balace through right agile methodology: BIRD framework as explaied below compares various agile methodologies agaist the key aspects to distributed developmet. This helps i choosig the right methodology for a orgaizatio. Some of the key aspects beig aalyzed are: requiremets volatility, umber of sites of deploymet, customer participatio, third part depedecy ad so o. It also provides the life cycle of Agile BI implemetatio. Followig modules are ivolved i the BIRD: Framework defiitio Specificatios Release: There will be multiple releases to productio. Withi each release there will be sprits for few weeks each. Susteace Co-actio setup: This ivolves settig up the followig: Ifrastructure: Tele-cofereces, Video-cofereces, Voice over IP, Dedicated etwork lik, Virtual Servers Egieerig: Maage requiremets, Code geeratio, Test data geerators, Test automatio, Build automatio 13

14 Collaboratio: Wiki, Discussio forums, Mailig lists, Commo repository Goverace: Real time metrics, Real time dashboards, Automated process improvemet. BIDS BI Competecy Cetre TCS provides a goverace model called BI Competecy Cetre (BICC). This provides guidelies to develop a goverig body withi a orgaizatio, which helps i defied process ad techology stadards. It also aligs the various projects based o busiess priorities. Sice Agile works best with small team, BICC ca help the orgaizatio to split requiremets ito various agile modules ad esure data stadards ad process stadard across various modules. It provides the followig beefits: Formulate Eterprise BI strategy ad user story prioritizatio Cosolidate best practices ad reusable compoets across teams Esure there are o coflictig developmets withi the team Ratioalizatio ad stadardizatio of techologies Shared BI ifrastructure ad services Cosolidate, develop ad retai competecies Cetre of Excellece for techology trackig Kowledge Maagemet TCS Tools There are several TCS products that ca be utilized i Agile Developmet for BI Projects. MAPAGILE helps i requiremet aalysis, review ad to geerate gap aalysis report. Testig tools such as DataXplod, Masketeer ad Testify ca be used for test data geeratio for testig automatio. Masketeer is a datamaskig tool, which esures privacy of data while maitaiig high data utility. DataXplod geerates ay volume of realistic data from a relatively small volume of sample data usig some simple but itelliget data-trasformatios. It ca read database schema ad data from heterogeeous data source. Testify is a software testig tool based o formal test specificatios that are processed for test data geeratio from scratch ad test drivers. Coclusio BI programs are oe of the areas where applyig agile priciples ca be extremely beeficial; this is because of the user ivolvemet ad cotiuous chage of requiremets i a typical BI project. The beefit of agile may ot be visible from the first sprit; it takes few cycles to reach a steady state where the retur ca be see as the scrum velocity icreases. It is advisable to start usig agile i a icremetal fashio. Oe of the areas where agile ca be first itroduced is to projects where Data warehouse is already i place ad ew requiremets lead to chages i data marts of reportig layer. Refactorig works well i this sceario as ehacemets are ot expected to make core chages i the core data model. Refereces [1] Extreme Programmig- [2] Scrum Methodology- [3] BIDS BI Rapid Delivery Framework [4] Beck, Ket; (2001). "Maifesto for Agile Software Developmet". Agile Alliace. [5] 14

15 About TCS HiTech Idustry Solutio Uit TCS' HiTech idustry Solutios Uit provides optimal, customized, ad comprehesive solutios across varied High Tech idustry segmets: Computer Platform ad Services Compaies, Software Firms, Electroics ad Semicoductor Compaies, ad Professioal Services Firms (Legal, HR, Tax & Accoutig ad Cosultig & Advisory/Aalyst firms). Buildig o its vast experiece i egieerig, busiess process trasformatio, iovatio ad IT solutios, TCS offers a comprehesive portfolio of services that maximize growth, maage risk, ad reduce costs. The TCS HiTech Idustry Solutio Uit parters with High Tech eterprises to provide ed-to-ed solutios which help realize operatioal excellece, iovatio ad greater profitability. For more iformatio, visit us at Cotact For feedback o this article ad more iformatio, please cotact us at: HiTech.Marketig@tcs.com Subscribe to TCS White Papers TCS.com RSS: Feedburer: About Tata Cosultacy Services (TCS) Tata Cosultacy Services is a IT services, cosultig ad busiess solutios orgaizatio that delivers real results to global busiess, esurig a level of certaity o other firm ca match. TCS offers a cosultig-led, itegrated portfolio of IT ad IT-eabled ifrastructure, egieerig TM ad assurace services. This is delivered through its uique Global Network Delivery Model, recogized as the bechmark of excellece i software developmet. A part of the Tata Group, Idia s largest idustrial coglomerate, TCS has a global footprit ad is listed o the Natioal Stock Exchage ad Bombay Stock Exchage i Idia. For more iformatio, visit us at IT Services Busiess Solutios Outsourcig All cotet / iformatio preset here is the exclusive property of Tata Cosultacy Services Limited (TCS). The cotet / iformatio cotaied here is correct at the time of publishig. No material from here may be copied, modified, reproduced, republished, uploaded, trasmitted, posted or distributed i ay form without prior writte permissio from TCS. Uauthorized use of the cotet / iformatio appearig here may violate copyright, trademark ad other applicable laws, ad could result i crimial or civil pealties. Copyright 2011 Tata Cosultacy Services Limited TCS Desig Services I M I 07 I 11