Business Intelligence Practice

Transcription

1 Business Intelligence Practice 1

2 Agenda BI Overview BI Current State Agile BI A new Methodology Agile BI Case Study Abercrombie & Fitch Questions 2

3 Business Intelligence Don Jackson draws on more than 27 years experience in the information systems industry to help businesses achieve greater profitability and reduce the cost of Business Intelligence within their IT departments. His team consists of over 70 dedicated professionals with years of BI experience and industry certifications. We have successfully implemented large DW projects for both public and private sector clients including: A & F AEP Abbott Nutrition Cardinal Health Nationwide Bank Nationwide Financial Nationwide Insurance NetJets State of Ohio Agencies Worthington Industries 3

4 Business Intelligence On-Shore Business Intelligence at Off-shore Prices Agile BI Methodology Information Factory Agile BI Development Business Sponsor / IT Alignment using Consensus Centers of Excellence (COE) Data Architecture Enterprise Data Integration Information Delivery/Presentation/Visualization Industry and partner certifications IBM, Informatica, MicroStrategy, Microsoft, Netezza, Oracle, SAP/Business Objects Gold level certification TDWI, DAMA (CBIP, CDMA industry-certified) 4

5 Applying Agile Principles to Business Intelligence 5

6 Agile Business Intelligence Lessons learned from the trenches 6

7 2009 Agile Projects Abercrombie & Fitch Abbott Nutrition Cardinal Health Nationwide Financial NetJets A&F Enterprise Data Warehouse 6 months DataStage, Netezza, and OBIEE Abbott Nutrition Customer Profitability 16 weeks Consensus, Informatica, Oracle, Cognos Cardinal Health MAPS 12 weeks Consensus, SSIS, SqlServer, Performance Point Nationwide Financial 12 weeks Informatica, Teradata, MicroStrategy NetJets Enterprise Data Warehouse 6 months Consensus, Business Objects, BODI, Oracle 7

8 BI Development Inefficient and Expensive Labor intensive: too much time spent gathering and organizing data Tools are not integrated: reporting and analysis are not linked with the processes that ensure accuracy Lack of consistency between monitoring, analysis, and reporting due to poor data lineage 8

9 Inefficient, labor intensive process without automation Use Case Manual rekeying? in Excel or Word by Business Analyst Business Object Definition Manual rekeying? Cube by OLAP Developer Reports by Report Developer User Documentation by Business Analyst Analysis & Reporting in Excel or Word by Solution Architect Entity Relationship Diagram in ERWin by Data Modeler Manual rekeying? Tables in SQL by DBA Manual rekeying? ETL Modules in PowerCenter by ETL Developer Manual rekeying? In SQL By DBA Manual rekeying? ETL Test Scripts in Excel or Quality Center Manual rekeying? ETL Documentation In Word or Visio Manual rekeying? Operations Documentation in Word Data Models Data Movement & Quality Source Data Requirements Analysis Design Build Test Deploy 9

10 BI SDLC: Waterfall Time to Value: 1 2 Years Solution Scoping Design Develop/Test 5 Data Analysis 9 ETL Design 11 ETL Development Justification 1 Business Case Assessment 2 Enterprise Infrastructure Evaluation Initiate 3 Project Planning 4 Project Requirements Definition 6 Application Prototyping 8 Database Design 12 Application Development 13 Data Mining 15 Implementation Implement 16 Project Evaluation 7 Metadata Repository Analysis 10 Metadata Repository Design 14 Metadata Repository Development Begin Gate Proceed Gate Define Gate Commit Gate Accept Gate Close Gate 10

11 Status Quo vs. Agile Manual, Paper Based, Slow, Error Prone Automated, Online, Reusable, Accessible, Governed 11

12 Agile Overview Will adopting an Agile Methodology solve our issues? 12

13 The Agile Manifesto We are uncovering better ways of developing software by doing it and helping others do it. Through this work we have come to value: Individuals and interactions over processes and tools Working software over comprehensive documentation Customer collaboration over contract negotiation Responding to change over following a plan That is, while there is value in the items on the right, we value the items on the left more. 13

14 Scrum Methodology Scrum is based on a "Sprint," which is typically a 30- day period focused on a specific deliverable. 1. Product backlog is determined 2. Backlog is prioritized day sprint is commenced 4. Daily standup meetings are directed by the Scrum Master 5. Product is delivered 14

15 Key Agile Principles! The Fundamentals To achieve breakthrough performance and cost savings would require a development process with the following: Mandatory Characteristics of Methodology: As business and IT friendly as possible As lean as possible As repeatable as possible With the highest quality possible At the lowest cost possible 15

16 Agile Alignment Information How does ICC s methods identify the right metrics for senior management? Process How does ICC s methods capture business rules, calculations and measures? Why is it faster? Technology How can you incorporate our current technology platform into an Enterprise Data Warehouse? Users and Tools How does the Agile process address divergent user needs for analysis and reports? Stewardship How does the Agile process ensure data accuracy and integrity? 16

17 Common Business Functions Executive Management Financial Performance Risk Management Compliance Management Finance Marketing Sales Customer Service Income Statement Drill-down Variance Operational Plan Variance Cash Flow/Working Capital Balance Sheet CapEx/Strategic Investments Treasury Market Opportunities Competitive Positioning Product Life Cycle Management Pricing Demand Generation Sales Results Customer/Product Profitability Sales Tactics Sales Pipeline Sales Plan Variance On-Time Delivery Information, Complaint, and Claims Service Benchmarks Service Value Shared Data Conformed Dimensions Product Development Operations Human Resources Information Technology Product & Portfolio Innovation Product Development Milestones Market & Customer Feedback Purchasing and Procurement Production and Capacity Inventory Management Distribution and Logistics Cost & Quality Management Process Efficiency Organization and Staffing Compensation Talent and Succession Training and Development Benefits Business Value Map IT Portfolio Management Project/SDLC Management IT Vendor Management IT Compliance Management 17

18 Headcount Expenses Revenue Plan Market Demand Capital Expenditures Financial Process Flow Finance Strategic Financial Planning & Forecasting Sales Sales Planning and Forecasting Financial Summary Balance Sheet Income Statement Operations Cash Flow Operating Expense Operations HR Headcount and Compensation Planning Capital Expenditure Planning Depreciation Expense Expense Planning and Control 18

19 Re-think Everything! People Process Platform Change the way we: Capture and document requirements Estimate and manage work Design the architecture and build models Develop and test ETL code Implement data stewardship Deploy BI Systems Build Enterprise Data Warehouse Systems 19

20 Business and Technical Value Chains A framework to organize Agile BI processes 20

21 Value Chains 21

22 Value Chain Deconstruction 22

23 Analysis & Reporting The Business Value Chain Objective Modeling is used to break down (analyze) and align broadly stated objectives (epics) into a hierarchy of manageable, easily understood questions and ideas (stories). Stories are shared with Data Architecture and Data Integration and linked to the Business Value Chain to provide continuous focus on delivering Business Value. These stories go through a five-step process to transform them into elements of a meaningful dimensional model (synthesis) 1. Business Questions Business Terms 2. Business Terms + Dimensional Functions (topic, category, measure) 3. Dimensional Functions are assembled into Multi-dimensional Information Packages 4. Queries & Dashboards are created from the Info Packages 5. Queries & Dashboards are reviewed to confirm they answer the target business questions

24 Data Architecture The Technical Value Chain After receiving the business terms from the business value chain, initial requirements and data architecture are determined. Major deliverables include: Initial Architecture Modeling (review existing models) Conceptual Data Models, Logical Data Models, Physical Data Models, Data Model Patterns and Reference Models (per release cycle) Coordination with enterprise architecture group is essential 24

25 Data Integration The Technical Value Chain The business and architecture value chains are used as inputs to create the data integration stories. These stories are used to identify target-to-source data mappings and identify the elements that will comprise the ETL rules for each sprint cycle. Major deliverables include: Data Quality Profiles Target to Source Maps ETL Process Flows ETL Reference Patterns ETL Test Scripts ETL Code 25

26 Scrum Story Cards

27 Horizontal vs. Vertical Slice Profitibility Process Flow Logical View C O N S E N S U S S O U R C E P S A (Staging Database ) T A R G E T (D a ta m a rts ) C O G N O S A cco u n tp a ya b le (M a in fra m e ) F la tf ile A cco u n tp a ya b le Staging Area (Oracle DB ) A cco u n tp a ya b le D a ta m a rt (Oracle DB ) P ro fitib ility D a ta m a rt (Oracle DB ) P & L R e p o rtin g P a cka g e P & L Report A cco u n t R e ce iva b le (M a in fra m e ) F la tf ile A cco u n t R e ce iva b le Staging Area (Oracle DB ) A cco u n t R e ce iva b le D a ta m a rt (Oracle DB ) C o g n o s F ra m e W o rk M o d e l P & L A d H o c Q u e rie s M icke y (L o tu s N o te s D B ) F la tf ile M icke y S ta g in g A re a (Oracle DB ) T IP S (L o tu s N o te s D B ) E xce l/f la tf ile T IP S Staging Area (Oracle DB ) S a le s A ctivity D a ta m a rt (Oracle DB) F re ig h tp a ym e n ts D a ta m a rt (Oracle DB) F in a n cia l A d ju stm e n t D a ta m a rt (Oracle DB) 27

28 Customer Profitability: Reuse Example Value Chain 1. AR Business Terms 2. AR Modeling Objects 3. AR Info Packages 4. AR Source to PSA Code/Audit 5. AR Source to PSA Code Tables 6. AR PSA to EDW 7. AR EDW to AR DM Code/Audit 8. AR EDW to AR DM Codes Tables 9. AR Data Mart 10.AR Cognos Framework ETL Patterns 1. Reset Audit Table (21) 2. Load Audit Table (21) 3. Reset PSA Table (4) 4. Load PSA Table (4) 5. Verify AR Table (6) 6. Verify AP Load (1) 7. Verify Load (13) 8. Load Staging Table (11) 9. Load Dim Table (3) 10.Final Load (9) 28

29 Case Studies 29

30 Sprint Cycles In a perfect world 2 week Sprint cycles In the real world Velocity and business requirements determine length of Sprint cycles 30

31 A&F - Retail Merchandising System November 10,

32 A&F ETL IF Project Abercrombie & Fitch ETL Information Factory Project: ETL transformations using IBM DataStage version 8 Multiple sources systems, ranging from Oracle RMS, PeopleSoft, and DB/2. Abercrombie team populates into Staging database. ICC Information Factory development to transform Staging to Netezza Data Warehouse foundation target. Around 300 ETL jobs delivered during the six month project. Sprint activities include: Mapping, Design Specifications, Development, Code Reviews, and QA Testing.

33 A&F Product Backlog Subject Areas

34 Maturity and Velocity 0 Current State 1 Defined 2 Architected 3 Implemented 4 Integrated 5 Optimized Combined People Speed Platform Velocity Process Quality

35 Efficiency Gains ETL Jobs per Day Productivity increased by 66% from Sprint 2 to Sprint Jobs per Day 70% 60% 50% 40% 30% 20% Efficiency 30% 66% 0.00 Sprint 2 Sprint 3 Sprint 4 Efficiency 10% 0% 0% Sprint 2 Sprint 3 Sprint 4 35

36 Cost per Sprint $350,000 $300,000 $339,960 Cost per Sprint $279,360 $259,560 $250,000 $200,000 $150,000 $100,000 $50,000 $- Sprint 2 Sprint 3 Sprint 4 36

37 Sprint 1 Training and Setup Three week Sprint 1 for training and prep: Information Factory members trained and educated on IBM DataStage and Netezza. Senior team members installed IBM DataStage environment and laptop configurations. Analysis & Design team members created Design Specifications and QA Test Plan documentation. Aggressive December timeline in order to start Development in Sprint 2. Team was flexible with their schedules. Helpful task: Performed team building exercises to establish group dynamics.

38 Sprint 2 Three ETL Subject Areas Five week Sprint 2 on three subject areas: Large hierarchy subject areas where our preparation and setup from Sprint 1 was put to the test. Helpful tool: Daily Scrum meeting to communicate and discuss tasks at hand. Helpful rule: If the source was not defined by the deadline, the ETL job was pushed into the next Sprint backlog. Helpful task: Unit Testing, Code Review, and plenty of QA testing to prevent defects and rework. Helpful communication: Mercury Interactive Quality Center to track and log defects.

39 Sprint 3 Four ETL Subject Areas Six week Sprint 3 with Reusable Code: Efficiencies kicked-in with consolidated A&F QA process, utilized previously developed reusable code, and split team into two pods. Helpful activity: Added Design Review with A&F internal team to prevent downstream QA bugs. Helpful task: Coding re-use of existing jobs, shared containers, documentation, and standards. Helpful environment change: Migrated from Windows to LINUX platform to speed up development without server lockups. Helpful team knowledge: Project experience allowed Sprint to be completed early.

40 Sprint 4 Transactional Data in Subject Areas Six week Sprint 4 to transform large data volume: Large data volume in Sprint required team to monitor performance and speed within the ETL jobs. Overcame hurdles with unexpected source design changes via a flexible SOW schedule. Final game-plan established for a strong finish to deliver by the end of May.

41 Unique and Successful Share a strong sense of purpose with a commitment to performance objectives: Experienced Project Managers, Architects, and Sr. Developers were responsible for ensuring successful project delivery. Junior developers completed the ICC Apprentice Program and ETL training to become highly skilled ETL developers. Created project specific library of reusable components. Test-driven development reduces defect rework which saves time and money. Continuous visibility of progress and work product review.

42 Summary 42

43 Process Agile Business Intelligence Program Management Business Justification, Financial Management, Portfolio Management Scrum Management Release Planning, Sprint Planning, Sprint Execution Business Modeling Business Questions, Business Terms, Business Objectives, Business Stories Solution Prototyping Topics, Categories, Measures, Information Packages Value Chain Deconstruction Business Analytics, Data Integration, Data Architecture Test-Driven Development Business Value validation, Technical Value Test Scripts, Automated & Continuous Integration/Regression Testing, Business-led Acceptance Testing, Production & Operation Support Training & Deployment, Service Level Agreements, Operations Support & Maintenance 43

44 Agile Alignment Information We quickly capture the right metrics and analysis requirements for each user group Process Our process provides dynamic business rule and data classifications, that are reusable Technology Users and Tools Stewardship We are certified in all major technologies and can assist enterprise architecture to ensure scalability and flexibility We are experts in delivering real value in terms of analytics, dashboard and detailed drill down reporting to all user audiences Our process implements stewardship from the start and ensures data accuracy and integrity 44

45 ICC Agile BI A Game Changer We have applied Agile principles to address the current inefficiencies in BI development We have created an integrated process that automates many of the coding activities We have connected the Business value with the Technical value so all parties understand their dependent relationship 45

46 There is no method or process silver bullet Methodologies continue to evolve and agile philosophies are a key component. 46

47 IT organizations need to find the balance between formality and agility. Adequate planning is a mix of long-term strategy and short-term execution. 47

48 Economic risk can be reduced by implementing Sprint cycles. At its core, Scrum is about people and deliverable-based project management. 48

49 Your Questions? 49