Extreme Scoping An Agile Project Management Approach for Data Warehouse Projects

Transcription

1 Extreme Scoping An Agile Project Management Approach for Warehouse Projects Larissa T. Moss Method Focus Inc. DAMA Central Ohio Chapter 2008 Copyright , Larissa T. Moss, Method Focus, Inc. 1 Outline Why traditional project management does not work on DW projects Software release concept with self-organizing project teams Different project planning process BI program management Organizational culture change Copyright , Larissa T. Moss, Method Focus, Inc. 2 1

2 Traditional Method Operational Systems (Legacy) L L Decision Support Systems MK FA Units Marketing Finance L L CS PS EN Customer Support Product Sales Engineering Copyright , Larissa T. Moss, Method Focus, Inc. 3 Uncoordinated Manager Chief Operating Officer Manager Chief Executive Officer Chief Information Officer IT Manager paired up Technology IT Manager Units KW KW KW KW KW KW KW Marketing Financial Product Customer Distribution Inventory Do You Know Your? IT IT IT IT IT IT IT Information Technology Units Sales swim lane development data redundancy process redundancy dirty data Copyright , Larissa T. Moss, Method Focus, Inc. 4 2

3 Waterfall Methodologies Need Project Plan Functional Requirements System External Internal System Client Approval Silos Production System Implementation Maintenance Post-Mortem Review Copyright , Larissa T. Moss, Method Focus, Inc. 5 Using the Waterfall Approach Requirements are well defined Scope is manageable Scope and users are confined to one business unit (department, business function) Technology infrastructure is known and proven volumes are relatively small activities are the same on every project activities are not cross-organizational Project schedules are relatively easy to estimate Mistakes are less expensive to fix early in the development process! Copyright , Larissa T. Moss, Method Focus, Inc. 6 3

4 Industrial-Age Mental Model highest to lowest priority Project Constraints Priority TIME SCOPE BUDGET PEOPLE QUALITY Scrap and rework (Larry English) Cost-based value proposition Cheaper, faster, better Automate as quickly as possible Copyright , Larissa T. Moss, Method Focus, Inc. 7 Time versus Quality Everyone wants quality, but rarely is the extra time given or taken to achieve it. Quality and time are polarized constraints. The higher the quality the more time it takes to deliver. Companies are driven by shorter and shorter schedules. SCOPE YAH TIME DDD Copyright , Larissa T. Moss, Method Focus, Inc. 8 4

5 The Time Trap Redundancy increases Maintenance increases Need for more staff increases Quality decreases time increases Employee burnout increases Inability to support business drivers User satisfaction decreases Productivity decreases value decreases Copyright , Larissa T. Moss, Method Focus, Inc. 9 Proliferation of Redundancy Operational Systems (Legacy) BI? Decision Support Systems Units L MK Marketing L L transformation? cleansing? FA CS Finance Customer Support L PS EN Product Sales Engineering Framework? Copyright , Larissa T. Moss, Method Focus, Inc. 10 5

6 Adding to the Chaos EDW DM DW and Marts DM DM DM transformation? cleansing? EDW DM DM DM DM Copyright , Larissa T. Moss, Method Focus, Inc. 11 The Lesson? You cannot keep doing what you have always done and expect the results to be different. That wouldn t be logical Spock, Star Trek Copyright , Larissa T. Moss, Method Focus, Inc. 12 6

7 From Chaos to Architecture Exploration Warehouse Mart Operational Store Enterprise Warehouse Mart Mart Oper Mart Oper Mart Oper Mart CRM Anal BI Access & Delivery Copyright , Larissa T. Moss, Method Focus, Inc. 13 Information-Age Mental Model highest to lowest priority Project Constraints Priority QUALITY TIME PEOPLE BUDGET SCOPE Reassemble reusable components (John Zachman) Investment-based value proposition Reassemble the entire enterprise Reuse assets from inventory Copyright , Larissa T. Moss, Method Focus, Inc. 14 7

8 Coordinated Units KW KW KW KW KW KW KW Sales Inventory Distribution Customer Product Financial Marketing IT IT IT IT IT IT IT Information Technology Units Discover, Communicate, Integrate, Document, Coordinate Enterprise Information Management Operational Environment Operational Systems ODS OM EDW DM DW bases Decision Support Environment cross-organizational development Copyright , Larissa T. Moss, Method Focus, Inc. 15 Cross-Organizational Commitment to data integration embedded in the methodology Cross-organizational program management Enterprise information management group Standards that include a common information architecture (enterprise data model) < principles < governance < resources < policies Coordinating the development/ processes< enforcement Disallowing stovepipe development Extracting and cleansing source data only once Reconciling data transformations and storing the reconciliation totals as meta data Copyright , Larissa T. Moss, Method Focus, Inc. 16 8

9 Outline Why traditional project management does not work on DW projects Software release concept with self-organizing project teams Different project planning process BI program management Organizational culture change Copyright , Larissa T. Moss, Method Focus, Inc. 17 Spiral BI/DW Methodologies Enterprise Infrastructure Assessment & Strategy Project Plan Opportunity Requirement Post-Impl. Review BI & DW s Integration Implementation Copyright , Larissa T. Moss, Method Focus, Inc. 18 9

10 Software Concept Extreme scoping - Larissa Moss First Projects Final Second feels like prototyping BI Reusable & Expanding Fifth Fourth Third Refactoring - Kent Beck Project = / Copyright , Larissa T. Moss, Method Focus, Inc. 19 Using the Software Approach Unstable requirements can be tested and enhanced in small increments Scope is very small and manageable Technology infrastructure can be tested and proven volumes (per release) are relatively small Project schedules are easier to estimate because the scope is very small activities can be iteratively refined, honed, and adapted Mistakes are less expensive to fix early in the development process! Copyright , Larissa T. Moss, Method Focus, Inc

11 Using the Software Approach Unstable requirements can be tested and enhanced in small increments Scope is very small and manageable Technology infrastructure can be tested and proven volumes (per release) are relatively small Project schedules are easier to estimate because the scope is very small activities can be iteratively refined, honed, and adapted And the quality of the release deliverables (and ultimately the quality of the BI applications) will be higher! And the development process will get faster and faster! Copyright , Larissa T. Moss, Method Focus, Inc. 21 Parallel Tracks Back-end DB + Construction Front-end Construction Meta Construction Management Delivery Meta Management Copyright , Larissa T. Moss, Method Focus, Inc

12 Team Organization BI Steering Committee EXTENDED Security Officer Track Project Management Appl. Track CORE Participation MDR Track Programmers EIM Skills Technical Skills Technical Services Auditor Operations BI Program Management Strategic Architect Sponsor End user Support Copyright , Larissa T. Moss, Method Focus, Inc. 23 Project Team Structure Project core team Self-organizing SWAT team Optimum team size: 3 5 people; never more than 7 Full-time involvement on the project from beginning to end Meet daily to control project activities track team Self-organizing XP (extreme programming) team Optimum team size: 2-3 people; never more than 5 Full-time involvement on their development track Meet daily to control development activities Extended team Involvement on an as-needed basis Support roles Copyright , Larissa T. Moss, Method Focus, Inc

13 Project Team Staffing Project core team One hands-on project manager One business representative with some decision-making authority One EIM person from IT (data administrator or enterprise information architect) One technical lead person from IT (senior developer or architect) track team One senior developer One developer or systems analyst Extended team sponsor Technical support team Operations Etc. Copyright , Larissa T. Moss, Method Focus, Inc. 25 Core Team Roles lead developer BI infrastructure architect representative administrator (EIM) quality analyst base administrator/designer lead developer Meta data administrator (EIM) Project manager Subject matter expert Copyright , Larissa T. Moss, Method Focus, Inc

14 Extended Team Roles developers BI support staff (helpdesk) sponsor Miner developers IT auditor or QA analyst Meta data repository developer Network services Operations Security officer Stakeholders Technical services Testers Tool administrator Web developers Webmaster Copyright , Larissa T. Moss, Method Focus, Inc. 27 Outline Why traditional project management does not work on DW projects Software release concept with self-organizing project teams Different project planning process BI program management Organizational culture change Copyright , Larissa T. Moss, Method Focus, Inc

15 Software Planning Process 1. Estimate development effort for entire BI application describes 592 Write query script Methodology specifications Write access interface programming specifications Write online help function programming specifications Step 11: Extract/Transform/Load 596 Build and unit test the process 542, Code the programs 598 If using an tool, write instructions for tool modules 599 Capture the technical metadata for the metadata repository 600 Write code to produce reconciliation totals, quality metrics, and load statistics 601 Unit test each individual program module 597, If using an tool, unit test each tool module 598, Write the scripts to execute the programs and the sort, merge, and load utilities 601, Integration or regression test the process Create a test plan with test cases for the process 606 Create test data for the programs 607 Integration or regression test the entire process 606, Log the actual test results and document any test issues from Intelligence Roadmap (Moss & Atre) refines Project Activities day-to-day work activities progress tracking and status reporting change control and impact analysis drives Work Breakdown Structure (WBS) activities dependencies deliverables roles defines Traditional Project Plan tasks sequence schedule resources milestones checkpoints overall effort, resources, cost Based on source: TDWI Copyright , Larissa T. Moss, Method Focus, Inc. 29 Software Planning Process 1. Estimate development effort for entire BI application 2. Break BI application into software releases 2006 Final First Second Nth BI Fourth Third Copyright , Larissa T. Moss, Method Focus, Inc

16 Software Planning Process 1. Estimate development effort for entire BI application 2. Break BI application into software releases 3. Create milestones from DDD to YAH for first release PROJECT reported First Highest level of project partitioning Major work effort of 1-3 weeks consisting of one or more tasks Milestone A unit of work that may be measured in days or hours Activity produces Task/Subtask Milestone Deliverable produces managed Task Deliverable Copyright , Larissa T. Moss, Method Focus, Inc. 31 Software Planning Process 1. Estimate development effort for entire BI application 2. Break BI application into software releases 3. Create milestones from DDD to YAH for first release 4. Organize and assign parallel development tracks Back-end DB + Construction Front-end Construction Meta Construction Management Delivery Meta Management Copyright , Larissa T. Moss, Method Focus, Inc

17 Software Planning Process 1. Estimate development effort for entire BI application 2. Break BI application into software releases 3. Create milestones from DDD to YAH for first release 4. Organize and assign parallel development tracks 5. Create detailed work assignments for all milestones Copyright , Larissa T. Moss, Method Focus, Inc. 33 Software Planning Process 1. Estimate development effort for entire BI application 2. Break BI application into software releases 3. Create milestones from DDD to YAH for first release 4. Organize and assign parallel development tracks 5. Create detailed work assignments for all milestones 6. Create Gantt chart showing milestones ID Milestone Name Duration Determine problem/opportunity Analyze cost/benefit of new project Determine critical success factors Analyze resources available Approval of project Copyright , Larissa T. Moss, Method Focus, Inc

18 Software Planning Process 1. Review development effort for entire BI application 2. Review and adjust BI application software releases 3. Create milestones from DDD to YAH for next release 4. Organize and assign parallel development tracks 5. Create detailed work assignments for all milestones 6. Create Gantt chart showing milestones Second Third Fourth Nth Final Copyright , Larissa T. Moss, Method Focus, Inc. 35 Project Dynamics Implementation Meta Mining Case Assessment BI/DW Project Meta base Planning and Requirements Meta Meta < parallel < iterative < dynamic Copyright , Larissa T. Moss, Method Focus, Inc

19 Implementatn Meta Mining Meta Reposit ory Case Assessment Post-Impl. Review Implementatn Meta Mining Planning Requiremts & Meta Reposi tory Meta Case Assessment Post-Impl. Review Mining Requiremts & Meta Case Assessment Meta Implementatn Post-Impl. Review Meta Reposi tory Testin g Planning Requiremts & Requiremts & Meta Meta Planning Requiremts & Requiremts & Meta Meta Implementatn Meta Mining Meta Reposi tory Implementatn Meta Mining Case Assessment Post-Impl. Review Meta Reposit ory Case Assessment Post-Impl. Review Planning Requiremts & Requiremts & Meta Meta Implementatn Meta Mining Planning Requiremts & Meta Meta Reposit ory Case Assessment Post-Impl. Review Requiremts & Meta Planning Requiremts & Meta Requiremts & Meta Iterative BI BI Copyright , Larissa T. Moss, Method Focus, Inc. 37 Outline Why traditional project management does not work on DW projects Software release concept with self-organizing project teams Different project planning process BI program management Organizational culture change Copyright , Larissa T. Moss, Method Focus, Inc

20 BI Program Management EXTENDED Security Officer Strategic Architect Track Appl. MDR Track Track Programmers Project Management CORE Participation Sponsor EIM Skills Technical Skills Technical Services End user Support EXTENDED Auditor Operations Coordinated Processes Common Meta Common Enterprise Information Architecture Security Officer Programmers Project Management CORE Participation Strategic Architect Track Appl. MDR Track Track Sponsor EIM Skills Technical Skills Technical Services End user Support Auditor Operations Common Methodology Common Enterprise Standards Common Ratified Rules Common Enterprise Infrastructure EXTENDED Security Officer Strategic Architect Track Appl. MDR Track Track Programmers Project Management CORE Participation Sponsor EIM Skills Technical Skills Technical Services End user Support Auditor Operations Copyright , Larissa T. Moss, Method Focus, Inc. 39 Enterprise Infrastructure Management Technical infrastructure (configuration management) Hardware Network RDBMS Tools Non-technical infrastructure methodology strategy with enterprise standards (modeling, naming, defining) Meta data (capture, maintenance, usage) and reconciliation Security Ratified business rules Roles and responsibilities Copyright , Larissa T. Moss, Method Focus, Inc

21 Enterprise Information Management governance program administration principles (data standardization) ownership stewardship profiling and data audits quality improvement processes certification Enterprise data models schemas (unnormalized transactions) Conceptual schemas (normalized business views) Logical schemas (refined and fully attributed views) Physical schemas (denormalized database views) Enterprise schema (integrated business views) Copyright , Larissa T. Moss, Method Focus, Inc. 41 Enterprise Meta Management Meta data strategy Buy or build Centralized or decentralized Meta data infrastructure Meta data sourcing meta data Technical meta data Meta data content management Meta data usage Impact analysis lineage Trend analysis of data statistics Query extensions Monthly Sales Report Month Produce US Sales $ Canada Sales $ Total Sales $ January February March Apples Bananas Coconuts Apples Bananas Coconuts Apples Bananas Coconuts 22,000 11,900 2,000 22,500 10,000 2,200 23,700 9,900 2,400 8,000 6, ,500 5, ,300 7, ,000 17,900 2,800 31,000 15,800 2,550 33,000 16,900 3,150 Meta as a query extension Quality & Load Statistics: 51 % of $ values not loaded 10 % of source records not loaded Copyright , Larissa T. Moss, Method Focus, Inc

22 BI Portfolio Management Project prioritization based on BI application business values Project scheduling based on project and resource interdependencies Project funding and staffing Resolution of issues among interdependent projects Copyright , Larissa T. Moss, Method Focus, Inc. 43 BI Steering Committee Composed of business executives and senior business managers Collective sponsorship from business executives Communicate enterprise-wide data integration principles to all lines of business [ongoing championship] Stand behind a BI strategy appropriate for the organization Fund an enterprise information management group Identify data owners and data stewards in all LOB Assign business representatives to project core teams Copyright , Larissa T. Moss, Method Focus, Inc

23 BI Program Manager Led by a BI program manager or BI director Perform periodic readiness assessments Identify new information needs Ascertain user satisfaction Create and enforce a common technical and nontechnical infrastructure ( Core Competencies ) Work with BI steering committee to prioritize BI and DW projects (BI portfolio management) Determine BI and DW project interdependencies Coordinate project resources and activities (Software release management) Copyright , Larissa T. Moss, Method Focus, Inc. 45 BI Maturity Model Source: Wayne Eckerson, TDWI Marts Warehouses Production Reporting Spreadmarts GULF Enterprise DW CHASM Analytic Services 1. Prenatal 2. Infant 3. Child 4. Teenager 5. Adult 6. Sage Cost BUSINESS VALUE ROI Value Copyright , Larissa T. Moss, Method Focus, Inc

24 Outline Why traditional project management does not work on DW projects Software release concept with self-organizing project teams Different project planning process BI program management Organizational culture change Copyright , Larissa T. Moss, Method Focus, Inc. 47 Organizational Impact No BI program management in place Cross-organizational tasks and responsibilities are not well defined New project dynamics are not understood Teams are still organized the traditional way people still don t participate on project activities Project scopes are still too big quality responsibility is not clear or ignored BI/DW projects are usually cost justified using the industrial-age mental model (no ROA) No reward for data sharing Resistance to change Copyright , Larissa T. Moss, Method Focus, Inc

25 Inevitable Culture Shift and IT collaboration ( partnership ) and business collaboration ( partnership ) IT and IT collaboration ( partnership ) Shifted roles and responsibilities Common architecture Enterprise standards Cross-organizational activities Software release concept Copyright , Larissa T. Moss, Method Focus, Inc. 49 Power Shift people must play an increased role and assume responsibilities for: ownership Information Architecture Enforcement of standardization Incentives Integration and standardization across all business units Cross-Organizational Disciplines Dispute resolution across all BI projects Leadership Governance BI Program Management Copyright , Larissa T. Moss, Method Focus, Inc

26 Charge-Back Policy System ownership by business unit Automate a business process from scratch Cost-based value proposition units pay for their own systems System = programs coupled with data old Information ownership by the enterprise new Reassemble the entire enterprise quickly Investment-based value proposition Organization pays for enterprise data units pool to pay for shared BI applications BI application = BI tools and programs decoupled from data Copyright , Larissa T. Moss, Method Focus, Inc. 51 New Incentive Policy Management and staff should be evaluated on: Team work and collaboration (IT and business) sharing (business) quality (business) Adherence to standards (IT and business) Adherence to cross-organizational development approach (IT and business) Rewards (salary increases and bonuses) should be tied to above performance measures Applicable to the business side as well as IT. Copyright , Larissa T. Moss, Method Focus, Inc

27 New Leadership CEO CFO collaboration COO CKO collaboration CIO LOB Execs DG...EA, IT Execs EIM, MDM, Governance DW, Chief BI, Knowledge CRM, Officer CDI, PIM, Copyright , Larissa T. Moss, Method Focus, Inc. 53 Thank You ISBN ISBN ISBN Larissa T. Moss Method Focus, Inc. ISBN Copyright , Larissa T. Moss, Method Focus, Inc