Agile and BI a perfect match Agile BI congres Marco Coopmans 13 december 2012?
What is Agile?
Why start working differently? A software development project Takes too long Is too expensive Does not deliver what is asked Does not deliver what is needed
The Agile Manifesto Values Individuals and interactions over processes and tools Working software over comprehensive documentation Customer collaboration over contract negotiation Responding to change over following a plan [Agile Manifesto, Utah, 2001]
The goal of agile The main goal of agile is: Delivering as much value as possible for the given time and money How? Only make realy needed functionality, the highest value first Eliminate waste (lean) only added value Mitigate risks show working, production feasible software
The characteristics of agile vs. waterfall Waterfall Agile The user knows what he needs The developer knows how to build it During the project nothing changes The user discovers during the project what he needs The developer discovers during the project how to build it There is a lot of dynamics and changes during the project
Variable Variable Fixed Fixed The devils triangle - Waterfall vs. Agile Waterfall Agile Functionality Time Quality Determines Money Time Quality Money Determines Functionality
Agile and architecture See the curve coming, but don t overdrive your headlights
Some architectural agile principles Agile is delivering working software as quickly as possible (business value) Keep it simple As simple as possible Eliminate waste Don t carry any waste as long as it is not necessary Don t invest (too much) in the future Postpone choices as long as possible Do things only when they are really needed Be a minimalist
Emerging architecture Architecture is not determined at the beginning, but emerges (form follows function) Determine a start architecture: a (minimal) set of guide lines That gives a lean-and-mean framework that grows during development Refactoring is a fact!
Agile is not short term solutions To avoid a dead end and minimalize the impact of refactoring you have to look ahead, but not too far! An experienced SCRUM team will not let this happen!
SCRUM is the most applied form of agile
Release 2 Release 1 The SCRUM process Product Vision Solution outline (architecture, tooling) Business Case / ROI Release planning (1st release = MVP) Priority Sprint planning Sprint (2 to 4 weeks) Daily stand up Demo Development Sprint Backlog Production feasible Product/software Release x Release.. Grooming/Poker sessions Feedback, new input Product Backlog
SCRUM Roles and Organisation Stake Holders Scrum Team Tune Tune Product Owner Supports Project Team Scrum Master
Roles and Organization Product Owner Product Owner Is a delegate of the business who has the mandate to determine what is on the product backlog and the prioritization of it Holder of the product vision Owner of the product backlog and user stories (requirements) Represents all stake holders (business, IT, external, ) Accepts or rejects the product of a sprint Decides release date and content Product Owner Product Owner sits in the driver seat
Roles and Organization Scrum Team Scrum Team Scrum Team is selforganizing Scrum Team A multidisciplinary team (designers, developers, testers, architects, ) all expertise necessary to deliver a production feasible product Design, develop and test the user stories Is self-organizing, without hierarchy Determines, together with the Product Owner, the content off a sprint and commits itself to it!
Roles and Organization Scrum Master Scrum Master Facilitates the SCRUM process of the team Ensures optimal conditions for the productivity of the team Solves impediments (blocking issues) Is part of the team, but is supporting (has no authority) Scrum Master Scrum Master is a facilitator
Roles and Organization Stake Holders Stake Holders Stake Holders Everybody who has an interest in the product (endusers, managers, security officers, architects, system administrators, controllers, supervisors, ) Are being represented by the Product Owner Stake Holders don t have direct access to the Scrum Team! takes place via the Product Owner
Agile and BI
Why start working differently? A software development project Takes too long Is too expensive Does not deliver what is asked Does not deliver what is needed According to Gartner research estimation is that 70 to Does 80% of this BI projects also fail. apply And for the BI number one reason cited for this disturbing rate of debacle? Poor projects? communication. (Patrick Meehan, President and Research Director at Gartner, 2011)
Agile BI is about tools? Agile BI = Delivering functionality as soon as possible It s not only about delivering Agile BI = Self-service data fast, it s also about the right data and Agile BI = Tools quality
Agile BI is only suited for the front-end? Front-end Agile Only 20/30% would be agile! Back-end Waterfall 70/80%
Agile BI is layer driven? Sprint 5 First visible result in sprint 5 Sprint 4 Sprint 3 Sprint 2 Sprint 1
From source to information Agile is delivering business value (each sprint)! business value But how are you gonna do this every sprint, from the beginning?
The minimal architecture Choose a minimal set of components to achieve the requested functionality. This depends on tools, source systems, transformation, integration,
2 ways - #1 Report driven (slicing) A report is growing with each sprint This will need a few sprints to get a sustainable report to get efficient feedback
2 ways - #2 Data / information driven (refining) Reports are refining / maturing with each sprint
The Mona Lisa model from raw data to information Initiate & Model Storming Base Sprint Sprint Sprint Sprint Production Raw Refine Making accessible Dimensional model (Facts and Dimensions) ETL Datastorage (DWH/DM/ ) Universe(s) Standard reports User documentation System- & Operatingdocumentation Set op a base version (dimensional) informationmodel Confronting users with data as soon as possible Unlock as much as possible (raw) data Refine content: - Refine Dimensional model - Refine Universe - Make basic report(s) - Add business rules, transformations, integrations, - Add history -. Refine to end users: - Make final rapports - Add security/ authorization - Finalize metadata - Finalize documentation -. Bring to Production Experienced Domainexperts Trained key-users
Organisation BICC and SCRUM team Business IV IT-supplier Business Intelligence Competence Center Proces/Data owner Power User End-User Key-User Management Reporting Authority BI Domain Expert BICC manager Enterprise WH Architect Service Coördinator BI/ETL Designer/Developer BI/Warehouse Consultant BI Tool Specialist ETL Designer/Developer Legacy Designer/Developer BI Business Modeler (BI Tool Specialist) Production Guard Technical Consultant (dba, network, infra, etc) Power User Business BI/ETL BI Domain Expert BI Business Modeler Enterprise WH Architect Developer/ Specialist Designer/Developer IT SCRUM Team Product owner Scrum master Project Team
Why agile is so well suited for BI BI issues Agile Unclear information need Management / business involvement Quality of data Definitions Reliability of information Organisation Agile assumes that requirements are not clear and/or change, that is part of the approach. By a constant feedback requirements become clear. Business is in the lead (Product Owner). Business is part of the team (domain experts and power users). Stake holders are being involved through demo s, grooming sessions, sprint plannings. Quality of data is being tackled at the moment the issue arrises. The issue is made clear from a concrete information need and is better negotiable than. Definitions are being established during development (by the business), when it is necessary, based upon real data/visible results. Reliability is being guaranteed by the participation of domain experts and power users in the SCRUM team (not by IT). They also give the demo! A SCRUM team fits realy good with a BICC. The business (power users, etc.) will grow into their role during the project.
Agile and BI are a perfect match!
Preconditions and Pitfalls
Preconditions - Culture Give confidence Learning organisation - making mistakes is allowed Transparancy - no judgement Entrepenearship and creativitity Belief Agile means a change of culture
Preconditions Process en organisation Committed Product Owner, with mandate Sufficient business (customer) participation Preferably Power Users IT Professionals (craftsmanship) (SCRUM) process is being followed meetings, stand-ups etc. are not for discussion The business sits in the driver seat
Preconditions Technique Enough complexity Emergent architecture Continuous build and deploy Automated testing Production(like) data available The SCRUM team determines the solution no one else
Pitfalls - Sprint No closed sprint New features, requirements added during the sprint No real commitment for a sprint Laying the bar too high Don t do too much at the same time Laying the bar too low Don t be satisfied too easily Too much production-oriented Communicating is production! It s not about the amount of code 10 to 30% preserved for grooming etc. No Definition of Done (DoD)
Pitfalls - Solution Holding on to standard solutions, architectures New techniques provide many opportunities, try them Change to generic / flexible solutions too quickly Generic datamodels, Datavault, Too little focus on quality short term solutions
Pitfalls - Leadership Agile is not about working late (on the contrary) Fall back on classical management The Team is self-organizing, even at bad weather It s not about working harder but working smarter
Pitfalls Client/Business No clear business objective, lack of product vision Why are we doing this? What are we trying to achieve? Insufficient coordination with stakeholders (by the Product Owner) Lack of support
Conclusion BI and Agile are a perfect match, but. It s a difficult game to play
Thank you Marco Coopmans Marco.coopmans@gnie.nl +31 6 11916413