M4A Extreme Scoping (Part 1) Agile BI versus Agile EDW

Transcription

1 M4A Extreme Scoping (Part 1) Agile BI versus Agile EDW June 23, 2014 Larissa T. Moss Method Focus Inc. World Conference Munich 2014 Copyright Copyright , , Larissa Larissa T. Moss, T. Moss, Method Focus, Inc. 1 Agile development has been popular in the operational systems world for over a decade and it is finally catching on in the data warehousing/business intelligence space. However, the results are not always positive. Agile development methods like Scrum seem to work for some projects, but not others. They seem to work for developers, but not for data management professionals. They seem to work for stand-alone BI applications, but not enterprise-class data integration projects. This seminar examines the reasons for the spotty success with agile methodologies in the EDW/BI space. It will describe four different BI scenarios that require different agile approaches. It will also introduce Extreme Scoping, which is a new agile approach designed specifically for enterprise-class data integration projects that require an enterprise perspective. It is a data-agile alternative to today s popular development-agile methodologies. 1

2 Outline Why traditional methodologies do not work for EDW/BI projects Spiral and agile methodologies Agile BI versus agile EDW Agile on the TDWI BI Maturity Model 2 Traditional methodologies and traditional project management practices don t work for enterprise-class EDW/BI projects. There are two major problems with these waterfall methodologies: 1. they do not include extensive data management activities 2. it takes much too long to deliver something tangible to the users Spiral data warehouse methodologies fix problem 1. These methodologies are very popular with enterprise information management professionals working on enterprise-class data integration projects. But the unintended consequence of spiral methodologies is that they add more data management activities to the development process, and now it takes even longer to deliver something to the users. Agile methodologies fix problem 2. These methodologies are very popular with developers working on BI applications. But these methodologies are code-centric, not data-centric. They do not include or even address the extensive data management activities on EDW/BI projects, which can take up as much as 80% of the entire development effort. This seminar will explain the differences between traditional, spiral, and agile methodologies. It will compare four different BI implementation scenarios and their use of agile methods. It will introduce a new data-centric alternative agile approach called Extreme Scoping, which was specifically designed for enterprise-class data integration projects like enterprise data warehousing (EDW) and master data management (MDM). This session will conclude with a mapping of different agile methods on the TDWI BI Maturity Model. 2

3 Business Intelligence Management is a cross-functional discipline and an enterprise architecture for an integrated collection of applications and databases, Delivery which provide the business community easy access to their business data, and allows them to make accurate business decisions. is not a system or a product 3 Business intelligence does not refer to a product you can buy or to a system you can build. Instead it is a new way of delivering and managing the decision support function in an organization. BI uses data like a strategic asset of the company, which when turned into information and applied as knowledge gives the company an advantage over its competitors. For data to become a strategic asset, each unique data element must be carefully managed and reused to provide maximum return on asset. That requires enforcing a set of disciplines across the organization. 3

4 Enterprise Warehouse Role in BI ERP Back Office CRM Front Office Other Legacy & Historical Web & External Order Entry Sales Automation Accounts Receivable Customer Demogr. Management Enterprise Warehouse Operational Store Delivery Exploration Warehouse Marketing Mart Operational Mart Mining CRM Analytical 4 Other BI apps. The enterprise data warehouse is an important building block in BI; it is the underthe-hood engine of BI. BI applications and products require consistent, clean, and integrated data in order to generate accurate information upon which business executives and managers can act. warehousing provides that basic pool of consistent, clean, and integrated data because of its two major objectives: data management and data delivery. 4

5 EDW/BI Goals and Objectives 1. Management data integration data quality 2. Delivery data access data analytics 4 BR DUPLEX 1 BR Business Intelligence Framework 5 Think of these two objectives (data management and data delivery) like a duplex. In a duplex you have two separate units under the same roof. If you were to physically separate the units, you would no longer have a duplex, but 2 single family residences. Similarly, a data warehouse is only a data warehouse as long as it addresses both objectives: data management and data delivery. If you remove either one of the objectives, it is no longer a data warehouse. It may be a data standardization initiative or a stovepipe data mart development activity, but not a data warehouse. Note that the size of the two units are not the same. management is like a 4 bedroom unit taking 80% of the development effort, while data delivery is like a 1 bedroom unit taking only 20% of the development effort. Yet, most organizations concentrate almost all of their efforts on data delivery and pay little if any attention to data management. There are several reasons for this: a) project schedules are very tight and there is no time for data management activities; b) enterprise information management (EIM) staff -like data administrators- are not effectively embedded with EDW/BI projects; c) the old, traditional system development lifecycle (SDLC) methodologies don t demand data management activities; d) departmental end users don t care about enterprise-wide data management if it slows down their projects. 5

6 BI Framework: Business Layer Business Requirements Program Management Development Source: The Warehousing Institute Business Value 6 The Warehousing Institute (TDWI) defines the overall BI framework in 3 layers, starting with the business layer. The business layer contains the components needed for BI to fit seamlessly into business organizations, processes, and activities. The business layer components are: Business requirements: The reasons to implement BI, and the kinds of results needed, including information needs, essential business metrics, etc. Business value: The benefits anticipated from or achieved by BI including such things as increased revenue, improved profit margins, risks mitigated or avoided, reduced costs, etc. Program management: The ongoing activities of managing the BI program for maximum business value including establishing enterprise structures and standards, synchronizing multiple and parallel projects, realigning with changing business needs, etc. Development: The project activities that create and deploy BI and EDW products and applications, including methodology, project decomposition, project team structure, project success measures, etc. 6

7 BI Framework: Administration & Operation Layer Business Requirements BI Architecture Program Management Resource Administration Development BI & EDW Operations Business Applications Source: The Warehousing Institute Business Value 7 The second layer is the administration and operation layer. This layer contains components that connect the technical components with the business components. The administration and operation layer components are: BI architecture: Frameworks, standards, and conventions that describe BI and EDW components and the relationships among them including Business architectures (process models) architectures (data models) Technology architectures (platform components and capacity planning) Organizational architectures (org charts, roles and responsibilities) Project architectures (portfolio management and project plans). Business applications: Business processes and procedures that access and/or receive information and employ that information to achieve business results. resource administration (aka Enterprise Information Management): Policies, procedures, and processes for data governance including data owner, steward, and custodial responsibilities. BI & EDW Operations: Executing EDW processes, monitoring BI functions, and maintaining acceptable quality, availability, and performance of the EDW and BI functions and services. 7

8 BI Framework: Implementation Layer Business Requirements BI Architecture Program Management Resource Administration Warehousing Sources Acquisition, Cleansing, & Integration Stores Information Services Information Delivery Business Analytics BI & EDW Operations Development Business Applications Source: The Warehousing Institute Business Value 8 The third layer is the implementation layer. This layer contains technical components needed to capture data, turn data into information, and deliver that information to the business people. This is the plumbing behind BI. The components are: warehousing: Systems, processes, and procedures to integrate data, store data, and prepare it to become information, which includes metadata management. Information services: Systems, processes, and procedures that turn data into information and deliver that information to the business people either as state-ofthe-art business analytics applications or state-of-the-practice information delivery, such as OLAP cubes and reports. 8

9 From Chaos to Architecture Operational Store Exploration Warehouse Enterprise Warehouse Management Mart Mart Inventory of shared data Mart Oper Mart Oper Mart Oper Mart CRM Anal BI Applications Delivery 9 One of the main objectives of enterprise data warehousing is to create a trusted set of standardized and integrated databases with minimal and controlled data redundancy. One of the main objectives of business intelligence is to ensure that all reports and analytics that run against any of the databases in the data warehouse are accurate and consistent. This requires a different approach to developing our decision support applications. It requires a substantial investment in data management activities, aka data governance, on every EDW/BI project. Customized silos i.e. stovepipe systems with duplicate and inconsistent data, which is stored across a multitude of independent data marts or Excel spreadsheets, are no longer acceptable because it takes too long to reconcile critical business information from them. 9

10 Traditional Development Method Operational Systems (Legacy) L Independent Marts MK Business Units Marketing L FA Finance L L CS PS EN Customer Support Product Sales Engineering 10 The traditional development methods simply do not support EDW/BI goals and objectives. While the early enterprise data warehousing efforts attempted to tame this spaghetti development approach, most were abandoned for the easier and faster silo (independent) data marts. Integration, reconciliation, removing redundancy, and cleansing data are neither quick nor easy endeavors, and they cannot be solved by buying one more technology solutions. It takes detailed analysis by humans across the entire organization to find and resolve the data discrepancies. That takes time. While some products, such as data profiling or data cleansing tools, can help in that effort, they cannot magically resolve all the problems without significant human involvement. 10

11 Uncoordinated Development Business Chief Operating Officer Chief Executive Officer Chief Information Officer Technology Business Units KW KW KW KW KW KW KW Marketing Product Financial Customer Distribution Do You Know Your Business? Sales Inventory Business Manager Business Manager paired up IT Manager IT Manager produces IT IT IT IT IT IT IT Information Technology Units swim lane development data redundancy process redundancy dirty data 11 Unfortunately, companies are not yet organized to build anything other than silos (stovepipe systems). When a business unit has a new requirement, they call their assigned IT unit and submit a service request. The IT technician then meets with the business person, gathers the requirements, and builds a system satisfying those requirements. Never during that time does the IT technician or the business person research where the same data already exists and what similar functionality is already part of another system. They also don t check with other end users of the same data (through a different system) to see if their needs can be combined and to ensure that they use the same definitions and domains for the same data elements. There is no group of people in place to help with this type of cross-functional integration and standardization, nor are there other data management disciplines in place to achieve that goal. Thus, everybody builds their systems any way they like with no coordination and integration between them. As a result, the data chaos of redundant and dirty data keeps growing faster and faster. Business managers at the lowest level in the organizational hierarchy are not very much affected by this because they rarely, if ever, need a cross-functional view of the entire organization. However, business executives at the highest level in the organizational hierarchy have to wait for analysts in IT and subject matter experts on the business side to reconcile the differences in their reports. This reconciliation effort can take weeks. Executives cannot wait for weeks, and so they often make unwise business decisions based on erroneous data. 11

12 Waterfall Methodologies Business Need Project Plan Functional Requirements System Analysis External Design Internal Design System Design Development Client Approval Silos Production System Testing Implementation Maintenance Post-Mortem Review 12 The traditional waterfall system development life cycle (SDLC) methodologies were invented in the 1970 s to build customized operational systems. They simply do not support EDW/BI goals and objectives. The activities and tasks in these methodologies are geared toward creating customized (stovepipe) systems because they contain no cross-functional enterprise integration activities that would involve other end users from other departments during requirements gathering and data analysis. In fact, there is no data analysis phase at all; there is only system analysis. analysis is a business-focused activity, which results in detailed source data requirements. analysis is performed by an enterprise information management (EIM) professional who works with data stewards on the business side. System analysis is a development-focused activity, which results in high-level design specifications. System analysis is performed by a systems analyst or a developer who works with the end user on the business side. In waterfall methodologies, each phase must be completed before the next phase can start, which means that half of the project time is spent on creating paper artifacts, such as voluminous requirements documents, process models and data models. Each project is believed to have a beginning and an end. A project is started because of a business need, and it ends when the customized system is put into production. After implementation, the stovepipe systems get maintained and occasionally enhanced by maintenance technicians, not the original developers. 12

13 Waterfall Methodologies Business Need Project Plan Functional Requirements System Analysis External Design Internal Design System Design Development Client Approval Testing Two problems: 1. No data management effort 2. Takes too long to deliver Implementation Maintenance Post-Mortem Review 13 There are two major problems when using waterfall methodologies to develop enterprise-class solutions: 1. Traditional methodologies were developed to create silo systems, not enterprise solutions. The tasks focus on turning user requirements into working code. These methodologies have no tasks for in-depth data analysis, business data modeling, data standardization, data cleansing, and data integration. These tasks are not needed when creating an operational data-entry system. 2. As we know, reconciling, rationalizing, and standardizing data across the enterprise takes much longer and involves many more users from other departments than simply coding a silo application for one user from one department. Since these waterfall methodologies do not include extensive data management activities (see problem 1), they are of little help when estimating enterprise-class data warehouse projects, and as a result EDW/BI efforts are chronically underestimated. 13

14 Using the Waterfall Approach Requirements are well defined Scope is manageable Scope and users are confined to one business unit (department, business function) Technology infrastructure is known and proven volumes are relatively small Development activities are the same on every project Development activities are not cross-functional Project schedules are relatively easy to estimate Mistakes are less expensive to fix early in the development process! 14 Waterfall methodologies were very popular because they organized the development activities into distinct phases. Each phase had to be completed and signed off before another phase could start. The thinking behind this was that mistakes can be caught early in the development process when they are still much less costly to fix. In the 1970 s and 80 s, that meant that finding a mistake on paper was much easier than fixing unstructured code in a 100,000-line Cobol program with hundreds of go-to statements. This old approach worked for stovepipe (silo) systems, because each system had well defined requirements, a relatively small scope, and it was confined to one set of users or one department. The underlying technology platform was usually known and proven (old mainframes), and data volumes were relatively small. And, the process was more-or-less repeatable because the development activities were almost the same on each project. 14

15 Industrial-Age Mental Model highest to lowest priority Project Constraints Priority PEOPLE BUDGET QUALITY SCOPE TIME QUALITY BUDGET PEOPLE EVERYTHING IS PRIORITY ONE! TIME SCOPE 15 What makes things even worse for us in IT is that most business people are still stuck in the industrial-age mental model. As an exercise, I ask end users to choose their priorities from 1 to 5 (1 being highest and 5 being lowest) among the constraints of people, budget, quality, scope, and time. Very often they list all of the constraints in the first column as their highest priority. They don t realize the contradiction when they assume that they can get all of their requirements (scope) in the shortest amount of time (120 days or less) with 100% quality on a fixed budget with limited resources. These expectations are quite unrealistic, but they are still very common. 15

16 Industrial-Age Mental Model highest to lowest priority Project Constraints Priority TIME SCOPE BUDGET PEOPLE QUALITY Scrap and rework (Larry English) Cost-based value proposition Cheaper, faster, better Automate as quickly as possible 16 A second try at the same exercise, usually yields this result: The traditional development approach and waterfall methodologies emphasized automation of manual business processes as cheaply and as quickly as possible. That made time (schedule) the highest priority on a project, immediately followed by scope, which always had to be a complete and fully functioning system. Budget and people were a distant third and fourth priority, with quality often not even on the map. Return on investment (ROI) calculations in this model are purely cost-based, comparing the business value of an automated process to the cost of building it. Since these systems were built independently of each other, the cost of scrap and rework never entered into the equation, and neither did the cost of adding more and more redundant applications and redundant databases to the IT portfolio. The problems with the industrial-age mental model are clear to everyone today. is duplicated hundreds of times with totally inconsistent values. Reports are produced with inconsistent numbers that must be explained to business executives after weeks of researching and reconciling. Historical data is not easily accessed and integrated because file structures have changed, and archived data ended up being stored on different storage devices. And worst of all, data is still not integrated between the stovepipe systems. At best, bridges are built between systems. However, now these bridges have to be maintained, which requires additional staff for additional costs. 16

17 Time versus Quality Everyone wants quality, but rarely is the extra time given or taken to achieve it. Quality and time are polarized constraints. The higher the quality the more time it takes to deliver. Companies are driven by shorter and shorter schedules. SCOPE YAH TIME DDD 17 When business people are asked about their requirements for data quality, they usually ask for 100%. But when the same people realize that quality-related tasks must be consciously executed during the development of an application, which invariably delays the delivery of the application, they often are not willing to extend the project schedule beyond 0%-10%. As a result, many organizations pay little or no attention to data quality. Time and quality are at two opposite ends of the same spectrum; the higher the quality the longer it takes to build a product. And, therein lies the quandary there s never enough time in our fast-paced environment. In fact, time is getting shorter and shorter while the scope of our projects is getting larger and larger. Yet, quality does not happen by osmosis. On the contrary, what is almost guaranteed to happen by osmosis is data deterioration and data disintegration. 17

18 The Time Trap Redundancy increases Maintenance increases Need for more staff increases Quality decreases Development time increases Employee burnout increases Inability to support business drivers User satisfaction decreases Productivity decreases Business value decreases 18 Giving in to the time pressures and ignoring data quality is a time trap in which many organizations are caught. The dirtier the data, the less users trust it, and the more they build their own silo systems. As redundancy increases, data quality gets even worse and user satisfaction decreases even more. At the same time, maintenance of the redundant systems increases, which leads to a need for more staff. Development time also increases because it takes longer to sift through the pool of redundant databases and applications. And because time is still of the essence, employees work longer and harder, which leads to employee burnout and that in turn leads to decreased productivity, and so on. Being seduced by the time trap leads to decreased business value and can potentially result in the inability to support the organization s business drivers and thus put a company out of business. 18

19 Proliferation of Redundancy Operational Systems (Legacy) BI? Independent Marts Business Units L MK Marketing L L transformation? cleansing? FA CS Finance Customer Support L PS EN Product Sales Engineering Framework? 19 As disheartening as the situation is with our existing systems, it is very discouraging to see and difficult to believe that organizations continue to build applications, even BI applications, in the same stovepipe manner as always, only adding to the data chaos. This is not a framework for enterprise-wide data standardization and integration, which, after all, is the goal of enterprise-class data warehousing. 19

20 Adding to the Chaos EDW DM EDW and Marts DM DM transformation? cleansing? DM EDW DM DM DM DM 20 Traditional methodologies are designed to produce stand-alone products. As long as we don t change our mental model as well as our development approach from building individual systems to building an integrated enterprise-wide decision support environment, we will continue to add more independent databases and more applications to the current data chaos, regardless what we call the new databases or what new technology we use. This is not true BI! 20

21 The Lesson? You cannot keep doing what you have always done and expect the results to be different. That wouldn t be logical Spock, Star Trek 21 The lesson should be obvious: It is completely illogical to expect the outcomes to be different if we do not change our ways to produce them. There are common root causes for why we have data chaos today. Once we know these root causes, it is up to us to remove them and to change our behavior in terms of system development, and to start treating data as the valued business asset that it is. This is the data management part of any EDW and it is the foundation for BI. 21

22 Fundamental Changes 1. Stop working in isolated swim lanes Standardize and share data 2. Centrally manage data like a business asset (Enterprise Information Management [EIM]) Standardize the business data Assemble data as needed from the data inventory (enterprise data model and metadata repository) Standardize and reconcile data transformations Standardize business metrics for BI applications 3. Modify your methodology to be based on cross-functional disciplines (e.g., EIM activities) 4. Scale down project scopes to incorporate EIM activities Use the release concept (agile approach) 22 You cannot be doing what you have always done, and expect the results to be different. Here are some of the fundamental changes we have to adopt: 1. First and foremost, we must stop building stovepipe solutions and coordinate our development efforts. 2. Second, and equally important, we must manage data as a business asset, which requires creating a centralized enterprise information management (EIM) function and embedding EIM staff with every EDW/BI project. 3. Third, we must modify SDLC methodologies and system development approaches to include cross-functional activities that will lead to data standardization and integration. 4. Fourth and finally, we must sharply reduce the scope of projects and develop applications using the release concept. The smaller the scope, the more time can be allocated to quality-related tasks and EIM activities. 22

23 Outline Why traditional methodologies do not work for EDW/BI projects Spiral and agile methodologies Agile BI versus agile EDW Agile on the TDWI BI Maturity Model 23 Traditional methodologies and traditional project management practices don t work for enterprise-class EDW/BI projects. There are two major problems with these waterfall methodologies: 1. they do not include extensive data management activities 2. it takes much too long to deliver something tangible to the users Spiral data warehouse methodologies fix problem 1. These methodologies are very popular with enterprise information management professionals working on enterprise-class data integration projects. But the unintended consequence of spiral methodologies is that they add more data management activities to the development process, and now it takes even longer to deliver something to the users. Agile methodologies fix problem 2. These methodologies are very popular with developers working on BI applications. But these methodologies are code-centric, not data-centric. They do not include or even address the extensive data management activities on EDW/BI projects, which can take up as much as 80% of the entire development effort. This seminar will explain the differences between traditional, spiral, and agile methodologies. It will compare four different BI implementation scenarios and their use of agile methods. It will introduce a new data-centric alternative agile approach called Extreme Scoping, which was specifically designed for enterprise-class data integration projects like enterprise data warehousing (EDW) and master data management (MDM). This session will conclude with a mapping of different agile methods on the TDWI BI Maturity Model. 23

24 Information-Age Mental Model highest to lowest priority Project Constraints Priority QUALITY TIME PEOPLE BUDGET SCOPE Reassemble reusable components (John Zachman) Investment-based value proposition Reassemble the entire enterprise Reuse assets from inventory 24 To start with, we have to replace the industrial-age mental model with the information-age mental model, which emphasizes the ability to reassemble the entire organization from reusable components, especially reusable data. This model demands that the quality of reusable components be as pristine as practical and puts quality on the top of the list followed immediately by time (schedule). People and budget are still a distant third and fourth priority, with scope being last. This requires a new release approach. Return on investment calculations in this model include return on asset (ROA) calculations, which are investment-based (asset-based). ROA calculations compare the business value of the new application to the cost-savings (lifetime value) achieved through reusability of components, including reusability of data. 24

25 Coordinated Development Business Units KW KW KW KW KW KW KW Sales Inventory Distribution Customer Product Financial Marketing IT IT IT IT IT IT IT Information Technology Units Discover, Communicate, Integrate, Document, Coordinate Enterprise Information Management Operational Environment Operational Systems ODS OM EDW DM EDW/BI Decision Support Environment cross-functional development 25 The previous discussion leads to one obvious conclusion: we must switch from stovepipe development to a coordinated cross-functional development approach. A cross-functional development approach includes tasks and resources to discover, communicate, integrate, document, and coordinate all interdependent project activities under one BI program management umbrella. This will ensure standardization and reusability of data and processes across the EDW/BI environment. The tasks include enterprise architecture principles, such as creating an inventory of reusable data and process components in the operational as well as decision support environment, as well as collectively architecting databases and processes with special attention to quality. This will require some fundamental changes to our system development habits, including the involvement and assistance of a centralized enterprise information management (EIM) group. This group is also known as data administration (DA), data resource management (DRM), and information resource management (IRM). 25

26 Cross-functional Development Commitment to data standardization and integration embedded in the methodology Cross-functional BI program management Enterprise information management group Standards that include a common information architecture (enterprise data model) Coordinating the development processes Disallowing stovepipe development Extracting and cleansing source data only once Reconciling data transformations and storing the reconciliation totals as metadata < principles < governance < resources < policies < enforcement 26 What does it mean to have a development approach with a cross-functional focus? First, it means that managing data as a strategic asset is an expressly stated and accepted business principle in the organization. Therefore, data integration and data standardization tasks are embedded in the methodology. Second, a coordinated development approach means that the requirements and the development of multiple applications for multiple users or departments is prioritized and coordinated under a BI governance program. Third, resources are made available to support and coordinate the cross-functional activities, such as an independent enterprise information management or data administration group that reports to a chief data officer. Fourth, standards, policies, and procedures for cross-functional integration are developed and published. Fifth, these policies are enforced. Enforcement can be a proclamation that stovepipe systems will no longer be allowed to be put into production. It can also mean that there is a mandate to go through a common coordinated staging area or a mandate to reconcile all databases. Compliance is ensured by tying it to people s performance appraisals, promotions, salary increases, and bonuses. 26

27 Solves problem 1 (data mgmt) Spiral Methodologies Assessment Assessment & Strategy Strategy Requirements Requirements Management Business Business Opportunity Opportunity Project Project Planning Planning Post-Impl. Post-Impl. Review Review Evolving Warehouse Integration Integration Analysis Analysis Implementation Implementation Application Application Prototyping Prototyping Testing Testing Iterative Development = One application at a time Development Development Delivery 27 Coordinated cross-functional development does not happen by accident. It needs an enterprise infrastructure with technical and non-technical components. One of the non-technical components should be a spiral methodology with activities and tasks that address cross-functional principles, governance, resources, policies, and enforcement of those policies. The reason it is called a spiral methodology is because it supports building the EDW/BI environment (databases and applications) in iterations, one BI application at a time. Spiral methodologies do not presume to build stand-alone systems with a distinct beginning and end. Instead they support building an evolving and expanding EDW/BI environment for an integrated collection of non-redundant (or little redundant) databases and applications. In other words, they solve problem 1 of waterfall methodologies, which is no data integration effort. Note that integrated is not the same thing as consolidated. To consolidate data simply means to collect it in one [logical] place. To integrate the data means to consolidate the data plus reduce redundancy plus standardize each data element with a unique name, definition, and domain plus build or rebuild the relationships between the business objects as they exist in the real world. This is accomplished with an enterprise data model. Unfortunately, the unintended consequence of spiral methodologies is that these additional cross-functional tasks add even more time to the development schedule, thereby exacerbating problem 2 of waterfall methodologies, which is takes too long to deliver. 27

28 Solves problem 2 (speed) Final Release Release Concept First Release Second Release Projects feels like prototyping BI Application Reusable & Expanding Fifth Release Fourth Release Third Release Refactoring One Application = Multiple Releases 28 The only way to solve problem 2 is with the release concept. We all agree that an integrated EDW/BI environment (EDW databases and BI applications) cannot be built in one big bang. While most organizations realize that they cannot build an entire enterprise data warehouse with all functionality all at once, they do not realize that they cannot and should not even build an entire BI application all at once either. The vendors promise that you can build a data mart in less than 30 days using their product. Sure you can - if you use the traditional development approach and build a stovepipe data mart solution, which is only the 20% effort of data delivery, and completely ignore the 80% effort of data management. Remember that a data warehouse is not a data warehouse if you don t address both data management and data delivery. Since data management involves standardizing and integrating data from multiple source files/databases, the data analysis effort alone can easily take 30 days or longer, if you try to delivery the entire scope of the application all at once. Therefore, if you want to stay true to building a standardized and integrated enterprise-class EDW and not revert back to building silo BI applications, and if at the same time you want to deliver something tangible every few months, you have to break each BI application request into multiple releases. By doing so you solve problem 2 of waterfall methodologies, which is takes too long to deliver. 28

29 Solves problem 2 (speed) Agile Methodologies Business Business Vision Vision (Srvc (SrvcReq) Product Backlog (Reqmts) Speculation Speculation (Planning) (Planning) Production System FOCUS on: Coding apps Exploration Exploration (Prototyping) (Prototyping) New development and refactoring Project Project Retrospective Retrospective (Review) (Review) Software Software (Implementation) (Implementation) 29 That leads us directly to agile methodologies. Agile methodologies do not recognize a service request for a new system to be the final set of requirements that the team has to deliver on a committed date, which was calculated based on their estimates (best guess) of how long it will take to turn the requirements into working software. Instead, they view the request as a business vision for a system that may or may not end up looking exactly like originally requested when it is finally delivered. With the participation of the user, the developers dissect the requirements into desired features, which are put as user stories on a product backlog. The users control the product backlog. They can add requirements to it and they can remove requirements. They are also responsible for prioritizing the requirements/features and selecting a few for the first (or next) release. Rather than come up with an estimate for the entire service request that will be cast in concrete, the team speculates how long it will take to turn the selected features into working software based on what is known to them at that point in time. The development effort for the selected features must fit into a predefined sprint (time-box of 10 or 29 days). If it doesn t, one or more features are de-selected. Progress for developing the requested features is measured by the number of features delivered and not by the number of tasks performed. Any tasks deemed necessary can be performed anytime by anybody during exploration (prototyping). Progress is tracked on a burn-down chart, which shows the total number of effort-hours (Y axis) plotted on an elapsed timeline (X axis). The chart reflects the effort speculated versus the effort spent. If the trajectory of the burn-down chart indicates that the deadline cannot be met, the scope is immediately adjusted or the project is cancelled. The deliverable of a sprint is production-worthy software for the selected features. In other words, it is a partially functioning system. Before the next sprint is started, the team takes one day to review lessons learned and to plan the next release. 29

30 Agile Manifesto We are uncovering better ways of developing software by doing it and helping others do it. Through this work we have come to value: Individuals and interactions over processes and tools Working software over comprehensive documentation Customer collaboration over contract negotiation Responding to change over following a plan That is, while there is value in the items on the right, we value the items on the left more. Source: 30 The Agile Manifesto challenges the traditional system development methodologies that place a lot of emphasis on following old system development processes and relying heavily on tools, creating extensive documentation, having users sign off on artifacts and contracts, following a detailed plan. Instead, the Agile Manifesto places much more emphasis on being flexible by placing more value on individuals working together, producing production-worthy deliverables frequently, involving the users throughout the development process to get immediate feedback, being open to scope changes. 30

31 Principles of Agile Development Business vision instead of final requirements Speculation instead of estimating Exploration instead of development Self-organizing project team (no interference) Co-location (sit in the same room with users) Daily stand-up meetings Pair programming (shared responsibility) Scrum Master (project manager) Product owner (user) participation Prioritize and re-prioritize requirements (scoping) Give constant and immediate feedback 31 The first important principle of agile development is to recognize that the initial set of requirements are seldom what the users really want. It s what they think they want when they request an application. But almost always, they change their own requirements during the development process. Therefore, the initial set of requirements are called nothing more than the initial business vision, which is expected to change. Since the term estimating has become to mean I guarantee these dates, the term speculation is used instead. And instead of following the waterfall phases of requirements definition, analysis, design, coding, and testing, all of these phases are combined into the concept of exploration or prototyping. Another important agile principle is to insulate the project team from management interference and interruptions. The team members alone decide what tasks to perform and how and when to perform them. To facilitate self-organization, colocation requires that all team members sit together with their users in the same room, like a project war room. Daily stand-up meetings replace the weekly status report meetings and keep the project team on target. The principle of pair programming is based on sharing the responsibility for quality control. And the role of the project manager is redefined as a Scrum Master (mixture of team leader, project manager, and Scrum guru) with specific responsibilities and duties. End users are called product owners because they pay for and own the system being built for them. 31

32 Principles of Agile Development Get physical as quickly as possible (deliver working code) Backlog with user stories (list of requirements) Time-boxed increments or sprints Freezing the scope of an increment Cadence is the release rhythm (Scrum 29 days, XP 10 days) KISS (simple, reusable, production-worthy code modules) Quality before quantity (write test cases before code) Refactoring (refinement = system evolution) Software releases instead of finished application Burn-down charts instead of Gantt or Pert charts 32 Another agile principle is to start coding as soon as possible rather than take months to document the requirements and design the system on paper. The requirements document is replaced by the backlog with user stories, which are controlled by the end users. They can add or remove requirements from the backlogs at will. Another important principle is the replacement of a detailed project plan for the entire set of requirements with time-boxed development increments. After each such increment, a portion of the requested system is ready for implementation in production. While the increment is being worked on, the scope is frozen. Cadence refers to the predetermined time span for the time-boxed development increments. In Scrum that rhythm is 29 days and in extreme Programming (XP) it is 10 days. The principle of KISS (keep it simple stupid) emphasizes the importance to write clean, uncomplicated, modularized, and reusable code. Another important agile principle is to put quality before quantity (referring to code quality and system quality). Refactoring is the concept of refining and enhancing a system over time. Another way to say it is to let the system evolve. This is accomplished through frequent software releases of a small number of system features rather than delivering all features of a finished application all at once. Another agile principle is to discard the cumbersome and complicated Pert charts and Gantt charts, which require hours on a project management tool to maintain. Instead they use burn-down charts, which is a plot of effort hours (Y axis) across an elapsed time line of days or weeks (X axis). 32

33 Using the Release Approach Unstable requirements can be tested and enhanced in small increments Scope is very small and manageable Technology infrastructure can be tested and proven volumes (per release) are relatively small Project schedules are easier to estimate because the scope is very small Development activities can be iteratively refined, honed, and adapted Mistakes are less expensive to fix early in the development process! 33 Just like the traditional methodologies, agile methodologies also support the belief that mistakes should be caught early in the development process when they are still much less costly to fix. But instead of breaking the project horizontally into traditional development phases with strict sign-offs for each phase (e.g., analysis, design, coding), agile methodologies achieve the same result by vertically controlling the scope of each release of an application. In other words, the entire service request is broken up into pieces (sprints) and each piece is fully developed into a production-worthy deliverable. Keeping the scope of each sprint (piece) extremely small gives you the same control for a high-quality deliverable. This is possible because the requirements, which are typically unstable, can be implemented in small increments, with each partial deliverable being a usable albeit incomplete portion of the application. This approach allows the end users to solidify their requirements as they learn more about the capabilities and limitations of the evolving system. Not only can the requirements be solidified this way, but the new technology components can be tried and tested in increments as well. And, breaking an application into several small releases gives you the opportunity to establish an efficient repeatable process. 33

34 Using the Release Approach Unstable requirements can be tested and enhanced in small increments Scope is very small and manageable Technology infrastructure can be tested and proven volumes (per release) are relatively small Project schedules are easier to estimate because the scope is very small Development activities can be iteratively refined, honed, and adapted And the quality of the release deliverables (and ultimately the quality of the applications) will be higher! And the development process will get faster and faster! 34 There are two additional benefits to using the release concept instead of trying to complete an application in one long project. 1. The quality of each release deliverable is much higher because you have time to address quality rather than rush to meet an impossible deadline to deliver all the data and all the functionality of an application all at once. In addition, you have a chance to refactor (refine) the deliverables in future releases. 2. You have the chance to review your development approach (tasks, schedules, procedures, skills, staffing, etc.) after each release and improve it. That means that your development process will get faster and faster. 34

35 Outline Why traditional methodologies do not work for EDW/BI projects Spiral and agile methodologies Agile BI versus agile EDW Agile on the TDWI BI Maturity Model 35 Traditional methodologies and traditional project management practices don t work for enterprise-class EDW/BI projects. There are two major problems with these waterfall methodologies: 1. they do not include extensive data management activities 2. it takes much too long to deliver something tangible to the users Spiral data warehouse methodologies fix problem 1. These methodologies are very popular with enterprise information management professionals working on enterprise-class data integration projects. But the unintended consequence of spiral methodologies is that they add more data management activities to the development process, and now it takes even longer to deliver something to the users. Agile methodologies fix problem 2. These methodologies are very popular with developers working on BI applications. But these methodologies are code-centric, not data-centric. They do not include or even address the extensive data management activities on EDW/BI projects, which can take up as much as 80% of the entire development effort. This seminar will explain the differences between traditional, spiral, and agile methodologies. It will compare four different BI implementation scenarios and their use of agile methods. It will introduce a new data-centric alternative agile approach called Extreme Scoping, which was specifically designed for enterprise-class data integration projects like enterprise data warehousing (EDW) and master data management (MDM). This session will conclude with a mapping of different agile methods on the TDWI BI Maturity Model. 35

36 Can agile be used for BI? Scrum XP Scenario 1 Scenario 2 Solves problem 2 FOCUS on: Coding apps BI BI BI architecture? integration? BI Apps Silos Staging Area base architecture? integration? BI Apps Delivery BI BI Independent BI applications Directly sourced from operational databases Developed by the same or by different BI groups Independent BI applications Sourced through a central staging area database containing copies of operational databases Developed by the same or by different BI groups 36 That brings us to the next question: Can agile be used for BI? Well, that depends on what you call BI and what you call agile. There is a growing number of companies that profess to be using agile methods on BI projects, in particular Scrum and XP. My research shows that those companies restrict their development effort to independent data marts with silo BI applications, very similar to the stovepipe operational systems that the agile practitioners are developing on the operational systems side. In other words, in scenarios 1 and 2, the BI teams don t deal with data standardization and integration or at least, not very effectively and not from an enterprise perspective. They only solve problem 2 of waterfall methodologies, namely takes too long to deliver. 36

37 Can agile be used for BI? Solves problem 2 Scrum XP Scenario 3 BI Apps FOCUS on: Coding apps ETL Staging Area base Enterprise Warehouse Mart Mart Mart Delivery CRM Anal Management Solves problem 1 FOCUS on: Enterprise Standardization and Integration Dependent BI applications Sourced from an integrated and standardized EDW Developed by the same or by different BI groups But, building/enlarging the EDW is a different project and a different team 37 Scenario 3 looks like the perfect textbook implementation of an enterprise-class EDW/BI environment, but in fact, it is worse than the last two scenarios because it separates the development of the EDW from the development of BI applications. Some BI teams wait for the data to be ready in the EDW and its dependent data marts before they develop selected BI features often using XP because that type of front-end effort can often be accomplished within days or weeks. Other BI teams cannot wait for the data to be ready in the EDW, and they develop their silo BI solutions by going directly against the source systems. I find that most of these types of projects use a form of Scrum, which gives them a little more time to deal with data issues in the ETL. Both of these approaches focus on writing code (BI application) and not on standardizing, integrating, cleansing, and documenting the data. In fact, I often hear complaints that the data is the problem on these agile BI projects. Clearly these companies don t understand that data management (content) is the most important aspect of BI and not just data delivery (functionality). Once again, they only solve problem 2 of waterfall methodologies, namely takes too long to deliver. But the worst thing about this scenario is that the BI team then puts pressure on the EDW team to use Scrum or XP for their data standardization and integration work. The only problem is that neither Scrum nor XP address data management activities. Scrum and XP were invented by developers for other developers to streamline their development work, i.e. coding software. Therefore, separating the teams and pinning them up against each other not only disrupts the cohesion of the effort but creates an unfair competition and even animosity between the teams! Some companies have gone so far as to separate the EDW team from the BI team and have both groups report to different managers. In many cases, I have seen their EDW/BI initiatives cancelled all together because of the war between the groups. 37

38 Can agile be used for EDW? Scenario 4 Scrum XP Architecture Exploration Warehouse Management Mart ETL Operational Store Enterprise Warehouse Mart Mart Oper Mart Oper Mart Oper Mart CRM Anal Dependent BI applications Sourced from an integrated and standardized EDW or from dependent data marts Developed by the same group Building or enlarging the EDW is the same project and the same team BI Applications Delivery 38 If your definition matches scenario 4 where BI includes building and loading the EDW components necessary to support the BI application, and having that effort be part of delivering the BI features, and if you want to apply agile to building the entire end-to-end solution (including data standardization, enterprise data modeling, and ETL), then using agile methods like Scrum or XP will not work because those development methods were never designed for data-centric enterprise solutions! They were designed for code-centric stand-alone (silo) systems. In other words, they do not address problem 1 of waterfall methodologies, which is no data integration effort. Using the proverbial 80/20 rule, 80% of our combined EDW and BI development effort is on the back-end data management side and only 20% of our effort goes into building the front-end BI application. To make things more complicated, while BI applications are separate and independent pieces of software, a collectively architected ETL process is not, which makes the ETL architecture as well as the ETL software extremely complex, not to mention the collectively architected EDW databases. By their own admission, agile practitioners concede that the more complex the system architecture and the software are, the more thinking (architecting) has to be done before coding. On EDW projects, the thinking alone can sometimes take weeks to avoid omissions and errors in the collective architectures, which later could result in rework measured in months. This is the main reason why BI teams who want to use Scrum or XP separate their front-end BI activities from all back-end EDW activities. But in my opinion, separating BI from EDW is the wrong solution. The right solution is to find a compromise that will include EDW activities in the delivery of BI features and to use an agile method that works for both groups. 38

39 Who invented it and why? Developers (mainly object-oriented) Abandon waterfall methodologies Officially adopted since Cutter Consortium Agile Project Management Practice ( Agile Alliance ( Famous authors: Ken Schwaber (Scrum) Jeff Sutherland (Scrum) Mike Beedle (Scrum) Kent Beck (XP) Not EDW/BI Professionals! Sanjiv Augustine (Agile PM) Jim Highsmith (Agile PM) 39 Many agile authors and practitioners are either project managers or senior developers, each with decades of experience developing operational systems - most written with object-oriented code. Individually, and collectively, they have compiled volumes of evidence why traditional waterfall methodologies do not work not even on operational systems and why those methodologies should be abandoned. They have applied and refined their agile techniques since the mid 1990 s and officially created and published the Agile Manifesto in the time frame. Most authors and practitioners are members of the Agile Alliance. However, these authors are not EDW/BI professionals, and none of the popular agile methodologies were developed with EDW or BI in mind. Writing software to create stand-alone operational systems does not require data integration efforts like data standardization, enterprise data modeling, business rules ratification by major business stakeholders, coordinated ETL data staging, common metadata, collectively architected (designed) databases, and so on. Therefore, it must be understood that all popular agile methodologies are focused on writing and delivering quality software (code) as quickly as possible without significant regard or focus on data. management activities are not included -or even consideredby these agile methodologies because data management is neither part of operational systems requirements nor part of operational software solutions. 39

40 Agile principles that work for EDW projects Business vision instead of final requirements Speculation instead of estimating Exploration (prototyping) instead of development Self-organizing project team (no interference) Daily stand-up meetings (course corrections) Pair programming (development track teams) Get physical as quickly as possible (deliver partial functionality or data) Time-boxed increments (but no cadence) 4 E () Quality before quantity Refactoring (refinement = system evolution) Releases instead of finished application 4 E s: Experimental Experiential Educational Evolutionary 40 However, that does not mean that enterprise-class EDW/BI projects cannot be agile. There are many agile principles that apply to EDW/BI projects. For example, the emphasis on treating the initial requirements as a business vision rather than cast-inconcrete requirements, or not committing to estimates, which are nothing more than best-guess speculations anyway. Exploration (prototyping with continuous refinements) makes much more sense than trying to get the whole application right from the start. Giving the project team full authority over their work, without interference from anyone outside the team, including the users and IT management, can be applied to EDW/BI projects. Daily meetings to identify risks and hurdles are much more productive than long weekly review meetings. Furthermore, nobody should work alone; all tasks and deliverables must be co-owned and co-developed (e.g., pair programming). Agile also places more importance on working deliverables than comprehensive documentation and makes frequent deliverables its primary goal. In order to manage projects without tracking task activities, all releases must be time-boxed into weeks or months not years. Quality comes before quantity and is applied to data quality on EDW/BI projects. Since systems evolve one release at a time, all deliverables can be refactored (refined, redesigned, simplified, improved) with each release. 40

41 Agile principles that do not work for EDW projects Freezing the scope of an increment EDW increments are too large to freeze, BDTP Balance must be renegotiated Scrum Master (project manager) Core team with BDTP Balance Product owner BI Steering Committee prioritize and reprioritize requirements Constant and immediate feedback < Business person on core team Select features from backlogs based on code complexity Create releases based on BDTP Balance Project backlog tied into program backlog (not user controlled) 41 But there are also some agile principles that only apply to the code-centric development of operational systems and not to data-centric development of enterprise-class EDW/BI solutions. For example, we don t have the luxury to simply freeze the scope because our EDW/BI releases are too large. You can freeze the scope for 10 days with XP, but you must allow scope changes on 3 to 4 months projects. The responsibilities and duties of a Scrum Master are totally inadequate on EDW/BI initiatives because a developer, which is usually who a Scrum Master is, does not have the combined expertise in the areas of business value, data effort, technical feasibility, project and program interdependencies. It is imperative to have BDTP Balance on an EDW/BI management team. Calling users product owners is not appropriate because the EDW is not owned by any one user. Instead, it is owned by the organization, which is represented by the BI Steering Committee. Therefore, each project has many product owners who will have to agree on all strategic and tactical project decisions. However, the primary business person who submitted the service request for a BI application can play a limited role of product owner in terms of giving constant and immediate feedback. Selecting features from a user-controlled backlog of requirements based on code complexity does not make any sense because it is the data effort required to support the selected features that determines how much can be delivered in a predetermined time-frame, not code complexity. Therefore, a EDW/BI release cannot and should not be based on code complexity. Instead, the scope and deliverable for each EDW/BI release is again totally dependent and defined by the BDTP Balance. In addition, the project backlog is tied into the overall program backlog. Therefore, one user cannot indiscriminately and continuously monopolize the backlog without eventually affecting other users who are waiting for the EDW/BI team to work on their service requests. 41

42 Agile principles that do not work for EDW projects Co-location Not possible because of geographic disbursement Cadence = development rhythm Not possible because of BDTP interdependency KISS (simple, reusable, production-worthy code modules) Integrated architecture for ETL and databases to deliver clean, standardized, integrated, ratified, reusable data Burn-down charts instead of Gantt charts or Milestone charts Limited internal use by development track teams 42 Co-location is a great idea and it does work. However, on enterprise-class EDW/BI projects, the team members and their users are often geographically disbursed across the country if not the world. Cadence (set rhythm) is another principle that cannot and should not be enforced on EDW/BI projects. Because of the vastly different data efforts that are required to deliver various BI features, it is entirely possible that some releases will take months to complete, while other releases can be finished in a few weeks. On EDW/BI projects, the data effort is the determining factor in defining the length of a time-box. The KISS principle is somewhat applicable to EDW/BI projects, but must be expanded to include the data standardization and integration activities. Another EDW/BI complexity that does not exist on stovepipe operational systems is that EDW databases and the ETL process is collectively architected in order to be reusable. That limits how uncomplicated the code can be. Burn-down charts are popular with developers. However, they appear to be quite unpopular with business managers and executives who much prefer to see the familiar Gantt chart, or at least a milestone chart. Gantt charts track activities, resources, and dependencies, all of which is not tracked in agile methodologies. Therefore, Gantt charts should be replaced with milestone charts. 42

43 No Silo and No Big-Bang Development Solves problems 1 & 2 Enterprise Business Infrastructure Business Vision Vision (Srvc (Srvc Req) Req) Business Business Opportunity Opportunity Post-Impl. Post-Impl. Product Review Review Backlog (Reqmts) Assessment Assessment & Strategy Strategy Speculation Speculation (Planning) (Planning) Merge Spiral with Agile! = Extreme Scoping Project Project Plan Plan Evolving Warehouse Requirement Exploration Requirement Exploration (Prototyping) (Prototyping) Business Business Analysis Analysis Integration Implementation Implementation Project Project Retrospective Retrospective (Review) (Review) Testing Testing Development Development Application Application Prototyping Software Prototyping Software (Implementatn) (Implementatn) 43 Since the code-centric agile methodologies currently on the market (e.g., Scrum, XP) were not created by EDW professionals, they consequently were never meant to be used on datacentric EDW/BI projects. Scrum and XP do not consider all of the data management tasks and complexities inherent to enterprise-class EDW/BI projects. Therefore, we must not adopt these agile methods blindly, but apply as many agile principles as we can to our own expanded spiral methodologies. We must also have the freedom to ignore and not implement those agile principles that do not support our EDW/BI projects. In other words, we should merge agile and spiral methodologies to deliver enterprise-class solutions. 43

44 Dependent BI applications Sourced from an integrated and standardized EDW or from dependent data marts Developed by the same group Building or enlarging the EDW is the same project and the same team Can agile be used for EDW? Architecture ETL Operational Store Oper Mart Exploration Warehouse Oper Mart Solves problems 1&2 Oper Mart Enterprise Warehouse BI Applications CRM Anal Mart Mart Extreme Scoping FOCUS on: Mart Enterprise Standardization and Integration NOT: Coding apps 44 One such merged approach is Extreme Scoping. It is an EDW/BI-specific agile method that focuses on enterprise data standardization and integration and not on coding applications. To create an inventory of reusable databases and shared BI applications, requires the following enterprise information management (EIM) activities to be performed: standardization Common data definitions Fully qualified data names Standardized business rules Enterprise data modeling Source data analysis cleansing rules These data management activities are the most crucial and the most time consuming of the back-end data preparation process. They must be performed diligently and with the participation of data owners, data stewards, and information consumers from other business units. These activities will determine the total effort and time needed to develop the EDW/BI releases. 44

45 (Trademark of Larissa T. Moss, Method Focus Inc.) Extreme Scoping 1. Speculate on the total effort for the new service request 2. Break the new service request into releases 3. Create list of activities and tasks for first release 4. Organize parallel development tracks for first release 5. Describe weekly milestones per track for first release 6. Schedule work assignments across weekly milestones 7. Create milestone chart for progress reporting 45 The Extreme Scoping planning process encompasses both the rigor of a spiral data-centric methodology and the agility of most agile principles. It is comprised of 7 steps. The first step is a diligent review of the entire soup-to-nuts spiral methodology to select the appropriate development steps for the entire set of initial requirements of a new EDW/BI service request. The second step is to break the new EDW/BI required into releases by carefully balancing business value, data effort, technical feasibility, project and program interdependencies. The third step begins with another review of the selected development steps (from step 1), but this time you select the appropriate activities or tasks for the first release only. In the fourth step, the release activities can now be organized into the appropriate number of development tracks for the first release only. In the fifth step, each development track team determines their weekly milestones. They start with the release deadline and work backwards using the list of activities or tasks (from step 3) as a guide. The sixth step is to schedule all the selected activities or tasks for each development track team across the weekly milestones and to create a micro plan for internal use by the core team members to manage the project. The seventh step is to create a macro project plan, which is an aggregated milestone chart, for reporting project progress to management. 45

46 Extreme Scoping Release Guidelines Frequent delivery: every one to four months (first release will take longer) Small scope and manageable deliverables (incl. EDW) Manage expectations during the evolution of the application Allow the self-organizing project teams to organize and manage their own projects Metadata must be an integral part of each release; otherwise, the releases will not be manageable Designs, programs, and tools must be flexible Small errors or defects are addressed with changed control procedures Large errors or defects are deferred to another release Nothing is cast in concrete; everything is [re-]negotiable 46 Final Release First Release Second Release BI Application Third Fifth Release Release Fourth Release Extreme Scoping release guidelines: A release does not equal a completed BI application. It may take several releases to implement a fully functioning BI application. Releases should be delivered every one to four months. The first release of an application should only deliver the basics. Business management must be willing to accept the evolution of an application and must stop insisting on minimum required deliverables, which usually means they want the entire BI application with all of its EDW components all at once. All expectations must be managed continuously. Let the self-organizing project teams put together their own project plans and allow them to manage their own projects in a way that is most effective for them. Releases must have a finite scope with small and manageable deliverables, which includes the EDW pieces. The infrastructure, both technical and non-technical, must be robust, especially the metadata solution. Developing releases feels like prototyping, but the process is more disciplined and controlled because it produces production-worthy deliverables. Small errors or defects are addressed during the project. Large errors or defects are deferred to a future release by removing the function or the data associated with the defect from the scope. Nothing is cast in concrete. Everything is negotiable. That includes scope, time, budget, resources, and quality. Designs, programs, and tools must be flexible to allow for occasional redesigns of databases and applications. 46

47 Agile Manifesto We are uncovering better ways of developing software by doing it and helping others do it. Through this work we have come to value: Individuals and interactions over processes and tools Tangible deliverables over comprehensive documentation Customer collaboration over contract negotiation Responding to change over following a plan That is, while there is value in the items on the right, we value the items on the left more. Source: 47 EDW/BI project teams can apply the Agile Manifesto as follows: Individuals and interactions over processes and tools. The most valued assets on an EDW/BI project are the team members. Presumably, you hired or promoted people with the right skills and positive attitudes. Give them credit and respect as they work through the challenges of EDW/BI projects. Don t bog them down with over-regulating their projects. Tangible deliverables over comprehensive documentation. Get physical as quickly as possible. Prototype everything. Documentation is important for communication, but most of it can be generated automatically from development tools. With agile, we produce much of our documentation after our deliverables are built, rather than before. Customer collaboration over contract negotiation. With traditional and spiral methodologies, every development step has to be signed off before the project team can proceed to the next step. That reflects an us versus them mentality. Change that. Put the business representative in the boat with you the boat being the core team. Deliberate and negotiate everything together on the core team. Responding to change over following a plan. Be flexible. Users don t always know what they want until they see it. Team members don t always know how something works until they do it. The only constant on EDW/BI projects is constant change. Allow change, but manage its impact on your releases and on other users. 47

48 Outline Why traditional methodologies do not work for EDW/BI projects Spiral and agile methodologies Agile BI versus agile EDW Agile on the TDWI BI Maturity Model 48 Traditional methodologies and traditional project management practices don t work for enterprise-class EDW/BI projects. There are two major problems with these waterfall methodologies: 1. they do not include extensive data management activities 2. it takes much too long to deliver something tangible to the users Spiral data warehouse methodologies fix problem 1. These methodologies are very popular with enterprise information management professionals working on enterprise-class data integration projects. But the unintended consequence of spiral methodologies is that they add more data management activities to the development process, and now it takes even longer to deliver something to the users. Agile methodologies fix problem 2. These methodologies are very popular with developers working on BI applications. But these methodologies are code-centric, not data-centric. They do not include or even address the extensive data management activities on EDW/BI projects, which can take up as much as 80% of the entire development effort. This seminar will explain the differences between traditional, spiral, and agile methodologies. It will compare four different BI implementation scenarios and their use of agile methods. It will introduce a new data-centric alternative agile approach called Extreme Scoping, which was specifically designed for enterprise-class data integration projects like enterprise data warehousing (EDW) and master data management (MDM). This session will conclude with a mapping of different agile methods on the TDWI BI Maturity Model. 48

49 Scrum XP CODE agile Production Reporting Spreadmarts BI Maturity Model Marts FOCUS on: GULF Coding apps Departmental Warehouses FOCUS on: Enterprise data CHASM standardization and integration Source: TDWI Maturity Model Extreme Scoping DATA agile Enterprise Warehouse Analytic Services 1. Prenatal/Infant 2. Child 3. Teenager 4. Adult 5. Sage Cost BUSINESS VALUE ROI Value 49 Excerpt from the TDWI Business Intelligence Maturity Model : The BI Maturity Model is shaped in a bell curve to indicate that most organizations today have reached Stages 2 and 3. Only a few are still stuck in the first two stages or have advanced to highly mature implementations in Stages 4 and 5. Organizations evolve at different rates through these six stages and may exhibit characteristics of multiple stages at a given time. Although it is possible to skip stages, it is unlikely. Organizations must learn critical lessons at each stage before they can move to the next. STICKING POINTS. Almost every organization gets stuck at two points in the life cycle. These are represented in the model as the Gulf and the Chasm. Most BI initiatives stall here. They have one foot stuck in the previous stage while the other is reaching out to the next, and they are unable to make a clean leap beyond. THE GULF. Executives must recognize that they need more than a production reporting system to make timely, effective decisions and that the dozens of spreadsheets and desktop databases (i.e., spreadmarts) that run the business undermine productivity and effectiveness. THE CHASM. The Chasm is deeper and wider than the Gulf and harder to cross. There are several reasons: Perception, ownership, consolidation, self-service, and mental silos. Code-agile methodologies (like Scrum) can be used for stages 1 and 2 because the focus is on writing software. -agile methodologies (like Extreme Scoping ) must be used in stages 3, 4 and 5 because the focus is on enterprise-wide data standardization and integration. 49

50 Thank You ISBN ISBN ISBN ISBN Larissa T. Moss Method Focus, Inc. ISBN/ ISBN Larissa Moss is founder and president of Method Focus Inc. She has over 30 years of IT experience, with a focus on data warehousing for over 20 years. She frequently speaks at conferences worldwide on the topics of data warehousing, business intelligence, project management, development methodologies, and other data asset management topics, such as enterprise architecture, data integration, and information quality. Her books include Warehouse Project Management, Impossible Warehouse Situations, Business Intelligence Roadmap, Strategy, and Extreme Scoping : An Agile Approach to Enterprise Warehousing and Business Intelligence. Her articles are frequently published interadata Magazine, TDWI Journal of Warehousing, TDWI Flash Point, Cutter IT Journal, and EIMInsight Magazine at Her white papers include Organizational and Cultural Barriers to Business Intelligence, Developing BI Decision-Support Applications: Not Business As Usual, Quality is Not Optional, The Importance of Modeling as a Foundation for Business Insight, Extreme Scoping: An Agile Approach to Enterprise Warehousing and BI, and The Role of Chief Officer in the 21 st Century. Her present and past associations include Third Party Influencers of Teradata, the IBM Gold Group, the Cutter Consortium, DAMA Los Angeles Chapter, the Relational Institute, and Codd & Date Consulting Group. She was a part-time faculty member at the Extended University of California Polytechnic University Pomona, and has been lecturing for TDWI, PESG, MIS Training Institute, the Cutter Consortium, IRM UK and Technology Transfer Italy. She can be reached at [email protected]. 50