The Necessary Skills for Advanced Analytics 4 Hugh J. Watson. BI Dashboards the Agile Way 8

Transcription

1 EXCLUSIVELY FOR TDWI PREMIUM MEMBERS Volume 17 Number 4 4th Quarter 2012 The leading publication for business intelligence and data warehousing professionals The Necessary Skills for Advanced Analytics 4 Hugh J. Watson BI Dashboards the Agile Way 8 Paul DeSarra Best Practices for Turning Big Data into 17 Big Insights Jorge A. Lopez Implementing Dashboards for a Large 22 Business Community Doug Calhoun and Ramesh Srinivasan Data Government Models for Healthcare 34 Jason Oliveira BI Q&A: Gaming Companies on the 40 Bleeding Edge of Analytics Linda L. Briggs Offloading Analytics: Creating a 43 Performance-Based Data Solution John Santaferraro BI Experts Perspective: Mobile Apps 49 Timothy Leonard, William McKnight, John O Brien, and Lyndsay Wise

2 BI Training Solutions: As Close as Your Conference Room We know you can t always send people to training, especially in today s economy. So TDWI Onsite Education brings the training to you. The same great instructors, the same great BI/DW education as a TDWI event brought to your own conference room at an affordable rate. It s just that easy. Your location, our instructors, your team. Contact Yvonne Baho at or ybaho@tdwi.org for more information. tdwi.org/onsite

3 volume 17 number 4 3 From the Editor 4 The Necessary Skills for Advanced Analytics Hugh J. Watson 8 BI Dashboards the Agile Way Paul DeSarra 17 Best Practices for Turning Big Data into Big Insights Jorge A. Lopez 22 Implementing Dashboards for a Large Business Community Doug Calhoun and Ramesh Srinivasan 34 Data Government Models for Healthcare Jason Oliveira 40 BI Q&A: Gaming Companies on the Bleeding Edge of Analytics Linda L. Briggs 43 Offloading Analytics: Creating a Performance-Based Data Solution John Santaferraro 49 BI Experts Perspective: Mobile Apps Timothy Leonard, William McKnight, John O Brien, and Lyndsay Wise 55 Intructions for Authors 56 BI StatShots BUSINESS INTELLIGENCE Journal vol. 17, No. 4 1

4 volume 17 number 4 tdwi.org EDITORIAL BOARD Editorial Director James E. Powell, TDWI Managing Editor Jennifer Agee, TDWI President Director, Online Products & Marketing Senior Graphic Designer rich Zbylut Melissa Parrish Bill Grimmer Senior Editor Hugh J. Watson, TDWI Fellow, University of Georgia Director, TDWI Research Philip Russom, TDWI President & Chief Executive Officer Senior Vice President & Chief Financial Officer Executive Vice President neal Vitale Richard Vitale Michael J. Valenti Director, TDWI Research David Stodder, TDWI Associate Editors Barry Devlin, 9sight Consulting Mark Frolick, Xavier University Troy Hiltbrand, Idaho National Laboratory Claudia Imhoff, TDWI Fellow, Intelligent Solutions, Inc. Barbara Haley Wixom, TDWI Fellow, University of Virginia Advertising Sales: Scott Geissler, sgeissler@tdwi.org, List Rentals: 1105 Media, Inc., offers numerous , postal, and telemarketing lists targeting business intelligence and data warehousing professionals, as well as other high-tech markets. For more information, please contact our list manager, Merit Direct, at or Reprints: For single article reprints (in minimum quantities of ), e-prints, plaques, and posters contact: PARS International, phone: , reprints@parsintl.com, Copyright 2012 by 1105 Media, Inc. All rights reserved. Reproductions in whole or in part are prohibited except by written permission. Mail requests to Permissions Editor, c/o Business Intelligence Journal, 1201 Monster Road SW, Suite 250, Renton, WA The information in this journal has not undergone any formal testing by 1105 Media, Inc., and is distributed without any warranty expressed or implied. Implementation or use of any information contained herein is the reader s sole responsibility. While the information has been reviewed for accuracy, there is no guarantee that the same or similar results may be achieved in all environments. Technical inaccuracies may result from printing errors, new developments in the industry, and/or changes or enhancements to either hardware or software components. Printed in the USA. [ISSN ] Product and company names mentioned herein may be trademarks and/or registered trademarks of their respective companies. Vice President, Finance & Administration Vice President, Information Technology & Application Development Vice President, Event Operations Chairman of the Board Christopher M. Coates Erik A. Lindgren David F. Myers Jeffrey S. Klein Reaching the Staff Staff may be reached via , telephone, fax, or mail. To any member of the staff, please use the following form: FirstinitialLastname@1105media.com Renton office (weekdays, 8:30 a.m. 5:00 p.m. PT) Telephone ; Fax Monster Road SW, Suite 250, Renton, WA Corporate office (weekdays, 8:30 a.m. 5:30 p.m. PT) Telephone ; Fax Oakdale Avenue, Suite 101, Chatsworth, CA Business Intelligence Journal (article submission inquiries) Jennifer Agee journal@tdwi.org tdwi.org/journalsubmissions TDWI Premium Membership (inquiries & changes of address) membership@tdwi.org tdwi.org/premiummembership Fax: BUSINESS INTELLIGENCE Journal vol. 17, No. 4

5 From the Editor Speed is on everyone s mind these days. From real-time data to on-demand reporting, BI professionals want up-to-the-minute information and they want it now. The authors in this issue of the Business Intelligence Journal understand. Agile development methodologies have long promised speedier delivery of new applications or features thanks to shorter development cycles and increased user collaboration. Paul DeSarra explains how an agile approach can be leveraged to meet the highly dynamic needs of business; he uses an agile dashboard project to illustrate his ideas. Dashboards are a quick and easy way to communicate key performance indicators, and Doug Calhoun and Ramesh Srinivasan provide tips and best practices for creating a successful dashboard design. An agile approach may also be what s needed for mobile development at a maternity clothes maker, the subject of our Experts Perspective. Timothy Leonard, William McKnight, John O Brien, and Lyndsay Wise offer their advice for getting mobile BI up and running quickly. Of the three leading characteristics of big data (the so-called 3 Vs: volume, variety, and velocity), it s the speed component that is often cited as its downfall. How can you process so much data without becoming bogged down? Jorge A. Lopez describes one approach. John Santaferraro discusses how analytics must be offloaded to separate analytics databases if big data is to provide accelerated queries, faster batch processing, and immediate access to a robust analytics environment. Senior editor Hugh J. Watson notes that studies suggest enterprises will soon face a shortage of data scientists. He explains that we will have to give business analysts and data scientists wider and more in-depth permissions and provide more training for existing staff if we re to keep up with current trends. Healthcare organizations face a variety of tough governance challenges. Jason Oliveira explores what can be learned from other governance and services organizations that have adopted business intelligence competency centers (BICCs) and how to apply that knowledge to help improve healthcare s BI disciplines. Speed can present challenges, which is why our Q&A explores how gaming companies are on the bleeding edge of analytics, using real-time information to improve gameplay (as well as up-sell or cross-sell products or services to players). How are you keeping up? We welcome your feedback and comments; please send them to jpowell@tdwi.org. BUSINESS INTELLIGENCE Journal vol. 17, No. 4 3

6 Advanced Analytics The Necessary Skills for Advanced Analytics Hugh J. Watson Hugh J. Watson is a C. Herman and Mary Virginia Terry Chair of Business Administration in the Terry College of Business at the University of Georgia. He is a Fellow of TDWI and senior editor of the Business Intelligence Journal. hwatson@uga.edu Analytics work requires business domain knowledge, the ability to work with data, and modeling skills. Figure 1 identifies some of the skills in each area. The importance of particular skills and the exact forms they take depend on the user and the kind of analytics involved. Let s take a closer look. It is useful to distinguish among business users, business analysts, and data scientists. Business users access analytics-related information and use descriptive analytics tools and applications in their work reports, OLAP, dashboards/scorecards, and data visualization. They have extensive business domain knowledge and are probably familiar with the data they are accessing and using but have a limited need for and understanding of modeling. BUSINESS DOMAIN Goals Strategies Processes Decisions Communication of results DATA Access Integration Transformation Preparation MODELING Methods, techniques, and algorithms Tools and products Methodologies Figure 1. Skills needed for analytics. 4 BUSINESS INTELLIGENCE Journal vol. 17, No. 4

7 Advanced Analytics Business analysts use analytical tools and applications to understand business conditions and drive business processes. Their job is to access and analyze data and to provide information to others in the organization. Most business analysts are located in the functional areas of a business (such as marketing) and perform analytical work (such as designing marketing campaigns), or they may work in a centralized analytics team that provides analytics organizationwide. Depending on their positions, business analysts work with some combination of descriptive, predictive, and prescriptive analytics. They tend to have a good balance of business domain knowledge as well as data and modeling skills. The data scientist title is taking hold even though it sounds elitist (I ve also heard the term data ninja). Data scientists typically have advanced training in multivariate statistics, artificial intelligence, machine learning, mathematical programming, and simulation. They perform predictive and prescriptive analytics and often hold advanced degrees, including Ph.D.s, in fields such as econometrics, statistics, mathematics, and management science. Companies don t need many data scientists, but they come in handy for some advanced work. Data scientists often have limited business domain knowledge, the ability to handle data related to performing analytics (e.g., data transformations), and strong modeling skills. They often move from project to project and are paired with business users and business analysts so that necessary domain knowledge is included on the team. Most companies have moved along the BI/analytics maturity curve and now have business users who understand and can employ descriptive analytics and business analysts who can deliver descriptive and some predictive analytics. Interest is now focusing on the organizational capability to perform predictive and prescriptive (that is, advanced) analytics to answer why things happen and propose changes that will optimize performance. This explains why enterprises are employing more data scientists. Successful advanced analytics requires a high level of business domain, data, and modeling skills, and a team of people is often required to ensure that all of the skills and perspectives are in place. As an example, consider the following experience. Southwire: Bringing the Skills Together Several years ago, I received a call from a manager at Southwire, a leading producer of building, utility, industrial power, and telecommunications cable products and copper and aluminum rods. He wanted help solving an impending problem associated with the production of copper, a key component of many of his company s products. My experience on that project (in particular, how the problem was approached and solved) provides a good example of the skills required to be successful with advanced analytics. I learned that the there is no set formula for manufacturing copper. A variety of ores and other ingredients are used depending on what is available. The current approach involved an expert who would look at what materials were on hand and decide what and how much of each ingredient should be used. It was critical that the ingredients produced copper and that the copper would be viscous enough to flow out of the smelter and refining furnace. The problem was that the expert was retiring soon and his expertise was going to be lost. A new solution approach was needed. Southwire assembled a team of business people, chemical engineers, IT, and me. We had individuals with business knowledge, subject area experts, people who were familiar with available data and systems, and members with modeling expertise. The team contained all the skills needed for advanced analytics. My role was to provide the modeling (data scientist) skills. I saw two possible modeling approaches. The first option was to create an expert/rules-based system based on the knowledge of the retiring expert. We would capture in an application the heuristics that the expert used in deciding what to put into the smelter each day. The model would be descriptive in that it would describe the expert s thinking. BUSINESS INTELLIGENCE Journal vol. 17, No. 4 5

8 Advanced Analytics The other alternative, and the one chosen, was to use linear programming. If you are familiar with linear programming, Southwire s problem was the classic production blending application. You create an objective function (that is, an equation) that you want to minimize with the sum of the cost of the various ingredients multiplied by the quantity of each ingredient. You also write constraint equations that reflect the conditions that the solution must satisfy. The output of the analysis is the quantity of each ingredient that will minimize costs while satisfying all of the constraints. A possible solution to the potential conflict over control versus flexibility is an analytical sandbox, whether it is internal to the warehouse or hosted on a separate platform. The writing of the constraint equations was fascinating to me. Remember that the solution had to produce copper and it had to be sufficiently viscous. These requirements were handled through the constraint equations and reflected what ingredients were available and the chemical reactions involved. The chemical engineers input was critical for developing these equations. Remember when you took chemistry in high school or college and studied valences (the number of bonds a given atom has formed, or can form, with one or more other atoms)? This and other factors (such as what ingredients were available each day) were incorporated into the constraint equations. Data scientists are not a one-trick pony when it comes to modeling. They are familiar with multiple modeling approaches and algorithms. They are able to identify and experiment with different models until they find the one that seems most appropriate. At Southwire, a linear programming modeling approach was selected over an expert/rules-based system. Once the objective function and constraint equations were developed, it was necessary for IT and me to select an appropriate linear programming package, enter the objective function and constraint equations, test the solution, develop a user interface that operational people could easily use for entering data (such as ingredients) and interpreting the output, implement the system, and train people to use it. Assembling the Skills Enterprises have the business domain knowledge for advanced analytics. However, as illustrated at Southwire, a key to success is to make sure that people with business domain knowledge are on the analytics team. Enterprises also have the required data skills, but a few changes may be necessary to accommodate their need for advanced analytics. Data scientists (and some business analysts) may need to have fewer restrictions on the data they can access and what they can do with it. They may need access to underlying data structures as well as the ability to join, transform, and aggregate data in ways necessary for their work. They may also need the ability to enter new data into the warehouse, such as from third-party demographic data sources. A possible solution to the potential conflict over control versus flexibility is an analytical sandbox, whether it is internal to the warehouse or hosted on a separate platform. Finding the required modeling skills is a trickier proposition. You can hire consultants, as Southwire did, or use a third-party analytics provider, but these options can become costly over time if your plans include extensive advanced analytics. You can probably coach some of your current business analysts. There are many conferences (such as those offered by TDWI), short courses, and university offerings that teach advanced analytics. As advanced analytics becomes better integrated into application software (for example, campaign management) and easier to use, it is likely that trained business analysts can take on tasks that have skill requirements typically associated with data scientists. 6 BUSINESS INTELLIGENCE Journal vol. 17, No. 4

9 Advanced Analytics You can also hire data scientists. This isn t a new approach; many companies have already done so and have data scientists scattered throughout their business units or in specialized groups such as analytics competency centers. Studies suggest that companies are planning to hire more data scientists and will face a shortage of such resources. For example, the McKinsey Global Institute predicts a shortfall of between 140,000 and 190,000 data scientists by 2018 (Manyika, et al, 2011). Many universities are gearing up to meet the need through degree programs, concentrations, and certificates. These offerings are usually in business, engineering, or statistics and the instructional delivery varies from on campus to online. skills, you will probably need to provide training for existing staff and bring in new people with specialized analytical skills. Reference Manyika, James, Michael Chui, Brad Brown, Jacques Bughin, Richard Dobbs, Charles Roxburgh, and Angela Hung Byers [2011]. Big Data: The Next Frontier for Innovation, Competition, and Productivity, McKinsey Global Institute, May. technology_and_innovation/big_data_the_next_ frontier_for_innovation One of the first and best-known programs is the Master of Science in Analytics at North Carolina State University. SAS has been an important contributor to the program, which is offered through the Institute for Advanced Analytics and has its own facility on campus. Deloitte Consulting has partnered with the Kelly School of Business at Indiana University to offer a certificate in business analytics for Deloitte s professionals. Just this year, Northwestern University rolled out an online Master of Science in Predictive Analytics offered through its School of Continuing Studies. Will students take advantage of these programs in large enough numbers? Advanced analytics is a tough study, and many students may not have the necessary aptitude, inclination, and drive to complete the programs, even though the career opportunities are great. Summary You have already been performing analytics under the BI umbrella. BI includes descriptive analytics, and you have probably also been performing predictive analytics. For more advanced analytics, however, you will need to ramp up your game a little. You have the business domain knowledge covered. For the data component, you will need to grant business analysts and data scientists wider or more in-depth permissions and you will likely need to extend and enhance your analytical platforms (such as appliances and sandboxes). For the modeling BUSINESS INTELLIGENCE Journal vol. 17, No. 4 7

10 Agile Dashboard Development BI Dashboards the Agile Way Paul DeSarra Paul DeSarra is Inergex practice director for business intelligence and data warehousing. He has 15 years of BI strategy, development, and management experience working with enterprises. Abstract Although the concept of agile software development has been around for more than 10 years, organizations only recently began to think about how this methodology can be applied to business intelligence (BI) and analytics. BI teams are continually evolving their rapid delivery of additional value through reporting, analytics, and dashboard solutions. These teams must also discover what types of BI solutions can reinvigorate a BI deployment and produce meaningful results. One way to reinvigorate BI deployments is to take the concept of agile software development and apply it to BI initiatives such as BI dashboard solutions, which can both re-engage the business and drive actionable intelligence and confident decision making. Agile BI replaces traditional BI project methods (heavy in documentation and process) with a user-driven approach. This article discusses an approach to building BI solutions and dashboards using an agile software development methodology. Introduction Although the concept of agile software development has been around for more than a decade, it s only been in the last few years that organizations have started to examine how this methodology can be applied to business intelligence and analytics. The constantly changing, highly dynamic needs of business today have increased the demands on BI environments and teams. The pressure to be more organized, turn projects around faster, and ensure user adoption at all levels is increasing. Teams need to be able to react to demands from the business and proactively develop ideas and solutions that give the business more creative ways to think about how to use data. Leveraging an agile software methodology as it applies to business intelligence is a great way to meet these constantly changing business needs. 8 BUSINESS INTELLIGENCE Journal vol. 17, No. 4

11 Agile Dashboard Development In a nutshell, using an agile software development methodology ( agile ) instead of a traditional development methodology allows end users to experience a version of the software product sooner. Instead of adhering to a strict and intensive requirements and design phase before development begins, agile employs a series of shorter development cycles to increase user collaboration. The agile approach welcomes changes during the development process to provide a better product that delivers measurable value quickly and efficiently. There are four guiding principles for agile software development (according to the Manifesto for Agile Software Development, These can also be applied to business intelligence development efforts. Principle #1: Value individuals and interactions over processes and tools Traditional BI development focuses on strong processes and tools to solve development challenges. As a result, many organizations end up creating silos among the business and IT teams. Each team silo focuses on individual responsibilities and objectives and, in effect, each team loses sight of the overall project goal of providing cohesive and comprehensive data-driven solutions that improve performance levels. When using an agile BI approach, all those involved in the BI initiative work together as one team with one goal and set of objectives. To accomplish this, many organizations create hybrid teams and a business intelligence competency center (BICC) composed of individuals with the necessary skills to define, architect, and deliver analytic solutions. In some cases, many of these teams are organized under business units outside of IT and the program management office. Principle #2: Value working software over comprehensive documentation Traditional BI development in a big-bang approach focuses on developing detailed documentation about common metrics, terminologies, processes, governance, support, business cases, and data warehouse architectures. Organizations may create a standardized enterprise data warehouse and then fail because they were focused on the documentation and lost touch with the business and the problems they were trying to solve. This doesn t mean we should stop creating detailed documentation. BI teams can and should continue to focus on creating documentation that emphasizes the vision and scope as well as the architecture for future support. With agile BI, the focus is not on solving every BI problem at once but rather on delivering pieces of BI functionality in manageable chunks via shorter development cycles and documenting each cycle as it happens. Agile employs a series of shorter development cycles to increase user collaboration. It welcomes changes during development to deliver measurable value quickly and efficiently. Principle #3: Value customer collaboration over contract negotiation Using an agile BI approach does not mean giving users an unlimited budget or tolerance for changes. Instead, users can review changes discussed in the last development cycle to ensure expectations and objectives are being met throughout the project. Traditional BI development teams use functional documentation to discuss what the solution will look like and how it will operate. Such an imagine this method often leaves users to try and visualize what they believe the solution will become. The resulting subjective expectations can quickly derail a BI project. In contrast, an agile methodology reviews progress during each development cycle using prototypes so that stakeholders and business users can see how the BI solution is expected to look and BUSINESS INTELLIGENCE Journal vol. 17, No. 4 9

12 Agile Dashboard Development RELEASE 1... N BUILD SCOPE PROTOTYPE The vice president of sales of a large manufacturing organization asked us to help his company gain better insight into its orders, shipments, and pipeline in order to hold the sales teams more accountable. Specifically, he wanted a dashboard that he and his executive team could use to meet accountability objectives. His vision for the dashboard was solid, and our role was to take that vision and boil it down to key metrics that would drive actions. STAKEHOLDER REVIEWS Figure 1. The agile process. function. Prototypes put a visual face to the project by showing what data is available, how it will be used, and how it will be delivered. Principle #4: Value responding to change over following a plan With an agile methodology, traditional BI projects that focus on huge project and resource plans are replaced by shorter development cycles designed to better incorporate changes and keep the project team focused and informed. For BI projects, changes should be expected and welcomed. When users see prototypes and gain a better understanding of what analytic capabilities and information are available, they are better able to communicate how they could use that information to make improved business decisions. Such revelations and ideas only strengthen the final product. An agile BI project still uses a plan, but its plan is short, manageable, and coupled with a prototype users can see and experience. Changes are jointly reviewed with business sponsors, users, and IT professionals at every project stage. Example: An Agile Dashboard To better understand how this methodology can be used, let s look at a real-world example of incorporating agile BI into a BI dashboard project for an executive sales team. After a few meetings with the vice president of sales and the IT sponsor (in this case, the IT director), we concluded that an agile BI dashboard project was the best approach. We ensured we had the needed sponsorship from both the business and IT teams. In addition, we confirmed the organization was using a BI tool that was capable of delivering the desired solution. We advanced the project using a hybrid approach to agile development, breaking the project into three phases to quickly and efficiently develop the scope, build prototypes, conduct reviews, develop the solution, and implement it quickly. An agile BI project still uses a plan, but its plan is short, manageable, and coupled with a prototype users can see and experience. Phase 1 This was the foundational phase for our project and focused on the third agile principle ( customer collaboration over contract negotiation ). Phase 1 should last no more than one week and involves identifying, at a high level, the scope of the BI dashboard to ensure that the executive sponsors are engaged and the internal teams are assembled. Phase 1 is essential because it is used to narrow the scope and prioritize what can be delivered in the set time frame. 10 BUSINESS INTELLIGENCE Journal vol. 17, No. 4

13 Agile Dashboard Development In the first week, we worked with IT and the vice president of sales to ensure that the team had the right people with the right skills who understood the project goals. We outlined roles and responsibilities, opportunity and vision, and the high-level scope all standard practices for an agile BI project. We worked with the vice president of sales along with several key business users to identify the metrics of greatest value. We worked diligently to understand what metrics were needed and how they influenced business decisions. A dashboard metric isn t enough; we strived to enable users to respond to each metric to achieve the best business results. For example, we examined what happened after the dashboard highlighted a large gap between what the customer relationship management (CRM) application identified as a sales opportunity and the revenue actually generated. We asked questions about the process of capturing these opportunities in the CRM to better understand leading factors that could influence revenue. Delving into these questions ensured that we understood the full sales engagement process. We didn t stop there. We identified about 10 metrics for invoicing, orders, shipments, and budgets across four different dimensions business area, Figure 2. Dashboard prototype examples. BUSINESS INTELLIGENCE Journal vol. 17, No. 4 11

14 Agile Dashboard Development product, customer, and date attributes. In our vision, the dashboard would allow the sales teams to focus more effectively on specific sales opportunities, better track budgets, and confidently predict and forecast sales throughout the year (and know where and how to make necessary adjustments). We held two meetings with the IT team to better understand the ability of the source systems to provide the data elements required. The prototyping tool may be separate from your BI tool, but it must be able to demonstrate visual elements as well as drill-up, drilldown, and interactivity. The Phase 1 deliverables included a high-level vision and scope document that clearly set the stage for the rest of the project. By quickly defining the vision and scope as well as establishing a short time frame, we removed one barrier (long contract negotiations and timelines) so we could focus on having the right people involved and the right team defined. Phase 1 was completed in one week. Phase 2 Phase 2 is where collaboration, rapid prototyping, whiteboard sessions, and interactive brainstorming take place. This phase applies three of the agile principles ( individuals and interactions, customer collaboration, and responding to change ). Phase 2 focuses on using prototyping methods in brainstorming sessions to quickly build and show business users how their ideas and needs are reflected in the proposed solution sometimes iteratively and on the fly. The prototyping tool may be separate from your BI tool, but it must be able to demonstrate visual elements as well as drill-up, drilldown, and interactivity. This phase requires collaboration between the sponsors, key business users, and IT. A key benefit of this phase is that users see the data in action and will know whether the data is being presented in a way that effectively delivers the information they need. In fact, the process often gives users new ideas for using the information to make business decisions (see Figure 2). In Phase 1 we created our vision and scope, outlined the business problem, and understood the set of metrics and dimensions necessary to reach the desired outcome. We approached Phase 2 with two goals in mind: Collaborate with the vice president of sales and the sales teams to define the look of the BI dashboard and the data interactions required to populate it. Work with the IT team to determine the data components and further understand what could be accomplished and delivered by the project deadline. (The overall project length was seven weeks, so we had only six weeks left.) The collaboration sessions were held with the vice president of sales, several key business users, and individuals from the IT team. The meetings started as whiteboarding sessions. Once we completed the initial design, we built a prototype with phony (but businesssensible) data and set up daily meetings to review and refine our development cycles. In each session, we identified how and why metrics were to be used and outlined the decisions that would be made using the data. We evaluated different ways to display information so it would be most useful to users. We also mocked up the drill-through detail analysis and reporting that would be available via the easy-to-understand dashboard and made sure only a single path led to the detail at each level. The resulting dashboard prototype 12 BUSINESS INTELLIGENCE Journal vol. 17, No. 4

15 Agile Dashboard Development had four quadrants, each of which was meant to answer a specific question: How are we performing today? Are we on plan and what is our updated forecast? Where are we winning and losing? There was a need to tie in a certain product category captured in a separate data source outside the ERP. The product category was required to ensure we were capturing the full picture. This product was set to be coded in the new ERP. We decided to pull in and map this information from the separate data source and also put in place a process to map it into the new ERP when the time was right. Who and what is not profitable? The mockup took the form of charts, regional maps, and dynamic and color-coded lists. It also included detailed drill-through paths and report examples to help guide users in making decisions. For example, a user could click on a troubled region on the map that identified a large revenue gap based on forecasting and get details on current activity within that region as well as open opportunities and win/loss details. All in all, we held about 10 different business sessions and kept coming back the next day with a refined prototype to generate ideas. As a result, throughout the entire process, users were engaged, excited, and willing to participate in the sessions. They also felt confident that their needs were being addressed and their ideas and feedback were incorporated. Users felt confident that their needs were being addressed and their ideas and feedback were incorporated. We uncovered more than 15 potential roadblocks to the initiative, and we worked through them all with the team. We kept everyone informed and made joint IT/business decisions to move forward accomplishing this with daily status meetings with IT and the business subject matter experts to address issues quickly and outline resolutions. Sponsors and stakeholders were also part of weekly checkpoint meetings. We simultaneously worked on the data components to map the vision to the actual data sources. To do so, we had to remove several roadblocks and make some tough decisions as a team (IT and business) in order to meet our deadline. As the team forged ahead, we uncovered several items that needed to be worked through as quickly as possible. A few financial metrics were not in the current ERP but would be implemented in an upgraded version, which was set to go live the following year. We worked with the business to outline the metrics and ultimately decided to put them on hold so that we could continue building the rest of the dashboard. After we removed all our technical and business roadblocks, we completed Phase 2 and delivered the prototype dashboard, drill-through mockups, and a Lean Requirements document that captured the requirements and outlined the assumptions and decisions we had made. We also built a Lean Design document that described the database design, data mapping, reporting designs, and ETL construct. Phase 2 was completed in four weeks. Phase 3 Phase 3 is the build phase and applies the second agile principle ( working software ). The foundation has been BUSINESS INTELLIGENCE Journal vol. 17, No. 4 13

16 Agile Dashboard Development set, the scope has been refined to ensure rapid delivery, IT and business are fully engaged, and now the time has come to take the prototype and construct it within the BI environment. Phase 3 should take no more than a few weeks and involves building integration and ETL procedures, security, and the BI solution itself. The project was a success because of the agile BI processes applied to every aspect of the project. One of the core success factors was the use of prototypes and interactive sessions. At this point, with two weeks left, we began to build the required dashboard, drill-through reports, and supporting data layers. Building everything in dynamic prototypes made it much easier to ensure expectations were in line as development progressed. During this phase, we continued to show the results of actual development of the dashboard every two days to the business sponsor and key users. Throughout this process, changes were still submitted. We reviewed all changes and put them into one of two buckets implement or put on hold and made notes in our change control log. Some of the change requests that flowed through in this phase revolved around adding different relative time-period buckets for revenue and margin analysis, some minor layout changes, and three changes that were put on hold for future phases around customizing various alternate drill paths from the dashboard based on a user s business unit and region. Phase 3 was completed in two weeks. Tips for Agile BI Success In the end, the initial phase of the dashboard was released in seven weeks. The project was a success because of the agile BI processes applied to every aspect of the project. One of the core success factors was the use of prototypes and interactive sessions. Using prototypes enabled us to keep all players involved from the beginning and provided a forum to exchange ideas, discuss issues, and actually see the solution as its development progressed. After reading the case study, perhaps you are now thinking, Can organizations really implement these types of BI solutions in seven weeks? You may be asking, What about data governance, load procedures, ETL, business rules, capacity planning, and security maintenance? The reality is that you must strike a balance when using the agile software development methodology for your BI initiatives. The process walks the line of ensuring that you are building a solid foundation that has longevity, speed, and strength to weather the dynamic and demanding needs of the business. The following ideas and concepts can help you implement an agile BI process. Tip #1: Start small, think big When you begin to build an agile BI solution, it doesn t matter if you have an enterprise data warehouse coupled with a large-scale, mega-vendor BI software stack or a small data mart managed with a niche tool. The key is to focus on the immediate business need and pain, then map that to the ultimate vision. Get the stakeholders to define and work with you to build out what it will look like. Once you have the vision, determine the best approach that completes the work quickly and keeps the long-term picture in mind. If you need to take shortcuts to get the work done, that s fine, as long as everyone approves the shortcuts and you have a process in place to close the loop at a future point. For example, if you have an ERP application and you have to group some of your sales data into a customized dimension (instead of modifying the ERP source of records) in order to deliver the BI solution, then do so, but ensure that you get approval and that everyone understands the costs and benefits. 14 BUSINESS INTELLIGENCE Journal vol. 17, No. 4

17 Agile Dashboard Development Although you are building a specific solution, you can still take steps to ensure it is repeatable, scalable, and fits into your overall data architecture. For example, in our case study, we were building a specific BI dashboard solution that was focused on shipments, orders, and pipeline processes specific to the sales functional area. However, in creating the solution, we built a data design that could scale outside of sales by building conformed dimensions and process-driven fact tables. If, for performance reasons, we had to create summary or aggregated tables to support specific business areas, we made sure these mapped back to lower-grain fact tables for data consistency and detail analysis. Tip #2: Remove the roadblocks Whether you face IT challenges or other obstacles, work systematically to overcome them. Typical roadblocks you may encounter in an agile BI project include: A narrowed scope. In some cases, it can be challenging to narrow the scope of a BI project so that a portion of the solution can be delivered in a shorter time frame. This is a slippery slope and requires the ability to prioritize and find common ground with business users and/or sponsors. If you can get the business sponsor to commit to a shorter time frame up front, it will be easier to narrow the scope. In addition, separate out the must-haves and the would-like-to-haves right away. Data gaps. In any BI project, data gaps are typical as users may not fully understand how information is collected or data anomalies are discovered. Agile BI is no different, and data profiling is a necessary step. In our case study example, we encountered data gaps that we had to eliminate or overcome by accepting risk, leaving out components, or implementing a temporary fix. Business commitment and time. Agile BI requires interaction with business stakeholders, sponsors, and users throughout the project s life. Secure commitment up front with everyone and clearly outline the project s benefits in terms of effective decision making. Managing expectations. There is often a gap in the expectations about what it takes to deliver a BI solution and the time it actually takes. Users may believe that much more can be done in a short amount of time, which can cause extreme tension between IT and the business. Managing these expectations requires strong communication skills and an individual on the team who can effectively bridge IT and business users. This individual should understand data modeling and architecture, reporting and analytics, and dimensional concepts and be able to articulate the challenges to business managers and sponsors in a language they understand. If you need to take shortcuts to get the work done, that s fine, as long as everyone approves the shortcuts and you have a process in place to close the loop at a future point. Rogue development. Agile BI still follows a process and a method. There is still documentation and a plan; success metrics are still defined at the beginning of the project. Project management is still a core component in this process. We recommend that you still use the following tools, documentation, and processes to help guide the project: A vision and scope document is used to define initial, critical success factors and get project approval. A requirements document outlines the core business problems and key data elements, metrics, and dimensions that are needed for the BI solution. The difference from traditional BI development is that this document focuses on the smaller and shorter deliverables and keeps it lean. BUSINESS INTELLIGENCE Journal vol. 17, No. 4 15

18 Agile Dashboard Development A design document describes the database design, data mapping, reporting designs, and ETL constructs. Again, different from a traditional BI project, this design should focus on bringing the technical team together on the architecture and for future support without getting lost in too many details. A project baseline plan for delivering a piece of functionality quickly, with the longer-term plan represented at a higher level. A change control log to track which changes are implemented and which are put on hold. An enhancement log to track enhancements that the team is unable to fit into the first release. and moved to the full development stage, keeping the key users involved continued to be highly important. During development, we still met at least twice a week to review progress and update our change control logs as we showed progress on the BI dashboard solution. The prototype became our guide to ensuring the development was on course and meeting all expectations. Summary As business becomes more dynamic and social in nature, BI environments need to be prepared to move fast and deliver value in creative ways. Intertwining BI best practices with the agile software methodology is one way to infuse speed, creativity, commitment, and value into any BI project. If you have obtained the right sponsorship at the start and ensured everyone has the same vision and understands the project, your ability to remove roadblocks will be improved. Inevitably, however, challenges will arise, so always keep one eye on the vision and one on the scope. Tip #3: Engage the business BI professionals sometimes get so focused on the technology that even after the initial meeting with business users they may flip back to thinking mostly about the tools and technology rather than the business s pains, needs, and objectives. In our case study, we used rapid prototyping and whiteboarding sessions to gather requirements and keep the right people involved and working in unison. We had daily brainstorming sessions to promote collaboration on the design, discuss the metrics and information needed to make business decisions, and show the BI dashboard prototype progress. From this, we built a requirements document that was focused on the key metrics and data elements, and we incorporated visuals from our prototype to ensure we had everything captured. Once we completed this phase 16 BUSINESS INTELLIGENCE Journal vol. 17, No. 4

19 big data insights Best Practices for Turning Big Data into Big Insights Jorge A. Lopez Jorge A. Lopez is senior manager, data integration, for Syncsort Incorporated. Abstract Big data is surfacing from a variety of sources from transaction growth and increases in machine-generated information to social media input and organic business growth. What does an enterprise need to do to maximize the benefits of this big data? In this article, we examine several best practices that can help big data make a difference. We discuss the role that extract, transform, and load (ETL) plays in transforming big data into useful data. We also discuss how it can help address the scalability and ease-of-use challenges of Hadoop environments. Introduction Growing data volumes are not a new problem. In fact, big data has always been an issue. Fifty years ago, big data was someone with a ledger recording inventory; more recently, it was a bank s mainframe processing customer transactional data. Today, new technologies enable the creation of both machine- and user-generated data at unprecedented speeds. With the growing use of smartphones and social networks, among other technologies, IDC estimates that digital data will grow to 35 zettabytes by 2020 (IDC, 2011). These new technologies have turned big data into a mainstream problem. In fact, it s not uncommon to see small and midsize organizations with just a few hundred employees struggling to keep up with growing data volumes and shrinking batch windows, just as large enterprises do. The viability of many businesses will depend on their ability to transform all this data into competitive insights. According to McKinsey (Manyika, et al, 2011), big data presents opportunities to drive innovation, improve productivity, enhance customer satisfaction, and increase profit margins. Although many CIOs and CEOs recognize the value of big data, they have struggled BUSINESS INTELLIGENCE Journal vol. 17, No. 4 17

20 big data insights (through no fault of their own) to handle this new influx of data. The problem isn t information overload; it s the failure to harness, prioritize, and understand the data flowing in. This is why data integration is a critical yet often overlooked step in the big data analytics strategy. Traditional IT approaches will not generate the results businesses expect in this era of big data. Therefore, IT organizations should look at the hype around big data as an opportunity to set a new strategy for harnessing their data to improve business outcomes. As a first step, organizations must examine their existing data strategies and ask: Are these data strategies helping us achieve the objectives of the business? Can our environment economically scale to support the requirements of big data? Can our infrastructure quickly adapt to new demands for information? To fully take advantage of new sources of information, organizations must cut through the buzz that big data creates. There are many definitions of big data, but most experts agree on three fundamental characteristics: volume, velocity, and variety. Another key aspect, often overlooked, is cost. Forrester, for instance, defines big data as techniques and technologies that make handling data at extreme scale affordable (Hopkins and Evelson, 2011). This touches on two critical areas that must be addressed to have a successful data management strategy: scalability and cost effectiveness. To scale data volumes 10, 50, or 100 times requires new architectural approaches to the data integration process. Doing so in a cost-effective way has been the biggest challenge to date for organizations. No matter what kind of IT environment you have or how you label your data (big or small), there are steps you can take to rearchitect and optimize your approach to data management, such as returning your attention to the data integration process in your quest for improved business insights. Not All Big Data Is Important Data Sometimes it s easy to get caught up in the hype about big data. However, trying to process larger data volumes can significantly increase the amount of noise, hindering your ability to uncover valuable insights. It s important to remember that not all data is created equal. Any big data strategy must include ways to efficiently and effectively process the required data while filtering out the noise. Data integration tools play a key role in filtering out the unnecessary data early in the process to make data sets more manageable and, ultimately, load only relevant data into the appropriate environment for analysis (whether that is a data warehouse, Hadoop, or an appliance). To scale data volumes 10, 50, or 100 times requires new architectural approaches to the data integration process. Doing so in a costeffective way has been the biggest challenge to date for organizations. Organizations can take three approaches: 1. Define a clear data strategy that identifies the users data requirements. (Why do I need this data? How will it help me accomplish my business objectives?) 2. Build an efficient data model that is adequate to the demands of the business. 3. Have the right data integration tools to do the job. Ultimately, the data integration tool is the critical component; it can help materialize the strategy and execute on it to build an efficient data model. The tool must have the right capabilities as well as the scalability and performance required to work effectively. A key component is the ease of use that allows developers to focus on business requirements instead of worrying about performance, scalability, and cost. 18 BUSINESS INTELLIGENCE Journal vol. 17, No. 4