Controlling Excel Chaos With Business Intelligence A White Paper by Diane Sklar
Table of Contents 1 Executive Summary 2 2 2 2 The Excel Challenge Spreadsheet Chaos Manual Labor Begets Errors On the Spreadsheet Audit Trail 4 4 4 5 5 6 11 WebFOCUS Excel Sourcebooks Setting up WebFOCUS Excel Sourcebooks There Is a Solution A Spreadsheet in Every Mailbox The Corporate Excel Sourcebook Deliver Advanced Functionality Answering the Ad Hoc Question 13 Conclusion
Executive Summary Business analysts develop Excel spreadsheets to assist with the day-to-day operational decisions their jobs demand. Pleased with the autonomy and sophisticated analysis that Excel supports, they share these innovations with colleagues who modify spreadsheet logic and manually tack on data from their own information silos. Over time, rogue spreadsheets with data from dubious sources is propogated throughout the organization. Executives find themselves making decisions based on untraceable, questionable data. The enterprise is at a loss to audit these numbers for themselves and for regulatory agencies. Meanwhile, the IT division has complete, auditable, and backed-up operational system data that is untapped by the Excel user community. IT executives are frustrated by the rampant spread of unreliable information and their inability to leverage corporate data stores. Information Builders has both push and pull strategies to serve up corporate data for use in Excel spreadsheets. These strategies buttress Excel s weaknesses as an enterprise business analysis tool while allowing users to continue to operate in their preferred analysis environment. 1 Information Builders
The Excel Challenge Spreadsheet Chaos Spreadsheets were the standard model for doing financial analysis long before enterprises automated accounting and finance functions. When Microsoft launched Excel, it mimicked the spreadsheet layout and made the business analysts transition from paper to computer a seamless one. Moreover, the easy-to-use, point-and-click interface running on the Windows platform gave Excel a market-share victory over competitors like Lotus and VisiCalc. Manual Labor Begets Errors Excel was designed for the individual analyst working alone on a computer with a limited amount of data. The original Excel model had a business analyst typing a small subset of corporate data into a spreadsheet, cell by cell, and then manually manipulating that data to solve a business problem. But manual data entry is an expensive, time-consuming misuse of the business analyst s time and error prone as well. Cell-by-cell data entry leads to simple mechanical errors such as mistyping a number or pointing to the wrong cell. Graver errors can also occur when the analyst makes errors in formulas because of bad keystrokes or bad logic. When data is available in an existing spreadsheet, it can be cut and pasted via the clipboard or merged via Excel functionality. This works best when the source and target ranges are formatted identically. If they are not identical, data and formulas can inadvertently be corrupted. Alternatively, ranges of data from an external Excel workbook can be referenced in place. The effectiveness of this technique is dependent on the source file remaining unchanged. Maintenance to spreadsheets chained in this way can be horrendous when there is no centralized change management policy. Ideally, data from corporate stores should be transferable directly to Excel spreadsheets, thereby insuring that all business analysts start their spreadsheet ventures with the one true version of enterprise data. But Microsoft s vision of interoperability has remained focused on Excel and other Microsoft desktop products. Outside connections must be established through ODBC or DDE connections. Beyond these options, one must transform data to the text-based CSV format and stage it on the Windows platform. Corporate Data Report Output Manual Data/ Formula Entry Styled Aggregated Spreadsheet Figure 1: Manual spreadsheet generation is a breeding ground for errors. 2 Controlling Excel Chaos With Business Intelligence
On the Spreadsheet Audit Trail For many companies Excel has grown beyond a small-scale analysis tool. It is instead the main vehicle for financial analysis in mission-critical departments, pushing Excel functionality beyond its original intent. Excel has no built-in mechanism for separating the presentation, calculation, analysis, and storage layers of an application. The onus of solid spreadsheet design is still left to the spreadsheet developer. Of course, skill levels in Excel and in programming methodology vary widely among analysts. In non-it departments untrained spreadsheet programmers can apply resourceintensive calculations at the wrong level, creating a performance bottleneck. Rigorous testing with sufficiently varied data samples is rarely done at the departmental level. These are just some examples of possible fallout from not applying IT methodology to the spreadsheet community. Accurate and traceable financial information is critical to making sound business decisions that move the enterprise forward. Auditable information is required outside the organization as well. The 2002 Sarbanes-Oxley Act requires disclosure in annual reports of internal controls over financial reporting. Spreadsheet errors can lead to expensive and embarrassing problems. The European Spreadsheet Risk Interest Group analyzes and quantifies the cost of spreadsheet errors worldwide. In the last six months alone, they have reported various situations, including: A public auditor s office in Minneapolis mistakenly reported the percentage change in unreserved fund balances; an analyst set up the formula for the column in the spreadsheet, dividing the difference between 2003 and 2004 balances by the 2004 total, instead of the 2003 figure A Housing and Urban Development (HUD) audit revealed that a local housing authority had to pay over $200,000 to cover expenses incurred when the authority overpaid some landlords due to a data-entry error In Nevada, a 2006 municipal budget was developed from a copy of the city s 2005 budgeting spreadsheet; in late 2005, a problem was revealed causing a $5 million deficit in the water and sewer fund; while fixing the problem with the water and sewer budget, other errors were uncovered and fixed A well-known medical and consumer imaging company had to amend its third-quarter loss by $9 million, announcing that the adjustment was needed because too many zeros were added to an employee s accrued severance on a spreadsheet; the company s CFO characterized the situation as an internal control deficiency True robust error protection and auditing for Excel is generally left to third-party products. The flourishing market of add-on tools bears witness to Excel s weakness in this area. Auditing with add-on tools becomes another training and standardization dilemma. Once again the enterprise is dependent on the skill, prudence, and discipline of the individual analyst to properly deploy audit tools. Despite these problems, usage of Excel is not diminishing. It is still the undisputed giant among business analysis tools. It can be found on every desktop in the enterprise. Its cost per seat is negligible, keeping it the first choice for on-going enterprise application needs. How can the organization protect itself from the rampant spread of uncontrolled and unsanctioned numbers in Excel spreadsheets? 3 Information Builders
WebFOCUS Excel Sourcebooks Setting Up WebFOCUS Excel Sourcebooks Since Excel usage shows no sign of waning, IT s best approach is to eliminate the resourceintensive, error-prone data entry process in traditional Excel deployment. In most enterprises, IT has already invested resources in creating a protected and scrubbed warehouse of summarized financial results. When necessary, IT can also provide links to extract relevant transaction-level data. A comprehensive study of all Excel user requirements can feed the design of a set of master Excel workbooks supplying content to all Excel users. Workbook structure should be flexible, allowing users to exploit any subset of workbooks, worksheets, or cell ranges. Ideally, data should be delivered to these master workbooks already summarized and styled. It should also be formatted for the specific version of Excel that users have on their desktops. Complex formulas should be pre-set to avoid errors. Excel structures that are difficult to develop, such as graphs and pivot tables, should be delivered analysis-ready. Users will have different data latency requirements as well. Some workbooks may need to be refreshed quarterly, other workbooks may require a monthly, weekly, daily or hourly feed. There Is a Solution WebFOCUS, Information Builders business intelligence suite of products, generates Excel spreadsheets across all generations of Excel. This includes XML-based formats with comprehensive Excel functionality as well as binary formats that optimize transfer speed. All WebFOCUS Excel spreadsheets have their data summarized and formatted as a complete spreadsheet on the WebFOCUS Reporting Server. The Reporting Server, typically located on the same platform as the target data, summarizes data as per the request, minimizing the volume of data transmitted and formatted on the desktop. By the time the spreadsheet arrives on the user s desktop it is ready to go, without wait-time for additional formatting. Delivery of corporate spreadsheet data to Excel users can be accomplished by several automated distribution strategies. For each distribution scenario, WebFOCUS ReportCaster schedules jobs and extracts data from known, auditable corporate sources. It makes data available at any interval, from every n years to every n minutes. The resulting extract can be formatted in several ways: As Excel spreadsheets, which are pushed to a distribution list of Excel users via e-mail As a Corporate Excel Sourcebook(s) on a central server users connect to the corporate Excel sourcebook and pull relevant data into their own Excel workbooks by referencing named ranges of cells As database extracts on a central server with a parameter-driven user interface; users supply parameter values that determine all content and stylistic characteristics for a target spreadsheet to answer an ad hoc question 4 Controlling Excel Chaos With Business Intelligence
A Spreadsheet in Every Mailbox When spreadsheet contents and format are uniform across a large population of users, a ReportCaster job can e-mail Excel spreadsheets to a distribution list of user addresses. The job is programmed once and tested rigorously by IT professional staff familiar with the corporate files. The job is run once and output is collected for all interested parties. E-mails with spreadsheet attachments are created for all entries on the distribution list. The distribution list can be developed manually or derived from Microsoft Active Directory or an LDAP directory. Each recipient on the distribution list can receive the entire report. Or, if the report contents are private, the report can be burst so that recipients receive content only for the sort group(s) relevant to them. For example, a company-wide profit and loss statement can be burst by cost center so that managers receive the statements only for their own cost centers. Corporate Directory (LDAP, Active Directory) France Spain Spreadsheet Distribution Schedule Global Excel Spreadsheet E-mail Brazil UK Figure 2: Bursting Excel spreadsheets saves system resources by running a job once for all users, but ensures privacy by distributing only the appropriate sections to each user. The Corporate Excel Sourcebook Sometimes spreadsheet content and format are highly customized and frequently change. In such an environment, a suitable strategy is to provide foundation spreadsheet data in a generic workbook or Corporate Excel Sourcebook. This workbook standardizes detail, summary, and calculated amounts. It is created and stored on a universally accessible server by a ReportCaster job that refreshes it as often as necessary. Organization of WebFOCUS Corporate Excel Sourcebooks is extremely flexible. Data can be structured on a single worksheet or multiple worksheets. If there are multiple worksheets, all sheets can have identical format but different ranges of data. Using the profit and loss statement example, a workbook could be organized with one sheet per cost center. 5 Information Builders
Another multiple worksheet solution is to consolidate unrelated reports in a single workbook. For example, one worksheet might contain inventory reports, the next sheet might contain human resources reports, and so on. No connection between the worksheets is necessary. To populate personal spreadsheets, users connect to the central server and reference the central workbook in their local spreadsheets. References can pluck data from the entire workbook, worksheet, or from a range of cells within a report. To facilitate selective data gathering, the Corporate Excel Sourcebook can be designed with named cell ranges. Named ranges are put in place when the WebFOCUS Excel job is set up. They shield users from needing to change their personal spreadsheets when report size and position change in the Corporate Excel Sourcebook. Named ranges can be set up for a sheet, report, column, row, or any rectangle of cells. With Corporate Excel Sourcebooks, users each take exactly the data they want, no more and no less. They can then style and manipulate that data in any way they please and the audit trail to controlled corporate data is preserved. Corporate Data Employees Customers Orders A B C D E F 1 A B C D E F 2 1 3 2 4 3 5 4 6 5 7 6 8 7 9 8 9 Purchasing Range Insurance Cost Range Sales Payroll Sheet Inventory Figure 3: A Corporate Excel Sourcebook contains corporate Excel data that all enterprise users can link to with their own spreadsheets. Deliver Advanced Functionality It is worth discussing the scope and variety of pre-formatted Excel features supported by WebFOCUS. Maximum automated data manipulation means minimum chance for human error. 6 Controlling Excel Chaos With Business Intelligence
WebFOCUS supports all Excel functionality, either directly or through the use of spreadsheet templates. The specific mix of Excel capabilities pre-set by WebFOCUS extract and distribution jobs is limited only by the standard versions of Excel in use. All features of every Excel release are supported. Following is a summary of stylistic and analytic features available. Summary Lines WebFOCUS generates report headings and footings, page headings and footings, and recap lines on sort breaks, all of which can have a combination of text and numeric totals. Worksheet and workbook names can also be automatically generated. Styling and Conditional Styling WebFOCUS controls the font family, size, color, and style for each and every element of a spreadsheet. Summary lines, column titles, column content, etc. can be styled separately. Conditional styling is also supported so that data meeting defined business criteria is highlighted by style or color. Drill Down Hypertext links to other spreadsheets, images, or HTML pages can be automatically embedded in distributed Excel output. Links can be positioned in summary lines or within the data of the report. Formulas and Calculation Logic Complex formulas can be implemented in the WebFOCUS extract job and passed to Excel. The benefit is that these complex calculations are written and tested centrally by IT professionals familiar with corporate data sources and with structured programming practices. The WebFOCUS self-documenting code becomes the audit trail for Sarbanes Oxley and other compliance regulations. Even relatively simple calculations such as growth rate and profitability can be incorrect when factors are reversed. Consider the multiplicity of factors in some calculations typically implemented in spreadsheets. In analyzing pay grades for example, human resources departments often rate performance based on manager performance evaluation, absenteeism, years of service, current salary within pay grade, and corporate guidelines for pay increases. If these formulas are incorrectly calculated, legal issues could arise. Insurance companies might base profitability projections for corporate health insurance contracts on rates charged, historic dollar amount of claims, employee population age, and numbers of dependents. All factors must be weighted and if a new factor is introduced, all weights must be re-calculated. 7 Information Builders
Calculating the cost of goods sold and the value of inventory on hand can also involve an array of factors. Inventory valuations typically involve a range of prices and costs for raw material, labor, shipping, etc. at different points in time, periodic production volumes, and LIFO or FIFO assumptions about the rotation of goods. A spreadsheet capturing these rules would quickly become a dangerous breeding ground for error. The potential for error is dramatically reduced when such calculations are done by a single, thoroughly tested and auditable module with access to the correct data sources. Pre-calculated formulas Formatted headings and footings Drill down to other reports, spreadsheets, or graphs Conditional styling when sales are greater than $100k Figure 4: A typical WebFOCUS Excel spreadsheet has predefined calculations, styling, and drill-downs. PivotTables PivotTables make it possible to reveal trends in a data sample by sorting, filtering, adding key elements, and charting. Despite a built-in Excel PivotTable Wizard, there are pitfalls to developing PivotTables. While most report consumers manipulate PivotTables effectively, far fewer users can readily produce sophisticated, styled PivotTables. 8 Controlling Excel Chaos With Business Intelligence
With WebFOCUS, PivotTables can be developed by IT professionals and built into the extract job. WebFOCUS PivotTable Developers work in a graphical user interface, building the initial view of the table as well as the lists of cached and page control fields. Figure 5: Most Excel users can manipulate PivotTables, but far fewer can design them. Templates For all spreadsheet functionality other than the features mentioned above, WebFOCUS can implement Excel functionality by leveraging templates. Additional functions might include print preview functions or graphing. Undoubtedly, spreadsheets exist that represent a great deal of effort spent in development of macros. If these macros have been tested and certified, they can be leveraged in place without re-writing. They can be applied to WebFOCUS formatted data by use of templates. Templates are skeleton workbooks containing any Excel functionality. During spreadsheet creation, the WebFOCUS Reporting Server locates the template, merges data into a designated sheet, and saves the results to a separate workbook file. The final spreadsheet has corporate data inserted into a pre-existing user approved format. Templates not only leverage all existing 9 Information Builders
functionality of Excel, they also position WebFOCUS to leverage future functionality that Microsoft will add to Excel. x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x xxxxxxx xxxxxxx xxxxxxx xxxxxxx xxxxx xxxxx xx xxxx Corporate Data Excel Templates WebFOCUS 14% 12% Populated Spreadsheets 5% 25% 30% Figure 6: WebFOCUS merges an Excel layout (template) with corporate data to provide the exact formatting and data manipulation required. Figure 7: An Excel template before and after it is populated with data by WebFOCUS. 10 Controlling Excel Chaos With Business Intelligence
Answering the Ad Hoc Question There will always be a need for new spreadsheets to address the latest business challenge. Latebreaking situations that require new subsets of data can be tackled with an interactive ad hoc reporting system. Such an application has a series of parameter selection screens that front-end appropriate corporate data stores. All characteristics of a target Excel spreadsheet are defined via the parameter selection screens. Which database fields and calculated fields are included, what are the summarization rules of the report, the sort criteria, the formatting, and which cascading stylesheet these are all criteria picked by users from menus. The resulting output is always a spreadsheet in the user s preferred Excel format. Corporate Data Inventory Forms A D A B D E A B CD E F B C E F C F Sales Orders Parameters: Sort Filter Aggregation Styling Figure 8: Answering the ad hoc question is handled with a nightly parameterized application that formats customized spreadsheet output. 11 Information Builders
Figure 9: Detail of a parameterized form and the report it generates. For example, the insurance company that historically computed health insurance rates based on historic claims, population age, and number of dependents could add salary to the rate calculation. Analysts want to model the effect of the new calculation on profitability of the contract. Analysts can multi-select salary, along with the other rate calculation factors and derive a new spreadsheet with factors including individual or aggregated salaries. This spreadsheet could then be manipulated off-line in various what-if scenarios but the foundation data is enterprise-certified. 12 Controlling Excel Chaos With Business Intelligence
Conclusion Enterprises can take control of the burgeoning number of unsanctioned spreadsheets by adopting a policy of centralized master spreadsheet generation. Excel is a powerful and ubiquitous analytical tool that can make a business analyst self-sufficient, but it lacks strong audit and error-checking capabilities. However, master spreadsheets drawn from the company s primary operational and warehoused data sources can provide an audit trail back to elemental transactions. WebFOCUS creates fully summarized, organized, and styled Excel spreadsheets with complex formulas already in place, lowering the potential for keystroke and logic error. WebFOCUSgenerated spreadsheets can be pushed out to business analysts in e-mail or staged on a central server where analysts fetch target data by hitching their own linked spreadsheets to the master corporate spreadsheet. Furthermore, on the corporate Excel form, analysts can screen, sort, and summarize samples of data to derive spreadsheet answers to ad hoc questions. With a little analysis and the right WebFOCUS business intelligence tools, you can finally have a corporate Excel spreadsheet strategy that works. 13 Information Builders
Sales and Consulting Offices North America United States Atlanta,* GA (770) 395-9913 Baltimore, MD Consulting: (703) 247-5565 Boston,* MA (781) 224-7660 Channels (800) 969-4636 Charlotte,* NC Consulting: (704) 494-2680 Chicago,* IL (630) 971-6700 Cincinnati,* OH (513) 891-2338 Cleveland,* OH (216) 520-1333 Dallas,* TX (972) 490-1300 Denver,* CO (303) 770-4440 Detroit,* MI (248) 743-3030 Federal Systems,* DC (703) 276-9006 Hartford, CT (860) 249-7229 Houston,* TX (713) 952-4800 Los Angeles,* CA (310) 615-0735 Metropolitan,* NY Sales: (212) 736-7928 Consulting: (212) 736-4433, ext. 4443 Minneapolis,* MN (651) 602-9100 New Jersey* (973) 593-0022 Orlando,* FL (407) 804-8000 Philadelphia,* PA (610) 940-0790 Pittsburgh, PA (412) 494-9699 St. Louis,* MO (636) 519-1411 San Jose,* CA (408) 453-7600 Seattle,* WA (206) 624-9055 Washington,* DC Sales: (703) 276-9006 Consulting: (703) 247-5565 Canada Information Builders (Canada) Inc. Calgary (403) 538-5415 Ottawa (613) 233-0865 Montreal* (514) 334-0404 Toronto* (416) 364-2760 Vancouver* (604) 688-2499 Mexico Information Builders Mexico Mexico City 52-55-5062-0660 Australia Information Builders Pty. Ltd. Melbourne 61-3-9631-7900 Sydney 61-2-8223-0600 Europe Belgium Information Builders Belgium Brussels 32-2-7430240 France Information Builders France S.A. Paris 33-14-507-6600 Germany Information Builders (Deutschland) Dusseldorf 49-211-522877-0 Eschborn 49-6196-77576-0 Munich 49-89-35489-0 Stuttgart 49-711-7287288-0 Netherlands Information Builders (Netherlands) B.V. Amsterdam 31-20-4563333 Portugal Information Builders Portugal Lisbon 351-217-230-720 Spain Information Builders Iberica S.A. Barcelona 34-93-344-32-70 Bilbao 34-94-425-72-24 Madrid 34-91-710-22-75 Switzerland Information Builders Switzerland AG Dietlikon 41-44-839-49-49 United Kingdom Information Builders (UK) Ltd. London 44-845-658-8484 Representatives Austria Raiffeisen Informatik Consulting GmbH Vienna 43-12-1136-3870 Brazil InfoBuild Brazil Ltda. São Paulo 55-11-3285-1050 China InfoBuild China, Inc. Shanghai 86-21-50805432 Finland InfoBuild Oy Espoo 358-207-580-843 Greece Applied Science Athens 30-210-699-8225 Guatemala IDS de Centroamerica Guatemala City 502-2361-0506 Gulf States Nesma Advanced Technologies Bahrain Kuwait Oman Qatar Yemen United Arab Emirates Riyadh 96-1-465-6767 India Divas Offshore Software Technologies Private Limited Gurgaon 91-124-501-8801 Israel NESS A.T. Ltd. Tel Aviv 972-3-5483638 Italy Selesta G C Applications S.P.A. Genova 39-010-64201-224 Milan 39-02-2515181 Torino 39-011-5513-211 Japan K.K. Ashisuto Osaka 81-6-6373-7113 Tokyo 81-3-5276-5863 Malaysia Elite Software Technology Sdn Bhd Kuala Lumpur 60-3-21165682 Norway InfoBuild Norway Oslo 47-23-10-02-80 Philippines Beacon Frontline Solutions, Inc. 63-2-750-1972 Saudi Arabia Nesma Advanced Technology Co. Riyadh 996-1-4656767 Singapore Automatic Identification Technology Ltd. 65-6286-2922 South Africa Fujitsu Services (Pty.) Ltd. Johannesburg 27-11-2335911 South Korea Unitech Infocom Co. Ltd. Seoul 82-2-2026-3100 Sweden Cybernetics Business Solutions AB Solna 46-7539900 Taiwan Galaxy Software Services Taipei 886-2-2586-7890 Thailand Datapro Computer Systems Co. Ltd. Bangkok 662-679-1927, ext. 200 Turkey Istanbul Veripark 90-212-283-9123** Venezuela InfoServices Consulting Caracas 58-212-763-1653 Toll-Free Numbers Sales and Information (800) 969-INFO ISV, VAR, and SI Partner Information (800) 969-4636 **Training facilities are located at these branches; additional locations are available. **Authorized to sell iway Software only. Corporate Headquarters Two Penn Plaza, New York, NY 10121-2898 (212) 736-4433 Fax (212) 967-6406 DN4601444.0706 www.informationbuilders.com askinfo@ibi.com Canadian Headquarters 150 York St., Suite 1000, Toronto, ON M5H 3S5 (416) 364-2760 Fax (416) 364-6552 For International Inquiries +1(212) 736-4433 Copyright 2006 by Information Builders, Inc. All rights reserved. [51] All products and product names mentioned in this publication are trademarks or registered trademarks of their respective companies. Printed in the U.S.A. on recycled paper