Open Source meets Business Intelligence Seminar Business Intelligence Winter Term 06/07 Monika Podolecheva University of Konstanz Department of Computer and Information Science Tutor: Prof. M. Sholl, Prof. H. Reiterer, S. Mannsman Abstract. This seminar paper concerns topics of the area business intelligence with focus on open source solutions for the business intelligence. First, an introduction to the concepts and objectives of the business intelligence solutions is given. Statistics and studies considering the overall business intelligence market situation and trends are presented, getting more focused on the open source vendors and their solutions and contribution on the worldwide market. Special attention is paid to the open source company Pentaho and their product Business Intelligence Suite. 1 Introduction In recent years, the influence of technology and growth of information volume have challenged the industry leaders to improve their performance to be able to meet the constantly changing customer demands and market dynamics. Business intelligence is a powerful instrument that can support them to sustain and improve their competitive position. The term business intelligence (BI) was first invented by Howard Dresner from the Gartner Group 1 in 1989 (now chief strategy officer at Hyperion 2 ). He described BI as a set of concepts and methodologies to improve decision making in business through use of facts and fact based systems. Nowadays it has become popularity and many definitions have emerged. One suggested by the Business Intelligence Institute 3 fully covers the main idea behind BI: BI is the management and analysis of vast amounts of information in order to gain valuable insights to drive strategic business decisions, and to support operational processes with new functions. BI includes technology practices like data warehouses, data marts, data mining, text mining, and on-line analytical processing. The objective of BI is to transform data into useful information and to help companies to become a more comprehensive knowledge of the factors affecting their 1 http://www.gartner.com/ 2 http://www.hyperion.com/ 3 http://www.bii.be/belgium/index.jsp
2 Monika Podolecheva business and help companies to make better competitive analysis and business decisions. Many companies already benefit from applying BI solutions and the BI tools market has experienced rapid growth over the past years. Some studies shown in the next section present an overview of the market situation which can give an overall idea of the worldwide BI tools adoption and trends. 2 Business Intelligence Market Overview 2.1 Market Segmentation Rough market segmentation is suggested by the market research and analysis firm International Data Corporation (IDC) 4 according to software packaging: database-embedded vs. standalone BI tools. More refined, the standalone BI tools market segment is composed first of end-user query, reporting and analysis software and second of advanced analytics software, which evolution on the market in years 2003-2005 are presented on Fig.??. The idea of this segmentation and the following geographic and operating environment breakdown of the BI market shown on Fig.?? is to provide a better market monitoring and thus possibly better forecasting. Fig. 1. BI market evolution from 2003-2005 (Source: IDC) In 2005 the market grew by 11.5% to reach $5.7 billion worldwide. A higher growth rate is observed by a database-embedded and the end-user standalone tools. The American market continues to be the largest one. Considering the revenue share by operating system, obvious leaders on the market is Windows. Linux is the fastest-growing platform and it is to expect that new open source BI initiatives are likely to sustain or force this trend. 4 http://www.idc.com/
Open Source meets Business Intelligence 3 Fig. 2. Worldwide BI Tools Revenue Share by Region (left) and by Operating Environment (right) in 2005 (Source: IDC) 2.2 Market Trends The BI market research has shown that 2005 was a turning point and the beginning of new wave of investments in BI. Until now, BI market was primary focused on delivery of information to analysts and managers; however they represent only 15-20% of the employees. The next wave of BI will reach out to these power users as well as other stakeholders on different levels in the organisations. Furthermore, BI should support the intercomany connectivity, or in other words the linking of business processes with partners, suppliers, and customers. The BI market is expected to growth because applying BI tools leads to improvement in the market analysis, the budgeting controlling and strategy planning. For an organisation that means higher rate of return, so the BI technology is something without organisations cannot succeed. Another point is that the broader adoption of BI software is expected to continue as more end users gain access to query and reporting tools and as organisations embed BI software into operational applications supporting all business processes. IDC research shows an optimistically trend as one can see on Fig.?? below: Fig. 3. Market forecast till 2010 (Source: IDC)
4 Monika Podolecheva There is an expected growth by more than 10% every year which means an overall growth rate about 50% from 2006 till 2010. A clear trend is that the BI market is continuously dominated by larger, full-service companies, such as IBM and Oracle, and specialized vendors, such as SAS, Cognos, Business Objects and Hyperion. Maybe not all of the current vendors will survive the next years in the BI market as independent vendors. The BI market seems to very turbulent and merger and acquisition activities are likely to continue and increase. Commercial Vendors To date, the BI market is dominated by the proprietary vendors, therefore let us take a look at the main commercial players in 2007 identified by Gartner Corp. Fig. 4. Magic Quadrant (Source: Gartner) On Fig.?? is presented the Magic Quadrant, a graphical representation of a marketplace at and for a specific time period. It depicts Gartners analysis of how certain vendors measure against criteria for that marketplace. Comparing the researches form IDC and Gartner, one can see clear similarity in the positioning of the vendors in the BI market.
Open Source meets Business Intelligence 5 Open Source BI Tools Vendors In the last years there are also sights that open source software is coming into the BI tools market. As the commercial vendors usually offer complete packages inclusive service and support, the open source vendors concentrate on stand-alone solutions but even more with tendency of providing software-stacks. Although, the impact of open source BI tools is forecasted to be limited over the next five years. After this period, open source may develop into a stronger competitive force. With open source, the software is usually freely. Therefore, organisations (and especially smaller organisations) can benefit from the eliminated licensing costs. The open source allows the user to focus on the needed tasks. Usually the commercial vendors offer complete solutions including reporting, analysis and other functions, but the normal user may simply want to execute reports and not to pay for OLAP. Furthermore, an organisation can access the project s source code and to embed it into an existing application or change it according to the organisational demands. These flexibility and simplicity lead to wider acceptance of business intelligence products and services. Vendors such as Pentaho, JasperSoft, and Actuate clearly display the first signs of potential market niche. There are many vendors and solutions in the market and in the following a short overview of them Area of application of the Open Source BI Tools Some of the well known open source BI tools vendors are Pentaho, JasperSoft, Jedox and Greenplum. Many communities and universities also offer open source BI solutions, as the University of Waikato, Neuseeland, SpagoBI, and the Eclipse Foundation. As next a more detailed application area of the different vendors and products is given: Data Gathering and Storage: most used in this area are MySQL, PostgreSQL, and Palo - database offered by Jedox Preprocessing ( ETL - Extraction, Transformation & Loading): Kettle, the CloverETL-Framework and Enhydra Octopus Data Mining: The most popular free OLAP-Server is the Java based Mondrian- Project; WEKA Reporting: JasperReports from JasperSoft, BIRT Furthermore JasperSoft and Pentaho provide complete BI solutions: JasperSoft - the JasperIntelligence-Suite and Pentaho - the Pentaho BI Suite. The next sections will present both open source BI market leaders and their solutions more focusing on the company Pentaho. 3 JasperSoft JasperSoft is founded 2004 in San Francisco, CA, U.S.A with less than 50 employees. The company is delivering Commercial Open Source in the area of BI.
6 Monika Podolecheva The JasperSoft reporting products have become the cornerstone of many mission critical application solutions in major market segments such as financial, retail and manufacturing, providing on-demand and real-time information delivery for critical applications such as auditing and reporting, customer self-service, compliance management and systems performance and tuning management. JasperSoft mission and success plan: make the BI more accessible to ordinary business users. 3.1 Products JasperSoft offers its customers the choice of a commercial or open source business intelligence suite. Following open source products are provided as stand-alone applications or combined in a complete solution: JasperReport - reporting tool that has the ability to deliver rich content onto the screen, to the printer or into PDF, HTML, XLS, CSV and XML files; can stand alone or be embedded directly into a user s application to give it advanced reporting capabilities. JasperServer - standalone server or web services reporting engine; dramatically reduces the time required to build and deploy server applications that need reports and analytics, giving operational decision-makers access to data previously accessible only to upper management. JasperAnalysis - provides data analysis (OLAP); JasperAnalysis can be used to explore trends, patterns, anomalies, and correlations in data by allowing users to dynamically slice and dice, pivot, filter, chart, drill-down, or roll-up a cube of data in real-time. JasperETL - integration tool for moving information from multiple databases into a data warehouse, and formatting the data so it can be analysed ireport - powerful graphical report designer Additionally, the company provides the commercial product line JasperDecisions that will offer complementary capabilities for advanced functionality to the JasperReports community. The JasperDecisions product line consists of: Scope Server: a java server-based operational reporting solution for interactive, self-serve reporting and analytics. Scope Designer: a swing-based report designer for Scope Server report development JasperDecisions is currently deployed in over 50 leading corporations and ISVs including IBM, British Telecom, Informatica and the US Department of Defense. Some of the open source products have been also adopted by many companies, especially the JasperReport which is currently ranked as the 7th most active project on Sourceforge.com, which hosts over 100,000 open source projects. An industry standard, it is now integrated with MySQL and JBoss.
Open Source meets Business Intelligence 7 4 Pentaho Pentaho is founded 2004 in Orlando, U.S.A by a team of industry veterans with long experience and successful projects for leading commercial vendors including Business Objects, Cognos, Hyperion, IBM, Oracle, and SAS. The company builds components for the Open Source community and enhances components developed by others. Pentaho integrates components into cohesive and flexible building blocks that Java developers can use to rapidly assemble custom solutions and other sides creates complete, out-of-the-box products and a comprehensive BI Platform for End-Users. Provide comprehensive technical support, release management, quality assurance, and enterprise services. Pentaho conception: Pentaho manages, facilitates, supports, and takes the lead development role in the Pentaho BI Project - a pioneering initiative by the Open Source development community to provide organisations with a comprehensive set of BI capabilities that enable them to radically improve business performance, efficiency, and effectiveness. Pentaho mission: not only to provide an Open Source alternative, but to surpass all commercial offerings in terms of features, functions, and benefits. 4.1 Products Pentaho offers a range of products that covers the application areas Reporting, Analysis, Dashboards, Data Mining, and Data Integration. Reporting As recognizing that reporting is an important issue for any organisation, Pentaho deployed its reporting tool as its first BI application: in January 2006 Pentaho have adopted JFreeReport - a free Java reporting library for embedded solutions as part of its BI Framework. The former known as JFreereReport and now as Pentaho Reporting allows organisations to easily access, format, and distribute information to employees, customers, and partners. It offers flexible deployment from standalone desktop reporting, to interactive web-based reporting to enterprise business intelligence. Further characteristics are: Broad data source support including relational, OLAP, or XML-based data sources Flexible output options including Adobe PDF, HTML, Microsoft Excel, Rich Text Format, or plain text Support for servlets (uses the JFreeReport extensions) Wizard-driven report design for fast, easy report creation Professional Edition available with additional deployment capabilities including clustering, subscriptions, directory integration, versioning, auditing, and more
8 Monika Podolecheva An example of reporting results can be seen on Fig.??. It presents business ratios for a specific company: in the current, the actual-budged variance for certain (here central) regions are reported. The creator and primary sponsor for Fig. 5. Reporting the Actual-Budget Variance: Central region (Source: Pentaho) the JFreeReport project has also joined Pentaho as Chief Architect of Reporting Solutions. This acquisition furthers Pentahos strategy of providing a comprehensive BI suite built on best-in-class technology delivered via a professional open source model. Analysing Pentaho Analysis is a powerful tool that helps users to operate with maximum effectiveness by gaining the insights and understanding they need to make optimal decisions, allowing them to explore business information by dragging, dropping, drilling into, and cross-tabulating data. It supports users through an intuitive interface, speed-of-thought response times to complex analytical queries, and user-by-user customization and preferences options. It provides extensive analysis capabilities that includes a pivot table viewers (JPivot), advanced graphical displays using SVG or Flash, integrated dashboard widgets, data mining integration, portal integration, and workflow integration. These capabilities come complete with scheduling, web services, content navigation and management, security, application integration, and auditing. Analysis can be performed on relational data source via Pentaho Analysis Services (based on Mondrian OLAP). An example of analysis results can be seen on Fig.?? and Fig.??. They present the same business ratios as in the reporting example. The first one shows the result from query for actual-budged variance for all operative regions and departments. The second one is more refined require regarding only the central
Open Source meets Business Intelligence 9 region with all his departments. Further filtering and customization is offered to the user for more detailed analysis of the business data. Fig. 6. Actual-Budget Variance: All regions (Source: Pentaho) Fig. 7. Actual-Budget Variance: Central region (Source: Pentaho)
10 Monika Podolecheva In addition, Pentaho Spreadsheet Services allows users to browse, drill, pivot and chart against Pentaho Analysis Services all from within Microsoft Excel. Dashboards Pentaho Dashboards provide immediate insight into individual, departmental, or enterprise performance. It provides comprehensive metrics management capabilities which allow for the definition and tracking of critical metrics at each organisational level. Appropriate visualizing helps users immediately see which business metrics are on track and alerts them which need more attention. Integration with Pentaho Reporting and Analysis allows users to drill to underlying reports and analysis to understand what factors are contributing to good or bad performance. At a glance: Provide re-usable display widgets (gauges, dials, charts etc) Integration of external content like web pages, 3rd party applications, and RSS feeds Integrate reporting, analysis, and dashboard content Provide configurable, common filtering controls Provide role-based security and filtering Provide user-by-user customization and preferences Data Mining Data Mining is the process of running data through sophisticated algorithms to uncover meaningful patterns and correlations that may otherwise be hidden. These can support the business understanding and predictive analytics and thus provide a truly sustainable competitive advantage and enable any organisation to maximize both its efficiency and effectiveness. Pentaho Data Mining incorporates Weka, a collection of machine learning algorithms including clustering, segmentation, decision trees, random forests, neural networks, and principal component analysis. These algorithms are combined with OLAP technologies to provide machine-intelligent data analysis to end users. Data mining tools can analyze historical data to create predictive models and then using Pentaho Reporting and Analysis companies can distribute this information to the appropriate people. Features and Benefits of Pentaho Data Mining: Enables embedding of recommendations in user applications Visualisation of the results; interactive output Filters for discretization, normalization, re-sampling, attribute selection, and transforming and combining attributes Provides role-based security and business rules Supports Java Single Sign-On/JOSSO and LDAP to integrate with existing enterprise security
Open Source meets Business Intelligence 11 Data Integration The data is available in any possible form and everywhere. Providing consistent data across all sources of information is one of the biggest challenges faced by IT organisations today. To overcome this problem Pentaho has adopted the open source project Kettle and nowadays delivers a powerful Extraction, Transformation and Loading (ETL) tool - Pentaho Data Integration. It provides an easy-to-use graphical, drag-and-drop environment and includes: Rich transformation library with over 50 out-of-the-box mapping objects Advanced data warehousing support for Slowly Changing and Junk Dimensions Export of databases to text-files or other databases Import of data into databases, ranging from text-files to excel sheets Data migration between database applications Exploration of data in existing databases (tables, views, etc.) Enterprise-class performance and scalability SAP Connector also available BI Platform All capabilities described above are integrated into a comprehensive BI Framework with a centralized repository, security and other platform-wide features. The Pentaho BI Platform provides the architecture and infrastructure required for building solutions to specific BI problems. The platform includes an embedded workflow engine and can be easily integrated into business processes. Core services including authentication, logging, auditing, workflow, web services, and rules engines are offered. The framework also includes a solution engine that integrates the former presented reporting, analysis, dashboards and data mining components to form a sophisticated and complete BI platform. An overview of the architecture can be seen on Fig.??. The server architecture has been built for the J2EE environment and complies with many of the associated specifications. The client design environment is built around the Eclipse workbench, with most of the end-user interaction delivered via HTML and other thin-client technologies. Having introduced the whole Pentaho product assortment, we are coming to the question: who can benefit from these products? Java developers can use project components for rapidly assemble custom BI solutions. Independent software vendors can enhance the value and capability of their solutions by embedding BI functionality. And least but not least - end-users can benefit from applying lower cost BI tools.
12 Monika Podolecheva Fig. 8. Pentaho Architecture (Source: Pentaho) 5 Conclusions An overview of the BI market and forecast for its evolution in the next years was given. Some studies and statistics were introduced with the purpose to show that BI market is expected to maintain a high level of growth. According to IDC, the last year was a turning point for the market and the start of a new wave of investment in BI. The market is maturing very slowly and until 2020 companies will continue to purchase BI with the aim of expanding its reach to more users both inside and outside the organisation. Further opportunities for BI application and chance for the BI vendors to explore new markets provide the so far ignored areas of chat rooms, blogs, wikis, and online communities. Open
Open Source meets Business Intelligence 13 source has crept into the market over the last few years as vendors like Pentaho and JasperSoft have carved out a niche, but IDC predicts the use of open source BI will be limited over the next five years because the BI market isn t large enough or generic enough to support significant open source offerings. After this time open source may develop into a stronger competitive force, especially because of lower costs, reduced dependence on software vendors, simplicity and flexibility. Open Source BI tools may not have the maturity and completeness as some commercial products. However the main reason for that is that they are still at the beginning of their development.
14 Monika Podolecheva References 1. Alexandra Kleijn: Business Intelligence mit Open Source Heise Open, published 2006 2. Martin LaMonica: Open source meets business intelligence, CNET News.com, published: April 23, 2006, 2006. 3. Barney Beal: IDC names BI market share leaders, predicts consolidation SearchCRM.com 27. Jul 2006 4. International Data Corporation: Worldwide Business Intelligence Tools 2005 Vendor Shares Source: http : //www.sas.com/news/analysts/idc b i 0706.pdf, July 2006 5. International Data Corporation: Worldwide Business Intelligence Tools 2005 Vendor Shares Source: http : //www.sas.com/news/analysts/idc analytics2 1006.pdf, October 2006 6. Rick Mortensen: The Open Source BI Trend Will Grow - Here s Why Source: http : //www.dmreview.com/article sub.cfm?articleid = 1050215, March 2006 7. Pentaho http : //www.pentaho.com/ 8. JasperSoft http : //www.jaspersoft.com/ 9. Hyperion http : //www.hyperion.com/