Building Open-Source Based Architecture of Enterprise Applications for Business Intelligence TOMAS CHLOUBA Department of Informatics and Quantitative Methods University of Hradec Kralove Rokitanskeho 62, Hradec Kralove, 500 03 CZECH REPUBLIC tomas.chlouba@uhk.cz http://fim.uhk.cz DANIEL KMINEK kminek@ders.cz Abstract: - The paper inquires into the area of enterprise application development and open-source based technologies. Basic principles of enterprise application development and open-source technologies are described. The paper then focuses on the possibilities of using open-source and open-source based technologies in the area of enterprise applications. The first part describes the methodology, which was created for the purpose of the study of open-source technologies and their usability for enterprise class applications. The second part contains an example of how the methodology was used in practice for the research of open-source technologies for system integration, with special focus on data mining, data analysis and representation in the form of management reports. This was used to build a Business Intelligence solution, integrated from opensource technologies. The methodology also reflects the current problem with the lack of skilled information technology professionals by focusing on simple and effective solutions, that can easily be integrated, managed and maintained. The results of the research are described in detail in the form of Best Practices and Recommendations for future researchers. Key-Words: - business intelligence, open-source, enterprise 1 Introduction In today's IT world, where highly skilled IT professionals are valued, the development of complex software becomes more and more expensive as the software complexity grows. The lack of software developers and also the price to keep one leads to finding new ways of decreasing costs with respect to quality and security. One of the steps to consider is using open-source technologies [3]. These technologies have advantages and disadvantages as described further, but the main concern is if these technologies can stand up to commercial solutions and are capable of competing with such solutions when picked carefully and put together. This paper describes the methodology for choosing the proper open-source technology with respect to several aspects, some of which are specific to open-source [4]. The methodology can then be used by anyone trying to put together a open-source project and it simplifies the start with open-source technologies. Another advantage of the methodology and the project itself is that a real life project has been put together and is working effectively based on the methodology. The examples written in this paper can serve as another source of information for anyone trying to achieve the same. 2 Methodology The methodology of choosing a suitable opensource technology for the needs of the organization is composed of several steps as described later in this chapter. This methodology is based on real life experience with picking suitable open-source tools to create a Business Intelligence solution. The choosing of components for the Business ISBN: 978-1-61804-126-5 343
Intelligence solution is later described and this methodology is applied. There are several aspects that have to be taken into consideration when choosing an opensource project to build a solution on it [5]. Whether the aspects noted here are relevant to the project must always be considered. While some aspects may be more important than others, weight may be attributed to the aspects as to increase or decrease its importance. This methodology does not include the specifics of the inner workings of the research and development team but rather focuses on the tasks required to build an open-source solution. This includes identifying the requirements of the project, mapping technological components to these requirements and then choosing the right technology with respect to the original requirements and other general aspects, including the integrability of the component. When considering using open-source technologies, there are some risks, that should be considered with every such technology. These are included in the aspects that follow. The aspects of every open-source software, that must be considered before choosing the proper technology for the job are support, security, quality, manageability, quick changes, backwards compatibility and integration. One must always consider whether the considered aspect is relevant to the task at hand. The aspects are described below. support Before picking a technology, the support for this technology should be considered. User and administration manuals, other available documentation, tutorials, user forums and the community size around the technology are the aspects to be considered here. security To find out if a technology is secure enough for the purpose intended, implementations of this technology should be identified and their security incidents considered. The amount and severity of known issues of the project will also tell us about the product a lot, however time to fix should be an aspect in this case as well. quality The quality of the product is hardly identifiable, but we can simply compare the product features with the identified requirements and consider if they are fulfilled. manageability Whether the product is self sustainable or requires continuous management and intervention. Managing the product may be more expensive than purchasing a commercial license. quick changes Open-source products have the advantage of often being developed very quickly, but we should be aware of the fact, that fast changes mean that a level of uncertainty is introduced and quick development may introduce more issues into the software. That would increase the maintenance costs. Usage of a snapshot can be considered, however that stops us from being able to upgrade the product. backwards compatibility As open-source software is often developed fast, backward compatibility may sometimes not be a concern for the developers, meaning that upgradability will decrease, because upgrading to a newer version of a product with new API for example may be very expensive, because programmatical intervention may be required, the software would have to be retested etc. integration When considering an open-source product as a component of a system, the possibilities of integration must be considered for such a product. Public API, input and output formats and common logging interface are few of the aspects that must be considered in this case. The above aspects are specific to open-source technologies, where they are usually more important than when using commercial products. There are of course other aspects, which should be considered, primarily whether the product fulfills the given requirements, if it is platform independent etc. The aspects in this methodology should thus be included among all the aspects being considered when picking an open-source technology. After all of the aspects are considered a comparison table should be put together and the most suitable product should be chosen. If the individual aspects are attributed weight, than some calculations first have to be made to get ISBN: 978-1-61804-126-5 344
the proper numbers for consideration. The above methodology is specific to open-source technologies, but not any specific area. As it is with every methodology, the real requirements are the basis for the picking of a technology, these general rules are guidelines of how to pick the right one, but sometimes some of the aspect may not be as important (or at all important) as others. (For example an intranet application may not have such high requirements for security as a public web application.) 3 Requirements To start the research of specific components needed for the system, the key requirements first have to be identified. Based on the functionality needed by the Business Intelligence solution, primary functional requirements were identified. Other nonfunctional and functional requirements rose from the demands of such a solution and are explained with each requirement. 3.1.General requirements The identified general requirements are: extensibility The possibility to interchange or add another component. Also includes the extensibility of the component by coding additional functionality. flexibility How quick can the changes be implemented under given conditions. integrability The possibility to connect with existing systems the customer uses. This includes identity management, portal technologies and notifications. open-source The code of the component is available under one of the open-source licenses. simple installation The main package should be installable simply and quickly. Under the assumption the source data and infrastructure are available, the standard functionality should be installable in one day. 3.2. Technological requirements The identified technological requirements are: data security Securing data against unauthorized access, changes and deletes. ETL (extract, transform, load) Job scheduling. OLAP (online analytical processing) Support for OLAP data views, drill through, drilldown. OLAP data views may be extended later. support for essential data For Business Intelligence, the essential data are time, organizational structure and chart of account. internet application The final application should be available with a thin client, not requiring the client to install anything for the end user. 3.3. Modules The identified modules are: operating metrics These are filters for every attribute where it makes sense (panel, page and metrics filters), charts and their different types, to display data, and also output formats (xls, pdf, etc.). modeling The possibility to create persistent modeling reports, which can later be viewed by an authorized viewer or exported into one of the output formats. modeling for CPM (Corporate Performance Management) The possibility of modeling data for CPM based on the entries from the end user. Done by whatif analysis, allowing for the change of input parameters (costs allocation, what-if analysis of past data, future planning). application administration Primarily consists of the working with jobs, their scheduling, evidence and controlling, with minimum programming knowledge. Jobs must be autonomous, log their activity and also allow to be merged with manual jobs in a single place. The requirement of minimum programming knowledge means that the jobs should be controlled by a well designed graphical user interface, which should inform about the job status, variable status of variables used by the jobs, subtasks of the jobs and the mode of the ISBN: 978-1-61804-126-5 345
job (run, step by step, out of scheduling). It should also allow the editing of job status (manual job), cancelling or pausing of the job, adding or deleting a job, work with variables after the job is running, skip subtasks, define configuration for the jobs, change the job mode and edit job configuration. The jobs must be well secured, so that unauthorized users do not get access to the jobs. 3.4. GUI requirements The identified GUI requirements are: general GUI standards This means that the solution is optimized for a specific screen resolution, every panel, page and metrics have help available, color standards are supported for charts, tables and problem areas. GUI structure The first level of the GUI is the dashboard, every panel composes of pages. Every page then contains the specific metrics. integrated search Data mining feature with a page where the user can search for non trivial, hidden and potentionally useful information from the data available. Can be compared to advanced search in common applications. The above are only examples of valid requirements for the application. These requirements have to be described in more detail in order for the solution to be designed better, however that is not in the scope of this paper. There are also component requirements for a Business Intelligence system [1]. The composition and base components of a Business Intelligence systems are discussed a lot and it is unclear how the base components should be identified. However most of the authors agree on these base components of a Business Intelligence system: Data Warehouses / Data Sources Data sources are necessary so that relevant data can be accessed and transformed, possibly to data warehouses to allow fast analytical processing of the relevant data. Decision Support Analysis Business Intelligence is about delivering information mined from data available. This information can then be used in planning tools to project futures for management and thus support their decision making process [2]. Integrated Reporting A single reporting tool along the entire organization is very powerful, it should allow the users to generate customized reports as well as offer high quality predefined reports. Executive Dashboard All of the information, available from the Business Intelligence system should be displayed in a way that is comprehensible and easily accessible, such as a web portal. 4. Components The goal is to build a Business Intelligence solution, which has to have several components. Not all of the components used are described here, a few were chosen as an example of how the technologies were picked according to the methodology. For example data storage technologies are not described here. Commercial software was also compared with the open-source solutions, but as it was not used in any of the areas in the end, it is not in the scope of this paper. 4.1. Data Warehouses / Data sources Part of the process of working with data are ETL (Extract, Transform, Load) processes. These processes serve as a mechanism for gathering data from the organization using the Business Intelligence tools, data processing and storing in the form of source data for reporting. Two tools were chosen for consideration, Talend Open Studio and Pentaho Data Integration (formerly Kettle). During the research, it was found that Pentaho Data Integration Solution is not open enough and does not fulfill all of the requirements. Talend Open Studio was chosen as the ETL component, because of its simple and wide integrability with other systems, and the possibility to simply extend this tool. 4.2. Decision Support Analysis In the area of decision support, mainly tools for planning were considered. No suitable tool was found, the only tools available provide only backend support. This may be due to the high ISBN: 978-1-61804-126-5 346
complexity of planning systems and the fact that these tools need to suit very specific needs and sometimes even contain artificial intelligence components for further analysis of future development. Planning tool thus has to be created from scratch. 4.3. Integrated Reporting For reporting, three tools were considered, Business Intelligence and Reporting Tools (BIRT), JasperReports and Pentaho Reporting. Business Intelligence and Reporting Tools (BIRT) is a Business Intelligence and reporting tool. It is a project of the Eclipse Foundation and is distributed under the Eclipse Public License as open-source. It is written in Java and targeted for web applications. BIRT has three main components - visual report designer (part of the Eclipse IDE), a runtime component for the generation of reports, which can be integrated into any Java environment and a module for the creation of charts. BIRT reports are generated in XML and can be connected to several data sources (SQL database, JDO data storage, JFire objects, POJO objects, XML, Web Services, etc.). The BIRT project has very good official and unofficial (user forums) support, it has a support for JDBC, XML, Web Services and CSV as input formats. Its analytical components include data reports, charts, two-dimensional display, the reports can be exported into HTML, PDF, XLS, DOC, PPT, Postscript and CSV. The implementation of the solution is simple and a tutorial is available. JasperReports is a reporting engine written in Java. It can be integrated with almost any data source and has a wide variety of output formats. Part of JasperReports is ireport, a tool for graphical creation of reports. JasperReports also has its own server module, JasperServer. Pentaho Reporting is part of the Pentaho Business Intelligence Suite and provides web access to reports in the portal. The Pentaho tool is very easy to use and the creation of reports is fairly simple. It has a smaller variety of output formats, but these are still sufficient. BIRT was the tool that was chosen among these three, because it fully fulfilled the original requirements, allowed great extensibility and simple integration with data sources, has an effective support, detailed documentation and large community. It can also be integrated in almost any portal solution. 4.4. Executive Dashboard The goal for the end users is to display all available information in a user friendly way, all in one place. To achieve this, a web portal is a suitable solution. Two tools were considered, Liferay and Jetspeed 2. Both of these portals are Java-based. Liferay is a portal solution, consisting of individual components - portlets. New portlets can be easily integrated into the portal and the administration of portlets and the portal itself is very simple. The support for Liferay is well available and Liferay also has a large community of users. Jetspeed 2 has two modes - portal solution or JetUI rendering engine based solution. JetUI allows the display of own components. Liferay was chosen as the portal solution, due to the lack of JetUI features to display all of the required components and the fatct, that Jetspeed has a significantly smaller user base. 5 Conclusion The paper describes the possibilities of using open-source products in enterprise class applications with the focus on the area of Business Intelligence. First the motivation for this paper is described. Than in the first part of the paper, the methodology, designed for choosing a proper open-source technology is presented. This methodology comes from practical usage of the open-source technologies and their choosing. It is a general description and it may simplify the choosing of an opensource technology for organizations, which are considering using them, but have no or little experience in doing so. The methodology has proven to be very effective and useful as can be seen in the second part of the paper, where the original methodology is applied to the problematic of picking suitable components for a Business Intelligence solution. The paper concludes with the description of the final solution, which has proven that it satisfies the ISBN: 978-1-61804-126-5 347
requirements for such a solution. The next step may be to apply the methodology to different areas of open-source and revise it to better reflect other risks and benefits of open-source.. References: [1] Donovan A. Schneider, Practical considerations for real-time business intelligence, Proceedings of the 1st international conference on Business intelligence for the real-time enterprises, 2006. [2] Jia Xiaoliang, Tian Xitian, Zhang Zhenming, Process planning data mining technology based on object-oriented modeling, Proceedings of the third conference on IASTED International Conference: Advances in Computer Science and Technology, 2007. [3] Fernando Almeida, José Cruz, Open source unified communications: the new paradigm to cut costs and extend productivity, Proceedings of the Workshop on Open Source and Design of Communication, 2012. [4] Vijay K. Gurbani, Anita Garvert, James D. Herbsleb, A case study of open source tools and practices in a commercial setting, Proceedings of the fifth workshop on Open source software engineering, 2005. [5] Uzma Raja, Evelyn Barry, Investigating quality in large-scale Open Source Software, Proceedings of the fifth workshop on Open source software engineering, 2005. ISBN: 978-1-61804-126-5 348