Business Intelligence for SUPRA. WHITE PAPER Cincom In-depth Analysis and Review

Transcription

1 Business Intelligence for A Technical Overview WHITE PAPER Cincom In-depth Analysis and Review SIMPLIFICATION THROUGH INNOVATION

2 Business Intelligence for A Technical Overview Table of Contents Complete Business Intelligence Solution Major Advantages Data Access for Application Program Interfaces Reports WHITE PAPER Cincom In-depth Analysis and Review Data Warehouse for ETL Extract Options ETL Steps ETL Jobs Warehouse Database Client Applications and Reports Analytics for Dimensional Data Developing Multi-dimensional Applications Application Programming Interfaces Visual OLAP Components Multi-dimensional Reports Microsoft Excel Support Glossary

3 1 Complete Business Intelligence Solution Business Intelligence for helps you to derive maximum value from your corporate data with high productivity and low risk. It provides an end-to-end solution for direct access to data, warehousing of data and multi-dimensional analysis of data. It allows you to quickly transform volumes of data from your transactional systems into decision-support and analytic information. Business Intelligence for is implemented in layers providing you with the flexibility to solve a number of business intelligence problems. You can directly access your host database using SQL-based tools and applications. You can reorganize your transactional data into a data warehouse to integrate with other data and provide historical information. You can also use multidimensional analysis to reveal product performance, profit trends or productivity comparisons. This is shown in the following image. Business Intelligence for Analytics for Data Warehouse for Data Access for Multidimensional Analysis Historical & cleansed data Direct SQL Access Major Advantages Business Intelligence for provides you with the following advantages. Optimized for PDM Accessing PDM data for decision support can be complex as implements specialized structures for navigation and performance optimization. Also, decisionsupport applications and tools are typically implemented on Windows, UNIX or Linux, and the data types used by the host PDM must be properly converted. Business Intelligence for understands the PDM data structures and how to optimize the extraction of host data. It understands the platform-specific data types and how to convert them to forms that are suitable for analysis. Cost-Effective Business Intelligence for is a cost-effective analytics solution for PDM. In comparison, third-party business intelligence products can be very expensive. They do not provide a solution that integrates with, so you also have the added cost of developing custom data access or data extraction procedures. Low Risk Business Intelligence for gives you the flexibility to try solutions and demonstrate the benefits without a large software investment. For example, a data-mart solution could be implemented where the analysis focuses on a single subject area such as customers, inventory or shopping baskets. The layered implementation is integrated using standard query languages, standard programming interfaces and standard protocols. This allows you to integrate with your existing tool sets and decision support software.

4 2 Data Access for Data Access for gives you direct access to your operational data through the SQL language. Direct data access provides you with the following advantages: You can access current information without the need to migrate data. Changes are not required to operational data or existing applications. Transactional access is available if needed. Third-party tools can access data using standard SQL and SQL-based interfaces. Data Access for provides a wide range of options for data access. Support for JDBC is provided enabling support for J2EE applications and tools. ODBC and OLEDB drivers are provided enabling support for.net applications and tools. A reporting framework is provided to allow you to design and deploy reports without the need to program. This is shown in the following image. Data Access for provides SQL support for PDM. This allows you to directly access PDM using standard relational APIs and access data using relational tools. In Data Access for, the metadata for the fields in a PDM file are mapped into an entity called a foreign table. Support is provided so that this foreign table can be treated as a standard SQL table by relational applications and tools. Data Access for optimizes the access and navigation of PDM files based on the information retrieved from the PDM directory (linkpaths, indices and control keys). Data Access for allows applications to access PDM files as standard SQL tables. To enable this, you define PDM files as foreign tables. An example is shown in the following figure. Data Access for Employee Foreign Table Empl-Dept Foreign Table EMPL PDM EMDE Jasper Reports J2EE Applications.Net Applications Department Foreign Table DEPT JDBC ODBC OLEDB Data Access Data Access for Foreign table definitions are somewhat like SQL view definitions. You define the fields to access in a PDM file, similar to the way you would define the columns to access in a table using an SQL view. In this example, you have an employee primary file called EMPL, a department primary file called DEPT and a related file called EMDE that relates multiple employees to a department. To access these, you would define a foreign table for each PDM file to Data Access for. In the foreign table definitions, you relate column names with field names like the following.

5 3 Employee employee-name. employee-numb. Empl-Dept emde-empl-numb.. emde-dept-numb... Department department-name.. department-numb.. EMPL EMPLROOT EMPLNAME EMPLNUMB EMDE EMDEEMPL EMDEDEPT DEPTLKEM DEPT DEPTROOT DEPTNAME DEPTNUMB DEPTLKEM You could then access these using SQL as in the following example: SELECT employee-name, employee-numb FROM Employee Application Program Interfaces Data Access for provides JDBC (Java Database Connectivity) to enable support for J2EE applications and tools. ODBC (Open Database Connectivity) and OLEDB (Object Linking and Embedding Database) are provided enabling support for.net applications and tools. JDBC is a standard Java interface for connecting to relational databases from J2EE applications. The supplied JDBC driver complies with the SQL92 Entry Level standard. ODBC is a standard interface for connecting to relational databases from.net applications. OLEDB is a set of Component Object Model (COM) interfaces providing applications with uniform access to data stored in diverse information sources. Like ODBC, OLEDB can be used for connecting to relational databases from.net applications. The ODBC driver and OLEDB Provider supplied with Data Access for comply with the SQL92 Entry Level standard. Reports To design and produce reports from query results, Data Access for supplies JasperReports (a reporting framework) to help you design and deploy reports without programming. Reports can be delivered in a variety of formats such as PDF, HTML, XLS, CSV and XML files. JasperReports formats data retrieved from a relational database through JDBC according to the report design defined in an XML file. The report design format provides a number of options such as: Using input parameters to drive report output. For example, input parameters can be used to change the columns used in an MDX query. Defining the layout of the report using sections and frames. Defining how data is grouped. Presenting data in charts such as bar charts, pie charts and line graphs. To create a report, you can develop the report design XML file or you can use ireport to visually develop the report. Using ireport, you can easily define data sources to use for relational data, how to format extracted fields, how to structure the report and how to present data in charts. As an example, in an inventory database, information may be kept for part costs, and various parts might be used by different projects. A report could be developed to organize part cost by project to evaluate project cost. The report could show: The parts used by each project For each part, the planned cost and current cost Using ireport, you can design a JasperReport that provides the summary of each part cost for a given project on a separate page. You could produce this in a PDF format as in the following example.

6 4 Data Warehouse for Data Warehouse for provides a data warehousing solution optimized to PDM. Building a data warehouse allows you to structure the data toward your business intelligence objectives. This provides a number of advantages over direct access to data. Data structures optimized for operational systems can be complex. A data warehouse allows you to restructure data into a schema that is easier for analysts to understand. Data can be combined from several sources into a unified relational schema. Data can be cleansed to meet the needs of analysis or changing standards. The warehouse data can be built incrementally to provide a historical basis for analysis. There may be less resource usage on operational systems as data is migrated to the data warehouse. Data Warehouse for provides the Extract- Transform-Load components to help you develop and automate the construction and updating of your warehouse data. Data can be directly extracted using Data Access for or it can be extracted using utilities. Data Warehouse for supplies a relational database with support for developing J2EE and.net applications. A reporting framework is also provided to allow you to design and deploy reports without the need to program. This is shown in the following image. Jasper Reports Data Warehouse for Data Warehouse Data Access for ETL J2EE and.net Apps ETL The Extract Transform Load (ETL) process provides the data basis for building your warehouse solution. The process extracts data from operational host systems and conforms data from disparate sources into a shape that is suitable for analysis. Data Warehouse for provides an ETL tool to help you develop and automate the construction and updating of your warehouse data. Facilities are provided to: Reshape your host data into the physical schema you design for the warehouse database. Conform heterogeneous data from multiple sources. Cleanse data to enforce your business rules. Handle large-volume initial loads and incremental updates. The ETL tool supports your design of the warehouse physical schema by providing the transformation steps commonly needed to reshape host data. In a data warehouse, the physical schema is optimized to provide efficient analysis. That is, the data is structured to allow fast analytic queries where the emphasis is on grouping and aggregating data. The host data used in your data warehouse solution may come from multiple, disparate data sources, which may not conform to the same business rules. And the extraction techniques may vary from one data source to another. Your data warehouse may be required to provide a cohesive data model that unifies the disparate data sources in your enterprise. Data warehouse designs refer to such an integrated model as having conformed dimensions and conformed measures. The ETL tool supports the integration of data by the following: Processes are provided to extract data directly from relational sources or from text files. For text files, a variety of format options are available. Data conforming processes are available to support your business rules. For example, look-up tables can be used to substitute values to conform to a standard. Data joining processes are provided to help match and join together different data sources. Host data may not have the accuracy needed for analysis. That is, some data sources may be incorrect, ambiguous or inconsistent. The design of your data warehouse may require the ETL process to cleanse the data by removing or correcting data that does not meet certain rules. The ETL tool provides processes that allow you to check various attributes of your data and filter out data or conform the data to a standard. Host Data

7 5 As the volume of data rises, the scalability of your ETL tool must meet your performance objectives. The Data Warehouse for ETL tool provides an architecture that scales with increasing volumes. This applies not only to the initial loading of your data warehouse but also to the ongoing incremental updates. Data Warehouse Extracting Data with Utilities Mainframe Extract Options A variety of options are available to extract host data for the warehouse. The Data Warehouse for the ETL component integrates with Data Access for to allow you to directly extract data, as shown in the following figure. ETL Tool (Transform and load) Text files Utilities (Extract) Data Warehouse Direct Extract ETL Tool Data Access for Data Access for allows the ETL tool to access PDM files as standard SQL tables. The ability to use the SQL language when extracting data from a host system can be a powerful advantage in the development of a data warehouse. During the extraction, data can be joined together using the navigation strategies of. Also, platform-specific data types are converted to ASCII text, which is suitable for loading a data warehouse. Also, data can be further converted and re-shaped using SQL in the extraction process. This can greatly simplify and speed up the ETL process. Data Warehouse for also provides a mainframe extract utility to unload PDM data into flat files. This can be useful for very large extracts where direct access to PDM across the network may not be practical. This is shown in the following figure: This provides some of the same advantages of extracting data with Data Access for, including the conversion of platform-specific data types to ASCII text and a limited join capability. However, the full power of the SQL language is not available for complex joins and data conversions. The ETL component also allows you to extract from other host sources and integrate the data with your data. You can: directly extract data from databases supporting JDBC, extract data from flat files allowing you to use data from any source that can unload data to comma-separated value files (CSV), and extract data from Excel spreadsheets and XML files.

8 6 ETL Steps A graphical tool is provided to help you construct and test transformation graphs. These graphs consist of connected steps categorized as input, transformation or output steps. For example, the following graph is used to show the loading of a client dimension table from two data sources. ETL Jobs A graphical tool is also available to help you construct and test ETL jobs. An ETL job is a set of connected job steps used to run scripts and transformation graphs, test for conditions and send alerts. For example, the following ETL job might be used to run the previously described transformation graph for loading the client dimension. In the example, the client dimension consists of a hierarchy of client and sales region information. The ETL process needs to join these two sources together and correct some problems in the process. 1. Client information is read from a client database table, and sales region information is read from a text file. 2. The client information is changed to conform to a different naming standard used for sales regions based on a corrections look-up file. 3. Sales region information is checked for the existence of a sales representative. Any error rows are sent to an error file. 4. The client information and sales region information is joined together based on the sales region name. 5. The joined information is written to the client dimension table in the warehouse database. Input steps are available to read text files, database tables, XML files or Excel files. A number of transformation steps are available for joining, filtering, grouping, merging, sorting, etc. Transformation steps are also available to help with common data warehouse needs such as surrogate keys and slowly changing dimensions. The sales region file is transferred from another system using FTP before running the transformation to load the client dimension. Also, a test is made to see if the transformation produced an error file. If an error file was produced, or if any of the previous steps fail, an alert is sent. Job steps are provided to execute SQL statements and run scripts, transformation graphs, FTP files, etc. You can also test for various conditions. Warehouse Database Data Warehouse for embeds PostgreSQL as the warehouse database. PostgreSQL is an open source database with a strong reputation for reliability, data integrity and performance. PostgreSQL s ease of administration and deployment make it an ideal choice for an embedded database server. However, Data Warehouse for does not require PostgreSQL and allows you the flexibility of using other relational databases. Client Applications and Reports As with Data Access for, you can use the JasperReports framework to produce reports from your warehouse data without programming. You can also use JDBC to enable support for J2EE applications and tools, and ODBC and OLEDB to enable support for.net applications and tools.

9 7 Analytics for Analytics for enables multi-dimensional analysis of information from your operational databases. This information is expressed in business measures that can reveal product performance, profit trends or productivity comparisons. Multi-dimensional analysis not only allows you to quickly reveal business performance and trends, it also allows you to explore new analysis areas. Ad hoc analysis can let you reveal trends and performance measures that would remain hidden if traditional querying and reporting were used. Analytics for builds on advantages provided by Data Warehouse for and Data Access for. An analytic server is provided allowing you to organize data using the measures and dimensions that are important to business intelligence objectives. Components are provided to allow the easy development of analytic applications. A reporting framework is provided to allow you to design and produce reports using analytic data. Also, tools are provided to help you develop analytic queries and to perform ad hoc analysis of data. This is shown in the following image. Analytic Reports Analytic Applications Analytic Tools Dimensional Data In multi-dimensional analysis, the data items to be examined are referred to as measures. The measures are described or categorized by dimensions. These are usually organized into a particular domain of inquiry as a sales performance or client purchases. The basic unit of organizing and storing the dimensions and the measures they contain is an OLAP cube. As an example, you might be interested in an inventory analysis measuring inventory quantity by location. This can be pictured as a two-dimensional cube as shown in the following: Rudder Wing flap Engine mount Prop Wing support Chicago, Bldg 1 New York, Bldg 3 Phoenix, Bldg The cells of the cube contain the Quantity measure while the rows and columns represent the part and location dimensions. You could continue to define dimensions for the cube. For example, you may want to measure inventory levels over time and define a time dimension. This can be pictured as a three-dimensional cube: Time Analytics for Analysis Chicago, Bldg 1 New York, Bldg 3 Phoenix, Bldg 2 Rudder Wing flap Engine mount Prop Wing support Data Warehouse for Data Access for

10 8 You define a cube to the analytic server by using a cube schema definition. This schema defines the measures and dimensions and how they are stored in the warehouse database. To create and maintain these definitions, a Cube Schema Builder tool is provided. For example, the Cube Schema Builder might display the definition of the above inventory cube as the following: The Inventory cube schema contains the Quantity measure and the dimensions: Time, Part and Location. The definition of the Quantity measure is shown. The Quantity measure is aggregated over dimensions as a sum. The measure is stored in a fact table in the column qty_on_hand. Also, formatting options are available to format the measure in analytic query results. The analytic server uses the cube schema definition to provide an organization for storing data in a memory cache. This is shown in the following diagram: The cube schema definitions allow the analytic server to map dimensions and measures in the cube to tables and columns in the warehouse database. The cube schema definitions also provide a cache organization for holding computed aggregations in memory so subsequent queries can access values without going to the warehouse database. In an analytic solution, the performance of grouping and aggregating data is critical to success. For very large volumes of data, the aggregates should be pre-computed when the warehouse database is loaded. Special aggregate tables can be constructed so that aggregates are stored and do not need to be computed by the analytic server. For aggregates that must be computed by the analytic server, the cache is used to hold results. The analytic server has a number of features for developing your logical data model: Multiple hierarchies can be defined for dimensions. For example, the Time dimension could be defined as a calendar hierarchy: year, month and week. It can also be defined as a fiscal calendar: fiscal year or fiscal quarter. Cubes can be mapped to warehouse databases with a star schema or a snowflake schema. Measures can be created with user-defined formulas. Aggregate tables in the warehouse database schema can be used to improve performance. Security can be defined for users to control access to analytic data. Application Warehouse cache Analytic WH Database Cube schema

11 9 Developing Multi-dimensional Applications Multi-dimensional applications allow analysis of multidimensional data where information is presented in a form that can immediately be understood by users. This is usually in a tabular or graphical form where more detailed information can be obtained by drill-down or breakdown lists. Applications are typically time-oriented to reveal past trends and patterns. These applications are often referred to as Online Analytical Processing (OLAP) applications. OLAP applications use the Multi-dimensional Expression language (MDX) to manipulate multi-dimensional data. MDX is oriented to analysis queries as it allows multiple dimensions, hierarchies of dimensions and aggregation of measures. Using this language, queries can be requested from an analytic server to provide measures that are organized and summarized by dimensions. You can develop analytic web applications using application programming interfaces to communicate MDX requests to the analytic server. You can also use GUI components that accept MDX queries and render visual components such as charts and pivot tables. For producing reports from MDX query results, you can use reporting frameworks to design and produce OLAP reports. You can also use multi-dimensional query tools to explore dimensions and measures and to design and test MDX queries. Multi-dimensional Applications and Tools Analytic Reports Application Programming Interfaces Analytics for provides several application programming interfaces to communicate requests to the analytic server. Applications can communicate MDX requests to an analytic server using XML for Analysis. XML for Analysis (XMLA) is a standard that allows clients to talk to analytic servers using Web Services. This allows opens access from a variety of platforms and languages to multi-dimensional data servers. Client requestor components are also supplied so that client applications can use XMLA without a detailed understanding of the Web Service protocol and technologies. This is shown in the following diagram: Userwritten OLAP Application XMLA Requestor soap Application XMLA Responder Analytic Java programs can also use the Java for OLAP (JOLAP) programming interface. JOLAP is a J2EE objectedoriented interface to analytic servers. Data Warehouse for provides an implementation of the interfaces. For example, you might use JOLAP to implement an OLAP servlet application as shown in the following diagram. Analytic Web Applications MDX Analytic Queries Application Analytic Web Browser http Servlet App Application JOLAP Analytic Analytic Tools

12 1 Visual OLAP Components Components are also provided to visually present analytic query results in the development of Java Pages applications. These include a: Navigator component to explore the dimensions and measures defined in a cube. Pivot table component to display analytic query results in a tabular form. The component allows slicing, dicing, drilling down and rolling up. Chart component to display analytic query results in a variety of formats. These include bar charts, line graphs, pie charts, etc. A Java Pages Tag library is provided to help you easily construct server pages to render visual components and to change options for components. As an example, your data warehouse may provide an inventory analysis cube. The cube could provide measurements for inventory quantities and inventory costs. The dimensions that are important for the analysis could include part names, part locations and year time periods. To understand the use of part storage space at different locations, you could design a query that shows: Part quantity measurement where the aggregation is the maximum Part quantity broken down by the location dimension Part quantity that is further broken down by time periods You could use the visual components to construct a Java Page to request your query and present a bar chart and pivot table as in the following image. For users who are familiar with OLAP technology, a navigator component can be used to change measures and add or remove dimensions. Also, other visual components can be presented to change the properties of charts and pivot tables. Toolbars can be presented to choose visual components. For example, you could use the visual components to construct a Java Page to present a navigator component and other options as a toolbar. The navigator can be used to add or remove dimensions to the inventory space analysis query and present a bar chart and pivot table as in the following image.

13 11 Multi-dimensional Reports To design and produce reports from MDX query results, you can use JasperReports. This allows you to use the dimensions and measures from analytic queries to produce reports in a variety of formats. You can also use ireport to help you visually design JasperReports. Using ireport, you can easily define data sources to use for analytic data, how to format extracted fields, how to structure the report and how to present data in charts. Microsoft Excel Support Microsoft Excel is an established tool for business analysis and reporting. You can use Microsoft Excel with Analytics for through third-party integration software. Excel allows you to dynamically create Excel Pivot Tables and charts using drag-and-drop operations. This allows you to explore and perform ad hoc analysis without knowledge of multi-dimensional query languages. Drill-down and summary operations are provided to further explore and understand trends. For example, in an inventory analysis, you might be interested in comparing the current and planned part costs for a project. You would also like to see the cost comparisons over time. With Excel, you can design a clustered column chart dragging measures and dimensions from a field list to the chart. The chart shows summary costs for past years and monthly costs for the current year.

14 12 Glossary Aggregate See Dimension Hierarchy. Cube A cube is an array of data cells arranged by dimensions. The data cells contain the measures or summary of the measures. For example, a spreadsheet is a twodimensional array with the data cells arranged by rows and columns where the rows and columns represent dimensions. Dimension A dimension is a structural attribute of a cube that is a list of members, all of which are of a similar type in the user's perception of the data. For example, all months, quarters, years, etc., make up a time dimension; likewise all cities, regions, countries, etc., make up a geography dimension. A dimension acts as an index for identifying values within a multi-dimensional array. If one member of the dimension is selected, then the remaining dimensions in which a range of members (or all members) are selected defines a sub-cube. If all but two dimensions have a single member selected, the remaining two dimensions define a spreadsheet (or a "slice" or a "page"). If all dimensions have a single member selected, then a single cell is defined. Dimensions offer a very concise, intuitive way of organizing and selecting data for retrieval, exploration and analysis. Dimension Hierarchy Dimensions can be organized with parent-child relationships. The parent represents an aggregation of its children. For example, a time dimension might be organized in a hierarchy of year and month. The data for year might be an aggregation of its children (months). This aggregation is typically a sum but can be more complex such as an average. The aggregation is sometimes referred to as a roll-up of data from children. Drill Down/Up Drilling down or up is a specific analytical technique whereby the user navigates among levels of data ranging from the most summarized (up) to the most detailed (down). The drilling paths may be defined by the hierarchies within dimensions or other relationships that may be dynamic within or between dimensions. For example, when viewing sales data for North America, a drill-down operation in the Region dimension would then display Canada, the eastern United States and the western United States. A further drill-down on Canada might display Toronto, Vancouver, Montreal, etc. Fact See Measure. Measure Measures, also referred to as facts, are data to be analyzed or examined. Measures are numeric and are usually additive. Multi-dimensional Array See Cube. ODBC ODBC, short for Open Database Connectivity, is a database access method for SQL databases. OLAP Online Analytical Processing designates a category of applications and technologies that allow the management of multi-dimensional data, with the goal of analysis. This is often used in sales and marketing analysis to study the volume of sales by products, location, time, etc. It is also used in decision support to forecast changes in income, expense and profit and in quality of service analysis. OLEDB OLEDB (Object Linking and Embedding Database) is an API designed by Microsoft for accessing different types of data stores in a uniform manner. It is a set of interfaces implemented using the Component Object Model (COM); it is otherwise unrelated to OLE. It was designed as a higher-level replacement for, and successor to, ODBC, extending its feature set to support a wider variety of nonrelational databases, such as object databases and spreadsheets that do not necessarily implement SQL. OLTP OLTP (online transaction processing) is a classification of programs that manage transactions. Data entry and data retrieval applications are examples.

15 13 Pivot Table A pivot table is a tool that allows you to visualize and explore the results of an analytic query. The results are presented as a spreadsheet of measures organized by rows and columns that represent dimensions. Pivot tables allow slicing and dicing, rotating and roll-up operations. Roll Up See Dimension Hierarchy. Rotate To change the dimensional orientation of a report or page display. For example, rotating may consist of swapping the rows and columns, moving one of the row dimensions into the column dimension, or swapping an off-spreadsheet dimension with one of the dimensions in the page display (either to become one of the new rows or columns), etc. A specific example of the first case would be taking a report that has Time across (the columns) and Products down (the rows) and rotating it into a report that has Product across and Time down. An example of the second case would be to change a report that has Measures and Products down and Time across into a report with Measures down and Time over Products across. An example of the third case would be taking a report that has Time across and Product down and changing it into a report that has Time across and Geography down. Star Schema A star schema is an organization of tables in a relational database optimized for OLAP. In the center is a fact table, whose columns contain the multi-dimensional measures. The branches of the star consist of dimension tables where each table contains the hierarchy information for a dimension. The dimension tables are linked to the fact table through foreign key relationships. Surrogate Keys Surrogate keys are primary keys that are substituted for the natural key of a table. A surrogate key is usually an integer and can help in the performance and updating of a warehouse database. Slice and Dice The user-initiated process of navigating by calling for page displays interactively, through the specification of slices via rotations and drill down/up. Slowly Changing Dimensions Slowly changing dimensions are dimensions that change over time. A variety of techniques can be used to update dimensions in a warehouse database and to record versioning information for changes. Snowflake Schema A snowflake schema is a variation of a star schema where the information for a dimension is normalized into multiple tables. This is usually used for very large dimensions.

16 Cincom, the Quadrant Logo,, and Simplification Through Innovation are registered trademarks of Cincom Systems, Inc. All other trademarks belong to their respective companies. 27 Cincom Systems, Inc. FORM DB /7 Printed in U.S.A. All Rights Reserved World Headquarters Cincinnati, OH USA US 1-8-2CINCOM Fax International