STEP BY STEP DATA TRANSFORMATION

Transcription

1 STEP BY STEP DATA TRANSFORMATION Step-by-step explanation of what BI4Dynamics does in a process of delivering business intelligence content to End-users

2 1 ST. STEP: CREATING A STAGING DATABASE A staging database is a separate storage area created for the purpose of providing continuous access to application data. Data converters are written to access business data sources and then cleanse them to be written into the staging database. It is updated daily with new or changed data, from AX (Microsoft Dynamics Axapta) or NAV (Microsoft Dynamics Navision), so the whole BI (Business Intelligence)* process takes less time. Main function of staging database is to provide data access for BI process, but these structures are NOT optimized for analytics. This step is in comparison to other steps rather faster in BI process as it contains a copy function. Example: If you change a sales quote or post an invoice today, changes will be written to the staging database during the night. BI4Dynamics copies AX or NAV business data which include: All main tables (Customer, Vendor, Item, GL Account, Locations, etc.), Ledger entries (Item ledger, Customer ledger, Purchase ledger, GL ledger, etc.), Live and posted documents (Quotes, Orders, Posted Invoices, Production Order, etc.), Other tables (actually much more than 100, etc.). FOR BETTER UNDERSTANDING Can we analyze data in staging database? No. It makes no sense. Is this a part of data warehouse? No. Data warehouse is created in next step. But stage tables are in the same SQL database as data warehouse tables (dimension, facts).

3 2 ND. STEP: CREATING DATA WAREHOUSE A data warehouse is a place where data is stored for archiving, analysis, and security purposes. Usually a data warehouse is either a single computer or many computers (servers) connected together to create one giant computer system. Data consists of raw data or formatted data. It can be on various types of topics including the organization s sales, salaries, operational data, summaries of data including reports, copies of data, human resource data, inventory data, external data to provide simulations and analysis, etc. On this step of BI* (Business Intelligence) process, data is structured in a special BI (Business Intelligence) way, that is the key and most important part of BI* (Business Intelligence) solution. Data in AX (Microsoft Dynamics Axapta) or NAV (Microsoft Dynamics Navision) aren t made for analysis in matter of speed and analytical features and needs to be processed. Example: Receivables balance is calculated from customer ledger: for every customer and customer grouping, for all dates from first posting date until, usually 1 year in advance, on a daily basis, for every other dimension such as Salesperson, Department, Sales Area This step is the most time consuming in the BI process and depends on a quantity of Microsoft Dynamics data, like customers, documents, dates and other dimensions. But for end-users these pre-calculated data is a great benefit, as the analysis could be done immediately. New data will be created and written to new database inside SQL server*. The main piece of data warehouse is a fact table, which can be equal to super ledger in AX (Microsoft Dynamics Axapta) or NAV (Microsoft Dynamics Navision). Simplified, entries from AX or NAV are copied, processed and written to fact tables inside of data warehouse. In this step structures are prepared, but not fully optimized for analytics.

4 FOR BETTER UNDERSTANDING Can we see data in data warehouse? Yes. It is a part of SQL server database Can we analyze data in data warehouse? Yes, but the transformation process isn t finished yet. Data is also transformed for the OLAP cubes. Could we speed up calculations by calculating on a weekly or monthly basis? Yes, we could speed up calculations considerably, but these calculations will not be accurate and new measures that would be calculated, based on data warehouse measures would be even less accurate.

5 3 RD. STEP: CREATING OLAP CUBES OLAP stands for Online (OL) Analytical (A) Processing (P). An OLAP cube (for online analytical processing) is a data structure that allows fast analysis of data. It can also be defined as the capability of manipulating and analyzing data from multiple perspectives. An OLAP cube is a multidimensional database that is optimized for data warehouse. In OLAP cubes, data (measures) is being categorized by dimensions and often pre-summarized across dimensions to drastically improve query time over relational databases. Dimensions vs. Measures Dimensions are those things you want to track, such as companies, dates, locations and other items which attributes are often non-numerical. Common dimensions are companies - you can specify multiple, eight global dimensions, two additional currencies for measures, analyze specific posting through source and reason code and analyze specific documents through multiple measures or dimensions. Measures are the quantities you want to measure, common measures are average, minimum and maximum value or quantity of stock and other items that can be quantified numerically. In this step data is being copied from data warehouse and structured again. This step takes some time, but much less than data warehouse calculations. Example: Calculations in Receivables cube create new measures or dimensions: Average Receivable Balance, Average Due Days, Average Open Days, Receivables Coefficient (in days), Don t forget - this is a data warehouse transformation: Receivables Age (Before due, Overdue y-2y).

6 A new database is created, this time in SQL* Analysis Services, which is also a part of SQL server*. Analysis services database has usually the same name as data warehouse database. Main pieces are known as OLAP cubes and one OLAP cube equals one application area (sales cube = sales area). Some BI* solutions have only one OLAP cube, but we find this very confusing and difficult to use, because there are too many combinations available and many of them will not work by default. These structures are fully optimized for analytics. FOR BETTER UNDERSTANDING Can we connect OLAP cubes in BI4Dynamics? Yes, we can easily connect cubes into new virtual cubes, because all cubes came from the same data warehouse; Example: a new virtual cube named Logistics is created as connection of Sales, Purchase and Inventory. Is all BI4Dynamics data balanced? Yes. It is balanced as it comes from the same data warehouse. Example: Sales information in sales cube is balanced with sales information in receivables or General Ledger cube Are all OLAP cubes balanced? No, there is no guarantee. In the case where OLAP cubes were created directly from business data without being processed to data warehouse or if a solution has a bad technical design there is an option that data may not be balanced.

7 4 TH. STEP: ANALYSING In this step we connect to OLAP cube with any front-end BI* product. Most of our customers use Excel as a front end tool as it is difficult to beat the affordability and power of Windows based product such as Excel. Analytical part of Excel is called Pivot table*. As whole process is standardized, we also offer predefined Excel sheets for each application area. OLAP delivers the simplest form of analysis, allowing everyone to slice and dice interrelated subsets of data or "cubes" with a click of a mouse. You can analyze data using standard OLAP features such as page-by, pivot, sort, filter and drill up/down to flip through a series of report views. OLAP Analysis offers you primary access to its data warehouses. Of course, these sheets can later be changed according to end-users need. This predefined Excel sheets give you many ideas of how to analyze your data.

8 FOR BETTER UNDERSTANDING Can an end-user see a BI4Dynamics as a solution? No, as BI4Dynamics is a so called backend solution. BI4Dynamisc is a program, a set of scripts that run inside SQL server. You can see the results of BI4Dynamics, but not the program itself. You can manage BI4Dynamics through Bi4Dynamics interface. LEGEND How much time does BI process take? For a very small AX or NAV database it can take from few minutes to more than 12 hours for database that is few hundreds of GB big, and has many dimensions (customers, items, vendors, etc.). * BI (Business Intelligence) is the process of making better decisions in an environment that provides data and reporting that is timely, reliable, consistent and understandable in a useful format or presentation. * SQL (often referred to as Structured Query Language) is a programming language designed for managing data in relational database management systems (RDBMS). * SQL server is a Microsoft product used to manage and store information. Technically, SQL Server is a relational database management system (RDMS). * FACT TABLE is the central table in a star schema of a data warehouse. A fact table stores quantitative information for analysis. Fact table contains measurements, metrics or facts of business process. If your business process is Sales, then a measurement of this business process such as "monthly sales number" is captured in the Fact table. * PIVOT TABLE is a data summarization tool found in data visualization programs such as spread sheets (eg. in Microsoft Excel) or business intelligence software.