Design a Distributed Data Warehousing based ROLAP with Materialized Views

Transcription

1 Research Article Internatinal Jurnal f Current Engineering and Technlgy ISSN INPRESSCO. All Rights Reserved. Available at Design a Distributed Data Warehusing based ROLAP with Materialized Views Akeela M. Al-Atrshi Ȧ* and Fared Mahmd Abdullah Ḃ Ȧ Department f Cmputer Science, Faculty f Science, University f Dhuk Ḃ Department f Cmputer Science, Institute f Science Dhuk Accepted 25 Nvember 2013, Available nline 01 December 2013, Vl.3, N.5 (December 2013) Abstract In tday s highly cmpetitive business, data became strategic resurces. Business cmpanies dealing with a huge and grwing amunts f data in different database frmats. whether these cmpanies need t understand the effectiveness f their marketing effrts and quickly maintain the large vlumes f data created each day. These challenges require a welldefined database system and data warehuse that can bring tgether disparate data with different dimensinality and granularity. The aim f this study is t design distributed data warehuse based n relatinal OLAP with materialize views fr the purpses f analysis. In rder t achieve this bjective, the current study adpted a case study apprach, including the applicatin f the prpsed system in General Cmpany fr the pharmaceutical industry. The study came ut a set f cnclusins f the mst imprtant is Data warehuse cnsidered the mst suited apprach f Infrmatin Integratin t build a Decisin Supprt System fr the business intelligence scenari. The implementatin shws that the system prpsed prvides the effective tls t retrieve data thrugh the: generate reprts r query the data n line, and building data cube. Keywrds: Distributed data warehuse, Materialize views, ROLAP, Rllup and Drill dwn, Slice and Dice peratins. 1. Intrductin 1 A data warehuse is a large repsitry f histrical data that can be integrated fr decisin supprt. The use f a data warehuse is markedly different frm the use f peratinal systems. Operatinal database cntain the data required fr the day-t-day peratins f an rganizatin. This peratinal data tends t change quickly and cnstantly. The table sizes in peratinal systems are kept manageably small by peridically purging ld data. The data warehuse, by cntrast, peridically receives histrical data in batches, and grws ver time [Griesemer,2009; Malinwski,2008]. OLAP is a term that describes a technlgy that uses a multi-dimensinal view f aggregate data t prvide quick access t strategic infrmatin fr the purpses f advanced analysis. OLAP supprts queries and data analysis n aggregated databases built in data warehuses. It is a system fr cllecting, managing, prcessing and presenting multidimensinal data fr analysis and management purpses. There are tw main implementatin methds t supprt OLAP applicatins: relatinal OLAP(ROLAP) and multidimensinal OLAP (MOLAP) [Wang,2006;Hwan,2007]. When a Business is distributed gegraphically r ver multiple differing prduct lines, infrmatin is needed at the level f crprate management are met by a central * Crrespnding authr: Prf. Dr. Akeela M. Al-Atrshi data warehuse where infrmatin is gathered. But there is als a need fr data at each lcatin f the rganizatin r need a separate data warehuse. Fr that, a distributed data warehuse is required at the level f the factry [Inmn,2005]. In the same cntext, the main prblem f the current research revlves arund the absence f applicatin f infrmatin technlgy represented by the peratinal databases and data warehuses as well as t retain the data fr lng times in industry. Als, its well-knwn amng sftware engineers that MOLAP applicatins require a lt f memry as the cube size increases, such as increasing in the number f dimensins, r increasing in the cardinality f the dimensins, r increasing in the amunt f data r a cmbinatin f sme r all these aspects. Therefre the study adpted ROLAP with materialized views t query the relatinal database especially that the applied envirnment f this study deals with large-scale database. Similarly, That fr, the Main bjective f this research is t design a Distributed data warehuse based n relatinal OLAP with materialized views. S, in rder t achieve the main bjective, there are sme sub bjective which have t be dne: Designing an peratinal database t stre data fr all the daily activities f the cmpany and becme the mst imprtant internal surces fr the data warehuse. 1838

2 Akeela M. Al-Atrshi et al Internatinal Jurnal f Current Engineering and Technlgy, Vl.3, N.5 (December 2013) Using ROLAP with materialized view t speed up data retrieval and making the time f query clse t the (Near real time). Reducing strage space, using (Views in the stage area) instead f string data in tables. 2. Distributed data warehuse cncepts The crnerstne f all Enterprise activities is infrmatin prcessing. This includes data cllectin, strage, transprtatin, manipulatin, and retrieval. The imprtance f gd infrmatin can be thught f as the difference in value between right decisins and wrng decisins, where decisins are based n that infrmatin. The larger the difference between right and wrng decisins, the greater the imprtance f having gd infrmatin [ Thmsen,2002]. Since the early 1990s, the data warehuse has becme the fundatin f advanced decisin supprt applicatins. Using sphisticated nline analytical prcessing (OLAP) and data mining tls, sme crpratins are able t explit insights gained frm their data warehuse t significantly increase sales, reduce csts, and ffer new and better prducts r services [Hwan,2007; Oracle,2009]. A. OLAP appraches OLAP perfrms multidimensinal analysis f enterprise data and prvides the capabilities fr cmplex calculatins, trend analysis and very sphisticated data mdeling. In additin, OLAP enables end-users t perfrm ad hc analysis f data in multiple dimensins, thereby prviding the insight and understanding they need fr better decisin making. An OLAP structure created frm the peratinal data is called an OLAP cube. OLAP cubes are data prcessing units cnsisting f the fact and the dimensins frm the database[ Naman,2000 ]. MOLAP is a multidimensinal data strage frmat that prvides a high perfrmance. The data feeding the cubes is kept with MOLAP n the OLAP server as a multidimensinal database. MOLAP is a specifically ptimized slutin fr multidimensinal data queries and due t this cause-andeffect it gives the best verall query perfrmance. MOLAP is s cnvenient fr small t medium-sized data sets. MOLAP needs cpying all data and cnverts its frmat cnveniently t fit the multidimensinal data stre. [Shia,2006]. The ROLAP prvides OLAP functinality by using relatinal databases apprach prvides the full analytical functinality f OLAP while maintaining penness and scalability.it can handle large amunts f data since n precnslidatin is dne. Of curse, query perfrmance in ROLAP systems may nt be as efficient as in MDDB systems since cmputatin f aggregates frm the base data has t be dne at query time. But perfrmance can be imprved by selectively precmputing certain aggregatins and string them in the warehuse using materialized views. Indexes can be built n these materialized views t imprve query perfrmance [ Ault, 2003, Wang, 2006]. B. Data Staging area (ETL) The extracted data cming frm several disparate surces needs t be changed, cnverted, and made ready in a frmat that is suitable t be stred fr querying and analysis. Three majr functins need t be perfrmed fr getting the data ready. These three majr functins f extractin, transfrmatin, and preparatin fr lading take place in a staging area [Pnniah, 2007]. Data is extracted frm the peratinal systems by extractin rutines. The extracted data is then cnverted int an intermediate schema and placed in a staging area. The surce data accumulated in the staging area is subjected t data cleansing, transfrmatin t the intermediate schema, and data aggregatin and finally laded int fact tables in the data warehuse [ Bhansali, 2010; Kimball, 2004]. Transfrm applicatin perfrms all data mdificatins t the Surce Data necessary t cnfrm it t the rules, layut, and frmat f a data warehuse. The transfrmed data is als captured n a platfrm that is cntrlled by the ETL applicatin. The pst-transfrm data is als captured n a Staging Platfrm r Staging Envirnment. In this pst-transfrm state, hwever, the staged data is called Lad Data. A Lad applicatin bridges the gap between the ETL and Staging Platfrms and the data warehuse platfrm. A Lad applicatin reads the Lad Data and perfrms the necessary inserts, updates, and deletes t a data warehuse. When the Lad applicatin has finished, the ETL applicatin has cmpleted. The missin f the ETL team at the highest level is t build the back rm f the data warehuse [Kimball, 2004; Silvers,2008]. C. Distributed Data Warehuses A distributed database system is ne in which data is stred at multiple ndes but all data is accessible t any authrized user. The three types f DDW are as fllws: [Inmn,2005]. Business is distributed gegraphically r ver multiple, differing prduct lines. In this case, there is what can be called a lcal data warehuse and a glbal data warehuse. The lcal data warehuse represents data and prcessing at a remte site, and the glbal data warehuse represents that part f the business that is integrated acrss the business. The data warehuse envirnment will hld a lt f data, and the vlume f data will be distributed ver multiple prcessrs. Lgically there is a single data warehuse, but physically there are many data warehuses that are all tightly related but reside n separate prcessrs. This cnfiguratin can be called the technlgically distributed data warehuse. The data warehuse envirnment grws up in an uncrdinated manner first ne data warehuse appears, then anther. The lack f crdinatin f the grwth f the different data warehuses is usually a result f plitical and rganizatinal differences. This case can be called the independently evlving distributed data warehuse. 1839

3 Akeela M. Al-Atrshi et al Internatinal Jurnal f Current Engineering and Technlgy, Vl.3, N.5 (December 2013) 3. Cnceptual Design f Prpsed System The Architecture f prpsed system is designed based n three cmplementary parts f physical design as illustrated in figure (1),. This study has fcused n the state cmpany fr drug industries and medical appliances in Ninavah/Iraq as a field fr applying the prpsed system( 2 ).Fr that and based n the Inmn strategies, prpsed system is designed s that it includes a data warehuse n tw levels fr each factry(sub), the ther at the firm level includes infrmatin n all cmpany units(main). The key cmpnents f the designed system that deals with hmgeneus databases, s all cmputers that are cnnected t the netwrk use the same cpy f the perating system. The prpsed system has a great ability t execute transactins in lcal sites and glbally within the netwrk: Lcal transactin t transfrm data frm peratinal database t Lcal Data warehuse. Glbal transactin t transfrm data frm Lcal Data warehuse t glbal data warehuse. cllecting systems that perates in a factry that has single data warehuse and make them deal with a single data warehuse called Bttm Up Design, which the design begins with base and ends with tp. 1. Building Operatinal Database Database in data warehuse represents a critical part in the architecture f the prpsed system in which it is specialized t stre large quantities f data up t (Terabyte) and fr lng perids f time which may exceed 10 years[ Lane,2005]. The peratinal database are designed accrding t the rules f the three frms f nrmalizatin t eliminate duplicative clumns using Oracle 11gR2, JDevelper 11g t execute SQL and t create tables, prcedures, functins and triggers and (PHP) t design web page and cnnect with Oracle. 2. Building Lcal Data Warehuse The lgical design prcess begins with the analysis f the data surce in the warehuse in rder t identify clearly the dimensins and the facts cntained in the star schema as data mdel fr data warehuse, it is based n this analysis and accrding t the nature f the supprted data the researcher uses ne fact table and (7) dimensin tables figure(3) explain Star schema f Lcal site as fllws: Figure1 Cnceptual structure f prpsed system Figure 3 Star schema f Lcal site Figure 2 Cmpnents f prpsed system at level f each side Mrever, the figure (2) indicates that there is a type f flexibility in adding the sites t the netwrk and the cmputers t the sites linked t Main Server thrugh the HUB because the system is applied t the LAN netwrk. Therefre, the designed system cnsists f tw separated systems linked thrugh a netwrk, the first level is specified fr sites while the secnd ne is specified fr main server. It shuld be nted that the prcess f 2 Researcher cnducted interviews with manager and related engineers f the state cmpany fr drug industries and medical appliances in Ninavah / Iraq A. Fact table : Represents the axis f the star schema, it s ne f the main tables in the database, this table includes three types f clumns, the first type is a freign key that is used t cnnect the fact table with the seven dimensin tables, by using (THR_ID, SALES_ID) t reset (Tw Views), and the thers are specialized (sales price and quantity, prduct cst and quantity) that represent as measures. B. Dimensin tables: ften cmpsed f ne r mre hierarchies that categrize data, the prpsed system deals with the seven dimensins cnnected with the fact table thrugh the freign keys. These tables include data f the prducts that the cmpany prduced hierarchies are lgical 1840

4 Akeela M. Al-Atrshi et al Internatinal Jurnal f Current Engineering and Technlgy, Vl.3, N.5 (December 2013) structures which use the rdered levels as a mean f rganizing data. A hierarchy can be used t define the data aggregatin. Fr example, a hierarchy f Dates_Dim will aggregate data frm the mnth level t the quarter level and t the year level. A hierarchy can als be used t define a navigatinal drill path and t establish a family structure. Determining the hierarchical sequence helps t shw a gradual visin and an analysis f the data frm several perspectives, and a cmparisn between the different levels. C. Designing the staging area cntents (ETL) : the ETL system is the essential part f the data warehuse the staging area is the interim lcatin fr the data between the surce system and the target database structure. The staging area will hld the data extracted directly frm the peratinal database, which will determine the physical structure f staging area [Malinwski,2008 ]. In the cntext f the current thesis, the researcher writes a set f (Prcedures, Functins and Cursres), using (PL/SQL), implementatin f these prcesses, are as fllws 4. The Implementatin f the Prpsed System In general, the prpsed system has been used as decisin supprt system, helps the users at the bth structure levels t build his SQL as fllw: A. On the level f each plant (Ninewa drugs Factry ) : Users can querying the database fr each plant and the Star schema using three tls : Querying Star schema using SQL t present a cmparisn f sales quantity f prduct ( Flu-ut) in January f (2010,2012) SELECT DATES_DIM.YEAR, DATES_DIM.QUARTER, CUSTOMERS_DIM.SECTOR, PRODUCTS_DIM.CATEGORY_NAME, SUM(SALES_QTY) SALES_QTY FROM THR_SALES_FACT_TBL,DATES_DIM,CUSTOMERS_DIM,PRODUCTS_DIM WHERE THR_SALES_FACT_TBL.DATE_ID = DATES_DIM.DATE_ID AND THR_SALES_FACT_TBL.CUST_ID=CUSTOMERS_DIM.CUSTOMER_ID AND THR_SALES_FACT_TBL.PROD_ID=PRODUCTS_DIM.PRODUCT_ID AND DATES_DIM.YEAR = 2010 AND DATES_DIM.QUARTER =1 AND CUSTOMERS_DIM.SECTOR = 'Private' AND PRODUCTS_DIM.CATEGORY_NAME = 'Ampules' GROUP BY(DATES_DIM.YEAR, DATES_DIM.QUARTER, CUSTOMERS_DIM.SECTOR,PRODUCTS_DIM.CATEGORY_NAME) 1. Extractin Create tw views in staging area, the first (Salescllect) is specified t extract the data frm five tables (Sales_tbl, Sales_details_tbl, PRODUCT_TBL, QUALITY_TBL, custmers_tbl) that relevant with sales. While the secnd (Thrughputcllect) is specified t extract data frm tables (THROUGHPUT, THROUGHPUT_DETAILS and PRODUCT_TBL, QUALITY_TBL, custmers_tbl). The main reasn f using views instead f the tables is t reduce the space and d nt need t full scan f the table t validate what data still nt extract. 2. Transfrmatin and Lading Using Frm : Select dimensins and Measures then click Preview Data buttn as shwn in the figure (5) This prcess is perfrmed t transfer data frm bth views (Thrughputcllect and Salescllect) in the same way t the star schema in a data warehuse by using the prcedure (DateIdExist) t extract the date f the sale and prductin int the units (day, week, mnth, quarter, year) as fllws: Extract(day frm dt), t_char(t_date(dt), W ) Extract(mnth frm dt), t_char(dt, MON ) T_char(dt_ Q1 ), Extract(year frm dt) T btain the transfrmatin and Lading peratin effectively, the researcher uses the explicit and implicit cursrs within the set f prcedures and functins as fllws: Extract data frm the views and stres in the cursrs. The tw main prcedures (CllectingdatafrmSales, CllectingdatafrmThrughput) call a set f sub prcedures t test the related dimensin tables, if the data extract frm the view presented in each dimensin table. If the data is nt existent, then a set f functins will call t fetch each rw t the related dimensin tables fr the sales and prductin. Figure 5 Brwsing Cube The result 1841

5 Akeela M. Al-Atrshi et al Internatinal Jurnal f Current Engineering and Technlgy, Vl.3, N.5 (December 2013) Using Analytical Wrkspace Manager(AWM) Rllup and Drill_dwn peratins : Using SQL Rllup Slice and Dice peratins: Fr slicing n Diseases dimensin, in Brwse Cube frm check Diseases level f diseases dimensin and the measure Sales Quantity then click Preview Data buttn, as shwn in figure(6 ). Fr dicing n Diseases.Type_Of_Diseases and Custmers.Sectr, in Brwse Cube frm check Diseases level f diseases dimensin, Sectr level f custmers dimensin and the measure Sales Quantity then click Preview Data buttn, as shwn in figure(7 ). SELECT DISTINCT D7_YEAR_A1, D5_CATEGORY_NAME, SUM(SUM_SALES_QTY) SALES_QTY FROM NDI_CUBE WHERE D7_YEAR_A1 = '2010' AND CHANNELS_D1 LIKE AND CUSTOMERS_D2 LIKE AND LOCATION_D4 LIKE AND D5_CATEGORY_NAME LIKE 'Ointments' AND QUALITIES_D6 LIKE AND DISEASES_D3 LIKE GROUP BY (sys_gid, D7_YEAR_A1, CHANNELS_D1, CUSTOMERS_D2, LOCATION_D4, D5_CATEGORY_NAME, QUALITIES_D6, DISEASES_D3) SELECT DISTINCT D7_YEAR_A1, D5_CATEGORY_NAME, D5_PRODUCT_NAME, SUM(SUM_SALES_QTY) SALES_QTY FROM NDI_CUBE WHERE D7_YEAR_A1 = '2010' AND CHANNELS_D1 LIKE AND CUSTOMERS_D2 LIKE AND LOCATION_D4 LIKE AND D5_CATEGORY_NAME LIKE 'Ointments' AND D5_PRODUCT_NAME!= ' ' AND QUALITIES_D6 LIKE AND DISEASES_D3 LIKE GROUP BY (sys_gid, D7_YEAR_A1, CHANNELS_D1, CUSTOMERS_D2, LOCATION_D4, D5_CATEGORY_NAME, QUALITIES_D6, DISEASES_D3) Figure 7 Dice n Custmers.Sectr and Diseases.type_f_diseases A. On Main server :The study prvide tw apprach t create glbal cube : First, create materialized view CREATE MATERIALIZED VIEW new_cube NOCACHE NOPARALLEL BUILD IMMEDIATE USING NO INDEX REFRESH ON DEMAND COMPLETE ENABLE QUERY REWRITE AS SELECT DATES_DIM.YEAR,DATES_DIM.QUARTER,DATES_DIM.MONTH,DATES_DIM.DAY, PRODUCTS_DIM.CATEGORY_NAME,PRODUCTS_DIM.PRODUC T_NAME, CUSTOMERS_DIM.SECTOR,CUSTOMERS_DIM.CUSTOMER_NA ME, SUM(SALES_QTY) SALES_QTY FROM DATES_DIM,PRODUCTS_DIM,CUSTOMERS_DIM,THR_SALES_FACT_TBL WHERE YEAR = 2011 AND MONTH = 1 AND SECTOR = 'Private' AND CATEGORY_NAME = 'Vials' AND DATES_DIM.DATE_ID = THR_SALES_FACT_TBL.DATE_ID AND PRODUCTS_DIM.PRODUCT_ID = THR_SALES_FACT_TBL.PROD_ID AND CUSTOMERS_DIM.CUSTOMER_ID = THR_SALES_FACT_TBL.CUST_ID GROUP BY ( DATES_DIM.YEAR,DATES_DIM.QUARTER,DATES_DIM.MONTH,DATES_DIM.DAY, PRODUCTS_DIM.CATEGORY_NAME,PRODUCTS_DIM.PRODUC T_NAME, CUSTOMERS_DIM.SECTOR,CUSTOMERS_DIM.CUSTOMER_NA ME) Secnd : create cube using the fllwing PL/SQL cde Figure 6 Slice n Diseases.type_f_diseases set serverutput n; declare MyCube varchar2(50); 1842

6 Akeela M. Al-Atrshi et al Internatinal Jurnal f Current Engineering and Technlgy, Vl.3, N.5 (December 2013) begin MyCube :=dbms_cube.create_mview('admin','new_cube', 'build=immediate' ); end; Rllup and Drill dwn glbal cube using AWM Sum f sales quantity at the year 2010 fr categries (Cancer_Capsules, Cancer_Tablets, Capsules, Tablets) in figures(8,9). Cnclusin Data warehuse cnsidered the mst suited apprach f Infrmatin Integratin t build a Decisin Supprt System fr the business intelligence scenari. Users able t see trends and patterns thrugh measures frm different perspectives, that can help t make better strategic decisins. The implementatin shws that the system prpsed prvides the effective tls t retrieve data thrugh the: generate reprts r query the data n line, and building cube. Als the results cncludes that using views instead f tables in staging area reduce the transfer time and memry space when the view size equals t zer after lading the data. This embdies ne f the sub bjectives f current thesis t make the time f transfer t near Real time, substantially reduce the respnse time f a query. That fr, the prpsed system is characterized by high flexibility in terms f the ability t add ther dimensins t star schema and add ther measures t the fact table. References Figure 8 ( Rllup) Decisin maker nted that the amunt f sales fr the department f tablets mre than the amunt f sales in the rest f the categries. Sum f sales quantity at the year 2010 fr categries (Cancer_Capsules, Cancer_Tablets, Capsules, Tablets) and drill_dwn year 2010 t Quarters.0 Figure 9 Rllup and Drill dwn ( Drill dwn ) Ault M. (2003), Oracle Data Warehuse Management, Rampant Techpress. Bhansali N. (2010), Strategic data warehusing: Achieving Alignment with Business. Griesemer, B. (2009), Oracle Warehuse Builder 11g Getting Started, Packt Publishing, Kimball R. and Caserta J (2004)., The Data Warehuse ETL Tlkit: Practica Techniques fr Extracting, Cleaning, Cnfrming, and Delivering Data, Wiley Publishing, Inc. Canada. Lane P. (2005), Oracle Database Data Warehusing Guide, 10g Release 2 (10.2). Malinwski E. and Zim anyi E. (2008), Advanced Data Warehuse Design Frm Cnventinal t Spatial and Tempral Applicatins, Springer-Verlag Berlin Heidelberg. Oracle Warehuse Builder Data Mdeling, ETL, and Data Quality Guide, 11g Release 2 (11.2), 2009 Pnniah P. (2007), Data Mdeling Fundamentals a Practical Guide fr IT Prfessinals, Jhn Wiley & Sns. Silvers F. (2008), Building and Maintaining a Data Warehuse, CRC Press. Thmsen (2002), E, OLAP Slutins: Building Multidimensinal Infrmatin Systems, Secnd Editin, Jhn Wiley & Sns, Inc., New Yrk W. H. Inmn (2005), Building the Data Warehuse, Furth Editin, Jhn Wiley & Sns, Inc., New Yrk. Wang, J. (2006), Encyclpedia f Data Warehusing and Mining, IDEA Grup Reference, Lndn. Hwan, M.I., and Hngjiang, X. (2007), The Effect f Implementatin Factrs n Data Warehusing Success: An Explratry Study, Jurnal f Infrmatin, Infrmatin Technlgy, and Organizatins Vlume 2. Naman A. Y. (2000), Distributed Data Warehuse: Architecture and Design, Dctr f Philsphy in Cmputer Science, University Manitba, Manitba, Canada. Shia, G. C. (2006), Design and Implementatin f Data Analysis Cmpnents, University f Akrn, Master thesis. 1843