A Performance Analysis of View Maintenance Techniques for Data Warehouses

Transcription

1 A Performance Analyss of Vew Mantenance Technques for Data Warehouses Xng Wang Dell Computer Corporaton Round Roc, Texas Le Gruenwald The nversty of Olahoma School of Computer Scence orman, OK 739 Guangtao Zhu nversty of Scence and Technology of Chna Department of Computer Scence and Technology Bejng, P.R. Chna 39 Abstract A data warehouse stores ntegrated nformaton as materalzed vews over data from one or more remote sources. These materalzed vews must be mantaned n response to actual relaton updates n the remote sources. Based on whether the current materalzed vews wll be used n computng the new vews, and whether the data warehouse wll query the remote data sources for addtonal data to do the computaton, the data warehouse vew mantenance technques are classfed nto four major categores: self-mantanable recomputaton, not self-mantanable recomputaton, self-mantanable ncremental mantenance, and not self-mantanable ncremental mantenance. Ths paper provdes a comprehensve comparson of the technques n these four categores n terms of the data warehouse space usage and number of rows accessed n order to propagate an update from a remote data source to a target materalzed vew n the data warehouse. The analyss shows that self-mantanable ncremental mantenance performs the best n terms of both space usage and number of rows accessed. Introducton A data warehouse stores ntegrated nformaton over data from one or more remote data sources for query and analyss [Hammer 95, Wener 96]. The ntegrated nformaton at the data warehouse s stored as materalzed vews. A vew s a vrtual relaton defned usng the actual relatons stored n the database. A materalzed vew s the result relaton of the evaluaton of the relatonal algebra expresson that defnes the vew relaton [Slberschatz 97]. sng these

2 materalzed vews, user queres can be answered qucly as the nformaton may be drectly avalable or can be calculated usng these materalzed vews on fly. A problem nown as the vew mantenance problem s how to mantan the materalzed vews so that they can be ept up to date n response to updates of the actual relatons n the remote data sources. There are numerous algorthms developed to solve the vew mantenance problem for tradtonal database systems. In these database systems, query expressons defned vews and actual relatons are stored at the same database. The database systems understand vew management and vew defntons and now what data s needed for propagatng updates to the vews. In a data warehouse, the query expressons that defne vews and actual relatons may be stored at dfferent database sources resdng at many stes. The sources may nform the data warehouse when an update occurs but they mght not be able to determne what data s needed for updatng the vews at the data warehouse. Therefore they may send only the actual data updates or the entre updated relatons to the data warehouse. pon recevng ths nformaton, the data warehouse may fnd that t needs some addtonal source data n order to update the vews. Then t wll ssue some queres to some of the sources to request the addtonal source data. Some of the sources may have updated ther data agan before they evaluate the requestng queres from the data warehouse. Therefore they wll send ncorrect addtonal data to the data warehouse, whch subsequently wll use the ncorrect data to compute the vews. Ths phenomenon s called dstrbuted vew mantenance anomaly. Solvng the vew mantenance problem n data warehouses s thus more complcated than that n tradtonal database systems. The objectves of ths paper are to provde a classfcaton of dfferent vew mantenance technques that have been proposed n the lterature and to conduct a comprehensve comparson

3 of these technques n terms of space usage and number of rows accessed usng the TPC benchmar for decson support queres. The rest of ths paper s organzed as follows. Secton classfes the exstng vew mantenance technques. Secton 3 presents a performance analyss of the technques. Fnally, Secton 4 provdes conclusons and future research. Classfcaton of Data Warehouse Vew Mantenance Technques Dependng on whether the current materalzed vews n a data warehouse are used n the computaton of ther new vews n response to updates that occur on the remote data sources, the exstng data warehouse vew mantenance technques can be classfed nto two broad categores: recomputaton and ncremental vew mantenance. Dependng on whether the data warehouse has to query the remote data sources n order to calculate the new vews, the technques can be further classfed as self-mantanable or not self-mantanable. The below subsectons dscuss these four categores.. The Self-Mantanable Recomputaton Category Materalzed vews can be computed from scratch by usng the vew defntons and other materalzed vews at the data warehouse. The current materalzed vews beng mantaned have no contrbuton to the calculaton of the new vews. Some technques replcate all or part of the remote data at the data warehouse. We can vew these replcated data as some nd of materalzed vews at the data warehouse. Others such as the self-mantenance warehouse approach dscussed n [Quass 96] store the remote relatons at the data warehouse as addtonal materalzed vews to provde data needed when the data warehouse computes the new vews. Therefore, the data warehouse wll never have to query the data sources for addtonal data. 3

4 A self-mantanable materalzed V vew can be defned n two ways. In the frst way, the materalzed vew V can be defned by usng other self-mantanable materalzed vews. In ths case, the vew V s defned as V Π proj cond v v v where all v 's are self-mantanable materalzed vews stored n the data warehouse. In the second way, vew V can be defned by usng the relatons resdng at the remote sources. In ths case, the vew defnton s V Π proj cond r r r where all r 's are self-mantanable relatons stored n the remote data sources. When there s an update occurrng at a remote data source, the data source nows what to do and wll send the update wth all related relatons n the vew defnton to the data warehouse n a transacton. Therefore, the data warehouse does not have to send a query requestng addtonal data to the remote data sources as all necessary nformaton s avalable. However, a self-mantanable data warehouse vew cannot be defned as V Π proj cond v v v r + r + r where all relatons v 's are self-mantanable materalzed vews resdng at the data warehouse, all relatons r 's are self-mantanable relatons resdng at the remote data sources, and there are totally relatons n the defnton of vew V. The reason s as follows. The data warehouse s stll self-mantanable when an update of relaton r l s propagated to the data warehouse as all v 's are avalable n the data warehouse. However, when an update of vew v s propagated to vew V, the relaton r l s not avalable at the data warehouse. Therefore, the data warehouse has to send 4

5 a query to the remote data sources to get the relaton r l n order to calculate vew V. Thus vew V s not self-mantanable. Although the above vew V s not self-mantanable, we can add some more selfmantanable materalzed vews and redefne vew V usng these self-mantanable vews to mae t self-mantanable. For example, n the above vew V where V Π proj cond v v v r + r + r suppose we can defne the followng self-mantanable vews v + Π proj cond r + r + r l and v + Π proj cond r l+ r l+ r Vew V can then be redefned as V Π proj cond v v v v + v + V s thus self-mantanable by defnton. The materalzed vew v at the data warehouse can be defned ether by other selfmantanable vews only, or by a collecton of self-mantanable relatons at the remote stes. We can vew the relatonshps among the data warehouse materalzed vews as a herarchy structure. The data warehouse may consst of many levels of materalzed vews. At the bottom level, there are materalzed vews defned by remote self-mantanable relatons. The vews defned by other self-mantanable materalzed vews are stayng on top of those defned by remote relatons. The materalzed vews n the data warehouse have to be mantaned n a correct order from the bottom level to the top level n the vew herarchy. The bottom level vews have to be mantaned frst. Then those vews n the second to the bottom level must be mantaned next, and so on. Fnally, the vews at the top of the herarchy are mantaned. Ths s the order that updates from the remote data sources are propagated to the data warehouse. 5

6 An advantage of the technques n ths category s that the vew mantenance anomaly problem s avoded as all necessary data are avalable at the data warehouse. The data warehouse nows the vew defntons and what to do wth the vews to eep them up to date. It elmnates accesses to the remote relatons, and therefore, t does not compete wth the remote data sources' local resources. The data warehouse mantenance operatons can then be totally separated from other OLTP operatons. Whether a remote data source s avalable or not wll not affect the data warehouse vew mantenance process. However, n order to mae the materalzed vews selfmantanable, addtonal materalzed vews that provde nformaton necessary for vew updates must be stored. Extra storage and tme are thus needed to mantan these addtonal vews.. The ot Self-Mantanable Recomputaton Category The data warehouse can recompute the materalzed vews from scratch usng the vew defntons, possbly some other materalzed vews at the data warehouse, and actual source relatons perodcally or whenever the source data s updated. That s, when an update occurs at the data source or perodcally, the source wll nform the data warehouse. Accordng to the query expresson that defnes the vew, the data warehouse may get part of data t wants from other materalzed vews at the data warehouse, and ssue queres to the sources to get the other data t does not have. The sources send the query results bac to the data warehouse. Based on the query results, the data warehouse calculates the vews and stores the results as materalzed vews n the data warehouse. The current materalzed vews have no contrbuton to the calculaton of the new vews. The data warehouse may replcate part of the remote relatons n the data warehouse. However, these data are not enough for mantanng the materalzed vews. Therefore, the data warehouse 6

7 wll have to query the remote data sources for addtonal data n order to mantan the vews. An extreme case s where the data warehouse does not replcate any remote relatons. If the vew mantenance process s not desgned carefully, the dstrbuted vew mantenance anomaly problem wll occur. Suppose that there s a data warehouse system where the remote data sources send updated relatons to the data warehouse whenever an update occurs at the data sources. pon recevng the nformaton, the data warehouse s ready to compute the new vews. But now let us assume that the data warehouse fnds that t needs some other relatons at some remote data sources to compute the new vews. It wll ssue queres to these data sources. Suppose the data sources that sent the updated relatons to the data warehouse update the relatons agan before they receve the queres from the data warehouse. The data sources answer the query and send the results to the data warehouse. These results mght contan extra nformaton that s ncorrect. The data warehouse wll then use the ncorrect data to compute the new vews, whch wll result n ncorrect new vews. There are a lot of solutons to the dstrbuted vew mantenance anomaly problem. The smplest soluton s as follows. Instead of sendng the updated relatons to the data warehouse, the data sources just smply nform the warehouse that there s an update that occurs at the data sources. The data warehouse wll ssue queres to the data sources to request all relatons requred n the vew defntons. After the data sources receve the queres, they wll send all requested relatons to the data warehouse. The anomaly problem s thus avoded. The not self-mantanable recomputaton approach s smple. The anomaly problem can be avoded easly. However, the recomputaton process s also tme and resource consumng. The data warehouse sends queres bac to the sources and wats for answers n order to computer 7

8 the new vews. Processng these queres consumes the sources local resources. If the sources are unavalable, the data warehouse wll not get the answers t needs..3 The Self-Mantanable Incremental Mantenance Category In ths category, the data warehouse vews are mantaned by usng the vew defntons, the materalzed vews, and the vew updates. The data warehouse wll never query the remote data sources as the nformaton at the data warehouse s enough for mantanng the vews. The data warehouse computes the vew updates, then adds them to the materalzed vews. The process s ncremental. ormally, only necessary remote relatons, or vews of the remote relatons are stored at the data warehouse as materalzed vews. In the extreme case, all remote relatons can be replcated at the data warehouse. The self-mantanable warehouse approaches dscussed n [Cu 99], [Hull 96] and [Quass 96] belong to ths category. Let us dscuss how to mantan a vew V that s defned as V Π proj cond v v v where each v s a materalzed vew and s defned as ether v Π proj cond v v v where each v j s a vew defned by other auxlary materalzed vews, or v Π proj cond r r r where each r j s a base relaton. In the frst case, each vew v j s defned ether as v j Π proj cond v v v where each v j s a vew defned by other auxlary materalzed vews, or 8

9 v j Π proj cond r r r where each r j s a base relaton. Fnally, at the lowest level of the vew herarchy dscussed earler n ths paper, vew v Mj can only be defned by relatons at the remote data sources as follows: v Mj Π proj cond r M r M r M where each r Mj s a base relaton. The above vew V s thus defned by M levels of the materalzed vews n the vew herarchy. In the second case, the vew can only be defned by base relatons r j. Each vew v j can be defned by v 3j. Vews can be defned by those vews at one level lower than t. The lowest level vews v Mj can only be defned by relatons at the remote data sources. Suppose an update j occurs at a data source, where represents the vew level n the vew herarchy and j represents the relaton n the vew defnton where the update occurs. The data source sends the update j along wth the related base relatons r, r, and r except r j to the data warehouse. The data warehouse calculates the vew update v -j usng the update j and relatons r, r, and r except r j as follows: v -j Π proj cond r r j r Then the vew update v -j s added to the vew v -j to produce the new vew. The vew v -j s thus mantaned. The vew update v -j s propagated to the vews at one level hgher than t. In ths case, t s v -j. The vew v -j s defned by self-mantanable vews as follows: v -j Π proj cond r - r - v -j r - We can calculate the vew update usng the followng formula: v -j Π proj cond r - r - v -j r - Ths process s repeated untl the top vew V n the vew herarchy s updated. 9

10 All ntermedate materalzed vews can be vewed as auxlary vews. These auxlary vews are self-mantanable. The materalzed vew V s self-mantanable by usng the update nformaton and addtonal nformaton from the auxlary vews. The data warehouse vews, ncludng vews such as V and auxlary vews, can be mantaned startng wth those vews that do not depend on any other auxlary vews, worng up to the fnal orgnal vew V. Fgure. Vew Herarchy Example All related materalzed prmary vews, auxlary vews and base relatons can be drawn n a herarchy structure as shown n Fgure. All leaves n the herarchy structure are those materalzed vews defned by the base relatons. In ths example, V s the prmary materalzed vew. Vews v, v and v are materalzed auxlary vews defned by other materalzed auxlary vews. Vews v 3, v, v 3, v 4, v 3 and v 4 are materalzed auxlary vews defned by the base relatons. All relatons r j 's are the base relatons. The vews n the leaves should be mantaned frst. Suppose an update for r 33 occurs n the data source. Vew v 4 should be mantaned frst. Then vews v and vew v must be mantaned next n that order. Fnally, the prmary vew V s mantaned. The data warehouse never needs to query the remote data source to get addtonal data. The data warehouse mantenance operatons can be totally separated from other OLTP

11 operatons. Whether the remote data source s avalable or not wll not affect the data warehouse vew mantenance process. However, n order to mae the materalzed vews self-mantanable, the auxlary vews are stored n the data warehouse to provde the addtonal nformaton. Extra storage and tme overhead are therefore requred to mantan the auxlary vews themselves. How to desgn materalzed vews at the data warehouse so that only necessary nformaton are stored at the data warehouse s a major ssue [Quass 96, Huyn 96a, 96b, 97b, 97c]..4 The ot Self-Mantanable Incremental Mantenance Category Instead of recomputng every vew from scratch, only parts of the warehouse that change are computed. However, the data warehouse has to query the remote data sources whenever necessary because the nformaton at the data warehouse s not enough to mantan the vew. A number of exstng approaches fall under ths category. Among them are the unrestrcted base access [Zhuge 95, 96, 97a] and runtme warehouse self-mantenance [Huyn 97a]..4. nrestrcted Base Access In the nrestrcted Base Access approach [Zhuge 95, 96, 97a], the data warehouse accesses the actual relatons from the data sources whenever necessary n order to mantan the materalzed vews. There are many proposed algorthms that follow ths approach. The Eager Compensatng Algorthm ECA s the smplest among them. It s also the fastest algorthm that wll let the data warehouse reman n a consstent state [Zhuge 98a and Zhuge 98b]. In ths algorthm, compensatng queres are sent bac to the sources to offset the effects of concurrent updates [Zhuge 95]. They are used only when the next update occurs at the sources before the sources receve the data warehouse queres. The data warehouse eeps a temporary table called COLLECT to eep the ntermedate answers t receves from the data sources. It also eeps a set called nanswered Query Set

12 QS to eep trac of those queres t sent to the data sources but has not receved ther answers yet. Suppose an update occurs at a data source. The data source sends to the data warehouse. Suppose the data warehouse wants to update the materalzed vew that s defned as V Π proj cond r r r where all relatons r 's resde at a sngle data source. The data warehouse determnes the query for calculatng the delta vew as Q Π proj cond r r r - r + r It then creates a temporary COLLECT table and QS set for processng ths specfc query, and sets both the COLLECT table and QS to empty. The data warehouse wrtes the query Q to the QS and sends the query Q to the data source. Suppose there s another update j that occurs at the same data source. The data source sends the update j to the data warehouse before t receves the query Q. The data warehouse now receves the update j. It nows that the upcomng answer for Q from the data source wll contan extra nformaton caused by smultaneous j update at the data source. Therefore, t has to offset ths extra nformaton. The data warehouse determnes the query Q j for updatng j as follows Q j Π proj cond r r r - r r + r j- j r j+ r Π proj cond r r r - r + r j- j r j+ r The frst part of Q j s the same as before; the second part s used to compensate for the extra nformaton that query Q wll receve. The data warehouse s eager to compensate before t receves the answer for query Q. Then t wrtes query Q j to QS and sends query Q j to the data source. ow the QS set contans two entres,.e., Q and Q j. The data source then processes

13 query Q and sends the answer to the data warehouse. The query answer A wll contan the followng extra nformaton due to the update j, Π proj cond r r r - r + r j- j r j+ r The data warehouse receves the answer A, removes entry Q from QS, wrtes the answer for Q to the temporary table COLLECT. ow QS contans only one entry Q j. The COLLECT table thus contans the followng data: A Π proj cond r r r - r + r j- r j r j+ r + Π proj cond r r r - r + r j- j r j+ r The data warehouse could not ntegrate the query answer nto the data warehouse at ths tme as t wll cause the data warehouse to contan nconsstent data. Therefore t has to wat for the query answer for query Q j to come bac. The COLLECT table s used to temporarly store query answers. After t receves the answer for the last unanswered query n QS n our case, Q j, t can ntegrate the data n the COLLECT table nto the data warehouse. The data source processes query Q j and sends answer A j to the data warehouse. A j contans the followng data: A j Π proj cond r r r - r r + r j- j r j+ r Π proj cond r r r - r + r j- j r j+ r The data warehouse receves the answer and removes the entry Q j from the QS set. ow QS s empty. The answer A j s ntegrated wth the data n the COLLECT table as follows: Π proj cond r r r - r + r j- r j r j+ r + Π proj cond r r r - r + r j- j r j+ r.+ Π proj cond r r r - r r + r j- j r j+ r Π proj cond r r r - r + r j- j r j+ r Π proj cond r r r - r + r j- r j r j+ r + Π proj cond r r r - r r + r j- j r j+ r 3

14 ow the COLLECT table contans the correct answer that the data warehouse wants. The data warehouse can then ntegrate the data n the COLLECT table nto the data warehouse. The nrestrcted Base Access approach s smple. Instead of dscardng old materalzed vews and calculatng new vews from scratch, ths approach calculates vew updates then adds them to the old vews n order to get the new vews. However, the data warehouse has to access remote source actual relatons n order to propagate one source update. In the eager compensatng algorthm dscussed above, all actual relatons have to be accessed f compensatng queres are used. In ths approach, the data warehouse may have to send queres bac to the sources and wats for answers n order to compute the vew updates. Therefore, ths approach has the same lmtaton as the not self-mantanable recomputaton approach. Computng these queres consumes remote sources local resources, and wll slow down other OLTP operatons. If the remote sources are unavalable, the data warehouse wll not get the answers t needs..4. Runtme Warehouse Self-Mantenance In the self-mantenance warehouse approach, at desgn tme, we determne vews to be selfmantanable and decde to add more auxlary vews for addtonal nformaton. Desgn-tme self-mantanablty s not flexble. It may be dffcult or mpossble for us to now the exact contents of the vews and ther updates at desgn tme. To solve ths problem, a run tme warehouse mantenance approach has been ntroduced [Huyn 97a]. The basc dea of the runtme self-mantenance approach s that the data warehouse generates the self-mantanable test for the vews to determne whether the vews are selfmantanable for a partcular update. At run tme, the self-mantanable test determnes the vews for self-mantanablty. If a vew s self-mantanable, then the vew can be mantaned by the 4

15 update nformaton, the vew tself, and the query expresson that defnes the vew. In ths case, the run-tme self-mantanable approach s smlar to the self-mantanable warehouse approach dscussed n the prevous secton. However, the data warehouse does not store and mantan any auxlary vews. If the vew s not self-mantanable, then the data warehouse has to query the remote data sources for those relatons t needs n order to update the vew. In ths case, ths approach s smlar to the unrestrcted base access approach. The runtme warehouse self-mantenance approach uses the vew mantanablty test frst before dong any mantenance. Ths creates the overhead n the data warehouse. However t mantans the vews by usng the vews n the data warehouse or some base relatons only, wthout requrng auxlary vews that are necessary n the self-mantanable ncremental mantenance approach. Ths saves the storage space n the data warehouse. 3 Performance Analyss We conduct an analyss to compare the performance of dfferent algorthms n the four categores. We consder only the problem of sngle vew mantenance n a sngle source envronment because the ECA algorthm n the not self-mantanable ncremental mantenance category can only be used n ths envronment [Zhuge 97a, Zhuge 96]. For self-mantanable vew mantenance technques, we also consder the case where all actual relatons are replcated at the data warehouse. 3. Performance Measurements In our analyss, only Select-Project-Jon vews are consdered. We measure the performances of the technques n terms of space and number of row accesses, whch are defned as follows: 5

16 !" Space: total space needed to store the data n the data warehouse, ncludng space for auxlary vews. We do not consder ndces.!" umber of rows accessed: the number of rows that must be accessed n the data warehouse and the data sources n order to ntegrate the updates nto the data warehouse. 3. Analyss Parameters The analyss parameters and ther default values are lsted n Table. The default values are calculated based on the TPC benchmar for decson support queres [TPC 99]. Meanng Symbol Default value Range nalty of vew V V 94 ~, Tuple sze of vew V n bytes tsv 43 ~ 5 umber of auxlary vews per vew av 3 ~ umber of base relatons n the vew 3 ~ 7 defnton nalty of base relaton r r 8,, ~,,, Tuple sze of base relaton r n tsr 6 ~ 8 bytes Selectvty: the fracton of tuples that.3. ~. satsfy the select condton The jon factor s roughly the fracton j.73. ~. of tuples n a relaton that jon wth others umber of tuples n a relaton that J j Calculated Calculated jon wth others r umber of nterferng updates for I.5 ~ each query nalty of update umber of tuple nsertons n a base relaton at data source update Table. Analyss Parameters 6

17 3.3 Comparson Based on Space eeded n the Data Warehouse 3.3. Self-Mantanable Recomputaton The technques n ths category do not query the remote data source for addtonal data n order to mantan the data warehouse materalzed vews. The data warehouse can replcate all or part of the remote base relatons at the data warehouse. These addtonal data tae space at the data warehouse. Here we consder the case where the materalzed vews are defned by other materalzed vews auxlary vews at the data warehouse, and all auxlary vews are replcated remote relatons. A vew V s defned as V Π proj cond v v v where all v 's are materalzed vews stored n the data warehouse. In the average case, the amount of space needed s: V ts V + av AV ts AV where av. If all auxlary vews are the same, the above formula can be reduced to V ts V + av AV ts AV av when none of the auxlary vews s needed. It means that the materalzed vew s a replcated remote relaton. Ths s the best case. The formula can be rewrtten as follows: V tsv If all auxlary vews are needed to mantan the vew, then av. Ths s the worst case. The formula can be rewrtten as follows: V ts V + AV ts AV If all auxlary vews are the same, the above formula can be reduced to V ts V + AV ts AV 7

18 3.3. ot Self-Mantanable Recomputaton Here we consder only the case where the data warehouse does not replcate any base relatons. Therefore, the data warehouse always has to query the remote data sources. The data warehouse stores only materalzed vews. In ths extreme stuaton, the amounts of space needed n the best case, the average case and the worst case are the same, and are equal to V tsv Self-Mantanable Incremental Mantenance Smlar to the self-mantanable recomputaton technques, the technques n ths category can replcate all or part of the remote data at the data warehouse. Here we consder only the case where the materalzed vews are defned by other materalzed vews auxlary vews at the data warehouse, and all auxlary vews are replcated remote relatons. A vew V s defned as V Π proj cond v v v where all v 's are materalzed vews stored n the data warehouse. The amount of space needed at the data warehouse s the same as that of the self-mantanable recomputaton technques. In the average case, the space needed s as follows: V ts V + av AV ts AV where av. If all auxlary vews are the same, the above formula can be reduced to V ts V + av AV ts AV In the best case, no auxlary vews are needed as the vew s self-mantanable. The formula can be rewrtten as follows: V tsv In the worst case, all auxlary vews are needed to mantan the vew. The formula can be rewrtten as follows: 8

19 V ts V + AV ts AV In the worst case, the auxlary vew's tuple sze and cardnalty are equal to those of the remote relaton. If all auxlary vews are the same, the above formula can be reduced to the followng: V ts V + AV ts AV ot Self-Mantanable Incremental Mantenance Here we consder the Eager Compensatng Algorthm ECA for ths category. In ECA, a temporary table COLLECT s used to store ntermedate query answers. For every update, the queres ncludng compensated queres are sent to the data source. ote that the COLLECT table s empty only when there s no query to the data source, or the answers for all the queres are returned to the data warehouse before a new update occurs at the data source. Ths s the best case. The space needed s as follows: V tsv In the average case, the sze of the COLLECT table for a specfc update wth all ts nterferng updates I s equal to the sum of the cardnaltes of the fnal query answers for all queres. The total number of queres sent to the data source can be calculated as below [Zhuge 97b]: q I j J In the worst case, the number of queres sent to the data source s: q. I r Let us derve the formula for the multple tuple update case. We follow the method used n [Zhuge 97b]. At frst, we have to fnd the number of wrapper queres sent from a vew 9

20 manager to the query processor [Zhuge 95, Zhuge 97b]. In the sngle tuple update case, A. In the multple tuple update case, A. The total number of wrapper queres to evaluate the update query s > : : : J q Assume all the s and Js are the same, the formula s smplfed to the followng: > : : J q The compensatng query s defned as: CQ Π proj cond n R n+ R n+ R There are n multple tuple updates and -n actual base relatons. The number of wrapper queres to evaluate the query s: > : : J j n q n n n n Then, we estmate the number of compensatng queres as I, where I s defned as the maxmum number of nterferng updates that occur between the tme when the data warehouse sends a query to the remote data source and the tme when the data warehouse receves the correspondng answer [Zhuge 98b]. In the worst case, there are I nterferng updates for each query. The number of wrapper queres s:

21 J j I q When, the above formula s reduced to the followng: J j I q whch s the formula of the sngle tuple update case [Zhuge 98b]. Suppose the cardnalty of the fnal query answer A for all nterferng updates are equal. J A for the sngle tuple update case [Zhuge 98b]. For the multple tuple update case, J A. When all s and Js are equal, J A. For a query wth n update relatons and n base relatons, n n n J j A, where n. The max cardnalty of the COLLECT table for updates at a tme s: J J j j I + J J j I. The space needed s V ts J J j I V ts V + +. In the worst case, when, j, J j r r. The space needed at the data warehouse s V ts r r I V ts V +.

22 3.3.5 Comparson Results In summary, for a gven vew V, the amounts of space needed n the data warehouse for the four categores are lsted n Table 3, where AV stands for Auxlary Vew. Category Self-Mantanable Recomputaton SMR ot Self-Mantanable Recomputaton SMR Self-Mantanable Incremental Mantenance SMIM ot Self-Mantanable Incremental Mantenance SIM Space needed n the data warehouse The best case: V tsv The average case: V ts V + av AV ts AV av The worst case: V ts V + AV ts AV V tsv The best case: V tsv The average case: V ts V + av AV ts AV av The worst case: V ts V + AV ts AV The best case: V tsv The average case: V ts V + + I j The worst case: V ts V + I r J J ts V r ts V Table. Space needed n the data warehouse We draw the graphs for the formulas derved above usng the parameters' defaults values lsted n Table. Due to space lmtaton, we wll only dscuss the results of the average case. In ths case, the not self-mantanable recomputaton approach does not requre extra space at the data warehouse. However, for the self-mantanable recomputaton and self-mantanable ncremental approaches, extra space s requred at the data warehouse to store the replcated

23 source data as auxlary vews. Ths extra space s proportonal to the total number of auxlary vews av, the auxlary vew cardnalty AV and the auxlary vew tuple sze tsav. For SIM, extra space s requred to store the COLLECT table. Suppose that the tuple sze of the COLLECT table s equal to the vew's tuple sze. In some cases, the space used to store the COLLECT table n SIM s small compared to the space used for storng the materalzed vews n SMR and SMIM. In other cases, the space used for storng the COLLECT table can be grown much larger than that of materalzed vews. Let us examne Fgures -4, where SMR s not shown snce t overlaps wth the x-axs. At frst, the space used for storng the COLLECT table s smaller and growng slower than that used for storng the materalzed auxlary vews when the cardnalty of auxlary vews s small Fgure. Then, the space used for storng the COLLECT table grows faster. At some pont AV 6. 7 n our case, the space used for storng the COLLECT table exceeds the space used for storng the materalzed auxlary vews Fgure 3. When the cardnaltes of auxlary vews or base relatons are large, the space used for storng the COLLECT table s much larger than that used to store the auxlary vews Fgure 4. From Table, we can see that the space needed for the SIM approach grows exponentally wth the select factor, jon factor, and cardnalty of updates. In summary, n the average case, as no addtonal space s requred at the data warehouse for the not self-mantanable approach, t s the best approach n terms of space used n the data warehouse. The amounts of space used for both the self-mantanable approaches, SMR and SMIR, are the same. When there are very few conflctng updates and the cardnaltes of the base relatons are relatvely small, the space used for the not self-mantanable approach s less than those of both the self-mantanable approaches. However, f there are a lot of conflctng 3

24 updates and the cardnaltes of the base relatons are large, the space used for the not selfmantanable approaches are much larger than those of both the self-mantanable approaches. Fgure. Space used n the data warehouse The average case Fgure 3. Space used n the data warehouse The average case 4

25 Fgure 4. Space used n the data warehouse The average case 3.4 Comparson Based on the umber of Rows Accessed To analyze the number of rows accessed at the data warehouse by the technques, we made the followng assumptons:!" The set of a prmary vew and ts auxlary vews f any s ndependent to other sets of prmary vews and ther auxlary vews.!" We do not consder ndces. Lnear search s thus used to chec f a record satsfes a select or jon condton.!" All auxlary vews are self-mantanable and are replcated base relatons.!" pdates to auxlary vews and prmary vews are for appendng only. 5

26 3.4. Self-Mantanable Recomputaton The data warehouse wll never query the remote data sources as all necessary data are avalable at the data warehouse. pdates from the remote data sources have to be propagated to the replcated relatons at the data warehouse frst, then the data warehouse recalculates the vew relaton and stores the result at the data warehouse as the new materalzed vew. In order to propagate an update to data warehouse replcated relaton, the number of rows to be accessed at the data warehouse s the cardnalty of the relaton tself plus the cardnalty of the update. That s, r +. Then the data warehouse recomputes the materalzed vew usng the vew defnton. Suppose the data warehouse materalzed vew s defned as: Q Π proj cond v v v The strategy such as the one descrbed n [Zhuge 97b] can be used to evaluate the query Q that defnes the vew V. Let us rename the actual relatons accordng to the jon order. The query Q that defnes the vew becomes Q Π proj cond R R R sng the equvalence rules [Slberschatz 97], the select condton cond can be decomposed nto a sequence of ndvdual selectons 's on ndvdual actual relatons R 's. Let T be the temporary relaton after s appled to R, and let T be the temporary relaton after the jon and selecton wth relaton R s done. T s the new vew. To compute T, each tuple n R has to be accessed for T - tmes. T T - R. T J T - and T 6

27 R. Each tuple n R s accessed once. R s accessed T R tmes. The number of tuples accessed for R and R s Thus, R + R R T J T J R. To get the new vew, the total number of tuples that must be accessed s R + R R + R 3 R J + + R R Π - J R + R + R 3 J + + R Π - J Suppose all the select and jon condtons, and cardnaltes of all base relatons are the same, the total number of tuples to be accessed n order to recompute the vew s equal to AV + AV + J + 3 J J - + AV AV J Ths s the average case. In the worst case, when, j, J j AV AV, the total number of tuples to be accessed s equal to AV + AV + AV + AV + + AV - AV + AV + AV + + AV - AV AV For update updates, the total number of tuples to be accessed n the average case s + update AV AV In the worst case, the formula s update AV AV J 7

28 Includng the number of rows that need to be accessed to mantan the vew and replcated data at the data warehouse, the total number of rows to be accessed n the average case s update V AV AV In the worst case, the number becomes update V AV 3.4. ot Self-Mantanable Recomputaton AV J Only source data s requred to be accessed. The reason s that the warehouse recalculates the full vew usng the source data each tme. It does not use the data warehouse data. Suppose the system locs all base relatons n order to evaluate the query expresson that defnes the vew. If the nested-loop jon method [Slberschatz 97] s used to evaluate t, the total number of rows to be accessed s r. Another strategy such as the one descrbed n [Zhuge 97b] can also be used to evaluate the query Q that defnes the vew V. It wll reduce the total number of rows to be accessed. Let us rename the actual relatons accordng to the jon order. The query Q that defnes the vew becomes Q Π proj cond R R R sng the same method presented n Secton 3.4., suppose all select and jon condtons, and cardnaltes of all base relatons are the same, the total number of tuples to be accessed n order to recompute the vew s equal to r + r + J + 3 J J - 8

29 + r r J Ths s the average case. In the worst case, when, j, J j r r, the total number of tuples to be accessed s equal to r + r + r + r + + r - r + r + r + + r - r r For update updates, the total number of tuples to be accessed n the average case s J + update r r. In the worst case, the formula s update r r Self-Mantanable Incremental Mantenance o queres are sent to the data sources for addtonal nformaton. Therefore, the number of rows accessed n the data source s equal to. For base relatons n a vew, av should be less than or equal to. In the worst case, av s equal to. At frst, the auxlary vew tself has to be mantaned before the prmary materalzed vew can be mantaned. Let stand for the cardnalty of update. Accordng to our assumpton that auxlary vews are self-mantanable and updates are used for appendng only, the number of rows needed to be accessed n order to mantan the auxlary vew s + AV. Let AV stand for the cardnalty of the auxlary vew update, whch s the same as. Then the update s propagated to the prmary vew. We need to calculate the 9

30 prmary vew update. The number of rows accessed can be estmated usng the same method presented n Secton Suppose the all select and jon condtons, and cardnaltes of all auxlary vews are the same, the total number of tuples to be accessed n order to calculate the prmary vew update s equal to AV + AV AV + J + 3 J + + av- J av- av + + AV AV J Fnally, the vew update s appended to the prmary vew. The total number of rows to be accessed n order to propagate a source update to the data warehouse s av V AV AV AV J Snce AV, we have av V AV AV AV J Ths s the average case. In the worst case, when, j, J j AV AV, av, the total number of tuples to be accessed s equal to V AV AV AV AV V AV AV AV In the sngle tuple update case,.e., AV, the total number of tuples to be accessed s equal to V AV AV For update updates, the average number of tuples to be accessed for the multple tuple update case s 3

31 av update V AV AV AV For the worst case, the total number of tuples to be accessed s update V AV AV ot Self-Mantanable Incremental Mantenance AV + J In the ECA algorthm, all tuples n the vew table have to be accessed n order to fnd a tuple to ntegrate wth the vew update. However, the data warehouse may have to access data from remote stes except for the best case. Parts of these queres are compensated. In Secton 3.3.4, we derve the number of wrapper queres correspondng to queres wth n relatons n the multple tuple update case as q I j J When there are totally update updates, the number of wrapper queres s q update I j J In general, for a query that s defned as Q Π proj cond R R R, the total number of tuples to be accessed n order to process the query s R + R + R 3 J + + R Π - J For a compensatng query Q Π proj cond R + R + R, there are multple tuple updates and - base relatons. Suppose all updates are evaluated frst, the number of rows to be accessed n order to evaluate a query s + + j j - + j - r Π - J 3

32 3 + + j j j - r + + j - r Π - J J r j j For the multple tuple update case, the total number of tuples to be accessed n order to propagate update updates from the remote source to the target data warehouse vew s: _ query per accessed tuples queres of number I update J r j j J j I update In the worst case, the total number s r r r I update Comparson Results The formulas to calculate the number of rows that need to be accessed n order to mantan a materalze vew for all four categores are lsted n Tables 3 and 4. We use the parameters' default values lsted n Table to draw the graphs showng the number of rows that need to be accessed to mantan a materalzed vew. Due to space lmtaton, here we wll dscuss the results for the average case only.

33 Category umber of rows accessed n data warehouse SMR The average case: update V + AV + + AV J umber of rows accessed n data source SMR SMIM The worst case: update V + + AV + update V AV The average case: update V + AV + AV + J av + AV av The worst case: update V + AV + AV + AV The best case: The average case: update r + r The worst case: update r r J Table 3. umber of Rows Accessed Part of 33

34 Category umber of rows accessed n data umber of rows accessed n data source warehouse SMIM update / I V The best case: The average case: update I + j j J + + j r The worst case: update I + r r + + J + r Table 4. umber of Rows Accessed Part of In the average case, the total numbers of rows accessed for the not self-mantanable recomputaton SMR approach and the self-mantanable recomputaton SMR approach are very smlar. As the same base relatons are replcated at the data warehouse n the selfmantanable recomputaton approach, actually these data wll be accessed twce. The frst tme s for mantanng the auxlary vews and the second tme s for mantanng the target materalzed vews at the data warehouse. When the cardnalty of the base relatons/auxlary vews, r, s small, the number of rows accessed for the self-mantanable recomputaton approach s somewhat larger than that of the not self-mantanable recomputaton approach Fgure 5. When r becomes larger, the dfference s small enough to be neglected Fgure 6. 34

35 Fgure 5. umber of rows accessed n the data warehouse The average case Fgure 6. umber of rows accessed n the data warehouse The average case 35

36 Fgure 7. umber of rows accessed n the data warehouse The average case Fgure 8. umber of rows accessed n the data warehouse The average case 36

37 When r s small, the numbers of rows accessed for self-mantanable recomputaton and not self-mantanable recomputaton grow faster than those for selfmantanable ncremental mantenance and not self-mantanable ncremental mantenance Fgure 7. When r becomes very large, the number of rows accessed for both the recomputaton approaches are much larger that those of both the ncremental approaches Fgure 8. In order to see the dfferences, we redrew Fgure 8 usng a logarthmc scale n the y-axs; the results are shown n Fgure 9. From Fgure 9, we can see that the number of rows accessed n SMR s the same as that n SMR, but s hgher than that of SIM, whch s hgher than that of SMIM. We also observe the smlar results when varyng the values of the select factor and jon factor j. Fgure 9. umber of rows accessed n the data warehouse The average case 37

38 4. Conclusons and Future Research Category Advantage Dsadvantage Self-Mantanable Recomputaton -Data warehouse vew mantenance operatons are totally separated from OLTP operatons; -navalable source wll not bloc the data warehouse vew mantenance process; -Data are replcated at data warehouse; -eed extra data storage for replcate data; -Have to mplement and mantan data transfer processes to transfer data from sources to data warehouse; ot Self- Mantanable Recomputaton -Very smple to mplement; -o replcate data at the data warehouse; -o extra data storage for replcate data; -Do not have to mplement and mantan data transfer processes to transfer data from sources to data warehouse; -navalable source wll bloc the data warehouse vew mantenance process; -Evaluatng queres at the data sources consumes local resources; -Data warehouse vew mantenance operatons are not separated from OLTP operatons; Self-Mantanable Incremental Mantenance ot Self- Mantanable Incremental Mantenance -Data warehouse vew mantenance operatons are totally separated from OLTP operatons; -navalable source wll not bloc the data warehouse vew mantenance process; -In the worst case, the number of rows accessed to mantan a vew s the lowest; -o replcate data at the data warehouse; -o extra data storage for replcate data; -Do not have to mplement and mantan data transfer processes to transfer data from sources to data warehouse; -Data are replcated at data warehouse; -eed extra data storage for replcate data; -Have to mplement and mantan data transfer processes to transfer data from sources to data warehouse; -navalable source wll bloc the data warehouse vew mantenance process; -Evaluatng queres at the data sources consume local resources; -Data warehouse vew mantenance operatons are not separated from OLTP operatons; -Have to desgn the vew mantenance process carefully to avod the anomaly problem; -In the worst case the number of rows accessed s the hghest; -Performance s down-graded rapdly; -eed extra storage for ntermedate data COLLECT tables; Table 5. Advantages and dsadvantages of the vew mantenance technques All data warehouse vew mantenance technques can be classfed nto four major categores. They are self-mantanable recomputaton, not self-mantanable recomputaton, self- 38

39 mantanable ncremental mantenance, and not self-mantanable ncremental mantenance. Ther advantages and dsadvantages are lsted n Table 5. Both self-mantanable recomputaton and self-mantanable ncremental mantenance approaches totally separate the data warehouse vew mantenance operatons from the OLTP operatons. Therefore, the vew mantenance operatons wll not consume data sources local resources. These operatons only consume the data warehouse's resources. Even f the remote data sources are not avalable, the data warehouse vew mantenance process can contnue runnng. However, a part or all source data are replcated at the data warehouse to mae the data warehouse vew mantenance process self-mantanable. These replcated data tae space. Data transfer processes are mplemented to transfer data from the remote data sources to the data warehouse. Desgn, mplement and mantan these processes are tme-consumng. A lot of unnecessary data may be duplcated at the data warehouse. However, these are the approaches that probably many large companes have to tae f they want to separate ther data warehouse vew mantenance operatons from ther OLTP operatons. Both the not self-mantanable recomputaton and not self-mantanable ncremental mantenance approaches suffer from some common dsadvantages. As the remote data sources have to process queres from the data warehouse that consume ther lmted local resources, the OLTP system wll be slow. Once a data source s unavalable, the data source wll not be able to answer queres sent from the data warehouse n tme. It wll bloc the data warehouse vew mantenance process. The not self-mantanable ncremental mantenance approach has some addtonal dsadvantages. To avod the anomaly problem, the vew mantenance process must be desgned carefully. If a lot of updates happen at the data sources, the data warehouse may ssue many compensatng queres. It s very possble that the data warehouse may never get the fnal 39

40 query results. Both approaches also have some common advantages. As there s no replcate data stored at the data warehouse, no data transfer process has to be mplemented and mantaned. There s no extra space for storng replcate data. Both approaches are good for small to mdszed companes whose OLTP database systems are not too busy. Among all the four categores, self-mantanable ncremental mantenance s the best n terms of space used n the data warehouse and number of rows accessed n order to propagate an update to the target materalzed vew n the data warehouse. As the cost of data storage becomes ncreasngly low, ths s the best approach to mplement a data warehouse. For future wor, we plan to consder another mportant performance measurement called the vew refresh tme, whch s defned as the elapse tme from the tme the system receves a source update to the tme the update s reflected n the vew. We wll also address the problem of multple vew mantenance n a multple source envronment. References [Cu 99] Y. Cu and J. Wdom. "Storng Auxlary Data for Effcent Vew Mantenance and Lneage Tracng." Techncal Report, Stanford nversty, 999. [Hammer 95] J. Hammer, H. Garca-Molna, J. Wdom, W. Labo, and Y. Zhuge, The Stanford Data Warehousng Project. IEEE Data Engneerng Bulletn, June 995. [Huyn 96a]. Huyn, Effcent Vew Self-Mantenance. Proceedngs of the ACM Worshop on Materalzed Vews: Technques and Applcatons, Montreal, Canada, June 7, 996. [Huyn 96b]. Huyn, Effcent Self-Mantenance of Materalzed Vews. Techncal ote, 996. [Huyn 97a]. Huyn, Mantanng Data Warehouse nder Lmted Source Access. Ph.D. Thess, Stanford nversty, August 997. [Huyn 97b]. Huyn, Explotng Dependences to Enhance Vew Self-Mantanablty. Techncal ote, 997. [Huyn 97c]. Huyn, Multple-Vew Self-Mantenance n Data Warehousng Envronments. Proceedngs of the 3rd VLDB Conference, Athens, Greece, 997. [Hull 96] R. Hull and G. Zhou, A framewor for supportng data ntegraton usng the materalzed and vrtual approaches, In SIGMOD 996. [Labo 97b] W. Labo, D. Quass, and B. Adelberg, Physcal Database Desgn for Data Warehousng. Proceedngs of the Internatonal Conference on Data Engneerng, Bnghamton, K, Aprl,