A TALE OF TWO ARCHITECTURE It was the est of times, it was the worst of times, it was the age of wisdom, it was the age of foolishness, it was the epoch of elief, it was the epoch of incredlity, it was the season of Light, it was the season of Darkness, it was the spring of hope, it was the winter of despair, we had everything efore s, we had nothing efore s,... Charles Dickens A Tale of Two Cities By W H Inmon It was the est of times. It was the worst of times. From an age of applications and the confsion over application ased information in the corporation arose the concept of a data architectre and data warehosing. Into the miasma came Bill Inmon s est selling ook - BUILDING THE DATA WAREHOUE. And there was Kimall s software company RedBrick ystems. And soon the world of data warehosing was orn. It was the late 1980 s and the world was aot to witness the rise of analytical processing, siness intelligence and a whole host of technologies never efore seen that wold change the world forever. THE CORPORATE INFORMATION FACTORY DATA WAREHOUE The indstry accepted definition of a data warehose a sect oriented, integrated, non volatile, time variant collection of data for management s decision making appeared in BUILDING THE DATA WAREHOUE. Later ooks y Inmon soon appeared which descried the architectre into which the data warehose fit. The architectre sometimes called the corporate information factory (or simply Inmon s architectre ) is seen in a simple form in Fig 1. INGLE VERION OF THE TRUTH The nexs of the corporate information factory and its fondation the data warehose - is the notion of the single version of the trth. Centered in the data warehose and descried y the definition of the data warehose is the granlar, integrated historical data - the single version of the trth - which is the essence of the corporate information factory. With the corporate information factory, the data
warehose has a place where there is a final word as to what data is right and what data is wrong. At the heart of the confsion over information that preceded the data warehose is the inaility of the organization to nderstand what data is correct and what data is not correct. It is hard to make proper decisions on data that is nreliale. Prior to the corporate information factory, organizations had a plethora of data, t they had no idea what data was correct and what data was incorrect. With the corporate information factory, there was a definitive sorce of data to which the corporation cold trn the single version of the trth. It is tre that the corporate information factory solved many other prolems. Bt the single most important aspect of the corporate information factory was that it contained the single version of the trth. The corporate information factory incldes an architectre that centers arond the data warehose. It is in the data warehose where the single version of the trth resides. Other featres of the corporate information factory architectre inclde legacy, operational systems, ETL and data s. ETL is the technology that reads in raw data from applications and writes ot corporate data (or data that constittes the single version of the trth ). s are those data ases created for the analytical needs for different departments and different grops of people doing analytical processing. In the corporate information factory the only sorce of data for the data s is the data warehose. The iggest isse in creating the corporate information factory is that of the integration of application data into corporate data. that comes from applications mst e recast into a corporate form and strctre. That is how the single version of the trth is created. The integration of old legacy, operational nintegrated data is a complex and time consming o. In many cases, old legacy data is ndocmented. In many cases old legacy data lies in technologies that have een nspported for years. In many cases old legacy applications mst e merged where a merger of application data was never an oective of the designer of the legacy application. In many cases the very definition of data sitting in an old application mst e recast. All of the work reqired for integration is tedios and mst e performed in a disciplined and in an exact manner. As sch, ilding a data warehose for the corporate information factory is not an easy or a fast thing to do. Bt the reslt is integrated data a single version of the trth for the organization. The focs of the corporate information factory is data across the enterprise. from many different places and applications the legacy systems environment is all integrated and inclded into the data warehose. One of the reasons why the corporate information factory is not ilt qickly is that data from lots of places needs to e integrated. For a small organization there may e very little integration that needs to occr. Bt for a large organization the process of integration can e laorios, tedios, and time consming. As a rle, the data in the corporate information factory data warehose is stored in a normalized relational format. Generally speaking, the data in the corporate information factory relational data ase is granlar, historical and is lightly denormalized. tated differently, ilding the corporate information factory is a long term proposition and the reslt is a long term infrastrctre that the corporation can rely pon.
The corporate information factory (and its evolved form - DW 2.0) is the architectre that is esposed and developed y Bill Inmon and expressed as the corporate information factory in his ook in 1999 and later in the ook DW 2.0 ARCHITECTURE FOR THE NEXT GENERATION OF DATA WAREHOUING. THE DIMENIONAL MODEL DATA WAREHOUE THE KIMBALL APPROACH Bt there was another related architectre that arose in roghly the same time frame. That architectre is the one that can e called the Kimall architectre. It is the Kimall architectre that is associated with Red Brick ystems. The Kimall architectre has evolved over time, like all architectres evolve. The first stage of evoltion of the Kimall architectre egan with what is known as a dimensional (or star schema) architectre. In the context of this paper we will call the first stage of evoltion of the Kimall architectre a simple dimensional model. Fig 2 shows the are one essence of a simple dimensional architectre. Fig 2 shows a fact tale srronded y several dimensions. In general, the facts are a clster of attrites that are physically colocated and the dimensions are the separate tales that descrie the facts. The fact tale and its dimensions form what is termed a star schema. As a rle there are many facts in the fact tales and relatively few occrrences of data in the dimensions. The data that comes from applications is placed in a star schema and is sed to create what is termed a data. The data that poplates the simple dimensional model comes directly from applications. In fact, Kimall draws a diagram that shows how application data enters the simple dimensional model. The diagram is taken from an article plished y Kimall in 2004, along with Margy Ross[1]. Fig 3 depicts the diagram showing how the simple dimensional model is sed to prodce mltiple data s from mltiple sorces. Order transaction ales data warehose Inventory snapshot Marketing data hipment transaction Financial data In Fig 3 it is seen that there are legacy applications and data s in the simple dimensional model. The many different data s are poplated directly from the many different applications. Kimall goes on
to give his definition of a data warehose. Kimall s definition relates to the first phase of the Kimall architectre the simple dimensional model - a data warehose is nothing more than the nion of the data s. *3+ Kimall refined the definition of the data warehose at a later point in time, saying that the definition of a data warehose was a a copy of the data specifically strctred for qery and analysis. *2+ It is easy and fast to merely copy data from one data ase to the next. Kimall s tage 1 simple dimensional architectre was never designed for enterprise integration. The Kimall tage 1 simple dimensional architectre was designed for immediate applications and immediate data s, where the scope of the effort was limited. Becase the scope of the Kimall tage 1 simple dimension architectre was limited and ecase only the copying of data was involved, Kimall s tage 1 simple dimensional data warehose is fast and easy to constrct. The iggest selling point of the Kimall simple dimensional architectre is the speed with which the data s can e constrcted. Indeed, arond the world, people like architectres that are easy to constrct and qick to e sed. The prolem with the simple dimensional architectre (and the nexs of the difference etween Inmon and Kimall) is that nowhere in the Kimall tage 1 simple dimensional architectre is there the notion of the single version of the trth. At est, Kimall says that application data shold e copied from the application environment. Inmon, on the other hand, sggests that a fndamental and rigoros transformation of legacy data is necessary in order to create the single version of the trth. When comparing the Kimall tage 1 simple dimensional architectre verss the Inmon corporate information factory, Inmon s data warehose reqires that there e a single version of the trth while Kimall s data warehose is a collection of data s consisting of data that has een copied from applications. And therein lies the difference etween the Inmon approach to data warehosing and the Kimall approach to data warehosing. DIFFERENCE BETWEEN THE MODEL The fndamental differences etween the Kimall tage 1 simple dimension architectre and the Inmon corporate information factory architectre can e smmed p as - The corporate information factory (Inmon) addresses the need for integration of data across the organization creating what can e called the single version of the trth. The focs of the Inmon corporate information factory is the integration of data across the corporation. - The Kimall tage 1 dimensional architectre is qick to ild and allows reports to e ilt qickly t does not reqire a single version of the trth e ilt, only that a copy of data from the legacy environment e made. The focs of the Kimall tage 1 simple dimensional model is on a few immediate applications from which data s can e ilt. ince the focs in the Kimall tage 1 dimensional architectre is on the speed with which a data can e prodced across a few applications, there is no time to ild a single version of the trth across the enterprise. There is no denying that a corporate information factory reqires mch more time and many more resorces to ild than a simple dimensional architectre, primarily ecase the scope of the corporate
information factory is enterprise wide. The Kimall style simple dimensional architectre is nqestionaly faster and easier to ild. Bt the Kimall tage 1 simple dimensional architectre does not contain the single version of the trth for the enterprise. For small organizations with a small amont of data the Kimall tage 1 simple dimensional architectre may e perfectly adeqate. Bt for larger organizations with larger amonts of data and a need for integration of data cross the enterprise, the Kimall tage 1 simple dimensional architectre soon ecomes prolematic. When the Kimall tage 1 simple dimensional architectre is applied to large systems, the lack of the single version of the trth and the lack of the aility to integrate data across the organization ecomes a large isse. THE IMPLE DIMENIONAL MODEL IN THE LARGE ENTERPRIE Consider what happens to the simple dimensional model in the face of a lot of data there are lots of legacy sorces and lots of data s. The model as illstrated y Kimall and Ross[1] in Fig 3 merely expands. In the face of a large organization, the diagram drawn y Kimall and Ross that depicts the simple dimensional model simply grows larger. And with that expansion comes some maor architectral prolems. Fig 4 depicts a Kimall tage 1 architectre for a large organization. It is at this point that the Kimall architectre egan to evolve into the next stage. Evoltion occrs ecase of the pain of prolems. And there were adeqate points of pain for large organizations that tried to implement the Kimall tage 1 simple dimensional architectre for an evoltion to occr. One of the motivations for evoltion is that there are many interface programs that are needed to spport a Kimall tage 1 simple dimensional architectre in a large organization. More pain arises when it comes time to maintain those interface programs. When the Kimall tage 1 architectre is ilt for a large organization, there is enormos redndancy of data, from one data to the next. Another
motivation for evoltion occrs when it is time to refresh data into the data s. The window of opportnity for refreshment contines to shrink on a nightly asis. Bt perhaps the most pain with the Kimall tage 1 simple dimensional architectre occrs ecase there is no corporately nderstood vale of data, no single version of the trth. In a large scale implementation of a Kimall tage 1 simple dimensional architectre, when an end ser wants to find a vale of data, the end ser literally has hndreds of places to trn to find that single vale of data. In the Kimall tage 1 simple dimensional architectre there is no one definitive place that states where a vale of data is or is not. Conseqently, a given vale of data can reside anywhere (or nowhere) in a Kimall tage 1 simple dimensional model. ince there is no definition of where there is a proper vale of data, there can e many versions of the same vale of data in a Kimall tage 1 simple dimensional model in a large organization. Needless to say, large confsion reslts when large organizations trn the Kimall tage 1 simple dimensional architectre into reality. If as Kimall sggests (in his own words) a data warehose is a nion of all the data s then there is a real prolem with the data warehose when it is ased on the Kimall tage 1 simple dimensional model. Fig 5 sggests the maor prolems that arise with the Kimall tage 1 data warehose for a large organization. (Note a small organization may not experience anywhere near the amont of grief that a large organization may experience. The size and the sophistication of the organization make a real difference in the amont of pain felt y an organization when it strggles with a Kimall tage 1 dimensional architectre.) - complex interface - no reconciliation - volmes of data ENTER THE CONFORMED DIMENION KIMBALL TAGE 2 ARCHITECTURE Recognizing the prolems that arise with realities of the implementations of the Kimall tage 1 simple dimensional model in large organizations, Kimall next sggests that what is really needed is a conformed dimension in addition to the star schema. The conformed dimension sets the stage for the next stage of evoltion of the Kimall architectre, the Kimall tage conformed dimension
architectre. The conformed dimension contains descriptive attrites and corresponding names. The prpose of the conformed dimension is to integrate the many data s that are prodced y the simple dimensional data model. With conformed dimensions Kimall starts to address the isse of integration. And with the isse of integration comes the isse of integration across the enterprise. And once the sect of integration across the enterprise is addressed, the speed with which the Kimall architectre can e implemented slows down exponentially. Yo simply cannot qickly and easily integrate data across the enterprise. o the attraction of speed of development of the Kimall architectre changes drastically in the face of a Kimall tage 2 conformed dimension architectre. In the face of a small organization, the need for integration across the organization may not e a large isse. Bt in the face of a large organization, the isse of integration across the organization is a very real and pressing isse. The reslt of introdcing conformed dimensions to the Kimall tage 1 dimensional architectre is the Kimall tage 2 architectre. Fig 6 shows the Kimall tage 2 conformed dimension architectre. With a conformed dimension The Kimall tage 2 conformed dimension architectre addresses the prolem of integration of data across the organization y introdcing conformed dimensions. With conformed dimensions it is possile to achieve a degree of integration. Bt there still are prolems with a Kimall tage 2 conformed dimension architectre. The prolem with the Kimall tage 2 conformed dimension architectre arises from the fact that conformed dimensions address only some attrites of the corporation, not all attrites of the corporation. There are many other attrites and data elements in the corporation that are not fond in conformed dimensions and those attrites need attention when it comes to integration. Bt conformed dimensions do not address all data elements, only some data elements. Fig 6 shows that in the portion of the Kimall tage 2 conformed dimension architectre that is not contained in conformed dimensions that there is tremendos redndancy of data, that there is a tremendos amont of nintegrated data, and that addressing conformed dimensions only addresses a small part of the general prolem of lack of integration of application data. In short, the data not fond in a conformed dimension is not integrated in a Kimall tage 2 conformed dimension architectre. Bt there was another maor isse with the tage 2 conformed dimension model. The prolem arises from the data s that are connected y a conformed dimension. The data s are process oriented
collections of data order processing, inventory, shipping and so forth. As sch, many data elements appear in more than one process oriented data. Even thogh the prolem of integration of some of the data elements were resolved y the creation of conformed dimensions, the prolem of integration of data elements that were not in the conformed dimension arose ecase of the process orientation of the data s. These isses with a Kimall tage 2 conformed dimension architectre are seen in Fig 7 ENTER MDM AND THE GOLDEN RECORD - redndancy within the data - no reconciliation While conformed dimensions are a first step to integration of corporate data, they are st that only a first step. What is needed is complete integration of ALL the corporate data needed for analytic processing. The key to creating a asis for all integration is MDM or master data management. With MDM there is the creation of what is sometimes referred to in MDM as the golden record. (NOTE: the term golden record is not a term that widely appears in the Kimall architectre, t is a term that appears in many other conversations regarding MDM. The term nevertheless descries the most salient aspect of MDM the need for a single, elievale sorce of corporate data.) The golden record in an MDM architectre is the place where the single version of the trth lies. Fig 8 shows a Kimall tage 3 MDM architectre. MDM
In the Kimall tage 3 MDM architectre it is seen that there is at last corporate, enterprise wide integration of data. With MDM, now the single version of the trth exists. At this point, the focs on speed of ilding is completely lost ecase trying to integrate data across the enterprise is not a speedy exercise nder any scenario. Even thogh the single version of the trth has een estalished in the Kimall architectre y the introdction of MDM, the evoltion of the Kimall architectre is not complete. Bt there is yet another prolem with the Kimall tage 3 MDM architectre. This isse presages a next stage of evoltion for the Kimall architectre. The prolem with the Kimall tage 3 MDM architectre is that many departments across the organization need to se the data fond in the non redndant MDM generated golden records for their analytic processing. In the world of MDM the orientation is to an organization arond integrated sect areas. is organized according to the maor sect areas of the corporation, sch as CUTOMER, PRODUCT, ORDER, HIPMENT and so forth. Across all of the MDM sect areas there is little or no redndancy of data. When organizations go to se the sect area data, they find that they need to recast the sect area data into a form and strctre for their own parochial processing needs. tated differently, even thogh the MDM sect area does spport the single version of the trth, the MDM golden records do not spport the many different ways that data needs to e viewed y the different departments of the organization. or this prpose, there is a simple architectral answer. In order to se the golden record across the organization for analytic processing in many different ways, departments may copy (t not pdate or otherwise alter) the data from the golden record. These cstomized copies of data from the golden record can e called data s. Those data s receive data that comes from the MDM golden records (i.e., the single version of the trth records) that are fond in the Kimall tage 3 integrated MDM data.) The data s are then recast into a form and strctre sitale for the individal departments that need to do analytical processing. The reslt is the predictale next evoltion of the Kimall architectre after the MDM has een estalished the Kimall tage 4 h and spoke architectre. Note that it is only a prediction that the Kimall tage 4 h and spoke architectre will evolve. Fig 9 depicts the predicted Kimall tage 4 h and spoke [4] architectre. The different stages of evoltion of the Kimall architectre can e seen in Fig 10.
The 4 stages of Kimall s architectre tage I tage II tage III tage IV Order transaction Inventory snapshot hipment transaction ales data warehose Marketing data Fin ancial data Dimensional model With a conformed dimension MDM with data s ome of the notale events/papers/ooks/definitions of the different stages of evoltion of the Kimall architectral approach are - 1992 Kimall tage 1 simple dimensional model phase Formation of Ralph Kimall Associates a data warehose is a nion of all its data s THE DATA WAREHOUE TOOLKIT, 1998 2002 Kimall tage 2 conformed dimension/master conformed dimension phase DATA WAREHOUE TOOLKIT: THE COMPLETE GUIDE TO DIMENIONAL MODELLING, 2002 Kimall Grop/Kimall University: Kimall Design tip #48, De-Clster with Jnk (Dimension), Ag 7, 2003 2007 Kimall tage 3 MDM phase Intelligent Enterprise: Kimall University, Pick The Right Approach To MDM Fe 2007 The Need For Master The Conformed Warehose The MDM Integration H The Enterprise MDM ystem For teps to MDM
THE EVOLVING KIMBALL ARCHITECTURE There is a certain irony here. Compare the predicted Kimall tage 4 h and spoke architectre with the corporate information factory architectre that was plished y Inmon a decade earlier and it is seen that they in fact are the same. The emphasis for the predicted Kimall tage 4 h and spoke architectre is now on integrated data, not on speed of development. The next irony is that the predicted Kimall tage 4 h and spoke architectre cannot e created qickly and easily. There has een a change in emphasis from Kimall tage 1 architectre to the predicted Kimall tage 4 architectre. In Kimall tage 1 the emphasis was on speed of development. Bt in the predicted Kimall tage 4 with the need for tre enterprise development and the creation of the golden record, ilding the Kimall tage 4 environment is no longer speedy. The emphasis on the tage 1 Kimall architectre is on a few legacy systems. The emphasis on the Kimall tage 4 architectre is on the enterprise. The emphasis for the predicted tage 4 Kimall model the need for integration across the enterprise - was the one that Inmon recognized 10 years earlier. PREDICTED KIMBALL TAGE 4 = CORPORATE INFORMATION FACTORY The predicted Kimall tage 4 architectre has evolved (and is still evolving) to the Inmon Corporate Information Factory. The Kimall tage 3 architectre and the predicted Kimall tage 4 h and spoke architectre is eing discssed in 2010. And the Inmon Corporate Information Factory was created in the 1990 s, more than a decade earlier. Over time, the asic Kimall dimensional architectre has ndergone several maor intellectal revoltions, all started y the realization that the asic dimensional architectre did not work in the face of large scale systems and that the simple dimensional model was not a tre enterprise soltion. That intellectal evoltion is depicted y Fig 11. Dimensional model Conformed dimension Master conformed dimension MDM ect area/data First there was the dimensional architectre. Then there was the conformed dimension. Then there was the master conformed dimension. Then there was MDM. Finally there is the predicted Kimall tage 4 h and spoke architectre. Throghot the renditions of the Kimall tage 1 tage 4 approach to data warehosing, the Kimall approach has een particlarly poplar with software vendors. In particlar the Bsiness Intelligence
data software vendors have een drawn to the original Kimall tage 1 simple dimensional architectre. There is a reason why data and Bsiness Intelligence vendors are drawn to the Kimall tage 1 simple dimensional architectre. That reason is the Bsiness Intelligence and data vendors care most of all aot making a sale. Consider the sales cycle for the data vendor in the face of an Inmon style corporate information factory architectre. In the Inmon architectre efore the data can e ilt, a data warehose has to e ilt. Bt ilding the Inmon style data warehose is going to take a while. Therefore, ilding an Inmon style data warehose gets in the way of the data vendor making a fast sale. On the other hand, with a Kimall dimensional model approach, the data is needed almost immediately. Is it any wonder then that the data, Bsiness Intelligence vendors gave all their spport to Kimall? It was in their own est interest to do so. tated differently, the data, Bsiness Intelligence vendors cared nothing for the long term architectral interests of their cstomers. All the data, Bsiness Intelligence vendors cared for was their own immediate ottom line making a qick sale, at the expense of their cstomers long term architectre. The Kimall dimensional tage 1 simple dimensional architectre was a natral fit for the fast ilding of data s. FITTING THE TWO ARCHITECTURE TOGETHER It is seen that there is a significant architectral difference etween the Inmon corporate information factory single version of the trth architectre and the Kimall tage 1 simple dimensional architectre. Despite the differences, there is a xtaposition of the two architectres that makes sense. Fig 12 shows this arrangement. Text Textal ETL Operational/ legacy applications MD ETL MD MD MD MD MD ETL MD ETLMD warehosemd Oper MD MD tar schema CDC MD ODMD MD Near line (alternate) Exploration warehose MD Relational model Archival MD
Fig 12 shows that in the center of the h is the Inmon corporate information factory. In the Inmon corporate information factory is the single version of the trth. The data here is granlar, historical and integrated. The data here is cast in the form of the relational model. rronding the single version of the trth are the data s. The data s are cast in the form of the Kimall star schema architectre. In the star schema architectre, each data is optimized to meet the analytical needs of the end ser. The sorce of data for each data is the data warehose. The asic architectre seen in Fig 12 meets the needs for a single version of the trth and for the different analytical needs of the different departments. And the architectre seen in Fig 12 lends the Inmon and Kimall architectre, taking the est featres of each architectre. However, the architectre seen in Fig 12 has een extended over the years into a mch more rost, mch more sophisticated architectre. The architectre seen in Fig 12 has een extended into what can e called DW 2.0. DW 2.0 Over the decade etween the creation of the corporate information factory and DW 2.0, the Inmon corporate information factory architectre has evolved as well. Today the Inmon architectre is est descried y the ody of work known as DW 2.0. Written in 2007, DW 2.0 is descried in a ook entitled DW 2.0 ARCHITECTURE FOR THE NEXT GENERATION OF DATA WAREHOUING. The essence of the DW 2.0 architectre is depicted in Fig 13.
DW 2.0 Architectre for the next generation of data warehosing Transaction data Interactive Very crrent A p p l A p p l A p p l Integrated Crrent++ Textal sects Internal, external Captred text Text id... imple pointer Detailed Continos snapshot data Profile data Linkage Text to s mmary Near line Less than crrent Textal sects Internal, external Captred text Text id... Linkage Text to s imple pointer Detailed mmary Continos snapshot data Profile data Archival Older Textal sects Internal, external Captred text Text id... imple pointer Detailed Continos snapshot data Profile data Linkage Text to s mmary The DW 2.0 architectre contains many different architectral components that have een added on to the asic corporate information factory. ome of the more salient aspects of the DW 2.0 architectre inclde
- Unstrctred data as an essential and granlar ingredient in the data warehose. - An exploration warehose - Near line (or alternate) storage - An archival component - Oper s - An OD - Metadata as an essential component of the architectre - Taxonomies - Changed data captre - Recognition of the life cycle of data within the data warehose. The DW 2.0 architectre then represents the evolving architectre for data warehose. It contains the est featres of the Inmon architectre and the Kimall architectre can e comined very adroitly. DW 2.0 represents a long term architectral leprint to meet the needs of modern corporations and modern organizations. Biliography Inmon BUILDING THE DATA WAREHOUE, John Wiley, 1991 THE CORPORATE INFORMATION FACTORY, John Wiley, 1999 OPERATIONAL DATA TORE, John Wiley, 1995 BUINE METADATA: CAPTURING ENTERPRIE KNOWLEDGE, Morgan Kafman, 2007 TAPPING INTO UNTRUCTURED DATA, Pearson, 2007 DW 2.0 ARCHITECTURE FOR THE NEXT GENERATION OF DATA WAREHOUE, Morgan Kafman, 2007 BUILDING THE UNTRUCTURED DATA WAREHOUE, Technics Plications, Nov 2010 Kimall [2][3] DATA WAREHOUE TOOLKIT, John Wiley, 1998 DATA WAREHOUE TOOLKIT: COMPLETE GUIDE TO DIMENIONAL MODELING, John Wiley, 2002 DATA WAREHOUE TOOLKIT: BUILDING THE WEB ENABLED DATA WAREHOUE, John Wiley, 2000 [1] Differences of Opinion: Comparing the Dominant Approaches to Enterprise Warehosing, Intelligent Enterprise magazine, 2004 [4] Internet Planning MDM and EDW with Dr Kimall for 2010- Informatica