2.1 Basics: Indexing. 2.1 Primary Index. 2.1 Secondary Index. 2.1 Secondary Index. 2.1 Indexes. 2.1 Indexes

Transcription

1 2. Architecture Data Warehousing & Mining Techniques Wolf-Tilo Balke Silviu Homoceanu Institut für Informationssysteme Technische Universität Braunschweig 2. Architecture 2.1 Basics 2.2 Storage structures 2.3 Tier architectures Data Warehousing & OLAP Wolf-TiloBalke InstitutfürInformationssysteme TU Braunschweig Basics Architecture of a DW Data is stored in a predefined database Maintenance of the database is performed as in OLTP by a DBMS Usual functionality of the database is ensured Storage, Update, Delete, Locate Individually Structured Departmentally Structured Organizationally Structured Source Systems 2.1 Basics: Databases & DBMS Databases & DBMS Application SELECT id FROM revenues WHERE val> DBMS Disk 1 Disk N ID VAL Data Warehousing & OLAP Wolf-TiloBalke InstitutfürInformationssysteme TU Braunschweig 3 Data Warehousing & OLAP Wolf-TiloBalke InstitutfürInformationssysteme TU Braunschweig Basics: Indexing DW is characterized by Large volume of information Mostly used for reading the information and not for updating or deleting operations outside the ETL phase This characteristics suggest indexes as a must have in DW so let s remember indexes 2.1 Basics: Indexing Indexes are additional data structures which help locating records in a DB Creation of indexes is part of the physical tuning task of the DB administrators Indexes can influence the actual location of storage for a record Sequential storage, or via a hash function If the location is determined by the index not all attributes can be directly indexed (primary vs. secondary indexes) Data Warehousing & OLAP Wolf-TiloBalke InstitutfürInformationssysteme TU Braunschweig 5 Data Warehousing & OLAP Wolf-TiloBalke InstitutfürInformationssysteme TU Braunschweig 6 1

2 2.1 Basics: Indexing 2.1 Primary Index Indexes are useful for speeding up access to the data They are ordered by indexing field (search key) search key is the attribute used to look up records into a file An index file consists of index entries Records of the form search key location Primary indexes Order data by some unique attribute as indexing field (primary key), store database records in this order An index record contains a pointer to the respective storage place Data Warehousing & OLAP Wolf-TiloBalke InstitutfürInformationssysteme TU Braunschweig 7 Data Warehousing & OLAP Wolf-TiloBalke InstitutfürInformationssysteme TU Braunschweig Secondary Index 2.1 Secondary Index Secondary indexes point to locations of records regarding non-ordering attribute Indexing does not affect storage order There can be multiple secondary indexes for the same DB file Secondary indexes are usually dense Characteristics of secondary indexes Speeds up retrieval, if secondary index on the searched attribute does not exist, the entire file has to be searched linearly Use more time and space, because they are dense Provide logical ordering Accessing records in this order might not be the most efficient way regarding block accesses In DW, due to the large amount of data, multilevel ordered indexes are used Data Warehousing & OLAP Wolf-TiloBalke InstitutfürInformationssysteme TU Braunschweig 9 Data Warehousing & OLAP Wolf-TiloBalke InstitutfürInformationssysteme TU Braunschweig Indexes Here s a great idea: Why not index every attribute? Have a physical index on the primary key, and logical indexes on every other attribute This results in good read efficiency, but really terrible write/update efficiency But Data Warehouses only need good read efficiency? 2.1 Indexes Whenever a DB is modified, most of the indexes have to be updated This result in a large amount of overhead on operations like insert, delete or update If the indexes are multi-level every level has to be updated Why should we care? We have a DW not an OLTP system The majority of the operations in a DW are reads But remember ETL? We should use considerable more indexes than in OLTP, but loading data into the DW should not last forever!!! Data Warehousing & OLAP Wolf-TiloBalke InstitutfürInformationssysteme TU Braunschweig 11 Data Warehousing & OLAP Wolf-TiloBalke InstitutfürInformationssysteme TU Braunschweig 12 2

3 2.1 Indexes in DW In DW the underlying technology has to support Creation and loading of new indexes Efficient access to the indexes Efficient access can be accomplished in different ways Using bit maps Having multi-leveled indexes Storing all or parts of an index in main memory Compacting the index entries when the order of the data being indexed allows such compaction Creating selective indexes and range indexes 2.1 Indexes in DW Recommended index structures are: B-tree indexes, on high cardinality attribute columns (due to the bushy nature of B-Trees) Bitmap indexes on all medium and low cardinality attributes Data Warehousing & OLAP Wolf-TiloBalke InstitutfürInformationssysteme TU Braunschweig 13 Data Warehousing & OLAP Wolf-TiloBalke InstitutfürInformationssysteme TU Braunschweig B-Trees Rel. DB 2: Basic structure of a B-Tree node Node contains key values and respective data (block) pointers to the actual data records Additionally, there are node pointers for the left respectively right interval around a key value Key Value Data Pointer Node Pointers Tree Node Data Warehousing & OLAP Wolf-TiloBalke InstitutfürInformationssysteme TU Braunschweig Bitmap indexes Bitmap indexes: Work well with small number of distinct values E.g., gender data Identifier Gender Bitmaps 1 Female Female Male Unspecified Male 0 1 Have a significant space and performance advantage over other structures for this type of data Useful in DW for joining a large fact table to smaller dimension tables Data Warehousing & OLAP Wolf-TiloBalke InstitutfürInformationssysteme TU Braunschweig 16 F M 2.1 Basic architecture 2.1 Basic architecture Architecture of a DW Data Sources Staging Area Operational System Operational System Flat files Summary Data Warehouse Metadata Raw Data Purchasing Sales Inventory Users Analysis Reporting Mining Sources Staging Warehouse Area The Data Staging Area Is both a storage and process area (the ETL process) It represents everything that happens between the operational source system and the data presentation area The key architectural requirement for data staging area is that it is off-limits to business users and does not provide query and presentation services Data Data Warehousing & OLAP Wolf-TiloBalke InstitutfürInformationssysteme TU Braunschweig 17 Data Warehousing & OLAP Wolf-TiloBalke InstitutfürInformationssysteme TU Braunschweig 18 3

4 2.1 Basic architecture Customers aren t invited to visit the kitchen Similar to a restaurant s kitchen, the data staging area should be accessible only to skilled professionals 2.1 Basic architecture The Data Presentation Area Data Sources Staging Area Warehouse Is where data is organized, stored and made available for queries, report writers, and other analytical processing This area is the Warehouse as far as the business community is concerned Data Warehousing & OLAP Wolf-TiloBalke InstitutfürInformationssysteme TU Braunschweig 19 Data Warehousing & OLAP Wolf-TiloBalke InstitutfürInformationssysteme TU Braunschweig Storage structures Storage structure After extraction from the operational data, in DW information is stored in databases The databases are operated by a DBMS Different database structures can be used for a DW: Relational model (RDB) operated by a RDBMS MultiDimensional model (MDB) operated by a MDBMS 2.2 Storage structures RDB and MDB are complementary and do not have to exclude each other In the staging area some RDBMS can be used, however it must be off-limits to user queries because of performance reasons By default, normalized databases are excluded from the presentation area, which should be strictly multi-dimensionally (MDBMS) Data Warehousing & OLAP Wolf-TiloBalke InstitutfürInformationssysteme TU Braunschweig 21 Data Warehousing & OLAP Wolf-TiloBalke InstitutfürInformationssysteme TU Braunschweig Relational DB DB in relational model A database is seen as a collection of predicates over a finite set of variables The content of the DB is modeled as a set of relations in which all predicates are satisfied Books Title ISBN (PK) Price Publisher(FK) Category (FK) Publisher Name ID (PK) BookCategory Cat_ID(PK) Description Data Warehousing & OLAP Wolf-TiloBalke InstitutfürInformationssysteme TU Braunschweig 23 A relation is defined as a set of tuples that have the same attributes It is usually described as a table Tuple 2.2 Relational DB Attribute Relation Data Warehousing & OLAP Wolf-TiloBalke InstitutfürInformationssysteme TU Braunschweig 24 4

5 A Multidimensional DB (MDB) is optimized for DW and OLAP applications They are created using input from the staging area Their purpose is to answer questions like How many Nokia 5800 have we sold so far this year in Braunschweig? MDBs are RDBS optimized for OLAP queries MDB are Designed for efficient and convenient storage and retrieval of large volumes of data Stored, viewed and analyzed from different perspectives called dimensions Data Warehousing & OLAP Wolf-TiloBalke InstitutfürInformationssysteme TU Braunschweig 25 Data Warehousing & OLAP Wolf-TiloBalke InstitutfürInformationssysteme TU Braunschweig 26 MDB example An automobile manufacturer wants to increase sale volumes Evaluation requires to view historical sale volume figures from multiple dimensions Sales volume by model, by color, by dealer, over time A relational structure of the given evaluation would be Model Color Sales volume Mini VAN Blue 324 Mini VAN Black 113 Mini VAN Red 18 Sedan Black 160 Sedan Blue 115 Sedan Red 6 Sports coupe Red 16 Sports coupe Black 16 Sports coupe Blue 12 Data Warehousing & OLAP Wolf-TiloBalke InstitutfürInformationssysteme TU Braunschweig 27 Data Warehousing & OLAP Wolf-TiloBalke InstitutfürInformationssysteme TU Braunschweig Multidimensional structure * Mini VAN Sedan Coupe Black Blue Red * The complexity grows quickly with the number of dimensions and the number of positions Example: 3 dimensions with 10 values each and no indexes If we consider viewing information in a RDB it would result in a worst case of 10 3 =1000 records view Data Warehousing & OLAP Wolf-TiloBalke InstitutfürInformationssysteme TU Braunschweig 29 Data Warehousing & OLAP Wolf-TiloBalke InstitutfürInformationssysteme TU Braunschweig 30 5

6 Now, if we consider performance For responding to a query when car type = Sedan, color = Blue, and dealer = Berg RDBMS has to search through 1000 records to find the right record MDB has more knowledge about where data lies The maximum of searches in the case of MDB is of 30 positions Average case 18 vs. 501 positions If the query is more relaxed Total sales across all dealers for all colors when car type = sedan RDBMS still has to go through the 1000 records MDB, however, goes only through a slice of 10x10 Data Warehousing & OLAP Wolf-TiloBalke InstitutfürInformationssysteme TU Braunschweig 31 Data Warehousing & OLAP Wolf-TiloBalke InstitutfürInformationssysteme TU Braunschweig RDB vs. MDB Performance advantages MDBs are an order of magnitude faster than RDBMSs Performance benefits are more for queries that generate cross-tab views of data (the case of DW) Conclusion The performance advantages offered by MDBs facilitates the development of interactive decision support applications like OLAP that can be impractical in a relational environment Any database manipulation is possible with both technologies MDBs however offer some advantages in the context of DW: Ease of data presentation Ease of maintenance Performance Data Warehousing & OLAP Wolf-TiloBalke InstitutfürInformationssysteme TU Braunschweig 33 Data Warehousing & OLAP Wolf-TiloBalke InstitutfürInformationssysteme TU Braunschweig RDB vs. MDB 2.2 RDB vs. MDB Ease of data presentation Data views are natural output of the MDBs Obtaining the same views in RDB requires a complex query Example with Walmart and Sybase: select sum(sales.quantity_sold) from sales, products, product_categories, manufacturers, stores, cities where manufacturer_name = Colgate and product_category_name = toothpaste and cities.population < and trunc(sales.date_time_of_sale) = trunc(sysdate-1) and sales.product_id = products.product_id and sales.store_id = stores.store_id and products.product_category_id = product_categories.product_category_id and products.manufacturer_id = manufacturers.manufacturer_id and stores.city_id = cities.city_id Ease of data presentation Top k queries cannot be expressed well in SQL Find the five cheapest hotels in Frankfurt SELECT * FROM hotels h WHERE h.city = Frankfurt AND 5 > (SELECT count(*) FROM hotels h1 WHERE h1.city = Frankfurt AND h1.price < h.price); Some RDBMS extended the functionality of SQL with STOP AFTER functionality SELECT * FROM hotels WHERE city = Frankfurt Order By price STOP AFTER 5; Data Warehousing & OLAP Wolf-TiloBalke InstitutfürInformationssysteme TU Braunschweig 35 Data Warehousing & OLAP Wolf-TiloBalke InstitutfürInformationssysteme TU Braunschweig 36 6

7 2.2 RDB vs. MDB Ease of maintenance No additional overhead to translate user queries into requests for data Data is stored as it is viewed RDBs use indexes and sophisticated joins which require significant maintenance and storage to provide same intuitiveness 2.2 RDB vs. MDB Performance Performance of MDBs can be matched by RDBs through database tuning Not possible to tune the database for all possible adhoc queries Aggregate navigators are helping RDBs to catch up with MDBs as far as aggregation queries are concerned Data Warehousing & OLAP Wolf-TiloBalke InstitutfürInformationssysteme TU Braunschweig 37 Data Warehousing & OLAP Wolf-TiloBalke InstitutfürInformationssysteme TU Braunschweig 38 When MDBs are in-appropriate? If the dataset types are not highly related, using a MDB results in a sparse representation Mini VAN 2.2 MDB Sedan Coupe Black Blue Red Smith James Fox MDB When MDBs are appropriate? In the case of highly interrelated dataset types MDBs are recommended for greatest ease of access and analysis Examples of applications Financial Analysis and Reporting Budgeting Promotion Tracking Quality Assurance and Quality Control Product Profitability Data Warehousing & OLAP Wolf-TiloBalke InstitutfürInformationssysteme TU Braunschweig 39 Data Warehousing & OLAP Wolf-TiloBalke InstitutfürInformationssysteme TU Braunschweig Tier architectures Popular DW architectures Generic Two-Tier Architecture Independent Data Mart Dependent Data Mart and Operational Data Store Logical Data Mart and Active Warehouse Three-Tier Architecture Other One-Tier Architecture N-Tier Architecture Web-based Architecture Generic Two-Tier Architecture Data is not completely current in the DW Periodic extraction Data Warehousing & OLAP Wolf-TiloBalke InstitutfürInformationssysteme TU Braunschweig 41 Data Warehousing & OLAP Wolf-TiloBalke InstitutfürInformationssysteme TU Braunschweig 42 7

8 Data analysis comes in two flavors Depending on the execution place of the analysis Thin Client Analytics are executed on the server Client just displays This architecture fits well for Internet/Intranet DW access Client HTTP, IIOP Server Fat Client The server just delivers the data Analytics are executed on the client Communication between client and server must be able to sustain large data transfers Client ODBC, JDBC, NFS Server Analysis Analysis Data storage Data storage Data Warehousing & OLAP Wolf-TiloBalke InstitutfürInformationssysteme TU Braunschweig 43 Data Warehousing & OLAP Wolf-TiloBalke InstitutfürInformationssysteme TU Braunschweig 44 Independent Data Mart Mini warehouses limited in scope Separate ETL for each independent Data Mart High access complexity Dependent Data Mart and Operational Data Store Single ETL for the DW are loaded from the DW More simple data access than in the previous case Data Warehousing & OLAP Wolf-TiloBalke InstitutfürInformationssysteme TU Braunschweig 45 Data Warehousing & OLAP Wolf-TiloBalke InstitutfürInformationssysteme TU Braunschweig DW vs. Scope Subjects Logical Data Mart and Active Warehouse The ETL is near real-time are not separate databases, but logical views of the DW DW Application independent Centralized, Planned DW Historical, detailed, summarized Lightly denormalized Specific DSS application Decentralized by user area Organic, possibly not planned Data Some history, detailed, summarized Highly denormalized DW Multiple subjects DW Many internal and external sources DW Flexible Data-oriented Long life Sources Onecentral subject Few internal and external sources Other characteristics Restrictive Project oriented Short life Large Start small, becomes large Single complex structure Multiple, semi-complex structure, together complex Data Warehousing & OLAP Wolf-TiloBalke InstitutfürInformationssysteme TU Braunschweig 47 Data Warehousing & OLAP Wolf-TiloBalke InstitutfürInformationssysteme TU Braunschweig 48 8

9 Generic Three-Tier Architecture Derived data Data that had been selected, formatted, and aggregated for DSS support Reconciled data Detailed, current data intended to be the single, authoritative source for all decision support Derived data Data Mart Reconciled data DW and ODS Operational data Operational data Data mart metadata DW metadata Operational metadata One-Tier Architecture Theoretically possible Might be interesting for mobile applications N-Tier Architecture Higher tier architecture is also possible But the complexity grows with the number of tier-interfaces Web-based Architecture Advantages: Usage of existing software, reduction of costs, platform independence Disadvantages: Security issues: data encryption/user access and identification Data Warehousing & OLAP Wolf-TiloBalke InstitutfürInformationssysteme TU Braunschweig 49 Data Warehousing & OLAP Wolf-TiloBalke InstitutfürInformationssysteme TU Braunschweig 50 In most cases the economics and technology greatly favor a single centralized DW But in some cases, distributed DW make sense Types of distributed DW Geographically distributed Local DW/global DW Technologically distributed DW Logically one DW, physically more DW Independently evolving distributed DW Uncontrolled growth Geographically distributed In the case of corporations spread around the world Information is needed both locally and globally A distributed DW makes sense When much processing occurs at the local level Even though local branches report to the same balance sheet, the local organizations are their own companies Data Warehousing & OLAP Wolf-TiloBalke InstitutfürInformationssysteme TU Braunschweig 51 Data Warehousing & OLAP Wolf-TiloBalke InstitutfürInformationssysteme TU Braunschweig 52 All IBM Europe Local DW Site A Local operational processing IBM/Teradata USA Local DW HQ Local operational processing Sybase Asia Local DW Site B Local operational processing Technologically distributed DW Placing the DW on the distributed technology of a vendor Advantages The entry cost is cheap large centralized hardware is expensive No theoretical limit to how much data can be placed in the DW we can add new servers to the network Global DW Data Warehousing & OLAP Wolf-TiloBalke InstitutfürInformationssysteme TU Braunschweig 53 Data Warehousing & OLAP Wolf-TiloBalke InstitutfürInformationssysteme TU Braunschweig 54 9

10 As the DW starts to expand network data communication starts playing an important role Example: Let s simplify and consider we have 4 nodes holding each data regarding the last 4 years Now let s consider we have a query which needs to access the data from the last 4 years: 2008 such a query arises the issue 2007 of transporting large amount of data between processors Independently evolving distributed DW In practice there are many cases in which independent DW are developed concurrently and uncontrolled in the same organization The first step many corporations make is to build a DW for financial or marketing Once it is successfully set up, other parts of the organization follow independently the process resulting in the coexistence of more independent DW in the same organization This problem will be addressed later Data Warehousing & OLAP Wolf-TiloBalke InstitutfürInformationssysteme TU Braunschweig 55 Data Warehousing & OLAP Wolf-TiloBalke InstitutfürInformationssysteme TU Braunschweig 56 Middleware-Systems Provide an inter-connectivity layer between heterogeneous platforms and the applications that come on top Application Application Application Platform-interface Platform -Hardware - Operating System APIs Middleware-System Platform-interface Platform -Hardware - Operating System Middleware in DW? DW usually implies Heterogeneous hardware, databases, operating systems, networks and applications Middleware serves both users and developers It shields both users and developers from differences in services and resources used by applications Without middleware Changes at the lower layers could imply propagating changes by updating the higher layers Data Warehousing & OLAP Wolf-TiloBalke InstitutfürInformationssysteme TU Braunschweig 57 Data Warehousing & OLAP Wolf-TiloBalke InstitutfürInformationssysteme TU Braunschweig 58 Roles of Middleware Assist the developer in ETL and populating the DW Assist DW users in accessing the DW It is therefore needed at different points in the life cycle Types: Copy management: data extraction, transformation, Gateways: DB and independent gateways Program to program: RPC, ORBs Message oriented Most common middleware technologies CORBA (Common Object Request Broker Architecture) DCOM (Distributed Component Object Model) J2EE in DW Data Warehousing & OLAP Wolf-TiloBalke InstitutfürInformationssysteme TU Braunschweig 59 Data Warehousing & OLAP Wolf-TiloBalke InstitutfürInformationssysteme TU Braunschweig 60 10

11 CORBA Mechanism for normalizing method-call semantics between application objects on the same host or on remote host ORB Is a middleware technology that manages communication and data exchange between objects in object-oriented programming and databases (client) main() Object reference Generated stub code ORB network (server) main() Object implementation reference Generated skeleton code ORB ORB vendor code ORB vendor-tool generated code User-defined application code Client app ORB Establish connection Client Service communication Object implementation (service) Data Warehousing & OLAP Wolf-TiloBalke InstitutfürInformationssysteme TU Braunschweig 61 Data Warehousing & OLAP Wolf-TiloBalke InstitutfürInformationssysteme TU Braunschweig How CORBA works Client The application program that invokes a method or operation on an object implementation Stub Precompiled interface between the client and the ORB, generated by the ORB tool ORB An interface containing help functions and APIs that can be used by a client or an object implementation BOA (Basic Object Adapter) Refers to the part of the ORB responsible for managing server-side operations Replaced by the POA (Portable Object Adapter) Skeleton The server-side analogue of stubs Implementation Called a service or method in object-oriented terminology, defines the operations supporting an interface definition language(idl) 1. the client makes the call through the stub to the ORB 2. the ORB dispatches the call to the BOA, that does the object activation 3. the implementation registers itself, if necessary, and declares itself ready 4. the BOA, now signaled ready, invokes the implementation via the skeleton from IDL 5. a response or exception propagates up to the client caller. 5 Client Stub ORB BOA Skeleton Implementation Data Warehousing & OLAP Wolf-TiloBalke InstitutfürInformationssysteme TU Braunschweig 63 Data Warehousing & OLAP Wolf-TiloBalke InstitutfürInformationssysteme TU Braunschweig 64 CORBA in DW Query Service Supports SQL and OQL Object Transaction Service Ensure correct state of transactional objects Distributed commit/rollback Guarantees ACID properties It is able to send copies of multidimensional data DCOM Microsoft's concurrent for CORBA Can access distributed stored data through ADO (ActiveX Data Objects) ADO uses for the actual database access OLE DB (Object Linking and Embeding DB) and ODBC (Open DB Conectivity) There is also a multidimensional ADO ADO MD It contains objects for communication of data cubes Data Warehousing & OLAP Wolf-TiloBalke InstitutfürInformationssysteme TU Braunschweig 65 Data Warehousing & OLAP Wolf-TiloBalke InstitutfürInformationssysteme TU Braunschweig 66 11

12 J2EE in DW Not fit for storage and analyze of a multidimensional DB JOLAP offers a programming interface for analytical access to the DW A Java community initiative, sustained by SUN and Oracle Lack of effective support OLAP4J Is simply put a multidimensional JDBC So why is middleware important? Heterogeneous Hardware, Data sources, Data targets, Platforms, Operating systems Communication protocols Connectivity Platform and Application independence Support of standard protocols and interfaces Data Warehousing & OLAP Wolf-TiloBalke InstitutfürInformationssysteme TU Braunschweig 67 Data Warehousing & OLAP Wolf-TiloBalke InstitutfürInformationssysteme TU Braunschweig 68 Next lecture Modeling Basics of data modeling Data models in DW Data Warehousing & OLAP Wolf-TiloBalke InstitutfürInformationssysteme TU Braunschweig 69 12