Data Warehousing. Technological Education Institution of Larisa in collaboration with Staffordshire University Larisa Dr.

Transcription

1 Data Warehousing Technological Education Institution of Larisa in collaboration with Staffordshire University Larisa Dr. Theodoros Mitakos

2 AGENDA DATA WAREHOUSES DATA MINING

3 INTRODUCTION There is a great need to provide decision makers with information at the correct level of detail to support decision making Often data comes from multiple sources. In comparison to traditional databases, data warehouses generally contain very large amounts of data from multiple sources that may include databases from different data models and sometimes files acquired from independent systems and platforms. Data warehouses have the distinguishing characteristic that they are mainly intended for decision-support applications. They are optimized for data retrieval, not routine transaction processing Data warehouses are nonvolatile. That means that information in the data warehouse changes far less often that traditional systems and may be regarded as non-real-time with periodic updating

4 Definitions Data warehouses have been developed in numerous organizations to meet particular needs, there is no single, canonical definition of the term data warehouse. A data warehouse is characterized as "a subject-oriented, integrated, nonvolatile, timevariant collection of data in support of management's decisions." Data warehouses provide access to data for complex analysis, knowledge discovery, and decision making

5 OLAP-DATA MINING-DSS-OLTP OLAP (online analytical processing) is a term used to describe the analysis of complex data from the data warehouse. DSS (decision-support systems) also known as EIS (executive information systems) support an organization's leading decision makers with higher level data for complex and important decisions. Data mining is the set of techniques used for knowledge discovery, the process of searching data for unanticipated new knowledge. OLTP (online transaction processing), includes insertions, updates, and deletions to Traditional databases, while also supporting information query requirements.

6 Characteristics of DW multidimensional conceptual view generic dimensionality unlimited dimensions and aggregation levels unrestricted cross-dimensional operations dynamic sparse matrix handling client- server architecture multi-user support accessibility transparency intuitive data manipulation consistent reporting performance flexible reporting

7 Types of DW Enterprise-wide data warehouses are huge projects requiring massive investment of time and resources. Virtual data warehouses provide views of operational databases that are materialized for efficient access. Data marts generally are targeted to a subset of the organization, such as a department, and are more tightly focused.

8 Data modeling Two dimensional matrix model Three dimensional cube model

9 Pivoting Changing from one dimensional hierarchy (orientation) to another is easily accomplished in a data cube by a technique called pivoting (also called rotation). In this technique the data cube can be thought of as rotating to show a different orientation of the axes.

10 Roll up Drill down functions Roll-up display moves up the hierarchy, grouping into larger units along a dimension A drill-down display provides the opposite capability, furnishing a finer-grained view

11 Multidimensional storage model The multidimensional storage model involves two types of tables: dimension tables and fact tables. A dimension table consists of tuples of attributes of the dimension. A fact table can be thought of as having tuples, one per a recorded fact

12 Multidimensional Schemas A star schema The star schema consists of a fact table with a single table for each dimension Dimension tables

13 Multidimensional Schemas A snowflake schema The snowflake schema is a variation on the star schema in which the dimensional tables from a star schema are organized into a hierarchy by normalizing them. Some installations are normalizing data warehouses up to the third normal form so that they can access the data warehouse to the finest level of detail. Dimension tables

14 Multidimensional Schemas A fact constellation A fact constellation is a set of fact tables that share some dimension tables

15 Building a DW Query language design Acquisition of data Storage of data Other design considerations

16 Acquisition of data The data must be extracted from multiple, heterogeneous sources Data must be formatted for consistency within the warehouse. Names, meanings, and domains of data from unrelated sources must be reconciled The data must be cleaned to ensure validity The data must be fitted into the data model of the warehouse. Data may have to be converted from relational, object-oriented, or legacy databases (network and/or hierarchical) to a multidimensional model. The data must be loaded into the warehouse. The refresh policy will probably emerge as a compromise that takes into account the answers to the following questions: How up-to-date must the data be? Can the warehouse go off-line, and for how long? What are the data interdependencies? What is the storage availability? What are the distribution requirements (such as for replication and partitioning)? What is the loading time (including cleaning, formatting, copying, transmitting, and overhead such as index rebuilding)?

17 Storage of data Storing the data according to the data model of the warehouse Creating and maintaining required data structures Creating and maintaining appropriate access paths Providing for time-variant data as new data are added Supporting the updating of warehouse data Refreshing the data Purging data

18 Other design considerations Usage projections The fit of the data model Characteristics of available sources Design of the metadata component Modular component design Design for manageability and change Considerations of distributed and parallel architecture

19 Typical functionality of DW Roll-up: Data is summarized with increasing generalization (e.g., weekly to quarterly to annually). Drill-down: Increasing levels of detail are revealed (the complement of roll-up). Pivot: Cross tabulation (also referred as rotation) is performed. Slice and dice: Performing projection operations on the dimensions. Sorting: Data is sorted by ordinal value. Selection: Data is available by value or range. Derived (computed) attributes: Attributes are computed by operations on stored and derived values.

20 Data Warehouses versus Views Data warehouses are different from views in the following ways: Data warehouses exist as persistent storage instead of being materialized on demand. Data warehouses are not usually relational, but rather multidimensional. Views of a relational database are relational. Data warehouses can be indexed to optimize performance. Views cannot be indexed independent of the underlying databases. Data warehouses characteristically provide specific support of functionality; views cannot. Data warehouses provide large amounts of integrated and often temporal data, generally more than is contained in one database, whereas views are an extract of a database.

21 Difficulties of implementing DW The administration of a data warehouse is an intensive enterprise, proportional to the size and complexity of the warehouse. A team of highly skilled technical experts with overlapping areas of expertise will likely be needed, rather than a single individual. A significant issue in data warehousing is the quality control of data. Both quality and consistency of data are major concerns. Every time a source database changes, the data warehouse administrator must consider the possible interactions with other elements of the warehouse. Usage projections should be estimated conservatively prior to construction of the data warehouse and should be revised continually to reflect current requirements. Because there is continual rapid change in technologies, both the requirements and capabilities of the warehouse will change considerably over time. Design of the management function and selection of the management team for a database warehouse are crucial.

22 Open Issues Old problems receive new emphasis; for example, data cleaning, indexing, partitioning, and views receive renewed attention. Academic research into data warehousing technologies will likely focus on automating aspects of the warehouse that currently require significant manual intervention, such as the data acquisition, data quality management, selection and construction of appropriate access paths and structures, self-maintainability, functionality, and performance optimization. Incorporation of domain and business rules appropriately into the warehouse creation and maintenance process may make it more intelligent, relevant, and self- governing.

23 Data mining Data mining refers to the mining or discovery of new information in terms of patterns or rules from vast amounts of data. To be practically useful, data mining must be carried out efficiently on large files and databases.

24 Data mining versus data warehousing The goal of a data warehouse is to support decision making with data. Data mining helps in extracting meaningful new patterns that cannot be found necessarily by merely querying or processing data or metadata in the data warehouse. Data mining applications should considered early, during the design of a data warehouse

25 The knowledge discovery process The KDD, process comprises six phases: cleansing, enrichment, data transformation or encoding, data mining, and the reporting and display of the discovered information.

26 Data mining results Association rules for example, whenever a customer buys video equipment, he or she also buys another electronic gadget. Sequential patterns for example, suppose a customer buys a camera, and within three months he or she buys photographic supplies, then within six months he is likely to buy an accessory item. This defines a sequential pattern of transactions. A customer who buys more than twice in the lean periods may be likely to buy at least once during the Christmas period. Classification trees for example, customers may be classified by frequency of visits, by types of financing used, by amount of purchase, or by affinity for types of items, and some revealing statistics may be generated for such classes.

27 Goals of Data Mining and Knowledge Discovery Prediction Data mining can show how certain attributes within the data will behave in the future. Examples of predictive data mining include the analysis of buying transactions to predict what consumers will buy under certain discounts, how much sales volume a store would generate in a given period, and whether deleting a product line would yield more profits Identification Data patterns can be used to identify the existence of an item, an event, or an activity. E.g. In biological applications, existence of a gene maybe identified by certain sequences of nucleotide symbols in the DNA sequence. Classification Data mining can partition the data so that different classes or categories can be identified based on combinations of parameters. For example, customers in a supermarket can be categorized into discount-seeking shoppers, shoppers in a rush, loyal regular shoppers, shoppers attached to name brands, and infrequent shoppers. Optimization One eventual goal of data mining may be to optimize the use of limited resources such as time, space, money, or materials and to maximize output variables such as sales or profits under a given set of constraints.

28 Types of knowledge The term "knowledge" is very broadly interpreted as involving some degree of intelligence Deductive knowledge deduces new information based on applying pre-specified logical rules of deduction on the given data. Inductive knowledge, which discovers new rules and patterns from the supplied data (data mining).

29 Describing knowledge in Data mining Association rules These rules correlate the presence of a set of items with another range of values for another set of variables. Examples: (1) When a female retail shopper buys a handbag, she is likely to buy shoes Classification hierarchies The goal is to work from an existing set of events or transactions to create a hierarchy of classes. Examples: (1) A population may be divided into five ranges of credit worthiness based on a history of previous credit transactions. Sequential patterns A sequence of actions or events is sought. Example: If a patient underwent cardiac bypass surgery for blocked arteries and an aneurysm and later developed high blood urea within a year of surgery, he or she is likely to suffer from kidney failure within the next 18 months. Patterns within time series Similarities can be detected within positions of a time series of data, which is a sequence of data taken at regular intervals such as daily sales or daily closing stock prices. Examples: (1) Two products show the same selling pattern in summer but a different one in winter. Clustering A given population of events or items can be partitioned (segmented) into sets of "similar" elements. Examples: (1) An entire population of treatment data on a disease may be divided into groups based on the similarity of side effects produced.

30 Association Rules Apriori Algorithm Sampling algorithm Frequent-Pattern Tree Algorithm Partition Algorithm Negative associations Multidimensional associations

31 Applications of data mining Marketing Finance Manufacturing Health care