Extract, Transform, Load (ETL)

Transcription

1 Extract, Transform, Load (ETL) Overview General ETL issues: Data staging area (DSA) Building dimensions Building fact tables Extract Load Transformation/cleaning Commercial (and open-source) tools The AJAX data cleaning and transformation framework

2 Design phase DW Phases Modeling, DB design, source selection, Loading phase First load/population of the DW Based on all data in sources Refreshment phase Keep the DW up-to-date wrt. source data changes The ETL Process The most underestimated process in DW development The most time-consuming process in DW development Often, 80% of development time is spent on ETL Extract Extract relevant data Transform Transform data to DW format Build keys, etc. Cleansing of data Load Load data into DW Build aggregates, etc.

3 DW Architecture other sources Operational DBs Data Staging Metadata Extract Transform Load Refresh Monitor & Integrator Data Warehouse OLAP Server Serve Analysis Query Reports Data mining Data Marts Data Sources Data Storage OLAP Engine Front-End Tools ETL process other sources Operational DBs Data Staging Extract Transform Load Refresh Data Warehouse Implemented using an ETL tool! Ex: SQLServer 2005 Integration Services

4 Data Staging Area Transit storage for data underway in the ETL process Transformations/cleansing done here No user queries (some do it) Sequential operations (few) on large data volumes Performed by central ETL logic Easily restarted No need for locking, logging, etc. RDBMS or flat files? (DBMS have become better at this) Finished dimensions copied from DSA to relevant marts ETL construction process Plan 1)Make high-level diagram of source-destination flow 2)Test, choose and implement ETL tool 3)Outline complex transformations, key generation and job sequence for every destination table Construction of dimensions 4)Construct and test building static dimension 5)Construct and test change mechanisms for one dimension 6)Construct and test remaining dimension builds Construction of fact tables and automation 7)Construct and test initial fact table build 8)Construct and test incremental update 9)Construct and test aggregate build 10)Design, construct, and test ETL automation

5 Building Dimensions Static dimension table Assignment of keys: production keys to DW using table Combination of data sources: find common key? Handling dimension changes Slowly changing dimensions Find newest DW key for a given production key Table for mapping production keys to DW keys must be updated Load of dimensions Small dimensions: replace Large dimensions: load only changes Building fact tables Two types of load: Initial load ETL for all data up till now Done when DW is started the first time Often problematic to get correct historical data Very heavy -large data volumes Incremental update Move only changes since last load Done periodically (../month/week/day/hour/...) after DW start Less heavy -smaller data volumes Dimensions must be updated before facts The relevant dimension rows for new facts must be in place Special key considerations if initial load must be performed again

6 Types of data sources Non-cooperative sources Snapshot sources provides only full copy of source Specific sources each is different, e.g., legacy systems Logged sources writes change log (DB log) Queryable sources provides query interface, e.g., SQL Cooperative sources Replicated sources publish/subscribe mechanism Call back sources calls external code (ETL) when changes occur Internal action sources only internal actions when changes occur (DB triggers is an example) Extract strategy is very dependent on the source types Extract phase Goal: fast extract of relevant data Extract from source systems can take a long time Types of extracts: Extract applications (SQL): co-existence with other applications DB unload tools: must faster than SQL-based extracts Extract applications sometimes the only solution Often too time consuming to ETL Extracts can take days/weeks Drain on the operational systems Drain on DW systems => Extract/ETL only changes since last load (delta)

7 Computing deltas Much faster to only ETL changes since last load A number of methods can be used Store sorted total extracts in DSA Delta can easily be computed from current+last extract + Always possible + Handles deletions - Does not reduce extract time Put update timestamp on all rows Updated by DB trigger Extract only where timestamp > time for last extract + Reduces extract time +/- Less operational overhead - Cannot (alone) handle deletions - Source system must be changed Load (1) Goal: fast loading into DW Loading deltas is much faster than total load SQL-based update is slow Large overhead (optimization,locking,etc.) for every SQL call DB load tools are much faster Some load tools can also perform UPDATEs Index on tables slows load a lot Drop index and rebuild after load Can be done per partition Parallellization Dimensions can be loaded concurrently Fact tables can be loaded concurrently Partitions can be loaded concurrently

8 Load (2) Relationships in the data Referential integrity must be ensured Can be done by loader Aggregates Must be built and loaded at the same time as the detail data Today, RDBMSes can often do this Load tuning Load without log Sort load file first Make only simple transformations in loader Use loader facilities for building aggregates Use loader within the same database Should DW be on-line 24*7? Use partitions or several sets of tables Overview General ETL issues: Data staging area (DSA) Building dimensions Building fact tables Extract Load Transformation/cleaning Commercial (and open-source) tools The AJAX data cleaning and transformation framework

9 Data Cleaning Activity of converting source data into target data without errors, duplicates, and inconsistencies, i.e., Cleaning and Transforming to get High-quality data! Why Data Cleaning and Transformation? Data in the real world is dirty incomplete: lacking attribute values, lacking certain attributes of interest, or containing only aggregate data e.g., occupation= noisy: containing errors or outliers (spelling, phonetic and typing errors, word transpositions, multiple values in a single free-form field) e.g., Salary= -10 inconsistent: containing discrepancies in codes or names (synonyms and nicknames, prefix and suffix variations, abbreviations, truncation and initials) e.g., Age= 42 Birthday= 03/07/1997 e.g., Was rating 1,2,3, now rating A, B, C e.g., discrepancy between duplicate records

10 Why Is Data Dirty? Incomplete data comes from: non available data value when collected different criteria between the time when the data was collected and when it is analyzed. human/hardware/software problems Noisy data comes from: data collection: faulty instruments data entry: human or computer errors data transmission Inconsistent (and redundant) data comes from: Different data sources, so non uniform naming conventions/data codes Functional dependency and/or referential integrity violation Why Is Data Cleaning Important? Data warehouse needs consistent integration of quality data Data extraction, cleaning, and transformation comprises the majority of the work of building a data warehouse No quality data, no quality decisions! Quality decisions must be based on quality data (e.g., duplicate or missing data may cause incorrect or even misleading statistics)

11 Types of data cleaning Conversion, parsing and normalization Text coding, date formats, etc. Most common type of cleansing Special-purpose cleansing Normalize spellings of names, addresses, etc. Remove duplicates, e.g., duplicate customers Domain-independent cleansing Approximate, fuzzy joins on not-quite-matching keys Data quality vs cleaning Data quality = Data cleaning + Data enrichment enhancing the value of internally held data by appending related attributes from external sources (for example, consumer demographic attributes or geographic descriptors). Data profiling analysis of data to capture statistics (metadata) that provide insight into the quality of the data and aid in the identification of data quality issues. Data monitoring deployment of controls to ensure ongoing conformance of data to business rules that define data quality for the organization. Data stewards responsible for data quality DW-controlled improvement Source-controlled improvement Construct programs to check data quality

12 Overview General ETL issues: Data staging area (DSA) Building dimensions Building fact tables Extract Load Transformation/cleaning Commercial (and open-source) tools The AJAX data cleaning and transformation framework ETL tools ETL tools from the big vendors, e.g., Oracle Warehouse Builder IBM DB2 Warehouse Manager Microsoft Integration Services Offer much functionality at a reasonable price (included ) Data modeling ETL code generation Scheduling DW jobs Many others Hundreds of tools Often specialized tools for certain jobs (insurance cleansing, ) The best tool does not exist Choose based on your own needs Check first if the standard tools from the big vendors are ok

13 ETL and data quality tools Magic Quadrant for Data Quality Tools, 2007 Some open source ETL tools: Talend Enhydra Octopus Clover.ETL Not so many open source quality/cleaning tools Application context Integrate data from different sources E.g.,populating a DW from different operational data stores Eliminate errors and duplicates within a single source E.g., duplicates in a file of customers Migrate data from a source schema into a different fixed target schema E.g., discontinued application packages Convert poorly structured data into structured data E.g., processing data collected from the Web

14 The AJAX data transformation and cleaning framework Motivating example (1) Publications(pubKey, title, eventkey, url, volume, number, pages, city, month, year) Authors(authorKey, name) Events(eventKey, name) PubsAuthors(pubKey, authorkey) Data Cleaning & Transformation DirtyData(paper:String)

15 Publications DirtyData Motivating example (2) QGMW96 Making Views Self-Maintainable for Data Warehousing PDIS null null null null Miami Beach Florida, USA 1996 Events PDIS Conference on Parallel and Distributed Information Systems Authors DQua Dallan Quass AGup Ashish Gupta JWid Jennifer Widom.. Data Cleaning & Transformation PubsAuthors QGMW96 DQua QGMW96 AGup. [1] Dallan Quass, Ashish Gupta, Inderpal Singh Mumick, and Jennifer Widom. Making Views Self-Maintainable for Data Warehousing. In Proceedings of the Conference on Parallel and Distributed Information Systems. Miami Beach, Florida, USA, 1996 [2] D. Quass, A. Gupta, I. Mumick, J. Widom, Making views selfmaintianable for data warehousing, PDIS 95 Modeling a data cleaning process Authors Duplicate Elimination DirtyTitles... DirtyEvents DirtyAuthors Extraction Standardization Formattin g Cities Tags DirtyData A data cleaning process is modeled by a directed acyclic graph of data transformations

16 Existing technology Ad-hoc programs written in a programming language like C or Java or using an RDBMS proprietary language Programs difficult to optimize and maintain Data transformation scripts using an ETL (Extraction-Transformation-Loading) or a data quality tool Problems of ETL and data quality solutions (1) Data cleaning transformations... App. Domain 1 App. Domain 2 App. Domain 3 The semantics of some data transformations is defined in terms of their implementation algorithms

17 Problems of ETL and data quality solutions (2) Clean data Rejected data Cleaning process Dirty Data There is a lack of interactive facilities to tune a data cleaning application program AJAX features An extensible data quality framework Logical operators as extensions of relational algebra Physical execution algorithms A declarative language for logical operators SQL extension A debugger facility for tuning a data cleaning program application Based on a mechanism of exceptions

18 AJAX features An extensible data quality framework Logical operators as extensions of relational algebra Physical execution algorithms A declarative language for logical operators SQL extension A debugger facility for tuning a data cleaning program application Based on a mechanism of exceptions Logical level: parametric operators View Map Merge Cluster Match Apply View: arbitrary SQL query Map: iterator-based one-to-many mapping with arbitrary user-defined functions Match: iterator-based approximate join Cluster: uses an arbitrary clustering function Merge: extends SQL group-by with user-defined aggregate functions Apply: executes an arbitrary user-defined algorithm

19 Logical level Authors Duplicate Elimination DirtyTitles... Extraction DirtyAuthors Standardization Formattin g Cities Tags DirtyData Logical level Authors Physical level Authors Merge Java Scan Duplicate Elimination Cluster TC DirtyTitles... Match DirtyTitles... NL DirtyAuthors DirtyAuthors Extraction Map Java Scan Standardization Map Java Scan Formattin g Map Cities Tags DirtyData SQL Scan Cities Tags DirtyData

20 AJAX features An extensible data quality framework Logical operators as extensions of relational algebra Physical execution algorithms A declarative language for logical operators SQL extension A debugger facility for tuning a data cleaning program application Based on a mechanism of exceptions Input: 2 relations Match Finds data records that correspond to the same real object Calls distance functions for comparing field values and computing the distance between input tuples Output: 1 relation containing matching tuples and possibly 1 or 2 relations containing non-matching tuples

21 Authors Example Merge Cluster Match MatchAuthors DirtyAuthors Duplicate Elimination Authors Merge Cluster MatchAuthors Match Example CREATE MATCH MatchDirtyAuthors FROM DirtyAuthors da1, DirtyAuthors da2 LET distance = editdistance(da1.name, da2.name) WHERE distance < maxdist INTO MatchAuthors DirtyAuthors Duplicate Elimination

22 Authors Merge Cluster Match DirtyAuthors MatchAuthors Duplicate Elimination Example CREATE MATCH MatchDirtyAuthors FROM DirtyAuthors da1, DirtyAuthors da2 LET distance = editdistance(da1.name, da2.name) WHERE distance < maxdist INTO MatchAuthors Input: DirtyAuthors(authorKey, name) 861 johann christoph freytag 822 jc freytag 819 j freytag 814 j-c freytag Output: MatchAuthors(authorKey1, authorkey2, name1, name2) johann christoph freytag jc freytag jc freytag j-c freytag... Implementation of the match operator s 1 S 1, s 2 S 2 (s 1, s 2 ) is a match if editdistance (s 1, s 2 ) < maxdist

23 Nested loop S 1 editdistance S 2... Very expensive evaluation when handling large amounts of data Þ Need alternative execution algorithms for the same logical specification A database solution CREATE TABLE MatchAuthors AS SELECT authorkey1, authorkey2, distance FROM (SELECT a1.authorkey authorkey1, a2.authorkey authorkey2, editdistance (a1.name, a2.name) distance FROM DirtyAuthors a1, DirtyAuthors a2) WHERE distance < maxdist; No optimization supported for a Cartesian product with external function calls

24 Window scanning S n Window scanning S n

25 Window scanning S n May loose some matches String distance filtering S 1 S 2 John Smit length- 1 length John Smith Jogn Smith length editdistance John Smithe length + 1 maxdist = 1

26 Annotation-based optimization The user specifies types of optimization The system suggests which algorithm to use Ex: CREATE MATCHING MatchDirtyAuthors FROM DirtyAuthors da1, DirtyAuthors da2 LET dist = editdistance(da1.name, da2.name) WHERE dist < maxdist % distance-filtering: map= length; dist = abs % INTO MatchAuthors Declarative specification DEFINE FUNCTIONS AS Choose.uniqueString(OBJECT[]) RETURN STRING THROWS CiteSeerException Generate.generateId(INTEGER) RETURN STRING Normal.removeCitationTags(STRING) RETURN STRING (600) DEFINE ALGORITHMS AS TransitiveClosure SourceClustering(STRING) DEFINE INPUT DATA FLOWS AS TABLE DirtyData (paper STRING (400) ); TABLE City (city STRING (80), citysyn STRING (80) ) KEY city,citysyn; DEFINE TRANSFORMATIONS AS CREATE MAPPING mapkedida FROM DirtyData Dd LET keykdd = generateid(1) {SELECT keykdd AS paperkey, Dd.paper AS paper KEY paperkey CONSTRAINT NOT NULL mapkedida.paper }

27 AJAX features An extensible data quality framework Logical operators as extensions of relational algebra Physical execution algorithms A declarative language for logical operators SQL extension A debugger facility for tuning a data cleaning program application Based on a mechanism of exceptions Management of exceptions Problem: to mark tuples not handled by the cleaning criteria of an operator Solution: to specify the generation of exception tuples within a logical operator exceptions are thrown by external functions output constraints are violated

28 Debugger facility Supports the (backward and forward) data derivation of tuples wrt an operator to debug exceptions Supports the interactive data modification and, in the future, the incremental execution of logical operators Architecture

29 To see it working... Workshop-UQ? Tomorrow at Tagus Park, 10H-16H