Session M6. Andrea Matulick, Acting Manager, Business Intelligence, Robert Davies, Technical Team Leader, Enterprise Data

Transcription

1 Strategic Data Management Conforming the Data Warehouse Session M6 September 24, 2007 Andrea Matulick, Acting Manager, Business Intelligence, Planning and Assurance Services, UniSA Robert Davies, Technical Team Leader, Enterprise Data Warehouse, ISTS, UniSA

2 Strategic Data Management Conforming the data warehouse Strategy Data Management Conforming data Why do we need strategic data management? To support the strategic planning cycle To maximize i performance and funding To support integrated processes using technology To become an analytic organisation 2

3 The standard business Strategic Planning cycle: Formulate the strategy (using decisions from evidence based data) Communicate the strategy Analyse scenarios Prepare plans and budgets Monitor, forecast, report against actual data Feedback the results for the next strategy cycle cle 3

4 Maximising performance and funding Measuring performance - examples of Key Performance Indicators Student Demand applications, preferences, TER Research Performance publications, research income, completions Student Staff Ratio student load (EFTSL), staff FTE Maximising funding - examples of funding formulae Commonwealth Grant Scheme (CGS) Research Training Scheme (RTS) Learning and Teaching Performance fund (L&TPF) Funding Agreement total$ = Sum of (CGS student (EFTSL) per cluster x cluster funding rate$ ) HEP s specific performance index = (HDR completions x 0.5) + (Research Income x 0.4) + (Research Publications x 0.1) Student demand (applications, load) Student experience (CEQ overall satisfaction, generic skills, good teaching) (GDS employment and further study) Student progression (success rate, retention rate, level of study)

5 Which data is important? focus on key issues and critical facts and measures do not overload with data from every aspect of the organisation data used in key performance indicators and funding formulae all data that underlies those indicators, right down to transactional level additional external competitor and benchmarking data standardised and conformed data the right information must be delivered to the right people at the right time and in the right context 5

6 Why transactional systems cannot help implement strategy transactional systems collect and store data on day to day operational activities they do not focus on key issues and critical facts data is siloed into source application areas (e.g. students, staff, finance) not combined into logical business processes data is not conformed (standardised) so that data may be aggregated across systems transactional data is not suitable for corporate analysis, reporting or dashboard applications 6

7 How can electronic decision support systems help with implementing strategy? tie together computer applications and business processes related to key measures be able to analyse and report on the results in a timely and accurate manner move beyond transactional processing systems, incorporate key managerial processes into technology by integrating applications and using decision support systems While 76 percent of executives cite strategic planning as the top management tool to improve long term performance and strengthen integration across an organisation, only 33 percent of executives use electronic decision i support tools that could help them in managing performance (Hackett, business survey 2002) 7

8 Electronic decision support systems to help implement strategy System Corporate Performance Management system (CPM) Dashboard system Benchmarking and Scorecarding system Budgeting and Forecasting system Business Intelligence presentation layer Data Warehouse Metadata system Master Data Management system Data Quality system Transactional systems Major Function Integrate strategic planning documents, organisational processes, targets t and responsibilities For senior managers to customise key views of performance in their area of responsibility Record and monitor key performance indicators against targets, indicate success, failure and alerts Using actual data to model scenarios and predict future trends Enable user access to information via data, analysis and reports Reorganise transactional data into logical business models for corporate analysis and reporting needs. Combine data required for KPI s, add value through external and conformed data, etc. Provide users context about the data, record data source, lineage, definitions, business rules, etc. Centrally store and maintain the major common data dimensions used by all areas of the organisation (e.g. org structure) Conform and standardise common data dimensions across the organisation, identify data errors and anomalies Collect source data and process day to day transactions for the organisation

9 How do resources affect strategic data management? transactional systems well resourced, supported, upgraded regularly other functions relatively unsupported financially, without sufficient experienced resources management continually complains about lack of reports and analysis, lagging timeliness of information, consistency and standardisation of results and definitions, lack of forecasting and scenario planning, lack of competitor data, benchmarking, etc. BI resources need to allocate appropriate resources to decision support systems

10 How does the standard and integrity of data affect corporate decisions? Data warehousing holds much promise to provide competitive advantage through derived business intelligence, but the promise cannot be realised unless you ensure the integrity of your data. You must have end-to-end controls and the ability to identify data anomalies in source data from many operational systems. These controls are an integral part of essential data management best practice. (Maurer, IBM, DMReview.com, July 2007) Data Quality software products assist in identifying data quality issues, but cannot fix data Most organisations do not factor data quality resources into any of their plans, it takes a very low priority. Organisations do not realise that this omission may be producing poor data on which they are basing their strategic decisions. 10

11 Why is conforming the data important? transactional systems contain very little data quality control free text fields mean questionable validity, consistency, standards reports and analysis become increasingly difficult, a computer does not recognise data to be the same unless it is identical (e.g. male and M, 1 and 01 are not the same to a computer) companies using Master Data Management systems to maintain common o data used by multiple transactional a applications consistency and standardisation of data and business rules across an organisation essential for the quality and usefulness of corporate analysis and reports 11

12 How does a data warehouse assist in organising, standardising and conforming data? A data warehouse reorganises transactional data into logical business models for corporate analysis and reporting needs Combines data from multiple systems and external data in a way that is meaningful to the business Uses standard business rules and conformed standard code sets Can only report across data from multiple systems if the dimensions are conformed and can be reused across the fact data Highlights the need for data quality frameworks and master data management systems to be part of the data management strategy 12

13 Data Warehouse Integration Matrix

14 Data that conforms well - external National Research Performance data Research funding (RTS) and the research quality framework (RQF) use publications, income and completions to measure performance. DEST provides national data on these measures as a series of reports in a spreadsheet. The data can be loaded into a data warehouse along with standard reference data to analyse trends, share, benchmarking, rankings etc. By combining the warehouse data in an OLAP cube, we can see how our university is performing in the sector. 14

15 external National Research Performance data

16 external National Research Performance data

17 Analysis vs Reports The original spreadsheet report is static and only shows one measure at a time In the warehouse we can add State, ATN and National benchmarking totals, share, rankings The OLAP cube provides the ability to analyse the data rather than just look at one report at a time However, this data is lagging by at least one year We have good data quality, context, but not timeliness Need to load our live research data into the warehouse daily to help make good strategic decisions 17

18 Data with conforming issues - Live Research Performance data need to see performance areas during the current year compared to previous years good and poor performing divisions and schools (org units) data comes from 3 transactional systems Research Master, Finance One, and Empower HR the data warehouse design was successful in bringing together the data from the 3 systems however, the issue of non conforming data proved to be a problem in a number of areas, including org structure which was supposedly controlled from a central master file 18

19 Live Research data Publications per FTE

20 Org Unit code conforming issues two transactional systems 20

21 Some alternative thoughts on data quality Blame everything on the source data and point out that fixing source systems is out of scope. Only use BI tools that let users export the reports to Excel where they can play with the data and produce information that looks much more accurate. (McBurney, Senior Consultant, 2006) 21

22 Overview Alternative approach to managing conformed data in the warehouse. 22

23 Agenda/Contents Background Challenges for a warehouse startup Additional challenges The two approaches Oracle Warehouse Builder Other tools 23

24 University Of SA, Data Warehouse The University data warehouse consists of data from source systems: Finance, HR, Student, Master data management system ie. Org Unit data Research administration i ti system -- data quality challenged Covers research and student related business areas consisting: 10 fact tables 2 snapshot fact tables 70 dimension tables Environment: Oracle 9i Rel2, OWB Rel 1, Cognos ver7 24

25 Challenges in a normal EDW start-up phase: Methodology and documentation standards Design and developing ETL technical infrastructure Master Data Management System one source of the truth ie. In-house built application to manage Org Unit data. Build the warehouse Design and build the BI layer. 25

26 Extra challenges to address Requirement for low on going support Some source systems include poor data quality Business processes and political environment not focused on data quality improvements 26

27 Containing the extra challenges through adaptive design How can we better manage the extra challenges? Establish Master Data Management system one source of data Don t want Data Quality (DQ) issues to destroy these gains. Therefore need a robust way to manage DQ issues in the warehouse with minimum impact and intervention. 27

28 Containing the extra challenges through adaptive design Managing reference data with DQ issues. Two approaches considered given our challenges: 1. Kimball recommended approach 2. University of SA, Hybrid Approach Oracle Warehouse Builder Release 1 does not provide an automatic means to manage SCD dimension tables. 28

29 Kimball recommended approach For an incoming fact row that has an unmatched dimensional value : automatically create a new dimension entry place holder as a result. assume at a later date the dimension row which matches the placeholder will arrive and overwrite the placeholder with a full row of attributes. 29

30 Kimball recommended approach Advantages: No Factual data is lost (?) Proven approach which works efficiently for large Fact tables Some ETL tools do this work for you. 30

31 Kimball recommended approach Disadvantages: more than one source of data for the dimension potentially more than one source of the truth. if the dimension is conformed then rubbish data is made available to all areas of the data warehouse, unless it is managed. If effective dating is involved, has the potential to corrupt contiguous date ranges. If only part of a placeholder is available (ie the code and no Efft Date) from the Fact row then Fact record gets written to a log file and dim key set to unknown or Fact row is rejected completely Either case probably requires manual intervention to resolve. 31

32 UniSA hybrid approach Capture Unknowns For an incoming fact row that has an unmatched dimensional value : 1. Store the unmatched business code in the core Fact table. 2. The dimension surrogate key within the fact record is set to -1. Hide business code from user reporting layer 32

33 UniSA hybrid approach Unmatched Business Code Fact Surrogate_key Org Unit Bus Code Org Code Key Org Key Version Fact Measure 1001 GPB ITU ITU

34 UniSA hybrid approach Reprocessing the Unknowns. At a later date: Copy the core fact rows into the staging table where the business code exists and the corresponding surrogate key =-1 Reconcile against the dimension table in order to obtain a known key. Reuse existing transformation mappings Merge the Fact record back into the core Fact table 34

35 UniSA hybrid approach Reprocessed Business Code Org Code Key Org Key Version Org Code Org Description Current_Flag -1 1 Unknown Y ITU Info Tech N ITU Information Tech Y GPB Grounds Y 35

36 UniSA hybrid approach Advantages: No Factual data is lost. The one data source controls the truth for each dimension table. Automatic poor data quality quarantine Data quality issues peculiar to the given source system are not propagated throughout the entire warehouse. No ongoing maintenance overhead with potential accumulation of rubbish data within dimensions. No need for an ever-expanding number of fix up scripts. DQ issues can be handled on a subject area basis, assisting in prioritization. Unknowns report per Fact subject area available for DQ department. 36

37 UniSA hybrid approach Disadvantages: Fact table requires extra processing on a regular basis in order to reconcile the unknown dim keys. Requires the raw business codes are present in the Fact (not necessarily visible for user reporting) Possibly not suitable for very large Fact tables where DQ is an ongoing issue, > 10 million fact table records, but Ok for Uni data volumes. 37

38 Oracle Warehouse Builder (OWB) Low cost Mappings automatically perform bulk inserts Excellent ETL auditing information available Process Flow allows forking of multiple database sessions 38

39 Use Emphasis on Graphics 39

40 OWB Process Flow - control 40

41 OWB Process Flow session forking 41

42 More alternative thoughts on data quality Default null values to the word unknown. If anyone questions this point out that unknown is used liberally throughout all the source systems and is more useful than not knowing that it is unknown. You will soon find that your information management projects are being delivered on time and are no less accurate than the source systems (McBurney, Senior Consultant, 2006) 42