How To Manage Data And Data Management

Transcription

1 Considerations for Data Management Handle unstructured data growth, in-flight /streaming (perishable) & data at rest parse, standardize and match/household to create holistic views Data governance Reference data management Data profiling Data cleansing Data matching/survivorship Data monitoring Data stewardship Master data repository leverage internal & external data partner & other 3 rd party sources 1

2 Considerations for Master Data Management Data Domains customer member patient citizen product partner materials employees location asset Data Sources Structured, semistructured, unstructured Enterprise applications Databases, text files, XML On premises, off premises (cloud) Data Latency and Granularity Real-time, transaction Bulk, batch Virtual, federated 2

3 Considerations for Master Data Management The conduit used to move or merge data across disparate processes & systems Dark Orange srgb (255,140,0): (FF8C00) Explains business information Describes business process results Ensures consistent meaning and interpretation Orange srgb (255,165,0): (FFA500) Supports identification of things uniquely identifies a business entity A set of common attributes that in combination are unique 3 3

4 The IM Supply Chain Considerations for AM formulate & answer questions deliver & act upon reports and intelligence acquire, explore & prepare data, ensure quality Validate, deploy & monitor models transform and integrate Data management Analytics management develop & evaluate reports & models Decision management 4

5 Considerations for Analytics Management centralize, organize, administer, evaluate & monitor analytical models and their outcomes push operations to database query, scoring, bulk load lifecycle and workflow management 5

6 Considerations for Decision Management Development, design, deployment Determine where source/target tables and transformations are used in jobs and downstream BI & analytics processes - understand the impact of a development change before making it Design jobs for grid or undistributed execution - understand job steps by modularizing procedure steps Visual flows, metadata & data quality monitoring & scorecarding Consumption Diversity support an increasingly diverse set of audiences, interactions, and devices in a scalable, contiguous & concurrent fashion batch, real-time and service-based job deployment 6

7 Key Point #2 An organization s ability to use information strategically is directly correlated to its IM competencies. 7

8 Agenda What is IM? Review business drivers for IM How has IM Evolved? - distinguishing ways to conduct IM Considerations for expanding & sharpening your IM arsenal Illustrations What you can do next 8

9 Information Management Illustrations 9

10 Master Data Management Reference data describes business entities Subject area specific Reflects enterprise business rules and values Critical in integrating operational systems Customers Employees Products Locations Suppliers Assets 10

11 Creating a Master Record service claims Cust. Id First Name William Middle Jim Last Name Sosewlsky DOB 04/12/39 SSN Address Oak St., Eves, IL billing Cust. Id First Name William Middle J. Last Name Sosulski DOB SSN Address 123 Oak St., Eves, IL contact us Cust. Id First Name j Middle s Last Name sosuluski DOB April 12 SSN 123 Address BubbaJ@bubbagroup.com ERP Cust. Id 3721B First Name Willaim Middle James Last Name Corp. DOB April 12 SSN Address 3224 Pkwy G, Los Osos data warehouse Cust. Id First Name William Middle James Last Name Sosulski DOB SSN Address 04/12/ Oak St.,Eves, IL B William James Sosulski 04/12/ Oak Street Eves IL

12 Master Data Management Illustration - clustering/matching 12

13 The IM Supply Chain Formulate & answer questions Deliver reports and intelligence Acquire, explore & prepare data, ensure quality Validate, deploy & monitor models Transform and integrate Data management Analytics management Develop & evaluate models Decision management 13

14 Illustrations Analytics Management Develop Candidate Models Compare Models Declare a Champion Model Validate Scoring Deploy Model or Scoring Function Validate Model Monitor Model Performance Request New Challenger Model Retire Model 14

15 Model Comparison Determine champion/challenger Lift ROC K-S (Kolmogorov- Smirnov) Profile Delta Custom reporting 15

16 Declare Champion Model Model promoted for scoring or used in production 16

17 Model Validation Perform scoring tests Create validation reports for the champion Validate using test data sources 17

18 Model or Scoring Deployment Publish or export to production system In-database scoring 18

19 Validate Scoring Perform score code validation independently from validation during model development Score code validated indatabase or in operational system 19

20 Monitor Model Performance Summarize model details Detect shifts in distribution of variable values over time in input and scored data Evaluate the predicted and actual target values for a champion model at multiple points in time Alerting & reporting 20