Data Governance for Big Data Analytics

Transcription

1

2 Data Governance for Big Data Analytics Dan Sholler Director, Product Marketing May 3, 2016

3 How Do We Drive Big Data Value? Archival & Storage Operational efficiency Discovery & Analytics Data-centric decisions Operational Analytics Real-time insights & actions Big data initiative often paid for by storage costs IT focus Rapid turnaround on traditional analysis Backlog build out Beginning self-service Data scientist focus Broad user participation Segmented by usage Self-service analytics are the majority case 3

4 Have you integrated your business glossary with your analytics tools? Respond at PollEv.com/Collibra Text COLLIBRA to once to join. Then text a KEYWORD Answers to this poll are anonymous 4

5 Have you integrated your business glossary with your analytics tools? Respond at PollEv.com/Collibra Text COLLIBRA to once to join. Then text a KEYWORD Answers to this poll are anonymous 5

6 Begin at the Beginning Archival & Storage Operational efficiency Discovery & Analytics Data-centric decisions Operational Analytics Real-time insights & actions Big data initiative often paid for by storage costs IT focus Rapid turnaround on traditional analysis Backlog build out Beginning self-service Data scientist focus Broad user participation Segmented by usage Self-service analytics are the majority case 6

7 Data as Grist for the Mill What did we start with? Operational efficiency Cheap storage Data is just stuff Put everything in Users know what they want Sharing due to colocation Native metadata What is the outcome? Few controls Users struggle 7

8 Emerging Data Lake Why did it make sense? Initial startups were limited in scope Silos and scale problems appeared later Storage and archiving Focus on Process Data Dictionary Security Classification Retention Policy Large-scale jobs (spark, Map/Reduce, R integration) 8

9 Data Haves and Have Nots The data literate create silos within the lake Purpose-built data sets and associated process and analytics Tolerate corporate governance The data poor are excluded because of lack of visibility and complexity Use traditional summarizations Reconstruct data warehouse concepts Copies and more copies for each variation 9

10 How Do Organizations Begin their Big Data Journey? Data as stuff : the origin of the data lake Data lakes were developed based on the traditional view of data. This assumes data is generic, and broadly useful if properly prepared. Manage the data lake based on that stuff theory Created basic data management procedures for data sets. Controls are seen as bad and no proper governance. Operational efficiency creates data inequality Current heavy users of data gain efficiency, but broad-based use is too difficult and is discouraged. The ROI model works, but the analytics are (mostly) a failure 10

11 Create Value From Big Data Analytics Archival & Storage Operational efficiency Discovery & Analytics Data-centric decisions Operational Analytics Real-time insights & actions Big data initiative often paid for by storage costs IT focus Rapid turnaround on traditional analysis Backlog build out Beginning self-service Data scientist focus Broad user participation Segmented by usage Self-service analytics are the majority case 11

12 Dictionary and Catalog Business Metadata Meaning, quality, integration Business users cannot use what they cannot find Copies cause confusion Quality viewpoint is critical to usability Integration across silos requires master data consistency What was the outcome? Ingestion controls for metadata basis 12

13 Data Helpdesk Data helpdesk is required Self-service users must be supported Lynchpin of data trust Processes of establishing trust Focus on suitability of data Problem/resolution Relative value Usage tracking Analytics assistance 13

14 Process at the Point of Use Identify relationships and views Access requests Data sharing agreements Catalog sharing Ingestion control Determine duplicates Identification Master data Use case associations 14

15 How to Enable Big Data Analytics? Dictionary and Catalog - Business Metadata Creating a comprehensive view of business and technical metadata reduces preparation and discovery time, and gives self-service users access to data and analytical views. Data Helpdesk to support self-service Any resource must be backed up by a well-understood process for helping users overcome challenges and for repairing anything that is not right. Focus processes on point of use Minimal ingestion control to populate basic metadata and sync master data. Apply controls at the point of consumption, except for obvious issues like duplicates. Enable analytics and discovery for scientists and analysts. 15

16 Deliver Governed Self-Service Across the Organization Archival & Storage Operational efficiency Discovery & Analytics Data-centric decisions Operational Analytics Real-time insights & actions Big data initiative often paid for by storage costs IT focus Rapid turnaround on traditional analysis Backlog build out Beginning self-service Data scientist focus Broad user participation Segmented by usage Self-service analytics are the majority case 16

17 Information Value Opportunity Value models give insight Prioritize helpdesk issues Discover monetization opportunities Improve management changes Value models are immature Begin with no assumption of precision, but relative value is important 17

18 Managing Big Data Policies Policy creation/modification New combinations of data reveal new insights, but can also create new risks Methodology for identifying and valuing risks Exception management must be part of helpdesk capability Flexible Policy Management Usability Trust Innovation 18

19 Segregation of Governance by Role User Type Needs Key Requirements Exception Process Basic (self-service) users Operational/ production Find appropriate data Avoid privacy and legal violations Protect confidentiality Business analysts Strategic/planning Crossorganizational data Data manipulation Data scientist Innovation Any data inside and outside the organization Formal and helpdesk based Helpdesk based VIP None, open sandbox 19

20 New Governance in the Big Data Analytics World Ethics Governance Behavioral Information Model Governance Big Data Governance Value Models 20

21 How to Operationalize Big Data Analytics? Policy management Create policies easily Determine those policies that apply and manage exceptions Identify need for new policies Segregation of governance by role Give basic users freedom within guardrails Expand guardrails for business analysts doing strategic work Allow data scientists to run without them Governance segregation by purpose Models and views PII data Behavioral information Enabling differentiation for role and purpose is the key challenge. 21

22 Big Data Analytics Journey Begin at the beginning Move quickly to create visibility using dictionary, business glossary, or catalog Span operational silos Make sure the have-nots have something Enable analytics Enhance metadata with quality, value, and other metadata Create a data helpdesk Initiate governance processes at the point of use where possible Deliver governed self-service across the organization Ethics policies Valuation Model and algorithm governance 22

23 Thank You Tell us how we did citizens.collibra.com/feedback

24