Operationalizing Your Data Science and Machine Learning Initiatives

Transcription

1 Gartner Data & Analytics Summit March 2019 / London, UK Operationalizing Your Data Science and Machine Learning Initiatives Erick Brethenoux 2019 Gartner, Inc. and/or its affiliates. All rights reserved. Gartner is a registered trademark of Gartner, Inc. and its affiliates. This publication may not be reproduced or distributed in any form without Gartner s prior written permission. It consists of the opinions of Gartner s research organization, which should not be construed as statements of fact. While the information contained in this publication has been obtained from sources believed to be reliable, Gartner disclaims all warranties as to the accuracy, completeness or adequacy of such information. Although Gartner research may address legal and financial issues, Gartner does not provide legal or investment advice and its research should not be construed or used as such. Your access and use of this publication are governed by Gartner s Usage Policy. Gartner prides itself on its reputation for independence and objectivity. Its research is produced independently by its research organization without input or influence from any third party. For further information, see Guiding Principles on Independence and Objectivity.

2 DS Teams Now Measure Success on Business Outcomes We use business KPIs 58% We run financial analysis on risk factors, ROI, etc. 47% Numbers of models in production Number of models developed 33% 32% We only roughly estimate We do not measure 2% 7% Base: All respondents, n = 302 Percentage of Respondents Q: How does your organization measure the success of its data science initiatives? Gartner, Inc. and/or its affiliates. All rights reserved. Gartner is a registered trademark of Gartner, Inc. and its affiliates.

3 Less Than Half of Models Are Fully Deployed Not Deployed 24% Fully Deployed 47% Partially Deployed 29% Base: Total, Excluding Not Sure; n = 250 Q. Thinking of your organization s data science projects over the past two years, what percentage was fully deployed, partially deployed or not deployed at all? ID: Gartner, Inc. and/or its affiliates. All rights reserved. Gartner is a registered trademark of Gartner, Inc. and its affiliates.

4 Production Is the Main Barrier Towards Delivering Business Value Difficulty deploying into business processes/applications 47 Management resistance/internal politics 36 Lack of DevOps or managerial skills 31 Unable to adequately secure or govern data and analytics inputs/outputs 27 Poor planning/unreasonable expectations Lack of funding/right tools Unable to adequately address (or mitigate) data quality and integrity issues Open-source pilot technologies are not production-grade Unable to demonstrate business ROI 13 Selected tooling didn't scale to production requirements Other 7 9 Percentage of respondents Base: n = 45 Gartner Research Circle Members/external sample. Excludes not sure. Asked if selected getting data and analytics projects into production at DA05. DA5b. Thinking about why you selected getting data and analytics projects into production as a challenge, please identify your organization s specific barriers to moving projects into production. Multiple responses allowed Gartner, Inc. and/or its affiliates. All rights reserved. Gartner is a registered trademark of Gartner, Inc. and its affiliates.

5 Key Issues 1. How do you optimize production success from the start? 2. What does a best practice operationalization process look like? 3. What are the main skills to support the production process? 4. How to build a sustainable DS production practice? Gartner, Inc. and/or its affiliates. All rights reserved. Gartner is a registered trademark of Gartner, Inc. and its affiliates.

7 A Battle-Tested Analytical Model Development Cycle Development Cycle Analysis Data Wrangling Data Sources Discovery Model Validation New Data Acquisition Business Understanding Publishing/ Deployment 1. Select Use Cases List use cases. Socialize with stakeholders. Establish KPIs. Select business owners. Research data sources. 2. Prioritize Use Cases Determine business value. Identify roadblocks. Estimate technical complexity. Are use cases: Business extenders. Game changers. 3. Stack-Rank Use Cases Business extended first. Game changers after three to six extenders. Start with lower technical complexity. High-rank cases with spinoff potential. Start Here! Gartner, Inc. and/or its affiliates. All rights reserved. Gartner is a registered trademark of Gartner, Inc. and its affiliates.

8 Art of the Possible: A Creative Approach to Use Cases Real-World Examples Could we adapt/extend what they ve done? What is important to us? What could the data tell us? Business Drivers? Information Assets (internal/external/composite) Actionable? Manageable? Technological? Economical? Ethical? Assess Feasibility << Possibility >> Prove With Data Science Gartner, Inc. and/or its affiliates. All rights reserved. Gartner is a registered trademark of Gartner, Inc. and its affiliates.

10 Development Is Academic, Production Is Economics Development Cycle Operationalization Cycle Analysis Model Validation Academics Data Wrangling New Data Acquisition Publishing/ Deployment Actionable Insights Economics Data Sources Discovery Business Understanding Start Here! Gartner, Inc. and/or its affiliates. All rights reserved. Gartner is a registered trademark of Gartner, Inc. and its affiliates.

11 Operationalization Phase 1: Release Cycle OP1: Test your models before releasing them in the wild. Development Cycle Analysis Model Validation Release Phase/ Operationalization Cycle KPIs Validation Instantiation Validation Integration Testing Data Wrangling New Data Acquisition Parameters Testing Data Sources Discovery Business Understanding Model Release Endpoints Identification Gartner, Inc. and/or its affiliates. All rights reserved. Gartner is a registered trademark of Gartner, Inc. and its affiliates.

12 Operationalization Phase 2: Activation Cycle OP2: Embed your models in business processes. Integration Testing Parameters Testing Endpoints Identification Analysis Data Wrangling Instantiation Validation Model Validation New Data Acquisition KPIs Validation Model Release Management & Governance Operationalization Cycle/Activation Model Behavior Tracking Retuning Challenging Explaining Visualizing Complying Production Audit Procedure Application Integration Data Sources Discovery Business Understanding Model Activation Model Deployment Gartner, Inc. and/or its affiliates. All rights reserved. Gartner is a registered trademark of Gartner, Inc. and its affiliates. Start Here!

13 Full DS Cycles Parameters Testing Operationalization Cycle/Release Establish a sustainable, repeatable, measurable and continuous data science process. Development Cycle Analysis Data Wrangling Integration Testing Instantiation Validation Model Validation New Data Acquisition KPIs Validation Endpoints Identification Model Release Management & Governance Operationalization Cycle/Activation Model Behavior Tracking Retuning Challenging Explaining Visualizing Complying Production Audit Procedure Application Integration Data Sources Discovery Business Understanding Model Activation Model Deployment Gartner, Inc. and/or its affiliates. All rights reserved. Gartner is a registered trademark of Gartner, Inc. and its affiliates. Start Here!

16 Core Skills for Operationalization Core DS Development Skills Core DS Operationalization Skills Data scientists Data engineers Software engineers Business experts Source system experts System architects System administrators Application developers Process engineers Gartner, Inc. and/or its affiliates. All rights reserved. Gartner is a registered trademark of Gartner, Inc. and its affiliates.

17 Where to Put the Knowledge Expertise? Business intimacy Operational synergy Knowledge sharing Agility Cross-functional view Expertise cross-pollination Proximity to process and data Strategy synchronization BT Business translator SME Subject matter expert DE Data engineer DS Data scientist Analytics Expertise/Data Science Lab/COE SME/BT LOB DE/DS SME/BT LOB IT LOB Corp./ CDO Steering Committee DS Lab Sandbox Gartner, Inc. and/or its affiliates. All rights reserved. Gartner is a registered trademark of Gartner, Inc. and its affiliates.

19 Decision Modeling Decision Intelligence Nondeterministic behaviors Sustainable decisions Autonomy of decision models Analytics Modeling Tactical and Strategic Decisions Decision Modeling Process Modeling Embedded Insights Smart Processes Real-Time Decisions Gartner, Inc. and/or its affiliates. All rights reserved. Gartner is a registered trademark of Gartner, Inc. and its affiliates.

20 The Gartner Decision Intelligence (GDI) Model Concept Capture (observe) Interpret (investigate) Comeback Context Act (execute) Outcomes Generate Response (contextualize) Fulfill Intent (model) Gartner, Inc. and/or its affiliates. All rights reserved. Gartner is a registered trademark of Gartner, Inc. and its affiliates.

21 Evolving Toward Explainability DARPA XAI Today: Black Box Training Data Machine Learning Process Learned Function Decision or Recommendation Task User Why did you do that? Why not something else? When do you succeed? When do you fail? When can I trust you? How do I correct an error? Tomorrow: Explainable Training Data New Machine Learning Process Explainable Model Explanation Interface Task User I understand why. I understand why not. I know when you succeed. I know when you fail. I know when to trust you. I know why you erred. Balance accuracy vs. explainability: Accuracy Accountability and fairness Stability and Trustworthiness Source: DARPA Gartner, Inc. and/or its affiliates. All rights reserved. Gartner is a registered trademark of Gartner, Inc. and its affiliates.

22 Recommendations Prioritize use cases by establishing an ongoing dialogue with business. Ensure the integrity, transparency and sustainability of models through a systematic process. Recruit operationalization core talents as early as possible. Establish a production committee to oversee the full data science cycle. Minimize the technical debt of deployed ML models by monitoring and revalidating their business value Gartner, Inc. and/or its affiliates. All rights reserved. Gartner is a registered trademark of Gartner, Inc. and its affiliates.

23 Recommended Gartner Research How to Operationalize Machine Learning and Data Science Projects Erick Brethenoux, Shubhangi Vashisth and Jim Hare (G ) Toolkit: How to Select and Prioritize AI Use Cases Using Real Domain and Industry Examples Svetlana Sicular and Others (G ) Staffing Data Science Teams: Map Capabilities to Key Roles Alexander Linden, Carlie Idoine and Others (G ) Decision Intelligence Is the Near Future of Decision Making: A Gartner Trend Insight Report Erick Brethenoux and Svetlana Sicular (G ) For information, please contact your Gartner representative Gartner, Inc. and/or its affiliates. All rights reserved. Gartner is a registered trademark of Gartner, Inc. and its affiliates.