Thinking in Big Data. Tony Shan. Nov 5, 2014

Thinking in Big Data Tony Shan Nov 5, 2014

Table of Contents Introduction Big Data Skillset Challenges Concept of Big Data Competency and Model Big Data Thinking Conclusion 2

In 1 minute 4

How to Deal with Complexity and Immaturity? 9

Need ACE Competency Architecture Engineering ACE 10

Critical Shortage of Qualified Resources 12

Survey Results 22% There is a big data talent shortage. No Shortage 40% very difficult to find and hire big data professionals No issue 78% 60% 3% senior marketers do not have the right talent Suffient talent level 30% there is a knowledge gap between big data workers and those commissioning the projects (e.g., managers and CIOs) 30% very difficult to find and hire business leaders and managers who could identify and optimize business applications in big data 97% No Gap No Problem 70% 70% - Survey by The Big Data London group - Survey by NewVantage Partners - Survey by CMO magazine 13

Seeking Holistic Approach 14

Definition Big Data Competency is a measurable set of knowledge, skills, abilities, behaviors, and other attributes that an individual needs to perform work roles or occupational functions effectively in the Big Data space. 16

Aspects Theoretical knowledge Behavior 17

Benefits of Big Data Competency Selection Training & Development Performance Management Career Paths Succession Planning Screen job candidates Provide a complete picture of the job requirements Increase the likelihood of selecting and interviewing only individuals who are likely to succeed on the job Minimize the investment (both time and money) Enable a more systematic and valid interview and selection process Help distinguish between competencies that are trainable after hiring and those are more difficult to develop Develop individual learning plans for individual or groups of employees Focus training and development plans to address missing competencies or raise level of proficiency Enable people to focus on the skills, knowledge and characteristics that have the most impact on job effectiveness Ensure that training and development opportunities are aligned with organizational needs Make the most effective use of training and development time and dollars Provide a competency framework for ongoing coaching and feedback, both development and remedial Provide regular measurement of targeted behaviors and performance outcomes linked to job competency profile critical factors. Provides a shared understanding of what will be monitored, measured, and rewarded Focus and facilitate the performance appraisal discussion appropriately on performance and development Provide focus for gaining information about a person s behavior on the job Enable effectiveness goalsetting around required development efforts and performance outcomes Develop stepping stones necessary for promotion and long term career-growth Clarify the skills, knowledge, and characteristics required for the job or role in question and for the follow-on jobs Identify necessary levels of proficiency for follow-on jobs Allow for the identification of clear, valid, legally defensible and achievable benchmarks for employees to progress upward Take the guesswork out of career progression discussions Careful, methodical preparation focused on retaining and growing the competency portfolios critical for the organization to survive and prosper Provide a method to assess candidates readiness for the role Focus training and development plans to address missing competencies or gaps in competency proficiency levels Allow an organization to measures its bench strength Provide a competency framework for the transfer of critical knowledge, skills, and experience prior to succession Inform curriculum development for leadership development programs 18

Big Data Competency Model Managerial (process, leadership) Organizational (vision, value, culture) Behavioral (collaboration, influence) Functional (tech expertise, method) 19

Big Data Capability Policies, Processes & Standards Measurement & Monitoring Organization Technology Strategy Big Data Program Communication 20

Big Data Capability Components Big Data Program 21

Big Data Thinking Scientific and engineering approach to Big Data problem solving Implementation Problem Diagnosis Logical sequence to troubleshoot and analyze the problem in a methodical fashion Pragmatic undertaking Prototype Facts Solution Analysis Hypothesis 23

Defining Problem Definition A problem is a situation that is judged as something that needs to be corrected implies that a state of "wholeness" does not exist Importance It is the job of the Big Data professionals to make sure they re solving the right problem. It may not be the one presented to us by the client. What do we really need to solve? Considerations Most of the problems are initially identified by our clients Defining the problem clearly improves focus it drives the analytical process Getting to a clearly defined problem is often discovery driven Start with a conceptual definition and through analysis (root cause, impact analysis, etc.) you shape and redefine the problem in terms of issues 24

Diagnosing Problem Definition Importance Considerations Examine the issues and challenges Root cause analysis Find the real problem Derive from the symptoms Investigate the pain points, barriers, and limitations Uncover the blind spots Leverage experience and patterns 25

Collecting Facts Definition Meaningful information (has merit not false) that is qualitative (expert opinions) or quantitative (measurable performance) to your decisions Importance Gathering relevant data and information is a critical step in supporting the analyses required for proving or disproving the hypotheses Considerations Know where to dig Know how to filter through information Know how to verify Has happened in the past Know how to apply Relates to what you are trying to solve 26

Conducting Analysis Definition Importance Considerations The deliberate process of breaking a problem down through the application of knowledge and various analytical techniques Analysis of the facts is required to generate, prove or disprove the hypotheses Analysis provides an understanding of issues and drivers behind the problem It is generally better to spend more time analyzing the data and information as opposed to collecting them. The goal is to find the golden nuggets that quickly confirm or deny a hypothesis Root cause analysis, storyboarding, and force field analysis are some of many analytical techniques that can applied 27

Formulating Hypotheses Definition: Hypothesis is a tentative explanation for an observation that can be tested (i.e. proved or disproved) by further investigation Importance Start at the end - Figuring out the solution to the problem, i.e. "hypothesizing", before you start will help build a roadmap for approaching the problem Considerations Hypotheses can be expressed as possible root causes of the problem Breaking down the problem into key drivers (root causes) can help formulate hypotheses 28

Designing Solution Definition Importance Considerations Solutions are the final recommendations presented to the clients based on the outcomes of the hypothesis testing and validation Solutions are typically the deliverables that the clients pay for. It is important to ensure the solution fits the client solutions are useless if they cannot be implemented Running an actual example through the solution is an effective way of testing the effectiveness and viability of the solution 29

Building Prototype Definition Importance Considerations For POC For bake-offs Clarify and refine requirements Validate assumptions Verify the suitability for adoption Flush out dependencies Pave the way for full adoption and migration Hugh Matrix ACH method 30

Developing Implementation Definition Implementing a runnable working prototype or fullblown application that realize the functions and non-functional requirements of the problem Importance Demonstration of the feasibility and costeffectiveness of the solution proposed or prescribed. Considerations It is important to ensure the solution fits the client solutions are useless if they cannot be implemented Running an actual example through the solution is an effective way of testing the effectiveness and viability of the solution 31

Summary 33

To Learn More Overview of Big Data and NoSQL Big Data Management and Governance Advanced Big Data and NoSQL Big Data Devops and Operations Training & Mentoring Big Data Science and Analytics Big Data Design and Architecture Big Data Engineering Big Data Technology 34

Contact: mail@tonyshan.com Copyright 2014, Tony Shan. All rights reserved. All materials, contents and forms contained in this presentation are the intellectual property of Tony Shan and may not be copied, reproduced, distributed or displayed without author's express written permission. Other streams of data and information from Internet are adapted and incorporated in the presentation for reference and illustration purposes. Not all sources are mentioned on the slides due to space and time constraints. The author does not warrant, either expressly or implied, the accuracy, timeliness, or appropriateness of the information contained in this deck. The author disclaims any responsibility for content errors, omissions, or infringing material, and disclaims any responsibility associated with relying on the information provided in this document. The author also disclaims all liability for any material contained in other resources linked to this file. 35