WHAT IS DATA SCIENCE? Grace Tang, Data Scientist, 99.co

Transcription

1 WHAT IS DATA SCIENCE? Grace Tang, Data Scientist, 99.co

2

3 WHAT IS DATA SCIENCE???

4 WHAT IS DATA SCIENCE??? What do YOU want to know about Data Science?

5 AGENDA Data Science in the Wild Data Analysis Big Data Tools of the Trade Data Scientists Data Science in Your Company

6 WHAT IS DATA SCIENCE? DATA SCIENCE IN THE WILD

7

8 WHAT IS DATA SCIENCE? RAW DATA VALUE Data Science is the extraction of VALUE from RAW DATA

9 TRENDS Source: mailchimp.com

10 DATA SCIENCE IN THE WILD DESCRIPTIVE Source: mailchimp.com

11

12 RECOMMENDATIONS Source: amazon.com

13 Source: linkedin.com

14 Source: 99.co

15 OUTLIER DETECTION Source: chase.com

16 A/B TESTING

17

18

19 WHAT IS DATA SCIENCE? DATA ANALYSIS

20 DATA MINING Extracting information and knowledge from data

21 DATA WRANGLING Extracting useable data from raw data csv JSO N Images Text

22 DATA ANALYSIS PIPELINE ANALYZE / INTERP

23 DATA ANALYSIS PIPELINE COLLECT EXTRACT CLEAN / AUGMENT ANALYZE / INTERP DATA WRANGLING

24 DATA ANALYSIS PIPELINE Data Collection What data will be useful? How will data be collected? How will it be stored?

25 DATA ANALYSIS PIPELINE Data Extraction Converting raw data into a more useable form

26 DATA ANALYSIS PIPELINE csv JSO N Images Text \

27 DATA WRANGLING - TEXT Source:

28 DATA WRANGLING - IMAGES Source: venturebeat.com

29 DATA CLEANING Inaccurate Information Error checking Typos: spell check Inaccuracies: outlier detection

30 DATA CLEANING Missing Information Imputation: Replace missing information with best guess User: Cheryl Interests: Clubbing, Instagram Gender:?? Age:??

31 DATA CLEANING Missing Information Imputation: Replace missing information with best guess Heuristic Mean Mode Regression

32 AUGMENTATION What is Cheryl s spending power?

33 AUGMENTATION

34 AUGMENTATION

35 DATA ANALYSIS PIPELINE Put everything together!

36 DATA ANALYSIS PIPELINE Exploit Analyze data to achieve business goals Descriptive

37 DATA ANALYSIS PIPELINE Exploit Analyze data to achieve business goals Descriptive Predictive

38 DATA ANALYSIS PIPELINE Exploit Analyze data to achieve business goals Descriptive Predictive Prescriptive

39 A/B TESTING

40 A/B TESTING

41

42

43 WHAT IS DATA SCIENCE? BIG DATA

44

45 BIG DATA Traditional data analysis Data Results Analyze

46 BIG DATA Big data Data Analyze

47 BIG DATA Big data Data

48 BIG DATA Too big for one machine - not enough storage - not enough memory / computational power

49 BIG DATA WHY IS IT SO BIG NOW? VOLUME VARIETY VELOCITY

50 BIG DATA WHY IS IT SO BIG NOW?

51

52 BIG DATA WHY IS IT SO BIG NOW?

53 BIG DATA WHY IS IT SO BIG NOW?

54 BIG DATA Need for scalable Storage Unstructured Lots of it Computation

55 BIG DATA Source: Google

56 BIG DATA Too Small Data Statistical significance Cold-start problem To predict user behavior, you must first have user behavior

57

58 WHAT IS DATA SCIENCE? TOOLS OF THE TRADE

59 TOOLS OF THE TRADE A tool for everything.. Data collection Storage Visualization Statistics A/B testing Machine learning

60 TOOLS OF THE TRADE Services Out-of-the-box Customer Support Paid Not customized Programs Expertise required Community Support Free Highly customizable

61 ANALYSIS PLATFORMS

62 ANALYSIS PLATFORMS Events Traffic User acquisition, flow, segmentation Ad campaign performance Basic A/B testing

63 ANALYSIS PLATFORMS $$$

64

65

66 A/B TESTING $$$

67

68 CLOUD SERVICES Cloud storage Cloud computing Machine Learning as a Service

69 ANALYSIS / VISUALIZATION $0

70

71 TOOLS OF THE TRADE - ANALYSIS / VISUALIZATION $0

72 BIG DATA $$$

73 WHAT IS DATA SCIENCE? DATA SCIENTISTS

74 source: grey s anatomy

75 DATA SCIENTISTS Skills Statistics / Research Business skills / Domain knowledge Programming / Computer science

76 DATA SCIENTISTS Data Engineers Programming Computer Science Storage architecture Data collection

77 Source: engineering.viki.com

78 DATA ENGINEERS How do I... make sure data is never lost? collect and store LOTS of different types of data? make it easy for people to access and analyze the data?

79 DATA SCIENTISTS Software developers Programming Computer Science Algorithm design Runtime optimization

80 Source: linkedin.com

81 SOFTWARE DEVELOPERS How do I... make accurate predictions? make real-time predictions fast?

82 DATA SCIENTISTS Sources Programming Computer Science Computer science background architecture, infrastructure artificial intelligence, algorithm design

83 DATA SCIENTISTS Business analysts Business skills Domain knowledge Specialized knowledge of the field Data visualization Communication

84

85 BUSINESS ANALYSTS How do I... visualize the data? generate actionable insights? communicate results?

86 DATA SCIENTISTS Sources Business skills Domain knowledge Business background

87 DATA SCIENTISTS Researchers Statistics Research Conduct experiments Evaluate effectiveness of features and algorithms

88 A/B TESTING

89 A/B TESTING

90 DATA SCIENTISTS How do I... formulate a hypothesis? test the hypothesis? determine if results are significant?

91 DATA SCIENTISTS Sources Statistics Research Academic background

92 WHAT IS DATA SCIENCE? DATA SCIENCE IN YOUR COMPANY

93 DATA SCIENCE IN YOUR COMPANY What problems do you want to solve? Do you need data scientists? What kind of data scientist(s) do you need? Big vs. small companies

94 DATA SCIENCE IN YOUR COMPANY What do you want to achieve with data science? Insights? Data visualization? Optimize your product? A/B testing? Predictive algorithms? Recommendation systems?

95 DATA SCIENCE IN YOUR COMPANY What do you want to achieve with data science? Insights? Data visualization? - Google Analytics Optimize your product? A/B testing? - Optimizely Predictive algorithms? Recommendation systems?

96 DATA SCIENCE IN YOUR COMPANY Data scientist vs services Data scientists Need a salary Understand your product and business goals intim Equipped to identify solutions for problems unique

97 DATA SCIENCE IN YOUR COMPANY Data scientist vs services Services Do not need a salary Horizontal: Build general solutions Still have to spend time learning tools and underst Still cost money

98 DATA SCIENTISTS Specialist? Statistics / Research All-rounder? Business skills Programming

99 DATA SCIENCE IN YOUR COMPANY Big companies More resources Dedicated teams for each part of the pipeline DATA ENGINEERS SOFTWARE DEVELOPERS - Storage architecture - Algorithm development - Data collection, cleaning, wrangling RESEARCHER - Feature evaluation

100 DATA SCIENCE IN YOUR COMPANY Small companies Fewer resources Data scientist might have to manage the entire pipe DATA SCIENTIST - Everything

101 DATA SCIENCE IN YOUR COMPANY Hiring Define what you need What problem do you want to solve? How much can your existing team already do? Determine required skill set Multi-tasker? Specialist?

102 WHAT IS DATA SCIENCE? Q&A

103

104 WHAT IS DATA