Big Data & Netflix. Paul Ellwood February 9th, 2015

Transcription

1 Big Data & Netflix Paul Ellwood February 9th, 2015

2 Who Am I? Director, Data Science & Engineering Also Leader, DataKind San Francisco chapter Formerly: Director, Product Netflix Formerly: Associate Partner, Analytics & Rosetta MS in Predictive Analytics from Northwestern BS in Systems & Control Eng. from CWRU

3

4 2007!!!

5 The Future of TV is Finally Here

6 Competition Winning the Moment of Truth

7 Stay focused and run fast

8 Culture of Freedom & Responsibility Context, not control Highly aligned, loosely coupled No brilliant jerks Professional sports team No time or expense policies

9 Netflix Expense Policy Act in Netflix s Best Interest

10 QA Team?

11 Accept things WILL break

12 Safety Nets

13 Give folks what they need

14 Eliminate Rules & Process

15 Tell vs. Ask

16 Conrad Gessner The modern world overwhelms people with data and this overabundance is both "confusing and harmful" to the mind.

17 Francis Bacon If a man will begin with certainties, he shall end in doubts; but if he will be content to begin with doubts, he shall end in certainties.

18 Innovation Cycle Formulate Hypothesis Productize Offline Experiment A/B Test

19 Analytics Teams Data Science & Engineering Science & Algorithms Business Stakeholders Consumer Insights Financial Planning & Analysis

20 (Mostly Cloud) Data Platform Source Source Systems Source Systems Source Systems Systems S3 Sting

21 Data Science & Engineering Enabling a data-driven culture

22 Metrics What does success look like?

23 Data Model How will people use it?

24 Data Integration How can we stitch it all together?

25 Data Visualization How should we present it?

26 Insights What does it mean?

27 Vertically Aligned Product Streaming Content Marketing Finance Messaging Content Delivery Content Buying Digital Customer Service Sign-up Flow Device Partners Originals TV Billing Social Streaming Supply Chain Out-of-Home Payments Search Infrastructure Consumer Device UIs Insights Rec Algos

28 Cross-functional Teams Horizontal Roles Skills Technologies Insights Data Scientists Data Analysts Metrics, Analysis, Data Mining, Modeling SQL, R, Python Data Visualization Data Visualization Engineers Design & Build Reports, Interactive Dashboards Microstrategy, Tableau, D3 Data Engineering Data Engineers Data Modeling, ETL Amazon S3, Hadoop, MapReduce, Pig, Python, Hive, Teradata, Redshift, SQL

29 Data Privacy VPPA EU DPA Judge Robert Bork ( ) 1988 Video Privacy Protection Act:...A video tape service provider who knowingly discloses, to any person, personally identifiable information concerning any consumer shall be liable for - Protecting personal data is a fundamental right - Personal data can only be kept in the context of the legitimate ordinary business activities - Right to be forgotten :

30 Marketing City-level metrics LTV modeling Media mix modeling Pioneering use of Google s digital stack Audience Targeting Originals Marketing Real Time Bidding (RTB) Optimization Looking for a new leader!

31 Content Content performance on service Target price for content Catalog coverage for current customers Catalog coverage for future customers Digital Supply Chain

32 Product Multi-channel Messaging Sign-up flow optimization Deeper personalization of recommendations New UI designs Kids Impact of social Search

33 Streaming Optimizing the Streaming experience ISP content delivery performance Device partnerships

34 FinOps Fraud Reduction Global payment processing Gift Cards Bill on Behalf Of Talent & Recruiting Analytics Customer Service -> Insights

35 Open Source Analytic Tools ScorePMML Time Series Anomaly Detection coming

36 Questions?

37 APPENDIX

38 Star Ratings Row Order Row Ranking Over 50% of hours from homepage Evidence

39 Smart TV

40 Extract, Transform, Load (ETL) Logging (Raw) Searches/clicks/plays (Clean Fact) Sums, Avgs (Clean Aggregate) Signals (Join Aggregate) Offline Analysis & Signal Discovery:

41 A/B Testing: Randomly allocate users to offline strategies Schedule daily job to update model(s) data Measure changes in key metrics over time vs. production Primary: Retention, streaming hours Secondary: Hours from titles discovered in search Genie: Lipstick: Quinto:

42 Determine what data is good enough Expect the unexpected (edge cases) Be vigilant (everyone involved with A/B test is looking at website/ui) Teamwork highly aligned, loosely coupled Data-driven decision making

43 The Challenge of Choice

44 Personalized Recommendations

45 Adaptive Row Ordering

46 Predicted Ratings

47 My List

48 Facebook Integration

49 User Experience

50 Price Tiers

51

52 Often product decisions are made by the most passionate person in the room

53 Sheer volume can get in the way of making the right decisions

54 Core Metrics What are we trying to accomplish?

55 Our Core Metrics Non-member Conversion rates Rate of paying for second month Member Retention now revenue weighted Streaming hours a great proxy for retention

56 Hypothesis Formulation Back to that Freedom & Responsibility thing

57 Qualitative Studies Voice of the Customer

58 Data Exploration Digging deep

59 Time Investment Data Visualization Tool Trade-offs Custom Web Products Sting Robustness

60 Handling the raw data

61 Analytics What does it mean?

62 Example Requests Launch Reporting & Analysis Reach Projections Usage Patterns Viewing Patterns Feature Mining for Algorithmic Models Predictive Models

63 Hypothesis Evaluation

64 Large Scale A/B Testing Let the customer decide

65 Product Strat Meetings A meeting, really??