10 Analytics & Optimization. From Code to Product gidgreen.com/course

Transcription

1 10 Analytics & Optimization From Code to Product gidgreen.com/course

2 Lecture 10 Introduction Data collection Website metrics Optimization Competitive intelligence Surveys Tools and books From Code to Product Lecture 10 Analytics Slide 2 gidgreen.com/course

3 Why analytics? Quantify success/failure For yourselves For investors Against competition Scientific decisions No blind faith Fewer arguments Avoid HiPPO = highest paid person s opinion From Code to Product Lecture 10 Analytics Slide 3 gidgreen.com/course

4 Good analytics Simple Few in number Relevant Unambiguous Actionable Instant (or nearly) Repeatable From Code to Product Lecture 10 Analytics Slide 4 gidgreen.com/course

5 AARRR Metrics for pirates Acquisition Activation Retention Referral Revenue Site visit or app download Registration or usage Repeat usage Brings other people Generate cash Dave McClure, 500 Startups From Code to Product Lecture 10 Analytics Slide 5 gidgreen.com/course

6 Some quotes What gets measured, gets managed. Peter Drucker The only metrics that entrepreneurs should invest energy in collecting are those that help them make decisions. Eric Ries, The Lean Startup From Code to Product Lecture 10 Analytics Slide 6 gidgreen.com/course

8 In-app analytics Home rolled or third party Store usage information locally Call home when online Privacy concerns Confirmation dialog? Complete access to device But you will be caught! Problem: slow iteration From Code to Product Lecture 10 Analytics Slide 8 gidgreen.com/course

9 In-app integration From Code to Product Lecture 10 Analytics Slide 9 gidgreen.com/course

10 Reporting app events From Code to Product Lecture 10 Analytics Slide 10 gidgreen.com/course

11 Web analytics All activity visible to site Users don t expect privacy Web servers log requests Also: Javascript solutions Page view centric Other events require integration Coffee break? Events not sessions From Code to Product Lecture 10 Analytics Slide 11 gidgreen.com/course

12 A web server log line [06/May/2012:08:13: ] "GET / HTTP/ " "Mozilla/5.0 (iphone; CPU iphone OS 5_1 like Mac OS X) AppleWebKit/ (KHTML, like Gecko) Mobile/9B179 Safari/ " From Code to Product Lecture 10 Analytics Slide 12 gidgreen.com/course

13 Javascript tracking code <script type="text/javascript > var _gaq = _gaq []; _gaq.push(['_setaccount', 'UA ']); _gaq.push(['_trackpageview']); (function() { var ga = document.createelement('script'); ga.type = 'text/javascript'; ga.async = true; ga.src = (' == document.location.protocol? ' : ' + '.google-analytics.com/ga.js'; var s = document.getelementsbytagname('script')[0]; s.parentnode.insertbefore(ga, s); })(); </script> From Code to Product Lecture 10 Analytics Slide 13 gidgreen.com/course

14 Web metrics alternatives Server logs Javascript Home-made Integration None Via HTML Server code Convenience Download + analyze Web-based access Up to you Delay None Up to 24 hours Up to you Reporting Varies Advanced Up to you Other events Hard Via API Easy Data leakage None Total! None From Code to Product Lecture 4 UI Design Slide 14 gidgreen.com/course

15 Track web users by IP address Given for every web request Good for geography But: proxies, classrooms, router resets Cookies Track user browser over long term But: clearing, multi-browsing, first request Customization of web server From Code to Product Lecture 10 Analytics Slide 15 gidgreen.com/course

16 Track web users by Log in Reliable for registered users But: anonymous users, multiple accounts Requires custom logging tools Solution: combine! Intelligently tie IPs, cookies and accounts Example: user registration Data always incomplete From Code to Product Lecture 10 Analytics Slide 16 gidgreen.com/course

18 Basic website metrics From Code to Product Lecture 10 Analytics Slide 18 gidgreen.com/course

19 Immediate questions When does one visit end? GA: 30 minutes without activity What makes a visitor unique? GA: Tracking cookie How is duration calculated? GA: Time between first and last pages What makes a visitor new? GA: Never visited your site before From Code to Product Lecture 10 Analytics Slide 19 gidgreen.com/course

20 Geography From Code to Product Lecture 6 BM Advertising Slide 20 gidgreen.com/course

21 Demographics From Code to Product Lecture 6 BM Advertising Slide 21 gidgreen.com/course

22 Frequency report From Code to Product Lecture 10 Analytics Slide 22 gidgreen.com/course

23 Sources of traffic Type-in (no referrer) Includes browser bookmarks Search engines Navigational search = type-in Referrals Website links or social media Paid advertising campaigns From Code to Product Lecture 10 Analytics Slide 23 gidgreen.com/course

24 The multitouch problem There s history before the referrer Who deserves the credit, e.g. affiliates So who gets the credit? Last click (standard) First click (unrealistic) Even split Split weighted to last From Code to Product Lecture 10 Analytics Slide 24 gidgreen.com/course

25 Search engine queries Also: internal site search From Code to Product Lecture 10 Analytics Slide 25 gidgreen.com/course

26 Popular pages From Code to Product Lecture 10 Analytics Slide 26 gidgreen.com/course

27 Landing/entry pages You can t choose your home page A. Kaushik From Code to Product Lecture 10 Analytics Slide 27 gidgreen.com/course

28 Clickmaps and heatmaps From Code to Product Lecture 10 Analytics Slide 28 gidgreen.com/course

29 Conversion funnel Source: From Code to Product Lecture 10 Analytics Slide 29 gidgreen.com/course

30 Sampling methods Popular site => lots of data Burden to collect, slow to analyze Don t record all events Choose important pages Random subset of visitors Random subset of pageviews Sub-sample when analyzing By page or visitor From Code to Product Lecture 10 Analytics Slide 30 gidgreen.com/course

31 Staleness due to changes in Content User familiarity Early adopters vs... Search engine rankings Market needs Devices Cookies From Code to Product Lecture 10 Analytics Slide 31 gidgreen.com/course

33 Optimization You don t know how users behave Example: show price early on? Small changes => big results But which small changes? Use a scientific methodology Easy to set up Easy to get report Statistical significance From Code to Product Lecture 10 Analytics Slide 33 gidgreen.com/course

34 Wording example Source: you_should_follow_me_on_twitter.html From Code to Product Lecture 10 Analytics Slide 34 gidgreen.com/course

35 A/B testing Two parallel variations Current vs challenger Assign randomly and evenly What about previous visitors? Repeat requests within a session? Set test length in advance Length of time or number of visits Chi-squared (or similar) test From Code to Product Lecture 10 Analytics Slide 35 gidgreen.com/course

36 Contingency table Product purchased Not purchased From Code to Product Lecture 10 Analytics Slide 36 gidgreen.com/course

37 Multivariate testing Source: From Code to Product Lecture 10 Analytics Slide 37 gidgreen.com/course

38 Multivariate testing Best to use third-party tool Full factorial vs partial factorial Certainty vs efficiency From Code to Product Lecture 10 Analytics Slide 38 gidgreen.com/course

39 Optimization pitfalls Preconception driven Too many similar tests Checking before it s done Wrong goal e.g. started vs completed purchases Unfair test Different time periods New vs returning users From Code to Product Lecture 10 Analytics Slide 39 gidgreen.com/course

40 More complex tests Non-binary outcomes Size of purchase, length of stay Cohort / longitudinal tests Whole-site multivariate testing Pricing How to prevent a riot? Spot diminishing returns Focus on registration, payment, etc From Code to Product Lecture 10 Analytics Slide 40 gidgreen.com/course

42 Finding competitors From Code to Product Lecture 10 Analytics Slide 42 gidgreen.com/course

43 Searches for product From Code to Product Lecture 10 Analytics Slide 43 gidgreen.com/course

44 But From Code to Product Lecture 10 Analytics Slide 44 gidgreen.com/course

45 Ranking for general searches From Code to Product Lecture 10 Analytics Slide 45 gidgreen.com/course

46 App Store searches From Code to Product Lecture 10 Analytics Slide 46 gidgreen.com/course

47 Online mentions From Code to Product Lecture 10 Analytics Slide 47 gidgreen.com/course

48 Website traffic From Code to Product Lecture 10 Analytics Slide 48 gidgreen.com/course

49 Website traffic From Code to Product Lecture 10 Analytics Slide 49 gidgreen.com/course

50 Downloads/installs From Code to Product Lecture 10 Analytics Slide 50 gidgreen.com/course

51 Registrations From Code to Product Lecture 10 Analytics Slide 51 gidgreen.com/course

52 Revenue Also: UK private companies From Code to Product Lecture 10 Analytics Slide 52 gidgreen.com/course

53 Revenue $200k From Code to Product Lecture 10 Analytics Slide 53 gidgreen.com/course

55 Why surveys? Customer feedback en masse Initiated by you ( /web) Avoid vocal minority Understand market Job descriptions Size of company Use of product How did you find me? From Code to Product Lecture 10 Analytics Slide 55 gidgreen.com/course

56 Why surveys? Help with strategic decisions Premium offerings Major new versions Customer satisfaction Quantify word of mouth Understand abandonment But hard to motivate response Open-ended feedback From Code to Product Lecture 10 Analytics Slide 56 gidgreen.com/course

57 Sources of bias Non-response bias Busy customer bad customer Response bias Word questions objectively Predictions vs facts Would you pay? How much? Snapshot in time Lots of data vs ongoing data From Code to Product Lecture 10 Analytics Slide 57 gidgreen.com/course

58 Getting users to survey Prominent link in product Prize giveaway Response to support Mass mailing Cold calling Bias bias bias... From Code to Product Lecture 10 Analytics Slide 58 gidgreen.com/course

59 Good survey design Keep it short! Focus on objectives Minimize burden on user Easy questions, especially at start Multiple choice Make it feel anonymous Social desirability bias Free text at end From Code to Product Lecture 10 Analytics Slide 59 gidgreen.com/course

60 Bad questions When did you last go online and buy something? Would you buy our superior product? Are you willing to pay for things online? If we created a reliable and bug-free product which had all of the features that you requested in response to the questions in this survey, would you be willing to pay us $10 per month for it? What are you looking for? From Code to Product Lecture 10 Analytics Slide 60 gidgreen.com/course

61 Analyzing survey data Manual review At least for free text field Histograms Pairwise correlations Especially against price Clustering Identify price points Decide who is worth serving From Code to Product Lecture 10 Analytics Slide 61 gidgreen.com/course

62 Pairwise correlation 1 Multiple Recipients? R² = $0 $20 $40 $60 $80 From Code to Product Lecture 10 Analytics Slide 62 gidgreen.com/course

63 Mini surveys From Code to Product Lecture 10 Analytics Slide 63 gidgreen.com/course

65 Analytics tools From Code to Product Lecture 10 Analytics Slide 65 gidgreen.com/course

66 Other tools From Code to Product Lecture 10 Analytics Slide 66 gidgreen.com/course

67 Books From Code to Product Lecture 10 Analytics Slide 67 gidgreen.com/course

68 We didn t cover Social media analytics Popularity Sentiment analysis Video analytics Attention Embeds Content reuse From Code to Product Lecture 10 Analytics Slide 68 gidgreen.com/course