Measuring CDN Performance. Hooman Beheshti, VP Technology

Transcription

1 Measuring CDN Performance Hooman Beheshti, VP Technology

2 Why this matters Performance is one of the main reasons we use a CDN Seems easy to measure, but isn t Performance is an easy way to comparison shop Nuanced Metric overload

3 Common mistakes Getting lost in data Focusing on one thing and one thing alone We like numbers! Forgetting about our applications Letting vendors influence us (I m fully aware of the irony!)

4 Goals Share measurement experiences What does the measurement landscape look like Help guide towards good testing plan Avoid pitfalls

5 Background

6

7 CDNs: delivery platforms We won t talk about Extra stuff: security, GSLB, TLS offload, etc We also won t talk about page- level optimizations We will focus on the delivery side Delivering HTTP objects

8 Delivery: static/cached objects CDN Node Client Origin

9 Delivery: dynamic/uncached objects

10 What we ll be focusing on How we measure Metrics to measure What to measure Some gotchas, misconceptions, and common mistakes

11 Measurement Techniques (how we measure)

12 Measurement techniques Pretend Users Synthetic tests Not actual users Real Users In the browser Actual users

13 Synthetic testing

14 Synthetic testing Usually a large network of test nodes all over the globe Highly scalable, can do lots of tests at once Many vendors that have this model Examples: Catchpoint, Dynatrace(Gomez), Keynote, Pingdom, etc

15 Synthetic testing Built to do full performance and availability testing Lots of monitors emulating what real users do DNS, Traceroute, Ping, Streaming, Mobile HTTP Object Browser Transactions/Flows Tests set up with some frequency to repeatedly test things Aggregates reported

16 Backbone nodes Test machines sitting in datacenters all around the globe Terrible indicators of raw performance No latency Infinite bandwidth But really good at: Availability Scale Backend problems Global reach

17

18 Backbone nodes Test machines sitting in datacenters all around the globe Terrible indicators of raw performance No latency Infinite bandwidth But really good at: Availability Scale Backend problems Global reach

19 Last mile nodes Test machines sitting behind a real home- like internet connection Much better at reporting what you can expect from users, but sometimes unreliable Also not as dense in deployment

20 backbone last mile

21 Synthetic testing Pros Geographic distribution Lots of options for testing Really good for on- the- spot troubleshooting Last- mile nodes can be pretty good proxies for performance Cons Not real users! Backbone nodes can be misleading

22 Real users (RUM)

23 RUM Use javascript to collect timing metrics Can collect lots of things through browser APIs Page metrics, asset metrics, user- defined metrics

24 Use test assets Use this model to initiate tests in the browser Some vendors: Cedexis, TurboBytes, CloudHarmony, more Usually, this isn t their business, but the data drives their main business objectives You can build this yourself too

25 Use real assets in the page Collect timings from actual objects Resource timing Vendors SOASTA, New Relic, most synthetic vendors Boomerang (open source) Google Analytics User Timings

26 DATA, DATA, DATA For either RUM technique, we need A LOT of data Too many variances Most vendors don t use averages Medians and percentiles

27 Real user measurements Pros Real users, real browsers, real world conditions If you use your own content, could be close to what your users experience With enough data, great for granular analysis Cons We need a lot of data If you do it yourself, data infrastructures aren t trivial

28 Measurement Metrics

29 Client Server

30 Client Server 1 x RTT

31 Client Server DNS DNS

32 Client Server DNS DNS TCP

33 Client Server DNS DNS TCP (TLS)

34 Client Server DNS DNS TCP (TLS) HTTP

35 Client Server DNS DNS TCP (TLS) HTTP Download

36 Time DNS TCP (TLS) TTFB Download (TTLB- TTFB)

37 Time DNS TCP (TLS) TTFB Download (TTLB- TTFB) DNS RTT to DNS server, DNS iterations, DNS caching and TTLs

38 Time DNS TCP (TLS) TTFB Download (TTLB- TTFB) DNS RTT to DNS server, DNS iterations, DNS caching and TTLs TCP RTT to cache server (CDN footprint & routing algorithms)

39 Time DNS TCP (TLS) TTFB Download (TTLB- TTFB) DNS RTT to DNS server, DNS iterations, DNS caching and TTLs TCP RTT to cache server (CDN footprint & routing algorithms) (TLS) RTT to cache server (or RTTs depending on TLS False Start), efficiency of TLS engine

40 Time DNS TCP (TLS) TTFB Download (TTLB- TTFB) DNS RTT to DNS server, DNS iterations, DNS caching and TTLs TCP RTT to cache server (CDN footprint & routing algorithms) (TLS) RTT to cache server (or RTTs depending on TLS False Start), efficiency of TLS engine TTFB RTT to where the object is stored + storage efficiency (different for requests to origin); lower bound = network RTT

41 Time DNS TCP (TLS) TTFB Download (TTLB- TTFB) DNS RTT to DNS server, DNS iterations, DNS caching and TTLs TCP RTT to cache server (CDN footprint & routing algorithms) (TLS) RTT to cache server (or RTTs depending on TLS False Start), efficiency of TLS engine TTFB RTT to where the object is stored + storage efficiency (different for requests to origin); lower bound = network RTT TTLB- TTFB Bandwidth, congestion avoidance algorithms (and RTT!)

42 Core object metrics Not every request experiences every metric: DNS: once per domain TCP/TLS once per connection HTTP/Download for every object (not already in browser cache) All techniques/tools measure and report these metrics

43 Resource timing timing/

44 Resource timing window.performance.getentries()

45 object metrics or page metrics

46

47

48 Download: 15Mbps Upload: 5Mbps Latency: 10 ms, 25 ms

49 10 msec 25 msec

50 10 msec 25 msec

51 onload Speed Index Start Render 10 msec 25 msec

52 What the??? We always assume all things equal Too many factors affect page load time 3 rd parties (sometimes varying), content form origin, layout, JS execution, etc Too much variance

53 3 rd parties Source: httparchive.org

54 To be clear Always use webpagetest (or something like it) to understand your application s performance profile Continue to monitor application performance, and always spot check Be extremely careful when using it to gage/compare CDN performance, it can mislead you If using RUM to measure page metrics, with lots of data, things become a little more meaningful (data volume handles variance)

55 What to measure (plus the right metrics)

56 Most commonly Pick a normal object e.g. some object on the home page Set up testing from multiple places (usually with a synthetic vendor) And hopefully not backbone! Compare either overall load time, or some object metrics

57 Totally application- dependent

58 Example: web application

59 Web application: objects Ratio of objects coming from CDN cache vs those coming from origin (through CDN) should determine objects to test If HTML is from origin, we must measure it Essential to critical page metrics

60 Web application: object sizes

61 Web application: metrics On any page DNS queries only happen a small number of times 6 TCP connections per domain Many many many HTTP fetches Core metrics TTFB Download (TTLB- TTFB) if important large objects Should have a good idea of DNS/TCP/TLS, but less critical

62 Web application If CDN only for static/cacheable objects: One or two representative assets TTFB and maybe download most important Client CDN Node

63 X-Cache: HIT

64 Web application If CDN also for whole site Sample of key HTML pages, delivered from origin TTFB will show efficiency of routing (and connection management) to origin TTLB will show efficiency of delivery Client CDN Node Web Server

65 Web application If CDN also for whole site Sample of key HTML pages, delivered from origin TTFB will show efficiency of routing (and connection management) to origin TTLB will show efficiency of delivery Client CDN Node CDN Node Web Server

66 Example: software download

67 Software download: objects Pick a standard file that users will be downloading Representative file size Also pick something you expect to be on the CDN but not fetched all that often More on this later.

68 Software download: metrics Ratio of TCP- to- HTTP is closer to 1-1 Especially if you have a dedicated download domain Could mean the same for DNS For large files, we care about download time Core metrics: TTFB (+ TCP and maybe DNS, if applicable) TTLB TTLB- TTFB will usually be a larger component

69 Bandwidth/Download measurements

70 Download time Careful about where you expect this download to happen in the lifetime of a TCP connection In the beginning of the connection Function of init_cwnd and TCP slow start (and RTT) Later in the connection Function of congestion avoidance and bandwidth Large files will experience both

71 bytes time

72

73 Download time Careful about where you expect this download to happen in the lifetime of a TCP connection In the beginning of the connection Function of init_cwnd and TCP slow start (and RTT) Later in the connection Function of congestion avoidance and bandwidth Large files will experience both

74 Cache hit ratios

75 Cache hit ratio: traditional calculation 1 - Requests to Origin Total Requests

76 Origin

77 Cache Origin

78 Cache TCP Origin

79 Cache HTTP Origin

80 Cache HTTP Origin

81 Cache HTTP Origin

82 Cache HTTP Origin

83 Cache HTTP Origin

84 Cache Origin HOT COLD

85 Cache Origin cache hit

86 Cache hit ratio: traditional calculation 1 - Requests to Origin Total Requests

87 Isn t this better? Hits Total

88 Isn t this better? Hits Hits +

89 Cache hit ratio 1 - Requests to Origin Total Requests vs. Hits Hits +

90 Cache hit ratio 1 - Requests to Origin Total Requests vs. Hits Hits + Offload

91 Cache hit ratio 1 - Requests to Origin Total Requests vs. Hits Hits + Offload Performance

92 Effect on long tail content

93 Effect on long tail content (long tail: Cacheable but seldom fetched)

94 Popular Medium Tail (1hr) Long tail (6hr)

95 Popular Medium Tail (1hr) Long tail (6hr) Connect (median) Popular 1hr Tail 6hr Tail 14msec 15msec 16msec

96 Popular Medium Tail (1hr) Long tail (6hr) 77,000+ measurements 38,000+ measurements 6,400+ measurements Popular 1hr Tail 6hr Tail Connect (median) 14msec 15msec 16msec

97 Popular Medium Tail (1hr) Long tail (6hr) 77,000+ measurements 38,000+ measurements 6,400+ measurements Popular 1hr Tail 6hr Tail Connect (median) 14msec 15msec 16msec

98 Popular Medium Tail (1hr) Long tail (6hr) 77,000+ measurements 38,000+ measurements 6,400+ measurements Connect (median) Wait (median) Popular 14msec 19msec 1hr Tail 15msec 26msec 6hr Tail 16msec 32msec

99 Isn t this better? Popular Medium Tail (1hr) Long tail (6hr)

100 Now

101 Does all this really matter?

102 Does all this really matter? Yes, but

103

104

105 The bigger picture It s really easy to lock in on a metric Performance absolutely matters True performance isn t always as easy to measure

106 Storage model and long tail content

107 Cache hit ratios for offload and performance

108 Footprint

109 HTTP vs TLS footprint

110 Caching something you didn t think you could cache

111 Serve stale content if necessary

112 Key takeaways Everything is application- dependent Evaluate how your application works and what impacts performance the most Don t get locked into a single number Always know your application performance and bottlenecks Be mindful of the bigger picture!

113 Thank you!