A Tale of Three CDNs: An Active Measurement Study of Hulu and Its CDNs

Transcription

1 A Tale of Three CDNs: An Active Measurement Study of Hulu and Its CDNs Vijay Kumar Adhikari, Yang Guo, Fang Hao, Volker Hilt and Zhi-Li Zhang University of Minnesota, Bell-Labs/Alcatel-Lucent Abstract We study the Hulu online video service via active measurements. It is known that Hulu utilizes multiple CDNs to serve users video requests. The focus of this study is on how Hulu selects CDNs and how each CDN allocates resources (i.e., servers) to serve user requests. Based on our analysis of measurement data, we find that Hulu frequently changes preferred CDNs for users. However, once a CDN is selected, Hulu clients try to stay with the same CDN during the entire length of the movie even when performance of that CDN degrades. While the preferred CDN selection is not fixed, we observe that Hulu attempts to divide video requests among CDNs to attain a fixed target ratio. In terms of CDNs, we find that different CDNs employ different amounts of resources (servers) to serve Hulu content. I. INTRODUCTION Large-scale video content delivery is becoming the single most important contributor to the Internet traffic. For instance, in [1] Labovitz et al. reports that Google is one of the largest source of Internet traffic and a significant amount of its traffic is due to the YouTube video delivery. Within North America, Netflix uses about 3% of the peak-time downstream bandwidth for fixed-access networks, according to a recent report by Sandvine [2]. Moreover, according to comscore[3], more than 82 percent of the U.S. Internet audience viewed online video in February 211; and Hulu (with more than 27, 257, unique viewers per month) is one of the largest online video service provider. Just as these online video services continue to grow, so does the demand for higher quality video viewing experience. Many of these services make money either by serving advertisement or through monthly subscription fees. Therefore, poor user viewing experience will likely impact their revenue. In order to serve video content at a sufficiently good quality, some providers, for example, Google, build their in-house content distribution systems. However, because of the formiddable infrastructure costs and the vast expertise needed to operate such a content distribution infrastructure, a large majority rely on one or more third-party content distribution network (CDN) providers such as and to help distribute content to users. How a content provider select CDNs and how many resources (e.g., video servers) each CDN utilizes to serve content for the content provider is one of the most important factors affecting user experience. Likewise, how an online video service reacts and adapts to diverse and changing This study was done while Vijay Kumar Adhikari was a summer intern at Bell Labs, Alcatel-Lucent. It was supported in part by the NSF grants , CNS-9537, CNS and CNS network conditions also plays a crucial role in providing good user viewing experiences. In this paper, we study the video delivery system employed by Hulu one of most popular online video services serving primarily TV shows via active measurements. It is known that Hulu utilizes multiple CDNs to serve users video requests. The focus of this study is on i) how Hulu selects CDNs and ii) how each CDN allocate resources (i.e., servers) to serve user requests. For this purpose, we have conducted extensive data collection, careful analysis and inference and systematical experimentation to verify and validate our inferences and findings. We find that Hulu appears to distribute user requests among the CDNs in accordance with certain predetermined ratio. The selection of the preferred CDN for a given user does not seem to be taking into account the current condition of the network performance (between the user and the selected CDN). Further, Hulu frequently changes preferred CDNs for each user. However, once a CDN is selected, Hulu clients try to stay with the same CDN during the entire length of the movie even when performance of that CDN degrades. The CND is only changed when the chosen CDN cannot serve the lowest quality level of the video. In terms of CDNs, we find that the CDNs employ varying number of servers at different locations to serve Hulu content. The CDN resources (servers) allocated to Hulu are also used for other online video services that employ the same video streaming protocol suite; they are not used for other online video services that employ different video streaming technologies. The rest of this paper is organized as follows. A brief overview of Hulu is presented in Section II. In Section III, we describe in detail how Hulu uses multiple CDNs and how it picks a CDN to serve a given request and in Section IV, we explore how the selected CDN decides which server resource to use to serve that request. We describe related work in Section V and conclude the paper in Section VI. II. OVERVIEW In this section we provide an overview of Hulu s video delivery system. We explain what technologies and protocols Hulu uses for video delivery and what bit-rates they support. The most popular service that Hulu offers is the free service for desktop users. In addition, Hulu also offers a subscription based service called HuluPlus which supports additional platforms such as set top boxes and mobile devices, and also supports HD video quality. Video advertisement is another

2 LiveStreaming technology[4]. Hulu advertisements are single.flv files. These files are small (a few megabytes) and are downloaded over single HTTP transactions. Fig. 1. High-level Hulu architecture major component of Hulu. These are typically short video clips delivered to users prior to the main video content. Although our study also covers various aspects of HuluPlus service and video advertisement delivery, the primary focus of this paper is on the free desktop version of Hulu because of it its popularity and impact. To study Hulu service, we play multiple videos on different web browsers from multiple locations with different ISPs, both with and without firewalls and proxy servers. We also corroborate several of these observations using what is published on Hulu s help pages and their blogs. For HuluPlus, to understand how it works on mobile devices, we play a large number of videos from multiple locations using an iphone, an ipad and a Nexus One phone. For these experiments, we use a laptop as WiFi hotspot and configure the mobile devices to connect to that WiFi network. We then run tcpdump on the laptop to capture the packets exchanged between the mobile devices and Hulu servers. A high level architecture of Hulu video delivery for desktop clients is shown in Figure 1. The client gets the HTML pages for the video from Hulu s front-end web server at It then contacts s.hulu.com to obtain a manifest file that describes the server location, available bit-rates and other details. The client uses the instruction in the manifest file to contact a video server to download the video. The client also periodically sends its status report to t.hulu.com. Bandwidth Requirements Hulu videos are streamed at 48Kbps and 7Kbps. A few videos can also be streamed at 1Kbps. HuluPlus subscribers can also access videos in HD quality when available. Clients can switch between bitrates during playback based on available bandwidth, as we will explain later. CDNs Hulu uses three CDNs to deliver video contents to users:, and. Based on manifest files, a Hulu client is first assigned a preferred CDN hostname, and then uses DNS to select a server IP address. Protocol Hulu uses encrypted RTMP to deliver movies to desktop browsers. Hulu videos can be delivered over raw RTMP on port 1935 or RTMP tunneled over HTTP (RTMPT). Our experiments reveal that prefers raw RTMP whereas and prefers RTMPT. All three CDNs use RTMPT when TCP port 1935 is blocked (by a firewall, for instance). HuluPlus on a desktop browser uses the same technologies and protocols. However, on mobile devices HuluPlus uses adaptive streaming over HTTP. For instance, on iphone and ipad, HuluPlus content is delivered over HTTP III. CDN UTILIZATION STRATEGY In Section II, we see that Hulu employs multiple CDNs to serve its content. A question that naturally arises is: how does Hulu use the three CDNs? For instance, are all three CDNs used during one video playback session, or just one CDN at a time? How does Hulu decide which CDNs to pick is it based on performance of the CDNs for a given client, or on overall performance of the CDNs across all users or are they selected at random? Is the choice static or dynamic? We attempt to answer such questions in this section. A. Rate and CDN adaptation We analyze the captured packet traces to find that during normal playback, when the network condition is relatively stable, Hulu uses only one CDN server throughout the duration of a video. But interestingly, it usually switches to a different CDN server for the next video. To further understand how network conditions affect player behavior and CDN selection strategy, we conduct the following experiment using a tool called dummynet[5]. In the beginning, servers from each CDN are allowed to send data at 151 Kbps. At the end of every minute, we reduce the available bandwidth for the current active CDN by 1 Kbps till it reaches 1 Kbps. As we lower the available bandwidth for the current CDN while leaving the other CDNs intact, we notice that instead of switching to a different CDN, which is not throttled, the client keeps lowering the bit-rate and stays with the original CDN. This indicates that Hulu adapts to changing bandwidth by adjusting the bit-rates and continues to use the same CDN server as long as possible. Only when the current CDN server is unable to serve the lowest possible bit-rate, it switches to a different CDN server. In summary, Hulu prefers to stay with the same CDN server than to keep maintaining maximum achievable bandwidth and video quality for the user. As a result, if a user is assigned to a CDN experiencing degraded performance at the beginning of a video, that user is most likely to remain with the same CDN for the duration of the video. We next try to explore how CDN selection is made for each video playback. B. Status Reporting Several factors can potentially affect CDN selection: bandwidth between the client and servers of different CDNs, past performance history of different CDNs, and non-technical reasons such as pricing and business contracts. We first try to find out what information is available to Hulu for making such decision. From the packet trace, we find that the Hulu player sends periodic reports to a server that includes detailed information about the status of the client machine at that time, the CDN servers for video content and advertisements, and any problems encountered in the recent past. These periodic

3 status reports are sent to t.hulu.com which maps to the same single IP address from all the locations in US. Using WHOIS[6] queries, we learn that the corresponding IP address, , is allocated to Hulu. Examples of detailed performance information contained in the periodic reports include: video bit-rate, current video playback position, total amount of memory the client is using, the current bandwidth at the client machine, number of buffer underruns, and number of dropped frames. When the client adapts bit-rate due to changed network conditions, the periodic reports also include details on why the bit-rate was changed. For instance, one of the messages reads Move up since avg dropped FPS < 2 and bufferlength > 1. In summary, it appears that Hulu has sufficient performance data for CDN selection for any given request, if they choose to do it based upon user experience. C. CDN Selection Strategy We try to infer how much, if any, of the performance data contained in the status reports influences CDN selection. Hulu clients follow the manifest files they receive from the server to decide which CDN server to request video content from. Since Hulu encrypts the manifest file sent to the client, it is not easy to read the contents of the manifest files from the network traces. Therefore, we collect and read the manifest files by using a tool called get-flash-videos[7]. A small section of the content of an example Hulu manifest file is shown in Fig. 2. The last line in the figure shows Hulu s CDN preference in that manifest file. When we examine CDN preferences in a few manifest files, we observe that the preferred CDN server included in the manifest file seems very dynamic. For instance, when we make two requests simultaneously (or within a few seconds) for the same video, the preferred CDNs for those two requests can be different. To better understand the CDN selection strategy employed by Hulu, we request one manifest file every second for the same video from the same computer for 1 seconds. In Figure 3, we show how CDN preference changes very frequently. In this figure, X-axis shows the time and the Y-axis shows the three CDNs. Each * in the plot represents the CDN selected for a given request. Since the network conditions on the tested Hulu client is fairly stable during the experiment, the above result indicates that Hulu CDN selection is not based on instantaneous network conditions. To further understand the impact of various factors such as client location, video and time on CDN selection, we use the get-flash-videos tool to collect manifest data for 61 different videos of different genres, length, popularity and ratings available on Hulu from 13 different locations across the United States over multiple days (up to 24 days at one of the locations). The client machines on these locations are connected to residential bradband networks or business high speed Internet services. They also cover a number of different ISPs including Comcast, AT&T, Verizon and CenturyLink. For a given video at a given location and time, we download the manifest file 1 times, with 1 second interval between two consecutive downloads. We call such 1 consecutive title => Soldier Girls, tp:ad_model => longform, tp:frame_rate => 25, dur => ms, tp:enableadblockerslate => true, tp:aspect_ratio => 4x3, tp:bugimageurl =>, tp:researchprogram =>, tp:comscoreid =>, tp:hasbug => false, tp:defaultbitrate => 65_h264, tp:primarysitechannelnielsenchannelid => 71, tp:cp_promotional_link =>, tp:cpidentifier => ContentFilm, tp:primary_category => Documentary and Biography, tp:adtype =>, tp:fingerprint => csel3_prod_iad115, tp:cp_promotional_text =>, tp:segments => T::11:54;22,T::26:29;9, T::38:49;27,T::57:37;18,T:1:3:15;2, T:1:17:12;3, tp:adtypeplus => SponsoredFilm, tp:tunein_information =>, tp:distributionpartnercomscoreid => 37, tp:secondarysitechannelnielsenid => 38, tp:cdnprefs => level3,akamai,limelight, Fig. 2. A section of Hulu manifest file downloads an experiment. Each downloaded manifest file assigns one CDN as preferred CDN. We count the number of times each CDN is preferred for each experiment. We refer to the percentage of times that a CDN is preferred in an experiment as preference percentage. This preference percentage essentially reflects the likelihood for a CDN to be selected by the clients. Overall CDN preference Figure 4 shows the distribution of preference percentage for the three CDNs based on results for all videos, locations, and time. The three curves representing the three CDNs are very close to Gaussian distributions. The mean preference percentage for, and are 25, 28 and 47, respectively. is the preferred CDN 47% of times, much more than the other two CDNs. Probability CDN preference percentage Fig. 4. Overall CDN preference distribution CDN preference over different locations Figure 5 shows

4 Time of request (in seconds) Fig. 3. CDN preference change in a short interval CDN preference observed from clients at different geographic locations. These 13 locations span different cities across eight US states. For this analysis, we combine data for all the videos collected at the same location and calculate the average preference percentage for each location. We observe that different CDNs have different popularity but the popularity do not change over different locations. Average preference % Fig Client location IDs CDN preference observed from different geographic regions CDN preference for different videos Figure 6 shows CDN preference for different videos. Here we aggregate the experiments for each video across location and time and calculate its average preference percentage. The small variation in preference percentage across different videos indicate CDN preference is independent of which video is being served. CDN preference over time Figure 7 shows CDN preference change over different days at the same location. This result is based on 24 days of experiments at a single location. Each data point represents the average preference percentage over all videos on each day for a given CDN. The results for for other locations (not shown here) are very similar. We observe that the CDN preferences do not change over time either. In summary, we conclude that Hulu selects the preferred CDN randomly following a fixed latent distribution for each of the playback requests. On average, one CDN () is preferred more than others, but such selection preference does Average preference % Average preference % Video IDs Fig. 6. CDN preference for different videos days between August 17 to September 9, 211 Fig. 7. CDN preference over time not seem to depend on instantaneous network conditions. It is also evident that CDN selection is not affected by client location at which the video is played. And the selection does not change over the 24 days that we measured. We conjecture that such CDN preference is most likely based on pricing and business arrangements and is not dependent upon

5 instanteneous bandwidth or past performance history of the CDNs. IV. CDN SERVER DISTRIBUTION Hulu server selection consists of two steps: (a) selecting a CDN provider, and (b) selecting a specific CDN server to serve the content. In this section, we study how the three CDNs utilize their server resources to serve different types of clients (desktop clients vs. mobile clients) and Hulu advertisement. We find that while three CDNs are used to serve desktop clients, only two CDN are used to serve mobile clients and Hulu advertisement. We further examine the number of CDN servers employed by individual CDNs and these servers locations. We find that different CDNs employ varying number of servers at different locations. In addition, the allocated servers for Hulu desktop clients, mobile clients, and for the advertisement are different and do not overlap with each other, suggesting CDNs may employ dedicated servers for different services. Finally, we briefly compare the CDN service for Hulu with that for Netflix and Amazon. We find that all three CDNs use locality-aware DNS resolutions. We analyze the packet traces collected by playing a large number of videos from multiple geographical locations and extract the hostnames of the servers. The same hostname may be mapped to different IP addresses when queried from different locations. In order to better understand how and where the video content is served by each CDN, we next try to identify all servers in each CDN and also cluster them based on locations. Besides the IP addresses extracted from packet traces, we resolve the extracted hostnames from 1 PlanetLab[8] nodes in the US. We also obtain a list of publicly available open recursive resolvers[9] and employ 7 of them (inside the US) to resolve these hostnames. A. CDN Servers for Hulu Desktop Clients Through experiments we find that and only use one hostname each, while uses 1, different hostnames for delivering Hulu video content to desktop clients (Table I). We also observe that uses last three digits of the internal ID of a video to decide what hostname is responsible to serve that video. For instance, a video with ID is mapped to host hulu-22.fcod.llnwd.net. Such mapping provides some limited means of balancing load among different servers because each of the 1 hostnames expect to responsible for an equal number of videos. However, since different videos have widely differing popularity, this approach can not be expected to equally divide locad among different servers. After we obtain the set of unique IP addresses for each CDN, we then measure ping latencies to all IP addresses. IP addresses serving Hulu content do not respond to ping probes. We therefore measure latencies to other nearby IP addresses. Assuming IP addresses from a /26 subnet will likely to placed nearby, we try to ping other IP addresses in the same /26 network as the original IP addresses when they do not respond to ICMP ping probes. Based on ping latencies TABLE I CDN SERVERS FOR HULU CDN Hostname(s) # IPs Clusters cp39466.edgefcs.net (Desktop) httpls.hulu.com (Mobile) ads.hulu.com (Advertisement) hulu-{-999}.fcod.llnwd.net (Desktop) ll.a.hulu.com (Advertisement) 18 9 hulufs.fplive.net (Desktop) 48 1 httpls-1.hulu.com (Mobile) to all the IP addresses from hundreds of PlanetLab nodes, we cluster the IP addresses in different groups. These clusters roughly represents the server locations from which the CDNs are serving Hulu content. The number of such clusters are also shown in Table I. In summary, uses the largest number of IPs (1178) and clusters (4), while the number of clusters for and is much smaller, at 9 and 1 clusters respectively. Interestingly, the number of IPs used by is close to that of, and is much larger than the IP addresses used by. These observations we made for servers serving Hulu content are also consistent with the previous studies [1], [11] on these three large CDNs architecture. B. CDN Servers for Mobile Clients (for HuluPLus) We find that only two CDNs are used to serve HuluPlus subscribers on ios devices: and. These two CDNs use only one hostname each: httpls.hulu.com for and httpls-1.hulu.com for. These canonical names are in turn mapped to other hostnames and finally to IP addresses. We find that and have 45 and 125 unique IP addresses, respectively. However, these IP addresses are different from IP addresses serving Hulu content to desktop browsers. On the other hand, the number of clusters is roughly the same as that for Hulu desktop CDN servers for both CDNs. C. CDN Servers for Hulu Advertisements We observe that CDNs and servers used for advertisements and regular video content are generally independently assigned. Hulu advertisements are only served from and CDNs. Two hostnames are used: ads.hulu.com for and ll.a.hulu.com for. ll.a.hulu.com is mapped to 2 IP addresses at each of the 9 clusters of CDN. Likewise, ads.hulu.com is mapped to approximately 45 IP addresses. There is again no overlap between servers for advertisements and regular content. The number of clusters is roughly the same as that for regular content. Overall, we observe that all CDNs employ locality-aware DNS resolution, while their server distribution and load balancing strategies differ in some cases. has many more distinct server IP addresses and locations compared to and. Note that this does not necessarily translate into more servers or better performance since there

6 can be multiple physical servers behind the same IP address, and also performance for video delivery mostly depends on throughput. A number of these results have been previously reported for generic CDN studies[1], [11], [12]. In contrast to these studies, we primarily focused on CDNs, hostnames and server IPs that serve Hulu content and advertisements. Additionally, and only rely on DNS to do load balancing; while also tries to distribute load by hashing video IDs into hostnames. We also observe that numbers of CDNs and servers for Hulu mobile clients and advertisements are smaller compared to those for regular videos served to desktop clients. This is most likely because the latter incurs significantly more traffic. D. Comparing With Other Services We next explore how Hulu differs from some other video service providers. Using active measurement for Amazon VoD, and data from a previous study [13] for Netflix, we compare their CDN and server selection strategy with those used for Hulu. Netflix uses the same three CDNs that Hulu uses. However, Netflix assigns CDNs according to user accounts statically. A Netflix subscriber uses the same CDN for extended periods of time. We also find that although the CDNs use approximately the same number of clusters to serve Netflix content as they do for Hulu, the IP addresses serving Netflix do not overlap with those serving Hulu. Amazon VoD uses only for content delivery. Hence no CDN selection is needed. Our analysis reveals that there is significant overlap between servers for Amazon VoD and those for Hulu. Our conjecture is that CDNs usually deploy different server infrastructure for different video delivery technologies. Both Amazon VoD and Hulu use RTMP, they share large number of servers. Netflix do not overlap with them since it uses HTTP adaptive streaming. V. RELATED WORK There are a number of research projects that try to understand server selection strategies for popular content providers. Adhikari et al. inferred YouTube video platform in [14] using active measurements whereas in [15], Finamore et al. used passively collected data to dissect server selection strategies for the same service. Similarly, past studies such as [1], [11] have explored CDNs in general. However, in this paper, we focused primarily from the point of view of a content-provider (Hulu) to see what resources the CDNs use to serve content for the specific content-provider. On the other hand, Krishnappa et al. studied how beneficial caching can be for Hulu in [16]. However, they do not focus on how Hulu uses multiple CDNs or how those CDNs select server to serve Hulu content. To the best of our knowledge, we are the first to study overall architecture of Hulu video delivery system and its server selection strategy. VI. CONCLUSIONS AND FUTURE WORK We have conducted a measurement study for the Hulu multi-cdn video delivery system. Our main observations are as follows: (1) Hulu CDN selection is done on Hulu s control servers and are communicated to the clients through the manifest files. The preferred CDNs are randomly assigned based on pre-determined probabilities, which are independent of location, video and time. (2) The preferred CDN is also independent of instantaneous available bandwidth observed at the client, although the client can adapt to network congestions by first lowering rate and then switching to different CDNs. (3) All CDNs employ locality-based DNS resolution, although their server distribution and load balancing strategies differ in some cases. (4) Different CDN customers may share same set of servers if they use the same delivery technology. For instance, Amazon VoD and Hulu share many servers since they both use RTMP; while they do not overlap with Netflix since Netflix uses adaptive streaming over HTTP. An interesting direction for future work may be to build a model for multi-cdn video delivery based on the measurement results contained in this paper and previous work on Netflix and YouTube, so that we can conduct a systematic study on how to improve large-scale video delivery performance. REFERENCES [1] C. Labovitz, S. Iekel-Johnson, D. McPherson, J. Oberheide, and F. Jahanian, Internet inter-domain traffic, SIGCOMM Comput. Commun. Rev., vol. 41, pp , August 21. [2] Sandvine, Global Internet Phenomena Report, Spring 211. [3] comscore, comscore Releases February 211 U.S. Online Video Rankings. [4] R. Pantos et al., 6.txt. [5] Dummynet, luigi/dummynet/. [6] L. Daigle, WHOIS Protocol Specification. [7] get-flash-videos, A command line program to download flash videos, [8] Planetlab an open platform for developing, deploying, and accessing planetary-scale services, [9] Public DNS Servers, public-dns-servers. [1] C. Huang, A. Wang, J. Li, and K. W. Ross, Measuring and evaluating large-scale CDNs (Paper withdrawn at Mirosoft s request), in Proceedings of the 8th ACM SIGCOMM conference on Internet measurement, ser. IMC 8. New York, NY, USA: ACM, 28, pp [11] B. Krishnamurthy, C. Wills, and Y. Zhang, On the use and performance of content distribution networks, in Proceedings of the 1st ACM SIGCOMM Workshop on Internet Measurement, ser. IMW 1. New York, NY, USA: ACM, 21, pp [12] B. Ager, W. Mühlbauer, G. Smaragdakis, and S. Uhlig, Comparing dns resolvers in the wild, in Proceedings of the 1th annual conference on Internet measurement, ser. IMC 1. New York, NY, USA: ACM, 21, pp [13] V. K. Adhikari, Y. Guo, F. Hao, M. Varvello, V. Hilt, M. Steiner, and Z.- L. Zhang, Unreeling Netflix: Understanding and Improving Multi-CDN Movie Delivery, in INFOCOM 12. IEEE, 212. [14] V. K. Adhikari, S. Jain, Y. Chen, and Z.-L. Zhang, Vivisecting YouTube: An Active Measurement Study, in INFOCOM 12 Mini-conference. IEEE, 212. [15] R. Torres, A. Finamore, J. R. Kim, M. Mellia, M. M. Munafo, and S. Rao, Dissecting video server selection strategies in the youtube cdn, in Proceedings of the st International Conference on Distributed Computing Systems, ser. ICDCS 11. Washington, DC, USA: IEEE Computer Society, 211, pp [16] D. K. Krishnappa, S. Khemmarat, L. Gao, and M. Zink, On the feasibility of prefetching and caching for online tv services: a measurement study on hulu, in Proceedings of the 12th international conference on Passive and active measurement, ser. PAM 11. Berlin, Heidelberg: Springer-Verlag, 211, pp