An Analysis of Facebook Photo Caching

Transcription

1 An Analysis of Facebook Photo Caching Qi Huang, Ken Birman, Robbert van Renesse, Wyatt Lloyd, Sanjeev Kumar, Harry C. Li Permission to make digital or hard copies of part or all of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for third-party components of this work must be honored. For all other uses, contact the Owner/Author. Copyright is held by the Owner/Author(s). SOSP 13, Nov. 3 6, 2013, Farmington, Pennsylvania, USA. ACM /13/11.

2 The problem 250 billion photos on Facebook in million more uploaded daily High availability and low latency required for delivery infrastructure.

3 Facebook's Photo-Serving Stack Client Browser cache Facebook storage and caches Akamai CDN

4 Facebook's Photo-Serving Stack Client Browser cache Facebook storage and caches Akamai CDN Focus exclusively on Facebook stack

5 Edge Caches (FIFO) Client Browser cache PoP Edge cache tens of independent edge caches across the US

6 Edge Caches (FIFO) Client Browser cache PoP Edge cache Purpose: reduce latency reduce bandwidth utilization between clients and datacenters

7 Global Origin Cache (FIFO) Data center Client Browser cache PoP Edge cache hash(id) Origin cache Data center a single global cache requests routed to specific data centers based on hash of photo id

8 Global Origin Cache (FIFO) Data center Client Browser cache PoP Edge cache hash(id) Origin cache Data center Purpose: traffic sheltering for I/O bound backend

9 Haystack backend Data center Client Browser cache PoP Edge cache hash(id) Origin cache Backend (Haystack) Backend (Haystack) Data center located in the same data centers as origin servers

10 Resizers Resizer Data center Client Browser cache PoP Edge cache hash(id) Origin cache R R Backend (Haystack) Backend (Haystack) Data center photos are served in different sizes to different users resized photos are treated as distinct objects by the cache layer

11 Methodology

12 Stack instrumentation Client PoP Data center Browser cache Edge cache Origin cache R Backend (Haystack) Instrumentation scope desktop clients only data for browser cache inferred by counting requests seen at browsers and edge caches

13 Sampling method request-based sampling: collect X% of requests object-based sampling: collect X% of objects

14 Sampling method request-based sampling: collect X% of requests object-based sampling: collect X% of objects by deterministic test on photo id

15 Object-based Sampling better coverage of unpopular photos simplified cross-stack analysis

16 Analysis

17 The collected data One-month long trace 77M requests from 13.2M user browsers 1.3M unique photos (2.6M photo objects)

18 Traffic sheltering Client PoP Data center Browser cache Edge cache Origin cache R Backend (Haystack) 77.2M 26.6M 11.2M 7.6M 65.5% 58% 31.8%

19 Popularity Distribution approximately Zipfian

20 Traffic share by photo popularity

21 Geographic Traffic Distribution

22 Client To Edge Cache Traffic Atlanta

23 Client To Edge Cache Traffic 5% New York 5% California 35% Washington D.C. 5% Dallas 20% Atlanta 10% Miami

24 Client To Edge Cache Traffic routing algorithm considers latency, edge capacity and ISP peering cost substantial remote traffic is normal

25 Edge Cache to Origin Cache Traffic Traffic flow based on content, not locality.

26 Cross-Region Traffic at Backend Caused by data migrations and hardware failures Only 0.2% of Origin Cache to Backend traffic

27 Potential Improvements

28 Cache Simulation replay the trace use the first 25% to warm the cache evaluate using the remaining 75% of the trace

29 Object-hit Ratios for Edge Cache S4LRU improves 9% at x over FIFO Still a lot of room for improvement

30 S4LRU Cache space L3 L2 L1 L0 Four queues are maintained at levels 0 to 3

31 S4LRU Cache space L3 L2 L1 L0 Missed objects inserted into the head of level 0 queue

32 S4LRU Cache space L3 L2 L1 L0 On a cache hit, the item is moved to the head of the next higher queue

33 S4LRU Cache space L3 L2 L1 L0 items evicted from the tail of a queue to the head of the next lower queue

34 Byte-hit Ratios for Edge Cache LFU fails to reduce bandwidth utilization over the current solution

35 Origin cache S4LRU improves Origin slightly more than Edge (14%)

36 Social-Network Analysis

37 Content Age Effect

38 Traffic share for non-profile photos by age

39 Traffic share for social activity groups. Number of followers

40