How to Train Your (Web) Dragon

Transcription

1 How to Train Your (Web) Dragon Optimize your China website in 45 minutes Steven Ryder [CDN Engineering. North America]

2

3 Observe, Identify, Focus, Design, Optimize. FLY

4 Dragon 1.0 Dragon 2.0

5 Web Dragon x3.0 Different Internet Service Providers China Network = Mainland China Hong Kong & Taiwan are treated separately Different Languages: Taiwan = Traditional Chinese Hong Kong = Cantonese Mainland China = Simplified Chinese Different Fixed & Mobile Networks.HK.TW.CN One Country. Three TLDs.

6 English Folklore Chinese Folklore 2012, Jan 23 -> Feb 9, 2013 Boundary dragons

7 China Internet Folklore In the beginning China Telecom (CT) created the Internet (for Mainland Chinese) ISPs Networks Today 200+ Branded ISPs / Resellers 16 Infrastructure providers 2x Main carriers: = 95% of both Fixed & Mobile Services And then China Telecom (CT) fixed line divided into: -China Telecom (CT) (Southern provinces) -China Netcom (CNC) (Northern provinces) China Mobile formed separately by China Telecom China Unicom (CU) formed by MIIT -to compete with China Mobile China Netcom and China Unicom merged -becoming China Unicom (CU) Today Three significant carriers all competing with each other over territory nationally. - China Telecom (CT) - China Unicom (CU) - China Mobile China Telecom (CT): Coverage = 63.66% of China Government owned Handles the middle and south of Mainland China Continues to expand into CU territory China Unicom (CU): Coverage = 31.66% of China Government owned Handles northern Mainland China Continuing to expand into more of CT territory Smaller / Regional ISPs China Mobile China s main mobile carrier with the 3 rd largest IP network. Introduced 2G to China, and leading China s mobile industry growth Tietong / China Railway Network Built with the railways expansion all over China Strong backbone with focus on national fibre infrastructure CERNet: / China Education & Research Network Focused around education industry (University & Research Organizations) within China Not directly connected to the Public Internet. CERNet->Public Web is limited. Connects all major universities of China together. Different from other carriers of China, & entirely separate from CT and CU

8 Local Broadband Intl. Landing Points Overall Average Still common < 1.0 Mbps ~512 Kbps International Data Gateways into China Beijing Shanghai Qingdao Guangzhou Major cities and province (Averages) - Shanghai: 4.82Mbps - Beijing: 3.46Mbps - Anhui: 3.11Mbps - Hubei: 2.98Mbps - Hainai: 2.84Mbps - Zhejiang: 2.69Mbps - Guangdong: 2.69Mbps

9 Mapping Geography

10 Volcanic Geography 1,500 active volcanos globally 90% located in the Ring of Fire

11 Cabling Geography Taiwan 2006 undersea quake: Dec 26 Damage: only 2 cables! People impacted: millions Affected: International connectivity Impacted Countries: CHINA, Japan and SEA Affected ISPs: China Telecom, China Unicom PCCW, NTT, KT, etc..

12 National History Museum Backbone Case Study London (ONLY) Hosted. No CDN. No Geo-Hosting. CNAME tarantula.nhm.ac.uk = Service requires 48 Successful Requests - 1x sub-domain ( only - 1x Index.html home page - 4x HTML includes - 3x Javascripts includes - 8x CSS includes - 32x Images

13 Registry (DNS, IPv4/v6, Whois) Authorities CNNIC (China Network Information Center) Provide to Carriers & Companies in large address blocks Carrier assigns IP blocks at province level internally Carrier manages routing policies APNIC (Asia Pacific Network Information Center) Same as CNNIC for the Asia Region

14 ICP (Internet Content Provider) License Purpose: Permit China based websites to operate within China Issued by MIIT (Ministry of Industry & Information Technology) Required for all locally hosted sites / those using local acceleration (eg: CDN) Joint Ventures are often formed to facilitate an ICP with a local Chinese partner ICP comes in 2x types: Informational (relatively easy to obtain) Operational (Requires Chinese partner / Local Registration / etc..) Obtaining an ICP for your website inside China Identify domains (registration is done by zone, not sub-domain) Talk to a Local China Expert (eg: ChinaCache) & confirm your ICP requirements Submit your application with the assistance of an in country licensed provider Confirm your ICP # from the official Gov.cn site: Publish ICP # in footer of all service pages

15 GFW (Great Firewall of China) Covers all external (globally facing) major Internet gateways/routers Recently also introduced at the province level (internal traffic) Local Data Centre may not allow (or take down) content without ICP # (limit potential liability) Notification of content issues provided to China based hosting provider (only). Site owner not directly informed Blocking rules have fluctuating behavior Filtering known to be more strict during significant dates Typical user experience: TCP Reset or Network Timeout Content to avoid to prevent being blocked Regular suspects Sensitive characters Linking / including content to/from blocked sites: YouTube videos Facebook Like Buttons

16 Web Service Site Types Informational Interactive Services E-Commerce Media Gaming Maps Mobile Services Generally more dynamic Accurate GSLB location important for customization Every Service. Every Domain. All Existing Assumptions. Measure again within China.

17 Measurement, Reliability, and Network Status Tools Network performance measurement platform tools: China Backbone, Last Mile, & Instant Testing Backbone vs Last Mile Data Usage: Server reliability vs ISP View. Backbone unrealistically fast. Compuware / Gomez: ( Last Mile Batch Test provider (same Last time URL comparison capability) NetworkBench ( More Local Agents / More choices / Rich Reporting & Analysis. CCINDEX (developed by ChinaCache): 3 rd most popular measurement platform in China Keynote: China Backbone Browser Analysis Tools HTTPWatch, Fiddler, YSlow, etc.. / Object waterfall charting DNS Lookup Tools: DIG, NSLOOKUP IP Query Tools: PING / TRACEROUTE (Mac), TRACERT/PATHPING (Win), TRACEPATH/MTR (Unix) Expanded Manual Testing Browser (unexpected versions), Eg: 26% = Qihoo 360 Client, IE6 = 22%). New clients (eg: IPTV) Mobile (phone, tablet, and smartphone devices)

18 CCIndex.cn App: Mobile - Network Awareness Platform Realtime China Local Testing: any URL. Cost: FREE! Realtime bandwidth testing from anywhere (check your device connectivity) Realtime ISP Network Statistics Free Mobile App: iphone & Android - install directly from via the links at the bottom of the page

19 CCIndex.cn Site: Desktop Analysis and Testing Tools URL: Realtime China Insights: any URL. Useful for bandwidth sampling Cost: FREE!

20 17ce.com - Get Ping TraceRoute DNS CDN Realtime Single Object China Local Testing: any URL. Cost: FREE! Chelsea Football Club (example:

21 Shared Network Performance Reporting Statistical data on the network of China s connection speed Provides connection insights broadband speed by province interconnect between carriers Example Data Providers (alphabetically ordered) Akamai Global Reports ( ) Cedexis ( ChinaCache CCINDEX ( Industry Performance Indexes Show market leaders with their page load time, and related metrics

22 Minimise Network Use Accept-Encoding Gzip, deflate (Compression) Connection Keep-Alive (TCP port reuse) Expires Wed, 10 Oct :15 (HTTP/1.0 Caching) Cache-Control Max-Age= (HTTP/1.1 Caching) W3C Logs Resolve Errors (Focus: 400/403/404/500) Cookie Free Domain Reduce SEND data (Simplify caching) (i1) Img Sub-domain Increase TCP Ports (Speed Page Load) ETag Remove for static files. (Use Last-Modified)

23 GSLB Inaccurate GeoData (Location) Impact IP Block Allocation Carriers allocate IP blocks first by provinces, and then by cities GSLB systems works based on static geographic databases (accuracy is thus very important) Local DNS Most provinces have their own LDNS servers Some provinces set geo-blocking to protect their DNS servers Some provinces set DNS forwarders only Common Problems / Mis-configured Client configuration Foreign Company inside China using VPN and foreign DNS to resolve local service domains China User using Google DNS ( ) or OpenDNS Where one or more GSLB services are used and do not have EDNS enabled Client has DNS for incorrect carrier (eg: CT user has CU LDNS IP configured) Fixed LDNS in Client config used when travelling DNS Forwarder causes wrong-location to be used

24 National Coverage Matters Hundreds of millions of users. Everywhere has remote connection congestion.

25 How to Improve GeoData Accuracy and Efficiency? Get accurate IP block allocation database in time Providers such as IP2Location.com do not have province level accuracy Leverage a local GSLB provider who specializes in accurate records Focus on client IP not LDNS IP: Use EDNS enabled provider Determine LDNS users are using to lookup your service domains Compare LDNS IP to Province and ISP Troubleshoot response problems with tools like

26 Site Domain DNS & China Timezones Have China specific service domains AND include domains Don't share IPs between your China and Non-China domains Use a www (or alternative) sub-domain Use a CNAME supportable DNS home/landing page (eg: "www ) Only CNAME records work with CDNs / GSLBs Have separate dynamic and static content domains Move Login pages off the "www" domain to eg: login/accounts.<domain> Peak Analysis! Expect TWO large peak traffic periods for popular content Local China Time: 09:30 (01:30 GMT) & 21:00 (13:00 GMT)

27 In Country vs Global Delivery Europe / US vs China Hosted European origins: Low availability & High latency into China In country hosting benefits Avoid external shared gateway congestion Avoid natural disasters related failed routes Origin Server Connectivity (Direct Peering Questions to Ask) What networks is your origin hosted in? What direct (BGP) routes exist from that DC to other ISPs? Regularly review traceroutes to your server from major China locations (eg: Beijing & Shanghai) CDN Edge Network Provider Benefits Eliminate need to build-up and manage BGP network connectivity & Geo-location Data Centers Leverage Dynamic acceleration for non-cachable content, and combine with static caching benefits

28 Video & Streaming Media Live streaming: - China delivery will require an overseas ingest point (unless locally replicated first) - Redundant international connections are necessary - Bitrate(MBR): 450Kbps for everyone 800Kbps for all broadband 1.5Mbps to have available for special users VOD: - Hot content benefits from pre-loading into the Edge - In country storage is recommended for large content sets - Content sync is rarely free, so consider in country ingestion strategy and costs - Bitrate(MBR): **same as for streaming***

29 International CDN +China Multi-Vendor Architecture GSLB systems support any DNS endpoint for traffic Different CNAMEs based on country, city/province, and ISP Load balancing by request %, time of day based rules, all to a single destination, or 100% direct to origin

30 Final Recommendations #1 Improve DNS LDNS Caching by increasing DNS TTLs Increase to 600 or higher (using your own readiness time to govern lower/upper limits) Extend HTTP Keep-Alive timeout settings to seconds to cover page load completion Apache v.2.5 has default of only 5 seconds. #2 Initially assume your China connectivity problems are NOT caused by the Great Firewall Countless content delivery, bandwidth, route, packet loss, and other issues are usually the cause Always start with PING, TRACE, and NSLOOKUP from within China (each and every time) #3 Don t skip obtaining an ICP license. Start to think about this now, as it takes time to obtain.

31 谢 谢 Thank you