CGN Architectures & Impacts Karthik Sundaresan 09/14/11 TXv6TF Summit
Outline Intrductin Transitin Technlgy Overview Impacts & CGN Testing Results CGN Architecture cnsideratins CGN Lgging Summary 2 Cable Televisin Labratries, Inc. 2011. All Rights Reserved. Prprietary/Cnfidential. 9/14/2011
Why Carrier Grade NAT (CGN)? IANA has run ut f IPv4 Addresses ARIN and all ther RIRs will sn be there Slutin t IPv4 address exhaustin is t deply IPv6 But IPv4 servers, devices will cntinue t exist n the Internet after IPv4 address exhaustin IPv6 nly hsts cannt reach IPv4 nly hsts ISPs will be frced t share IPv4 addresses amng subscribers : address multiplexing technlgies (CGN) 3 Cable Televisin Labratries, Inc. 2011. All Rights Reserved. Prprietary/Cnfidential. 9/14/2011
Carrier Grade NAT Each subscriber is assigned a private address A NAT situated in ISP's netwrk translates between private and public addresses Share a smaller number f Glbal Unicast Addresses (GUA) amngst a larger number f end-sites 4 Cable Televisin Labratries, Inc. 2011. All Rights Reserved. Prprietary/Cnfidential. 9/14/2011
Intrductin TRANSITION TECHNOLOGY OVERVIEW 5 Cable Televisin Labratries, Inc. 2011. All Rights Reserved. Prprietary/Cnfidential. 9/14/2011
What is NAT444? NAT Table 76.121.26.3:2001<->10.1.0.2:1025 CGN builds NAT mapping using public and private IPv4 MSO DHCPv4 Server NAT Table 10.1.0.2:1025<->192.168.0.2:1025 Changes DA and frwards IPv4 packet t hst Changes SA f packet, sends upstream IPv4 Internet CGN changes DA CGN f packet, sends packet t HGW IPv4 Packet SA 76.121.26.3:2001 CMTS CM/eRuter IPv4 Packet SA 10.1.0.2 IPv4 Hst 192.168.0.2 IPv4 Packet SA 192.168.0.2 IPv4 Packet DA 76.121.26.3:2001 IPv4 Packet DA 10.1.0.2 IPv4 Packet DA 192.168.0.2 6 Cable Televisin Labratries, Inc. 2011. All Rights Reserved. Prprietary/Cnfidential. 9/14/2011
What is Dual-Stack Lite? NAT Table 76.121.26.3:2001<-> 2001::1 192.168.0.2:1025 IPv4 Internet CGN builds NAT mapping using IPv6, IPv4, and prt, then perfrms NAT Mdified DHCP Server DHCP lease cntains IPv6 addr, CGN Remves IPv6 header, frwards IPv4 packet t hst Encapsulates packet with IPv6 header, sends Hst btains it t CGN address frm eruter CGN translates DA, CGN adds IPv6 tunnel CMTS header, sends IPv4 Packet packet t HGW SA 76.121.26.3:2001 IPv6 Header CM/eRuter IPv4 Packet SA 192.168.0.2 IPv4 Hst 192.168.0.2 IPv4 Packet SA 192.168.0.2 IPv4 Packet IPv6 Header DA 76.121.26.3:2001 IPv4 Packet DA 192.168.0.2 IPv4 Packet DA 192.168.0.2 7 Cable Televisin Labratries, Inc. 2011. All Rights Reserved. Prprietary/Cnfidential. 9/14/2011
IPv6 Internet What is 6RD? 6RD BR remves IPv4 header and rutes IPv6 packet nt IPv6 Internet MSO DHCP Server Remves IPv4 DHCP lease header, cntains frwards IPv6 IPv4 BR packet t hst addr Autmatic Prefix delegatin frm CE t CPE 6rd delegated prefix = the 6rd prefix + all r part f CE public IPv4 address Encapsulates IPv6 packet with IPv4 header, Hst btains sends it t 6RD BRIPv6 Prefix frm 6RD CE 6RD BR (Brder Relay) IPv6 Packet SA 2001:db8::1001 6RD BR adds IPv4 tunnel header, sends packet t 6RDCE CMTS IPv4 Header CM + 6RD CE (Custmer Edge) IPv6 Packet SA 2001:db8::1001 IPv6 Hst 2001:db8::1001 IPv6 Packet SA 2001:db8::1001 IPv6 Packet IPv4 Header DA 2001:db8::1001 001 IPv6 Packet DA 2001:db8::1001 IPv6 Packet DA 2001:db8::1001 8 Cable Televisin Labratries, Inc. 2011. All Rights Reserved. Prprietary/Cnfidential. 9/14/2011
Timeline f Access Technlgy Transitin Cnnectivity Type Time IPv4 Native Native, NAT444 * Native, NAT444 * Native, DS-Lite * IPv6 Nne 6RD * Native Native * sme may skip this step
What is brken CGN IMPACTS & TESTING RESULTS 10 Cable Televisin Labratries, Inc. 2011. All Rights Reserved. Prprietary/Cnfidential. 9/14/2011
CGN Challenges Pr quality f experience fr advanced services Netflix, peer-t-peer, vide streaming, gaming, etc. Change t Drpped Negative impact t targeted advertising/ge-lcatin Lgging fr lawful intercept : Petabytes f data Overlapping Addressing / NAT Znes Impacts netwrk architecture/traffic engineering 11 Cable Televisin Labratries, Inc. 2011. All Rights Reserved. Prprietary/Cnfidential. 9/14/2011
Overview f CGN test scenaris CableLabs is cnducting CGN testing n multiple vendr devices Bth NAT444 and DS-Lite c-existence technlgies Single and dual ISP netwrks with ne r mre users n multiple hme netwrks Test applicatins include 12 Vide services e.g. Netflix, YuTube, iclips, Jst, Silverlight Audi streaming e.g. Pandra, Internet Archive Peer-t-peer utrrent Online gaming & Peer t peer Gaming - Xbx FTP large file transfers SIP calls e.g. X-Lite, Skype, PJSIP Vide chat e.g. Skype, OV Scial netwrking e.g. Facebk, Webkinz Web cnferencing e.g. GTMeeting, WebEx Cable Televisin Labratries, Inc. 2011. All Rights Reserved. Prprietary/Cnfidential. 9/14/2011
Client devices and gateways used fr testing Laptps running Vista, XP, Win 7 and MAC OS X-bx gaming cnsles ipad iphne and Andrid smartphnes CE devices LG Blu Ray player with Apps Hme ruters Netgear, Linksys, Cisc, D-Link, Apple Airprt Express 13 Cable Televisin Labratries, Inc. 2011. All Rights Reserved. Prprietary/Cnfidential. 9/14/2011
CGN Testing Issue #1 SIP Call Test Scenari SIP client t client call is initiated Acrss tw hme netwrks n single ISP Acrss dual ISPs, each supprting a single hme netwrk P2P client: PJSIP User Agent, SIP Prxy: X-Lite client Observed Behavir Peer-t-peer calls cannt be initiated r received (PJSIP) In cases where a prxy is used, there is n issue Inside device needs t initiate traffic first Only ne SIP user was active behind a single IPv4 address. 14 Cable Televisin Labratries, Inc. 2011. All Rights Reserved. Prprietary/Cnfidential. 9/14/2011
CGN Testing Issue #2 X-Bx P2P Gaming Test Scenari X-Bx gaming sessins are initiated fr tw users Acrss tw hme netwrks n single ISP (hairpinning) Observed Behavir Cannt start game between cnnected users X-Bx Sessin errrs ut: Game sessin n lnger available 15 Cable Televisin Labratries, Inc. 2011. All Rights Reserved. Prprietary/Cnfidential. 9/14/2011
CGN Testing Issue #3 Vide Streaming Test Scenari Netflix vide streaming sessins are initiated using a variety f client devices Observed Behavir Slwer buffering times were nted fr mst client devices On sme hme ruters first sessin degrades after the secnd sessin was started (excessive rebuffering) within same hme Vide streaming was smth behind ther hme ruters Micrsft Smth Streaming had similar issues 16 Cable Televisin Labratries, Inc. 2011. All Rights Reserved. Prprietary/Cnfidential. 9/14/2011
CGN Testing Issue #4 utrrent Test Scenari utrrent seeding initiated n client A (inside CGN) utrrent leeching initiated n client B (n internet) Acrss tw hme netwrks n single ISP Observed Behavir utrrent peer-t-peer seeding des nt wrk Succeeds using prxy server NAT444 : smetimes utrrent redirects t a different seed 17 Cable Televisin Labratries, Inc. 2011. All Rights Reserved. Prprietary/Cnfidential. 9/14/2011
CGN Testing Issue #5 Slwer Dwnlad Rates Test Scenari Dwnlads are initiated using multiple types f clients FTP f large size file Internet Archive audi/vide files are dwnladed Observed Behavir Dwnlad rate is nticeably slwer Windws clients perfrmed at a cnsistently higher rate 250Kbps fr Windws vs. 120Kbps fr Mac 18 Cable Televisin Labratries, Inc. 2011. All Rights Reserved. Prprietary/Cnfidential. 9/14/2011
Testing Takeaways Imprvement since testing last year CGN device imprvements Cntent prvider updates (X-Bx live, Netflix Streaming) Applicatin updates (Prxy Servers fr X-Bx, utrrent) Trubleshting applicatin issues is difficult Test befre yu deply Yur mileage may vary Depending n yur CGN vendr, CGN architecture etc. 19 Cable Televisin Labratries, Inc. 2011. All Rights Reserved. Prprietary/Cnfidential. 9/14/2011
Deplyment Cnsideratins CGN ARCHITECTURE 20 Cable Televisin Labratries, Inc. 2011. All Rights Reserved. Prprietary/Cnfidential. 9/14/2011
CGN architecture Centralized architecture Places all CGN functinality in ne central lcatin within the carrier s netwrk. Distributed architecture Places CGN functinality as clse t the subscriber as pssible. Hybrid apprach Falls smewhere between these tw extremes 21 Cable Televisin Labratries, Inc. 2010. All Rights reserved. Prprietary/Cnfidential 9/14/2011
Areas f cnsideratin fr CGN architecture Relative deplyment cst (day 1 cst) Ease f implementatin Impact n ruting: Changes required in current ruting infrastructure Traffic Engineering: Allws MSO t distribute/rute traffic Lad Balancing: Sharing lad between different devices Scalability: Respnse t increased traffic/subscriber grwth Subscriber IP addressing Size f Private Subnet needed Number f Public Addresses used Ge-lcatin: Granularity f gelcatin infrmatin btained On-net CDN deplyments: Ease f placement f CDN servers 22 Cable Televisin Labratries, Inc. 2011. All Rights Reserved. Prprietary/Cnfidential. 9/14/2011
Architecture Centralized Head End CMTS CMTS RTR Central Lcatin IPv6 Internet CGN CGN Head End CMTS CMTS Head End RTR Cre Netwrk RTR IPv4 Internet CMTS CMTS RTR 23 Cable Televisin Labratries, Inc. 2011. All Rights Reserved. Prprietary/Cnfidential. 9/14/2011
Architecture - Distributed Head End CMTS CMTS CGN RTR IPv6 Internet Head End CMTS CGN CMTS RTR Cre Netwrk RTR IPv4 Internet Head End CMTS CMTS CGN RTR 24 Cable Televisin Labratries, Inc. 2011. All Rights Reserved. Prprietary/Cnfidential. 9/14/2011
Architecture Hybrid (Phased apprach) Head End CMTS CMTS Large Head End CMTS CGN CMTS RTR RTR Cre Netwrk Reginal Peering Pint CGN RTR IPv6 Internet IPv4 Internet Head End CMTS CMTS RTR 25 Cable Televisin Labratries, Inc. 2011. All Rights Reserved. Prprietary/Cnfidential. 9/14/2011
Ruting CGN Traffic NAT444 CGN traffic must be identified and ruted t crrect CGN Withut frcing all traffic thrugh a ptential bttleneck. Best apprach Classify nce at the edge Use native ruting r tunnels in the cre 26 Cable Televisin Labratries, Inc. 2011. All Rights Reserved. Prprietary/Cnfidential. 9/14/2011
Ruting CGN Traffic - Optins Decisin Criteria Classificatin Transprt Destinatin Address Default Gateway Surce Address TOS DNS Xlatin Virtual Ruter DOCSIS service flw Surce Ruting Plicy Ruting Native Ruting VLAN MPLS Tunnel (GRE?) Plicy Ruting 27 Cable Televisin Labratries, Inc. 2011. All Rights Reserved. Prprietary/Cnfidential. 9/14/2011
Redundancy Onsite device redundancy and site-t-site failver Device redundancy 1+1 redundancy : ne device directly supprting anther N+1 redundancy : single device prviding backup fr multiple (N) devices. Backup device can be active all the time (active/active) r it can be drmant until needed (active/passive) Apply N+1 redundancy at reginal and large headend (50k+ CGN subs) lcatins Extra capacity is readily available when a single CGN fails Rely n dynamic ruting t carry CGN traffic t alternate lcatin when an entire CGN lcatin fails 28 Cable Televisin Labratries, Inc. 2011. All Rights Reserved. Prprietary/Cnfidential. 9/14/2011
Lad balancing and Scalability Sizing individual CGN devices (blades / pizza-bxes ) Factrs t determine number f subscribers supprted Device thrughput New cnnectins per secnd per subscriber (peak) Number f cncurrent cnnectins per subscriber (average) Necessary thrughput per subscriber (average) Lgging vlume Once capacity f each CGN device is understd, develp a detailed lad balancing and scaling strategy 29 Cable Televisin Labratries, Inc. 2011. All Rights Reserved. Prprietary/Cnfidential. 9/14/2011
Lad balancing and Scalability Example Example regin with 1millin ttal CGN subscribers Reginal hub serving 500k CGN subs at varius headends w/ut lcal CGNs Reginal Hub Tw devices prvide 10% additinal capacity fr entire regin (including headend CGNs) Device prviding N+1 redundancy fr Hub Headend with 50k CGN subs 30 Headend with 40k CGN subs Headend with 100k CGN subs Cable Televisin Labratries, Inc. 2011. All Rights Reserved. Prprietary/Cnfidential. 9/14/2011
Server Lcatin and NAT Bypass Optimizing lcal traffic and subscriber access t advanced services apprached differently fr each CGN technlgy NAT444 DS-Lite 6RD Place advanced service servers inside f CGN, add IPv6 Use dynamic ruting r a VPN t bypass NAT fr ther servers Enable IPv6 n all servers All IPv4 ges thrugh CGN All IPv4 is Native r NAT444, IPv6 servers shuld be utside 6RD tunnel end pints 31 Cable Televisin Labratries, Inc. 2011. All Rights Reserved. Prprietary/Cnfidential. 9/14/2011
IP Addressing Need bth inside and utside addressing plans Design determined by number f CGN subscribers and cmpressin algrithm Expected number f CGN subscribers Assuming 6% grwth/year fr 3 years Industry can expect excess demand f 20% subs (High Watermark) Prpse t start with 8-10x cmpressin and scale if needed 32 Cable Televisin Labratries, Inc. 2011. All Rights Reserved. Prprietary/Cnfidential. 9/14/2011
IP Addressing: Public utside addressing Finding utside addresses Re-purpse, acquire, r reserve Des nt need t be cntiguus space Primary cncern is the cmpressin rati, Defined as rati f subscribers using a CGN t the number f public IPv4 addresses supprting them Prt restrictins likely nt an issue at 8 subs per IP Ptential fr min/max f 8k prts 33 Cable Televisin Labratries, Inc. 2011. All Rights Reserved. Prprietary/Cnfidential. 9/14/2011
IP Addressing: Inside Addressing Use a single netwrk-wide pl f inside addresses /10 Shared Transitin Space (in IETF last call) Out f larger pl, assign lcal (per headend) blcks (/16 per CGN device = 65k addresses) Fr peratinal clarity, lgging, the ability t insert lcal CGNs, and ptential ge-lcatin benefits DS-Lite: Any addresses are acceptable and can be reused per tunnel 34 Cable Televisin Labratries, Inc. 2011. All Rights Reserved. Prprietary/Cnfidential. 9/14/2011
Gelcatin Prcess f identifying physical lcatin f a user primarily thrugh surce IP address. Used by targeted advertising and lcatin aware services Users behind a CGN d nt have a unique public address it becmes mre difficult t determine lcatin Lcal headend CGNs will ffer rughly equal granularity t what is available tday Reginal CGNs will dilute ge-lcatin data 35 Cable Televisin Labratries, Inc. 2011. All Rights Reserved. Prprietary/Cnfidential. 9/14/2011
Issues arund Lawful Intercept and Lgging CGN LOGGING 36 Cable Televisin Labratries, Inc. 2011. All Rights Reserved. Prprietary/Cnfidential. 9/14/2011
Lawful Intercept & CGN Lgging ISPs respnd t law enfrcement requests r attack/abuse reprts Infrmatinal Lg messages 173 542 bytes (DS-Lite) 150 450 bytes (NAT444) Template fr lgging is currently nt cnfigurable Data fields include: Time Stamp, Surce IP/Prt, NAT IP/Prt, Destinatin IP/Prt, Hst Name 37 Cable Televisin Labratries, Inc. 2011. All Rights Reserved. Prprietary/Cnfidential. 9/14/2011
CGN Lgging (NAT444) 38 Cable Televisin Labratries, Inc. 2011. All Rights Reserved. Prprietary/Cnfidential. 9/14/2011
CGN Lgging (DS-Lite) 39 Cable Televisin Labratries, Inc. 2011. All Rights Reserved. Prprietary/Cnfidential. 9/14/2011
Lawful Intercept : Identity Traceback Illustratin DHCP Lg A:10.0.0.1 B:10.0.0.2 C:10.0.0.3 Webserver lg 204.57.36.2:4567 - - [10/Oct/2010:13:55:36-0700] "GET /illegal.html HTTP/1.0" 200 2326 : 10.0.0.1 : 10.0.0.2 : 10.0.0.3 CGN Lg 10.0.0.2:1234 -> 204.57.36.2:4567 [10/Oct/2010:13:55:36-0700] 10.0.0.1:1234 -> 204.57.36.2:2345 [10/Oct/2010:13:55:38-0700] 10.0.0.3:4356 -> 204.57.36.2:3456 [10/Oct/2010:13:55:45-0700] 10.0.0.2:3456 -> 204.57.36.2:4569 [10/Oct/2010:13:55:47-0700]
IP Address Reputatin IP blacklisting is mre prblematic with multiple subscribers behind a single utside IP All subs behind that IP are affected Any sub behind that IP can cause the listing Examples: Secure transactins (Banking, Strefrnts, etc.) Email spam lists (Spamhaus, etc.) Individual website blcking (cmment spam, etc.) Difficult t trublesht 41 Requires CGN lgging Cable Televisin Labratries, Inc. 2011. All Rights Reserved. Prprietary/Cnfidential. 9/14/2011
CGN Lgging Generates Huge Data Vlumes Subscribers generate arund 33,000 cnnectins/day 150-byte lg messages x 33,000 cnnectins/day= 5MB/day/sub 1M-sub ISP will generate 150 TB f data/mnth (1.8 PB/yr) Assuming 50,000 subs/cgn => 23 Mbps f lgging traffic Imagine DB search times Cnclusin: per-cnnectin CGN lgging is nt scalable 42 Cable Televisin Labratries, Inc. 2011. All Rights Reserved. Prprietary/Cnfidential. 9/14/2011
SUMMARY 43 Cable Televisin Labratries, Inc. 2011. All Rights Reserved. Prprietary/Cnfidential. 9/14/2011
CGN Design Summary Optimizing ruting Intrducing redundancy Balance lad efficiently and scale Psitining internal servers t avid traversing CGN Efficient use f scarce IPv4 addresses Imprving targeted advertising results Facilitating LEA respnse Secure CGN against DS, rute leaking Managing IP address reputatin issues with partners Reducing lgging requirements 44 Cable Televisin Labratries, Inc. 2011. All Rights Reserved. Prprietary/Cnfidential. 9/14/2011
Wrapping up Fr many peratrs CGN will be inevitable NAT 444, DS-Lite Experience gap between CGN and nn-cgn subscribers Testing is needed! CGN disrupts and degrades services Prperly designed CGN system can manage impacts Hybrid - Phased apprach seems a gd chice 45 Cable Televisin Labratries, Inc. 2011. All Rights Reserved. Prprietary/Cnfidential. 9/14/2011