Carrier Grade NAT44 on IOS-XR Deployment Experience

Carrier Grade NAT44 on IOS-XR Deployment Experience Nicolas Fevrier Rajendra Chayapathi Syed Hassan

Agenda Introduction NAT Principles and Mechanisms Bulk-Port Allocation Port limit Static Port Forwarding ALG Logging Hardware Deployment feedback Routing consideration and Best Practices Redundancy 3

INTRODUCTION

Introduction Do you think CGN is evil? Yes but it s a necessary one IPv4 address exhaustion End-to-end IPv6 traffic, are you ready? The same cards can be used for: NAT44 But also for smooth transition to IPv6 Let s jump directly into the deep end 5

Facts About IPv4 Shortage LIR are allocating their last blocks On 14 September 2012, the RIPE NCC started allocating from the last /8 of IPv4 addresses received from IANA IPv4 grey/black market is flourishing 6

Cisco s Strategy: 3 Pillars Cisco s strategy relies on three pillars Preserve (Business Continuity) NAT44 / CGN Optimize the IPv4 resource and allow growth Prepare (Encourage Adoption) Offer IPv6 to the customers 6rd: transport IPv6 on top of a IPv4 infrastructure Prosper (Interworking) DS-Lite, MAP-T/E: transport of the remaining IPv4 traffic on top of a IPv6 backbone NAT64: translate to the IPv4 at the border Among IOS-XR products, the ISM and VSM (ASR9000) and CGSE and CGSE+ (CRS) cards are the tools used to build these three pillars. 7

Vocabulary i2o / o2i: inside to outside / outside to input NAT/NAPT: Network Address (and Port) Translation CGx: carrier grade (CGN: Carrier Grade NAT) LSN: Large Scale NAT ALG: Application Layer Gateway GRT: Global Routing Table SL/SF: Stateless/Stateful 8

Translation Protocols Illustrated Stateful vs Stateless Example: we have 16 public addresses 4 4 4 4 Stateless translation 1 external IP : 1 internal IP No multiplexing no DB needed Stateful translation 1 external IP : n internal IP Multiplexing DB to maintain 9

Stateless vs Stateful Stateless 1:1 translation i2o or o2i initiated sessions are treated equally Allows asymmetrical traffic Better convergence time Potential inline implementation No logging required Protocols: NAT64SL, 6rd, MAP-T Stateful 1:n translation (port multiplexing) Needs translation DB maintenance Logging scalability can be an issue Need static port forwarding or PCP to accept o2i initiated sessions May need ALGs In case of failover, we need to reestablish sessions on a new device Protocols: NAT44, NAT64SF, DS Lite 10

NAT44 Introduction Preserve the investment: buy time to prepare migration to IPv6 Not the solution but meets a vast majority of user current needs NAT vs NAPT Defined since 2001 (RFC3022, RFC4787, RFC5382, RFC5508) Unicast TCP/UDP/ICMP Stateful Permit multiplexing : several internal hosts will use the same external address, maximizing the IPv4 resource ALGs 11

NAT44 Overview IPv4 Traffic Source Address = 10.1.1.10 Outside Address = 170.0.0.1 IPv4 Backbone CGN IPv4 Internet Stateful translation protocol from an IPv4 space to another IPv4 space IPv4 space public or private Usually, from private (RFC1918) to public but not necessary Translation table or database (DB) maintained on the CGN card

NAT444 or Double NAT44 IPv4 Traffic Source Address = 10.1.1.10 Outside Address = 170.0.0.1 CPE IPv4 Backbone CGN IPv4 Internet Double step stateful translation: At CPE level Between home network and ISP access network At CGN level Between ISP network and public address network From CGN perspective: NAT44 = NAT444 Translated Address = 10.8.1.111

NAT PRINCIPLES and MECHANISMS

NAT Mechanisms Inside VRF NAT Engine Outside VRF Web Client 10.10.10.2 Source 10.10.10.2:2493 Destination 5.20.3.2:80 Source 100.2.1.24:8442 Destination 5.20.3.2:80 Web Server: 5.20.3.2 Translation table Collector 10.10.10.2:2493 100.2.1.24:8442 Logging Record Syslog Netflow Inside VRF NAT Engine Outside VRF Web Client 10.10.10.2 Source 5.20.3.2:80 Destination 10.10.10.2:2493 Source 5.20.3.2:80 Destination 100.2.1.24:8442 Web Server: 5.20.3.2

NAT Principles EIM/EIF vs EDM/EDF EIM: End-point Independent Mapping destination address and port for i2o traffic not tracked If multiple destinations but source address and port are the same no other entry created Sometime referred as full cone NAT Dest A:80 Dest B:80 Source Y:1430 Dest B:80 Inside Outside Destination X:4828 Y:1430 * EDM: End-point Dependent Mapping Opposite of EIM Destination info is maintained in DB Source X:4828

NAT Principles EIM/EIF vs EDM/EDF EIF: End-point Independent Filtering Once entry is present in the table For o2i traffic, we don t verify source address/port Better scalability and larger support EDF: End-point Dependent Filtering Opposite of EIF Check the source addresses for o2i traffic Required in some situation: bill shock effect Source A:80 Dest Y:1430 Dest X:4828 Source B:4234 Inside X:482 8 Source C:80 Outsid e Destinatio n Y:1430 *

NAT Principles Paired IP Address Assignment We use the same external IP address mapping for all sessions associated with the same internal IP address (RFC4787) Each inside odd port is mapped to an outside odd port number Each inside even port is mapped to an outside even port number Outside Inside Source X:2104 Source A:10302 Source A:11238 Source X:23342 Source A:10985 Source X:48271 Source B:1045 Source B:1491 Source B:1228 Source Y:29301 Source Y:43017 Source Y:1024 Inside Outsid e X:2104 A:1030 2 X:2334 2 X:4827 1 A:11238 A:1098 5 Y:29301 B:1045 Y:43017 B:1491 Y:1024 B:1228

NAT Principles Hair Pinning Two endpoints on inside NAT can communicate to each others using external NAT IPv4 addresses and ports. Outside A:10302 B:11237 Inside Source X:2104 Source Y:11003

NAT Principles Address Allocation First flow per Inside source address CGN picks an Outside address that has at least 1/3 of its ports free All subsequent Flows from that Inside source will use the same Outside address. No? NAT IP1 IP2 IP3 IP4 IP5 IP6 IP7 IP8? Used port Free port Ok

NAT Principles Address Allocation If that Outside address is completely exhausted, then a random selection is made from the remaining addresses, repeated until an address is chosen or it is determined that none are available (which results in an ICMP error message)? NAT NAT Used Free? port port? IP1 IP2 IP3 IP4 IP5 IP6 IP7 IP8 IP1 IP2 IP3 IP4 IP5 IP6 IP7 IP8 ICMP error No

NAT Principles Port Allocation Ports are randomly picked from the list of available (unused) ports associated with the chosen Outside IP address Each port is allocated once, regardless of which L4 protocol (UDP, TCP) is being used in the Flow CGN creates a Translation binding (state) between Inside source IP address + port and Outside source IP address + port NAT? IP1? Inside Outside IPa:2104 Used port Free port IP1:10302

NAT Principles Port Allocation If the randomly chosen port is already being used, the selection increments (around a ring) until an available port is found; if none are available then an ICMP error message is sent If the Inside source already has a number of Flows equal to the configured per-user limit, then the allocation is rejected and an ICMP message is returned NAT? IP1 port-limit=8 Inside Outside IPa:Pa IP1:P1 IPa:Pb IP1:P2 IPa:Pc IP1:P3 IPa:Pd IP1:P4 IPa:Pe IP1:P5 IPa:Pf IP1:P6 IPa:Pg IP1:P7 IPa:Ph IP1:P8 IP1 ICMP error IPa:Pi No

Algorithm-based / Predefined NAT Often referred as Deterministic NAT, coming in future releases Opposite approach than random allocation mechanisms described before Allows predictable mapping of source addresses/ports between the inside and outside world Based on an algorithm, each internal address will be allocated an external address and range Predefined NAT is still stateful (translations are still stored in DB) Main benefit: logging is no longer necessary (but will still be possible) Main flaw: sub-optimal address allocation Addresses and port ranges are allocated regardless of the presence or usage of the internal users To meet requirements of certain ALGs, it will be necessary to allocate contiguous ports SDNAT (stateless) draft has been discontinued

BULK PORT ALLOCATION

Bulk Port Allocation Aims at reduces data generated by logging Bulk port allocation behavior A subscriber creates the first connection N contiguous ports are pre-allocated (ex: 2064 to 2080 if N=16) Bulk-allocation message (NFv9 and/or syslog) is logged for the port-range Additional connections (up to N) will use one of the pre-allocated ports New pool allocated if subscriber creates > N concurrent connections Bulk-delete message is logged when subscriber terminates all sessions from pre-allocated pool NAT Outside IP1 Logging Record Collector Syslog Netflow

Bulk Port Allocation When bulk size is changed, all current dynamic translations will be deleted Ports below dynamic start range (< 1024) are not allocated to bulk It can take one of the following values: 16, 32, 64, 128, 256, 512, 1024, 2048, 4096 (8 in IOS XR 4.3.1) port-limit / 4 bulk-port-alloc port-limit x 2 Recommendation: closest value to half the port-limit Orthogonal with Destination Based Logging, can NOT be configured together Port range allocation is random, in following examples we picked 1024-1039 and 1040-1055 for the sake of simplicity only

BPA Illustrated Example Bulk=16 Source Address = 10.1.1.1 IPv4 Traffic NAT CGN Outside Address from pool = 99.0.0.1 IPv4 Internet 10.1.1.2

BPA Illustrated Example Bulk=16 1 10.1.1.1 10.1.1.2 NAT 1 packet from 10.1.1.1 to 30.1.1.1:80 IPv4 Internet 1 2 1 packet from 10.1.1.1 to 30.1.1.1:25 2 3 1 packet from 10.1.1.2 to 40.1.1.1:80 3

BPA Illustrated Example Bulk=16 10.1.1.1 10.1.1.2 NAT IPv4 Internet 1 1 packet from 10.1.1.1 to 50.1.1.1:80 1 2 1 packet from 10.1.1.1 to 60.1.1.1:80 2

BPA Illustrated Example Bulk=16 Same rules for init and active timeout apply for bulk ports 1 2 3 4 5 6 7 1 packet from 10.1.1.1 to 30.1.1.1:80 BPA=16 can reduce the logging volume MUCH more than by 16 7

Bulk Port Allocation With NAT444, it s very likely that at least one device is connected behind the CPE at any given time Consequently, logging for the port allocation is generated once and the port block is never de-allocated or de-allocated many weeks or months later It s exactly what the protocol is supposed to do, but it creates some issues Potential issue with logging collector correlator Another issue could be the security. It makes one CPU always use the same port range and reduces the scope for attackers Workaround: DHCP lease time reduced to re-assign a different IP to the CPE every couple of weeks.

Bulk Port Allocation: Configuration Config parser will enforce the selection respecting: 8, 16, 32, 64, 128, 256, 512, 1024, 2048, 4096 port-limit / 4 bulk-port-alloc port-limit x 2 Recommendation: closest value to half the port-limit service cgn POC-1 service-type nat44 nat44-1 inside-vrf Inside-1 bulk-port-alloc size 256

PORT LIMIT

Per-user Port Limit For stateful translation protocols (NAT44, NAT64 SF, DS Lite), each user can be assigned a maximum number of ports. It prevents a single user to consume all port resources port-limit=8 IP1 NAT Inside IPa:Pa IPa:Pb IPa:Pc IPa:Pd Outside IP1:P1 IP1:P2 IP1:P3 IP1:P4? IPa:Pe IPa:Pf IPa:Pg IPa:Ph IPa:Pi IP1:P5 IP1:P6 IP1:P7 IP1:P8 No

Per-user Port Limit Port-limit can be defined per protocol But also per VRF allows different treatment for different type of customers Finding the proper port-limit is a very tricky exercise No simple rule of the thumb Different for each type of customer (ADSL, Mobile, Cable, Enterprise ) Different for each theater (Asia, Europe, Russia, Americas ) Scripts can be used to collect average and maximum port usage

Per-user Port Limit on CGSE Exceeding the port limit will trigger a syslog message: [Portblockrunout 17 10.1.11.202 ivrf- 2005 - - ] Portblockrunout: event name signifying the port limit hit event 17: it was hit by a UDP packet requesting the translation 10.1.11.202: is the subscribers private IP ivrf: name of the inside VRF 2005: private port number These messages are throttled For 10.1.11.202, once we report this message, we will not repeat them for the same subscriber until it goes below 70% of max limit and then goes up again and hits the port limit Can be used to quickly user consuming a lot of ports

Configuring Port-Limit It s a safety net preventing one user to use all resources For stateful translation protocols each user can be assigned a maximum number of ports NAT44 and NAT64SF will use keyword portlimit We can use every value between 1 to 65535, default is 100 Defined per protocol or globally since 4.3.1 service cgn demo service-location preferred-active 0/1/CPU0 service-type nat44 nat44-1 portlimit 512 inside-vrf ivrf1 portlimit 256 inside-vrf ivrf2!!

STATIC PORT FORWARDING and PCP

Session Initiated From the Outside? 10.1.1.1 IPv4 Traffic Map pool = 99.0.0.0/24 CGN IPv4 Internet 30.0.0.1 0 Inside Outside TCP state 2 No entry in the NAT DB, o2i packets are discarded 1 1 With stateful translation mechanisms, a traffic initiated from the outside will be discarded Static Port Forwarding or Port Control Protocol necessary

Static Port Forwarding 10.1.1.1 IPv4 Traffic Map pool = 99.0.0.0/24 CGN IPv4 Internet 30.0.0.1 1 0 Inside Outside TCP state service cgn demo service-type nat44 nat1 inside-vrf insidevrf1 protocol tcp static-forward inside address 10.1.1.1 port 80 2 6000 entries max Inside Outside TCP state 11.1.1.1:80 99.0.0.1:80 static 4 3 Static-port-forwarding creates an entry in the NAT DB

Verifying Static-Port-Forwarding External address is picked by the system, not the user (based on hashing of inside address) RP/0/RP0/CPU0:R#sh cgn demo inside-translation protocol tcp inside-vrf Inside inside-address 10.12.0.250 port s 10000 e 10000 Inside-translation details --------------------------- CGN instance : demo Inside-VRF : Inside -------------------------------------------------------------------------------------------- Outside Protocol Inside Outside Translation Inside Outside Address Source Source Type to to Port Port Outside Inside Packets Packets -------------------------------------------------------------------------------------------- 100.0.0.58 tcp 10000 15819 static 0 0 RP/0/RP0/CPU0:R#

Port Control Protocol PCP client on private network 10.1.1.1 IPv4 Traffic Map pool = 99.0.0.0/24 CGN IPv4 Internet 30.0.0.1 Host on public network PCP Server PCP allows applications to create mappings from an external IP address+proto+port to an internal IP address+proto+port PCP Server is a software instance via which clients request and manage explicit mappings PCP Client issues requests to a server A PCP Client can issue PCP requests on behalf of a third party device A PCP request is transported on UDP(v4/v6) packet with destination port 5351 Supported on CGSE cards for NAT44, NAT64 and DS-Lite http://tools.ietf.org/html/draft-ietf-pcp-base-29

Port Control Protocol 10.1.1.1 IPv4 Traffic Map pool = 99.0.0.0/24 CGN IPv4 Internet 1 MAP Request 99.0.0.1 TCP 80 MAP Response 2 0: SUCCESS 4 FIN or RST 0 3 5 Inside Outside TCP state Inside Outside TCP state 10.1.1.1:80 99.0.0.1:80 pcp_explicit Inside Outside TCP state 10.1.1.1:80 99.0.0.1:80 pcp_explicit

Port Control Protocol IPv4 Traffic PCP Req/Resp 10.1.1.1 Map pool = 99.0.0.0/24 CGN IPv4 Internet 1 MAP Request 99.0.0.1 TCP 80 MAP Response 2 11:CANNOT_PROVIDE_EXTERNAL Available external port: 84 0 Inside Outside TCP state 10.1.1.1:80 99.0.0.1:80 dynamic Other result codes could be: 1:UNSUPP_VERSION 2:NOT_AUTHORIZED 3:MALFORMED_REQUEST 4:UNSUPP_OPCODE 5:UNSUPP_OPTION 6:MALFORMED_OPTION 7:NETWORK_FAILURE 8:NO_RESOURCES 9:UNSUPP_PROTOCOL 10:USER_EX_QUOTA 11:CANNOT_PROVIDE_EXTERNAL 12 ADDRESS_MISMATCH 13:EXCESSIVE_REMOTE_PEERS

Port Control Protocol 10.1.1.1 IPv4 Traffic Map pool = 99.0.0.0/24 CGN IPv4 Internet 0 Inside Outside TCP state 1 PEER Request 99.0.0.1 TCP 80 PEER Response 2 0: SUCCESS 4 FIN or RST 3 5 Inside Outside TCP state 10.1.1.1:80 99.0.0.1:80 pcp_implicit Inside Outside TCP DB entry removed state

APPLICATION LAYER GATEWAYS

Need for ALG ALG are features allowing upper layer inspection to track a particular behavior (port negotiation, ) and make sure the protocol will be unaffected by the translation Cisco s position is to discourage the pursue of ALGs Applications are regularly rewritten and keeping track of each change is challenging NAT traversal is more generally handled at the application level Supported ALGs in CGN cards Active FTP (passive FTP doesn t need ALG) RTSP (used for some streaming services) PPTP (for legacy VPN applications)

Active FTP ALG In active mode FTP the client connects from a random unprivileged port (N > 1023) to the FTP server's command port 21 then, client starts listening to port N+1 and sends the FTP command PORT N+1 to the FTP server the server will then connect back to the client's specified data port from its local data port, which is port 20 ALG converts the network Layer address information found inside an application payload Note: Passive FTP Mode does NOT need any ALG

RTSP ALG Real-Time Streaming Protocol is not a streaming protocol It s a remote control protocol for streamers (which use RTP/RTCP or RDT) a text-based protocol based on methods (like requests) and transported on port554 RTSP session is not a connection per say since it s not tied to a transportlevel connection, even if transported by TCP Our implementation considers the server is located outside and clients are inside RTSP is used in many streamers like QuickTime or RealNetworks (less and less used with generalization of HTML5)

PPTP ALG Point to Point Tunneling Protocol is used by legacy VPN solutions Encapsulate PPP packets in IP GRE Translation of PPTP packet is challenging because we don t translate source ports but a peer caller ID field contained in the GRE header PAC: PPTP Access Concentrator, in the public side (Outside) PNS: PPTP Network Server, in the private side (Inside) PPTP PNS NAT IPv4 Internet PAC Control Connection (TCP1723)

Configuring ALGs We currently support three ALGs types for NAT44 (none for NAT64SF and only FTP for DS Lite) ActiveFTP (not needed for PassiveFTP) RSTP (for Real Audio G2 and windows media player), default port is 554 PPTP (for legacy VPN systems) service cgn demo service-type nat44 nat44-1 alg ActiveFTP alg rtsp port 10000 alg pptpalg!

Verify ALG Activity When a translation database entry will be allocated based on ALG, it will appear like: RP/0/RP0/CPU0:R#sh cgn demo inside-translation protocol tcp inside-vrf Inside inside-address 10.13.0.29 port s 1 e 65535 Inside-translation details --------------------------- CGN instance : demo Inside-VRF : Inside -------------------------------------------------------------------------------------------- Outside Protocol Inside Outside Translation Inside Outside Address Source Source Type to to Port Port Outside Inside Packets Packets -------------------------------------------------------------------------------------------- 100.0.0.221 tcp 1043 41493 dynamic 51 55 100.0.0.221 tcp 55000 26236 dynamic 6 5 100.0.0.221 tcp 55001 16300 dynamic 6 5 100.0.0.221 tcp 55002 28942 alg 23 22 100.0.0.221 tcp 55003 4373 dynamic 5 5 RP/0/RP0/CPU0:R#

LOGGING

Need for Logging Entries in NAT table are of temporary nature Any Stateful protocol (NAT44, NAT64SF, DS-Lite) requires logging Directive 2006/24/EC - Data Retention: EU Law Logging preserves the mapping information between an internal and external CGSE and ISM cards supports Netflow v9 and Syslog NAT Logging Record IPv4 Internet Syslog Netflow

What CGN information needs to be stored by ISPs? Source IP address and port translation history to be able to reliably identify the private IP translated to public IP at one precise moment further inspection of RADIUS or DHCP database can be performed to provide the identity of subscriber (e.g. MAC address of device or username) Format of the information (as long as translation can be inverted based on the input parameters): ASCII format Compressed text/binary files or relational database that contain translation history details Outcome of an algorithmic mapping of private IP address to public IP address/port

Dynamic or Pre-defined NAT? No definitive and easy answer The logging solutions Dynamic NAT Per-session logging (w/syslog or w/netflow) Bulk Port Allocation logging (w/syslog or w/netflow) Destination Based Logging w/syslog or w/netflow Pre-defined NAT Each choice is optimizing subset of requirements at the expense of others Pre-defined NAT

Destination-Based Logging DBL permits to specifically log destination address and port X1 X2 A Internal NAT External X3 Logging Record X4 Syslog Netflow Tim e Inside IP/Port Outside IP/Port Destination IP/Port T1 A:Pa IP1:P1 X1:Pd1 T2 A:Pb IP1:P2 X2:Pd2 T3 A:Pc IP1:P3 X3:Pd3 T4 A:Pd IP1:P4 X4:Pd4

Destination-Based Logging Why would you like to use DBL? Legal regulations in country Many web servers are not logging port information for each session (not respecting RFC6302 Logging Recommendations for Internet-Facing Servers) Others Need for data analytics solution e.g. Offers very detailed info on user behavior Why should you avoid using DBL? Privacy considerations Country regulations Interpretation of EU directive Conflicts with Bulk Port Allocation and Deterministic NAT Increased storage requirements 6 additional bytes in NFv9 to store A+P draft-ietf-behave-lsn-requirements REQ-12: A CGN SHOULD NOT log destination addresses or ports unless required to do so for administrative reasons

Destination-Based Logging The CGN card will generate templates 271 for Add records and templates 272 for Delete records service cgn POC-1 service-type nat44 nat44-1 inside-vrf Inside-1 map address-pool 150.0.0.0/17 external-logging netflow version 9 server address 172.16.255.254 port 5000 session-logging!

Syslog or Netflow v9? Two options in CGN cards today: Syslog Netflow v9 Netflow is preferred since lighter Some customers select syslog: existing collection infrastructure based on syslog to guarantee multi-vendor interoperability Format Netflow v9 Binary Template based format Syslog ASCII RFC52432 Transport UDP UDP Sequence number Scalabilit y Yes in header High (tested) IPFIX doesn t bring anything to the CGN logging hence isn t considered Both NFv9 and Syslog can be configured simultaneously in a CGN system No Need BPA

Syslog or Netflow v9? Keep in mind before selecting your collector Traditional use of NFv9 or syslog requires much lower data rates (< 50k fps) NAT is still a relatively new application using NF hence there is no existing data analysis tool box available NAT requires the records to be stored in a Database Most NF collectors store only the analysis results in a DB, but not the records themselves and are therefore not suitable Templates for NAT44 NAT64SF DS Lite with or without Bulk-Allocation Destination-based-logging.

Syslog for CGN Message needs to comply to RFC5424 format Field are separated by space and non-applicable field are - <Priority> <Version> <Time stamp> <host name> - - <Application name (NAT44 or DSLITE)> - [Record 1][Record 2] [EventName <L4> <Original Source IP> <Inside VRF Name> <Original Source IPv6> < Translated Source IP> <Original Port> <Translated First Source Port> <Translated Last Source Port>] Example: NAT44 with Bulk-Port-Alloc 1 2011 May 31 10:30:45 192.168.2.3 - - NAT44 - [UserbasedA - 10.1.32.45 INVRFA - 100.1.1.28-12544 12671]

Netflow v9 for CGN Netflow v9 supports flexible field definition Light weight transport via UDP NFv9 records are in binary Based on templates containing IPFIX entities (http://www.iana.org/assignments/ipfix/ipfix.xml) Supported since the first days on CGN Different behavior than Netflow on routers Record creation / deletion of NAT entries Doesn t count packets Doesn t sample packets headers Generated by the CGN card and not the MSC in the CRS or the LC in ASR9K

Netflow v9 templates for CGN A few examples

Netflow Packet Generation With default path MTU = 1500B, one netflow packet can hold around 50 creation records Generation is handled at the CPU core level An event (new translation or deletion of an existing one) will trigger the creation of a NF packet but it s not sent directly If other events happen for the same core, records are added to the NFv9 packet Packet is sent if we reach the MTU size or if we exceed one second

Configuring NFv9 Options NFv9 is supported for all stateful translation protocol. Only a single server can be defined for instance Templates are regenerated and sent by default every 500 packets or 30 minutes service cgn ISM service-type nat44 nat44-1 inside-vrf Inside-1 external-logging netflow version 9 server address 1.2.3.4 port 123 path-mtu 2000! can be configured from 100 to 2000 refresh-rate 100! Regenerate NF record with template flowset every 100 logging packets timeout 10! Regenerate NF record with template flowset every 10 minutes session-logging! Session logging Enable Flag!

HARDWARE

Service Cards on IOS XR Routers Carrier Grade Service Engine (CGSE) for all CRS routers CGSE-PLUS for CRS-3 and CRS-X routers Integrated Service Module (ISM) for ASR9000 routers Virtualized Service Module (VSM) for ASR9000 routers with RSP440 Same form-factor than any Line Card No physical port / interfaces (except CGSE+ and VSM for future usage) Multi-purpose cards, they can be used for different applications Very similar to Intel server, they run a Linux distribution Use virtual interfaces to communicate with the rest of the system VSM introduces the Virtual Machines and the service chaining capability

Carrier Grade Service Engine (CGSE) Supported with CRS-1 / CRS-3 / CRS-X fabric 4-slot / 8-slot / 16-slot single/multi chassis Up to 12 cards in the 16-slot chassis Multi-purpose service card CGN Arbor TMS Monte Vista Linux distribution but configuration via IOS-XR 20M translations 1M sessions established per second 20Gbps

Carrier Grade Service Engine (CGSE) GLIK FPGA GLIK FPGA PLA M I D P L A N E ipse EgressQ epse IngressQ FabQs M I D P L A N E F A B R I C CGSE PLIM Paired with MSC40 or FP40. MSC40/FP40

Load-Balancing Traffic inside CGSE Cores 64 cores are available an each CGSE card (2^6) and LB decision is performed by the egress PSE ASIC (emetro) For i2o traffic, the least 6 bits of the source IP address will be used For o2i traffic, the least 6 bits of the destination IP address will be used. It implies that we can not assign a map pool prefix longer than /26 to use each core of the system: /26: each core will handle a single IP address from the map pool range (outside) /24: each core will handle 4 IP addresses from the map pool range (outside)

Carrier Grade Service Engine PLUS (CGSE+) Supported with CRS-3 / CRS-X fabric 8-slot / 16-slot single/multi chassis Up to 12 cards in the 16-slot chassis Multi-purpose service card CGN Arbor TMS (future) DPI / Analytics (future) Monte Vista Linux distribution but configuration via IOS-XR Current supports: NAT44 / 6rd 80M translations 1M+ sessions established per second 70+ Gbps

Carrier Grade Service Engine PLUS (CGSE+) DDR 16GB DDR 16GB Netlogic NPU Netlogic NPU Beluga PLA M I D P L A N E ipse EgressQ epse IngressQ FabQs M I D P L A N E F A B R I C CGSE+ PLIM MSC140/FP140 Paired with MSC140 or FP140 in a CRS-3 or CRS-X chassis Not supported in CRS-1 chassis

Integrated Service Module (ISM) Supported with RSP2 and RSP440 9006 and 9010 chassis (not in 9001 or 99xx) Multi-purpose service card CGN CDS-IS/TV (discontinuated) RedHat Linux distribution but configuration via IOS-XR 20M translations 1M sessions established per second 14Gbps

ISM Architecture PPC DRAM 24GB 24GB Intel CPU I/O Hub Bridge Bridge Bridge Bridge Fabric ASIC B A C K P L A N E Application Domain IOS-XR Domain

Virtualized Service Module (VSM) Supported with RSP440 (and future RSPs) All 9x00 chassis except 9001 Multi-purpose service card CGN IPsec Mobile GW Service chaining KVM virtualized environment Current CGN Supports: NAT44 60M translations 10M+ sessions established per second 60Gbps

VSM Architecture SFP+ SFP+ SFP+ SFP+ Quad PHY 32GB DDR3 32GB DDR3 32GB DDR3 32GB DDR3 Crypto/DPI Assist Ivy Bridge Ivy Bridge Crypto/DPI Assist Crypto/DPI Assist Ivy Bridge Ivy Bridge Niantic Niantic Niantic Niantic Niantic Niantic Niantic 48 ports 10GE Typhoon NPU Typhoon NPU XAUI PCIe Fabric ASIC 0 Fabric ASIC 1 B A C K P L A N E Crypto/DPI Assist Application Processor Module (APM) Service Infra Module (SIM)

Performance / Scalability CGSE CGSE+ ISM VSM Sessions 20M 80M 20M 60M Target: 80M+ Establishment Rate Bandwidth (IMIX) Physical Interfaces 1M/s 1M/s 1M/s Up to 13M/s 20Gbps 70Gbps 14Gbps 60Gbps No 2x10G (future) No 4x10G (Future) Platform CRS 4-slot CRS 8-slot CRS 16-slot CRS Multi- Chassis ASR9001 ASR9006 ASR9010 ASR9922 9k nv Satellite 9k nv Cluster CGN Card 3 x CGSE or CGSE+ 6 x CGSE or CGSE+ 12 x CGSE or CGSE+ Supported since 4.3.1 Not supported 3 x ISM or VSM 6 x ISM or VSM VSM only VSM is compatible VSM support targeted for 5.2.0

DEPLOYMENT FEEDBACK

Deployment Tips CGSE(+) PLIM are considered high powered PLIMs Their power consumption is higher But more important, they generate more heat than other PLIMs (heat will naturally go up) In 16-slot chassis, their position must be thought carefully Some PLIMs are considered Thermally sensitive and can not be positioned above high powered PLIMs : CRS-1 OC768 (C/L-band) DWDM PLIM CRS-1 OC768 DPSK C/L-BAND STD CHAN PLIM So, CGSE should be positioned ideally in upper shelf If necessary, they can be positioned in lower shelf but in that case it s important to make sure another high-powered PLIM is inserted above it in upper shelf. upper shelf lower shelf

Key Deployment Takeaways Most majority of the ISM and CGSE deployments are done for NAT44 6rd Some new customers or customers with internal IPv4 shortage issues are now looking at DS-Lite (and MAP) MAP is interesting (stateless in the router / inline performance at 240G per card) but not much CPE yet DS-Lite is stateful (implies logging) but CPEs are very common Many customers are testing NAT64 but some applications are not supported at all on IPv6 (ex: Skype) Logging both syslog and netflow are used Some customers using both simultaneously Mobile are usually using far less ports (true for handheld, not for dongles)

Monitoring Options Prime Performance Manager supports CGSE/ISM NAT44/NAT64 monitoring Active Translation / Creating Rate I2O and O2I Forward Rate I2O Drop Port Limit Exceeded I2O Drop System Limit Reached Pool address totally free / used Expect scripts can be used to collect counters from show commands More scripts can be used to figure out the port user port usage (very important to figure out the proper port-limit) First, Get all IP outside addresses in use with a sh cgn nat44 NAT statistics Then, for each IP address, run a sh cgn nat44 NAT outside-translation proto $Prot outside-address $IP port start 1 end 65535 with $Prot: TCP/UDP/ICMP Logs can be used to spot customers exceeding the limits

Scripts Script will collect info on used ports per external address RP/0/RSP0/CPU0:R1#sh cgn nat44 nat44 pool-utilization inside-vrf IN address-r$ Public address pool utilization details ----------------------------------------------- NAT44 instance : nat44 VRF : IN ----------------------------------------------- Outside Number Number Address of of Free ports Used ports ----------------------------------------------- 1.2.3.0 65528 7 1.2.3.4 65525 10 1.2.3.8 65529 6 1.2.3.12 65533 2 1.2.3.16 65517 18 1.2.3.20 65535 0 1.2.3.24 65534 1 1.2.3.28 65469 66 1.2.3.32 65522 13 1.2.3.36 65530 5 1.2.3.40 65529 6 1.2.3.48 65533 2 1.2.3.52 65534 1

Scripts RP/0/RP0/CPU0:R1#sh cgn nat44 NAT-1 outside-translation protocol tcp outside-address 196.219.0.3 port start 1 end 65535 -------------------------------------------------------------------------------------------- Inside Protocol Inside Outside Translation Inside Outside Address Source Source Type to to Port Port Outside Inside Packets Packets -------------------------------------------------------------------------------------------- 10.193.114.195 tcp 1114 46599 dynamic 110 129 10.193.114.195 tcp 1525 59248 dynamic 26 26 10.193.208.195 tcp 1691 54882 dynamic 6 4 10.193.114.195 tcp 1845 46393 dynamic 6 6 10.193.169.131 tcp 1980 63344 dynamic 12 21 10.193.248.131 tcp 2581 51821 dynamic 25 29 10.193.254.67 tcp 2873 1469 dynamic 12 15 10.193.117.67 tcp 2958 50417 dynamic 12 11 10.193.24.131 tcp 3016 50279 dynamic 8 8 10.193.247.3 tcp 3248 32869 dynamic 27 32 10.193.114.195 tcp 3479 58883 dynamic 29 28 10.193.114.195 tcp 3664 49916 dynamic 6 6

Scripts 10.193.114.195 10.193.114.195 10.193.208.195 10.193.114.195 10.193.169.131 10.193.248.131 10.193.117.67 10.193.24.131 10.193.247.3 10.193.114.195 10.193.114.195 Sort 10.193.24.131 10.193.114.195 10.193.114.195 10.193.114.195 10.193.114.195 10.193.114.195 10.193.117.67 10.193.169.131 10.193.208.195 10.193.247.3 10.193.248.131 46599 59248 54882 46393 63344 51821 1469 50417 50279 32869 58883 Divide by BPA and round down 45 57 53 45 61 50 1 49 49 32 57 Count 1 1 32 1 45 2 49 2 50 1 53 1 57 2 61 1 10.193.24.131 1 10.193.114.195 5 10.193.117.67 1 10.193.169.131 1 10.193.208.195 1 10.193.247.3 1 10.193.248.131 1 Per user port usage - Top X users - Average - Port-limit tweaking For a BPA=1024 - Number of ports used per block ID - Top X blocks - Average usage - BPA tweaking

Sizing the Port-Limit and BPA No rule of thumb to define port-limit, BPA, timers Example for a broadband ISP in LATAM (using a script) 18 ports average per user Can not be used to determine the best port-limit i2o 50kpps per card o2i 70kpps per card Avg i2o packet size: 200B Avg o2i packet size:1200b percentage of users using less than X ports (starts at 99.8%)

Impact of CGN on Applications? Several customers have been testing extensively the most popular applications and successfully, for example: TFTP, SSH, Telnet IPSec VPN (Cisco Client), SSL VPN (AnyConnect Client) HTTP/HTTPS on popular sites (CNN, Facebook, Youtube, Google services, ) WebMail (Java) Skype, SkypeUpdate, Audio/Video/FileTransfer/Chat MSN Bit Torrent Netflix Video web sites like Crunchyroll.com, ign.com itunes store browsing, upgrade, Sony Media Go Store Steam Install and Update StarCraft 2, World of Warcraft, MineCraft,

Application Impacted by CGN A vast majority of users only need their internet connection for Web surfing Emails Skype Mobile Phone Apps on Wifi Occasionally p2p download These customers will never realize they are NATed Per complaint behavior: When customers are complaining about their connection (latency, applications not working mainly for hardcore gamers who need to be a node for multiplayer games), the ISP move them into a different VRF which is not NATed

Service Impacted by CGN Geo-localization services IP tracking services (advertisement system, not based on cookies)

ROUTING CONSIDERATION AND BEST PRACTICES

Types of Routing Two types of routing should be differentiated Intra-chassis routing Packets candidate for translation or tunnel encapsulation/decapsulation, when received on the router, should be forwarded to and from the CGN card Static routes and Access-List Based Forwarding will be use Extra-chassis routing Packets should also be attracted by the CGN system able to handle them properly Dynamic routing protocols (BGP or IGP) will be used to advertise the prefix

CGN Routing IPv4 Backbone Te0/0/0/0 ServiceApp1 inside VRF CGN Card ServiceApp2 outside VRF Te0/1/0/0 IPv4 Internet IGP Static Static Intra-Chassis Extra-Chassis ABF IGP/BGP

Intra-Chassis Routing Aimed at forwarding packets candidate for translation or tunnel encapsulation/decapsulation, to and from the CGN card For i2o traffic, two methods available Based on destination: static routes to the serviceapp interface in the global table to the serviceapp in the global table to the serviceapp in a named VRF in a named VRF table to the serviceapp should be advertised in IGP and/or ibgp Based on source or destination: Access-list Based Forwarding applied in ingress on the interface, could be VRF-aware or not For o2i traffic usually, we will rely on static routes to advertised a route back to the map pool range into the outside serviceapp should be advertised in external IGP or BGP

Extra-Chassis Routing It s necessary to attract traffic to the CGNAT device and determine which traffic is actually candidate to translation Asymmetrical traffic is not possible with CGNAT routing, o2i must follow the path of the i2o traffic That s why it s mandatory to advertise the map pool ranges to the external world to guarantee the symmetry Some example: Default Map pool Access BNG NAT CGN Core Public IP Internet

Extra-Chassis Routing A few other examples Core Private IP Default Full Table NAT Map pool CGN Peering Network Aggregate Internet L3VPN VRF Default Map pool Full Table NAT CGN Internet

NAT44 Static Route Configuration Create one static route in each VRF (inside and outside) All packets arriving in vrf inside should be directed to the CGN card through the serviceapp1 interface All packets arriving in vrf outside and targeted to addresses in the map pool range should be directed to the serviceapp2 interface RP/0/RSP0/CPU0:router(config)# router static vrf inside address-family ipv4 unicast 0.0.0.0/0 ServiceApp1! vrf outside address-family ipv4 unicast 100.0.0.0/24 ServiceApp2! Te0/0/0/0 ServiceApp1 inside VRF Translate to 100.0.0.0/24 CGN Card ServiceApp2 outside VRF Te0/1/0/0

NAT44 Static Route Configuration In many situations, physical interfaces can not be in a inside VRF but must be in the global routing table We could simply use a static default in the global ipv4 table pointing to serviceapp in the inside VRF, but a global default route is not recommended: ALL traffic with no route in the RIB will be attracted if the router has a full BGP table, no packets will be routed to serviceapp1 Translate to 100.0.0.0/24 RP/0/RSP0/CPU0:router(config)# router static address-family ipv4 unicast 0.0.0.0/0 vrf inside ServiceApp1! Te0/0/0/0 ServiceApp1 inside VRF CGN Card ServiceApp2 outside VRF Te0/1/0/0

NAT44 ABF Configuration Routing based on ACL enables decision based on source addresses Public sources can avoid NAT // Private can be sent for NAT translation RP/0/RSP0/CPU0:router(config)# ipv4 access-list ABF 10 permit ipv4 10.0.0.0 0.255.255.255 any nexthop1 vrf inside ipv4 1.1.1.2 20 permit ipv4 any any interface ServiceApp1 vrf inside ipv4 address 1.1.1.1/30 service cgn demo service-type nat44! interface TenGigE0/0/0/0 ipv4 address 20.1.1.1/24 ipv4 access-group ABF ingress 10.0.0.0/8 Te0/0/0/0 ServiceApp1 inside VRF Translate to 100.0.0.0/24 CGN Card ServiceApp2 outside VRF interface ServiceApp2 vrf outside ipv4 address 2.1.1.1/30 service cgn demo service-type nat44! interface TenGigE0/1/0/0 ipv4 address 30.1.1.1/24 Te0/1/0/0 30.0.0.0/8

NAT44 ABF Configuration Return traffic When you configure ABF for the i2o traffic, you don t need to do it for the o2i traffic o2i traffic must be routed to the correct Inside (default) VRF when it comes out of the Inside Service App RP/0/RSP0/CPU0:router(config)# router static vrf inside address-family ipv4 unicast 10.0.0.0/8 vrf default 20.1.1.2 10.0.0.0/8 20.1.1.2 Te0/0/0/0 ServiceApp1 inside VRF Translate to 100.0.0.0/24 CGN Card ServiceApp2 outside VRF RP/0/RSP0/CPU0:router(config)# router static address-family ipv4 unicast 100.0.0.0/24 vrf outside serviceapp2 Te0/1/0/0 30.0.0.0/8

NAT44 ABF Limitations What if the next-hop address in GRT isn t reachable (interface down for example)? RP/0/RSP0/CPU0:router(config)# router static vrf inside address-family ipv4 unicast 10.0.0.0/8 vrf default 20.1.1.2 10.0.0.0/8 20.1.1.2 Te0/0/0/0 ServiceApp1 inside VRF Translate to 100.0.0.0/24 CGN Card ServiceApp2 outside VRF RP/0/RSP0/CPU0:router(config)# router static address-family ipv4 unicast 100.0.0.0/24 vrf outside serviceapp2 Te0/1/0/0 30.0.0.0/8 Even if another path is available to reach 10.0.0.0/8 in the GRT, traffic is lost

NAT44 ABF Limitations What if the next-hop router points to the CGN router to reach 10.0.0.0/8? RP/0/RSP0/CPU0:router(config)# router static vrf inside address-family ipv4 unicast 10.0.0.0/8 vrf default 20.1.1.2 10.0.0.0/8 20.1.1.2 Te0/0/0/0 ServiceApp1 inside VRF Translate to 100.0.0.0/24 CGN Card ServiceApp2 outside VRF RP/0/RSP0/CPU0:router(config)# router static address-family ipv4 unicast 100.0.0.0/24 vrf outside serviceapp2 Te0/1/0/0 30.0.0.0/8 In this case, the traffic will eventually find it s way to 10.0.0.0/8 but via a suboptimal path

NAT44 ABF Limitations ABF is performed before MPLS labels are stripped from packets Consequently, packets are not matched Example, the CGN in PE case Workaround: loop fiber P Te0/6/0/2 0/0/CPU0 VRF Inside-1 0/1/CPU0 VRF Inside-2 SA1 SA3 Translate to 151.0.0.0/24 CGN Card Translate to 151.0.1.0/24 CGN Card SA2 251.5 250.5 SA4 51.5 52.5 Global Global 2 Labels Transport VRF 1 Label VRF PE

NAT44 ABF Limitations Other example, the CSC case (CGN in CE) P PE Te0/6/0/2 0/0/CPU0 VRF Inside-1 0/1/CPU0 VRF Inside-2 SA1 SA3 Translate to 151.0.0.0/24 CGN Card Translate to 151.0.1.0/24 CGN Card SA2 251.5 250.5 SA4 51.5 52.5 Global Global 3 Labels Transport VRF CSC 2 Labels VRF CSC 1 Label CSC CE

REDUNDANCY

CGSE/ISM Redundancy On both CRS/CGSE and ASR9000/ISM, we support 1:1 warm standby redundancy (not supported on CGSE+ today) Warm-standby translation state is not synchronized between active and standby, all connections will be re-established Pros: simple to configure, a single map pool is used Cons: only 1:1, one card on two will not be used 99% of the time An alternative with ABF is available Pros: offers more options like n:1 redundancy, converges very quickly Cons: we can not re-use the same map pool range, so we need to configure a second range

1:1 Warm-Standby Redundancy Configuration RP/0/RSP0/CPU0:CGN(config)# service cgn demo service-location preferred-active 0/1/CPU0 preferred-standby 0/3/CPU0 RP/0/RP0/CPU0:CGN#show services redundancy Service type Name Pref. Active Pref. Standby -------------------------------------------------------------------------------- ServiceInfra ServiceInfra1 0/1/CPU0 Active ServiceInfra ServiceInfra2 0/3/CPU0 Active ServiceCgn demo 0/3/CPU0 Standby 0/1/CPU0 Active RP/0/RP0/CPU0:CGN#

CGSE/ISM Redundancy service cgn mets-cgn service-location preferred-active 0/1/CPU0 service-type nat44 nat44-1 inside-vrf Inside-1 map address-pool 151.0.0.0/24! service cgn mets-cgn-2 service-location preferred-active 0/3/CPU0 service-type nat44 nat44-2 inside-vrf Inside-2 map address-pool 151.0.1.0/24! service cgn mets-cgn-backup service-location preferred-active 0/7/CPU0 service-type nat44 nat44-backup inside-vrf ibackup map address-pool 151.0.2.0/24 Te0/6/0/2 10.1.1.1/24 VRF Inside-1 VRF Inside-2 VRF ibackup SA1 SA3 Translate to 151.0.0.0/24 CGN Card Translate to 151.0.1.0/24 CGN Card SA2 251.5 250.5 CGN Card SA4 51.5 52.5 SA5 Translate to 151.0.2.0/24 SA6 53.5 54.5 Global Global Global Te0/6/0/3 100.1.1.1/24

CGSE/ISM n:1 Redundancy ipv4 access-list ABF 10 permit ipv4 10.2.0.0/24 any nexthop1 vrf Inside-1 ipv4 192.168.251.6 nexthop2 vrf ibackup ipv4 192.168.53.6 20 permit ipv4 10.2.1.0/24 any nexthop1 vrf Inside-2 ipv4 192.168.51.6 nexthop2 vrf ibackup ipv4 192.168.53.6 100 permit ipv4 any any! router static address-family ipv4 unicast 110.1.0.0/16 100.1.1.2 description Ixia-i2o-Default 151.0.0.0/24 ServiceApp2 description Ixia-o2i-ABF 151.0.1.0/24 ServiceApp4 description Ixia-o2i-ABF 151.0.2.0/24 ServiceApp6 description Ixia-o2i-ABF VRF Inside-1 SA1 Translate to 151.0.0.0/24 CGN Card SA2 251.5 250.5 VRF Outside-1 Packets sourced from 150.0.0.x 10.2.0.0/24 Te0/6/0/2 10.1.1.1/24 VRF Inside-2 SA3 Translate to 151.0.1.0/24 CGN Card SA4 51.5 52.5 VRF Outside-2 Te0/6/0/3 100.1.1.1/24 110.1.0.0/16 10.2.1.0/24 VRF ibackup SA5 Translate to 151.0.2.0/24 CGN Card SA6 53.5 54.5 Default

CGSE/ISM n:1 Redundancy ipv4 access-list ABF 10 permit ipv4 10.2.0.0/24 any nexthop1 vrf Inside-1 ipv4 192.168.251.6 nexthop2 vrf ibackup ipv4 192.168.53.6 20 permit ipv4 10.2.1.0/24 any nexthop1 vrf Inside-2 ipv4 192.168.51.6 nexthop2 vrf ibackup ipv4 192.168.53.6 100 permit ipv4 any any! router static address-family ipv4 unicast 110.1.0.0/16 100.1.1.2 description Ixia-i2o-Default 151.0.0.0/24 ServiceApp2 description Ixia-o2i-ABF 151.0.1.0/24 ServiceApp4 description Ixia-o2i-ABF 151.0.2.0/24 ServiceApp6 description Ixia-o2i-ABF VRF Inside-1 SA1 Translate to 151.0.0.0/24 CGN Card SA2 251.5 250.5 VRF Outside-1 10.2.0.0/24 Te0/6/0/2 10.1.1.1/24 VRF Inside-2 SA3 Translate to 151.0.1.0/24 CGN Card SA4 51.5 52.5 VRF Outside-2 Te0/6/0/3 100.1.1.1/24 110.1.0.0/16 10.2.1.0/24 VRF ibackup SA5 Translate to 151.0.2.0/24 CGN Card SA6 53.5 54.5 Default Packets sourced from 150.0.2.x

CGSE/ISM n:1 Redundancy Limitations 1:1 warm standby redundancy N:1 ABF based redundancy Convergence time Up to 7s <1s CAPEX Impact on other resources (address map pools) Preemption when the first card gets back online Static port forwarding Needs a standby card for every active one No map pool necessary for the backup card No preemption, the new active card stays active No problem, the standby re-populates the table with the static entry Needs only a single backup card per router No map pool necessary for the backup card The initial active card regains the active role and create a 2 nd impact Since the backup card uses a different map pool, a new static entry will be created 111

Extra-Chassis Redundancy ipv4 access-list ABF-1 10 permit ipv4 any any nexthop1 vrf Inside-1 ipv4 192.168.251.6 nexthop2 vrf Inside-2 ipv4 192.168.51.6 nexthop3 ipv4 10.10.1.1 Te0/6/0/2 10.1.1.1/24 0/0/CPU0 VRF Inside-1 0/1/CPU0 VRF Inside-2 SA1 SA3 Translate to 151.0.0.0/24 CGN Card Translate to 151.0.1.0/24 CGN Card SA2 251.5 250.5 SA4 51.5 52.5 Global Global Te0/6/0/3 100.1.1.1/24 If routers are not directly connected, a GRE tunnel can be used to avoid routing loops Te0/0/0/0 10.10.1.1/24 0/0/CPU0 VRF ibackup SA5 Translate to 151.0.2.0/24 CGN Card SA6 53.5 54.5 Global Te0/0/0/1 100.1.2.1/24

Logging Redundancy CGN cards are generating syslog and NFv9 on UDP No mean to send backpressure if the server can t cope One single destination per type and inside-vrf Workarounds exist at the collector level: Virtual IP addresses on the collector Port SPAN on the switch were is connected the collector to replicate the logging flow (second server needs some tweaking to accept the trafffic) Directed-Broadcast on the last router (ex: the last interface is 10.100.1.1/30 and we will generate the logging traffic to 10.100.1.4, the broadcast address of this network. Only 10.100.1.0/24 will be advertised in IGP) RAID / DB redundancy is highly recommended at the server level

CONCLUSION

Conclusion CGN offers tools to buy time for your IPv6 preparation The same line cards can also be used for IPv6 migration (NAT64, 6rd, DS-lite) For the vast majority of usages: it just works Deployment must be considered carefully for Routing Logging infrastructure for collection and storage Timers, BPA, Port-Limit,

Complete Your Online Session Evaluation Complete your online session evaluation Complete four session evaluations and the overall conference evaluation to receive your Cisco Live T-shirt 116

BACKUP SLIDES UNDERSTANDING TIMERS

Stateful Protocols Understanding the Stateful Translation NAT44 (like NAT64SF and DS Lite) performs a stateful translation Packet source address and port are rewritten Details are stored in a translation database A new packet from inside to outside will create a new entry in the table No activity during a configurable period of time will trigger the suppression of this entry We use different timers for different packet types and different situations

NAT44: TCP Establishment Source Address = 10.1.1.1 1 5 Src: 10.1.1.1:12345 Dst: 30.0.0.1:80 ACK SYN 0 2 4 IPv4 Traffic NAT CGN Inside Outside TCP state Inside Outside TCP state 10.1.1.1:12345 99.0.0.1:1025 Inactive Inside Outside TCP state 10.1.1.1:12345 99.0.0.1:1025 Active Outside Address from pool = 99.0.0.1 Src: 99.0.0.1:1025 Dst: 30.0.0.1:80 IPv4 Internet 30.0.0.1 SYN/ACK 3 Now, as long as TCP traffic is received in any direction within the active timer, state is maintained as Active. This behavior can be changed by configuration, considering only the i2o traffic to refresh the timers.

NAT44: End of TCP Session Source Address = 10.1.1.1 Src: 10.1.1.1:12345 Dst: 30.0.0.1:80 0 2 IPv4 Traffic NAT CGN Inside Outside TCP state 10.1.1.1:12345 99.0.0.1:1025 Active Inside Outside TCP state Outside Address from pool = 99.0.0.1 Src: 99.0.0.1:1025 Dst: 30.0.0.1:80 IPv4 Internet FIN or RST 30.0.0.1 1 3 Default timers: TCP init: 120s ACK 4 10.1.1.1:12345 99.0.0.1:1025 Inactive Inside Outside TCP state 3 Initial timer expires DB is cleaned up 10.1.1.1:12345 99.0.0.1:1025 Inactive Note: We are not checking the sequence numbers in the NAT engine.

NAT44: TCP Initial Timeout Source Address = 10.1.1.1 1 Src: 10.1.1.1:12345 Dst: 30.0.0.1:80 SYN 0 2 IPv4 Traffic NAT CGN Inside Outside TCP state Inside Outside TCP state Outside Address from pool = 99.0.0.1 Src: 99.0.0.1:1025 Dst: 30.0.0.1:80 IPv4 Internet 30.0.0.1 10.1.1.1:12345 99.0.0.1:1025 Inactive Default timers: TCP init: 120s 4 Inside Outside TCP state 10.1.1.1:12345 99.0.0.1:1025 Inactive 3 Initial timer expires DB is cleaned up Note: we are checking all timers every 10ms to clean up the time-outs

NAT44: TCP Active Timeout Source Address = 10.1.1.1 Src: 10.1.1.1:12345 Dst: 30.0.0.1:80 0 IPv4 Traffic NAT CGN Inside Outside TCP state Outside Address from pool = 99.0.0.1 Src: 99.0.0.1:1025 Dst: 30.0.0.1:80 IPv4 Internet 30.0.0.1 10.1.1.1:12345 99.0.0.1:1025 Active No traffic matching the DB entry flows through the system Default timers: TCP active: 1800s 2 Inside Outside TCP state 10.1.1.1:12345 99.0.0.1:1025 Inactive 1 Initial timer expires DB is cleaned up Note: We are not sending any FIN/RST to either side (inside nor outside), the translation entry is simply removed from the table.

NAT44: Security Behavior Source Address = 10.1.1.1 Src: 10.1.1.1:12345 Dst: 30.0.0.1:80 0 IPv4 Traffic NAT CGN Inside Outside TCP state Outside Address from pool = 99.0.0.1 Src: 99.0.0.1:1025 Dst: 30.0.0.1:80 IPv4 Internet 30.0.0.1 If we send TCP data packet before a complete TCP handshake 1 TCP Data 2 this packet is considered invalid and dropped without ICMP being generated.

NAT44: Security Behavior Source Address = 10.1.1.1 1 Src: 10.1.1.1:12345 Dst: 30.0.0.1:80 SYN 0 2 IPv4 Traffic NAT CGN Inside Outside TCP state Inside Outside TCP state Outside Address from pool = 99.0.0.1 Src: 99.0.0.1:1025 Dst: 30.0.0.1:80 IPv4 Internet 30.0.0.1 10.1.1.1:12345 99.0.0.1:1025 Inactive If we receive a TCP data packet before a complete TCP handshake 4 Inside Outside TCP state 10.1.1.1:12345 99.0.0.1:1025 Inactive TCP Data 3 this packet is translated back and passed to the host, but table state isn t changed from Inactive to Active. It stays at Inactive.

NAT44: UDP Packets Source Address = 10.1.1.1 Src: 10.1.1.1:12345 Dst: 30.0.0.1:80 0 IPv4 Traffic NAT CGN Inside Outside UDP state Outside Address from pool = 99.0.0.1 Src: 99.0.0.1:1025 Dst: 30.0.0.1:80 IPv4 Internet 30.0.0.1 1 UDP 2 Inside Outside UDP state 10.1.1.1:12345 99.0.0.1:1025 Inactive UDP 3 4 Inside Outside UDP state 10.1.1.1:12345 99.0.0.1:1025 Active Now, as long as UDP traffic is received in any direction within the active timer, state is maintained as Active.

NAT44: UDP Timeout Case 1 Source Address = 10.1.1.1 0 Src: 10.1.1.1:12345 Dst: 30.0.0.1:80 UDP 0 IPv4 Traffic NAT CGN Inside Outside UDP state 10.1.1.1:12345 99.0.0.1:1025 Inactive Outside Address from pool = 99.0.0.1 Src: 99.0.0.1:1025 Dst: 30.0.0.1:80 Only I2O traffic passes through CGN, UDP state is Inactive 1 Now, no more I2O UDP traffic is received IPv4 Internet 30.0.0.1 Default timers: UDP init: 30s 4 Inside Outside UDP state 2 Initial timer expires DB is cleaned up 10.1.1.1:12345 99.0.0.1:1025 Inactive

NAT44: UDP Timeout Case 2 Source Address = 10.1.1.1 0 Src: 10.1.1.1:12345 Dst: 30.0.0.1:80 UDP 0 IPv4 Traffic NAT CGN Inside Outside UDP state 10.1.1.1:12345 99.0.0.1:1025 Active Outside Address from pool = 99.0.0.1 Src: 99.0.0.1:1025 Dst: 30.0.0.1:80 IPv4 Internet 30.0.0.1 UDP 0 1 Now, both I2O and O2I UDP stop flowing through the CGN Default timers: UDP active: 120s 4 Inside Outside UDP state 2 Initial timer expires DB is cleaned up 10.1.1.1:12345 99.0.0.1:1025 Active

NAT44: ICMP Source Address = 10.1.1.1 Src: 10.1.1.1 Dst: 30.0.0.1 IPv4 Traffic 0 NAT CGN NAT Info Outside Address from pool = 99.0.0.1 Src: 99.0.0.1 Dst: 30.0.0.1 IPv4 Internet 30.0.0.1 1 ICMP No state in ICMP translation 2 NAT Info Only a DB entry. 10.1.1.1 99.0.0.1 ICMP ICMP 3 Now, as long as ICMP traffic is received in any direction within the timer, this entry will be maintained in the DB.

NAT44: ICMP Timeout Case Source Address = 10.1.1.1 1 Src: 10.1.1.1 Dst: 30.0.0.1 ICMP IPv4 Traffic 2 0 NAT CGN NAT Info NAT Info Outside Address from pool = 99.0.0.1 Src: 99.0.0.1 Dst: 30.0.0.1 IPv4 Internet 30.0.0.1 10.1.1.1 99.0.0.1 ICMP Now, no more I2O and O2I ICMP flow through the CGN Default timers: ICMP: 60s 4 NAT Info 3 ICMP timer expires DB is cleaned up 10.1.1.1 99.0.0.1 ICMP

Fine Tuning Timers For stateful translation protocols (NAT44, NAT64 SF, DS Lite), the NAT DB maintains timers for each entry service cgn demo service-type nat44 nat44-1 protocol udp session initial timeout 10 session active timeout 30 protocol tcp session initial timeout 30 session active timeout 120 protocol icmp timeout 30 service cgn demo service-type nat64 stateful nat64-1 protocol udp timeout 30 v4-init-timeout 10 protocol tcp session initial timeout 30 session active timeout 120 protocol icmp timeout 30 service cgn demo service-type ds-lite ds-lite1 protocol udp session active timeout 30 session init timeout 10 protocol tcp session active timeout 120 session init timeout 30 protocol icmp timeout 30 Default Initial Active TCP 120s 1800s UDP 30s 120s ICMP 60s

Refresh Direction Timers are refreshed when packets are translated in i2o or o2i direction. But an external attacker could send regularly one packet for every DB entry and eventually create a resource depletion To change this default behavior, we can make the timer refresh to only take into consideration Inside-to-Outside (i2o) packets This feature is not available for DS Lite service cgn POC-1 service-type nat44 nat44-1 refresh-direction Outbound! service-type nat64 stateful nat64-1 refresh-direction Outbound!

BACKUP SLIDES LOAD BALANCING

Load-balancing Traffic Between CGSEs SPA SPA SPA SPA SPA SPA BRIDGE BRIDGE BRIDGE BRIDGE PLA PLA PLA PLA M I D P L A N E ipse Egress Q ipse Egress Q ipse Egress Q epse epse epse IngressQ FabQs IngressQ FabQs IngressQ FabQs M I D P L A N E F A B R I C At egress PSE level: Hashing on source address to loadbalance traffic between 64 cores At ingress PSE level: Two static routes for one NH address pointing to two serviceapps interfaces (L3 or L4 LB is used depending on the configuration) ABF is possible too and is a better option. Note: using static routes will break the principle of same external IP address mapping for all sessions associated with the same internal IP address (RFC4787) we recommend ACL Based Forwarding.

DDR 16GB DDR 16GB SPA SPA SPA SPA SPA SPA Netlogic NPU Netlogic NPU BRIDGE BRIDGE PLA PLA PLA PLA M I D P L A N E ips E Egress Q ips E Egress Q ips E Egress Q epse epse epse IngressQ FabQs IngressQ FabQs IngressQ FabQs M I D P L A N E F A B R I C RP/0/RP0/CPU0:router(config)# router static vrf inside address-family ipv4 unicast 0.0.0.0/0 ServiceApp11 192.168.11.2 0.0.0.0/0 ServiceApp21 192.168.21.2 0.0.0.0/0 ServiceApp21 192.168.21.3 0.0.0.0/0 ServiceApp21 192.168.21.4 0.0.0.0/0 ServiceApp21 192.168.21.5! vrf outside address-family ipv4 unicast 100.0.0.0/24 Translate ServiceApp12 to 100.0.0.0/24 100.1.0.0/16 ServiceApp22 ServiceApp11 192.168.11.1/2 4 inside VRF ServiceApp21 192.168.21.1/2 4 CGSE CGSE PLUS Translate to 100.1.0.0/16 outside VRF ServiceApp12 192.168.12.1/2 4 ServiceApp22 192.168.22.1/2 4

DDR 16GB DDR 16GB Netlogic NPU Netlogic NPU PLA ips E Egress Q epse IngressQ FabQs RP/0/RP0/CPU0:router(config)# + ACL definition here + ABF applied on ingress interface here! vrf outside address-family ipv4 unicast 100.0.0.0/24 ServiceApp12 100.1.0.0/16 ServiceApp22 SPA SPA SPA SPA SPA SPA BRIDGE BRIDGE PLA PLA PLA M I D P L A N E ips E Egress Q ips E Egress Q epse epse IngressQ FabQs IngressQ FabQs M I D P L A N E F A B R I C ServiceApp11 192.168.11.1/2 4 inside VRF ServiceApp21 192.168.21.1/2 4 Translate to 100.0.0.0/24 CGSE CGSE PLUS Translate to 100.1.0.0/16 outside VRF ServiceApp12 192.168.12.1/2 4 ServiceApp22 192.168.22.1/2 4

Load-balancing Traffic inside ISM 24Gb 24Gb Based on the number of cores, we can t allocate a range more specific than /30 (4 public addresses) Load-balancing is different on the ISM than CGSE: First, it s performed by the ingress NPU (Trident or Typhoon on in the ingress card) where lookup is performed and a VQI is assigned for the destination Each VQI is linked to a particular Niantic port, hence to a particular dispatcher process on a CPU. (2 CPUs, 2 dispatchers running on 2 different ports 4 options). Second, the dispatcher process will determine which CGv6 application process should be handle this packet: - i2o traffic: hash is performed on the source address 32 bits - o2i traffic: hash is performed on the destination address 32 bits For DS-Lite, hash will be done on the B4 ipv6 address for i2o traffic and on the destination ipv4 address for o2i traffic.

BACKUP SLIDES NAT CONFIGURATION

Virtual Service Interfaces Interconnecting CGSE/ISM card to the rest of the system Configuration is only needed on the router/xr side, addresses on the CGN/Linux side will be automatically created To direct traffic into the CGN card, we ll need one or several of these options: static routes redistribution ACL based forwarding rules ServiceInfra interface For CGN card management One per card mandatory ServiceApp interfaces To interconnect GRT address-family Physical Interface VLAN or VRF inside and outside to the CGN card ServiceApp1 VRF or address-family ServiceInfra1 CGN Card ServiceApp2 VRF or address-family Physical Interface VLAN

NAT44 Configuration To avoid routing loops, VRF are mandatory with NAT44 Inside VRF must be non-default Outside VRF is optional, we can use the Default or a named VRF RP/0/RSP0/CPU0:Router(config)# vrf inside address-family ipv4 unicast! vrf outside address-family ipv4 unicast! interface te0/0/0/0 vrf inside ipv4 add 10.1.1.1/24! interface te0/1/0/0 vrf outside ipv4 add 100.1.1.1/24! Te0/0/0/0 interface ServiceApp1 vrf inside ipv4 address 1.1.1.1 255.255.255.252 service cgn demo service-type nat44! interface ServiceApp2 vrf outside ipv4 address 2.1.1.1 255.255.255.252 service cgn demo service-type nat44 ServiceApp1 inside VRF CGN Card ServiceApp2 outside VRF Te0/1/0/0

NAT44 Configuration Create a nat44 instance nat1 and associate an outside pool (Public IPv4 addresses) to a given inside VRF A single nat44 instance can be created per CGN card Several mechanisms exist to push traffic in2out into ServiceApp1 A static route with the map pool range will be necessary to send out2in traffic to the CGN card via ServiceApp2 service cgn demo service-type nat44 nat1 inside-vrf inside map address-pool 100.0.0.0/24! Mapping to the default VRF in public side service cgn demo service-type nat44 nat1 inside-vrf inside map outside-vrf outside address-pool 100.0.0.0/24! Mapping to the VRF outside in public side ServiceApp1 Inside VRF Translate to 100.0.0.0/24 CGN Card ServiceApp2 outside VRF or Default

NAT44 Configuration Tips In current XR release, we can not configure two map pools under one VRF inside (coming in the near future) RP/0/RP0/CPU0:Router(config-cgn-invrf)#show Fri Jun 15 16:54:52.430 PDT service cgn demo service-type nat44 nat44-1 inside-vrf Inside-2 map address-pool 151.0.0.0/24! RP/0/RP0/CPU0:Router(config-cgn-invrf)#map address-pool 151.0.1.0/24 RP/0/RP0/CPU0:Router(config-cgn-invrf)#show Fri Jun 15 16:56:23.669 PDT service cgn demo service-type nat44 nat44-1 inside-vrf Inside-2 map address-pool 151.0.1.0/24! RP/0/RP0/CPU0:Router(config-cgn-invrf)#

NAT44 Configuration Tips To overcome this limit we can configure several inside VRFs: RP/0/RP0/CPU0:Router(config-cgn-invrf)#show Fri Jun 15 16:54:52.430 PDT service cgn demo service-type nat44 nat44-1 inside-vrf Inside-1 map address-pool 151.0.0.0/24! inside-vrf Inside-2 map address-pool 151.0.1.0/24! RP/0/RP0/CPU0:Router(config-cgn-invrf)# Challenge will now reside in directing the traffic to both inside VRF Total of all map pools can not be larger than 65535 addresses It doesn t need to be into a single /16 or contiguous ranges

NAT44 Show Commands RP/0/RP0/CPU0:Router#show cgn demo stat sum Statistics summary of NAT44 instance: demo' Number of active translations: 2250000 Number of sessions: 11500028 Translations create rate: 0 Translations delete rate: 0 Inside to outside forward rate: 12600 Outside to inside forward rate: 0 Inside to outside drops port limit exceeded: 0 Inside to outside drops system limit reached: 0 Inside to outside drops resorce depletion: 0 No translation entry drops: 0 PPTP active tunnels: 0 PPTP active channels: 0 PPTP ctrl message drops: 0 Number of subscribers: 0 Drops due to session db limit exceeded: 0 Pool address totally free: 25268 Pool address used: 7500 ------------------------------------------------- External Address Ports Used ------------------------------------------------- 160.0.0.8 300 160.0.0.36 300 160.0.0.52 300 Translation entries allocated in DB Additional flows inside these translations Rate in sessions per second Rate in packets per second Packets dropped because of port-limit for inside user is reached Packets discarded because we reached the limit of 20M sessions or 1M internal users Packets dropped because no public L4 Port could be allocated

NAT44 Show Commands RP/0/RP0/CPU0:Router#show cgn demo stat sum Statistics summary of NAT44 instance: demo' Number of active translations: 2250000 Number of sessions: 11500028 Translations create rate: 0 Translations delete rate: 0 Inside to outside forward rate: 12600 Outside to inside forward rate: 0 Inside to outside drops port limit exceeded: 0 Inside to outside drops system limit reached: 0 Inside to outside drops resorce depletion: 0 No translation entry drops: 0 PPTP active tunnels: 0 PPTP active channels: 0 PPTP ctrl message drops: 0 Number of subscribers: 0 Drops due to session db limit exceeded: 0 Pool address totally free: 25268 Pool address used: 7500 ------------------------------------------------- External Address Ports Used ------------------------------------------------- 160.0.0.8 300 160.0.0.36 300 160.0.0.52 300 out2in drops because of no entry in the translation DB PPTP/GRE sessions/tunnels info Private addresses having at least one active translation Packets dropped after exceeding the 20M sessions Addresses available in the pool Addresses used in the pool External addresses and ports allocated

NAT44 Show Commands Pool utilization statistics RP/0/RP0/CPU0:Router#show cgn demo pool-utilization inside-vrf Inside address-range 100.0.0.90 100.0.0.95 Public address pool utilization details ------------------------------------------------------- CGN instance : demo VRF : Inside ------------------------------------------------------- Outside Number Number Address of of Free ports Used ports ------------------------------------------------------- 100.0.0.90 64512 0 100.0.0.91 64512 0 100.0.0.92 63139 1373 100.0.0.93 63138 1374 100.0.0.94 64512 0 100.0.0.95 64512 0...

NAT44 Show Commands Translation statistics from an inside address perspective RP/0/RP0/CPU0:router#sh cgn demo inside-translation protocol tcp inside-vrf Inside inside-address 10.12.0.29 port start 1 end 65535 Inside-translation details --------------------------- CGN instance : demo Inside-VRF : Inside -------------------------------------------------------------------------------------------- Outside Protocol Inside Outside Translation Inside Outside Address Source Source Type to to Port Port Outside Inside Packets Packets -------------------------------------------------------------------------------------------- 100.0.0.93 tcp 1405 58529 dynamic 7 4 100.0.0.93 tcp 1406 34188 dynamic 7 4 100.0.0.93 tcp 1407 41851 dynamic 7 4 100.0.0.93 tcp 2156 38317 dynamic 7 4 100.0.0.93 tcp 2157 30504 dynamic 7 4 100.0.0.93 tcp 2158 40039 dynamic 7 4 100.0.0.93 tcp 2907 42745 dynamic 7 4...

NAT44 Show Commands Translation statistics from an outside address perspective RP/0/RP0/CPU0:router#sh cgn demo outside-translation protocol tcp outside-vrf Outside outside-address 100.0.0.93 port start 1024 end 65535 Outside-translation details --------------------------- CGN instance : demo Outside-VRF : Outside -------------------------------------------------------------------------------------------- Inside Protocol Outside Inside Translation Inside Outside Address Destination Destination Type to to Port Port Outside Inside Packets Packets -------------------------------------------------------------------------------------------- 10.12.0.221 tcp 1032 56742 dynamic 7 4 10.12.0.157 tcp 1033 43804 dynamic 7 4 10.12.0.157 tcp 1055 54299 dynamic 7 4 10.12.0.157 tcp 1206 41550 dynamic 7 4 10.12.0.157 tcp 1274 64801 dynamic 7 4 10.12.0.221 tcp 1306 10243 dynamic 7 4 10.12.0.221 tcp 1359 8738 dynamic 7 4...

BACKUP SLIDES CONFIGURATION AND TROUBLESHOOTING TIPS

Protecting ServiceInfra Interface w/ an ACL ServiceInfra interfaces are virtual tunnels between the router and the CGN card and are mandatory to boot and manage it Even if the prefix used for this card isn t supposed to be advertised outside of the router, it s recommended to configure a filter to protect it from potential DoS attack RP/0/RP0/CPU0:router(config)# ipv4 access-list ServiceInfraFilter 100 permit ipv4 host 1.1.1.1 any 101 permit ipv4 host 1.1.1.2 any! interface ServiceInfra1 ipv4 address 1.1.1.1 255.255.255.0 service-location 0/0/CPU0 ipv4 access-group ServiceInfraFilter egress!

Sending Logging Reports in a VRF ServiceInfra interfaces are part of the global routing table and they are the source interfaces for syslog or netflow messages. If the collector is located in the Inside VRF, it s not possible to send it any reports by default We need to use ABF to overcome this limitation interface GigabitEthernet0/3/1/0 vrf Inside ipv4 address 10.1.0.1 255.255.255.0! service cgn cgn1 service-location preferred-active 0/0/CPU0 preferred-standby 0/2/CPU0 service-type nat44 NAT44 inside-vrf Inside map address-pool 110.0.0.0/20 external-logging syslog server address 10.1.0.3 port 3000 session-logging

Sending Logging Reports in a VRF We define and apply an ABF on the serviceinfra interface ipv4 access-list acl1 10 permit udp 101.100.11.0/24 host 10.1.0.3 nexthop1 vrf Inside 20 permit ipv4 any any! interface ServiceInfra2 ipv4 address 101.100.11.1 255.255.255.0 service-location 0/2/CPU0 ipv4 access-group acl1 ingress!! router static vrf Inside address-family ipv4 unicast 0.0.0.0/0 ServiceApp1 10.1.0.3/32 GigabitEthernet0/3/1/0!!

Dynamic Port Range For stateful translation protocols, the dynamic translations start from 1024. We can change this starting value from 1 to 65535 service cgn POC-1 service-type nat44 nat44-1 dynamic-port-range start 2000!

ICMP Rate-Limiting We can define an ICMP rate-limiter for CGN card (ISM, CGSE) For CRS/CGSE: should be a multiple of 64, less than 65472 For ASR9K/ISM: should be a multiple of 8, less than 8184 It can be 0 (zero) service cgn ISM protocol icmp rate-limit 8184!!

Using these Features Creatively How to reduce the number of users per external address? A customer requested to limit the number of internal users allowed to used each external addresses of their map pool. Only for NAT44 (no dynamic-range config in DS-Lite) Step 1: define port-limit and bulk-port-range to the same value. Ex: 4096 ports: rounddown[(65535-1024)/4096]=15 potential inside addresses for each external address Ex: 2048 ports: rounddown[(65535-1024)/2048]=31 BPA=1024 63 BPA=512 126, Step 2: if we need to reduce the number of users to something smaller than 15, let define the dynamic-port-range to an higher value Ex: BPA/port-limit=4096, dynamic-range start=24575 rounddown[(65535-24575)/4096]=10

Changing Logging DSCP Marketing Not possible to change the DSCP marking of syslog or netflow packets generated by ISM or CGSE card. But a remarking can be done at the egress interface level with the proper QoS policy RP/0/RP1/CPU0:Yanks#show policy-map interface gig 0/6/3/0.2 GigabitEthernet0/6/3/0.2 direction input: Service Policy not installed GigabitEthernet0/6/3/0.2 output: NF Class NF Classification statistics (packets/bytes) (rate - kbps) Matched : 37991/53199036 838 Transmitted : 37991/53199036 838 Total Dropped : 0/0 0 Queueing statistics Queue ID : 23 Taildropped(packets/bytes) : 0/0 Class class-default Classification statistics (packets/bytes) (rate - kbps) Matched : 0/0 0 Transmitted : 0/0 0 Total Dropped : 0/0 0 Queueing statistics Queue ID : 23 High watermark (bytes)/(ms) : 0/0 Inst-queue-len (bytes)/(ms) : 0/0 Avg-queue-len (bytes)/(ms) : 0/0 Taildropped(packets/bytes) : 0/0 RP/0/RP1/CPU0:Yanks#sh run policy-map Wed Sep 5 03:46:20.324 PDT policy-map NF class NF set dscp cs5! class class-default! end-policy-map!

Changing Logging DSCP Marketing Syslog / CS5 NetFlow v9 / CS5

Troubleshooting Tips Makes sure the traffic is indeed pushed to and from the CGN cards Show interface serviceapp * is always expressed from the router perspective, so Pkts out: going into the CGN cards Pkts in: coming from the CGN cards into the router RP/0/RSP0/CPU0:Nets#sh int serviceapp * accounting ServiceApp1 Protocol Pkts In Chars In Pkts Out Chars Out IPV4_UNICAST 2810763 348534612 37102220124 37515911766210 ServiceApp2 Protocol Pkts In Chars In Pkts Out Chars Out IPV4_UNICAST 36742436201 37162233422198 0 0 RP/0/RSP0/CPU0:Nets#

Troubleshooting Tips We can use show interface serviceapp * accounting rates to get some trends on the traffics going through the system

Troubleshooting Tips When using ABF: configure hardware count in ABF in order to see ABF match statistics You should see Hits increase as ingress traffic is directed to ServiceApp NH interface TenGigE0/0/5/0 vrf LOOPBACK ipv4 address 12.1.7.10 255.255.255.0 load-interval 30 ipv4 access-group ABF ingress hardware-count! RP/0/RP0/CPU0:router#show access-lists ABF hardware ingress detail location 0/0/CPU0 ACL name: ABF Sequence Number: 10 Grant: permit Logging: OFF Per ace icmp: ON Next Hop Enable: ON VRF Table Id: 4096 Next-hop: 1.1.1.2 Default Next Hop: OFF Hits: 4063640803 Statistics pointer: 0x7ff5f Number of TCAM entries: 1

Troubleshooting Tips on ISM Be extra careful with the unix level commands, one is very useful though: RP/0/RSP0/CPU0:BNG#run attach 0/5/cpu0 Sat Dec 22 06:33:02.403 UTC attach: Starting session 1 to node 0/5/cpu0 # # # show_nat44_stats CORE-ID #SESSIONS(%UTIL) #USERS(%UTIL) ------------------------------------------------------------------------ 0 563100(19.6%) 1877(1.43%) 1 561000(19.5%) 1870(1.43%) 2 563400(19.6%) 1878(1.43%) 3 562500(19.6%) 1875(1.43%) 4 0(0.0%) 0(0.00%) 5 0(0.0%) 0(0.00%) 6 0(0.0%) 0(0.00%) 7 0(0.0%) 0(0.00%) ------------------------------------------------------------------------ Total Sessions: 2250000 Total users: 7500 Main DB size is 2875008 and User DB size is 131072 #exit RP/0/RSP0/CPU0:BNG#

Troubleshooting Tips on CGSE # show_nat44_stats CORE ID #SESSIONS(UTIL) #USERS(UTIL) ----------------------------------------------------------------- 0 40194(11.2%) 5109(31.18%) 1 40541(11.3%) 5085(31.04%) 2 44626(12.4%) 5143(31.39%) 3 42984(12.0%) 5121(31.26%) 4 44286(12.3%) 5171(31.56%) 5 43361(12.1%) 5154(31.46%) 6 43394(12.1%) 5048(30.81%) 7 39203(10.9%) 5124(31.27%) 8 43285(12.0%) 5122(31.26%) 9 44728(12.4%) 5091(31.07%) 10 41258(11.5%) 5128(31.30%) 11 43362(12.1%) 5108(31.18%) 12 44791(12.5%) 5218(31.85%) 13 44026(12.2%) 5147(31.41%) 14 41399(11.5%) 5146(31.41%) 15 45238(12.6%) 5148(31.42%) 16 45989(12.8%) 5087(31.05%) 17 42037(11.7%) 5068(30.93%) 18 40363(11.2%) 5125(31.28%) 19 39819(11.1%) 5136(31.35%) 20 44321(12.3%) 5133(31.33%) 21 40380(11.2%) 5159(31.49%) 22 44183(12.3%) 5137(31.35%) 23 43153(12.0%) 5164(31.52%) 24 44762(12.5%) 5098(31.12%) 25 44317(12.3%) 5092(31.08%) 26 45482(12.7%) 5153(31.45%) 27 38451(10.7%) 5127(31.29%) 28 40848(11.4%) 5149(31.43%) 29 44388(12.3%) 5116(31.23%) 30 42729(11.9%) 5120(31.25%) 31 41428(11.5%) 5081(31.01%) 32 43292(12.0%) 5028(30.69%) 33 40294(11.2%) 5077(30.99%) 34 40734(11.3%) 5066(30.92%) 35 43167(12.0%) 5083(31.02%) 36 43519(12.1%) 5110(31.19%) 37 42372(11.8%) 5116(31.23%) 38 44425(12.4%) 5035(30.73%) 39 42546(11.8%) 5063(30.90%) 40 40284(11.2%) 5072(30.96%) 41 42166(11.7%) 5068(30.93%) 42 40136(11.2%) 5110(31.19%) 43 44040(12.3%) 5084(31.03%) 44 38744(10.8%) 5115(31.22%) 45 37815(10.5%) 5078(30.99%) 46 42205(11.7%) 5075(30.98%) 47 42783(11.9%) 5068(30.93%) 48 40146(11.2%) 5105(31.16%) 49 40471(11.3%) 5080(31.01%) 50 40798(11.4%) 5107(31.17%) 51 44311(12.3%) 5110(31.19%) 52 40794(11.3%) 5119(31.24%) 53 40354(11.2%) 5136(31.35%) 54 41776(11.6%) 5016(30.62%) 55 42932(11.9%) 5115(31.22%) 56 43001(12.0%) 5022(30.65%) 57 40488(11.3%) 5026(30.68%) 58 41422(11.5%) 5072(30.96%) 59 39293(10.9%) 5064(30.91%) 60 43408(12.1%) 5044(30.79%) 61 44388(12.3%) 5083(31.02%) 62 40447(11.3%) 5100(31.13%) 63 42022(11.7%) 5073(30.96%)

Online Diagnostics Optionally, configure Diagnostics on the CGSE card If we use redundant cards, active being in 0/0/CPU0 RP/0/RP0/CPU0:CRS(config)# service-plim-ha location 0/0/CPU0 datapath-test service-plim-ha location 0/0/CPU0 core-to-core-test service-plim-ha location 0/0/CPU0 pci-test service-plim-ha location 0/0/CPU0 coredump-extraction service-plim-ha location 0/0/CPU0 linux-timeout 500 service-plim-ha location 0/0/CPU0 msc-timeout 500! An error detected will trigger the reload of the PLIM. If the card is in stand-alone (no redundancy), we add the configuration: RP/0/RP0/CPU0:CRS(admin-config)# hw-module reset auto disable location 0/0/CPU0!

Online Diagnostics Optionally, configure Diagnostics on the ISM card RP/0/RP0/CPU0:ASR9000(config)# service-cgv6-ha location 0/2/CPU0 puntpath-test service-cgv6-ha location 0/2/CPU0 datapath-test!

Performance / Scalability Per Blade Limits CGSE CGSE+ ISM VSM NAT44 instances supported 1 per card 1 per card 1 per card 1 (at FCS) DS Lite instances supported 64 per chassis N/A 64 per chassis Future 6rd instances supported 64 per chassis 64 per chassis? Future NAT64 instances supported 64 per chassis N/A? Future Number of service infra 1 1 1 1 Number of service app 890 (2000 per system) IP pool supported /16 to /26 (max 65535 addresses)? 244 (per system) 4096 /16 to /26 (max 65535 addresses) Future: longer prefix /16 to /30 (max 65535 addresses) /16 to /30 (max 65535 addresses) Max Static Port forwarding 2K tested 6K 6K 6K Max number of NAT users 1M 1M (2M) 1M 4M

Comparing the CGN Platforms Parameter CGSE CGSE+ ISM VSM Configuration CLIs Same Same Same Same Uses SVI Yes Yes Yes Yes Network Processor Yes (Metro) Yes (Pogo) No, handled by a dedicated process Packet distribution One level: NAT44 load-balancing on egress Metro One level: NAT44 load-balancing on egress Pogo Two levels a) by ingress LC using VQI b) NAT44 load-balancing within Dispatcher process Yes (Typhoon) Egress FIB Lookup On imetro On ipogo Within CGv6 App On ServiceApp placement Anywhere Anywhere Associated with Niantic port/vqi? Associated with NP ports / Niantic ports # of CGv6 instances 64 (4 octeons) 8 (2 Westmeres) 48 (in 2 logical groups) Stateless protocols (in CGN card) 6rd, NAT64SL 6rd, (NAT64SL future) 6rd, MAP-T/E Future: 6rd, MAP-T/E Inline support No No Yes for SL protocols Future

BACKUP SLIDES PPTP ALG DETAILS

PPTP ALG PNS Control Connection (TCP1723) PPTP NAT IPv4 Internet PAC Inside Call-ID Outgoing Call Reply Outgoing Call Request Inside Call-ID Outside Call-ID Outbound Call Translation DataBase Two tuples are mapped and an entry is created in the translation DB

PPTP ALG PNS Control Connection (TCP1723) PPTP NAT IPv4 Internet PAC Incoming Call Request Inside Call-ID Outside Call- ID Outside Call-ID Incoming Call Reply Inbound Call Translation DataBase Two tuples are mapped and an entry is created in the translation DB

PPTP ALG PNS Control Connection (TCP1723) PPTP NAT IPv4 Internet PAC Inside Call-ID Call Disconnect Notify Call Clear Request Outside Call-ID Disconnect Translation DataBase Depending on the side initiating the disconnection, the Inside-Call-ID or Outside-Call-ID tuple will be marked for deletion from the translation DB