Enterprise QoS. Tim Chung Google Corporate Netops Architecture Nanog 49 June 15th, 2010

Transcription

1 Enterprise QoS Tim Chung Google Corporate Netops Architecture Nanog 49 June 15th, 2010

2 Agenda Challenges Solu5ons Opera5ons Best Prac5ces

3 Note: This talk pertains to Google enterprise network only, not google.com

4 Challenges WAN connec5ons are bandwidth boflenecks QoS essen5al for latency and jifer sensi5ve real 5me apps Many applica5ons require different QoS level Applica5on classifica5on granularity necessary on WAN Links Interac5on with different MPLS provider requires different egress policies Mul5- Vendor network Diverse Hardware: 4 Queue vs 8 Queue systems QoS over IPSec poses interes5ng challenge Performance: Need line rate Tput QoS MF Classifica5on changes/update on weekly basis Sta5s5cal analysis for capacity planning

5 Classifica5on as Close to Edge as Possible

6 MF Classifica5on on LAN Switching To trust or not to trust Decide where and what you want to classify LAN switches typically use port based classifica5on Permit tcp any any eq 80 Permit tcp any eq 80 any Permit udp any any range Permit udp any range any Permit udp any range any range Port based classifica5on is not ideal but may be sufficient

7 QoS Classifica5on at the WAN Links

8 MF Classifica5on on WAN links Conges5on at the WAN links More Granular classifica5on needed to minimize misclassifica5on Port based classifica5on not sufficient Granular classifica5on prevents flows from gaming the system by ge_ng into a befer QoS class Precious WAN bandwidth is u5lized more effec5vely Classifica5on based on Layer 4 is imperfect!

9 MF Classifica5on Granularity term qos- App1 { from { source- address { /24; /25; /20; /22; /19; /18; /17; } des5na5on- address { /32; /32; /24; } protocol udp; source- port ; des5na5on- port ; } then { count CS4- app1; forwarding- class CS4; } }

10 Juniper Commit Scripts Goal: Manage and deploy one generic MF classifier on all edge plaiorm Challenges 4 Queue Systems - M7i/M10i 8 Queue Systems J/M120/MX/M7i/10i with CFEB- E Unique Rou5ng- instances on a subset of routers Unique Business logic/requirements Use commit script to modify forwarding- class and PLP to tailor it for 4 Queue vs 8 Queue plaiorms Use commit scripts to customize MF classifier for routers with rou5ng instances

11 Juniper Commit Scripts Use commit script to customize shaping rate based on Bandwidth statement, not Port speed Use commit scripts look for bandwidth statement, interface tag, plaiorm, and WAN provider and tailor policer/shaper Remove/modify policer/shaper if input values change Business logic encoded inside commit scripts

12 Policers (Cascading markdown) Traffic agreement with transit provider may require sop policers on your CE router E.g., If exceed 500 Mbps budget of CS4, then drop Place control back on customer (CE) side Use a sop policer to mark down excess traffic over 500 Mbps to a lower classs Account and graph bps of markdown

13 Policers (Cascading markdown) Example policer config set firewall family inet filter markdown interface- specific set firewall family inet filter markdown term count- CS3- pre from forwarding- class CS3 set firewall family inet filter markdown term count- CS3- pre then count count- CS3- pre set firewall family inet filter markdown term count- CS3- pre then next term set firewall family inet filter markdown term police- CS3 from forwarding- class CS3 set firewall family inet filter markdown term police- CS3 then policer markdown- CS3 set firewall family inet filter markdown term police- CS3 then next term set firewall family inet filter markdown term count- CS3 from forwarding- class CS3 set firewall family inet filter markdown term count- CS3 then count count- CS3- post set firewall family inet filter markdown term count- CS3 then accept Calculate BPS markdown (count- CS3- pre - count- CS3- post) * 8 = xxxx Mbits marked down by policer interval (sec)

14 Policers (Cascading markdown) set firewall policer markdown- CS3 filter- specific set firewall policer markdown- CS3 if- exceeding bandwidth- limit 500m set firewall policer markdown- CS3 if- exceeding burst- size- limit set firewall policer markdown- CS3 then loss- priority low set firewall policer markdown- CS3 then forwarding- class CS2 set firewall policer markdown- CS2 filter- specific set firewall policer markdown- CS2 if- exceeding bandwidth- limit 600m set firewall policer markdown- CS2 if- exceeding burst- size- limit set firewall policer markdown- CS2 then loss- priority high set firewall policer markdown- CS2 then forwarding- class CS1

15 Opera5onal aspects of QoS Requirements Need to effect QoS changes globally in 10 mins Need to account for traffic in each class Solu5on Use Capirca Meta Language to generate Juniper and Cisco syntax and push ACLs to all edge devices Use firewall counters to track byte and packet count Use SNMP to poll counters and graph them Counters provide alterna5ve look at u5liza5on graph compared to neilow exports

16 Capirca (hfp://code.google.com/p/capirca/) Built by Google Security team for ACL management Open sourced under Apache2 license in Sept 2009 Same tool used to generate QoS MF classifica5on ACL as security ACLs One policy file, generate many vendor specific ACLs ( Out of One, Many

17 ACL Genera5on Process (high level) Network Definition Policy File Service Definition cisco generator juniper generator iptables generator Generated ACL Filter

18 Network Defini5on File RFC1918 = /8 # non-public /12 # non-public /16 # non-public INTERNAL = RFC1918 LOOPBACK = /32 # loopback ::1/128 # ipv6 loopback NYC_OFFICE = /24 # new york office SFO_OFFICE = /24 # san francisco office CHI_OFFICE = /24 # chicago office OFFICES = NYC_OFFICE SFO_OFFICE CHI_OFFICE

19 Service Defini5on File WHOIS = 43/udp SSH = 22/tcp TELNET = 23/tcp SMTP = 25/tcp VOICE = /udp MAIL_SERVICES = SMTP ESMTP SMTP_SSL POP_SSL DNS = 53/tcp 53/udp

20 Example Policy File header { comment:: ACL for QoS classification" target:: juniper qosfilter not-interface-specific } term qos-voice-cs4 { comment:: Voice Traffic" source-address:: CORP_VOICE destination-address:: CORP_VOICE source-port:: VOICE destination-port:: VOICE protocol:: udp counter:: CS4-Voice qos:: CS4 policer:: markdown-cs4 action:: accept }

21 firewall { family inet { replace: /* ** ACL for QoS classification" */ filter qosfilter { /* ** Voice Traffic */ term qos-voice-cs4 { from { source-address { /16; } destination-address { /16; } protocol udp; source-port ; destination-port ; } then { count CS4-Voice; policer markdown-cs4; forwarding-class CS4; accept; } } Generated Vendor ACL

22 QoS over IPSec You will always have offices that need to IPSec back to headquarters due to circuit availability Use BGP as routing protocol to glue all offices together Juniper Make sure DSCP bit are written to inner header and outer header Juniper services PIC by default copies inner DSCP to outer Rewrite rule necessary on inside of SP interface This is slightly counterintuitive. Inside or outside is from the perspective of the PFE, not the services PIC MF Classifier necessary on inside interface of SP if you run BGP over IPSec to learn routes; otherwise, BGP gets dumped into BE. Cisco Use the pre-classify knob to classify flows before encryption

23 Best Prac5ces Need to know your traffic pafern How many Apps do you care to classify? Move applica5ons in higher QoS class to the top of the firewall filter Need to know your Plaiorm Juniper Flows matching the last term of a complex 100 term firewall filters with firewall counters enabled will not yield Line rate Tput at 64 byte frames Check to see if you have info cell drops Cisco Be aware of TCAM size limita5ons

24 Best Prac5ces - Con5nue Test everything! You need your own Qualifica5on Lab/Testbed You need to weigh in on cost vs performance Work with vendor on bugs/feature requests before going live in produc5on Monitor everything Snmp poll appropriate QoS MIBs Neilow Know your usage pafern/trend

25 References Google Capirca hfp://code.google.com/p/capirca/ Default Rou5ng Engine Protocol Queue Assignment hfp:// cos/default- rou5ng- engine- protocol- queue- assignments.html#id Understanding ACL on Catalyst 6500 hfp:// products_white_paper09186a00800c9470.shtml Configuring QoS for Virtual Private Networks hfp://