Understanding Latency in Software Defined Networks

Transcription

1 Understanding Latency in Software Defined Networks Junaid Khalid Keqiang He, Sourav Das, Aditya Akella Li Erran Li, Marina Thottan 1

2 Latency in SDN Mobility Centralized Controller Openflow msgs 1 flows Time taken to install 1 rules? Can Some be as XXX large msecs?? as 1 secs!!! 2

3 Do applications care about latency? Intra-DC Traffic Engineering Latency is critical to many applications MicroTE routes predictable traffic on time scales of 1-2s Route Limits updates the applicability can take as long of SDN as applications.5s Latency can undermine MicroTE s effectiveness Fast Failover Applications assume latency is low and constant Reroute the affected flows quickly in face of failures Longer update time increases congestion and drops Latency can inflate failover time by nearly 2s! 3

4 Factors contributing to Latency? Robust Control Software Design and Distributed Controllers Speed of Control Programs and Network Latency Not received much attention Latency in network switches 4

5 Our Work Two contributions: Systematic experiments to explore latencies in production switches Latency in network switches Design a framework to overcome the impact of latencies 5

6 Outline Motivation Elements of Latency Measurement Methodology Inbound Latency Outbound Latency Insertion Modification Deletion 6

7 Outline Motivation Elements of Latency Measurement Methodology Inbound Latency Outbound Latency Insertion Modification Deletion 7

8 PCI Elements of Latency 1. Inbound Latency Controller Switch Switch Fabric CPU board I1 Memory DMA Forwarding Engine Lookup I2 ASIC SDK CPU I3 No Match Hardware Tables OF Agent Inbound Latency I1: Send to ASIC SDK I2: Send to OF Agent I3: Send to Controller PHY PHY Packet 8

9 PCI Elements of Latency 1. Inbound Latency 2. Outbound Latency Controller O1 Switch Switch Fabric CPU board Memory Forwarding Engine DMA Lookup ASIC SDK CPU OF Agent O4 Hardware Tables O2 O3 Outbound Latency O1: Parse OF Msg O2: Software schedules the rule O3: Reordering of rules in table O4: Rule is updated in table PHY PHY 9

10 Outline Motivation Elements of Latency Measurement Methodology Inbound Latency Outbound Latency Insertion Modification Deletion 1

11 Latency Measurements - Setup POX Controller Pktgen Libpcap eth eth1 eth2 Control Channel Flows IN Flows OUT Openflow Switch Switch CPU RAM Flow table size Data Plane Vendor A 1 Ghz 1 GB *1Gbps + 4*4Gbps Vendor B 2 Ghz 2 GB 496 4*1Gbps +4*4Gbps 11

12 Outline Motivation Elements of Latency Measurement Methodology Inbound Latency Outbound Latency Insertion Modification Deletion 12

13 Inbound Latency Increases with flow arrival rate CPU Usage is higher for higher flow arrival rates Flow Arrival Rate (packets/sec) Low Power CPU Mean Delay per packet_in (msec) CPU Usage ( % ) vendor A switch 13

14 inbound delay (ms) inbound delay (ms) Inbound Latency Increases with interference from outbound msgs flow # (a) with flow_mod/pkt_out 3 25 Low Power 2CPU 15 1 Software Inefficiency flow # (b) w/o flow_mod/pkt_out vendor A switch 2 flows/sec 14

15 Outline Motivation Elements of Latency Measurement Methodology Inbound Latency Outbound Latency Insertion Modification Deletion 15

16 Outbound Latency Latency for three different flow_mod operations Insertion Modification Deletion Impact of key factors on these latencies Table occupancy Rule priority structure 16

17 Outline Motivation Elements of Latency Measurement Methodology Inbound Latency Outbound Latency Insertion Modification Deletion 17

18 insertion delay (ms) insertion delay (ms) Insertion Latency Priority Effects (a) Burst size 1, same priority (b) Burst size 1, increasing priority Vendor B switch

19 insertion delay (ms) insertion delay (ms) Insertion Latency Priority Effects (a) Burst size 1, same priority (b) Burst size 1, increasing priority Vendor B switch x TCAM 19

20 insertion delay (ms) insertion delay (ms) Insertion Latency Priority Effects (a) Burst size 1, same priority (b) Burst size 1, increasing priority Vendor B switch x 99 TCAM 2

21 insertion delay (ms) insertion delay (ms) Insertion Latency Priority Effects (a) Burst size 1, same priority (b) Burst size 1, increasing priority Vendor B switch x 11 TCAM 21

22 insertion delay (ms) insertion delay (ms) Insertion Latency Priority Effects (a) Burst size 1, same priority (b) Burst size 1, increasing priority Vendor B switch x TCAM 22

23 insertion delay (ms) insertion delay (ms) Insertion Latency Priority Effects (a) Burst size 8, same priority (b) Burst size 8, decreasing priority Vendor A switch

24 insertion delay (ms) insertion delay (ms) Insertion Latency Priority Effects (a) Burst size 8, same priority (b) Burst size 8, decreasing priority Vendor A switch x TCAM 24

25 insertion delay (ms) insertion delay (ms) Insertion Latency Priority Effects (a) Burst size 8, same priority (b) Burst size 8, decreasing priority Vendor A switch x TCAM 25

26 insertion delay (ms) insertion delay (ms) Insertion Latency Priority Effects (a) Burst size 8, same priority (b) Burst size 8, decreasing priority Vendor A switch x TCAM 26

27 insertion delay (ms) insertion delay (ms) Insertion Latency Priority Effects (a) Burst size 8, same priority (b) Burst size 8, decreasing priority Vendor A switch x TCAM 27

28 insertion delay (ms) insertion delay (ms) Insertion Latency Priority Effects (a) Burst size 8, same priority (b) Burst size 8, decreasing priority Vendor A switch x TCAM 28

29 avg completion time (ms) avg completion time (ms) Insertion Latency Table occupancy Effects Rule Priority Structure and Table Occupancy Affected by the types of rules in the table table 1 table 4 2 TCAM Organization and 18 # Of Slices Switch Software Overhead burst size (a) low priority rules into a table with high priority existing rules table 1 table burst size (b) high priority rules into a table with low priority existing rules Vendor B switch 29

30 Outline Motivation Elements of Latency Measurement Methodology Inbound Latency Outbound Latency Insertion Modification Deletion 3

31 modification dely (ms) modification delay (ms) Modification Latency Higher than Insertion latency for vendor B Not affected by rule priority but affected by table occupancy Poorly optimized software in vendor B (a) 1 rules in the table (b) 2 rules in table Vendor B switch 31

32 Outline Motivation Elements of Latency Measurement Methodology Inbound Latency Outbound Latency Insertion Modification Deletion 32

33 deletion delay (ms) deletion delay (ms) Deletion Latency Higher than Insertion latency for both vendor A and B Not affected by rule priority but affected by table occupancy (a) 1 rules in table (b) 2 rules in table Vendor B switch 33

34 deletion delay (ms) deletion delay (ms) Deletion Latency Higher than Insertion latency for both vendor A and B Not affected by rule priority but affected by table occupancy Deletion is causing TCAM 6 Reorganization (a) 1 rules in table (b) 2 rules in table Vendor A switch 34

35 Summary Latency in SDN is critical to many applications Assumption: Latency is small or constant Latency is high and variable Varies with Platforms, Type of operations, Rule priorities, Table occupancy, Concurrent operations Key Factors: TCAM Organization, Switch CPU and inefficient Software Implementation 35

36 Summary Latency in SDN is critical to many applications Assumption: Latency is small or constant Latency is high and variable Varies with Platforms, Type of operations, Rule priorities, Table occupancy, Concurrent operations Key Factors: TCAM Organization, Switch CPU and inefficient Software Implementation 36

37 Backup Slides 37

38 Impact of Concurrent CPU jobs Impacts insertion delay. E.g. Polling Statistics Polling stats impacts more when table occupancy is higher table occupancy table occupancy Low Power CPU delays concurrent jobs flow stats polling count Vendor B 38