Data Center Quantized Congestion Notification (QCN): Implementation and Evaluation on NetFPGA June 14th

Transcription

1 Data Center Quantized Congestion Notification (QCN): Implementation and Evaluation on NetFPGA 2010 June 14th Masato Yasuda (NEC Corporation) Abdul Kader Kabanni (Stanford University)

2 Background In data centers, there is a movement of Network Convergence The spec of Converged Enhanced Ethernet (CEE) has been discussed in IEEE standard committees Separate Network Converged Network Internet terminals storage Internet terminals storage LAN (TCP) SAN (Fibre Channel: FC) Converged DC transport Converged Enhanced Ethernet servers servers Page 2

3 Congestion Control Mechanism in CEE In CEE, it has Congestion Control Mechanism Non-TCP traffic (storage, media) can avoid network congestion TCP FC (for storage) UDP (for media) IP FCoE Converged Enhanced Ethernet (CEE) Congestion Notification (CN) The spec of Congestion Control Mechanism in CEE It is specified in IEEE802.1Qau in Data Center Bridging Task Group We proposed QCN (Quantized Congestion Notification) and finally accepted as a standard in March IP Support Congestion Control at Layer 2 Page 3

4 Significant restrictions/requirements for CN No per-packet packet ACKs Links can be paused Sources can start at line late Stable Simple No way to know round trip time. Not automatically self-clocked like TCP. (The algorithm need to have counters for self-clocking) Links can be paused by pause mechanism but CN is needed to avoid congestion spreading (pause spreading to innocent paths). Unlike TCP slow start, sources can come on at the full line rate of 10Gbps We must avoid queue oscillation which causes overflow (causes link pause) or underflow (lose utilization) The algorithm should be simple enough to be implemented in hardware (for 40G or 100G processing) Page 4

5 QCN Overview Page 5

6 QCN (Quantized Congestion Notification) QCN: Congestion Control Mechanism at Layer 2 Proposed by our group and accepted in IEEE 802.1Qau Terminology: Congestion Point: Where congestion occurs, mainly switches Reaction Point: Source of traffic, mainly rate limiters in Ethernet NICs Reaction Point Congestion Point Feedback Message NIC Traffic from other sources Fb Switch Page 6

7 QCN Algorithm: Congestion Point (CP) Feedback Processing Calculate Feedback value (Fb) from Switch Queue length Send feedback to Reaction Point (RP) Fb sampling probability No Congestion: Pmin(1%) Under Congestion: Pmax(10%) at maximum Fb Calculation F b Q ( Q off Qoff: Offset Q off Q eq w Q delta ) Reflection Probability Pmin Q Qdelta: Variance Q Q delta Q old Qoff Qdelta Qeq Qold P max Fb Fb : Feedback (quantized to 6bits) Q eq : Qlen in equibrium w: fixed parameter (= 2) Q: Current Queue Length Q old : Queue length at previous processing Page 7

8 Rate Control QCN Algorithm: Reaction Point (RP) Rate Decrease: Feedback from CP Rate Increase: Increase by itself Rate Increase Algorithm: Averaging Principle Based on BIC-TCP: Works without ACK Target Rate (TR) Current Rate (CR) Rate Fast Recovery TR TR CR (CR TR)/2 Active Increase Hyper-Active Increase Receive Feedback TR CR CR CR(1 G d F b ) Trigger Timeout or Byte Counter TR TR R AI CR (CR TR)/2 TR CR TR (CR ir HAI TR)/2 (G d F bmax 1/2) Time Page 8

9 Implementation Page 9

10 Current QCN Implementation World s s first Hardware QCN Full implementation It is fully functional We confirmed that the result matches OMNet++ simulation result! QCN-NIC Multiple Reaction Points Improved Token Bucket Rate Limiter QCN-Switch Multiple Congestion Points Compliant with Pseudo Code v2.3 [1] 1Gpbs Platform (NetFPGA) [1] QCN Pseudo Code v2.3: Page 10

11 Packet Format Data Frame (Normal Ethernet UDP/IP Frame) 63 0 DST MAC[47:0] SRC MAC[48:32] SRC MAC[31:0] Type =16 h0800 IP and UDP Header Extract as flowid[7:0] at QCN Switch QCN Field[15:0] QCN Fb Frame(64Bytes) flowid[7:0] qcn_fb[5:0] 63 reserved(2bits) 0 DST MAC[47:0] SRC MAC[31:0] SRC MAC[48:32] Type = 16 hxxxx QCN Field[15:0] Ethernet Padding Page 11

12 QCN NIC Block Diagram SRAM User Data Path Delay Line Reaction Point 3 Reaction Point 2 Reaction Point 1 Reaction Point 0 Input Arbiter Demux Demux UDP Pkt Gen Timer Token Bucket Rate Limiter Rate Calculator Byte Count Mux Mux Demux Receive Fb Demux qcn_fb, flowid Packet parser QCN Terminator Page 12

13 Token Bucket Rate Limiter Token bucket algorithm avoiding burstiness Bucket Size B Decrease packet size tokens in sending packet tokens Increase token_fill size tokens If refresh timer has expired Wait for being allowed to send Packet Previous implementation Send Send Condition: tokens tokens in in the the bucket bucket > 0 Causing burstiness! Current implementation Send Send Condition: tokens tokens in in the the bucket bucket >= >= token token size size Working Fine Page 13

14 QCN Switch Module Structure SRAM User Data Path Congestion Point Fb Calculator Demux 4port Rate Limiter Prioritized Mux Fb Generator Prioritized Mux Input Arbiter Port Byte Count src/dst mac src/dst port Qlen Monitor Prioritized Mux Fwd Port Packet Parser Receive Block Output Queues Prioritized Mux Page 14

15 Evaluation Page 15

16 QCN Hardware Evaluation System QCN-NIC0 RP0 4RPs RP1 RP2 RP3 Mux Delay Line MAC Port1 MAC QCN-Switch Demux Fb Receive Output Queue (Port0) QCN-NIC1 RP0 Mux Fwd Port CP Rate Limiter Port0 MAC drop 4RPs RP1 RP2 RP3 Mux Delay Line MAC Port2 MAC Receive Number of Reaction Points (RPs): 1-8 Each RP generates 1Gbps UDP Traffic Limit Rate at Switch: Limit Rate (950M 200M 950M) to check flow control behavior 950Mbps(3.7sec) 200Mbps(3.7sec) 950Gbps(3.7sec) Page 16

17 Parameters (QCN) NIC FAST_RECOVERY_THRESHOLD = 5 AI_INC = 0.5 Mbps HAI_INC = 5 Mbps BC_LIMIT = 150 KB (30% randomness) TIMER_PERIOD = 25 ms (30% randomness) MIN_RATE = 0.5 Mbps GD = 1/128 Switch Quantized_Fb: 6 bits Q_EQ = 33 KB W = 2 Based on QCN Pseudo Code Parameters Base marking = 150 KB, and varies according to the lookup table in the pseudo code (30% randomness) Page 17

18 Evaluation Result Page 18

19 Hardware Evaluation Result (1RP), RTT=100usec NIC: Rate Switch: Queue Length Rate [Mbps] Queue Size [KBytes] Evaluation Time [sec] Evaluation Time [sec] The NIC rate behavior well follows the limited Switch rate by QCN Algorithm Queue length rises temporarily in limiting Switch rate but go back to Qeq in shorter time. Page 19

20 OMNet++ Simulation Result (1RP), RTT=100usec NIC: Rate Switch: Queue Length Rate [Mbps] Simulation Time [sec] Queue Size [KBytes] Simulation Time [sec] Simulation result well well matches the the Hardware Evaluation Result!! Page 20

21 Hardware Evaluation Result (8RPs), RTT=1msec NIC: Rate Switch: Queue Length Rate [Mbps] Queue Size [KBytes] Evaluation Time [sec] Evaluation Time [sec] The aggregate NIC rate behavior still well follows the Switch limited rate Wiggling because of longer delay but still keeps the same queue length. It It also also matches OMNet++ Simulation Result with with the the same parameter Page 21

22 Demonstration Page 22

23 Conclusion We presented/demonstrated QCN theory and implementation QCN Implementation on NetFPGA Board All QCN Functions are implemented (Pseudo Code v2.3) QCN-NIC Multiple RPs are ready Improved Token Bucket Rate Limiter QCN-Switch Multiple CPs are ready Binary Search Logic for Fb Calculation We have proved that our QCN implementation achieves expected performance. The result matches OMNet++ Simulation Page 23