Ethernet over SONET Dr. John S. Graham University of London Computer Centre
Ethernet vs SONET Feature Ethernet SONET Guaranteed Delivery No (Packet Switched) Yes (Circuit Switched) Guaranteed BW No Yes Performance Monitoring CRC Count BER Protection None (STP?) APS Fault Location None AIS Access Costs Low High
SONET/SDH Model MUX DXC R R MUX Section Layer Section Layer Section Layer Section Layer Line Layer Line Layer Path Layer Connection
SONET Network Elements
SONET STS-1 Frame 1 4 30 59 90 3 Section PTR Path SPE 5 Line 86 columns Payload Capacity = 756 Bytes = 48.384 Mb/s
Section Overhead Bytes Section Framing A1 BIP-8 B1 Data Com D1 Framing A2 Orderwire E1 Data Com D2 STS-1 ID C1 User F1 Data Com D3
Line Overhead Bytes Line BIP-8 B2 Data Com D4 Data Com D7 Data Com D10 Sync Stat Z1 APS K1 Data Com D5 Data Com D8 Data Com D11 Line FEBE Z2 APS K2 Data Com D6 Data Com D9 Data Com D12 Orderwire E2
Path Overhead Bytes Path Path Trace J1 BIP-8 B3 Signal Label C2 Path Status G1 User Channel F2 Multi-frame H4
Accurate Bit-Level Timing Line Coding AMI, AMI RZ HDB3 Block Coding 4B/5B (FDDI) 5B/6B (E3) 8B/10B (Gigabit Ethernet) Scrambling Frame Synchronous Self Synchronous
8B/10B Encoding (1/2) Ensures equal number of 1 s and 0 s for Clock recovery DC Balance Detection of single bit transmission errors Predefined comma (0011111 and 1100000) characters
8B/10B Encoding (2/2) Data (D) and commands (K) encoded separately Octet split into 5 MSB and 3 LSB Code groups generated from lookup tables Decoded Encoded A B C D E F G H A B C D E i F G H j 0 1 0 0 0 0 0 0 1 0 1 1 0 1 0 1 0 0 2 0 D2.0
Control Words ( Ordered Sets ) Symbol Encoding Purpose F K28.5 D21.5 LINK_NOT_AVAILABLE C K28.5 D10.5 config_reg LINK_CONFIGURATION I1 K28.5 D5.6 Idle I2 K28.5 D16.2 Idle S K27.7 SoF T K29.7 EoF R K23.7 CARRIER_EXTEND H K30.7 ERROR
Scrambling in SONET/SDH SDH STM-1 uses x 7 + x 6 +1 polynomial
Concatenation STS-3 155 Mb/s 145.152 Mb/s STS-3c/VC-4 155 Mb/s 149.76 Mb/s STS-12 622 Mb/s 580.608 Mb/s VC-4-4 622 Mb/s 599 Mb/s STS-12c 622 Mb/s 599.04 Mb/s
Provisioning Example A 1??? 3-5 F B 1 12 (622 Mb/s) E C 10-12 7 D 9
Bandwidth Fragmentation Drop Interface (Node F) STS-3c Blocked! Ring STS-12
Virtual Concatenation Drop Interface (Node F) STS-3c STS-1-3v Ring STS-12
The Magic of VCAT Protocol Client Rate Contiguous Concatenation Virtual Concatenation Fast Ethernet 100 STS-3c 65% STS-1-2v 98% ESCON 160 STS-12c 26% STS-1-4v 78% Fibre Channel 850 STS-48c 34% STS-3c-6v 92% Gigabit Ethernet 1000 STS-48c 40% STS-3c-7v 92% Fibre Channel 2 1700 STS-48c 66% STS-3c-12v 92%
Differential Delay Source STS-1-12v Core Network (Partial Mesh) Sink T X T y
Multiframe Indicator Bit 1 Bit 2 Bit 3 Bit 4 Bit 5 Bit 6 Bit 7 Bit 8 MFI2 MSBs (bits 1-4) 0 0 0 0 MFI2 LSBs (bits 5 8) 0 0 0 1 CTRL 0 0 1 0 GID 0 0 1 1 0 1 0 0 0 1 0 1 CRC-8 0 1 1 0 CRC-8 0 1 1 1 MST 1 0 0 0 MST 1 0 0 1 1 0 1 0 1 0 1 1 1 1 0 0 1 1 0 1 SQ MSBs (bits 1 4) 1 1 1 0 SQ LSBs (bits 5 8) 1 1 1 1
VCAT Example STS-1-4v 1 2 3 4 5 6 7 8 9 10 11 12 GID-a, SQ#=0 GID-a, SQ#=1 GID-a, SQ#=2 GID-a, SQ#=3 GID-b, SQ#=1 GID-b, SQ#=2 GID-b, SQ#=3 STS-12 STS-1-3v
VCAT Puzzle An STS-1-2v between Chicago and London One STS-1 sent via geostationary satellite The other STS-1 sent via transatlantic fibre Will it work? ;-)
VCAT: Buffer Requirements VC Path Type STS-1 VC-3 STS-1 VC-3 STS-3c VC-4 STS-3c VC-4 Transport Signal STS-12 STM-4 STS-48 STM-16 STS-12 STM-4 STS-48 STM-16 Number Paths Buffer Size 12 18 MB 48 71 MB 4 18 MB 12 55 MB
Link Capacity Adjustment Scheme Modify membership of a VCG On-demand and hitless bandwidth changes Automatic removal of failed VCG members Interworking of LCAS-enabled VCG with non-lcas VCG Old-style bridge and roll requires double bandwidth
LCAS Messaging Member Status (MST) Re-Sequence Acknowledge (RS-Ack) Control (CTRL) Fixed Add Norm EOS Idle DNU Group ID CRC
LCAS Signalling Flows (1/3) CTRL = NORM CTRL = EOS SQ = 1 SQ = 2 CTRL = NORM SQ = 1 CTRL = EOS SQ = 2 CTRL = IDLE SQ = 255 MST = FAIL (#3) CTRL = NORM SQ = 1 CTRL = EOS SQ = 2 CTRL = ADD SQ = 255 MST = OK (#3) CTRL = NORM SQ = 1 CTRL = NORM SQ = 2 CTRL = EOS SQ = 3 RS-ACK CTRL = NORM SQ = 1 CTRL = NORM SQ = 2 CTRL = EOS SQ = 3 Source PTE STS-1 #1 STS-1 #2 STS-1 #3
LCAS Signalling Flows (2/3) CTRL = NORM SQ = 1 CTRL = IDLE SQ = 255 CTRL = NORM SQ = 2 CTRL = NORM SQ = 2 CTRL = EOS SQ = 3 CTRL = EOS SQ = 3 MST = FAIL (#1) RS-ACK CTRL = NORM SQ = 1 CTRL = NORM SQ = 1 CTRL = EOS SQ = 2 CTRL = EOS SQ = 2 Source PTE STS-1 #1 STS-1 #2 STS-1 #3
LCAS Signalling Flows (3/3) CTRL = NORM SQ = 1 CTRL = NORM SQ = 1 CTRL = NORM SQ = 2 CTRL = NORM SQ = 2 CTRL = EOS SQ = 3 CTRL = IDLE SQ = 255 MST = FAIL (#3) RS-ACK CTRL = NORM SQ = 1 CTRL = NORM SQ = 1 CTRL = EOS SQ = 2 CTRL = EOS SQ = 2 Source PTE STS-1 #1 STS-1 #2 STS-1 #3
Gigabit Ethernet Implementations
Gigabit Ethernet (802.3z) ANSI X3T11 Fibre Channel IEEE 802.3 Ethernet IEEE 802.3z Gigabit Ethernet FC-4 Upper Layer Mapping FC-3 Common Services FC-2 Signalling FC-1 Encode/Decode FC-0 Interface & Media IEEE 802.2 LLC IEEE 802.3 CSMA/CD IEEE 802.3 PHY IEEE 802.2 LLC IEE 802.3 MAC PCS PMA PMD MDL
1000BASE-LX/SX
Sequence of Events LINK_NOT_AVAILABLE ~ LINK_NOT_AVAILABLE LINK_CONFIGURATION ~ LINK_CONFIGURATION IDLE ~ IDLE SOP PREAMBLE SFD DA SA LENGTH LLC Data PADDING FCS
Generic Framing Procedure ITU-T G.7041 and ANSI T1.105.02 Defines mapping for many types of service onto SONET/SDH or OTN: Ethernet, IP/PPP GbE, Fibre Channel (inc. DVB), FICON etc Excellent bandwidth utilization; efficiency tailored to suit different client types Simple delineation and robust error control Extensible
ATM Motivation for GFP (1/2) Cell overhead causes 10% bandwidth inflation Adaptation functions needlessly complex Packet over SONET (POS) Requires all frames to be converted to PPP over HDLC Byte stuffing causes non-deterministic bandwidth inflation QoS hard to monitor or guarantee
Motivation for GFP (2/2) Minimal overhead Transport of client PDUs in native format Designed for optimized processing Easy aggregation of frames from multiple client and multiple protocols into shared bandwidth channels Low latency capabilities for SAN
GFP: Types of Frame GFP Client Frames Control Frames Data Frames Management Frames Idle Frames OA&M Frames
GFP: Functional Model Source = IEEE Communications Magazine, May 2002
GFP: Frame Structure Source = IEEE Communications Magazine, May 2002
Payload Types Value of UPI Field Protocol GFP Mode 0x01 Ethernet F 0x02 PPP (IP & MPLS) F 0x03 Fibre Channel T 0x04 FICON T 0x05 ESCON T 0x06 GbE T 0x07 reserved 0x08 MAPOS F 0x09 0xFE reserved 0x00 & 0xFF unavailable
GFP: Client-Independent Processes Synchronization Bit Level Frame Delineation Scrambling Multiplexing GFP Frame Client PDU
GFP: Synchronization No CHEC Match Hunt chec Match No 2 nd chec Match Presync chec Mismatch 2 nd chec Match Sync chec Match
GFP: Frame Delineation PLI chec Payload PLI chec Payload PLI chec Payload PLI chec Payload PLI chec Payload PLI chec Payload
GFP: Client PDU Multiplexing Source Sink 1 1 2 GFP Framer STS-3c-2v GFP Framer 2 3 1 3 2 3
GFP-F Encapsulation (1/2) PLI (MSB) Ethernet MAC Frame PLI (LSB) chec (MSB Preamble 7 chec (LSB) Transmission Order SFD Destination Source Length Data 1 6 6 2 Payload Header Payload X = 4 to 64 0 to 65,535 - X FCS 1 8 Bit Transmission Order 4 FCS 4
GFP-F Encapsulation (2/2) PLI (MSB) PLI (LSB) PPP/HDLC Frame chec (MSB chec (LSB) Flag (0x7E) 1 Payload Header X = 4 to 64 Transmission Order Address Control PPP Type Data 1 1 2 Payload 0 to 65,535 - X FCS 1 8 4 FCS 4 Bit Transmission Order
GFP-T: 64B/65B Payload Octet Number 64B Sequence 000 001 010 011 100 101 110 111 D1 K1 D2 D3 D4 K2 D5 D6 Octet Number F 000 001 010 011 100 101 110 111 65B Sequence 1 1 001 C1 0 101 C2 D1 D2 D3 D4 D5 D6
GFP-T Error Detection Leading flag bit is errored Error affects Last control-code indicator Control-code location address received in error Error causes 4-bit control code to be modified
GFP-T: Superblocks 536-bit Byte-Aligned 1,1 1,2 1,7 1,8 1,1 1,2 8,1 8,2 8,1 8,2 8,7 8,8 1,7 1,8 8,7 8,8 CRC (MSB) CRC (LSB)
GFP-T: Bandwidth Requirements Client Signal Client Signal Bandwidth Minimum Transport Channel Size Nominal Transport Channel Bandwidth Number Superblocks per GFP Frame ESCON 160 Mb/s STS-1-4v VC-3-4v 193.536 Mb/s 1 Fibre Channel 850 Mb/s STS-3c-4v VC-4-6v 898.56 Mb/s 13 Gigabit Ethernet 1000 Mb/s STS-3c-7v VC-4-7v 1.04832 Gb/s 95
Pros and Cons: GFP-F Higher bandwidth efficiency Higher Latency More buffer memory required Core header fields must be calculated
Pros and Cons: GFP-T Supports many protocols Low latency Ingress core header fields need not be calculated Less bandwidth efficient More logic required
10 G Ethernet 802.3ae Full Duplex 802.3 MAC XGMII 64B/66B PCS Serial PMA WIS Serial PMA PMD E L S E L S PMD 10GBASE-R 10GBASE-W S = 850 nm MM 300 m L = 1310 nm SM 10 km E = 1550 nm SM 40 km
10 GbE Overview Usual 802.3 MAC and frame format Jumbo frames not included in standard Full duplex only No shared media No CSMA/CD Only optical fibre No copper interface LAN-PHY or WAN-PHY at various reaches
The STS-192c WIS Frame J1 B3 Idle 802.3 Frame Idle C2 G1 F2 H4 Fixed Stuff 9.58464 Gbs -1 Capacity Z3 Z4 Z5 Idle 802.3 Frame Idle 63 columns Synchronous Payload Envelope (SPE) = 16704 Columns
A1 B1 D1 Section & Line Overhead 1 2 192 193 194 384 385 386 576 H1 B2 D4 D7 D10 S1 A1 A1 A2 A2 A2 H1 B2 Z1 Section H1 B2 Z1 E1 D2 H2 K1 Overhead J0 F1 D3 Z0 Z0 D5 D6 Line D8 Overhead D9 D11 Z2 H2 Z2 H2 M1 H3 K2 D12 E2 H3 H3 Supported by WAN PHY Defined by WAN PHY as Fixed Value Unused by WAN PHY Optional for SONET/SDH Unused by WAN PHY Undefined for SONET/SDH
NetherLight Connectivity
UKLight NetherLight Today UKLight MetroDirector NetherLight 15454 NetherLight HDXc UKLight CoreDirector CERN 15454 UKLight MetroDirector
UKLight NetherLight Tomorrow? NetherLight 15454 NetherLight HDXc UKLight CoreDirector CERN 15454 UKLight MetroDirector Requires that MSPPs are GFP-F enabled
Peering Using 10 G WAN-PHY Chicago StarLight Force10 10GBASE-SW UKLight CoreDirector STM-64 UKLight CoreDirector 10GBASE-SW London UKLight Switch