Data Center Optimization: Component Choice. Innovative Design. Enabling Infrastructure

Transcription

1 Data Center Optimization: Component Choice and Innovative Design Combine to Allow the Creation of an Enabling Infrastructure

2 Data Center Spaces Data Center Spaces as Defined by the TIA 942 Standard

3 Source: TIA 942 Data Center Spaces: TIA 942

4 Data Center Spaces: Definitions Entrance Facility (ER) Location of the interface with campus and carrier entrance facilities Main Distribution Area (MDA) Location of main cross-connect (MC) Horizontal Distribution Area () Location of horizontal cross-connect (HC) Zone Distribution Area (ZDA) Location of zone outlet (ZO) or consolidation point (CP) Equipment Distribution Area (EDA) Location of equipment cabinets and racks

5 Data Center Logical Spaces The Web or Internet Gateway LAN SAN A SAN B DC Core HPC DC Aggregation DC Access High Performance Computing Data Center LAN Data Center SAN

6 Logical Architectures Core Layer Provides the high-speed connectivity between the data center and the campus network Central t l Patching Aggregation Layer (EoR or ) Provide a point where all server area devices can share common applications such as firewalls, cache engines, load balancers and other value added services Common path to Central Patching Access Layer (Edge or EDA) Provides the connectivity between the aggregation layer shared services and the server farm Uplinks to Aggregation Layer

7 Data Center Spaces The Computer Room Server Environment: EoR / CoR / ToR

8 Data Center Spaces

9 Data Center Spaces MDA ZDA EDA EDA EDA EDA

10 Data Center Spaces Core Layer MDA ZDA Aggregation Layer Aggregation Layer Aggregation Layer Access EDA Layer Access Layer EDA Access EDA Layer Access Layer EDA

11 Data Center Spaces Top of Rack In it s simplest form, Top of Rack Cisco switches are shown. Benefits include copper only in the server racks, fiber trunks to the /MDA. Each server rack has fiber patch field to SAN Core Ethernet ToR switching implemented utilizing fiber optic backbone to Ethernet Core Source: Cisco

12 Data Center Spaces End of Row Elevations Network Copper with SAN Fiber and Centralized EoR Fiber Switch. Each server rack has fiber / copper patch field to EoR Cabinet Ethernet ToR Centralized switching implemented utilizing fiber optic backbone to Ethernet Core

13 Typical Data Center Floor plan layout with Highlighted Server Row & MDA Topology

14 Typical Data Center Floor plan layout with Highlighted Server Row Row CoR & MDA & MDA Topology

15 Typical Data Center Floor plan layout with Highlighted Server Row Row CoR EoR & MDA & MDA Topology

16 Implementation Options Equipment Location & Associated Media Choices

17 Implementation Options End of Row (EoR) Implementation Utilizing Traditional ToR Switch Placement

18 EoR Design Concepts: Option 1 Switches or Fex (Example N2K) located in each server cabinet. Aggregation g switch (Example N5K) Located in EoR. C S ( ) Aggregation or Core Switch (Example N7K) Located in MDA.

19 Server to (N2K or N5K) Design ToR or located in each server cabinet. Aggregation switch (N5K) located in EoR ToR ToR ToR ToR ToR ToR ToR ToR ToR R10 B 5K EoR ToR ToR ToR ToR ToR ToR ToR ToR ToR ToR R9 R8 5K EoR ToR ToR ToR ToR ToR ToR ToR ToR ToR ToR B 5K EoR ToR ToR ToR ToR ToR ToR ToR ToR ToR A 5K R10 EoR ToR ToR ToR ToR ToR ToR ToR ToR ToR ToR 5K R9 EoR ToR ToR ToR ToR ToR ToR ToR ToR ToR ToR A 5K R8 EoR ToR ToR ToR ToR ToR ToR ToR ToR ToR ToR R7 5K EoR ToR ToR ToR ToR ToR ToR ToR ToR ToR ToR 5K R7 EoR R6 Net. 7K Core Switch Net. 7K Core Switch R6 B MDA A

20 Server to (N2K or N5K) Media Server to Fex (ToR) Media Choices: Cat 6A Patch Cable:

21 Server to (N2K or N5K) Media Server to Fex (ToR) Media Choices: SFP+ Patch Cable:

22 Server to (N2K or N5K) Media Server to Fex (ToR) Media Choices: Fiber Optic Patch Cable:

23 (N2K or N5K) to Aggregation Switch (N5K or N7K) Design ToR or (N2K) located in each server cabinet. Aggregation switch (N5K) located in EoR Cabling from ToR (N2K) to Aggregation switch (N5K) in EoR ToR ToR ToR ToR ToR ToR ToR To R ToR R10 R9 B B 5K EoR ToR ToR ToR ToR ToR ToR ToR To R ToR ToR R8 B B 5K EoR ToR ToR ToR ToR ToR ToR ToR To R ToR ToR R7 R6 Net. 7K Core Switch 5K EoR ToR ToR ToR ToR ToR ToR ToR To R ToR ToR B 5K EoR MDA ToR ToR ToR To R ToR ToR ToR ToR ToR B A B 5K R10 EoR ToR ToR ToR To R ToR ToR ToR ToR ToR ToR B A B 5K R9 EoR ToR ToR ToR ToR ToR To R ToR ToR ToR ToR Net. 7K Core Switch 5K R8 EoR ToR ToR ToR ToR ToR To R ToR ToR ToR ToR A 5K R7 EoR R6

24 (N2K or N5K) to Aggregation Switch (N5K or N7K) Media Fex to Aggregation Switch Media Choices: Cat 6A Patch Cable:

25 (N2K or N5K) to Aggregation Switch (N5K or N7K) Media Fex to Aggregation Switch Media Choices: SFP+ Patch Cable: Design Note: 7 Meter Max Length

26 (N2K or N5K) to Aggregation Switch (N5K or N7K) Media Fex to Aggregation Switch Media Choices: Fiber Optic Patch Cable:

27 (N2K or N5K) to Aggregation Switch (N5K or N7K) Media Fex to Aggregation Switch Media Choices: Fiber Optic Trunk Cable: Connector choice: MTP

28 Aggregation Switch (N5K or N7K) to Core (N7K) Design ToR or located in each server cabinet. Aggregation switch (N5K) located in EoR Aggregation switch (N5K) to Core (N7K) located in the MDA ToR ToR ToR ToR ToR ToR ToR ToR ToR R10 R9 B 5K EoR ToR ToR ToR ToR ToR ToR ToR ToR ToR ToR R8 B 5K EoR ToR ToR ToR ToR ToR ToR ToR ToR ToR ToR 5K EoR ToR ToR ToR ToR ToR ToR ToR ToR ToR ToR R7 R6 B Net. 7K Core Switch 5K EoR MDA To R To R To R To R To R To R ToR ToR ToR A 5K R10 EoR To R To R To R To R To R To R ToR ToR ToR ToR A 5K R9 EoR To R To R To R To R To R To R ToR ToR ToR ToR 5K R8 EoR To R To R To R To R To R To R ToR ToR ToR ToR Net. 7K Core Switch A 5K R7 EoR R6

29 Aggregation Switch (N5K or N7K) to Core (N7K) Media Aggregation Switch to Core Media Choices: Fiber Optic Trunk Cable: Connector choice: MTP

30 Different Physical Implementation of ToR Switch Port Utilization Number of servers in any single cabinet is insufficient to allow full utilization of switch Available power limits the number of servers installed in a cabinet Scaling of DC Data Center build out is in a phased strategy Co-location or hosting facility will scale as required by client needs Administration of System Client would like to limit it Switch Sprawl

31 Implementation Options: Centralized Switch Placement Center of Row (CoR) Implementation Utilizing Centralized Switch Placement

32 CoR Design Concepts: Switches in Example Switches or (Example N2K) located in CoR (). Aggregation switch (Example N5K) Located in CoR (). Aggregation or Core Switch (Example N7K) Located in MDA.

33 Server Patch Panel Design Cat 6A or Fiber Patch Panel located in each server cabinet. and Aggregation switch (N5K) located in CoR P/P R10 P/P R9 P/P R8 P/P R7 P/P P/P P/P P/P P/P P/P P/P P/P P/P P/P P/P P/P P/P P/P P/P P/P P/P N5K N5K B A P/P P/P P/P P/P P/P P/P P/P P/P P/P N5K P/P P/P P/P P/P P/P P/P P/P P/P P/P N5K B A P/P P/P P/P P/P P/P P/P P/P P/P P/P N5K R10 P/P P/P P/P P/P P/P P/P P/P P/P N5K P/P P/P P/P P/P P/P P/P P/P P/P N5K P/P P/P P/P P/P P/P P/P P/P P/P N5K R9 R8 R7 R6 Net. 7K Core Switch Net. 7K Core Switch R6 B MDA A

34 Server Patch Panel Media Server Cabinet Patch Panel Choices: CAT 6A (HD) Patch Panel:

35 Server Patch Panel Media Server Cabinet Patch Panel Choices: MTP to LC Duplex Cassette:

36 Server Patch Panel Media Server Cabinet Patch Panel Choices: MTP to MTP 40GbE Cassette:

37 Server to Patch Panel Media Server to Patch Panel Media Choices: Cat 6A Patch Cable:

38 Server to Patch Panel Media Server to Patch Panel Media Choices: Fiber Optic Patch Cable:

39 Server to Patch Panel Media Server to Patch Panel Media Choices: Fiber Optic Patch Cable:

40 Server Cabinet Patch Panel to (N2K or N5K) Design Cat 6A or Fiber Patch Panel located in each server cabinet. and Aggregation switch (N5K) located in CoR Cabling from Patch Panel to (N2K) in CoR P/P R10 P/P R9 P/P R8 P/P R7 R6 P/P P/P P/P P/P P/P P/P P/P P/P P/P N5K B P/P P/P P/P P/P P/P P/P P/P P/P P/P N5K P/P P/P P/P P/P P/P P/P P/P P/P P/P N5K B P/P P/P P/P P/P P/P P/P P/P P/P P/P N5K Net. 7K Core Switch B SAN MDA P/P P/P P/P P/P P/P P/P P/P P/P N5K A R10 P/P P/P P/P P/P P/P P/P P/P P/P N5K P/P P/P P/P P/P P/P P/P P/P P/P N5K A P/P P/P P/P P/P P/P P/P P/P P/P N5K Net. 7K Core Switch A SAN R9 R8 R7 R6

41 Server Cabinet Patch Panel to (N2K or N5K) Media Server Cabinet Patch Panel to Choices: CAT 6A (Trunk Cabling is suggested):

42 Server Cabinet Patch Panel to (N2K or N5K) Media Server Cabinet Patch Panel to Choices: Fiber Optic Trunk Cable:

43 (2NK) to Aggregation (N5K) Design and Aggregation switch (N5K) Colocated in CoR. Cabling from Patch Panel to (N2K) in CoR Inter-cabinet cabling utilized for (N2K) to Aggregation (N5K) Switch P/P R10 P/P R9 P/P R8 P/P R7 R6 P/P P/P P/P P/P P/P P/P P/P P/P P/P N5K B P/P P/P P/P P/P P/P P/P P/P P/P P/P N5K P/P P/P P/P P/P P/P P/P P/P P/P P/P N5K B P/P P/P P/P P/P P/P P/P P/P P/P P/P N5K Net. 7K Core Switch B MDA P/P P/P P/P P/P P/P P/P P/P P/P N5K A R10 P/P P/P P/P P/P P/P P/P P/P P/P N5K P/P P/P P/P P/P P/P P/P P/P P/P N5K A P/P P/P P/P P/P P/P P/P P/P P/P N5K Net. 7K Core Switch A R9 R8 R7 R6

44 (2NK) to Aggregation (N5K) Media to Aggregation Media Choices: Cat 6A Patch Cable:

45 (2NK) to Aggregation (N5K) Media to Aggregation Media Choices: SFP+ Patch Cable:

46 (2NK) to Aggregation (N5K) Media to Aggregation Media Choices: Fiber Optic Patch Cable:

47 (2NK) to Aggregation (N5K) Media to Aggregation Media Choices: Fiber Optic Patch Cable:

48 Aggregation Switch (N5K or N7K) to Core (N7K) Design Fiber Patch Panel located in each CoR cabinet. and Aggregation g switch (N5K) located in CoR Aggregation switch (N5K) to Core (N7K) located in the MDA P/P R10 P/P R9 P/P R8 P/P R7 R6 P/P P/P P/P P/P P/P P/P P/P P/P P/P N5K B P/P P/P P/P P/P P/P P/P P/P P/P P/P N5K P/P P/P P/P P/P P/P P/P P/P P/P P/P N5K B P/P P/P P/P P/P P/P P/P P/P P/P P/P N5K Net. 7K Core Switch B MDA P/P P/P P/P P/P P/P P/P P/P P/P N5K A R10 P/P P/P P/P P/P P/P P/P P/P P/P N5K P/P P/P P/P P/P P/P P/P P/P P/P N5K A P/P P/P P/P P/P P/P P/P P/P P/P N5K Net. 7K Core Switch A R9 R8 R7 R6

49 Aggregation Switch (N5K or N7K) to Core (N7K) Media Aggregation Switch to Core Media Choices: Fiber Optic Trunk Cable: Connector choice: MTP

50 Traditional ToR Switch Placement / Centralized Switch Placement Two basic approaches to server networking are traditional Top-of-Rack and Centralized Switch Placement. These concepts utilize the same basic building blocks to accomplish the same primary function of creating a distributed switch architecture. Traditional ToR is normally seen with Fabric Extenders utilized at the server cabinet with aggregation switches at the. When Traditional ToR architecture is not implemented, centralized switching has every server cabled back to a set of switches ( & Aggregation) dedicated di d to a row of server racks.

51 Traditional ToR Switch Placement / Centralized Switch Placement An upside of the centralized approach is a simplified management environment, with less switches to manage. It should be noted when ToR switches / fabric extenders are located within the and not located directly within the server cabinet it may allow for more efficient switch port utilization. It s important to point out that both approaches are valid designs and the ultimate choice is driven by broader architectural requirements

52 Traditional ToR Switch Placement / Centralized Switch Placement It is important to design with a forward looking migration strategy. As such backbone connectivity design between switches should allow for a simple non-disruptive migration strategy to allow implementation of 40GbE & 100GbE bandwidth capabilities. Implementation of MTP / MPO multi-fiber terminations and interconnect components are recommended. Traditional & Centralized cabling can include: Copper Trunks Copper Jumpers Fiber Trunks Fiber Jumpers

53 Data Center Spaces - Equipment MPO & MTP Multi-Fiber Connector Technology

54 Parallel Optics Transmission Parallel Optics

55 The Need for Speed High Bandwidth: 10 Gig 40 Gig 100 Gig = Parallel Optics

56 IEEE 802.3ba Cabling Media Options Physical Layer specifications which support 40 Gb/s operation over: at least 10km on SMF at least 100m on OM3 (2Ghz) MMF at least 150m on OM4 (4.7Ghz) MMF at least 7m over a copper cable assembly Physical Layer specifications which support 100 Gb/s operation over: at least 40km on SMF at least 10km on SMF at least 100m on OM3 MMF (2Ghz) MMF at least 150m on OM4 MMF (4.7Ghz) MMF at least 7m over a copper cable assembly

57 Utilization of Fibers IEEE Ethernet Channel Layout Data Rate Laser Type Fiber Type Connector # of Fibers 10Gb/s VCSEL OM3 LC x Gb/s VCSEL Array OM3 / OM4 12-fiber MPO Gb/s VCSEL Array OM3 / OM4 2x12f MPO or 1x24f MPO 24 Schematic TX TX TX TX RX RXRXRX Single Port TX TX TX TX TX TX TX TX TX TX (recommended) RX RX RX RX RX RX RX RX RX RX 40GBASE-SR4 100GBASE-SR10 or Side-by- Side Ports TX TX TX TX TX TX TX TX TX TX RXRXRXRXRXRXRX RXRXRX Source: ethernet alliance

58 40G (SR4) Ethernet Parallel Optics 12MMF MTP 10 GbE X 4 Fibers each direction = 40 GbE 12F MTP Connectors 1 Rx 1 Tx 2 Rx 2 Tx 3 Rx 3 Tx 4 Rx 4 Tx Tx 10 Tx 11 Tx 12 Tx Fibers: Tx 9 Rx Fibers: not used 10 Rx Fibers: Rx 11 Rx 12 Rx 1 12

59 100G (SR10) Ethernet Parallel Optics 24MMF MTP 10 GbE X 20 Fibers each direction = 100 GbE 1 2 Rx 3 Rx 4 Rx 5 Rx 6 Rx 7 Rx 8 Rx 9Rx 10 Rx 11 Rx Tx 3 Tx 4 Tx 5 Tx 6 Tx 7 Tx 8 Tx 9Tx 10 Tx 11 Tx Fiber Fibers: 2 thru 11 Tx MTP Connectors Fibers: 1&12 not used Tx 15 Tx 16 Tx 17 Tx 18 Tx 19 Tx 20 Tx 21 Tx 22 Tx 23 Tx Rx 15 Rx 16 Rx 17 Rx 18 Rx 19 Rx 20 Rx 21 Rx 22 Rx 23 Rx 24 Fibers: 14 thru 23 Rx Fibers: 13 & 24 not used

60 Future: MMF 100 GbE with Less Fibers 100GBASE-SR4 Uses 4 fibers to transmit and 4 fibers to receive 25 GbE per fiber Same 12F / 24F trunk you install today reused again Target reach of 100m on OM4 Target price of 100GBASE-SR10 4x 25 GbE transceiver not really there yet

61 Summary Summary Points

62 Design for the Future, Adapt for Today Current Infrastructure Media Choices Allow for an Extended Installed Product Lifecycle Accelerating Technology Development Cycles to Continue Data Centers Require Adaptability Live Long and Prosper Scalability Migratability Design and Product Choice is Critical to Optimized ROI Model (Migrate and Improve ROI)

63 Ensure The Quality of All Infrastructure Aspects Ensure a Standards Based Design Ensure Product Longevity: Quality Vendor Choice Ensure Quality Installation is Performed: Best Practice & Warranty Look for Recognized Third-Party Certifications: Trust in International Certification Labs

64 Questions?