DATA CENTER OPTICS AFCOM MIDLANTIC CHAPTER MEETING OCTOBER 10, 2014

Transcription

1 DATA CENTER OPTICS AFCOM MIDLANTIC CHAPTER MEETING OCTOBER 10, 2014 Lisa Huff, CDCP Principal Analyst, Discerning Analytics

2 AGENDA Discerning Analytics, LLC Data Center Networking Trends Data Center Optics Fiber versus Copper in the Data Center Vertical Market Summary 2

3 DISCERNING ANALYTICS Discerning Analytics is an independent data center market research and site selection firm. We produce custom studies and training to help clients distill complex data center concepts into actionable business strategies. General Market Research Reports Inside Data Center Optics (Published Oct. 2014) Custom Consulting & Assessments Data Center Technology Strategy / Growth Strategy Market & Site Selection Advisory Services Certified Data Center Professional on staff Staff with direct data center and LAN experience 3

4 DISCERNING ANALYTICS Common Requests: Site Selection Buy versus build Market analysis - evaluating the entire value chain Market forecasts Connectivity and equipment market research Optical components market research Competitive analysis Technical writing Product and design recommendations Best practices Training 4

6 DATA CENTER NETWORKING TRENDS Software Defined Networks Higher speed networks Server trends Data center power and cooling 6

7 SOFTWARE DEFINED NETWORKING (SDN) OSI Model SDN Model Source: SSF-NOW 7

8 SDN HYBRID IMPLEMENTATION Oracle s Hybrid SDN Solution 8

9 SDN METRO IMPLEMENTATION Vello Systems Metro SDN Solution 9

11 DATA CENTERS DEFINED Facility Purpose Typical Data Rates TR Cross-Connections & Switches 100 Mbps 1Gbps Typical # of Racks Typical Business < 5 All Small DC (Reduced) Medium DC (Basic) Servers, CPU, UPS, Switches Servers, CPU, UPS, POE, Local SAN, Switches 1 Gbps 5-20 Small Financial Hospitals 1-10 Gbps Banks Drug Co. Universities Large DC (Distributed) Servers, CPU, UPS, POE, Local & Remote SANs, Switches, Asset Management 1-100Gbps > 100 Large Financial Retail Insurance Large Research Labs Co-location Facilities TR: Telecommunications Room 11

12 LAN AND DATA CENTER GROWTH DRIVERS Government Initiatives Data Policy Directive (DPD) European Union (E.U.) Basel Capital Accord (2006) HIPAA (U.S.), PIPEDA (Canada) Sarbanes-Oxley Act (US) HITECH (US) Broadband Initiative (US) Redundancy to protect against catastrophic loss Web 2.0/Internet Data Centers Cloud Computing Both corporate and consumer Enterprise Storage Growth Source: Discerning Analytics Million Terabytes 12

13 DATA CENTER NETWORK ARCHITECTURE 10G 40G 100G Multiple 1G 10G 40G 100M/1G 1G 10G 13

14 TOP OF RACK (TOR) SWITCHING Source: Extreme Networks 14

15 DATA CENTER NETWORK TRENDS END OF ROW (EOR) SWITCHING Source: Brad Hedlund, Cisco Systems 15

16 DATA CENTER NETWORK TRENDS ETHERNET NETWORK ARCHITECTURE Top of Rack (ToR) End of Row (EoR) 16

17 DATA CENTER NETWORK TRENDS LAN DATA RATE PROGRESSION $9.00 $8.00 $7.00 $6.00 Billions USD $5.00 $4.00 $ G 40G 10G 1G $2.00 $1.00 $ Year Source: Discerning Analytics, LLC 17

19 X86 SERVER PORT DATA RATE PROGRESSION Milions of Ports Year 100M 1G 10G 40G 100G Source: Discerning Analytics 19

20 X86 SERVER FORECAST BY DATA RATE Millions of Ports G 10G 40G 100G Source: Discerning Analytics 20

21 SERVER TRENDS - MARKET PENETRATION BY DATA RATE Ethernet Speed (Mbps) Time to Reach (years) Million Port Sale (after Ratification) 60% Market Penetration , , , ,

22 SERVER TRENDS FORM FACTORS BLADE SERVERS Servers on a card, containing processors, memory, integrated network controllers, an optional Fibre Channel host bus adapter (HBA) and other input/output (IO) ports. Rack Server Blade Server 22

23 SERVER TRENDS Percentage of Servers 100% 90% 80% 70% 60% 50% 40% 30% 20% 10% 0% Year Pedestal Rack Blade 23

24 SERVERS VIRTUALIZATION IMPACT ON THE NETWORK Virtualization is the creation of a virtual (rather than actual) version of something, such as an operating system, a server, a storage device or network resources Drastically reduces the number of servers needed in the data center Impacts the amount of cabling needed Impacts the network architecture need faster server-to-server connections Source: CA Associates,

25 SERVER TRENDS BANDWIDTH DENSITY (Gbps/sqft or /rack) Server virtualization Pushing network connection data rates up Changing network traffic patterns from hierarchal to horizontal Server Form Factor Blades Source: HP 25

27 DATA CENTER TRENDS - POWER CONSUMPTION Source: Oracle 27

28 DATA CENTER HEAT LOAD TRENDS kw/ft 2 kw/rack ASHRAE, Datacom Equipment Power Trends and Cooling Applications, American Society of Heating, Refrigerating and Air-Conditioning Engineers, Inc., 28

29 MODULAR DESIGNS BASED ON COOLING REQUIREMENTS Hot/Cold aisle containment POD with APC in-row cooling Hot/Cold aisle containment POD with Liebert XDV overhead spot cooling Source: Oracle/Sun Microsystems 29

30 DATA CENTER TRENDS COLD AISLE CONTAINMENT 30

31 COOLING TRENDS MOVING CABLING TO ABOVE CABINETS 31

32 COOLING TRENDS STRUCTURED CABLING 32

34 GIGABIT ETHERNET OPTICAL VARIANTS Ethernet Variant 1000-BASE- Min. Reach (m) Form Factors Media Wavelength SX 300 GBIC, SFP LOMF 850nm z Standard IEEE LX 5000 GBIC, SFP SMF 1310nm ZX 70,000 GBIC, SFP SMF 1550nm Defacto GBIC SFP Source: Avago Technologies 34

35 10-GIGABIT ETHERNET OPTICAL VARIANTS Ethernet Variant 10GBASE- Data Rate (Gb/s) Min. Reach (m) Form Factors SR XENPAK, XFP, X2, SFP+ LR 10, pin, XENPAK, XFP, X2, SFP+ Media Wavelength LOMF 850nm SFM 1310nm LX4 4x XENPAK, X2 FDDI-grade MMF 850nm ER 10 40,000 XENPAK, XFP, X2 SMF 1550nm ZR 80,000 XFP, SFP+ SMF 1550nm LRM 220 XFP, SFP+ FDDI-grade MMF 1310nm Standard IEEE ae-2002/-2005 aq-2006/ pin XENPAK X2 XFP SFP+ 35 Source: Finisar

36 40-GIGABIT ETHERNET OPTICAL VARIANTS Ethernet Variant 40GBASE- Data rate (Gb/s) Min. Reach (m) Form Factors Media Wavelength Standard IEEE SR4 4x10 100/125 QSFP+ LOMF 850nm ba LR4 4x10 10,000 QSFP+, CFP SMF 1310nm FR 40 2,000 CFP SMF 1310/1550nm bg ER4 4x10 40,000 QSFP+, SFP SMF 1310nm bm QSFP+ CFP Source: JDSU Source: Reflex Photonics 36

37 100 GIGABIT ETHERNET OPTICAL VARIANTS Ethernet Variant 100GBASE- Data rate (Gb/s) Min. Reach (m) Form Factors Media Wavelength Standard IEEE SR10 10x10 100/150 CXP, CFP, CPAK LOMF 850nm ba LR4 4x25 10,000 CFP, CFP2, CFP4, QSFP28, CPAK SMF 1310nm ER4 4x25 40,000 CFP, CPAK? SMF 1310nm LR10 10x10 2,000 CFP SMF 1550nm MSA SR4 4x QSFP28, CFP2, CFP4, CPAK? LOMF 850nm bm LR4-lite 4x25 2,000 QSFP28, CFP2, CPAK? SMF 1310/1550nm MSAs CXP CFP CPAK CFP2 CFP4 QSFP28 Source: Avago Technologies Source: Oclaro Source: Cisco Systems Source: Avago Technologies Source: Fujitsu OC Source: ColorChip 37

38 DATA CENTER ETHERNET OPTICAL DEVELOPMENT STANDARDS IEEE P802.3bs 400Gb/s Task Force Objectives At least 100 m over MMF At least 500 m over SMF At least 2 km over SMF At least 10 km over SMF 16x25G current solution - CDFP CDFP AOC Source: TE Connectivity 38

39 DATA CENTER ETHERNET OPTICAL ROADMAP Source: Ethernet Alliance, OFC

40 FIBRE CHANNEL IN THE DATA CENTER 40

41 INFINIBAND AND HIGH PERFORMANCE COMPUTING Today s Supercomputer Source: Sun/Oracle 41

42 INFINIBAND ROADMAP Switched fabric communications link primarily used in high-performance computing. Aggregate Data Rates (Gbps) SDR DDR QDR FDR EDR HDR 1X 2.5 (1x2.5) X 10 (4x2.5) X 30 (12x2.5) Supercomputer Juropa (Jülich) #10 worldwide Source: IBM Source: FCI 42

43 PRE-TERMINATED FIBER ASSEMBLIES EXAMPLE Source: Brocade Source: Corning 43

45 2012 DATA CENTER CABLING MIX Data Center Media Mix SAN Media Mix 37% 6% 63% 94% Copper Fiber Copper Fiber Source: DataCenterStocks.com 2012 Research 45

46 2017 DATA CENTER CABLING MIX Data Center Media Mix SAN Media Mix 60% 40% 2% 98% Copper Fiber Copper Fiber Source: Discerning Analytics 2014 Research 46

47 AIRFLOW: HOT AISLE/COLD AISLE CEILING CABLE TRAYS or LADDER RACK 3-4 CABLE TRAYS or LADDER RACK CABLE TRAYS or LADDER RACK Front Cabinet or Rack Cabinet or Rack Cabinet or Rack Rear Rear RAISED FLOOR CABLE TRAYS POWER CABLES PERFED TILES CABLE TRAYS POWER CABLES PERFED TILES 47

48 DATA CENTER LAYOUT Source: 42u.com 48

49 DATA CENTER COPPER CABLING UNDER RAISED FLOOR 49

50 DATA CENTER COPPER CABLING ABOVE CABINETS 50

51 AIRFLOW: HOT AISLE/COLD AISLE Wire Basket Air flow channel Cat 6 UTP solution 242 cables at 50% Fill Ratio 240 channel Fiber solution 50/125 LOMF MDP 10 cables, Fill Ratio < 1 % *channels include 2 fibers Source: Berk-Tek, A Nexans Company 51

52 DATA CENTER CABLING FIBER VERSUS COPPER COST COMPARISON SUMMARY Connection Type Port Cost Patch Cords Horizontal Cable and Connectivity Installation Labor per Channel Total per Channel Cost Fiber SM FT $7, $ $ $ $ 7, Fiber LOMF FT $ 3, $ $ $ $ 3, Fiber LOMF PT $ 3, $ $ $ 6.25 $ 3, Copper UTP $ 1, $ $ $ $ 2, Copper FTP $ 1, $ $ $ $ 2, Model details: Distribution prices based on 16-port 10-Gig switch blades for Cisco Catalyst 6509 WS-X T-3CXL WS-X G-3CXL w/x2 10GBASE-SR modules Patch cords are 2m CAT6A, OM3 LOMF or SMF Labor rate of $60/hour Port cost computed by total cost of switch blade and transceivers/number of ports 52

53 FIBER VERSUS COPPER COST COMPARISON WITH POWER CONSUMPTION 48-port GigE Blade Blade Watts # Blades/ Chassis 16-port 10GigE Blade Blade Watts # Blades/ Chassis Total Watts W/port WS-X6748-SFP (Fiber) WS- X G-3CXL WS-X6748-GE- TX (Copper 1) WS- X T-3CXL WS-X6548-GE- TX (Copper 2) WS- X T-3CXL Optical solution saves 1.4 kw/rack 1W power savings yields 2.84W total Total possible savings = 4kW/rack 53

54 FIBER VERSUS COPPER THE CASCADE EFFECT Source: DVL 54

55 FIBER VERSUS COPPER ENERGY SAVINGS ECONOMICS SWITCH PORTS Save about $ /rack/year with optical solution (at $0.12/kWh) ROI would take about 32 years To get a 10 year payback Electricity would have to cost $0.77/kWh Installed optical port needs to be < $ more than an installed copper port Why not analyze a server rack? For further study Quantify airflow inefficiency with copper cabling 55

56 OTHER COMPELLING REASONS TO INSTALL FIBER Latency nanoseconds versus microseconds Lower power consumption at higher data rates Smaller will allow better airflow Higher density Longer reach Less weight in cable trays Not susceptible to EMI 56

57 DATA CENTER FIBER CABLING TYPES PROGRESSION 100% 90% 80% 70% 60% 50% 40% 30% 20% 10% 0% OM2 OM3 OM4 OS2 57

59 VERTICAL MARKET SUMMARY Typical Business Small Financial Manufacturing K12 Schools Size (sqft) # of Racks <1, Typical Data Rates (Gbps) 1 4 Networking Technologies Ethernet, iscsi, FC Banks Drug Co. Small Hospitals Higher Education 1,000-10, Ethernet, iscsi, FC Large Financial Retail Insurance Large Research Labs >10,000 > Ethernet, iscsi, FC, InfiniBand Medium Healthcare Web 2.0/IDCs ISPs Co-location 59

60 VERTICAL MARKET BREAKDOWN DATA CENTER OPTICS 25% 62% 13% Co-location Internet Enterprise Source: Discerning Analytics, LLC 2012 Research 60

61 SUMMARY Gigabit Ethernet still the workhorse of the typical data center but this is quickly changing to 10G 10G Ethernet is the largest revenue generator application in the data center for the next five years SFP+ 10GBASE-SR OM3 and OM4 Fiber SFP+ DAC 10GBASE-CR RJ45 10GBASE-T CAT6A/7 Twisted Pair 40G Ethernet is becoming mainstream for access to aggregation/distribution and core connections 100G Ethernet is the fastest growing networking technology in the data center over the next five years Ethernet networks in the data center will continue to evolve into flatter topologies Higher data-rates will be pushed closer to the server SDN could change all of the above dependent on how fast it is implemented 61

62 APPENDIX Case Study Contributors and References

63 DATA CENTER CASE STUDY WELLSPAN HEALTH Facilities Data centers support 2 hospitals, 10 health centers, 14 imaging centers, 16 laboratory locations, 55 physician offices State-of-the-art 6,000 sq ft. data center in York, PA built in MW, 1000 A Cooled by CRACs Continuously updated 5,100 sq. ft. additional space State-of-the-art 1,400 sq ft. mirror data center in York, PA built in MW, 800 A Cooled by CRACs Continuously updated 63

64 WELLSPAN HEALTH Network Three-tiered Ethernet network: Access, Distribution/Aggregation, Core Access Gigabit Distribution Multiple Gigabit Core Multiple 10G, looking at 40G Topology Main Gigabit-rated cross-connect Transitioning to Top-of-Rack switching using Cisco Nexus 2000 Cisco Nexus 5000 in Distribution/Aggregation and Nexus 7000 in Core Multi-wavelength CWDM/DWDM between buildings Separate FC SAN Looking at FCoE still too expensive 64

65 WELLSPAN HEALTH Cabling Using TIA-942 as much as possible Copper in Access Category 6 Fiber in Distribution/Aggregation Pre-terminated LOMF OM2 and OM3 MPO assemblies with cassettes Fiber in Core SMF between data centers and buildings LOMF and SMF in SAN 65

66 ACKNOWLEDGEMENTS 66

67 RESOURCES STANDARDS ANSI/BICSI , Data Center Design and Implementation Best Practices FC-PI, Fibre Channel Physical Layer Standards IBTA InfiniBand Architecture Specification IEEE 802.3, Standard for Information Technology TIA-942, Telecommunications Infrastructure Standard for Data Centers TIA-568, Commercial Building Cabling for Telecommunications TIA-1179, Healthcare Facility Telecommunications Infrastructure Standard 67

68 RESOURCES ARTICLES, WHITE PAPERS, BLOGS, MARKET RESEARCH Huff, L. (2014), Data Center Optics, Discerning Analytics, LLC Huff, L. (2013), Ethernet Networks Progress to 400G, Connector Supplier Debiec, T. & Huff, L.(2013)Structured Cabling Technology and Market, Connector Supplier Huff, L. (2012) Multi-Gigabit Datacom Network Connectivity, Bishop & Associates, Inc. Huff, L. (2011) The Battle for the Converged Data Center Network, Data Center Knowledge.com Gross, D., & Huff, L. (2011). DataCenterStocks.com Blog. Huff, L. (2011). Optical Components Blog. Huff, L. (2011) Data Center Cabling Copper versus Fiber Webinar Ponemon Institute (2011), Calculating the Cost of Data Center Outages, Ponemon Institute and Emerson Network Power Hewlett Packard (2011) Building Virtualization-Optimized Data Center Networks, Gosai, Birendra (2011) Building the Next Generation Data Center, CA Technologies Web site 68

69 RESOURCES ARTICLES, WHITE PAPERS, BLOGS, MARKET RESEARCH Huff, L. (2010). Data Center Cabling Cost Analysis. DataCenterStocks.com. Huff, L. (2010). DAC Report. DataCenterStocks.com. Huff, L. (2010). 40/100 GigE Markets: 2010 and Beyond. Communications Industry Researchers. Huff, L. (2010). New Revenue Opportunities for Optical Interconnects: A Market and Technology Forecast. Communications Industry Researchers. Cisco (2009), Data Center Top-of-Rack Architecture Design, Cisco Cisco (2007), Cisco Data Center Infrastructure 2.5 Design Guide, Cisco APC (2005), Determining Total Cost of Ownership for Data Center and Network Room Infrastructure, TechRepublic.com 69

70 MULTI-SOURCE AGREEMENTS (MSAS) AND SFF 10G SFP+: SFF G QSFP+: SFF G CXP: InfiniBand Supplement CFP/CFP2/CFP4: CFP MSA 10x10: 10x10 MSA QSFP28: SFF G CDFP 70