Optics Technology Trends in Data Centers

Transcription

1 Optics Technology Trends in Data Centers Marc A. Taubenblatt IBM OFC Market Watch 2014

2 Outline/Summary Data movement/aggregate BW increasing and increasingly important Historically, per channel data rate increases underlie gains in optical interconnect cost and power efficiency It s getting harder to keep doing that Proliferation of alternate technology choices More Customized (e.g. Technology choice, EOE optimization, packaging) Market fragmentation, fight for volumes ( cost) Make the most of limited PHY resources higher up the stack (i.e. SDN) or new networks (OCS?)

3 Data Movement Becoming Critical Compute density increasing (multi-core, VM s), BW need increases EW traffic increasing* (Distributed workloads, virtualization, big data, random workloads, e.g. graph analytics) 44% CAGR in DC traffic, 76% within DC (Cisco 2012*) 80%+ of Google traffic now internal facing (B. Koley, Google OI Conf. 2012) Every 1kb of external traffic entering the datacenter generates 930kb of internal traffic (N. Farrington, Facebook OI Conf. 2013) Flatter topologies for lower latency ( higher radix, longer links) Resiliency (redundancy, path diversity, hot backup ) Caveat: Not all datacenters have the same needs (e.g Mega-center vs Enterprise) Interconnect Volumes vs Network Hierarchy Aggregation Decreasing volumes Flatter network Access Source: after B.Booth/Microsoft, Nov 2013, IEEE 400G study group TOR MOR *

4 Will networks dominate DC cost and power concerns? Amortized Cost after * H.Liu, Google, IEEE Summer Topicals 2013) after L.A. Barrosom, DataCenter as Computer, 2013 after A.Greenberg, Microsoft, ACM Sigcomm 2009 Growing concern on power and cost of network, but it s not the biggest piece today: Optics becoming most of network cost (H.Liu*) Growing faster than compute and memory But DC efficiency is gated by the network Latency, bottlenecks, predictable latency So has been easy to exploit increased optics/network cost for performance efficiencies at the DC level Leverage of benefits have to level off when?

5 $/Gbps Challenges for Physical Layer Optical Interconnects: Cost and Power Historically, gains in $/Gbps and mw/gbps driven by increased data rates: Aggregate data rates easier to manage with fewer channels e.g. 400GE advantageous at 8x50Gbp PCIe Gen5 likely to be 32Gb/s, Gen6 could be 64Gb/s. 64G Fibre Channel expected to be 56.10Gb/s OIF has defined 3 serial 56Gb/s interfaces for future optical interconnects But NRZ increasingly difficult at these data rates and beyond: E packaging losses, driver and receiver performance, Link budget CMOS logic not getting any faster, increased power for mux/demux M 266M 100M 1G 10G 1G 12x2.5G 100G (10x10) 2G 4G 8G 12x5G Year 4x5G 4x10G 12x10G 12x10G FC Ethernet AOC SNAP12 $/Gbps Data courtesy of Ken Jackson, prices at volume production Aggr Raw Data Rate (Gbps) FC Ethernet AOC SNAP12 Power (Ethernet) Power (AOC)

6 What are the technology choices to stay on trend? MultiMode (MM) VCSELs Getting faster but distance limited SingleMode (SM) Si Photonics Expensive but getting cheaper with Si Ph, no distance issues, link budget and cost (packaging!) more challenging SM VCSELs and MM Si Photonics Best of both worlds or worst of both worlds? WDM, PAM4, DMT Trades fiber cost for Trx cost & power (e.g. NRZ 56G a few db better link budget than PAM4) Fwd Error Correction (FEC) likely needed, adds some latency (e.g. ~100ns for 4-6dB gain*) and don t forget Copper 25G possible for a few meters, and it s cheaper than optics! *T.Wang et al, Huawei IEEE 400GE study group, Nov 13

7 Competing Technology Space Channel Speed (Gbps) SM Optics MM Optics Copper 1 10 Distance (m) OFC 2014 Th3C.2: D. Kuchta et al 64Gb/s Transmission over 57m MMF using an NRZ Modulated 850nm VCSEL (also >250m at 40G and > 100m at 60G) Beware that incumbant technologies are not standing still (e.g. VCSELs, Copper)

8 Data Rate & Power: A Packaging Challenge High data rates cause high electrical to optical link power tighter packaging integration of optics Standard edge of drawer module: Easy to use, replace and cable but electrical link will be costly/expensive at higher channel data rates (e.g. need SerDes in module, PAM4 ) Logic: mproc, memory, switch, etc. First-level package optical module Mid-board optics: Harder to cable and replace, but mitigates electrical link power and cost, opportunities for EOE optimization Logic: mproc, memory, switch, etc. First-level package optical module connector jumper On Package Optics: Difficult to design, cable and replace, but best opportunity for lower power and cost more amenable to HPC and highly custom needs Logic: mproc, memory, switch, etc. First-level package

9 Summary/Questions Staying on the cost and power trend is creating more technology choices and more possibilities for customization Proliferation of alternate technology choices Volumes are key to lower costs, could too many choices could inhibit volume? Cause longer time for market to adopt? More opportunities for technology niches with reasonable volumes? More Customized (e.g. Technology Choice, EOE optimization, packaging) Different DC s will have different needs, cost tolerance and focus areas Advanced packaging favors more sophisticated users, harder to do this with one size fits all designs Does market fragmentation dominate OR open HW standards?