Photonic Networks for Data Centres and High Performance Computing Philip Watts Department of Electronic Engineering, UCL Yury Audzevich, Nick Barrow-Williams, Robert Mullins, Simon Moore, Andrew Moore Computer Laboratory, University of Cambridge
Motivation Future digital services are reliant on power efficient computers Data centres, large servers, scientific computing Power management is now the primary issue 200 W/chip thermal limit reached Interconnect accounts for majority of power consumption Transistors scale, wires don t Can t continue to exploit Moore s Law without dealing with interconnect power issue Power of Computation vs. Communication Transfer 32b across chip Transfer 32b off-chip 130nm CMOS (2002) 20 computations 260 computations Polymer Waveguides 45nm CMOS (2008) 57 computations 1300 computations Increase +285% +500% Many developments are addressing this issue: 3D stacked memory Non-volatile unified memory Photonic interconnect MEMORY CACHE CORES PHOTONIC NETWORK 3D Computer Chip Optical PCB Optical Backplane
DEMUX Components of a Photonic C2C Interconnect Hybrid Si Laser (UCSB/Intel) Si/SiO 2 Waveguides 400 x 200 nm, Bend Radius few μm Germanium Photodiodes integrated with Silicon waveguides 3 x 1 μm (Intel) Chip #1 Chip #2 Tx Silicon ring resonators can modulate, filter and switch Tx MUX X Tx 16 x 16 integrated SOA switch (Cambridge/Eindhoven) Polymer waveguides in standard FR4 PCB (Cambridge)
Characteristics of Photonic Networks Photonics offers very high bandwidth (WDM) over long distances with lower power Photonics favours circuit switching No viable optical buffer exists End-to-end paths required Computer architectures favours packet switching Message sizes in Shared Memory systems 64-bits Design implications of complete computer systems with photonic interconnect are not clear Traditional Electronic Networks 3D Mesh Fat Tree Edge Buffered Photonic Network Optical Network 4
On-chip Lasers or Off-chip Power Supply On chip laser approach Avoid losses in distribution and getting light onto chip Can be rapidly power gated ( ns) Photonic power supply (PPS) approach Only compact and low drive power components need to be on-chip Single optical power supply can be used for multiple devices We can t store photons any generated light is dissipated mainly onchip Unlikely that efficiency of optical power supply will be >20% Efficiency of electronic power supply can be >80% and is naturally dynamic PHOTONIC POWER SUPPLY On-chip laser approach LASER LASER LASER Photonic Power Supply approach MOD MOD MOD 5
Photonic Networks for Data Centres - Vision Scalability of the all-optical network is limited to around 64 ports, rack network Optical power and attenuation Latency and complexity of Arbitration/Contention resolution Expandable using O sections Optical Network Optical Network Optical Network
Methodology Multi- Wavelength Source Power Request (optional) Processor Chip Network Adapter Switch Chip Allocator Receiver Modelled the power consumption of the photonic path using the photonic power supply approach Created SystemVerilog models of the network control circuits and synthesized using a low leakage 45 nm CMOS process Evaluated the power consumption of circuit switched and time slotted networks covering a single rack How much power is disipated on-chip? What can be power gated?
Networks: Circuit Switching Electronic Network is required for residual flows Analysis of the traffic patterns generated running the PARSEC benchmark suite on simulated 32-core x86 system running Linux Observe the effect of changing the circuit decision time Circuit decisions are ideal (a priori knowledge of traffic)
Energy (J/bit) Energy per bit Circuit Switching Only 1.0 3.2 packets on average per circuit 10-9 10-10 10-11 10-12 Setting up circuits on μs timescales required 10-7 10-6 10-5 10-4 10-3 10-2 10-1 10 0 Circuit Decision Time (s) Ring resonator Clos switch assumed Current Injection - Min Current Injection - Max Capacitive - Min Capacitive - Max Electronic Packet Switch No significant advantage in static case
Time Slotted Networks Scheduled Speculative Time slots are 10 ns, sized for transmission of a single 32B cache line Speculative does not allow power gating of PPS Intra-Rack Network Latency
Total Energy, J/b Processor Chip Energy, J/b Time Slotted Network Time slotted networks can be > order of magnitude lower power than electronic mesh network 10-8 10-9 SOA Switch Ring Resonator Switch SOA switch, Dynamic OPS) Electronic total energy 10-8 10-9 SOA Switch Ring Resonator Switch SOA switch, Dynamic OPS) Electronic total energy 10-10 10-10 10-11 10-11 10-12 10 5 10 6 10 7 10 8 Packet Arrival Rate, pkts/s/port 10-12 10 5 10 6 10 7 10 8 Packet Arrival Rate, pkts/s/port 11
Sources of Power - Time Slotted Network Total Network Power, including PPS (W)
Sources of Power Time Slotted Network Power Dissipation on the Processor Chip, including PPS
Summary Photonic time slotted networks can reduce power consumption by > order of magnitude Total network power Processor chip dissipation Further work required to understand how these networks scale under realistic data centre traffic patterns Transceiver power consumption is significant proportion of on-chip disipation Yury will tell you more in next talk