January 16-17, 2013 Santa Clara

Transcription

1 January 16-17, 2013 Santa Clara

2 Open Rack Workshop Day 1 Matt Corddry, Facebook Pierluigi Sarti, Facebook Steve Mills, Facebook Richard Chan, Delta Products Patrick MacDonald, Sanmina Corp Eric Wilcox, Emerson Network Power

3 Introductions Matt Corddry, Open Rack Project Chair Manager, Gloria Lau Manager, Pierluigi Test Lead, Rack & Power, Facebook Mills Mech Samir Rack & Power Team, Facebook Sarti Technical Steve Facebook Hardware Design Engineer, Rack & Power, Facebook Kala Engineer, Rack & Power, Facebook

4 Introductions Richard Chan Data Center Solution R&D Manager, Delta Products Patrick MacDonald SVP, Mechanical System Division, Sanmina Corp Eric Wilcox VP of Engineering and Operations, Emerson Network Power

5 How did we get here? Open Compute Rack Overview Matt Corddry, Facebook Pierluigi Sarti, Facebook

6 19 Racks: Complex Deployment

7 19 Racks: Some Assembly Required

8 19 Racks: Loose Standards

9 Open Compute Rack v1: The Triplet Ø Great for Compute Servers

10 Open Compute Rack v1: The Triplet Ø Great for Compute Servers Ø Fixed at 1.5U Increments

11 Open Compute Rack v1: The Triplet Ø Great for Compute Servers Ø Fixed at 1.5U Increments Ø Custom PSU in every Server

12 Open Compute Rack v1: The Triplet Ø Great for Compute Servers Ø Fixed at 1.5U Increments Ø Custom PSU in every Server Ø Tool-free Server Installation

13 Open Compute Rack v1: The Triplet Ø Great for Compute Servers Ø Fixed at 1.5U Increments Ø Custom PSU in every Server Ø Tool-free Server Installation Ø Limited Flexibility

14 Colocation Rack Open Compute Rack v1 in Colocation Sites Matt Corddry, Facebook Steve Mills, Facebook

15 Colocation Rack Why? Some facilities cannot accept our Triplet racks Ø 95 Tall, 66 Wide, 2600 lbs, 480v power

16 Colocation Rack Why? Some facilities cannot accept our Triplet racks Ø 95 Tall, 66 Wide, 2600 lbs, 480v power We needed a standard rack form factor for our OCP servers

17 Colocation Rack What? A modified DAMAC 19 server rack

18 Colocation Rack What? A modified DAMAC 19 server rack Shelves and side panels from an OCP triplet rack

19 Colocation Rack What? A modified DAMAC 19 server rack Shelves and side panels from an OCP triplet rack Off-the-shelf power strips from Server Technology for 208v power

20 Colocation Rack What? A modified DAMAC 19 server rack Shelves and side panels from an OCP triplet rack Off-the-shelf power strips from Server Technology for 208v power New power cords connecting the OCP power supply to the new power strips

21 Colocation Rack When? Deployed in Facebook datacenters today

22 Colocation Rack When? Deployed in Facebook datacenters today Will be replaced with Open Rack this year Ø Open Rack works in all facilities

23 Colocation Rack Q&A (please state name & company)

24 Open Rack Update We ve Come A Long Ways Matt Corddry, Facebook Pierluigi Sarti, Facebook Steve Mills, Facebook

25 Why Open Rack? Lower Total Cost of Ownership Disaggregation of technologies Maximize component life Infrastructure cost per node is reduced Larger volume allows more nodes per rack Flexibility allows multiple generations Rack Level Standards Modular and Serviceable components Simplified data center deployment Potential Component Lifecycle in Years: CPU 4 FANS 5 Boot Disk 5 Switches 5 Memory 6 PSU 6 Rack 10

26 Open Rack Overview Ø Well-defined Mechanical API between the server and the rack

27 Open Rack Overview Ø Well-defined Mechanical API between the server and the rack Ø Accepts any size equipment 1U 10U

28 Open Rack Overview Ø Well-defined Mechanical API between the server and the rack Ø Accepts any size equipment 1U 10U Ø Wide 21 equipment bay for maximum space efficiency

29 Open Rack Overview Ø Well-defined Mechanical API between the server and the rack Ø Accepts any size equipment 1U 10U Ø Wide 21 equipment bay for maximum space efficiency Ø Shared 12v DC power system

30 Open Rack: Dimensions AC Rack dimensions: 600mm (23.6 ) wide 1067 mm (42 ) deep 2100 mm (82.7 ) tall DC 600mm Rear View Front View 538mm

31 Open Rack: Equipment Bay Total IT equipment bay space: 538mm (21 ) chassis width x 830mm (32.7 ) max chassis depth 538mm (21 ) Front view Power Zone 13 OU Equipment Bay (10 OU)

32 Open Rack: Equipment Bay Total IT equipment bay space: 538mm (21 ) chassis width x 830mm (32.7 ) max chassis depth Rack slot height measured in 48mm OpenU (OU) 538mm (21 ) Front view Power Zone 13 OU Equipment Bay (10 OU)

33 Open Rack: Equipment Bay 538mm (21 ) Total IT equipment bay space: 538mm (21 ) chassis width x 830mm (32.7 ) max chassis depth Rack slot height measured in 48mm OpenU (OU) Server space: three equipment bay at 10 OU each; IT switch space is 2 OU Power Zone 13 OU Front view Equipment Bay (10 OU)

34 Open Rack: Equipment Bay 538mm (21 ) Total IT equipment bay space: 538mm (21 ) chassis width x 830mm (32.7 ) max chassis depth Rack slot height measured in 48mm OpenU (OU) Server space: three equipment bay at 10 OU each; IT switch space is 2 OU Adjustable chassis slot height (1 OU to 10 OU, with 0.5 OU increments) Power Zone 13 OU Front view Equipment Bay (10 OU)

35 Open Rack: Mounting Server Shelf: Movable within the rack to support gear of any height (e.g. 1.5 OU, 2 OU, etc.) at 0.5 OU step increments

36 Open Rack: Mounting Server Shelf: Movable within the rack to support gear of any height (e.g. 1.5 OU, 2 OU, etc.) at 0.5 OU step increments Tool-less mounting and removal from front rack, without pulling out chassis from next slot

37 Open Rack: Mounting Server Shelf: Movable within the rack to support gear of any height (e.g. 1.5 OU, 2 OU, etc.) at 0.5 OU step increments Tool-less mounting and removal from front rack, without pulling out chassis from next slot Supports compute chassis as small as 1 OU height

38 Open Rack: Mounting Server Shelf: Movable within the rack to support gear of any height (e.g. 1.5 OU, 2 OU, etc.) at 0.5 OU step increments Tool-less mounting and removal from front rack, without pulling out chassis from next slot Supports compute chassis as small as 1 OU height Wide range of chassis height is supported (from 1 OU and up to 10 OU)

39 Open Rack: Power Power Zones: One power zone is one equipment bay (10 OU) + one power shelf (3 OU) Front view Power Zone 13 OU Equipment Bay 10 OU

40 Open Rack: Power Power Zones: One power zone is one equipment bay (10 OU) + one power shelf (3 OU) Three power zones in the column, 13 OU high (each) Front view Power Zone 13 OU Equipment Bay 10 OU

41 Open Rack: Power Power Zones: One power zone is one equipment bay (10 OU) + one power shelf (3 OU) Three power zones in the column, 13 OU high (each) Each power zone has a max of 3 bus-bar pairs (+/-) for 12V distribution, rated 350A max (4.2KW) Power Zone 13 OU Front view Equipment Bay 10 OU

42 Open Rack: Power Power Zones: One power zone is one equipment bay (10 OU) + one power shelf (3 OU) Three power zones in the column, 13 OU high (each) Each power zone has a max of 3 bus-bar pairs (+/-) for 12V distribution, rated 350A max (4.2KW) Power shelf is a single-phase power solution (3x) Power Zone 13 OU Front view Equipment Bay 10 OU

43 Open Rack: Power Power Zones: One power zone is one equipment bay (10 OU) + one power shelf (3 OU) Three power zones in the column, 13 OU high (each) Each power zone has a max of 3 bus-bar pairs (+/-) for 12V distribution, rated 350A max (4.2KW) Power shelf is a single-phase power solution (3x) Up to 7 power supplies per shelf in N+1 configuration Power Zone 13 OU Front view Equipment Bay 10 OU

44 Open Rack 3 bus-bars Efficiency Test Setup Each Motherboard power (Load Tray) is 12V (23.3A) 15 Load Trays each Power Zone (15 x 280W) = 4200W Each bus-bar pair is loaded at 1400W Three bus-bar pairs are used, for each Power Zone TOR CABLES TOR Switches 280W load 280W load 280W load 280W load 280W load 280W load 280W load 280W load 280W load 280W load 280W load 280W load 280W load 280W load 280W load Power Shelf 4.2KW 280W load 280W load 280W load 280W load 280W load 280W load 280W load 280W load 280W load 280W load 280W load 280W load 280W load 280W load 280W load Power Shelf 4.2KW 280W load 280W load 280W load 280W load 280W load 280W load 280W load 280W load 280W load 280W load 280W load 280W load 280W load 280W load 280W load Power Shelf 4.2KW

45 Bus-Bar pair Voltage 117A and Efficiency Worst case tot voltage drop = 44mV V V 0.03W 10 OpenU tall bus-bar pair (1400W tot) V 0.05W This setup is (x3) V 0.07W 1400W 1400W 1400W V 0.18W 1.05W 117A V 3.51W V 4.2KW Power Shelf Total power loss of the whole 12V distribution system at full load, including all CLIPs, is 6.5W Max efficiency loss is 0.5%

46 Open Rack 1 bus-bar pair Efficiency Test Setup Each Motherboard power (Load Tray) is 12V (70A) 5 Load Trays each Power Zone (5 x 840W) = 4200W The central bus-bar pair is loaded at 4200W Only the central bus-bar is used, for each Power Zone TOR CABLES TOR Switches 840W load 840W load 840W load 840W load 840W load Power Shelf 4.2KW 840W load 840W load 840W load 840W load 840W load Power Shelf 4.2KW 840W load 840W load 840W load 840W load 840W load Power Shelf 4.2KW

47 Bus-Bar pair Voltage 350A and Efficiency V V Worst case total voltage drop = 102mV 0.14W 10 OpenU tall bus-bar pair (4200W total) V 0.28W This setup is (x1) V V 1.26W 3.92W 7.70W 350A not used 4200W not used V 19.6W V 4.2KW Power Shelf Total power loss of the whole 12V distribution system, including all CLIPs, is 47.6W Max efficiency loss is 1.15%

48 Open Rack: Bus Bar Clip 12V 100A Clip Clip Assembly Installed to the IT chassis, it plugs directly to the Open Rack 12V bus-bars Installed in a Compute Chassis

49 Open Rack: Bus Bar Clip 12V 100A Clip Clip Assembly Installed to the IT chassis, it plugs directly to the Open Rack 12V bus-bars Contacts are Nickel plated, max 0.3mOhms Installed in a Compute Chassis

50 Open Rack: Bus Bar Clip 12V 100A Clip Clip Assembly Installed to the IT chassis, it plugs directly to the Open Rack 12V bus-bars Contacts are Nickel plated, max 0.3mOhms Rated 80A, 100V, up to +105C Installed in a Compute Chassis

51 Open Rack: Bus Bar Clip 12V 100A Clip Clip Assembly Installed to the IT chassis, it plugs directly to the Open Rack 12V bus-bars Contacts are Nickel plated, max 0.3mOhms Rated 80A, 100V, up to +105C Power loss < 80A when mated to a nickel plated 3mm Copper bus-bar Installed in a Compute Chassis

52 Open Rack: Bus Bar Clip 12V 100A Clip Clip Assembly Installed to the IT chassis, it plugs directly to the Open Rack 12V bus-bars Contacts are Nickel plated, max 0.3mOhms Rated 80A, 100V, up to +105C Power loss < 80A when mated to a nickel plated 3mm Copper bus-bar Floating panel for self blind mating (no extra guiding pins necessary on chassis) Installed in a Compute Chassis

53 Open Rack: Bus Bar Clip 12V 100A Clip Clip Assembly Installed to the IT chassis, it plugs directly to the Open Rack 12V bus-bars Contacts are Nickel plated, max 0.3mOhms Rated 80A, 100V, up to +105C Power loss < 80A when mated to a nickel plated 3mm Copper bus-bar Floating panel for self blind mating (no extra guiding pins necessary on chassis) Chassis maybe installed at any height in the power zone Installed in a Compute Chassis

54 Open Rack: Bus Bar Clip 12V 80A Clip Clip Assembly Installed to the IT chassis, it plugs directly to the Open Rack 12V bus-bars Contacts are Nickel plated, max 0.3mOhms Rated 80A, 100V, up to +105C Power loss < 80A when mated to a nickel plated 3mm Copper bus-bar Floating panel for self blind mating (no extra guiding pins necessary on chassis) Chassis maybe installed at any height in the power zone Released into OCP and available from Methode today Installed in a Compute Chassis

55 Blind Mating: self-guiding pins for blind-mate hot-plug OCP released CLIP assembly AWG 6 cables CM choke Floating Plate: +/- 3mm horizontally +/- 4mm vertically CLIP component (2x) installed on a floating plate

56 Open Rack: Variations Cold Storage Single Power Domain Very Low Power Density

57 Open Rack: Variations EMI Shielded Front EMI panel Front and rear panels Rack-level EMI shielding For locations with this requirement Front and rear panels attach to rack frame with 10mm bolts

58 Open Rack: Variations Seismic Zone 4 Rear Cross Bracing Racks Ganged Together

59 Open Rack Q&A (please state name & company)

60 Break Check out the Open Rack!

61 Creating a Standard Interface between Node and Rack Steve Mills Facebook Mechanical Engineer

62 What was released since last time Open Rack V1.0 Covers the technical detail to produce Open Rack

63 What was released since last OCP Design Guide 0.5 Provides detail on how to create a generic chassis that is compatible with the Open Rack design

64 Design Guide 0.5 Overview Front Side Top

65 Retention Features for Equipment 12x18 mm rectangle Spring Cam Cage latch surface nut

66 Retention Features 5mm tall lance every 48mm

67 Reference Design for Chassis DC Power clip location Chassis size Latching location

68 Creating a Standard Goal: Define the minimal amount of specification to ensure interoperability between the rack and the IT equipment to be installed. Goal: Enable the rack and IT equipment ecosystem to innovate within this well defined interface.

69 What Should be Part of the Standard? DC Power Connection Maximum depth of contact mate on the busbar Electrical properties a function of power draw Alignment Amount of float prior to mate Clip alignment method keep-out zone Cycle life Connection to PWA in the system

70 What Should be Part of the Standard? Busbar Mechanical Location in space relative to retention location Reserved Zone for clip interface Cycle Life Electrical properties Max ambient temperature

71 What Should be Part of the Standard? Equipment retention feature Shape and location of the retention Max forces the location can handle Equipment Support Bracket Width and length of supporting surface Loading rating per pair Retention Method in the rack

72 What Should be Part of the Standard? Rack interfaces to the data center Rack Width Bolt down locations for seismic applications Baying interfaces Hot-aisle containment features

73 Day One is Done! See you 9AM

74 Delta Open Rack Richard Chan

75 Sanmina Open Rack Patrick MacDonald

76 Emerson Open Rack Eric Wilcox

77 Thanks!

78 January 16-17, 2013 Santa Clara

79