The AMD Opteron CMP NorthBridge Architecture: Now and in the Future

Transcription

1 The AMD Opteron CMP NorthBridge Architecture: Now and in the Future Pat Conway & Bill Hughes August, 2006

2 AMD Opteron The Industry s First Native Dual-Core 64-bit x86 Processor Integration: Two 64-bit CPU cores 2MB cache On-chip Router & Memory Controller Bandwidth: Dual channel DDR (128-bit) memory bus 3 HyperTransport (HT) links (16-bit each x 2 GT/sec x 2) Usability and Scalability: Socket compatible: Platform and TDP! Glueless SMP up to 8 sockets Memory capacity & BW scale w/ CPUs Power Efficiency: AMD PowerNow! Technology with optimized power management Industry-leading system level power efficiency 2

3 AMD Opteron The Industry s First Native Dual-Core 64-bit x86 Processor 3

4 A Clean Break with the Past Chip X Chip X Chip X Chip X Chip X Chip X Chip X Chip X MCP MCP MCP MCP MCP MCP MCP MCP SRQ Crossbar Mem.Ctrlr HT SRQ Crossbar Mem.Ctrlr HT USB USB PCI PCI Hub Hub Memory Controller Hub Hub PCI-E PCI-E Bridge PCI-E Bridge PCI-E Bridge PCIe Bridge PCIe TM TM Bridge Bridge SRQ Crossbar Mem.Ctrlr HT 8 GB/S SRQ Crossbar Mem.Ctrlr HT 8 GB/S 8 GB/S XMB XMB XMB XMB XMB XMB XMB XMB PCIe PCIe TM TM Bridge Bridge PCIe PCIe TM TM Bridge Bridge 8 GB/S Hub Hub USB USB PCI PCI Legacy x86 Architecture 20-year old traditional front-side bus (FSB) architecture CPUs, Memory, all share a bus Major bottleneck to performance Faster CPUs or more cores performance AMD64 s Direct Connect Architecture Industry-standard technology Direct Connect Architecture reduces FSB bottlenecks HyperTransport interconnect offers scalable high bandwidth and low latency 4 memory controllers increases memory capacity and bandwidth 4

5 4P System Board Layout 5

6 System Overview 12.8 GB/s 128-bit DRAM MCT Core 0 Core 1 SRI DRAM MCT Core 0 Core 1 SRI ncht HT-HB HT-HB XBAR HT HT XBAR HT HT cht HT HT XBAR HT HT XBAR HT-HB HT-HB 4.0GB/s per 2GT/s Data Rate Core 0 Core 1 SRI MCT Core 0 Core 1 SRI MCT NorthBridge DRAM DRAM 6

7 Northbridge Microarchitecture Overview Core0 Core1 Virtual Channel Use Data Request Read Non-posted Write Y Cache Block Commands Posted Request Posted Writes Y System Request Interface (SRI) APIC (interrupts) Probe Response Broadcast probes Read Response Probe response Completion N Y Crossbar (XBAR) Memory Controller (MCT) DRAM Controller 2 DDR2 channels HT0 HT1 HT2 7

8 Northbridge Command Flow Core 0 Victim Buffer (8-entry) Write Buffer (4-entry) Instruction MAB (2-entry) Data MAB (8-entry) Core 1 All buffers are 64-bit command/address to Core System Request Queue 24-entry Address MAP & GART HT0 Input HT1 Input HT2 Input Memory Command Queue 20-entry to DCT Router Router Router Router Router 10-entry Buffer 16-entry Buffer 16-entry Buffer 16-entry Buffer 12-entry Buffer XBAR HT0 Output HT1 Output HT2 Output 8

9 Northbridge Data Flow Core 0 All buffers are 64-byte cache lines HT0 input HT1 input HT2 input Core 1 Victim Buffer (8-entry) Write Buffer (4-entry) from Host Bridge from DCT 8-entry Buffer 8-entry Buffer 8-entry Buffer 5-entry Buffer 8-entry Buffer XBAR XBAR HT0 output HT1 output HT2 output System Request Data Queue 12-entry Memory Data Queue 8-entry to Core to Host Bridge to DCT 9

10 Lessons Learned #1 Allocation of XBAR Command buffer across Virtual Channels can have big impact on performance MP traffic analysis gives the best allocation e.g. Opteron Read Transaction Request (2 visits) Probe (3 visits) Response (8 visits) Memory 0 3: RD 4: D 8:SD 5: D 4: RP0 3: PI1 3: PI2 P 0 P 2 3: PI0 P 1 2: RD 4: PI1 5: RP1 7: SD 1: RD P 3 Memory 2 6: D 4: RP2 5: RP0 Memory 1 Memory 3 10

11 Lessons Learned #2 Memory Latency is the Key to Application Performance! System Performance Performance (single 2.8GHz core, 400MHz vs DDR2 Average PC3200, 2GT/s HT Memory with 1MB cache in MP Latency system) (single 2.8GHz 7 core, 400MHz DDR2 PC3200, 2GT/s HT with 1MB cache 120% in MP system) System Performance OLTP1 2 OLTP2 1 SW99 0 SSL JBB Performance vs Average Memory Latency OLTP1 OLTP2 SW99 SSL JBB 1N 2N 4N (SQ) 8N (TL) 8N (L) 1 Node P0 P0 P1 4 Node Square 1N 2N 4N (SQ) 8N (TL) 8N (L) AvgD 0 hops 1 hops 1.8 hops Latency x + 0ns x + 44ns (124 cpuclk) x + 105ns (234 cpuclk) P2 P3 P0 P1 P2 P3 100% 80% 60% 40% 20% 0% 8N Ladder P4 Processor Performance P5 P6 P7 120% 100% 80% 60% 40% 20% 0% Processor Performance 1 Node 4 Node Square P0 8N Twisted Ladder P2 P4 P6 8N Ladder P0 P0 2 Node P0 P1 P2 P1 P3 P5 P7 P0 P2 P4 P6 0.5 hops 1.5 hops P1 P3 x + 17ns (47 cpuclk) x + 76ns (214 cpuclk) P1 P3 P5 P7 11

12 Looking Forward

13 HyperTransport -based Accelerators Imagine it, Build it Open platform for system builders ( Torrenza ) 3rd Party Accelerators Media FLOPs XML SOA AMD Opteron Socket or HTX slot HyperTransport interface is an open standard see hypertransport.org Coherent HyperTransport interface available if the accelerator caches system memory (under license) Other Other Optimized Silicon Silicon SOA Accelerator PCI-E PCI-E Bridge Bridge Hub Hub 8 GB/S 8 GB/S 8 GB/S 8 GB/S Native Dual-Core SRQ Crossbar Mem.Ctrlr HT USB USB PCI PCI XML Accelerator Native Dual-Core SRQ Crossbar Mem.Ctrlr HT FLOPs Accelerator 13

14 AMD s Next Generation Processor Technology Ideal for 65nm SOI and beyond Native quad core die Expandable shared L3 cache IPC enhanced CPU cores 32B instruction fetch Improved branch prediction Out-of-order load execution Up to 4 DP FLOPS/cycle Dual 128-bit SSE dataflow Dual 128-bit loads per cycle Bit Manipulation extensions (LZCNT/POPCNT) SSE extensions (EXTRQ/INSERTQ, MOVNTSD/MOVNTSS) Enhanced Direct Connect Architecture and Northbridge Four ungangable x16 HyperTransport TM links (up to 5.2GT/sec) Enhanced crossbar Next-generation memory support FBDIMM when appropriate Enhanced power management and RAS 14

15 Balanced, Highly Efficient Cache Structure Efficient memory handling reduces the need for brute force cache sizes L1 L3 Core 1 Cache Control C2 64KB L1 512KB 2+MB Dedicated L1 Locality keeps most critical data in the L1 cache Low latency bit data paths 2 loads per cycle C3 L1 C4 L1 15

16 Balanced, Highly Efficient Cache Structure Efficient memory handling reduces the need for brute force cache sizes L1 L3 Core 1 Cache Control 64KB 512KB 2+MB Dedicated C2 C3 L1 L1 Sized to accommodate the majority of working sets today Dedicated to help eliminate conflicts common in shared caches C4 L1 16

17 Balanced, Highly Efficient Cache Structure Efficient memory handling reduces the need for brute force cache sizes L1 L3 Core 1 Cache Control 64KB 512KB 2+MB Shared L3 Coming Soon C2 C3 L1 L1 Allocation policy which optimizes movement, placement and replication of data for multi-core Ready for expansion C4 L1 17

18 Additional HyperTransport Ports Enable Fully Connected 4 Node (four x16 HT) and 8 Node (eight x8 HT) Reduced network diameter Fewer hops to memory Increased Coherent Bandwidth more links cht packets visit fewer links HyperTransport3 Benefits Low latency because of lower diameter Evenly balanced utilization of HyperTransport links Low queuing delays Four x16 HT OR Low latency under load Eight x8 HT 18

19 4 Node Performance 4 Node Square 4 Node fully connected P0 16 P2 P0 16 P2 P1 P3 P1 P3 4N SQ (2GT/s HyperTransport) Diam 2 Avg Diam 1.00 XFIRE BW 14.9GB/s + 2 EXTRA LINKS 4N FC (2GT/s HyperTransport) Diam 1 Avg Diam 0.75 XFIRE BW 29.9GB/s (2X) W/ HYPERTRANSPORT3 4N FC (4.4GT/s HyperTransport3) Diam 1 Avg Diam 0.75 XFIRE BW 65.8GB/s (4X) 19 XFIRE ( crossfire ) BW is the link-limited all-to-all communication bandwidth (data only)

20 8 Node Performance 8 Node Fully Connected P0 8N Twisted Ladder 16 P2 P4 P P Node 8 6HT 2x4 2x P2 1 2 P0 P1 8 P2 P3 OR P0 P5 P4 8 P7 P1 P3 P5 P P P P4 P5 P6 P7 P1 P2 P3 P6 8N TL (2GT/s HyperTransport) Diam 3 Avg Diam 1.62 XFIRE BW 15.2GB/s 8N 2x4 (4.4GT/s HyperTransport3) Diam 2 Avg Diam 1.12 XFIRE BW 72.2GB/s (5X) 8N FC (4.4GT/s HyperTransport3) Diam 1 Avg Diam 0.88 XFIRE BW 94.4GB/s (6X) 20

21 Why Quad-Core? x Max Frequency Performance Power Baseline is 2 Node x 2 Core blade running OLTP 21

22 Increasing Frequency x x Increased Freq (+16%) 1.00x Max Frequency Performance Power Baseline is 2 Node x 2 Core blade running OLTP 22

23 Decreasing Frequency x x 1.00x 0.87x x Performance Increased Freq Max Frequency Decreased Freq Power (+16%) (-16%) Baseline is 2 Node x 2 Core blade running OLTP 23

24 Quad-Core Higher Performance within a Fixed Power Budget x 1.73x x 1.00x 0.99x Quad-Core Performance Increased Freq Max Frequency Decreased Freq Power (+16%) (-16%) Baseline is 2 Node x 2 Core blade running OLTP 24

25 Clock and Power Planes Core 0 Core 1 VDD PLL Core 2 PLL PLL Core 3 PLL VRM DIMMs SVI Misc VDDIO, VTT PLLs Northbridge PLL VDDA VHT PLLs HyperTransport Links VDDNB 25

26 DICE: Dynamic Independent Core Engagement Ability to dynamically and individually adjust core frequencies to improve power efficiency 100% Workload 100% Workload 100% Power State 100% Workload 100% Workload 26

27 DICE: Dynamic Independent Core Engagement Ability to dynamically and individually adjust core frequencies to improve power efficiency 100% Workload 33% Workload 60% Power State 33% Workload 33% Workload 27

28 DICE: Dynamic Independent Core Engagement Ability to dynamically and individually adjust core frequencies for improved power efficiency 100% Workload 50% Workload Halted Halted 45% Power State 28

29 Enjoy the rest of the conference! 29

30 Trademark Attribution AMD, the AMD Arrow, AMD Opteron, AMD PowerNow logo and combinations thereof are trademarks of Advanced Micro Devices, Inc. HyperTransport and HTX are trademarks of the HyperTransport Consortium. PCIe is a trademark of the PCI-SIG. Other names used in this presentation are for informational purposes only and may be trademarks of their respective owners. 30