Table Of Contents. Page 2 of 26. *Other brands and names may be claimed as property of others.

Transcription

1 Technical White Paper Revision 1.1 4/28/10 Subject: Optimizing Memory Configurations for the Intel Xeon processor 5500 & 5600 series Author: Scott Huck; Intel DCG Competitive Architect Target Audience: IT Managers/Server End Users (NDA not required) Legal Disclaimer: This document contains information on products in the design phase of development. The information here is subject to change without notice. Do not finalize a design with this information. Contact your local Intel sales office or your distributor to obtain the latest specification before placing your product order. INFORMATION IN THIS DOCUMENT IS PROVIDED IN CONNECTION WITH INTEL PRODUCTS. EXCEPT AS PROVIDED IN INTEL S TERMS AND CONDITIONS OF SALE FOR SUCH PRODUCTS, INTEL ASSUMES NO LIABILITY WHATSOEVER, AND INTEL DISCLAIMS ANY EXPRESS OR IMPLIED WARRANTY RELATING TO SALE AND/OR USE OF INTEL PRODUCTS, INCLUDING LIABILITY OR WARRANTIES RELATING TO FITNESS FOR A PARTICULAR PURPOSE, MERCHANTABILITY, OR INFRINGEMENT OF ANY PATENT, COPYRIGHT, OR OTHER INTELLECTUAL PROPERTY RIGHT. Intel products are not intended for use in medical, life saving, or life sustaining applications. Intel may make changes to specifications, product descriptions, and plans at any time, without notice. All products, dates, and figures are preliminary for planning purposes and are subject to change without notice. Designers must not rely on the absence or characteristics of any features or instructions marked reserved or undefined. Intel reserves these for future definition and shall have no responsibility whatsoever for conflicts or incompatibilities arising from future changes to them. The Intel Xeon processors may contain design defects or errors known as errata which may cause the product to deviate from published specifications. Current characterized errata are available on request. The code names Penryn, Harpertown, Dunnington, Nehalem, Westmere presented in this document are only for use by Intel to Identify products, technologies, and services in development, that have not been made commercially available to the public, i.e., announced, launched or shipped. They are not commercial names for products or services and are not intended to function as trademarks. Copies of documents which have an order number and are referenced in this document, or other Intel literature, may be obtained by calling , or by visiting Intel s website at Performance tests and ratings are measured using specific computer systems and/or components and reflect the approximate performance of Intel products as measured by those tests. Any difference in system hardware or software design or configuration may affect actual performance. Buyers should consult other sources of information to evaluate the performance of systems or components they are considering purchasing. For more information on performance tests and on the performance of Intel products, visit or call (U.S.) or Intel, Xeon, Core are trademarks or registered trademarks of Intel Corporation or its subsidiaries in the United States and other countries. Copyright , Intel Corporation. All rights reserved. Page 1 of 26

2 Table Of Contents 1 Introduction/Objective Revision History Background Summary DIMM Differences UDIMM vs. RDIMM vs. FB-DIMM DDR2 vs. DDR3 vs LV-DDR DIMM Rank (Single vs. Dual vs. Quad) Memory Population Options DDR3 UDIMM Memory Population Options DDR3 RDIMM Memory Population Options Supported Memory Frequency by processor SKUs Memory RAS Features x4 and x8 SDDC Memory Lockstep Memory Mirroring Memory Lockstep vs. Memory Mirroring Memory Population for Best Performance Definitions and Guidance Balanced Memory Configurations Near Balanced Memory Configurations Un-Balanced Memory Configurations Best 4GB Solutions Memory Interleaving Performance based on Memory Bandwidth Other Memory Performance Details step guide to picking the optimal memory configuration Memory Population Guidelines Intel Xeon processor 5600 series differences Summary References Page 2 of 26

3 1 Introduction/Objective The purpose of this paper is to answer a broad range of questions around the memory subsystem of the Intel Xeon processor 5500 & 5600 series (code named Nehalem EP & Westmere EP respectively). Topics include, but are not limited to, how to populate memory, which memory population options provide the best power or performance, details about the various RAS features etc. Unless noted otherwise, the information in this document applies to both the 5500 series and 5600 series processors. Chapter 8 addresses how memory sub-system is different between the 5500 series and the 5600 series processors. 1.1 Revision History Revision History Revision Changes Date 1.0 Initial Release 8/10/ Added Chapter 8 to address differences between the 5500 and 5600 series processors, removed references to MetaRAM (no longer supported). Tables updated/added to reflect differences between the 5500 and 5600 series processors (changes noted in pink text) 4/28/ Background The Intel Xeon processor 5500 series (code named Nehalem EP) has an integrated memory controller and three channels of DDR3 memory. Since many of these features are a first for Intel, the existing rules of thumb for memory optimizations may not be valid, so this document will identify how to populate memory and how to populate for best performance. Figure 1 shows the basic block diagram of a typical 2-socket Xeon 5500 series platform (processor and memory). Intel Xeon Processor 5500 Series QPI Intel Xeon Processor 5500 Series Memory Channels CH 0 CH1 CH2 Memory Channels CH 0 CH1 CH2 DIMM 2 DIMM 2 DIMM 2 DIMM 2 DIMM 2 DIMM 2 DIMM 1 DIMM 1 DIMM 1 DIMM 1 DIMM 1 DIMM 1 DIMM 0 DIMM 0 DIMM 0 DIMM 0 DIMM 0 DIMM 0 Figure 1: Memory sub-system block diagram (Intel Xeon Processor 5500 series) Page 3 of 26

4 1.3 Summary Below is a list of the various topics covered in this document. - Difference between DDR2 and DDR3 (and FB-DIMM) - Difference between UDIMMs and RDIMMs - What are Ranks? - Supported memory population options - Supported memory RAS features - Differences in memory support for the Intel Xeon processor 5500 and 5600 series - Populating for best performance based on various attributes including application, capacity, price, bandwidth 2 DIMM Differences This section discusses the various DIMMs which can be supported in the Xeon 5500 series platform, and how they are the same/different from each other and other types of memory. 2.1 UDIMM vs. RDIMM vs. FB-DIMM Memory DIMMs are made up of primarily memory devices plus a few other devices. It is these other devices, which distinguish unbuffered memory (UDIMMS) from registered memory (RDIMMS) and from fully-buffered memory (FB-DIMM). Below is a brief table summarizing the difference between DIMMs based on DDR2 technology. Differences between DDR2 and DDR3 are detailed in the next section. Feature UDIMM (DDR2) RDIMM (DDR2) FB-DIMM (Gen 1) Device Used DDR2 DDR2 DDR2 Buffered Address & No Yes Yes Command Signals? Buffered Outputs No No Yes Available with ECC? Yes Yes Yes Available w/o ECC? Yes No No On-board clock device No No Yes Data Interface Parallel Parallel Serial Power ~5W UDIMM + ~.75W RDIMM + 3-5W** Table 1 -- UDIMM vs. RDIMM vs. FB-DIMM ** Rough approximation. Power will vary by DIMM manufacturer and configuration. In general the main difference between UDIMMs and RDIMMs, is the address and command inputs are buffered in a register before being sent to the devices (see Figure 1 and Figure 2 below). Figure 2 shows a simplified bus layout for a UDIMM device. The figures below show two DIMMS connected on a single channel. Other memory channels are not shown for simplicity, but would be identical. Page 4 of 26

5 Processor/ Mem Controller Memory Data Bus (Channel 0) 72 Memory Channel 0 Memory Address & CS (Channel 0) Figure 2 -- UDIMM Bus Connections Processor/ Mem Controller Memory Data Bus (Channel 0) 72 Memory Channel 0 Memory Address & CS (Channel 0) Register Register Figure 3 -- RDIMM Bus Connections (note the registers) Each memory channel has a set of address and command/chip Select (CS) signals which go to each DIMM. In a UDIMM these signals connect directly to each device on the DIMM. Thus the electrical loading on these signals is quite high (Figure 2). In comparison, on a RDIMM device (Figure 3), there is a register which buffers the address and command signals thus reducing the electrical loading on these signals. By reducing the electrical loading from each Page 5 of 26

6 DIMM, more DIMMS can be populated on a given memory channel. Typically, the max number of UDIMM s per channel is 2. Where as RDIMMs can typically support up to 4 DIMMs/channel. Although these numbers may vary with frequency (fewer DIMMs per channel as the memory frequency goes up). Buffering the address and command signals with the register adds 1 cycle of latency to a memory request. Although if you have numerous requests pipelined, this latency typically can be hidden. While FB-DIMMs use the same DDR2 devices, the similarities stop there. FB-DIMM s use a completely different interface on the outputs than UDIMM or RDIMMs, and have an entirely different protocol for tracking memory requests. For the sake of time, I will not discuss the details of how FB-DIMMs work, but if interested, you may refer to another Tech-Tid-bit which will provide these details (see the reference section for link to the FB-DIMM document) UDIMMs are lowest in power since they have the fewest devices. RDIMMs typically are ~0.75W more power than an equivalent UDIMM. FB-DIMMs have he most devices and thus dissipate the most power. On average, an FB-DIMM dissipates 3-5W more power than an equivalent RDIMM. All three types of memory (based on DDR2 technology), use essentially the same DDR2 connector, however the DIMM key location is different for FB-DIMM technology preventing the user from inserting UDIMMs or RDIMMS into a motherboard designed to support FB-DIMMs. A given platform can support both UDIMMs or RDIMMs (if designed as such). But not both at the same time. If a platform is designed to support FB-DIMMs it can only support FB-DIMMs (UDIMMs and RDIMMs are not supported). 2.2 DDR2 vs. DDR3 vs LV-DDR3 The key differences between DDR2 and DDR3 are lower voltage, and higher frequency. LV- DDR3 is even lower voltage. The table below summarizes these differences: Feature DDR2 DDR3 LV-DDR3** Voltage 1.8V 1.5V 1.35V On-Die Termination Yes Yes Yes On-Die Thermal Sensor No Yes Yes Separate Vref for Address No Yes Yes & Data Adjustable Output No Yes Yes Impedance Frequencies Supported (per JEDEC spec may vary by platform) 400, 533, 667, , 1066, 1333, , 1066, 1333, 1600 **Supported by the Intel Xeon processor 5600 Series (not supported on the Intel Xeon processor 5500 Series.) Table 2 -- DDR2 vs. DDR3 Most of the differences between DDR2 and DDR3 improve the electrical signaling properties and thus allow the part to run at the higher frequencies. Power is not so cut and dry. If both DDR2 and DDR3 DIMMs were operating at the same frequency (800MHz), then DDR3 would have lower power. However since most DDR3 DIMMs will be operating at higher frequencies, typically a 1066MHz DDR3 DIMM will dissipate slightly more power than a DDR2 667MHz DIMM. As you go to higher frequencies, the DDR3 DIMM will typically dissipate little more power. LV-DDR3 DIMMs run at a lower voltage than standard DDR3 DIMMs. The lower voltage allows the DIMMs to dissipate lower power. LV-DDR3 DIMMs are available in both UDIMM and RDIMM configurations. Power savings will vary by DIMM/DIMM manufacturer, but you should expect at least the following power savings per DIMM: - ~ 0.5 Watts (at idle) - ~ Watts (at 100% load) Page 6 of 26

7 2.3 DIMM Rank (Single vs. Dual vs. Quad) DIMMs can be designed as either single rank (SR), dual rank (DR) or quad rank. The differences are essentially how many rows of device are used to provide a given DIMM capacity. Single Rank: One device is connected to a given data bus pin Dual Rank: Two devices are connected to a given data bus pin Quad Rank: Four devices are connected to a given data bus pin In a single Rank DIMM, one x8 device, for example, will be responsible for driving 8 of the 64 data bits. (See Figure 4) In a dual rank DIMM, two x8 devices will be connected to the same 8 data bits, and based on a chip select signal, will determine which drives the data onto the data bus. (See Figure 4) In a quad rank DIMM, four x8 devices will be connect to the same 8 data bits, and based on chip select signal, will determine which drives the data onto the bus. (See Figure 5) The effect of a more ranks on a DIMM is a higher electrical loading on the Address/command, and data buses (Quad more than Dual, dual more than single), but slightly lower power (Quad less than dual dual less than a single rank) since only half of the devices on the DIMM need to be active at any given time. A dual rank DIMM also allows you to have twice the amount of memory on a DIMM (vs. a single rank DIMM using the same memory technology) (and Quad twice as much as a dual). More ranks mean you have higher electrical loading in a given system. Dependent on the maximum frequency supported by the platform, you often cannot populate as many quad rank DIMMs as dual rank DIMMs and/or must run at lower frequencies with QR DIMMs vs. DR DIMMs. Figure 4 -- Typical connection for SR and DR DIMMs (ECC w/x8 ) Page 7 of 26

8 Figure 5 -- Typical connection for QR DIMMs (ECC w/x8 ) See Figures 4 & 5 for the typical layout of a single, dual and quad rank DIMM. Note that the physical size of the DIMM module does not grow as shown in the figures. The physical size of the DIMM module remains the same for all SR, DR and QR DIMMS, however to fit all of the DIMMs required by a DR or QR DIMMs, the DIMMs will often be placed on both sides of the DIMM module and/or sometimes be stacked on top of each other. One of the advantages of a DR or QR DIMM is you can often use older, less dense and thus cheaper memory devices to create a large capacity memory module. Other advantages of a DR and QR DIMM is they are often lower power (because only ½ or ¼ of the devices are being accessed at any given time) and they also tend to have higher performance, since you can have multiple pages simultaneously open, thus reducing latency. 3 Memory Population Options The Intel Xeon processor 5500 & 5600 series support a number of different memory configurations. And while these numerous options allow the customer to customize the memory configuration to their advantage, the many options often leave people paralyzed, since they don t know which is best for their particular situation. When reading this section, and helping your customer to decide which configuration is best, remember that it doesn t have to be confusing. The many options available are meant to allow the customer to determine exactly which configuration is optimal for their situation based on the various criteria including capacity, bandwidth, price and power. And there are probably many right answers for each customer. Memory configuration caveats: 1. The configuration information below is what Intel designed the components to support. The actual configurations supported by each platform, by each OEM may vary (some may support more or less than shown here). You will have to confirm with the OEM to determine if a specific memory configuration is supported on their particular platform. 2. Memory support tends to change over time and newer products become available, additional validation is done, etc. The below information is valid as of April 17, The platform supports the following different memory module options: - DDR3 Unbuffered DIMMs (ECC or non-ecc) (aka UDIMM) Page 8 of 26

9 - DDR3 Registered DIMMs (ECC Only) (aka RDIMM) - DDR3 800, 1066 or 1333MHz - Single Rank, Dual Rank or Quad Rank DIMMs (QR only on RDIMMs) - 1, 2, 4, 8 or 16GB DIMMs (16GB was added ~4/17/09) - DDR3 MetaS (MetaRAM) is no longer supported by this platform The sections below provide additional details on exactly which DIMMS are supported and in what configurations, frequencies etc. 3.1 DDR3 UDIMM Memory Population Options DDR3 UDIMM & LV DDR3 UDIMM Detailed Memory Population Options (with or w/o ECC) Platforms with UDIMM Routing Only (max 2 DIMM slots on board) # DIMMS populated per channel Supported Speeds (MHz) Ranks per DIMM (any combination) Population Rules 1 800, 1066, 1333 SR or DR 1. Any combination of x8 and x16 UDIMMs, with 2 800, 1066, 1333** SR, DR (Mixing okay) 1Gb or 2Gb density. 2. Populate DIMMs starting with slot 0 (slot furthest from the CPU). ** MHz only supported on Intel Xeon processor 5600 series with 1.5V DIMMs. Platforms with Combo UDIMM / RDIMM Routing (max 3 DIMM slots on board) # DIMMS populated per channel Supported Speeds (MHz) Ranks per DIMM (any combination) Population Rules 1 800, 1066, 1333 SR or DR 1. Any combination of x8 UDIMMs, with 1Gb or 2 800, 1066, 1333** SR, DR (Mixing okay) 2Gb density. 2. Populate DIMMs starting with slot 0 (slot furthest from the CPU). ** MHz only supported on Intel Xeon processor 5600 series with 1.5V DIMMs. Table 3 -- DDR3 UDIMM Population Options Additional Notes on UDIMM Population Rules: - 256Mb, 512Mb, 4Gb technologies are NOT supported - x4 on UDIMM are NOT supported - Quad rank UDIMM are NOT supported x16 is NOT supported on combo routing (routing that supports either UDIMM or RDIMM) All channels in a system will run at the fastest common frequency No mixing of registered and unbuffered DIMMs Mixing ECC & non-ecc UDIMMs anywhere on platform will force system to run in non-ecc mode No RAS support for non-ecc UDIMMs Supported UDIMMs are available in 1GB, 2GB and 4GB sizes No x4 SDDC support with UDIMM w/ecc, however, x8 SDDC is supported in lock step mode with x8 UDIMMs w/ecc Above table and notes are valid for 1.35V LV UDIMMs with the following additional notes: - LV-UDIMMs are only supported on Intel Xeon processor 5600 series - No plans to validate LV UDIMMs w/o ECC - If 1.35V and 1.5V DIMMs are mixed, the DIMMs will run at 1.5V MHz is NOT supported with LV-UDIMMs. Page 9 of 26

10 3.2 DDR3 RDIMM Memory Population Options # DIMMS populated per channel DDR3 RDIMM Detailed Memory Population Options (only with ECC) Supported Speeds (MHz) Ranks per DIMM (any combination) Population Rules 1 800, 1066, 1333 SR or DR 1. Any combination of x4 and x8 RDIMMs, 1 800, 1066 QR Only with 1Gb or 2Gb density. 4Gb density supported only on Intel Xeon processor 5600 series , 1066, 1333** SR, DR (Mixing okay) SR, DR, QR (Mixing okay) SR, DR (Mixing okay; no QR) Table 4 -- DDR3 RDIMM Population Options 2. Populate DIMMs starting with slot 0 (slot furthest from the CPU). ** MHz only supported on Intel Xeon processor 5600 series with 1.5V DIMMs. Additional Notes on RDIMM Population Rules: - 256Mb, 512Mb technologies are NOT supported - 4Gb technologies are supported only on only Intel Xeon processor 5600 series - x16 on RDIMM are NOT supported All channels in a system will run at the fastest common frequency No mixing of registered and unbuffered DIMMs Mixing quad rank RDIMMs in one channel and 3 DPC in another on same CPU socket is not supported If a QR RDIMM is mixed with a SR or DR DIMM in a channel, QR DIMM must be populated in slot 0 (furthest from the CPU) Supported RDIMMs are available in 1GB, 2GB, 4GB, 8GB and 16GB sizes As noted above, QR DIMMS are not supported at 1333MHz (only 800 and 1066MHz) and QR DIMMs are not supported with 3 DPC (only 1 or 2 DPC). This is because of the higher electrical loading of the QR DIMMs. All 1333MHz QR DIMMs, if installed, will be clocked down to 1066MHz (1 DPC) or 800MHz (2 DPC) Page 10 of 26

11 # DIMMS populated per channel LV DDR3 RDIMM (Intel Xeon processor 5600 series only) Detailed Memory Population Options (only with ECC) Supported Speeds (MHz) Ranks per DIMM (any combination) Population Rules 1 800, 1066, 1333 SR or DR 1. Any combination of x4 and x8 RDIMMs, QR Only with 1Gb or 2Gb density , Populate DIMMs starting with slot 0 (slot SR, DR furthest from the CPU). (Mixing okay) SR, DR, QR (Mixing okay) Table 5 -- LV DDR3 RDIMM Population Options (Xeon 5600 only) Additional Notes on RDIMM Population Rules: - 256Mb, 512Mb and 4Gb technologies are NOT supported - x16 on RDIMM are NOT supported All channels in a system will run at the fastest common frequency No mixing of registered and unbuffered DIMMs If a QR RDIMM is mixed with a SR or DR DIMM in a channel, QR DIMM must be populated in slot 0 (furthest from the CPU) Supported RDIMMs are available in 1GB, 2GB, 4GB, 8GB and 16GB sizes LV RDIMMS are supported only at 1 and 2 DPC. 3 DPC is not supported with LV RDIMMs As noted above, QR DIMMS are not supported at 1333MHz or 1066MHz (only 800MHz) and QR DIMMs are not supported with 3 DPC (only 1 or 2 DPC). This is because of the higher electrical loading of the QR DIMMs. All 1333MHz or 1066MHz QR DIMMs, if installed, will be clocked down to 800MHz 3.3 Supported Memory Frequency by processor SKUs The Intel Xeon processor 5500 and 5600 series have a number of different models or SKUs. Not all SKUs support all memory frequencies. Tables 6 & 7, below, show the different SKUs, and the memory frequencies supported by each SKU for the Intel Xeon processor 5500 & 5600 series respectively. Core Frequency Memory Frequency Model (GHz) 800MHz 1066MHz 1333MHz X Yes Yes Yes X Yes Yes Yes X Yes Yes Yes E Yes Yes No E Yes Yes No E5520/L Yes Yes No E Yes No No E Yes No No E5504/L Yes No No E Yes No No E Yes No No Table 6 -- Intel Xeon processor 5500 series memory support options (by SKUs) Page 11 of 26

12 Core Frequency Memory Frequency Model (GHz) 800MHz 1066MHz 1333MHz X Yes Yes Yes X Yes Yes Yes X Yes Yes Yes X Yes Yes Yes X Yes Yes Yes X Yes Yes Yes W Yes Yes Yes W Yes Yes Yes W Yes Yes Yes W Yes Yes Yes L Yes Yes Yes L Yes Yes No L Yes Yes No E Yes Yes No E Yes Yes No E Yes Yes No Table 7 -- Intel Xeon processor 5600 series memory support options (by SKUs) 4 Memory RAS Features The Intel Xeon processor 5500 & 5600 series supports the following memory RAS features - x4 SDDC - x8 SDDC - Lock Step - Mirroring Notes: 1. Mirrored mode and lockstep mode both require that channel 0 and channel 1 are exact copies of each other Channels must have DIMMs that are identical in organization (#ranks, #rows, #cols, #banks) Can run with differently rated DIMMs and use the settings of the slowest DIMM on both channels Channel 2 is not usable in either of these modes 2. With memory installed behind both sockets of a DP system, both sockets must be running the same RAS mode and paging policy - Only exception is when mirroring redundancy is lost on one socket, this will not force the loss on the other socket 3. Memory sparing was initially going to be supported on the Xeon 5500 series platform, but was dropped. Sparing is expected to return with the Westmere EP platform. 4.1 x4 and x8 SDDC SDDC (Single Device Data Correction) is a memory RAS feature which allows for an entire device to fail on a DIMM, yet the system will continue to operate as normal without any loss of data. The Intel Xeon processor 5500 series supports x4 and x8 SDDC, however x8 SDDC is only supported in lockstep mode. SDDC x4 or x8 is supported on UDIMMS with ECC or RDIMMS. x4 SDDC means that a 4-bit wide (x4) device may fail, and the system will continue to operate normally. x8 SDDC means that an 8-bit wide (x8) device (or 2 x4 devices) may fail and the system will continue to operate normally. Most DIMMs today are made up of either x4 or x8 devices. x4 SDDC can only be supported if all the DIMMS installed use x4 devices. x8 SDDC can be supported if the DIMMS installed all use x4 or x8 devices. Page 12 of 26

13 4.2 Memory Lockstep Multiple memory channels are accessed simultaneously, but different contents written to each. Typically a cache line is split across both memory channels, so different parts of the same cache line are stored in multiple places. When reading from memory, the processor then needs to reassemble the cache lines which add additional latency. Figure 6 below shows a typical lockstep configuration. Lockstep Ch 0 & 1 operate in lockstep Ch 2 unused CPU Figure 6 -- Typical Memory Lockstep Configuration (1 processor shown) The following apply when in memory lockstep mode: - 2 channels operating in lockstep (cache line is split across both channels), 3rd channel is unused - Both channels must be populated identically - x8 SDDC is supported (x8 SDDC is supported only in lockstep mode) - No mirroring or sparing support in this mode - Vs. a 3-channel configuration with only x4 SDDC, reliability increases, but performance drops and capacity drops (only 2 channels used) 4.3 Memory Mirroring Memory contents are written to two different places in physical system memory. 100% redundancy is assured, but it comes at cost of reduced capacity and performance. When memory mirroring is enabled (as shown in Figure 7), the memory capacity of the system is on ½ of what is populated. Memory performance will be reduced when memory mirroring is enabled since effectively, only 1 channel of memory is used (the second is used for redundancy). Both mirroring and lock-step will have reduced performance vs. all other configurations. However we believe mirroring will have slightly better performance than lock-step. Mirroring Ch 2 unused CPU Figure 7 -- Typical Memory Mirroring Configuration (1 processor shown) The following apply when in memory mirroring mode: - 2 memory channels operating as mirrors of each other o Same content is written to both channels simultaneously - Both channels must be populated identically - Only half the populated memory is usable as system memory - No sparing, lockstep or x8 SDDC in this mode (x4 SDDC is supported) - Increased reliability vs. lockstep, but max memory capacity drops (you effectively have only one channel of memory both from a capacity standpoint) - Higher reliability than memory lockstep or x4 SDDC Page 13 of 26

14 4.4 Memory Lockstep vs. Memory Mirroring So what is the difference, and why choose one of the other? Both mirroring and lockstep provide higher reliability than the other memory configurations (which only provide x4 SDDC). However, with the additional reliability, there are tradeoffs. Both mirroring and lockstep will reduce the performance of the memory subsystem and the amount of memory available in the system by 2/3 for mirroring and by 1/3 for lockstep. Lockstep is probably lower performance (vs. mirroring). The real difference is that lockstep will allow the system to run un-interrupted if a single x8 (or 2 x4) devices on a DIMM fail, while memory mirroring will continue to operate, uninterrupted, if an entire DIMM fails. Thus Mirroring is a higher reliability solution. 5 Memory Population for Best Performance When trying to find the optimal memory configuration for a given application, there are many variables to consider including: Capacity, cost, power and application being run on the server. Because there are so many variables, it is often difficult to create simple guidelines to determine exactly which memory configuration is best for each application. For example, how sensitive is your application to memory? Is it most sensitive to overall capacity, latency or memory bandwidth. Depending on the application s sensitivities, different memory configurations may have different results. And which processor is being used determines which memory frequencies can be supported. Only the X W and one L series processors support 1333MHz memory. If you are choosing to use an E series processor, then 1066MHz or 800MHz is the max frequency supported and different options may provide better performance. Caveats: When we say one configuration has better performance, sometime it may be a few percent better, other times it may be 20-30% better. So while one configuration may provide say 3-5% better performance, it may cost 40% more. Is the added performance worth it? It is up to the user to decide. That said, the sections below attempt to provide some guidelines and examples on how to choose the best memory configuration for different situations. 5.1 Definitions and Guidance For this document we will refer to a memory configuration as either being balanced, unbalanced or near-balanced. Below is a brief description of each. See the sections below for more detailed definitions. Memory Configuration All DIMMs Identical All Memory Channels Populated All Channels Identically Populated Balanced Yes Yes Yes Un-Balanced Maybe Maybe No Near-Balanced No Yes Yes Table 8 -- DIMM Configuration Definitions The recommendation, for best performance, is to populate memory in a balanced configuration. The reason is that a balanced configuration will not only provide the best memory sub-system performance, but will ensure all regions of memory have identical performance. When populating in an un-balanced memory configuration, it is likely that some regions of memory will have different performance than other regions of memory. This is due to the fact that if the memory configuration is not balanced then the memory controller will need to configure different regions of memory differently. Page 14 of 26

15 For example, some regions may be configured with a higher performance 3-way interleaving scheme, while other regions of memory may need to be configured with a lower performing 1-way interleaving scheme (the details of why this is done is beyond the scope of this document, however section 5.5 does discuss why higher memory interleaving provides higher performance). Consequently, some regions of memory might perform differently than other regions of memory. How an application is affected by this varied memory performance is unknown and may vary depending on what portion of the application is assigned which what region of memory, and/or may change over time as the memory footprint of the server, OS and other applications change over time. Say a database stretches across different performing memory regions, and one portion of the database may perform differently than another portion of the same database. Suffice it to say, the more unbalanced the memory configuration gets, the lower performance one should expect from the memory sub-system. 5.2 Balanced Memory Configurations If at all possible, for best performance all memory channels (for both processors) should be populated, with identical number/size/speed/ranked DIMMs. Such memory configurations we call balanced and are shown in Table 9 for UDIMMs and Table 10 for RDIMMs. These configurations should be used if at all possible, and will usually provide the best performance for almost any application. Which configuration you choose depends on the capacity needed, and the cost sensitivity. Fewer, but larger DIMMs per channel (DPC) typically will provide better performance, since the memory bus can operate at a higher frequency as you reduce the number of DIMMs per channel. However, the larger DIMMs typically cost more. Total System Memory (2 CPU Sockets Populated) 6GB 12GB 18 GB 24GB 36GB 48GB 72GB 96GB 144GB 1GB UDIMMs 6 x 1GB 12 x 1GB 2GB UDIMMs 6 x 2GB 12 x 2GB 4GB UDIMMs 6 X 4GB 12 x 4GB. Configuration supports DDR3 1333, 1066 or 800MHz. Configuration supports DDR or 800MHz (or 1333MHz with Intel Xeon processor 5600). Configuration supports DDR3 800MHz Only Table 9 -- Balanced UDIMM Memory Configurations Total System Memory (2 CPU Sockets Populated) 6GB 12GB 18 GB 24GB 36GB 48GB 72GB 96GB 144GB 192GB 1GB RDIMMs 6 x 1GB 12 x 1GB 18 x 1GB 2GB RDIMMs 6 x 2GB 12 x 2GB 18 x 2GB 4GB RDIMMs 6 X 4GB 12 x 4GB 18 x 4GB 8GB RDIMMs 6 x 8GB 12 x 8GB 18 x 8GB 16GB RDIMM 1 6 x 16GB 12 x 16GB. Configuration supports DDR3 1333, 1066 or 800MHz. Configuration supports DDR or 800MHz (or 1333MHz with Intel Xeon processor 5600). Configuration supports DDR3 800MHz Only GB configuration (18 x 16GB) supported only on Xeon 5600 with DR DIMMs (4Gb density) Table Balanced RDIMM Memory Configurations Page 15 of 26

16 5.3 Near Balanced Memory Configurations As stated above, balanced memory configurations have all memory channels populated with identical number/size/speed/ranked DIMMs. You may vary from this guidance have performance which is most likely equal to or better than any un-balanced configuration and in some cases might approach the performance of a balanced configuration. Data to quantify these statements are provided in section 5.5. Near balanced configurations as I will call them have the following attributes: - All DIMMs capable of the same speed - All memory channels are used - All memory channels are identically populated - However you may use DIMMs with different sizes and/or ranks within a channel The Near balanced configuration should operate equal to or better than any un-balanced configuration and in some cases will approach the performance of a balanced configuration The key is that all memory channels must be identically populated. An example of this near balanced memory configuration is below (DIMM0 is furthest from the processor, the 2 nd processor is populated same as below) Near Balanced Config (18GB) (Config #1a) CH1 CH2 CH3 DIMM0 2GB 2GB 2GB DIMM1 1GB 1GB 1GB DIMM The advantage of near balanced configuration is it gives one the ability to hit more memory sizes than those offered with a truly balanced configuration. Below are example near balanced memory configurations (balanced configs in blue italics are also shown). Note: Some memory sizes have multiple near-balanced configuration options. For example: six 8GB DIMMS could be replaced with twelve 4GB DIMMs. For brevity sake, in Table 11 below, configuration options with 16GB DIMMs are not shown, and some memory sizes below do not list all possible near-balanced configuration options. Page 16 of 26

17 Total Capacity DIMMs used Notes 6 GB 6 x 1GB Balanced 12 GB 6 x 2GB Balanced 12 GB 12 x 1GB Balanced 18 GB 6 x 1GB & 6 x 2GB Near balanced 24 GB 12 x 2GB Balanced 24 GB 6 x 4GB Balanced 30 GB 6 x 4GB & 6 x 1GB Near Balanced 36 GB 18 x 2GB Balanced 36 GB 6 x 4GB & 6 x 2GB Near Balanced 42 GB 6 x 4GB & 6 x 2GB & 6 x 1GB Near Balanced 48 GB 6 x 8GB Balanced 48 GB 12 x 4GB Balanced 54 GB 6 x 8GB & 6 x 1GB Near Balanced 54 GB 12 x 4GB & 6 x 1GB Near Balanced 60 GB 6 x 8GB & 6 x 2GB Near Balanced 60 GB 12 x 4GB & 6 x 2GB Near Balanced 66 GB 6 x 8GB & 6 x 2GB & 6 x 1GB Near Balanced 72 GB 18 x 4GB Balanced 72 GB 6 x 8GB & 6 x 4GB Near Balanced 78 GB 6 x 8GB & 6 x 4GB & 6 x 1GB Near Balanced 84 GB 6 x 8GB & 6 x 4GB & 6 x 2GB Near Balanced 96 GB 12 x 8GB Balanced 102 GB 12 x 8GB & 6 x 1GB Near Balanced 108GB 12 x 8GB & 6 x 2GB Near Balanced 120 GB 12 x 8GB & 6 x 4GB Near Balanced 144 GB 18 x 8GB Balanced Table Near Balanced RDIMM Memory Configurations 5.4 Un-Balanced Memory Configurations If you must hit an exact memory size, and any of the balanced or near balanced configurations listed above are not possible for what ever reason, for example the customer requires exactly 16GB of memory it the system, then use the guidelines below to populate an unbalanced configuration for best performance. Note, for performance reasons, I would urge your customer to use one of the above balanced or near balanced configurations first and use an unbalanced configurations only as a last resort. For example, a 60GB near balanced config will likely perform much better than an unbalanced 64GB config. If you must have exactly 8, 16, 32 or 64GB memory configurations, below are likely the best options for achieving such configurations and which are likely the best. The below is an example for 16GB, but by changing the DIMM sizes as documented in the notes, the below applies to 8 or 32GB configurations and similar logic can be applied to 64GB configs. Page 17 of 26

18 Config #2a Config #3a Config #4a. CH1 CH2 CH3 CH1 CH2 CH3 CH1 CH2 CH3 DIMM0 4GB 2GB 2GB 2GB 2GB 2GB 1GB 1GB 1GB DIMM GB GB 1GB 1GB DIMM GB 1GB --- Notes: 1. CH1 = Memory Channel 1, CH2 = Memory Channel 2, CH3 = Memory Channel 3 2. DIMM population shown above is for a single CPU socket Assume the 2 nd CPU socket is populated identically as above. 3. Conclusions below are valid for 8 & 32GB memory config as well: 8GB: Replace the 2GB and 4GB DIMMs in the above diagrams, with 1 and 2GB DIMMs, respectively, and reduce the number of 1GB DIMMS by half. 32GB: Replace the 1GB, 2GB and 4GB DIMMs with 2, 4 and 8GB DIMMs, respectively. Assumptions: - Objective is for a system to have exactly 16GB of memory - Memory frequency will cause the results to vary see below for different assumptions Performance Comparison: - If all three configs are operating at 800MHz, performance is probably similar for all three configurations although Config #4a might be slightly better than #3a and #3a slightly better than #2a due to more DIMMs per channel (more pages open, thus better memory interleaving). - If Config #2a and #3a are both operating at 1066MHz, their results will be very similar and depending on the benchmark, one might be slightly better than the other (see the next section for specific results between these two configs). Both would likely be better than Config #4a since #4a can only run at 800MHz - If all configs are running at their fastest frequency, Config #2a (1333MHz) will be best, followed by Config #3a (1066MHz) followed by Config #4a (800MHz) Note: All of the configurations in the above example have exactly 16GB of memory. However, the near balanced configuration shown on the previous page (6 x 2GB and 6 x 1GB identified as Config #1a) will likely outperform configurations #2a, #3a & #4a when all are running at the same frequency (either 800 or 1066MHz where possible). And as the data in the next section shows, config #1a (near balanced), running at 1066MHz will likely outperform even Config #2a (un-balanced) running at 1333MHz Best 4GB Solutions What is the best 4GB memory solution? When using a 32-bit OS or application that can t access more than 4GB of memory, the best performing option is to use a balanced configuration and populate all memory channels equally. This will mean you are populating more than 4GB of memory, but it will ensure the best performance because the 4GB of memory that is used, will be configured in a 3-way interleave. If you populate exactly 4GB, you will, at best, have a 2-way interleave (2 DIMMs per processor) which is lower performance than 3-way interleave. Below are the best options for 4GB solutions: With 2 processors: With 1 processor: 6 x 1GB DIMMS (one DIMM in each of the 6 memory channels) 3 x 2GB DIMMS (one DIMM in each of the 3 memory channels) Page 18 of 26

19 5.5 Memory Interleaving Interleaving memory is an important way to get the most performance out of the memory subsystem. Memory interleaving is the process of organizing memory so that adjacent sections of memory (cache lines) are physically located on different memory channels. For example, let s say you have a single processor system with 3GB of memory. If you 3 x 1GB DIMMs (one DIMM in each channel) the BIOS will configure the memory in a 3-chanel interleave. Meaning the first cache line (64Bytes) in memory will be placed in on memory channel 0. The second cache line will be placed on memory Channel 1 and the third cache line on memory channel 2. The 4 th, cache line will then be placed on Channel 0 and so on. Thus interleaving the memory on all three memory channels. Note: The processor typically accesses data from memory on a cache line basis. The reason for interleaving memory is that memory needs to be pre-charged every so often, and if you continue to read from the same memory channel, and thus the same DIMM, you often will have to wait for the memory to pre-charge before you read your data. This waiting will delay your access to memory. However if you interleave your memory accesses you can be accessing from one cache line, while the other cache line is pre-charging. The more channels interleaved, the less likely you will have to wait for a pre-charge. The specific memory interleaving is determined at power on. The processor s BIOS looks at how the memory is populated, and determines the optimal memory interleaving scheme. Note depending on how the memory is populated, different section of memory will have different interleaving setups. For example let s look at the three different memory configurations below: Config #1c Config 2c Config #3c. CH1 CH2 CH3 CH1 CH2 CH3 CH1 CH2 CH3 DIMM0 1GB 1GB 1GB 1GB 1GB 1GB 1GB 1GB 1GB DIMM GB GB --- 1GB DIMM Config 1c: BIOS will configure memory as follows: 0 3GB 3-way interleave (best performance) Config 2c: BIOS will configure memory as follows: 0 2GB 2-way interleave (okay performance) (DIMM 0 CH1/CH2) 2 4GB 2-way interleave (okay performance) (DIMM 1 CH2/ DIMM0 CH3) Note: The BIOS chooses the above interleaving for an unbalanced configuration to allow similar performance for all regions of memory. The other option possible, but not implemented in the BIOS reference code, was to designate 0-3GB as a 3-way interleave, and 3-4GB as a 1-way interleave. Config 3c: BIOS will configure memory as follows: 0 3GB 3-way interleave (best performance) 3 5GB 2-way interleave (okay performance) Config 1c is best, since the entire memory region will be configured in a 3-way memory interleave. The other two, however, will have some portions of memory which perform okay (2-way interleave), but not great. Thus it is most important to populate all memory channels equally so that all regions of memory can be configured with a 3-way interleave, and thus deliver the best possible performance. Page 19 of 26

20 Interleaving of memory behind the 2 nd processor in a 2-socket system is determined independent of the first processor. For example: if your memory is configured as shown below for both processor. Processor #1 Processor #2. CH1 CH2 CH3 CH1 CH2 CH3 DIMM0 1GB 1GB 1GB 1GB 1GB --- DIMM DIMM BIOS will likely configure memory as follows: Processor #1: 0 3GB 3-way interleave (best performance) Processor #2 3 5GB 2-way interleave (okay performance) 5.6 Performance based on Memory Bandwidth What if memory bandwidth is most important? Below are two tables (one each for the Intel Xeon processor 5500 and 5600 series, respectively) with a number of different configurations, with the STREAM benchmark results listed. Use these results as a guideline for memory bandwidth intensive applications. The below results may not be representative of a workload which is not memory bandwidth intensive. STREAM Triad Result (MB/s) Memory Frequency DIMM Population 4.8GT/s QPI 5.86GT/s QPI 6.4GT/s QPI (MHz) (CPU 0 / CPU 1) Balanced Configs / , / ,218 33, / ,912 33, / ,265 26,750 27, / ,866 25,844 26, / ,052 26,305 27,208 Unbalanced Configs / , / , / 2-2-2** , / , / , / 2-2-2** , Table Memory bandwidth for different memory configurations (Xeon 5500 Series) Notes: means one DIMM in each memory channel. (all identical DIMMs) means two DIMMs in each memory channel (all identical DIMMs) means two DIMMs in the first two channels, and no DIMMs in the third memory channel. - Source: Intel internal measurements March 2009 (Xeon E5540 w/5.86gt/s QPI, Xeon X5570 w/6.4gt/s QPI) - ** These are actually balanced configurations, but listed here for easy reference to show the bandwidth differences between balanced and unbalanced configurations Page 20 of 26

21 STREAM Triad Result (MB/s) Memory Frequency DIMM Population 4.8GT/s QPI 5.86GT/s QPI 6.4GT/s QPI (MHz) (CPU 0 / CPU 1) Balanced Configs / , / , / ,897 TBD / ,497 TBD / ,459 TBD TBD / ,171 TBD TBD / TBD TBD TBD Table Memory bandwidth for different memory configurations (Xeon 5600 Series) Notes: means one DIMM in each memory channel. (all identical DIMMs) means two DIMMs in each memory channel (all identical DIMMs) means two DIMMs in the first two channels, and no DIMMs in the third memory channel. - Source: Intel internal measurements Feb 2010 (Xeon E5507 w/4.8gt/s, Xeon E5640 w/5.86gt/s QPI, Xeon X5680 w/6.4gt/s QPI) - Unbalanced configurations are not shown/tested, but relative performance impact should be similar as seen with the 5500 series shown in Table Other Memory Performance Details All things being equal, other variables may affect application performance. Below is a list of some of these variables. UDIMM vs. RDIMM UDIMM s tend to have slightly better performance than RDIMMs due to lower latencies, but the difference is very small and does vary by application. For example: with 1 DPC, the two have nearly identical bandwidth. However with 2 DPC, the RDIMM actually provides more max memory bandwidth (~10%) than the UDIMM solution. Also, UDIMMs have 1N timing with 1 DPC and 2N timing for 2 DPC. RDIMMs have 1N timing at all times (1, 2 or 3 DPC). Hence UDIMM performance tends to drop when you go to 2 DPC. (1N/2N refers to the number of cycles the memory controller allows for the DIMM to effectively latch the address and command data being sent to the DIMM and thus determines how many cycles until it expects to see data returned by the DIMM on the data pins. 1N = 1 cycle, 2N = 2 cycles. Thus UDIMMs operating with 2N timing will have 1 cycle additional latency vs. 1N timing.) UDIMMs tend to have lower power (~.75 watts per DIMM). UDIMMs typically are lower cost at lower densities (1GB and 2GB) Non-ECC UDIMMs are not recommended for server platforms Page 21 of 26

22 Memory Rank (Single Rank vs. Dual Rank vs. Quad Rank) - More ranks, typically provide better performance due to more pages potentially being open at a given time thus better response/lower latency for some memory accesses (see notes below) - More ranks, typically dissipate lower power However, as noted below, Quad rank DIMMs, due to their additional electrical loading, will run at a lower frequency than similar memory configurations. As such, Dual ranked DIMMs are typically the best option for best performance. Notes about Quad rank DIMMs: - The max memory frequency with a QR DIMM is 1066MHz - If a QR DIMM is installed in the platform the BIOS will reduce the frequency to 1066MHz if 1 DPC and 800MHz if 2 DPC (DPC = DIMM per channel) (even if the DIMM supports 1333MHz) - 3 DPC is not supported with QR DIMMs - So while QR DIMMs may have higher performance than SR or DR DIMMs (at the same frequency), if you have to run the QR DIMMs at a lower frequency than equivalent DR or SR configurations, the QR DIMMS probably will not provide higher performance. DIMM vs. LV DIMM (Xeon 5600 only) LV (Low Voltage) DIMMs simply run at a lower voltage, and thus lower power. While the power difference will vary between specific products and vendors, we expect the power savings to be: Power savings at Idle: Power savings at 100% utilization: ~0.5W per DIMM ~1.5-2W per DIMM All other things being equal, an LV DIMM will perform the same as a regular voltage DIMM. One exception is that LV DIMMs are limited to a maximum of 2 DPC (DIMMS per channel). 3 DPC operation is not supported with LV-DIMMS due to their lower voltage and thus lower signal drive capabilities. DIMM/ Configuration (x4 vs. x8 vs. x16) - No performance difference between a x4 vs. x8 vs. x16 - DIMMs build with x16 devices tend to be lower power than x8, which tend to be lower power than x4 6 4-step guide to picking the optimal memory configuration The Intel Xeon processor 5500 & 5600 series have many options for configuring memory. Consequently there are many possible combinations and identifying the best configuration for a given customer is often difficult since it very much depends on many variables and what is most important. The variables typically are, but are not limited to the following: - Capacity - Application - # DIMMs supported on the platform - Speed - Sensitivity to latency - Desired capacity - Cost - Sensitivity to bandwidth - Power - Upgradability Below is a suggested 4 step process for determining the optimal memory configuration for a given application: Page 22 of 26

23 Step 1: Answer the following questions: - Is my application most sensitive to capacity or memory bandwidth? o If more memory bandwidth sensitive: Stick to balanced or near balanced configurations Stick to higher frequency, lower DIMM per channel (DPC) configs o If capacity sensitive: Lower frequency configs may be okay Higher DPC solutions may to offer better price/performance by using more DIMMS, but lower capacity per DIMM (due to better interleaving with more DIMMs) - Which processor am I going to use (not all processors support all memory frequencies)? o If using an Advanced processor SKUs All memory options are available o If using Standard processor SKU s 1333MHz memory options are not available Configurations with 2 DPC could be better than configurations with 1 DPC (lower cost and higher performance) (both will run at 1066MHz, and more DPC means better memory interleaving) o If using Basic processor SKUs Only 800MHz memory options are available Configurations with 3 DPC could be better than configurations with 1 or 2 DPC (lower cost and higher performance) (both will run at 800MMHz, and more DPC means better memory interleaving) - How sensitive am I to cost and power? o Higher density DIMMs tend to cost more (especially 8GB DIMMs) o But higher density DIMMS, and thus fewer total DIMMs in the system, tend to dissipate less power o Higher density DIMMs also tend to have more ranks and higher ranked DIMMS tend to have lower power (fewer devices are active at a given time), higher performance (better interleaving) and lower cost (older and cheaper silicon technology used) Once you have answered the above questions, you probably have narrowed down the choices and it is easier to start making optimizations within your acceptable solution set. Step 2: Pick a balanced memory configuration which best meets your criteria. If you can pick one you are done. If not, go to step 3. Step 3: Pick a near balanced memory configuration which best meets your criteria. If you can pick one you are done. If not, go to step 4. Step 4: Pick an un-balanced configuration using the following guidelines: - All things being equal, faster memory will result in better performance (e.g is better than 800) - Populate equal number of DIMMs on each processor - Try to populate at least one DIMM in each memory channel - If you have un-equal numbers of DIMMs in the memory channels, never allow the difference in DIMM count, between channels, to be more than one. (e.g. don t have 3 DIMMS in one channel, and 1 DIMM in another channel) (exceptions are if you are using a Lock-step or Mirroring configuration where the 3 rd memory channel isn t used) - All things the same (e.g. same speed and same total memory size), more DIMMs per channel will likely give equal to better performance than fewer DIMMs per channel. (e.g. 12 x 2GB running at 1066MHz is probably equal to better performance than 6 x 4GB at 1066MHz). This is because with more DIMMs on each channel, more memory pages are maybe open, resulting in reduced latency from better memory interleaving performance Page 23 of 26