Kingston Technology Server Architecture and Kingston Memory Solutions Ingram Micro May 2015 Mike Mohney Senior Technology Manager, TRG 2015 Kingston Technology Corporation. All rights reserved. All trademarks and registered trademarks are the property of their respective owners.
Kingston Server Upgrades Introduction to Kingston Kingston Solutions and Memory Roadmap Memory Configurations for Intel server platforms 2
Kingston Snapshot Founded in 1987, 27+ years Privately Held Company Financially Secure For 8 years, revenue exceeding $4B 4,000 Employees Worldwide Worldwide Manufacturing & Testing Focused Business: Memory Modules, Flash Products & Tech Solutions Delivering over 1M units per day Products distributed in 125 Countries and available in over 30,000 locations worldwide; widely accepted brand 3 Source: Kingston Technology
Kingston Global Infrastructure Manufacturing Centers are all ISO 9001:2008 certified 2014 worldwide capacity: Over 30 million units per month Over 50 SMT manufacturing lines in 4 global facilities, producing both DRAM & Flash products * * * * ** ** * 4 Kingston Technology Manufacturing Facilities Worldwide ** Panram and OSE are subcontractors to Kingston 4 Source: Kingston Technology, June 2014
Kingston Investments & Expertise Kingston manages all six processes required to build modules, cards, and drives. Wafer Production Wafer Testing Wafer Processing/Chip Production Component Testing Module, Card & Drive Production Module, Card & Drive Testing 5
Strategic Alliances: Server Compatibility Check Kingston works with many facets of technology to ensure server memory compatibility. Industry Leading OS, Software, Hardware PC OEM s Motherboard Manufacturers 6
Intel and Kingston DDR3 DRAM chip validation testing on new Advantest tester (DDR4 capable) Checking early FB-DIMM prototype against specifications Since 1998, Intel and Kingston have jointly developed memory technology validation testing processes for PC133, RDRAM, DDR, DDR2, FB-DIMM, DDR3, LRDIMM, and DDR4. 7
Kingston Performance Upgrades Introduction to Kingston Kingston Solutions and Memory Roadmap Memory Configurations for Intel server platforms 8
Reliability: The Challenges of Memory Testing Memory Capacity 1GB 2GB 4GB 8GB 16GB Number of Cells Over 8.5 billion cells Over 17 billion cells Over 34 billion cells Over 68 billion cells Over 136 billion cells Kingston Quality Standard: Zero bad cells 1 bad cell = defective module 9
Semiconductor Reliability This curve of semiconductor failures over time is called a bathtub curve Number Of Field Failures (in units) Early-Life Failures Useful Life Of Semiconductor A typical useful life would be 20 years or more Failure curve for lower-quality semiconductor products End-of-Life Failures Time Ship out from Factory to user Typically the first few months of usage 10
Kingston Production Testing and Dynamic Server Burn-In 100% Testing No spot testing! Kingston memory is 100% tested prior to leaving our factories, as we have always done Dynamic Server Memory Burn-In Shrinking DRAM lithography increases sensitivity to temperature, forcing higher error rates and exposing weak memory cells Kingston s advanced burn-in process screens these by simulating 3 months of heavy server use at higher temperatures under full system environment and frequency stress Proprietary Production Testers Kingston s test engineers designed the first automated production testers in the memory industry New patented automated testers build upon our test engineering legacy to meet the more stringent demands of evolving memory technology 35 Quality Control Checkpoints Kingston uses AQL sampling methods to re-inspect and re-test every lot to ensure our quality standards are being met Kingston holds 29 patents related to memory testing 11
Memory Roadmap (1987 2017) 1987-1990 FAST PAGE MODE EDO 1997-1999 PC66 PC100 PC133 2000 RDRAM PC600 PC700 PC800 2001-2003 DDR200 DDR266 DDR333 DDR400 RDRAM PC1066 2004-2005 DDR2-400 DDR2-533 DDR2-667 2006 DDR2-800 FB-DIMMs 2007-2009 DDR3-800 DDR3-1066 DDR3-1333 2010-2013 DDR3L 1.35V DDR3-1600 DDR3L-1600 DDR3-1866 LRDIMM 2014 DDR4-2133 DDR4-2400 2015 2016 DDR4-2666 2017 DDR4-2933 Technology Data Rate Module Classification Peak Bandwidth DDR3 (1.5V) DDR3L (1.35V) DDR3U (1.25V) DDR4 (1.2V) 800 DDR3-800 / PC3-6400 6400 MB/s or 6.4 GB/s 1066 1333 1600 DDR3-1066 / PC3-8500 DDR3L-1066 / PC3L-8500 DDR3-1333 / PC3-10600 DDR3L-1333 / PC3L-10600 DDR3-1600 / PC3-12800 DDR3L-1600 / PC3L-12800 8500 MB/s or 8.5 GB/s 10600 MB/s or 10.6 GB/s 12800 MB/s or 12.8 GB/s 1866 DDR3-1866 / PC3-14900 14900 MB/s or 14.9 GB/s 2133 DDR4-2133 / PC4-2133 17000 MB/s or 17 GB/s 2400 DDR4-2400 / PC4-2400 19200 MB/s or 19.2 GB/s 2666 DDR4-2666 / PC4-2666 21300 MB/s or 21.3 GB/s 2933 DDR4-2933 / PC4-2933 23400 MB/s or 23.4 GB/s 1 2
New Module Key Location Form Factors 288-pin DIMM 260-pin SODIMM Performance 1600MT/s, 1866MT/s, 2133MT/s, 2400MT/s, 2666MT/s, 2933MT/s 4000MT/s (possible extension) 4266MT/s (possible extension) Power 1.2V with improved refreshes, ½K page for x4 12V for NVDIMM (4-pins) DRAM Densities/DIMM Capacities 4Gb, 8Gb, 16Gb (x4, x8, x16) Future DIMM capacities of 64GB, 128GB, 256GB New Features 3D chip stacking support (up to 8 stacked dies) Point-to-Point Memory Channel addressing More stable than DDR3 with improved error correction code Slightly thicker for improved performance Curved edge for better socket connection 13
Low Voltage (LV) DDR3L = Low Voltage 1.35V JEDEC (industry spec) Modules are classified as PC3L-xxx Reducing memory power consumption 10% by dropping module voltage requirement, lowering energy costs to data centers Kingston DDR3L modules are dual-voltage supporting 1.35V / 1.5V Supported on current and legacy server platforms Intel SNB/IVB/HSW (Xeon E3+ Series) Intel WSM-EP/SNB-EP/IVB-EP/HSW-EP (Xeon 5600/E5 v1/v2 Series) Intel WSM-EX/IVB-EX (Xeon E7/v2 Series) AMD Magny-Cours+ (Opteron 6100/6200/6300 Series) AMD Lisbon+ (Opteron 4100/4200/4300 Series) Supported on mobile/desktop platforms from Intel and AMD Skylake (DDR3L Only), Broadwell, Haswell, & Ivybridge (Core i5 / Core i7) AMD A-Series DDR3U (1.25V) Ultra Low Voltage JEDEC Spec for Mobile, Server Low Voltage spec for DDR4 deemed not possible by DRAM semiconductors 14
Load-Reduced DIMMs for Servers JEDEC Industry Standard Specification LRDIMMs overcome the impact of Ranks (Loads) upon memory speed Parallel memory bus impacted by memory module Ranks Enable greater memory capacity and speeds while reducing load Require special buffer chips Eight-Rank modules now possible with less loading on memory bus Will enable faster 32GB, 64GB, 128GB LRDIMMs Supported on 2012+ Intel and AMD servers Included in DDR4 spec with new, more efficient design Q1 2015 Kingston launched Q1 2013 DDR3 LRDIMMs KVR13LL9Q4/32 KVR16LL11Q4/32 KVR18L13Q4/32 DDR4 LRDIMMs KVR21L15Q4/32 KVR21L15Q4/64 Coming soon 15 Source: Intel IDF, September 2011
Load-Reduced DIMMs for DDR4 New Design, Reduced Latency 16 Source IDT: www.idt.com/document/whp/ddr4-lrdimms-both-memory-capacity-and-speed
Non-Volatile DIMM (NVDIMM) What Is It? A non-volatile memory module directly accessed by the memory controller (approximately 10x faster than PCIe interface). Silicon based memory technology is reaching its perfomance and efficiently limit. Types of NVDIMM standards NVDIMM-F An all FLASH memory module accessible by the memory controller. NVDIMM-N A DRAM memory module made persistent by the use of NAND flash silicon. NVDIMM-N2 Similar to NVDIMM-N but with system accessible Flash. NVDIMM-M MRAM based memory module accessible by the memory controller. NVDIMM-x New technology. PRAM? RRAM? HMC? Viking ArxCis-NV DDR3 NVDIMM 17 Source: Google Images Diablo/SanDisk ULLtra DIMM SK Hynix 16GB DDR4 NVDIMM
Kingston Memory Product Lines ValueRAM System Specific o o Generic / JEDEC Specification Ideal for system builders or small datacenters o o o Also known as Branded Guaranteed compatibility Ideal for users who want to upgrade OEM machines OEM / Private Label HyperX o Customized in box solutions o Enthusiast gaming market o Contracted supply agreements o Overclocking o Brand focused 18
System Specific Memory Guaranteed Compatible Testing conducted in the system in which the memory will be installed Importance of Branded Memory Different chips provide varying performance levels Chip Compatibility can vary Possible constraints in module height or width Presence Detect Configuration 19
ValueRAM Product Decoder Kingston ValueRAM is built to industry standard specifications outlined by J.E.D.E.C., the Joint Electron Device Engineering Council, on which Kingston holds a seat. Intel, AMD, and other computing chipset architects adhere to these standards for industry agreement and cross-platform compatibility. J.E.D.E.C. provides the specifications for Pin Count, Form Factor, Speed, Power, and Memory Technology Types 20
ValueRAM Server Focused Programs Intel Certified intended specifically for customers who are using Intel branded server/workstation motherboards or systems Qualified and supported by Intel Controlled builds (BOMs) PCNs with 45-90 Day Notice Server Premier intended for server builders with fully controlled BOM requirements (PCB, chip brand/die revision, etc) regardless of brand of motherboard used Fully BOM controlled PCNs with 45-90 Day Notice Easily manage what is ordered No price premium 21
ValueRAM Qualifications In addition to the strategic partnerships Kingston has with ODMs for submitting memory for qualification, we also work with independent memory test labs AVL and CMTL Labs for specific platform, motherboard, or system qualification. Test certificates are available on our website. http://www.kingston.com/us/memory/valueram/server Example: 22
Module Types For Server Platforms Registered DIMM Load Reduced DIMM ECC Unbuffered DIMM 288-pin DIMM Registered DIMM Load Reduced DIMM ECC Unbuffered DIMM 240-pin DIMM Optimizing Memory: DDR4 is NOT compatible with DDR3 based systems (module will not fit) Install memory in kits of identical modules according to memory architecture Configure all processors the same way for most bandwidth Do Not Mix Module Types within a server. Mixing DDR3 (1.5V) and DDR3L (1.35V) will default the memory to 1.5V. 23
DRAM Chip Types Used in Servers Multi Bit Error Detection and Correction x4 Registered DIMMs LRDIMMs 1 2 3 4 Single Bit Error Detection and Correction x8 1 2 3 4 5 6 7 8 Registered DIMMs LRDIMMs Unbuffered ECC DIMMs Unbuffered DIMMs SODIMMs 24
64-bit Memory Loads (aka Ranks) Single Rank (1R) 64bits Dual Rank (2R) 64bits 64bits Quad Rank (4R) Ranks are virtual modules on a single DIMM, designed to save space on the motherboard Only one Rank is active at a time, the other ranks on the DIMM go into standby saving power 2R and 4R DIMMs consume less power than 1R DIMMs Memory lanes on the motherboard become congested with more memory loads, so server systems are limited by the number of ranks that can be used, as the memory slows down with more ranks populated, just line cars in heavy traffic 64bits 64bits 64bits 64bits 25
Why Add Channels and Increase The Memory Speed? 120 DDR3 & DDR4 Aggregate Server Memory Bandwidth 100 80 GB/s 60 40 DDR3-800 DDR3-1066 DDR3-1333 20 DDR3-1600 DDR3-1866 0 Single Channel (1 DIMM per CPU) 2 x 16GB = 32GB Dual Channel (2 DIMMs per CPU) 4 x 16GB = 64GB Quad Channel (4 DIMMs per CPU) 8 x 16GB = 128GB DDR4-2133 26 Benchmark by SiSoftware SANDRA 2013.05.19.44 on Intel DDR3 and DDR4 platforms
Memory Configuration Options by Processor Memory Supported Memory Configuration Xeon 5500 Series DDR3-1333 Triple-Channel = K3 kits Xeon 5600 Series DDR3-1333 / DDR3L Triple-Channel = K3 kits Xeon E5 Series Xeon E5 v2 Series DDR3-1600 / DDR3L-1333 DDR3-1866 / DDR3L-1600 Quad-channel = K4 kits Triple-channel = K3 kits Quad-channel = K4 kits Triple-channel = K3 kits Xeon E5 v3 Series DDR4-2133 Quad-channel = K4 kits Xeon 6500 / 7500 / E7 Series DDR3-1066 DDR3L-1066 on E7 Series Dual-channel = K2 / K4 kits Xeon E7 v2 Series DDR3-1600 / DDR3L-1600 Quad-channel = K4 kits Xeon W3500 / 3600 / E3 Xeon E3 v2 / v3 Series DDR3-1066 / DDR3L (W3600) DDR3/L-1600 / DDR3-1866 Dual-channel = K2 / K4 kits Opteron 8400 / 2400 Series DDR2-800 Dual-channel = K2 kits Opteron 6100-6300 Series DDR3 / DDR3L 1333-1866* Quad-channel = K4 kits Opteron 4100-4300 Series DDR3 / DDR3L 1333-1866* Dual-channel = K2 kits * Subject to motherboard compatibility 27
Kingston Performance Upgrades Introduction to Kingston Kingston Solutions and Memory Roadmap Memory Configurations for Intel server platforms 28
Microarchitecture Roadmap 1600 1866 2133 2400 29
Intel Xeon E5 / E5 v2 / E5 v3 EP Series Server Memory Configuration HP ProLiant Gen 8 / Gen 9 Cisco UCS M3 IBM System xseries M4 Dell PowerEdge 30 Picture sources: Google images
Intel Xeon E5 Series Summary 31 Source: Intel public marketing
32 Intel Xeon E5-2600/4600 v1/v2/v3 Architecture Quad Channel up to 12 DIMMs per CPU (24 per 2-CPU System, 48 per 4-CPU System)
Intel Xeon E5-2400 v1/v2 Architecture Triple Channel up to 6 DIMMs per CPU (12 per 2-CPU System) 33
Haswell-EP Intel Xeon E5-2600 v3 DDR4 Memory Speeds Module Type (No Mixing in a Server) 1 DIMM per Channel 1 DPC 1.2V 2 DIMMs per Channel 2 DPC 3 DIMMs per Channel 3DPC Registered DIMM (RDIMM) 2133 1866 1600 Load Reduced DIMM (LRDIMM) 2133 2133 1600 2133 Processor Model Determines Maximum Memory Speed 1866 1600 34
Optimizing Intel Xeon E5 v3 Servers DDR4 Server Intel S2600WT Ch 3 Ch 4 Ch 2 Ch 1 128GB 3DPC 1DPC 1DPC 3DPC 3DPC 1DPC 1DPC 3DPC 1 2 3 1 2 3 3 2 1 3 2 1 1 2 3 1 2 3 3 2 1 3 2 1 Ch 3 Ch 4 16 x KVR21R15S4/8 8GB DDR4-2133 RDIMMs 1Rx4 1.2v Ch 2 Ch 1 Unbalanced Configuration (128GB) Memory Bandwidth 67.464 GB/s E5-2650v3 CPU1 E5-2650v3 CPU2 31.41% (30.89 GB/s) Ch 3 Ch 4 Ch 2 Ch 1 Ch 3 Ch 4 Ch 2 Ch 1 2DPC 2DPC 2DPC 2DPC 2DPC 2DPC 2DPC 2DPC 1 2 3 1 2 3 3 2 1 3 2 1 1 2 3 1 2 3 3 2 1 3 2 1 E5-2650v3 E5-2650v3 Balanced Configuration (128GB) CPU1 CPU2 2 DIMMs Per Channel Memory Bandwidth 98.357 GB/s 35 Benchmark by SiSoftware SANDRA 2013.05.19.44 on Intel S2600WT
Memory Speed Performance Gains for Intel DDR4 Servers Bandwidth comparison using 32GB DDR3 and DDR4 Load Reduced DIMMs 2200 2133 2133 2000 1800 1600 1866 13% 25% 1600 1600 1400 1200 1000 Xeon E5 v3 35% Xeon E5 v2 1066 1 DPC 2 DPC 3 DPC 256GB 512GB 768GB 36 Source: http://www.intel.com/performance
Intel Xeon E5 v2 Processors Processor Model Determines Maximum Memory Speed 37 http://www.intel.com/content/www/us/en/processors/xeon/xeon-e5-brief.html
Ivybridge-EP Intel Xeon E5-2600 v2/4600 v2 DDR3 RDIMM Memory Speeds 1.5V Module Type (No Mixing in a Server) System Configuration Module Ranks 1R/2R=Single/Dual 4R = Quad 1 DIMM per Channel 1 DPC 2 DIMMs per Channel 2 DPC 3 DIMMs per Channel 3DPC Registered DIMM (RDIMM) 1 Socket Per Channel 2 or 3 Sockets Per Channel 1R/2R 1866 4R 1066 1R/2R 1866 1600 1066 4R 1066 800 NO QR 1.35V Module Type (No Mixing in a Server) System Configuration Module Ranks 1R/2R=Single/Dual 4R = Quad 1 DIMM per Channel 1 DPC 2 DIMMs per Channel 2 DPC 3 DIMMs per Channel 3DPC 1DPC 2DPC 3DPC Registered DIMM (RDIMM) 1 Socket Per Channel 2 or 3 Sockets Per Channel 1R/2R 1600 4R 800 1R/2R 1333 1333 800 4R 800 800 NO QR 38
Ivybridge-EP Intel Xeon E5-2600 v2/4600 v2 DDR3 LRDIMM & ECC UDIMM Memory Speeds 1.5V / 1.35V Module Type (No Mixing in a Server) Load Reduced DIMM (LRDIMM) Module Ranks 4R = Quad 1 DIMM per Channel 1 DPC 2 DIMMs per Channel 2 DPC 3 DIMMs per Channel 3DPC 4R at 1.5V 1866 1600 1066 4R at 1.35V 1600 1600 1066 1DPC 2DPC 3DPC Module Type (No Mixing in a Server) ECC Unbuffered DIMM (ECC UDIMM) System Configuration 1 Socket Per Channel 2 or 3 Sockets Per Channel 1 DIMM per Channel 1 DPC 1.5V 1.35V 2 DIMMs per Channel 2 DPC 3 DIMMs per Channel 3DPC 1 DIMM per Channel 1 DPC 1866 1600 1866 1600 NO UDIMM 2 DIMMs per Channel 2 DPC 3 DIMMs per Channel 3DPC 1333 1333 NO UDIMM 39
Optimizing Intel Xeon E5 v1/v2 Servers 256GB 16 x KVR16LR11D4/16 16GB DDR3-1600 RDIMMs 2Rx4 1.5v Unbalanced 34.88% Balanced 40 Benchmark by SiSoftware SANDRA 2013.05.19.44 on Intel S2600GZ
Kingston Summary Top buyer of memory chips/only the best chips Ranked #1 in DRAM Ranked #1 in USB drives Listed on America s largest private companies Worldwide presence/financially solid Channel leadership OEM, channel, Retail, etail Established relationships with technology leaders Value-Add to Flash products Major investments in technology Most extensive testing in the industry Guaranteed compatibility/certifications Managing the quality process from wafer to module 41
42 Questions?
43