SUN FIRE X4150, X4250, AND X4450 SERVER ARCHITECTURE. Breakthrough Density Using Sun and Intel Technology

SUN FIRE X4150, X4250, AND X4450 SERVER ARCHITECTURE Breakthrough Density Using Sun and Intel Technology White Paper September 2008

Table of Contents Executive Summary................................................ 1 Managing Datacenter Capacity and Complexity.......................... 2 Introducing the Sun Fire X4150, X4250, and X4450 Servers................... 2 Comparing the Sun Fire X4150, X4250, and X4450 servers.....................5 Intel Advantage................................................... 9 Multi-Chip Packaging................................................ 10 Intel Core Microarchitecture........................................ 11 Sun Fire X4150, X4250, and X4450 Server Architectures................... 13 Sun Fire X4150 System-Level Architecture................................ 13 Sun Fire X4150 Server Overview........................................ 14 Sun Fire X4250 System-Level Architecture................................ 16 Sun Fire X4250 Server Overview........................................ 16 Sun Fire X4450 System-Level Architecture................................ 18 Sun Fire X4450 Server Overview........................................ 19 System Processors and Chipsets........................................ 21 Memory Subsystem................................................. 22 I/O Subsystem.................................................... 23 Enclosure Features................................................. 27 Service Processor and System Management.............................. 29 RAS Features...................................................... 32 Enterprise-Class Software........................................... 34 The Solaris Operating System........................................ 34 Linux Environments................................................. 36 Microsoft Windows Environments...................................... 36 VMware Support................................................... 36 Conclusion...................................................... 37

1 Executive Summary Executive Summary In recent years, emerging applications and expanding web services have fueled an escalating demand for mission-critical IT resources, with increasing numbers of collaborative business initiatives focused on information sharing and continuous data availability. As a result, IT data centers experienced rapid growth as the number of applications, the population of users, and the quantity of transactions multiplied. Today IT managers face the task of administering a complex storage and computing infrastructure one that typically contains many small servers added over time to meet ongoing resource demands. Since administrative, energy, and real estate costs continue to skyrocket, data center sprawl can severely strain IT budgets. To curb operational expenses, many companies are turning to consolidation and virtualization to improve resource utilization and enhance business agility. Sun s expertise in delivering mission-critical computing solutions and its focus on ecoresponsibility are evident in designs of a new platform family the Sun Fire X4150, X4250, and X4450 servers. Using high performance Intel Xeon processor technology, Sun has engineered powerful 1U and 2U systems that are ideal for HPC, grid computing, database, web infrastructure, and consolidation and virtualization initiatives. These systems offer breakthrough compute, memory, storage, and I/O density. At the same time, they feature tremendous energy efficiency as well as flexible support for multiple operating systems the Solaris Operating System, Linux, Windows, and VMware. The Sun Fire X4150, X4250, and X4450 servers showcase Intel s talent for creating highperformance commodity chip sets along with Sun s innovative engineering and quality system design. In addition to remarkable expandability and density (such as over 2 terabytes of internal storage or up to four six-core Intel Xeon processors in a single 2U chassis), these servers feature several redundant and hot-swappable components, efficient front-to-back air flow, highly efficient power supplies, and built-in system management tools. Engineered for mission-critical application availability, the Sun Fire X4150, X4250, and X4450 servers strive to conserve valuable energy resources and lower operational expenses while delivering high performance and density in an extremely small footprint.

2 Managing Capacity and Complexity Chapter 1 Managing Capacity and Complexity In today s competitive global marketplace, companies place ever-increasing demands on the IT infrastructure for compute and storage resources. As a result, most datacenters have grown exponentially in recent years, with large numbers of servers and storage racks continuously added to satisfy business requirements and meet escalating resource demands. Such rapid growth has produced its own set of challenges. The cost of maintaining large numbers of servers can prove to be formidable, especially at a time when real estate costs are rising, energy costs are skyrocketing, and department budgets are flat or declining. Managing the complexity that results from sprawling server resources puts tremendous strain on IT staff and budgets. To successfully meet business needs, IT management must control both cost and complexity while simultaneously scaling processing and storage capacity. Today the responsibility of managing datacenter capacity and rapid growth commonly fuels IT initiatives like consolidation and virtualization. Typically, these initiatives strive to improve resource utilization, reduce administrative complexity, and drive down IT costs. Consolidating many small servers into fewer powerful systems helps to minimize administrative workloads while increasing capacity and conserving valuable floor space. It can also help to reduce energy costs, vastly improving available performance relative to the amount of energy consumed. In addition, when IT organizations consolidate under-utilized resources, virtualization technologies can balance IT resource demands, applying key resources as needed to business-critical applications to enhance agility. Introducing the Sun Fire X4150, X4250, and X4450 Servers To help IT managers address the challenge of increasing capacity while managing growth, Sun offers new Intel Xeon processor-based systems the Sun Fire X4150, X4250, and X4450 servers. These systems feature high performance and unprecedented density in energy-efficient and compact 1U and 2U form-factors. With capabilities that complement the rest of the Sun server product line, the Sun Fire X4150, X4250, and X4450 servers raise the bar for 32- and 64-bit enterprise-class computing. These systems offer: Best-in-class performance. The Sun Fire X4150, X4250, and X4450 servers feature dual-, quad-, and six-core Intel Xeon 5000 and 7000 Sequence processors. With large integrated caches and high clock speeds, these processors offer high system performance and throughput compared with systems based on earlier generation Intel chipsets. In high-end configurations, Sun Fire X4150, X4250, and X4450 servers can house multiple Intel Xeon processors for example, the Sun Fire X4450 server

3 Managing Capacity and Complexity supports up to four six-core Intel Xeon X7450 processors clocked at 2.66 GHz, for a maximum of 24 cores and impressive performance in a compact 2U chassis. Remarkable density. Density is the cornerstone of the Sun Fire X4150, X4250, and X4450 server designs. When populated in a 40-rack unit (RU) enclosure, the 1U Sun Fire X4150 server facilitates a single rack with up to 320 cores, 680 DIMM slots, and 120 PCI Express (PCIe) slots. The 2U Sun Fire X4450 server enables a single rack with up to 480 cores maximum. In addition, the Sun Fire X4250 server can house over 2 terabytes internally using sixteen 2.5-inch SAS drives (via a PCIe Host Bus Adapter) in a single 2U chassis. The Sun Fire X4150, X4250, and X4450 servers provide the type of density needed to achieve consolidation and virtualization efficiencies. Such density facilitates the consolidation of many smaller servers, helping to conserve real estate, lower energy expense, and reduce costly administrative talent. In addition, these servers support multiple operating systems, which helps to simplify consolidation efforts and diminish server sprawl. Extensive system expandability. The ability to expand a server over time reduces the need for additional capital acquisitions and lowers application lifecycle costs. The Sun Fire X4150 and X4250 servers feature two CPU sockets per system and 64GB of memory (using 4-gigabyte FB-DIMMs), while the Sun Fire X4450 houses four sockets and up to 128 gigabytes of memory. The Sun Fire X4150 and X4450 servers support a maximum of 1 terabyte of internal storage, while the Sun Fire X4250 server allows over 2 terabytes of internal storage. Four GigabitEthernet ports are standard on all systems. Because of breakthrough system densities, these servers can scale to support new users, more transactions, or new 32-bit or 64-bit applications, enhancing longevity and increasing overall return on investment (ROI). Improved energy efficiency. Sun offers a portfolio of eco-responsible products and computing solutions to address customer IT needs. In the Sun Fire X4150, X4250, and X4450 servers, Intel Xeon processors incorporate new technologies that minimize power use and enhance energy efficiency. The Intel Core Microarchitecture optimizes processor performance relative to the power consumed. Power management capabilities limit power to unused execution units in each core, which helps to reduce power and cooling requirements. High-efficiency power supplies in the server chassis lessen overall power consumption. Variable speed fans, disk carrier design, and front-to-back air flow in the chassis help to effectively cool the system and maintain appropriate processor and system ambient temperatures. Enterprise-class high availability. The Sun Fire X4150, X4250, and X4450 servers are designed with enterprise-class RAS (Reliability, Availability, and Serviceability) features. To maximize uptime, systems include redundant hot-swappable fans and can be configured with redundant hot-swappable power supplies. Using a Sun StorageTek SAS RAID Host Bus Adapter (HBA), internal SAS disk drives can be configured for RAID 0, 1, 1E, 10, 5, 5EE, 50, 6, and 60 when mirroring is implemented, drives are also hot-swappable. Four integrated GigabitEthernet ports

4 Managing Capacity and Complexity enhance network availability without consuming a PCIe slot and can be implemented in failover configurations. On-board system management tools encourage remote, proactive monitoring and intervention. Simplified system management. To support out-of-band management, the Sun Fire X4150, X4250, and X4450 servers incorporate a service processor that features robust lights-out management capabilities. This built-in, hardware-based functionality allows administrators to monitor and manage systems remotely, allowing them to take corrective action as necessary and minimize unplanned downtime. The Sun Fire X4150, X4250, and X4450 servers combine best-in-class performance with noteworthy compute, memory, and I/O capacities. As a result, these systems are designed to scale up, scale out, and scale within, enabling deployment in a wide range of application architectures: Scale-up architectures: With multiple cores, these servers are well-suited to scale for growing workloads that deliver web, database, and other key infrastructure services. Scale-out architectures: With large memory capacities, internal storage, four GigabitEthernet ports, and high-bandwidth PCIe expansion for high-speed system interconnects (such as fiber channel and InfiniBand), these servers can scale to solve complex computing problems demanding intensive computing power and data bandwidth. Scale-within: With the ability to support Solaris 10 Virtualization and VMware, Sun Fire X4150, X4250, and X4450 servers are ideal systems to host virtualization technologies and consolidate multiple applications within a single extensible platform.

5 Managing Capacity and Complexity Figure 1 shows the 1U Sun Fire X4150 server and the 2U Sun Fire X4250 and X4450 server enclosures. Sun Fire X4150 Server Sun Fire X4250 Server Sun Fire X4450 Server Figure 1. Sun Fire X4150, X4250, and X4450 servers Comparing the Sun Fire X4150, X4250, and X4450 servers Table 1 summarizes features of the Sun Fire X4150, X4250, X4450 server platforms. These enterprise class systems are good solutions for applications where high density is needed to satisfy fast-growing workloads such as OLTP, database, and web service delivery. Table 1. Feature Comparison for Sun Fire X4150, X4250, and X4450 servers Feature Sun Fire X4150 Server Sun Fire X4250 Server Sun Fire X4450 Server Chassis 1U 2U 2U Number of CPU sockets 2 2 4 Supported processor types Dual-core Intel Xeon Processor 5200 Series, or quad-core Intel Xeon Processor 5400 Series Dual-core Intel Xeon Processor 5200 Series, or quad-core Intel Xeon Processor 5400 Series Dual-core Intel Xeon Processor 7200 Series, or quad- or six-core Intel Xeon Processor 7400 Series Number of cores per 40 RU rack enclosure Up to 320 cores Up to 160 cores Up to 320 cores Processor system bus Dual Front-Side Bus (FSB) Dual Front-Side Bus (FSB) Quad Front-Side Bus (FSB)

6 Managing Capacity and Complexity Feature Sun Fire X4150 Server Sun Fire X4250 Server Sun Fire X4450 Server Number of memory slots 16 16 32 Memory capacity Up to 64 GB (using 4 GB FB-DIMMs) Up to 64 GB (using 4 GB FB-DIMMs) Up to 128 GB (using 4 GB FB-DIMMs) Memory type DDR2 FB-DIMM DDR2 FB-DIMM DDR2 FB-DIMM Internal storage Up to 8 2.5-inch SAS (with HBA) Up to 16 2.5-inch SAS (with HBA) Up to 8 2.5-inch SAS (with HBA) Removable media 1 EIDE DVD/RW 1 EIDE DVD/RW 1 EIDE DVD/RW Number of PCIe slots Three total (x8) Six total (x8) Six total (two x8 and four x4) Number of GigE ports 4 on-board 4 on-board 4 on-board Number of USB ports 4 external plus 1 internal 4 external plus 1 internal 4 external plus 1 internal System management RAS components Integrated service processor and built-in lights-out management Hot swappable and redundant power supplies, fans, disk drives Integrated service processor and built-in lights-out management Hot swappable and redundant power supplies, fans, disk drives Integrated service processor and built-in lights-out management Hot swappable and redundant power supplies, fans, disk drives As Table 1 shows, the systems share a number of features, including: Intel Xeon-based architecture supporting multiple dual-, quad-, or six-core processors Large memory capacities using FB-DIMM memory modules Multiple high-bandwidth Front Side Bus (FSB) system interconnects Large-capacity internal storage Expandability using PCIe cards Built-in quad GigabitEthernet support An integrated service processor for lights-out system management Enterprise-class RAS features such as redundant, hot-swappable power supplies, fans, and disk drives Support for multiple operating systems Notable differences between the Sun Fire X4150, X4250, and X4450 servers include: Chassis enclosure (1U versus 2U) Number of processor sockets (2 versus 4) Supported processor types (Intel Xeon Processor 5000 Sequence versus Intel Xeon Processor 7000 Sequence) Memory capacity (up to 64GB versus 128GB, using 4GB FB-DIMMs) Internal storage capabilities (up to 8 or 16 SAS drives maximum, depending on server model) PCI expansion capabilities (3 PCIe slots versus 6 PCIe slots) Multiple off-the-shelf configurations of each platform are available, along with a wide spectrum of options to tailor each system for specific workload requirements. The Sun Fire X4150, X4250, and X4450 servers offer the density and configurability necessary to realize operational, administrative, and energy cost-savings the goals of many IT strategic plans.

7 Managing Capacity and Complexity A Choice of Operating Systems To optimize flexibility and investment protection, Sun Fire X4150, X4250, and X4450 servers support a choice of operating systems, including: Solaris Operating System (OS) Linux operating systems (64-bit Red Hat or SuSE Linux) Microsoft Windows VMware ESX Server Chapter 4 describes the OS releases supported as of this writing. Please see sun.com/ x64 for the latest information on supported operating systems and environments. Chassis Design Innovations Sun Fire X4150, X4250, and X4450 servers share common chassis design characteristics with other Sun x64 and SPARC server platforms. This approach not only provides a consistent look and feel across the product line, but simplifies administration through consistent placement and shared components. Beyond consistency, this approach provides a design focus that places key technology where it can make a difference in the datacenter. Enhanced System and Component Serviceability Finding and identifying servers and components in a modern datacenter can be a challenge. Sun Fire X4150, X4250, and X4450 servers are optimized for lights-out datacenter configurations with easy to identify servers and modules. Color-coded operator panels provide easy-to-understand diagnostics and systems are designed for deployment in hot-isle/cold-isle multi-racked deployments, with both front and rear diagnostic LEDs to pinpoint faulty components. Fault Remind features help to identify failed components. Consistent connector layouts for power, networking, and management make moving between Sun systems straightforward. All hot-plug components are tool-less and easily available for serviceability. For instance, an integral hinged lid provides access to dual fan modules so that fans can be serviced without exposing sensitive components or causing unnecessary downtime. Robust Chassis, Component, and Subassembly Design Sun s volume servers share chassis designs that are carefully engineered to provide reliability and cool operation. Even features such as the honeycomb-shaped chassis ventilation holes help to provide the best compromise for strength, maximum airflow, and maximum electronic attenuation. Next-generation hard disk drive carriers enhance chassis ventilation, enabling greater storage density while increasing system airflow. A removable disk cage in each system plugs directly in front of the fan tray assemblies, allowing airflow to be directed both above and below disk drives, and above and below memory FB-DIMMs and mezzanine boards to efficiently cool the system. Dual cooling

8 Managing Capacity and Complexity fan modules are isolated from the chassis to avoid transferring rotational vibration to other system components. Also, the integration of the fan power board into the Fan Tray assembly protects users from electrical shock during fan removal or insertion. In spite of extreme computational, I/O, and storage density, Sun servers are able to maintain adequate cooling using conventional technologies. Efficient modular fan assemblies keep the chassis within an effective operating temperature range. Minimized DC-to-DC power conversions also contribute to overall system efficiency. By providing 12 volt power to the motherboard, power conversion stages are eliminated. This approach reduces generated heat, and introduces further efficiencies to the system. Minimized Cabling for Maximized Airflow To minimize cabling and increase reliability, a variety of smaller boards and riser cards are used: Power distribution boards (PDBs) distribute system power from the dual power supplies to the motherboard and to the disk backplane (via a connector board). Connector boards eliminate the need for many discrete cables, providing a direct card plug-in interconnect to distribute control and most data signals to the disk backplane, fan boards, and the PDB. Fan boards provide connections for power and control for both the primary and secondary fans in the front of the chassis. No cables are required since every dual fan module plugs directly into one of these PCBs which, in turn, plug into the Connector board. PCIe riser cards plug directly into the motherboard, allowing PCIe cards to be easily installed. The disk backplane mounts to the disk cages in the enclosure, delivering disk data through one or two 4-channel discrete mini-sas cables from the installed HBA card. An 8-disk backplane is offered for the Sun Fire X4150 and X4450 servers while the Sun Fire X4250 server supports a 16-disk backplane. Also provided via the disk backplane are two USB connections to the front of the system. Chapter 3 gives more details on the chassis, system features, and internal components, highlighting similarities and differences between the Sun Fire X4150, X4250, and X4450 servers.

9 The Intel Advantage Chapter 2 The Intel Advantage Sun Microsystems, Inc. and Intel Corporation are collaborating in an alliance targeted at building a comprehensive family of servers based on Intel Xeon processors and at optimizing performance for the Solaris Operating System (OS). With the Intel processorbased Sun Fire X4150, X4250, and X4450 servers, Sun extends its existing X64 system portfolio and complements its powerhouse SPARC processor-based platforms. Moving forward, Sun and Intel continue to participate in joint engineering efforts aimed at enhancing the Solaris OS, Java technologies, and systems built using Intel Xeon processor designs. In this new family of servers, Sun s well-known engineering expertise combines with an emphasis on performance, quality, reliability, and eco-responsibility. The Sun Fire X4150, X4250, and X4450 servers leverage dual-, quad-, or six- core Intel Xeon processors that feature key Intel technologies for virtualization, I/O acceleration, and energy efficiency. These 64-bit processors are compatible with the legacy of IA-32 software, instantly making available a large volume of existing 32-bit applications as well as emerging 64-bit applications. Figure 2. 50W Quad-core Intel Xeon L5420 Processor. This chapter introduces the Intel Xeon processors and chipsets used in these new Sun models. The Sun Fire X4150 and X4250 servers have two processor sockets that can be populated with either dual- or quad-core Intel Xeon Processor 5000 Series for a maximum of 8 cores. In contrast, the Sun Fire X4450 server contains four processor sockets supporting up to four dual-, quad-, or six-core Intel Xeon Processor 7000 Series, for a maximum of 24 cores in a single 2U chassis. The related system chipsets include a Northbridge Memory Controller Hub (MCH) and a Southbridge Input/Output Hub (IOH). For an overview of Intel chipsets used in these Sun servers, see Chapter 3. For detailed information on the Intel chipsets, see the web sites: www.intel.com/products/processor/xeon7000/ www.intel.com/products/processor/xeon5000/

10 The Intel Advantage Multi-Chip Packaging The Intel Xeon Processor 5000 Sequence processors use a Multi-Chip Package (MCP) to deliver dual- and quad-core configurations, while the design of the Intel Xeon Processor 7000 Sequence enables dual-, quad-, and six- core configurations. The processor packaging approach increases yields and lowers manufacturing costs, helping Intel and Sun to deliver higher performance and throughput at lower price points. Intel Xeon Processor 5200 and 5400 Series Based on the Intel Core Microarchitecture, the Intel Xeon Processor 5200 and 5400 Series employ a second-generation 45nm manufacturing process that incorporates smaller transistors, allowing the processors to consume less power, to achieve faster switching times, and to provide greater density. (Earlier generation 5100 and 5300 processors used a 65nm manufacturing process.) The Intel Xeon 5200 and 5400 Series processors feature microarchitecture enhancements, including larger Level-2 caches, in comparison with earlier generation processors. The Intel Xeon Processor 5200 and 5400 Series processors provide performance for multiple application types and user environments in a substantially low power envelope. The dual-core 5200 Series processors provide significant performance headroom for multithreaded applications and can boost system utilization through virtualization and application responsiveness. The quad-core 5400 Series processors maximize performance-per-watt, providing increased density for datacenter deployments where energy efficiency is a key goal. Figure 3 shows logical diagrams for the 5000 Sequence. The 5200 Series processors include dual execution cores, each with a 64K Level-1 cache (32K instruction/32k data). The dual cores share a Level-2 cache (6 MB on the 5200 Series) to increase cache-toprocessor data transfers, maximize memory-to-processor bandwidth, and reduce latency. The 5400 Series processors provide four execution cores. Each pair of cores shares a 6 MB Level-2 cache, for a total of 12 MB. With two sockets in Sun Fire X4150 or X4250 servers, 5400 Series processor designs enable a maximum of 8 processor cores in a single 1U or 2U form factor. Figure 3. Intel Series 5200 use a single die in one package while the quad-core Series 5400 processors use two die

11 The Intel Advantage Intel Xeon Processor 7200 and 7400 Series In Sun Fire X4450 servers, the dual-core Intel Xeon Processor 7200 Series processor incorporates two die per processor package, with each die containing a single processor core (Figure 4). The Intel Xeon Processor 7400 Series is a monolithic design that supports up to six cores. In a Sun Fire X4450 server configuration with four Intel Xeon 7400 Series processors, this enables a maximum density of 24 execution cores in a compact 2U enclosure. Figure 4. Intel Xeon 7200 Series uses two die per package while the Intel Xeon 7400 Series is a monolithic design supporting up to six cores The 5000 and 7000 Sequence processor families share these additional features: An integrated Level-1 (L1) instruction and data cache (64KB per core) Large integrated Level-2 (L2) cache. The Intel Xeon 7400 Series processors also feature up to 16 MB of shared Level 3 cache. Multiple, independent Front Side Buses (FSBs) that act as high-bandwidth system interconnects. The Intel Xeon 5000 Sequence processors support both 1066 MT/sec and 1333 MT/sec Front Side Buses enabling theoretical data transfer rates of 8.66 GB/sec (at 1066 MT/sec) or 10.5 GB/sec (at 1333 MT/sec). The 7000 Sequence supports Front Side Bus interconnects at 1066 MT/sec. Like the Intel Xeon Processor 5200 and 5400 Series, the 7400 Series processors use a new, second-generation 45nm manufacturing process along with an enhanced microarchitecture. (Earlier generation 7300 processors used a 65nm manufacturing process.) The 45nm manufacturing process leverages the use of smaller transistors, enabling higher densities, lower power consumption, and faster switching times. Microarchitecture enhancements support up to six cores, large Level-2 caches, and a shared Level-3 cache of up to 16 MB. As a result, 7400 Series processors offer fast performance and innovative features that are ideal for addressing server consolidation and virtualization initiatives.

12 The Intel Advantage Intel Core Microarchitecture WIth a goal of reducing the processor energy footprint, the Intel Core Microarchitecture delivers high performance with low power consumption. The microarchitecture includes several features designed specifically to enhance performance while maintaining energy efficiency: Out-of-order execution conserves processor-to-memory bandwidth, improving memory access and increasing overall processing performance Speculative prefetches move data to the L2 cache before an L1 cache request occurs Large L2 caches (6 MB on dual-core 5200 Series, 12 MB on quad-core 5400 Series, 8 MB on dual-core 7200 Series, 6 MB on quad-core 7400 Series, and 9 MB on six-core 7400 Series) help to increase system memory efficiency and enhance performance. On the Intel Xeon 7400 Series, up to 16 MB of shared Level-3 cache can enable ready access to the working set for performance-critical applications. Power management capabilities modulate power delivery to execution cores, limiting power to unused components These features help to enhance performance and conserve processor power, which can translate into energy savings and lower operational costs. Table 2 and Table 3 list typical power envelopes for Intel Xeon processors that are configurable in the Sun Fire X4150, X4250, and X4450 servers as of this writing. Additional processor characteristics are also included. Table 2. Power Envelope for Intel Xeon Processor 5000 Sequence used in Sun Fire X4150 and X4250 servers Processor Speed L2 Cache L3 Cache FSB Power Dual-core X5260 3.33 GHz 6 MB None 1333 MT/sec 80 W Quad-core E5410 2.33 GHz 12 MB None 1333 MT/sec 80 W Quad-core E5440 2.83 GHz 12 MB None 1333 MT/sec 80 W Quad-core E5450 3.00 GHz 12 MB None 1333 MT/sec 80 W Quad-core X5460 3.16 GHz 12 MB None 1333 MT/sec 120 W Quad-core L5420 2.50 GHz 12 MB None 1333 MT/sec 50 W Table 3. Power Envelope for Intel Xeon Processor 7000 Sequence used in Sun Fire X4450 servers Processor Speed L2 Cache L3 Cache FSB Power Dual-core E7220 2.93 GHz 8 MB None 1066 MT/sec 80 W Quad-core E7420 2.13 GHz 6 MB 16 MB Shared 1066 MT/sec 90 W Quad-core E7440 2.40 GHz 6 MB 16 MB Shared 1066 MT/sec 90 W Six-core L7455 2.13 GHz 9 MB 16 MB Shared 1066 MT/sec 65 W Six-core E7450 2.40 GHz 9 MB 16 MB Shared 1066 MT/sec 90 W Six-core X7460 2.66 GHz 9 MB 16 MB Shared 1066 MT/sec 130 W

13 Sun Fire X4150, X4250, and X4450 Server Architectures Chapter 3 Sun Fire X4150, X4250, and X4450 Server Architectures The Sun Fire X4150, X4250, and X4450 servers are designed to provide best-in-class performance with high reliability and low power consumption. This chapter details physical and architectural aspects of the systems, highlighting similarities and differences between the server designs. Sun Fire X4150 System-Level Architecture Figure 5 contains a system-level block diagram for Sun Fire X4150 servers. The architecture of the Sun Fire X4150 server is similar to that of the Sun Fire X4250 and X4450 servers multiple host processors connect to a Northbridge Memory Controller Hub (MCH) which in turn connects to a Southbridge I/O Hub (IOH). Details about key subsystems (system chipsets, memory subsystems, I/O subsystems, etc.) are given later in this chapter. Intel Xeon 5200/5400 TM XEON TM XEON Intel Xeon 5200/5400 B3 B2 B1 B0 FSB 1333 MT/s FSB 1333 MT/s C0 C1 C2 C3 10.5 GB/s Dual FSB to MCH 10.5 GB/s DIMMs Channel B 5.3 GB/s Channel C 5.3 GB/s DIMMs MCH Blackford 5000P DIMMs 5.3 GB/s Channel A 5.3 GB/s Channel D A3 A2 A1 A0 PCI-E x8 PCI-E ESI (PCI-E) PCI-E x8 PCI-E SAS/RAID Controller PCI-E x8 USB IOH ESB-2 USB to IDE PCI-E x4 PCI 32-bit 33 MHz 2x USB IDE USB Hub AST2000 Q62611.1 GP 0608 TAN A2 CD/DVD 1x Internal USB 2.0 2x Rear USB 2.0 2x Front USB 2.0 2x 1GB Ethernet 2 & 3 2x 1GB Ethernet 0 & 1 Serial RJ-45 Management 10/100 Ethernet VGA Video D0 D1 D2 D3 DIMMs PCI-E x16-2 PCI-E x16-1 PCI-E x16-0 8x SAS HDDs Figure 5. Block Diagram of Sun Fire X4150 Server with SAS drives

14 Sun Fire X4150, X4250, and X4450 Server Architectures Sun Fire X4150 Server Overview The Sun Fire X4150 server includes these major components: One or two dual- or quad-core Intel Xeon Processor 5000 Series Up to 64 GB of memory populated in 16 Fully-Buffered Dual Inline Memory Module (FB-DIMM) slots (1 GB, 2 GB, or 4 GB FB-DIMMs are supported) Four on-board 10/100/1000 Mbps Ethernet ports Three low-profile PCIe slots, each 8-lane Up to eight internal 2.5-inch SAS drives (via a PCIe Host Bus Adapter) Five USB 2.0 ports An integrated service processor with embedded functionality for lights-out management Up to two hot-swappable, high-efficiency power supply units (PSUs) for N+N redundancy Seven hot-swappable, variable speed fan modules (for N+1 redundancy), each containing two fans operating under environmental monitoring Sun Fire X4150 System Enclosure The Sun Fire X4150 server enclosure is designed to occupy one rack unit in a standard 19-inch rack (Table 4). Table 4. Dimensions and weight of the Sun Fire X4150 server Dimension U.S. International Height 1.73 inches (1 RU) 44 millimeters Width 16.75 inches 17.47 inches (including ears ) 425.5 millimeters 443.9 millimeters (including ears )) Depth 28 inches 711.2 millimeters 28.99 inches (including PSU handles) 736.4 millimeters (including PSU handles) Weight 30.93 pounds minimum 40.60 pounds maximum 14.028 kilograms minimum 18.418 kilograms maximum

15 Sun Fire X4150, X4250, and X4450 Server Architectures Sun Fire X4150 System Front and Rear Perspectives Figure 6 illustrates the front and rear panels of the Sun Fire X4150 server. Component status indicators System status indicators DVD drive USB ports Hard disk drives Redundant (N+1) power supply units PCIe slots System status indicators Serial and network management ports 10/100/1000 Ethernet ports USB ports VGA port Figure 6. Sun Fire X4150 server, front and rear panels External features and connections include: Front and rear status indicator lights, reporting locator (white), service required (amber), and activity status (green) for the system and components Up to eight hot-plug SAS disk drives, which insert through the front panel One slimline, slot-accessible DVD-RW, accessible through the front panel Four USB ports, two on the front panel and two on the rear. (A fifth internal USB port is used to attach internal boot devices.) Up to two power supply units (for N+1 redundancy) with integrated fans, with each power supply having a single, independent AC plug on the rear panel Rear power-supply indicator lights, showing the status of each hot-swappable power supply Four 10/100/1000BaseT autosensing Ethernet ports, accessible on the rear panel Three PCIe slots, in which low-profile cards can be installed from the rear panel Two management ports (one 10/100BaseT Ethernet port and one RJ-45 serial management port on the rear panel) for default connections to the service processor VGA video port with an analog HD-15 VGA connector on the rear panel

16 Sun Fire X4150, X4250, and X4450 Server Architectures Sun Fire X4250 System-Level Architecture Figure 7 contains a system-level block diagram for Sun Fire X4250 servers. The architecture of the Sun Fire X4250 server is similar to that of the Sun Fire X4150 server the motherboard contains dual host processors that connect to a Northbridge Memory Controller Hub (MCH) which in turn connects to a Southbridge I/O Hub (IOH). Details about key subsystems (system chipsets, memory subsystems, I/O subsystems, etc.) are given later in this chapter. Intel Xeon 5200/5400 B3 B2 B1 B0 FSB 1333 MT/s DIMMs Channel B 5.3 GB/s DIMMs 5.3 GB/s Channel A A3 A2 A1 A0 USB USB to IDE PCIe x4 IDE USB Hub CD/DVD 1x Internal USB 2.0 2x Rear USB 2.0 2x Front USB 2.0 2x 1GB Ethernet 2 & 3 TM XEON TM XEON Intel Xeon 5200/5400 FSB 1333 MT/s C0 C1 C2 C3 10.5 GB/s Dual FSB to MCH 10.5 GB/s Channel C 5.3 GB/s DIMMs D0 D1 D2 D3 MCH Blackford 5000P 5.3 GB/s Channel D DIMMs + PCIe x8-3 PCIe x8 PCIe ESI (PCIe) PCIe Switch + + + PCIe x8-0 + PCIe x8-4 PCIe x8 PCIe Switch + + + PCIe x8-1 PCIe SAS/RAID Controller + PCIe x8 PCIe Switch + PCIe x8-5 IOH ESB-2 + + PCIe x8-2 + 2x USB SAS Expander PCI 32-bit 33 MHz AST2000 Q62611.1 GP 0608 TAN A2 + 2x 1GB Ethernet 0 & 1 Serial RJ-45 Management 10/100 Ethernet VGA Video 16x SAS HDDs + + Figure 7. Block Diagram of Sun Fire X4250 Server with SAS drives Sun Fire X4250 Server Overview The Sun Fire X4250 server includes these major components: One or two dual- or quad-core Intel Xeon Processor 5000 Series Up to 64 GB of memory populated in 16 Fully-Buffered Dual Inline Memory Module (FB-DIMM) slots (1 GB, 2 GB, or 4 GB FB-DIMMs are supported) Four on-board 10/100/1000BaseT Ethernet ports Six low-profile PCIe slots, all 8-lane Up to sixteen internal 2.5-inch SAS drives (via a SAS Expander and a PCIe Host Bus Adapter) Five USB 2.0 ports

17 Sun Fire X4150, X4250, and X4450 Server Architectures A built-in service processor with integrated functionality for lights-out management Up to two hot-swappable, high-efficiency power supply units (PSUs) for N+1 redundancy Six hot-swappable, variable speed fan modules (for N+1 redundancy), each with two fans operating under environmental monitoring and control Sun Fire X4250 System Enclosure The Sun Fire X4250 server enclosure is designed to occupy two rack units in a standard 19-inch rack (Table 5). Table 5. Dimensions and weight of the Sun Fire X4250 server Dimension U.S. International Height 3.46 inches (2 RU) 87.85 millimeters Width 16.75 inches 17.55 inches (including ears ) 425.5 millimeters 445.7 millimeters (including ears )) Depth 28 inches 711.3 millimeters 28.88 inches (including PSU handles) 733.7 millimeters (including PSU handles) Weight 43.00 pounds minimum 58.45 pounds maximum 19.506 kilograms minimum 26.415 kilograms maximum Sun Fire X4250 System Front and Rear Perspectives Figure 8 illustrates the front and rear panels of the Sun Fire X4250 server. System status indicators Component status indicators Hard disk drives DVD drive USB ports Redundant (N+1) power supply units PCIe slots System status indicators Serial and network management ports 10/100/1000 Ethernet ports USB ports VGA port Figure 8. Sun Fire X4250 server, front and rear panels

18 Sun Fire X4150, X4250, and X4450 Server Architectures External features and connections include: Front and rear status indicator lights, reporting locator (white), service required (amber), and activity status (green) for the system and components Up to sixteen hot-plug SAS disk drives, which insert through the front panel One slimline, slot-accessible DVD-RW, accessible through the front panel Four USB ports, two on the front panel and two on the rear. (A fifth internal USB port is used to attach internal boot devices.) Up to two power supply units (for N+1 redundancy) with integrated fans, with each power supply having a single, independent AC plug on the rear panel Rear power-supply indicator lights, showing the status of each hot-swappable power supply Four 10/100/1000BaseT autosensing Ethernet ports, accessible on the rear panel Six PCIe slots, in which low-profile cards can be installed from the rear panel Two management ports (one 10/100c Ethernet port and one RJ-45 serial management port on the rear panel) for default connections to the service processor VGA video port with an analog HD-15 VGA connector on the rear panel Sun Fire X4450 System-Level Architecture Figure 9 contains a system-level block diagram for Sun Fire X4450 servers. The architecture is similar to the Sun Fire X4150 and X4250 servers system-level architecture except that four processor sockets interface to the Northbridge MCH and Southbridge IOH, and the system contains 32 memory slots. Key subsystems (the chipset, the memory subsystem, the I/O subsystem, etc.) are discussed later in this chapter.

19 Sun Fire X4150, X4250, and X4450 Server Architectures Intel Xeon 7200/7400 TM XEON TM XEON TM XEON TM XEON Intel Xeon 7200/7400 B7 B6 B5 B4 B3 B2 B1 B0 FSB 1066 MT/s FSB 1066 MT/s FSB 1066 MT/s FSB 1066 MT/s C0 C1 C2 C3 C4 C5 C6 C7 DIMMs Channel C Channel B 8.5 GB/s 8.5 GB/s DIMMs D0 D1 D2 D3 D4 D5 D6 D7 MCH Clarksboro 7000P DIMMs Channel A Channel D PCI-E x4 DIMMs PCI-E x8-3 PCI-E x8 PCI-E x16-0 A7 A6 A5 A4 A3 A2 A1 A0 PCI-E ESI (PCI-E) PCI-E SAS/RAID Controller PCI-E x4 PCI-E x8 PCI-E x4 PCI-E x4 PCI-E x8-4 PCI-E x16-1 PCI-E x8-5 PCI-E x16-2 USB IOH ESB-2 USB to IDE PCI-E x4 2x USB Figure 9. Block Diagram of Sun Fire X4450 Server with SAS Drives PCI 32-bit 33 MHz IDE USB Hub AST2000 Q62611.1 GP 0608 TAN A2 8x SAS HDDs CD/DVD 1x Internal USB 2.0 2x Rear USB 2.0 2x Front USB 2.0 2x 1GB Ethernet 2 & 3 2x 1GB Ethernet 0 & 1 Serial RJ-45 Management 10/100 Ethernet VGA Video Sun Fire X4450 Server Overview The Sun Fire X4450 server includes the following major components: Up to four dual-core, quad- or six-core Intel Xeon Processor 7000 Series Up to 128 GB of memory in 32 Fully-Buffered Dual Inline Memory Module (FB-DIMM) slots (1GB, 2GB, or 4GB FB-DIMMs supported) Four on-board 10/100/1000BaseT Ethernet ports Six low-profile PCIe slots, two 8-lane and four 4-lane Up to eight internal 2.5-inch SAS drives (via a PCIe Host Bus Adapter) Five USB 2.0 ports A service processor that features embedded functionality to support integrated lights-out management Up to two hot-swappable, high-efficiency power supply units (PSUs) for N+1 redundancy Six hot-swappable, variable speed fan modules (for N+1 redundancy), each with two fans operating under environmental monitoring and control

20 Sun Fire X4150, X4250, and X4450 Server Architectures Sun Fire X4450 System Enclosure The Sun Fire X4450 server enclosure occupies two rack units in a standard 19-inch rack (Table 6). Table 6. Dimensions and weight of the Sun Fire X4450 server Dimension U.S. International Height 3.46 inches (2 RU) 87.85 millimeters Width 16.75 inches 17.55 inches (including ears ) 425.5 millimeters 445.7 millimeters (including ears )) Depth 28 inches 711.3 millimeters 28.88 inches (including PSU handles) 733.7 millimeters (including PSU handles) Weight 42.13 pounds minimum 56.28 pounds maximum 19.109 kilograms minimum 25.528 kilograms maximum Sun Fire X4450 System Front and Rear Perspectives Figure 10 illustrates the front and rear panels of the Sun Fire X4450 server Component status indicators System status indicators DVD drive USB ports Hard disk drives Redundant (N+1) power supply units PCIe slots System status indicators Serial and network Management ports 10/100/1000 Ethernet ports USB ports VGA port Figure 10. Sun Fire X4450 server, front and rear panels External features include: Front and rear status indicator lights, reporting locator (white), service required (amber), and activity status (green) for the system and components Up to eight hot-plug SAS, which insert through the front panel

21 Sun Fire X4150, X4250, and X4450 Server Architectures One slimline, slot-accessible DVD-RW, accessible through the front panel Four USB ports, two on the front panel and two on the rear. (A fifth internal USB port is used to attach internal boot devices.) Up to two power supply units (for N+1 redundancy) with integrated fans, with each power supply having a single, independent AC plug on the rear panel Rear power-supply indicator lights, showing the status of each hot-swappable power supply Four 10/100/1000BaseT autosensing Ethernet ports, accessible on the rear panel Six PCIe slots, in which low-profile cards can be installed from the rear panel Two management ports (one 10/100BaseT Ethernet port and one RJ-45 serial management port on the rear panel) for default connections to the service processor VGA video port with an analog HD-15 VGA connector on the rear panel System Processors and Chipsets Although the Sun Fire X4150, X4250, and X4450 servers are based on different Intel Xeon processors and associated chipsets, they share a similar system-level architecture. In these systems, multiple host processors interface to the Northbridge MCH over multiple Front-Side Buses (FSBs) two FSBs in the Sun Fire X4150 and X4250 servers and four in the Sun Fire X4450 server. The Southbridge IOH interfaces to the I/O devices, enabling expandability along with high I/O throughput. Each chipset is designed to match processor performance with memory capacity, I/O expandability, and interconnect bandwidth. Intel Xeon 5000 Sequence Chipset Refer back to the Sun Fire X4150 and X4250 server block diagrams earlier in this chapter. The Sun Fire X4150 and X4250 servers incorporate the Intel Xeon Processor 5000 Sequence chipsets, which consist of: One or two dual-core Intel Xeon 5200 Series processors or quad-core Intel Xeon 5400 Series processors The Northbridge Intel 5000P Memory Controller Hub (MCH) The Southbridge Intel 6321ESB Input/Output Handler IOH (ESB-2). The ESB-2 is interconnected to the MCH using one ESI (Enterprise South Bridge Interface) link and one PCIe link. ESI is based on x4 PCIe interconnect with proprietary extensions and offers a 2 GB/sec transfer rate. Dual independent Front Side Buses (FSBs) act as a system interconnect between each processor socket and the MCH. Operating at either 1066 MT/sec or 1333 MT/sec, the 64-bit wide FSBs are capable of peak bandwidths up to 8.5 GB/sec or 10.5 GB/sec respectively. For more information on the Intel Xeon Processor 5000 Sequence chipset, see www.intel.com/products/processor/xeon5000/.

22 Sun Fire X4150, X4250, and X4450 Server Architectures Intel Xeon 7000 Sequence Chipset Refer back to the Sun Fire X4450 server block diagrams (earlier in this chapter). The Sun Fire X4450 server uses an Intel Xeon Processor 7000 Sequence chipset, which includes: Up to four dual-core Intel Xeon 7200 Series processors, or up to four quad- or six-core Intel Xeon 7400 Series processors The Northbridge Intel 7000P Memory Controller Hub (MCH) The Southbridge Intel 6321ESB Input/Output Handler IOH (ESB-2) With the 7000 Sequence chipset, four FSB buses connect between the four processor sockets and the MCH. Running at 1066 MT/sec, the 64-bit wide FSBs are capable of peak bandwidths of 8.5 GB/sec each. For more information on the Intel Xeon Processor 7000 Sequence chipset, see www.intel.com/products/processor/xeon7000/. Memory Subsystem To address memory-intensive applications, multiple FB-DIMM memory channels help to support large memory densities and high bandwidth in the Sun Fire X4150, X4250, and X4450 servers. The memory subsystems are based on similar Intel MCH controller designs that feature four independent memory channels. In the Sun Fire X4150 and X4250 servers, each memory channel supports up to four FB- DIMMs, enabling up to 16 FB-DIMMs per system for a maximum capacity of 64GB using 4GB modules. In the Sun Fire X4450 server, each channel supports up to eight memory modules for up to 32 FB-DIMMs per system, allowing a memory capacity of up to 128GB in the 2U chassis using 4GB FB-DIMMs. In all systems, the four memory channels are organized into two branches. Modules of identical size must be populated in pairs, starting with memory slot 0 on each channel (slot 0 on each channel is marked with white handles). The same 667 MHz, PC2-5300 DDR2 FB-DIMM modules in either 1 GB, 2 GB, or 4 GB capacities are supported in the three server models. Peak read bandwidth to the FB- DIMMs is 5.3 GB/sec per channel (21 GB/sec total with the 1333 MT/sec system bus), and peak write bandwidth is 2.7 GB/sec per channel (10.7 GB/sec total with the 1333 MT/sec system bus). Memory modules feature Error Checking and Correcting (ECC) with Chipkill technology for high reliability.

23 Sun Fire X4150, X4250, and X4450 Server Architectures I/O Subsystem Designed for the headroom needed to expand systems and scale applications, the Sun Fire X4150, X4250, and X4450 servers feature a PCIe expansion bus, integrated storage, four on-board Intel Gigabit Network Interface Controllers (NICs), and five USB ports. The same Southbridge ESB-2 chip is used on the three server models. In each case, it is interconnected to each system s Northbridge MCH using one ESI link and one PCIe link. The ESB-2 provides two built-in GigabitEthernet NICs going to external NIC ports 0 and 1. Two additional GigabitEthernet NICs (port 2 and 3) are connected to a Dual Gigabit Intel Ophir 82571 chip that interfaces to the ESB-2 using a 4-lane PCIe link. The ESB-2 also supports all USB functionality two USB ports go from the ESB-2 to the rear of the system, one is routed to an internal USB hub for the two front USB connections, and one USB port is inside the chassis for internal boot devices. The ESB-2 also supports a USB-to-IDE interface that enables the connection of an optional internal EIDE DVD/RW drive, and connects to the service processor via a 32-bit, 33MHz PCI channel. System Network Interfaces Multiple on-board GigabitEthernet connections promote flexible connectivity, as well as configurations that support network interface failover. Each server features four 10/100/1000 Mbit/sec Ethernet ports on the rear panel, numbered in sequence from left to right. Each port auto-negotiates its link connection, and LEDs above the port indicate the speed of the established link (green signifies that the established link is 1000 Mbit/sec). All four Ethernet interfaces support PXE boot for network interface booting. PCIe Expansion Bus The Sun Fire X4150, X4250, and X4450 servers include a PCIe expansion bus that accommodates 64-bit low-profile cards. On each server model, three right-angle risers plug directly into the motherboard to enable PCI expansion (Figure 11). On the Sun Fire X4150 server, the risers support one slot per riser, for a total of three internal 8-lane PCIe slots (all with x16 mechanical connectors). In contrast, on the Sun Fire X4250 and X4450 server, each riser supports two PCIe card slots. On the Sun Fire X4250 server, each riser provides two 8-lane (electrical and mechanical) slots, for a total of six x8 slots in the 2U chassis. On the Sun Fire X4450 server, each riser supports two card slots with one x8 and one x16 mechanical connector, also for a total of six slots in a 2U chassis.

24 Sun Fire X4150, X4250, and X4450 Server Architectures x8 electrical x16 mechanical PCI Riser for Sun Fire X4150 Server (1 slot per riser) x8 mechanical x8 electrical PCI switch Active PCI Riser for Sun Fire X4250 Server (2 slots per riser) x8 mechanical x16 mechanical PCI Riser for Sun Fire X4450 Server (2 slots per riser) Figure 11. Right-angle PCIe risers for Sun Fire X4150, X4250, and X4450 servers Of note, the three PCIe risers for the Sun Fire X4250 server are active riser cards that is, a IDT PES24N3A PCIe switch resides on the riser card (Figure 12). The PES24N3A switch provides high-performance I/O connectivity and switching functions between a x8 PCIe upstream port and two x8 downstream ports. It can operate either as a storeand-forward switch or as a cut-through switch (depending on packet size) and supports switching of memory and I/O transactions. It supports eight Traffic Classes (TCs) and one Virtual Channel (VC) with sophisticated resource management algorithms (including round robin, weighted round-robin, and strict priority schemes). These algorithms support sufficient bandwidth allocation and latency for critical traffic classes in applications such as high throughput 10 Gigabit I/Os, SATA controllers, and Fibre Channel HBAs.