Cisco Unified Computing System (UCS) A Complete Reference Guide to the Data Center Visualization Server Architecture Silvano Gai Tommi Salli Roger Andersson Cisco Press 800 East 96th Street Indianapolis, Indiana 46240 USA
Contents Preface Nomenclature xvi xvi Chapter 1 Introduction 1 Data Center Challenges 1 Environmental Concerns "Green" 2 Server Consolidation 3 Virilization 4 Real Estate Power and Cooling 5 Cabling 5 Disaster Recovery 7 Network Virtualization 8 Desktop Virilization 9 Cloud Computing 10 Evolution of Data Centers 10 Stand-Alone Servers 11 Scale-Up 12 Scale-Out 12 Scale-Up vs. Scale-Out 12 Rack-Optimized Servers 13 Blade Servers 14 Server Sprawl 15 Virtualization 17 Server Deployment Today 18 Unified Computing System (UCS) 18 Chapter 2 Server Architectures 23 The Processor Evolution 24 Sockets 24 Cores 25 Threads 27 Intel Hyper-Threading Technology 27 Front-Side Bus 28 Dual Independent Buses 29 Dedicated High-Speed Interconnect 30 Intel QuickPath Interconnect 31 The Memory Subsystem 33 SRAMs 34 DRAMs 34 SDRAMs 35 DIMMs 36 ECCand Chipkill 38
x Contents Memory Ranks 39 UDIMMs and RDIMMs 40 DDR2 and DDR3 41 The I/O Subsystem 43 PCI Express 43 Intel Microarchitectures 45 Platform Architecture 46 CPU Architecture 49 Virilization Support 56 Advanced Reliability 59 Advanced Encryption Standard 60 Trusted Execution Technology 61 Chip Design 61 Chipset Virtualization Support 63 Intel VT-d for Direct I/O 64 Intel VT-c for Connectivity 65 VMDirectPath 68 Chapter 3 UCS Enabling Technologies 69 Unified Fabric 69 10 Gigabit Ethernet 71 Lossless Ethernet 72 Terminology 72 PFC (Priority-Based Flow Control) 72 DCBX: Data Center Bridging exchange 73 Bandwidth Management 74 FCoE (Fibre Channel over Ethernet) 75 Virtualization 81 Server Virtualization 81 SR-IOV 83 The IEEE Standard Effort 83 Port Extenders and Virtualization 84 VNTag 86 Fabric Extenders 88 VN-Link 90 Memory Expansion 93 Speed vs. Capacity 94 Capacity vs. Cost 94 How Much Memory Is Required? 95 NUMA 98 The UCS Approach 98 The UCS Advantage 101 Chapter 4 I/O Adapters 103
Contents xi Disclaimer 104 The Intel Approach 104 10 Gigabit Ethernet NIC Solutions 104 Intel 82598 10 Gigabit Ethernet Controller (Oplin) 105 Support for Multi-Core CPUs 108 Hardware-Assisted Virtualization 109 Advanced Features for Storage over Ethernet 109 Intel 82599 10 Gigabit Ethernet Controller (Niantic) 109 Improved Performance 112 Hardware-Assisted Virtualization 113 Support for DCB (Data Center Bridging) 114 Storage over Ethernet 115 Fibre Channel over Ethernet (FCoE) 115 Time Sync IEEE 1588 115 Double VLAN 116 Security 116 Intel's NetEffect iwarp Controller (NE020) 116 iwarp and RDMA 117 N2020 Architecture 119 Performance 120 Summary 122 Converged Network Adapters (CNAs) 123 Cisco Palo 124 Emulex 129 EmulexOneConnectOCm10102-FC 130 FCoE Features 132 Ethernet Features 133 Functional Architecture 133 Deployment in UCS 133 Management of OneConnect UCNAs 134 Benefits of OneConnect UCNAs 136 QLogic 137 8000 Series First Generation CNA 137 8100 Series Second Generation CNA 138 Broadcom 144 BCM57711 Dual-Port 10GbE Controller 144 Advanced Integration 145 High-Performance Hardware Offload 146 Broadcom and UCS 151 Chapter 5 UCS B-Series Blade Servers 153 Components Overview 153 UCS Manager 154
xii Contents UCS 6100 Series Fabric Interconnects 154 UCS 2100 Series Fabric Extenders 155 UCS 5100 Series Blade Server Chassis 156 UCS B-Series Blade Servers 157 I/O Adapters 159 Overall Organization 160 UCS C-Series Rack Servers 161 Detailed Description UCS 6100 Series Fabric Interconnects 161 UCS 2104XP Fabric Extender 168 UCS 5108 Blade Server Chassis 172 Two-Socket Blade Architecture 172 UCS B200 Two-Socket Server 179 UCS B250 Extended Memory Server 181 Four-Socket Blade Architecture 182 UCS B440 Four-Socket Server 186 Description of Communication Flows 187 The Boot Sequences 187 Fabric Interconnect and UCSM 189 Fabric Extender 190 Baseboard Management Controller 190 Chapter 6 UCS C-Series Rack Servers 193 UCSC200 194 UCSC210 195 UCSC250 199 UCS C460 202 Processors 207 Adapters 212 Hard Disk 213 Management 213 Software 219 Physical Parameters 220 C200 220 C210 220 C250 220 C460 221 Weights 221 Chapter 7 UCS Manager 223 UCSM Overall Architecture 223 System Components 223 UCSM Is a Model-Driven Framework 227 161
Contents xiii Management Information Model 232 Available Integration Points 233 Interfaces 233 Standard (Cut-Through) Interfaces in a UCS 235 Standard Interfaces in a UCS 237 Native Interfaces in UCS 239 Operating Principles 240 Configuration Policies 242 Operational Policies 243 Global vs. Local Policies 244 Pools 244 Manual Population of Pools 250 Automatic Population of Pools 250 Service Profiles 252 Service Profile Templates 253 Organizations 255 Hierarchical Pool and Policy Resolution 256 Role-Based Access Control (RBAC) 257 Locales 258 Users and Authentication 259 UCSM and VMware's vcenter Integration 259 Integration Architecture 260 Virtualization Support 260 Management Plane Integration 261 Port-Profiles 261 vnic Template 262 Runtime Policy Resolution for Dynamic VIFs 262 UCS Manager and VM in GUI 264 Basic System Management with UCSM 264 Hardware Management 264 Example of a Chassis Discovery Process 265 Retirement of Hardware 266 Firmware Management 266 Firmware Download Formats 267 Firmware Life Cycle 268 Management Firmware Pack Policy 272 Host Firmware Pack Policy 272 The Stateless Computing Deployment Model 272 The Basic Computing Deployment Model 273 System Setup Initial Setup 274 The Default Computing Deployment Model 274 The Stateless Computing Deployment Model 275 Requirements for Stateless Service Profiles 280
xiv Contents System Logging 287 Faults and Events 289 Audit Log 291 Backup and Restore of UCS Manager 292 Full State Backup 292 Configuration-Only Backup 292 Backing Up the UCS 292 Restoring a Configuration-Only Backup 293 Integrating with UCS 294 UCS Manager XML API 295 UCS XML API Object Naming 296 Method Categories 296 UCS Platform Emulator 301 Chapter 8 Third-Party Management Software 307 BMC 307 Just-in-Time Provisioning 308 Embedded System Management 309 Business Service Provisioning 313 Composite Packaging 315 Configuration Management 316 Granular Access Control 318 Compliance 318 Vision for Automated and Efficient IT 319 CA Management Integration with Cisco UCS 322 Integration Point 323 CA Infrastructure Management Integration 323 Discovery, Fault, and Service Modeling 324 Performance Management and Analytics 327 Automation 328 Change and Configuration Management 329 Service Profile and Application Templates 330 Automated Provisioning 331 Policy-Based Automation 332 User Self-Service 333 Private Cloud Deployments 333 EMC lonix Products for Cisco UCS 337 Unified Infrastructure Manager (UIM) 337 Data Center Insight (DCI) 341 IBM Tivoli Software Integration with Cisco UCS 344 Microsoft System Center 346 VMware vcenter 347 Communications 348
Contents xv Chapter 9 Planning Configuration of the DVS 350 Virtual Machine Adapters 350 Resource Checks for DRS, HA, and FT 350 a UCS Blade Server Installation 353 The Owner of the UCS Domain 353 User Authentication 354 Power and Cooling 354 Physical Sizing and Environmental Requirements 357 Connectivity 359 Choosing the Right Cables 361 Twinax 361 Fiber 361 Bandwidth 362 OS and Application Support 363 Supported Storage Devices and Protocols 363 Planning for Redundancy 364 Power Supply Redundancy 364 I/O Redundancy 365 Ethernet Interfaces Redundancy 365 Fibre Channel Interfaces Redundancy 367 Bibliography 369 PCI Express 369 IEEE 802.3 369 Improvements to Ethernet 369 IEEE 802.1 Activities 369 FCoE 370 TRILL 370 Virtualization 370 Memory Subsystem 371 Intel Processors 371 Data Centers 372 Green 372 Cloud Computing 372 Glossary 373 Index 379