» This session come to elaborate on role of PCIe in enterprise/data centers. First of all short introduction to PCI Express will be done : evolution

Transcription

1 Alex Umansky Leading Israeli PCI Express experts, active participant and contributor to PCI-SIG. Mr. Umansky also serves as Logtel's senior lecturer and trainer on PCI Express and USB 3.0 protocols for leading companies in Israel. Now Alex served as Senior ASIC architect, developing leadingedge Multicore SoC for QoS purposes and Switch Fabrics. Alex has previously served as Senior HW Architect at LucidLogix - multi-gpu SoC solution provider, based on enchanced PCI Express Switch. During his employment in IBM, he worked on the development of several PCIe IP cores and Infiniband products. Alex worked as a VLSI

2

3 » This session come to elaborate on role of PCIe in enterprise/data centers. First of all short introduction to PCI Express will be done : evolution of PCI /PCI-X to PCIe, basic terminology, gen1-4.» Session will focus in three main areas where PCIe is very important for enterprise market : IO virtualization, NVM Express for SSD and PCIe as Data center mini-cluster interconnect

4 » PCI Express evolution from PCI to Terabits fabric» IO virtualization SR-IOV reality and MR-IOV dreams» PCIe SSD and NVMe» PCIe in disaggregated world DC cluster interconnect

5 » PCI Express evolution from PCI to Terabits fabric» IO virtualization SR-IOV reality and MR-IOV dreams» PCIe SSD and NVMe» PCIe in disaggregated world DC cluster interconnect

6 » PCI Express showed up in 2001 as a collection of subprotocols, with standards defining a physical layer, a data link layer, transaction layer and configuration registers space Data LINK Layer Physical Layer

7 » De-facto only connection between CPU and IO s world GPUs and Accelerators/Offloads Ethernet NICs, IB HCAs, FC HBAs SSDs» PCI Express is an interconnect solution for devices belonging to the same board Connectors enable the connection of expansion boards

8 » PCI Express is based on high data rate point-to-point links» PCIe able to double bandwidth roximately every four years, to keep pace with increasing processor performance» Scalable link width» Full duplex operation» Low power» Reliable data transfer on link» Optional E2E CRC» Hot-plug supported» Packet-based protocol, MTU 4KB, in real world ~256B» 8 TCs CoS» ~20y of SW development (BIOS/Kernel/Drivers)

9 » The lane rate : 2.5Gpbs (Gen1), 5 Gbps (Gen2), 8Gbps (Gen3) and 16Gbps (Gen4)» A link can include 1, 2, 4, 8 or 16 lanes Gen1/Gen2 uses 8b10b coding scheme Gen3/4 : 128/130b scheme» Gen3 link bandwidth Number of Lanes x1 x2 x4 x8 x16 Link BW 8Gbps 16Gbps 32Gbps 64Gbps 128Gbps

10 CPU Root Complex x8-16 Gen3 SWITCH Intel Z97 Chipset block diagram by INTEL IO IO IO

11 PCI PCI-X PCIe Gen1 PCIe Gen2 PCIe Gen3 PCIe Gen4 CPU/Cache performance Ordering Enhancements Atomic Operations TLP Processing Hints Power Optimization DPA LTR OBFF IO Virtualization ATS SR-IOV MR-IOV

12 » PCI Express evolution from PCI to Terabits fabric» IO virtualization SR-IOV reality and MR-IOV dreams» PCIe SSD and NVMe» PCIe in disaggregated world DC cluster interconnect

13 OS1 OS2 OS3 OS1 OS2 OS3 IO device IO device IO device Virtualization Intermediary IO (NIC) IO (FC)» Independent Operating Systems (OS)» Each OS owns dedicated physical adapter» Each OS owns the interconnect to communicate with IO Example: Chipset PCI Express bus NIC» Independent Operating Systems (OS)» Each OS owns subset of Shared IO resources» VI responsible for resources allocation and isolation» One physical IC to shared IO, thru RC or Switch

14 13» The SR-IOV standard enables the efficient sharing of PCIe devices among virtual machines running on the same host» SR-IOV defined HW-based IO-sharing mechanism to bypass Hypervisor SW emulated virtualization» That allow to achieve I/O performance that is comparable to native (bare-metal) performance Guest OS VM 0 vnic Driver Guest OS VM 1 vnic Driver Guest OS DOM PF Driver Guest OS HOST A VM 1 VF Driver Guest OS VM 2 VF Driver vswitch Device Driver VMM VMM bypass Physical Device (CFG,BARs,DMAs) Network Link PF (CFG/BARs) VF1 (CFG/BARs) VF2 (CFG/BARs) DMAs, L2 classifiing and switchg Ethernet MAC and PHY

15 14» The MR-IOV defines sharing of PCIe devices among virtual machines running on the different host» New kind of IO devices (SR-IOV is not enough)» New kind of Switches (multi Upstream ports, multi hierarchy) HOST A HOST B Guest OS Guest OS Guest OS Guest OS PF Driver VF Driver VMM PCIe Root Port MR IOV Switch PF Driver VF Driver VMM PCIe Root Port Switch 0 Switch 1 PF0 (CFG/BARs) PF0 (CFG/BARs) VF1 (CFG/BARs) VF1 (CFG/BARs) VF2 (CFG/BARs) VF2 (CFG/BARs) DMAs, L2 classifiing and switchg Ethernet MAC and PHY

16 » PCI Express evolution from PCI to Terabits fabric» IO virtualization SR-IOV reality and MR-IOV dreams» PCIe SSD and NVMe» PCIe in disaggregated world DC cluster interconnect

17 » PCI Express evolution from PCI to Terabits fabric» IO virtualization SR-IOV reality and MR-IOV dreams» PCIe SSD and NVMe» PCIe in disaggregated world DC cluster interconnect

18 » PCIe is High Performance Full duplex, multiple outstanding requests, and out of order processing Scalable port width (x1 to x16) Scalable link speed (2.5 GTps, 5 GTps, 8 GTps) Low latency (no HBA overhead or protocol translation)» PCIe is Low Cost High volume commodity interconnect Direct attach to CPU subsystem eliminates HBA cost PCIe Provides Effective Power Management Direct attach to CPU subsystem eliminates HBA power Link power management Optimized Buffer Flush/Fill (OBFF) Power Budgeting and Dynamic Power Allocation Slot Power Limit CPU SATA SSD CPU PCIe SSD

19 » NVM Express (NVMe) is a register level interface that allows host software to communicate with a non-volatile memory subsystem» Optimized command issue and completion path All parameters for 4KB command in single 64B command» Support for parallel operation deep queues up to 64K commands within a single I/O queue to the device» Enterprise capabilities like end-to-end data protection (T10 DIF and DIX standards), enhanced error reporting, and virtualization» NVM Express 1.0 specification published in March 2011» Latest Spec : Rev 1.1b, July 2, 2014

20 NVM Express is based on a paired Submission and Completion Queue mechanism Commands are placed by host software into a Submission Queue Completions are placed into the associated Completion Queue by the controller Multiple Submission Queues may utilize the same Completion Queue. Submission and Completion Queues are allocated in host memory

21 An Admin Submission and associated Completion Queue exist for the purpose of controller management and control Examples : creation and deletion of I/O Submission and Completion Queues, aborting commands, etc. Only commands from Admin Command Set may be submitted to the Admin Submission Queue An I/O Command Set is used with an I/O queue pair The host selects one I/O Command Set that is used for all I/O queue pairs.

22

23 Command Completion 5. Controller writes completion to Completion Queue 6. Controller generates MSI-X interrupt 7. Host processes completion 8. Host writes updated Completion Queue head pointer to doorbel

24 Optimized Commands Set Only 10 Admin and 3 IO commands mandatory

25 » PCI Express evolution from PCI to Terabits fabric» IO virtualization SR-IOV reality and MR-IOV dreams» PCIe SSD and NVMe» PCIe in disaggregated world DC cluster interconnect

26 #Cores/servers Memory Storage Network Today s Rack ~100(20-40) ~1TB ~100TB(SSD/HDD) 10Gbps /Server 2020, Rack-scale Computer ~100,000s (1,000s) ~100s TB ~100s PB (NVM) 1Tbps/Server New requirement from rack interconnect: Very high bandwidth to blades Aggregated / shared resources Memory & Storage Shared IO to DC network FC HCA, 40/100G NIC Low latency : VM to VM, VM to storage Modularity and Simplification

27 Pooled compute Pooled storage Pooled memory Pooled network (uplink) From Rack Scale Architecture Platform and Management, IDF14

28 » PCIe is Server s gateway to the external world» All IO devices use PCIe» SSD move to PCIe» Direct connection no NICs/HCA/HBA CPU CPU CPU CPU CPU CPU CPU CPU PCIe Fabric NIC HCA SSD SSD SSD SSD MEM MEM Uplink to DC

29 » PCIe topology is a tree Fabric requires progressive topologies : CLOS, fat tree etc» Host-to-Host connection is not PCIe native» Congestion avoidance is not part of PCIe» Error on bus (unpluged device as example) may cause to CPU hang» Multi-host sharing of SR-IOV devices» DCB protocols map to new fabric» Enterprise-class of security, isolation and management Developed for Ethernet

30 » Need more information?» Trainings / Seminars /Consulting : PCI Express (Fundamentals/DeepDive) IO Virtualization NVMe» Alex Umansky alex.umansky@gmail.com

31 » Thanks