How to build a high speed PCI Express bus expansion system using the Max Express product family 1

Transcription

1 Applications The Anatomy of Max Express Cable Expansion How to build a high speed PCI Express bus expansion system using the Max Express product family 1 By: Jim Ison Product Marketing Manager One Stop Systems 2235 Enterprise Street Escondido, CA Date: August 2005 The power of PCI Express cabling The evolution of high-speed serial buses in the desktop PC is redefining how everyone must think about the commercial and industrial computer. Data between a CPU complex and an add-in card is no longer constrained to a printed circuit board where it is broadcast across a shared parallel PCI bus. With the advent of PCI Express (PCIe), data now races across a dedicated device-to-device serial link at 2.5 gigahertz per lane. Along with this increase in speed, the PCI Express bus breaks out of the printed circuit board and can travel up to seven meters and beyond over a shielded copper cable. Having the host computer bus running at 2-80 Gbps on the printed circuit board or on a cable opens many new configurations of the traditional computer. Now one or many add-in boards can reside over 23 feet away from the controlling computer without the use of a high latency network or proprietary expansion product. In other words, PCIe transmitted over a cable provides full host bus performance along with full software transparency. 1 Statements in this document refer to the One Stop Systems, Max Express implementation of PCI Express Cable standard governed by the PCI-SIG. At the time of publishing, the PCI-SIG standard is in draft form and is therefore subject to change by the current active committee until final release. Expected release date is early 2006, shortly after the PCI Express Cable Base Standard 2.0.

2 The concept of using PCIe to the max forms the cornerstone of a new breed of products aimed at taking advantage of these new capabilities. Max Express products, from One Stop Systems, open the world of the industry standard PCIe cable with an entire family of products aimed at high-density expansion, bus conversion, bus extension and high-speed interconnect. PCIe Features Max Express Cable products allow for maximum flexibility in expanding the capabilities of any host system. With an ever-expanding family of cable interfaces and accessories for many industrial form factors, virtually any type of host system can be expanded. First, there are some basic concepts of expansion that will be helpful to define. Lanes and Links PCI Express, in its most basic form, is a serial point-to-point, packet-based device interconnect using both a transmit (Tx) pair and receive (Rx) pair of wires. This four wire interconnect, known as a PCI Express lane, communicates at 2.5 Gbps on each of the Tx and Rx pairs of wires simultaneously for a total bandwidth of 5 Gbps. Each lane embeds its own clock signals into the data stream using a technique called 8- bit/10-bit encoding. Due to this encoding, 10 bits of information is transmitted for each 8-bits (1 byte) of data. Thus a 2.5Gbps transmit rate results in 250MB/s of data throughput. By grouping many lanes together between the same two devices, higher bandwidth interconnects are created up to 32 lanes wide or 160Gbps (16GB/s) total bandwidth. Some applications of PCIe require an external reference clock, such as when transmitting over a cable. Along with other signals such as reset, power-on and wake, the clock is part of side band signals needed to enhance PCIe. The PCIe lanes between devices along with the sideband signals create what is know as a PCIe link. Links are designated x1 (pronounced by one ), x4, x16, etc. based on the number of lanes forming the link. PCI Express 2.0 (Gen 2) signaling available in 2007 doubles the clock rate of the lanes to 5 GHz effectively doubling the bandwidth of every size link (i.e. PCIe 2.0 x1 link = 5Gbps each direction). Upstream vs. downstream The concept of upstream and downstream in basic PCI Express is simple but can become more complex in larger expansion systems depending on the point of view that is being considered. Figure 1: Upstream/Downstream Page 2 of 2

3 Basically, upstream refers to a direction closer to the root complex of the primary host system. Conversely, downstream is the direction farther from the root complex. As shown in Figure 1, from the point of view of device A, the root complex is upstream and device B is downstream. In other words, device A is upstream of device B because it is closer to the root complex. With PCIe defined as a point-topoint interconnect between two devices, device A in Figure 1 is likely a repeater or two-port PCIe switch. Quality of Service Quality of service (QoS) is a term used to define many features combining to provide predictive allocation of available bandwidth. It is a critical feature to PCIe that is not available in shared parallel busses and provides a powerful tool to first and secondgeneration PCIe developers. QoS assures the data is available to the application when it is needed whether it transmitted across a backplane or over a cable. QoS is best understood with an example that is familiar, such as Internet access. Suppose there is a 1Gbps Internet backbone entering a mixed-use high-rise building. In this building there are 15 residential subscribers to the link at $15/month, 5 small business subscribers at $50/month and 1 large business subscriber at $500/month. If all of the subscribers are downloading files at the same time, exceeding the 1Gbps bandwidth, which deserves the priority to for the available bandwidth? Considering the large business subscriber pays more than everyone else combined, the ISP controlling the bandwidth would like to allocate the largest share of the available bandwidth to the large company. This would give the large company a higher quality of service, the small businesses a lower QoS, and the residential subscribers the lowest QoS. PCI Express includes both hardware and software methods for a system designer to designate QoS levels for data. Virtual Channel (VC) buffers in PCIe hardware devices allow traffic to be routed at one of 8 priority levels along the same physical link. Devices can be programmed to let higher priority traffic such as interrupts to be sent along a link before lower priority traffic such as chassis management information. Software controlled prioritization occurs similarly through the use of Traffic Classes (TCs). There are up to 8 traffic classes that can be designated in the data packet header for priority routing. When used together VCs and TCs provide up to 64 possible priority levels for traffic routing. Other QoS features include enhanced flow control and poison packet tagging. Flow control is built into PCIe hardware devices as a way to prevent data overflow along a PCIe link. The downstream device on a link such as a switch can communicate with upstream devices to prevent more data from entering the device than it can process with available buffers. Poison packet tagging is a single bit that can tag a bad packet at the first device. This allows other devices along the path to ignore the bad Page 3 of 3

4 packet. All of these features are designed to increase throughput and reduce latency over traditional proprietary cable interconnects. Many of these QoS enhancements are built-in to the PCIe hardware with no additional software. Software Transparency Arguably the most compelling feature of PCI Express is the software transparency. PCI-based operating systems including Windows and Linux automatically support PCIe devices and drivers. Many applications written for standard PCI devices are fully functional with similar PCIe devices. Software tools used to manipulate PCI devices at a more intimate level continue to be used for PCIe devices. The high-speed serial link with built-in quality of service features and software transparency provides a compelling cable expansion solution. Max Express cable expansion products are a natural extension of the PCIe host bus architecture instead of an adapter to some other cabled expansion architecture. While other architectures use valuable time converting from the host PCIe bus to the cable bus and back again, Max Express is busy moving your PCIe data across the cable in native, low-latency PCIe format. PCIe transmitted over a cable provides full host bus performance along with full software transparency. Max Express Expansion Basics Max Express cable expansion products consist of three major building blocks, the host interface, the cable and the expansion interface. The upstream host interface and downstream expansion interface form the endpoint devices of the PCIe bus on the cable. These endpoints are PCIe devices such as a switch, bridge, FPGA or any other chip with a PCIe interface. This section is devoted to the basic building blocks of an expansion system. More advanced expansion systems and applications of the Max Express products are described in later sections. Figure 2: Max Express HIB Page 4 of 4

5 Host Interface Board The Host Interface Board (HIB) is the upstream device of the cable. The most common form factor Max Express HIB is the familiar desktop style PCIe card shown in Figure 2. Other Max Express HIB models are designed to interface with industrial form factors such as CompactPCI, CompactPCI Express and other industry standards. A PCIe switch on the HIB allows the board to act as the both the downstream device to the host PC and the upstream device to one or more downstream, cabled PCIe busses. This is illustrated in Figure 3. From the point of view of the Figure 3: HIB Functional Diagram HIB board, the PCIe bus to the root CPU complex is on the upstream port of the switch and travels through the card edge connector. The PCIe bus leaving the front panel via a cable connector is on a downstream port of the switch. In this basic two-port configuration, the switch is acting as a repeater between the upstream PCIe bus and the downstream PCIe bus. This repeater function is necessary to maintain the noise requirements for the cabled PCIe bus as mandated by the PCI-SIG PCI Express Cable standard. Some Max Express HIB models are outfitted with multiple downstream PCIe busses for internal or external cabling as well as special features such as creating a high-speed link between two hosts, which is discussed later. PCI Express Cable The Max Express PCIe cable forms the physical means of transmitting the PCIe bus from the host to the end point device. The maximum cable length possible for internal or external PCIe communications is purposely not defined by the PCI-SIG Cable standards, although the maximum noise budgets across a cable and signal strength are well documented. Measured noise levels in a 7 meter cable have been reported by PCI-SIG members to be a practical maximum. At the time of publication, the PCI-SIG cable standard is in draft form awaiting the latest release of the PCI Express Cable Draft Revision 0.5. This 0.5 draft revision goal is to solidify the mechanical connector design for PCIe 1.1 and PCIe 2.0 Page 5 of 5

6 signaling and allow for industry design 2. An example draft revision 0.4e x4 cable with the corresponding connector/shield combination is shown in Figure 4. The cable is designed with keying features on the cable shell and mating shield to designate PCIe 1.x and 2.0 standards. The connector also has positive latching mechanisms designed to remain within the connector width. The cable itself Figure 4: PCI Express x4 Cable contains the differential PCIe signal pairs as well as additional differential and single-ended side band signals. The side band signals include a differential 100MHz reference clock as well as single-ended reset, board present and power on signals. There is also a single ended signal return and several grounds. Optional signals on a x4 cable include +3.3V power and power return for active connector noise reduction (not bussed over the cable) and a wake signal. There are also two reserved signals that are not bussed over a cable on the x4 and up to 12 reserved signals on a x16. This assortment of PCIe signals and sidebands produce a cable with a minimum of 18-conductors for x1 up to 36- conductors for a x4, 68-conductors for a x8 and 136-conductors for a x16. It is necessary to consider overall system design, especially when using the larger external cables. One Stop Systems expert engineers provide the OEM designer a powerful ally in assuring accurate system design before, during and after the sale. Expansion Link Boards The board or device at the downstream end of the cable may be any device with the corresponding cable connector. Such devices are planned or in production from various manufacturers including high speed instrumentation devices, disk drive arrays, high-speed host-to-host communications and bus expansion devices. The primary Max Express product family providing bus expansion are Expansion Link Boards (ELBs) as shown in Figure 5. The ELB is available in many form factors Figure 5: Max Express ELB 2 7 meters is the rule of thumb with an unpowered connector at 2.5Ghz PCIe 1.1 (generation 1) signal. Optionally powered connectors, for active equalization, may allow cables supporting up to 10 meters. PCIe 2.0 (generation 2) 5Ghz will be somewhat shorter due to a lower noise budget. Page 6 of 6

7 including industry standards such as SHB Express and CompactPCI Express as well as custom embedded form factors. The ELB serves two primary functions in bus expansion. First, the ELB is the downstream end of the cabled PCIe Bus. Second, the ELB provides fan-out to one or more downstream add-in board slots. This is accomplished through the use of a PCIe switch as shown in Figure 6. The ELB generally has more ports (i.e. a larger switch) than an HIB for the fanout to multiple add-in board slots. Max Express Cable Expansion System The HIB, Cable and ELB are the heart of the Figure 6: ELB Slot Fan Out PCIe expansion system but the functionality of the entire system depends on another critical component, the expansion backplane. The most common application of PCIe cable expansion is to increase the number of PCIe add-in board slots available using a single host. The goal of Max Express is to use standards-based building blocks in fulfilling this application. In this case, a standard PCIe HIB, standard cable and SHB Express ELB form the basis of this system. SHB Express is a PICMG industry standard (PICMG 1.3) chosen for its generous mapping of up to 20 lanes of PCIe bus to the host slot in a backplane. The ELB plugs into the host slot of this SHB Express expansion system backplane allowing the host at the upstream end of the cable to transparently control the expansion system. The SHB Express standard allows for flexible backplane configurations to meet a variety of applications. With methods long understood in the industrial computer market through the evolution of the PICMG 1.x family of standards, backplanes from 4-slots in a 1U enclosure, up to 20 slots in a 4U enclosure are possible in 19 rack mount configurations. An example 20-slot backplane is shown in Figure 7. This backplane allows the upstream host to control up to 19 additional standard PCIe add-in boards via the ELB. This is possible by the addition of switches on the backplane for fan-out to all 19 slots of PCIe. Figure 7: 20-Slot SHB to PCIe Backplane Page 7 of 7

8 Figure 8: 11-Slot SHB to PCI/PCI-x/PCIe Backplane To change the character of the expansion system, only the backplane needs to change. If, for example, only 10 expansion slots are required and a mixture of PCI, PCI-X and PCIe are needed, then the 11-slot backplane shown in Figure 8 need be the only part substitution in the system configuration described above. In this case, some of the PCIe lanes from the ELB are bridged to PCI and PCI-X slots directly on the backplane. When the components described in this section are combined the result is a powerful expansion system. Figure 9 shows a demonstration expansion system consisting of a standard PCIe motherboard expanded with an HIB to an expansion system. The expansion system consists of a single slot PCIe backplane that has a PCIe SATA HDD controller. This demonstration system simulates a PCI Express disk drive array as large amounts of data are transferred across the cable transparently to the system host. Figure 9: Demonstration PCIe Expansion System Page 8 of 8

9 Advanced Max Express Expansion Every OEM has a different set of requirements that may deviate from the basic expansion system above. This section introduces other Max Express products used to expand systems using industrial form factors or large numbers of add-in boards. Also, there are Max Express products to remotely mount a single add-in board or to link two host systems together over a high-speed PCIe link. PCI Express Rack Mount Switches Nineteen add-in boards may not satisfy the requirements of some applications. One Stop Systems announced the first 8-port rack mount PCIe switch shown in Figure 10 in August 2005 to address these requirements. The rack mount switch acts in a similar fashion to other rack mount switches used for various applications, such as Ethernet, Infiniband and Fiber Channel. The key difference with PCIe is the device is switching the native host add-in board bus and not a secondary network or storage interconnect. This feature means the PCIe switch is extending the reach of the host root complex across several other chassis with each cable segment up to 7 meters long. An end result is the host bus can be expanded to over 100 addin boards across a room transparently to the host. Bus Conversion Figure 10: 1U 8-Port PCIe Switch The basic bus expansion system is the most popular way to use the Max Express Cable products but the PCIe cable also serves another class of application. It is referred to as bus conversion. Bus conversion adds expansion slots of a different form factor to a host system. For example, a standard motherboard with PCI and PCIe slots can be expanded across a cable with CompactPCI Express slots or CompactPCI slots. Conversely, a CompactPCI host system can be expanded with off the shelf PCI or PCIe cards. A list of Max Express model designations for all cable endpoint cards is in Table 1. To construct an expansion system between any bus combination, simply choose a product from the upstream list to match your host form factor then choose a product from the downstream list to match your desired expansion form factor. Page 9 of 9

10 Table 1: Max Express expansion and conversion products Model Designator 3 Bus Type Form Factors Supported Upstream Boards HIB PCI Express PCIe HBB PCI-X PCI, PCI-X HEB CompactPCI, PXI 3U, 6U SW CompactPCI Express 3U, 6U Downstream Boards ELB-SH SHB Express PCIe, PCI, PCI-X HEB CompactPCI, PXI 3U, 6U ELB-xE CompactPCI Express 3U, 6U HIB-T PCI Express PCIe ECA PCI Express PCIe When one of the form factors involved in bus conversion is PCI or PCI-X based instead of PCIe, the corresponding interface board uses a PCI(-X)-to-PCIe bridge in place of a switch to access the PCIe cable. This allows the latest high-speed PCIe motherboards to power legacy PCI/PCI-X devices preserving prior investments of an OEM. With the inherent compatibility of PCIe to PCI/PCI-X, this remains a driverless and transparent operation. Below is a summary of the bus conversion products in Table 1: HBB: This is a host bridge board that plugs into a PCI or PCI-X slot in the host system. A PCI(-X)-to-PCI bridge converts the parallel PCI based signals into serial PCIe signals for transmission over the PCIe cable. HEB: The Host/Expansion bridge board performs the same function as the HBB above for the CompactPCI bus. A key feature of the HEB is the reversible bridge that allows the board to be used in upstream host or downstream expansion mode. When the HEB is plugged into a CompactPCI peripheral slot it operates as the upstream host cable interface. When plugged into a CompactPCI system slot it operates as a downstream expansion interface. SW: It is common to use a switch board in a CompactPCI Express switch slot to allow for fan-out to more slots in a system than the system master can directly support. The Max Express switch boards are equipped with a cable interface to allow for cable expansion to additional backplane slots or expansion systems. 3 More than one model may be available for the listed form factors in various lane configurations. All models may not be in current production and more models for other form factors may be available. Refer to for the latest product availability. Page 10 of 10

11 ELB-xE: The xe model ELB boards plug into a CompactPCI Express system slot. Like the basic SHB Express ELB model, a PCIe switch on the board connects the cable to multiple slots on the CompactPCI Express backplane. HIB-T: While the ELB-SH is designed to support multiple PCIe, PCI or PCI-X addin boards, a special version of the HIB can be used to support a single PCIe add-in board. The HIB-T is configured as the downstream device with a special backplane for single add-in board. This is referred to as a slot extension application and is a powerful way to use high-speed PCI Express cabling to replace expensive and possibly slower fiber channel, SCSI or Infiniband busses to a drive array as shown in Figure 5. ECA: The embedded cable adapter enables any PCIe board to be plugged into a cable. This versatile board provides card-edge to cable connector conversion as well as power to the PCIe board in a compact form factor. The small adapter allows any PCIe card to stand alone in an embedded application without the use of a backplane. High-speed host-to-host link There is another application of PCI Express cabling besides expansion and conversion called host-to-host link. Almost every application using a computer network can take advantage of the higher speeds of PCIe cabling. By using two HIB boards, each in a host system a 10Gbps link can be established between the two systems. The Max Express HIB supports the physical link between the two host busses through the use of a non-transparent port on one of the HIB boards. Since this is linking two host busses and not a network, additional software is required in the link application to transmit data across the PCIe cable. A diagram of a host-tohost link is shown in Figure 11. Figure 11: High-speed host-to-host link Page 11 of 11

12 Internal PCI Express Cable Expansion Internal PCIe cabling is available on various MaxExpress models to facilitate extension or isolation of certain boards in the same chassis. If the chassis is providing the proper EMI shielding and the cable lengths are less than 7 meters, an industry standard Serial Attached SCSI (SAS) cable is employed. While this not defined in the PCI-SIG PCIe Cable standard, extensive modeling and testing have been conducted to assure the integrity of the PCIe bus. Test data is available to One Stop Systems customers by contacting Technical Support at (760) The Future of Max Express Expanding Universe The use of cable expansion, conversion, extension and linking opens up a new world of possibilities for the host computer bus that was only the domain of other protocols of the past. Many other cable end-point devices for cable expansion will be available for several different form factors as product need arises. Advanced Mezzanine Card (AMC) and Switched Mezzanine Card (XMC) form factors are other PCIe based technologies poised to use the cable interface. Mainstream motherboards and laptop computers are in development by various manufacturers to incorporate PCIe cable connections much like USB ports. This will make connection to the Max Express expansion chassis very simple. Advanced Switching A natural extension of PCIe is Advanced Switching (AS). AS extensions for PCIe give additional functionality to the PCIe transaction layer. By adding these extensions the PCIe bus is transformed to more of a network topology with all of the associated network features. With AS, there is no longer the concept of upstream and downstream allowing true peer-to-peer networking at incredible speeds reaching 160Gbps by The most compelling feature remains the backward compatibility to PCIe. This compatibility allows the host bus and computer network of unlimited nodes to communicate in the same PCIe-based protocol without the need of conversion to a non-pcie protocol such as Ethernet. The Next Generation Second generation PCIe signaling, formerly called Gen2 and officially released as PCI Express Base Specification 2.0, doubles the PCIe clock rate to 5GHz. This effectively doubles the bandwidth of every PCIe portion of the system, weather it is Page 12 of 12

13 the host bus or a cabled expansion bus. PCIe 2.0 affects cabling in two ways. First, cable lengths are reduced due to a lower noise budget than with 2.5GHz signaling. This will likely limit the cable to around 3-4 meters without powered active equalization to the cable connectors. Second, different physical cables will be required to handle the higher frequency. The PCIe 2.0 cables will have connectors keyed to allow only these PCIe 2.0 compatible cables to be used in PCIe 2.0 connectors. The keying will also allow the PCIe 2.0 cables to be used in PCIe 1.x connectors but not vice versa. The latest Max Express products are found at It is recommended to check the latest motherboard and downstream device compatibility list to find MaxExpress tested configurations. It is also recommended to update your boards with the listed BIOS and drivers from the respective manufacturers on the list. This list is available from One Stop Systems Technical Support at (760) Page 13 of 13