Dlphin Intercnnect Slutins Dlphin Express IX Reflective Memry / Multicast Whitepaper Dlphin Engineering 3/24/2015
DISCLAIMER DOLPHIN INTERCONNECT SOLUTIONS RESERVES THE RIGHT TO MAKE CHANGES WITHOUT FURTHER NOTICE TO ANY OF ITS PRODUCTS AND DOCUMENTATION TO IMPROVE RELIABILITY, FUNCTION, OR DESIGN. DOLPHIN INTERCONNECT SOLUTIONS DOES NOT ASSUME ANY LIABILITY ARISING OUT OF THE APPLICATION OR USE OF ANY PRODUCT OR DOCUMENTS. Ntes This dcument is based n infrmatin available at the time f publicatin. While effrts have been made t be accurate, the infrmatin cntained herein des nt purprt t cver all details r variatins in hardware and sftware. Trademarks SCRAMNet is a registered trademark f Systran Crpratin. GE FANUC is a registered trademark f GE Fanuc Autmatin Inc. Windws is a registered trademark f Micrsft Crpratin. Page 1 Dlphin Express IX reflective memry
Table f Cntents DISCLAIMER...1 Ntes...1 Table f Cntents...2 Intrductin...3 Multicast implemented in hardware...3 Traditinal reflective memry...3 PCI Express reflective memry...4 Multicast memry and multicast grups...4 Using PCI Express reflective memry...5 Transmitting data t reflective memry...5 Reading Data frm reflective memry...6 Interrupts...6 Significant benefits prvided by PCI Express...6 Perfrmance...7 Hardware cnfiguratin and installatin...8 Reflective Memry Cmparisn...8 Radmap and future plans...9 SISCI API...9 SISCI API Cde examples...9 Reference and mre infrmatin... 10 Page 2 Dlphin Express IX reflective memry
Intrductin The Dlphin Express IX prduct family supprts multicast peratins as defined by the PCI Express Base Specificatin 2.1. Dlphin has integrated supprt fr this functinality int the SISCI API specificatin t make it easily available t applicatin prgrammers. The cmbinatin f Dlphin PCI Express hardware and the SISCI API creates a slutin fr custmers seeking multi-cast r reflective memry type functinality. The first Dlphin Express prduct line was intrduced in 1994 and has been fllwed by several generatins f shared memry slutins. The Dlphin Express IX prduct is ur secnd generatin f intercnnect prducts supprting a real hardware based multicast implementatin. PCI Express multicast enables a single bus write transactin t be sent t multiple remte targets r in PCI Express technical terms - multicast capability enables a single TLP t be frwarded t multiple destinatins. Dlphin cmbines PCI Express multicast with ur SISCI API. The cmbinatin allws custmers t easily implement applicatins that directly access and utilize PCI Express reflective memry functinality. Nw, applicatins can be built withut the need t write device drivers r spend time studying PCI Express chipset specificatins. The advantage f the PCI Express reflective memry apprach is lwer latency and higher bandwidth. Dlphin benchmarks shw end-t-end latencies as lw as 0.99 micr secnds and ver 2,650 Megabytes /sec dataflw at the applicatin level. These benchmarks are included in the SISCI develper s kit. By using PCI Express based reflective memry functinality, custmers can easily slve their real time, distributed cmputing perfrmance requirements. Multicast implemented in hardware Reflective memry systems (in cmputer literature als referred t as mirrr memry systems, replicated shared memry, multicast r replicated memry systems) implement transparent and autmatic updates f remte memry areas. Reflective memry is typically mapped int an embedded system applicatin and enables similar applicatins n ther ndes t share updated data withut invlving any traditinal netwrking prtcl and verhead. Data f any size is autmatically transmitted t all ndes directly by functinality implemented in hardware. Typical applicatins can range frm a tw-nde fail ver pair t large distributed shared memry applicatins like aircraft, ship and submarine simulatrs, autmated testing systems, industrial autmatin, electrnic trading, cntrl, nline and high-speed data acquisitin and distributin. Because f their inherent replicatin they are especially gd fr fault tlerance. Traditinal reflective memry Other reflective memry type slutins typically implement reflective memry by prviding a plug-in adapter card with nbard device memry. Applicatins can write t this memry and the data is autmatically frwarded thrugh t all ther ndes cnnected. Applicatins reads data frm the lcal adapter card device memry. A ring netwrk tplgy cnnects the systems tgether. A typical 4 nde cnfiguratin can be seen in the figure belw. Page 3 Dlphin Express IX reflective memry
Figure 1 : Alternative types f reflective memry implementatin PCI Express reflective memry The Dlphin slutin is unique as it is able t utilize the cmputer system s standard main memry. This, cmbined with regular PCI Express technlgy running at wire speeds f 40Gbps gives significant perfrmance imprvements. Figure 2 Dlphin Express IX reflective memry setup The figure abve visualizes a typical Dlphin Express setup. Dlphin IXH610 and IXH620 cards are cnnected thrugh an IXS600 switch. Neither the IXH610 nr IXH620 card has any memry used fr string reflective memry data, resulting in significant perfrmance and cst benefits. The IXS600 switch prvides a mechanism fr simultaneus multi-cast f data t all cnnected prts with a measured prt t prt latency less than 200 nansecnds. Multicast memry and multicast grups Dlphin Express IX supprts up t 4 independent glbal multicast grups / memry segments. This enables SISCI prgrams t use up t 4 independent reflective memry regins and cntrl which ndes receive the multicast data. This differs frm ther reflective memry slutins frm ther vendrs which nly supprt a single multicast grup. Page 4 Dlphin Express IX reflective memry
If a multicast grup segment des nt exist in a system, the multicast data will be silently drpped withut any ntificatin. Multicast data filtering is dne by each cnnected adapter. Ndes can be rebted and multicast segments can be added r remved at any time withut any synchrnizatin with the ther ndes. Each multicast segment can be up t 2 Gigabytes with a ttal reflective memry size f 8 Gigabytes. The current driver release - DIS 4.4.3 - supprts a max segments size f 64 Megabytes and a ttal reflective memry size f 256 Megabytes. The Dlphin driver allcates the reflective memry segment memry frm main memry during driver startup. The upcming DIS 5.0 sftware distributin will supprt the 8 Gigabyte ptin. A PC server with large PCI BARS is required t supprt the 8 Gigabyte ptin. If yu wuld like t use reflective memry segments larger than 256 Megabytes, yu shuld ask yur system vendr t cnfirm the system BIOS supprts memry mapped I/O abve 4GB (large Base Address Register supprt per the PCIe specificatin). Using PCI Express reflective memry The majr difference between traditinal reflective memry slutins and PCI Express apprach t reflective memry is that the PCI Express slutin utilizes tw different addresses, ne fr reading and a different address fr writing data. The SISCI API prvides these addresses during initializatin. The write address is inside the PCI Adapter address space. Any write t this address space will typically trigger an address translatin inside the PCI adapter and cause PCIe transactins t be sent t the IXS600 switch and ther ndes. The result f reading this address is undefined. Transmitting data t reflective memry Data can be transferred t ther ndes using the reflective memry slutin in the fllwing ways: CPU: Data can be sent t reflective memry using ne r mre CPU psted write instructins. Using SISCI, applicatins the flexibility t d a standard memcpy() using the reflective memry as a target r d a regular pinter assignment. The fully hardware based memry mapped data transmissin des nt rely n any perating system service r kernel driver functinality and prvides the best pssible deterministic data transmissin latency and jitter. PCIe device: custmers can use the SISCI API t cnfigure and enable GPUs, FPGAs etc. (any PCIe master device) t send data directly t reflective memry. (Aviding the need t first stre the data in lcal memry). Onbard DMA: The Dlphin Express IX adapter card includes an efficient scatter / gather DMA engine that can be engaged t send small r larger amunts f Figure 3: FPGA direct transmissin data t reflective memry. This functinality is available with the DIS 4.4.4 r newer sftware release frm Dlphin. The figure belw shws the flw f data (indicated by the red arrw) frm the CPU f cmputer 1 - t a lcal memry address allcated fr a specific reflective memry grup ID. Data will be transmitted by the PCI Express hardware int the main memry f all ther ndes in the netwrk that has allcated a reflective memry segment fr the same grup ID. All f this is easily managed thrugh the SISCI API. In this example grup ID includes cmputers 3 and 4, des nt include cmputer 2. Page 5 Dlphin Express IX reflective memry
Reading Data frm reflective memry T read data received frm ther ndes, the applicatin needs t use the read address, this pints t the allcated segment in lcal main memry. If a lcal reflective memry update is needed, applicatin prgrammers need t cpy the sent data t the lcal buffer as well. This is a very lw cst peratin as the data is already in the CPU cache. Interrupts The SISCI API prvides functinality t register and trigger applicatin interrupt s int a remte nde. Please cnsult the SISCI Users guide fr details n using SISCI interrupts. Significant benefits prvided by PCI Express The PCI Express based reflective memry slutins prvides significant imprvements ver alternative slutins: Data in main memry: The Dlphin Express IX reflective memry slutins utilize main memry t stre data. This has several significant benefits: Reading data in main memry is significantly faster than slutins string data in specialized PCIe device memry lcated in the cmputer IO system. Main memry is cached: This means that the slutin will benefit frm the standard CPU cache when reading data. Reflective memry updates frm remte will autmatically invalidate the CPU cache and ensure full data cnsistency. Specialized device memry is nrmally very expensive vs main memry mdules. Yu dn t need t specify the reflective memry size when buying hardware. The size f Dlphin Express IX reflective memry is user cnfigurable a prperty set by the applicatin during initializatin f the system. Data is multicast by a centralized switch. Each IXS600 switch will send data ut n all cnnected prts simultaneusly. This means that all ndes will receive data virtually simultaneusly when cnnected t a single switch. When multiple switches are used, each switch hp will add less than 200 nansecnds delay t the distributin f the data. Page 6 Dlphin Express IX reflective memry
Alternative slutins using a ring tplgy t distribute data have significant delays between when the first and the last nde in the netwrk receives the data. Each nde will typically intrduce a fixed delay; the ttal delay in the netwrk varies depending n the number f ndes. The minimal delay intrduced by Dlphin Express IX reflective memry enables real-time applicatins t benefit frm a significantly reduced ttal cmmunicatin time allwing the applicatin t run at a faster simulatin frequency r spend mre time n cmputatin. Dead ndes r unplugging cables will nt stp the entire netwrk; all ndes that remain cnnected t the netwrk will be able t cmmunicate withut interruptin. Hardware based CRC and retransmissin. PCI Express implements a reliable data transmissin by calculating a CRC fr every data packet. Crrectable link errrs will autmatically cause a hardware retransmit. Fair arbitratin and sharing f bandwidth. Hard real-time systems shuld nrmally be cnfigured t avid narrw bttlenecks in the netwrk. PCI Express uses a fair, rund rbin allcatin f resurces and prvides a very deterministic data transmissin even under maximum lad. Perfrmance The Dlphin IXH610 adapter and IXS600 switch utilizes standard x8 PCI Express link enabling custmer applicatins t take advantage f the exceptinal 40Gb/s link bandwidth. Dlphin reflective benchmarks included in the SISCI develper s kit can be used t measure the reflective memry perfrmance f yur system. The actual perfrmance will slightly vary dependent n the cmputers IO system, but typically yu shuld expect end t end latencies as lw as 0.99us and ver 2,650 Mega Bytes per secnd dataflw at the applicatin level as shwn n the figure belw. The SISCI reflective memry example reflective_bench can be used t measure the thrughput vs message blck size. The prgram is included in the Dlphin sftware distributin package. PCIe Multicast Thrughput 3000 2500 2000 1500 1000 500 0 4 8 16 32 64 128 256 512 1K 2K 4K 8K 16K 32K 64K 128K 256K 512 1M 2M 4M 8M Figure 4: Reflective_bench results Page 7 Dlphin Express IX reflective memry
Hardware cnfiguratin and installatin T create a reflective memry system with Dlphin prducts, each nde must have a Dlphin Express IXH610/ IXH611 r Dlphin Express IXH620 XMC adapter card in NTB mde installed. A Dlphin IXS600 can be used t cnnect multiple systems. Up t 8 systems can be cnnected t a single IXS600 8 prt PCI Express Gen2 switch. Fr larger reflective memry systems, IXS600 switches are cascaded t create larger tplgies. Initially up t 20 ndes are supprted with cmplete SISCI functinality. Up t 56 ndes are supprted when limited t just reflective memry functinality. Please refer t the actual sftware release nte fr cnfiguratin details. The reflective memry functinality is nly available when an IXS600 switch is cnnected. Tw adapter cards can cmmunicate using a direct cable using the standard SISCI unicast functinality (write t nly ne remte nde). SISCI reflective memry supprt is targeted at Linux, Windws r RTX perating systems. The ndes can be running any f the abve perating systems and inter-cmmunicatin between Linux, Windws and RTX systems is fully supprted. VxWrks 6.9 supprt is planned fr Q2 2015. PCI Express reflective memry is nt limited t server ndes. PCI Express devices are als supprted. Data frm attached GPUs, FPGAs etc can be distributed t multiple remte ndes simultaneusly by simply attaching the PCIe device t a regular PCI Express slt in any f the cmputers. Additinal infrmatin can be fund in the reflective_device.c example prgram included in the Dlphin sftware distributin package. Reflective Memry Cmparisn The varius reflective memry systems available have different prperties. Generally, PCI Express based reflective memry has significant lwer latency, higher thrughput but currently supprts less ndes and distance. Details fr sme ppular reflective memry slutins can be fund in the table belw. Feature Dlphin Express IX GE Fanuc SCRAMNet GT Standard PCI Express Prprietary Prprietary Netwrk speed 40 Gigabit/s 2.12 Gigabit/s 2.5 Gigabit/s Netwrk tplgy Switch Ring Ring Max ndes 56 / 20 *3 256 256 Max distance end t end 600 meter Up t 10 km Up t 30 km Cables ipass Cpper r fiber Fiber SFP cpper r fiber Data Deliver Jitter 200 ns pr switch hp 1 us pr nde Less than 1 us pr nde 8 ndes 1us 8 us < 8 us 20 ndes 1.4 us 20 us < 20 us 56 ndes 1.4 us 56 us < 56 Transfer methds PIO, DMA *1, PCIe PIO, DMA PIO master Write perfrmance PIO 2650 Megabytes/s 26 Megabytes/s 210 Megabytes/s Write perfrmance DMA *1 170 Megabytes/s NA Read perfrmance PIO 20 Gigabytes/s *2 6 Megabytes/s Read perfrmance DMA 3400 Megabytes/s *1 408 Megabytes/s NA Number f multicast grups 4 1 1 Max Memry cnfiguratin 4 x 2 Gigabytes 256 Megabytes 128 Megabytes Type f Memry System main memry Device memry Device memry Fixed memry settings N, sftware cnfigurable Yes, card is rdered with a specific memry size Yes, card is rdered with a specific memry size Memry is cacheable Yes N N Remte interrupts Yes Yes Yes The data in the table is fund by ggling fr reflective memry and SCRAMNet. Please let us knw if the data is incrrect. 1)The DIS 4.4.3 supprts up t 4 x 64 Megabytes, 256 Megabyte reflective memry segments. The 4x 2 Gigabyte ptin and DMA peratins are available with the DIS 5.0 r newer sftware release. Please cntact Dlphin fr mre infrmatin. 2) Actual thrughput depends n the lcal system memry t memry Page 8 Dlphin Express IX reflective memry
bandwidth. 3) Scalability, the IX hardware limits the number f ndes that can be used fr general purpse, unicast, interrupts t 20. The reflective memry functinality nly is limited t 56 ndes. Dlphin is wrking with PCIe chip vendrs t ensure future slutins will scale t 256 ndes r mre. Radmap and future plans Dlphin s reflective memry slutin utilizes the standard multicast functinality as defined by the PCI Express Base Specificatin 2.1. Upcming PCI Express Gen3 and future PCI Express Gen 4 chipsets will further increase the perfrmance and scalability fr applicatins utilizing PCI Express multicast. Dlphin is cmmitted t maintain a stable SISCI API t enable custmers an easily upgrade t new future PCI Express based multicast slutins. SISCI API The SISCI API (Sftware Infrastructure Shared-Memry Cluster Intercnnect) cnsists f driver and API sftware, tls, dcumentatin and surce needed t develp yur wn embedded applicatin utilizing the lw latency and high perfrmance f a PCI Express Cluster. The SISCI API prvides a C system call interface t ease custmer integratin f PCI Express ver cable slutins. SISCI enables custmer applicatins t easily and safely bypass the limitatins f traditinal netwrk slutins, aviding time cnsuming perating system calls, and netwrk prtcl sftware verhead. SISCI resurces (memry maps, DMA engines, Interrupts etc) are identified by assigned IDs and managed by a resurce manager enabling prtability and independent applicatins t run cncurrently n the same system. The SISCI API has been defined in the Eurpean Esprit prject 23174 as a de fact industry standard Applicatin Prgramming Interface (API) fr shared memry based clustering. In additin t the reflective memry/multicast functinality, the SISCI API prvides functinality t access remte memry fr unicast (single remte read r write), Direct Remte DMA (RDMA) using the nbard DMA engine. The API als includes supprt fr sending and receiving remte interrupts and errr checking. SISCI als supprt PCIe peer t peer cmmunicatin ver the PCIe cable. SISCI API Cde examples The SISCI Develpers kit cntains several basic cde examples t demnstrate the use f SISCI and the reflective memry functinality. A gd starting pint fr reflective memry is reflective.c (click t pen the surce). Please cnsult the SISCI API reference manual fr mre details. Page 9 Dlphin Express IX reflective memry
Reference and mre infrmatin Please visit www.dlphinics.cm fr additinal infrmatin n the Dlphin Express IX prduct family. Additinal infrmatin including the SISCI Users guide and the nline SISCI API reference manual can be fund at http://www.dlphinics.cm/prducts/embedded-sisci-develpers-kit.html Additinal white papers n the Dlphin Express technlgy are currently available: SuperSckets fr Linux SuperSckets fr Windws Dlphin Express Reflective Memry / Multicast (This dcument) Dlphin Express Peer t Peer cmmunicatin Direct PCIe Please cntact pci-supprt@dlphinics.cm if yu have any questins. Page 10 Dlphin Express IX reflective memry