Lessons learned from Run2 C-RORC/Clusterfinder Development

Transcription

1 Lessons learned from Run2 C-RORC/Clusterfinder Development Heiko Engel

2 C-RORC Hardware Timeline Kickoff Meeting Start PCB Layout Purchase Preparations Pre-Series Boards done Schematics done First Prototypes MS done IT started No (real) technical / hardware problems Only administrative / contractual delays MS: Market Survey IT: Invitation to Tender ATLAS joins 2 nd batch Prototypes Production Contract Series Production completed

3 C-RORC Hardware Timeline Virtex-6 was available as Engineering Sample when the project started In the meantime: Stratix-V Arria-10 (Stratix-10) Kintex-7 Virtex-7 (Ultrascale) FPGA was outdated before first C-RORC was produced Support countless hours of preparing framework of firmware, driver, testapps,... to verify correct hardware behavior manufacturer support & user support

4 Cluster Finder in VHDL Online pre-processing of TPC data in the RORC FPGA significant performance edge over software implementation data compression factor Run1: ~ 1.3 potentially higher: move SW compression into FW FPGA resource usage for single Cluster Finder instance Fast Cluster Finder Post-Synthesis Results H-RORC RCU1 (Virtex-4 / DDL1 / 166 MHz) C-RORC RCU1 (Virtex-6 / DDL1 / 160 MHz) C-RORC RCU2 (Virtex-6 / DDL2 / 320 MHz) LUTs ~ (LUT4) ~ (LUT6) ~ (LUT6) FFs ~ ~ ~ BRAMs 39 (RAMB16) 19 (RAMB36) 15 (RAMB36) DSPs % FPGA Resources ~ 30 % ~ 5 % ~ 8 %

5 Scaling Problem Resources available in FPGAs V7 UltraScale 440 MPC-X (4x Stratix-V) V7-V2000 Arria10 Stratix-V V6-LX760 V6-LX240 (C-RORC) V4-LX40 (H-RORC) ClusterFinder Instance Million FlipFlops Development took full PhD Data processing capabilities with VHDL/Verilog don't scale with device capabilities lack of PhDs and time ;) definitely a lack of code maintainability!...you don't write analysis code in Assembler, don't you?

6 Higher-Level Data Processing There are frameworks available that allow a description of hardware on a higher level We are evaluating dataflow computing: describe dataflow in a Java-like syntax compiler creates a dataflow graph and a big pipeline generating a >1000 stage pipeline is no problem for a compiler hand-writing the same thing correctly is nearly impossible compiler runs coregen/megawiz, creates VHDL and runs regular vendor tools compiler handles flow control & latency adjustments data-valid, enable, full, empty, write-to/read-from FIFO etc... comes with a full environment of device driver, PCIe/IB API and hardware building blocks can integrate custom HDL

7 Dataflow Example Example Dataflow Description: last step in Clusterfinder, floating point division same in VHDL: obvious what to do! how much code will it be? how long will it take you to write and verify it?...now change fixed-point precision of input vectors...

8 Clusterfinder in Dataflow Description Reference VHDL Implementation: RCU1 FCF Resource Usage: VHDL Dataflow Registers ~ ~ LUTs ~ ~ RAMs DSPs 0 8 Lines of Code ~10k ~2.3k Comparable resource usage significantly smaller code base much easier to maintain build for different FPGAs without changing the code

9 Summary Avoid custom board development if possible Outdated before produced Don't underestimate hardware verification/support Use commercial hardware, protocols & interfaces get rid of custom interfaces where possible Keep an eye on higher-level hardware description OpenSPL, Xilinx Vivado HLS, Altera OpenCL,... Vendors provide supported hardware, evaluation boards,... supported hardware likely has PCIe but unlikely *TCA