QLogic SRP Module on Linux for OpenFabrics and InfiniPath Version 1.3.0.0.15 This software license applies only to QLogic customers. QLogic Corporation. All rights reserved. 1. Introduction 2. OS Support 3. Important Files 3. Package Installation 4. OFED Package Requirements 5. Sample Configuration File 6. Contacting Support Table of Contents 1. Introduction The the QLogic SRP module on Linux for OpenFabrics and InfiniPath, version 1.3.0.0.15 provides the following capabilities: Specify individual adapters to which you wish to connect. Specify multiple connections(session) per adapter. Configure adapter to use multiple connections as either a failover or roundrobin (load balancing) mechanism. Hot swap Fibre Channel bridge modules. 2. OS Support Supported OS Distributions: Red Hat Enterprise Linux 4 Update 4/5/6 Red Hat Enterprise Linux 5/5.1 SuSE Linux Enterprise 10/SuSE Linux Enterprise 10 SP1 Supported Open Fabrics Enterprise Distributions, version 1.3 Supported InfiniPath Releases, version 2.2 3. Important Files Configuration file: /etc/sysconfig/qlgc_srp.cfg Tools: qlgc_srptools is a package of tools for discovering and connecting to SRP targets on InfiniBand fabrics. qlgc_srptools contains following executables and scripts. ib_qlgc_srp_query ib_qlgc_srp_cfg ib_qlgc_srp_update (background daemon for FibreChannel bridge module hotswap) ib_qlgc_srp_build_cfg ib_qlgc_srp_stats ib_qlgc_srp_control To run ib_qlgc_srp_query, you must load the ib_umad module, as well as an appropriate low-level driver for the installed IB hardware ( for example, ib_mthca for a Mellanox HCA, ib_ipath for a QLogic HCA, or mlx4 for ConnectX.) Using ib_qlgc_srp_query: ib_qlgc_srp_query [-v #] QLogic SRP Module on Linux for OpenFabrics and InfiniPath Readme File 1 of 6
With no command line parameters, ib_qlgc_srp_query displays information about SRP targets in humanreadable form: # ib_qlgc_srp_query QLogic Corporation. Virtual HBA (SRP) SCSI Query Application, version 0.0.0.0.0 1 IB Host Channel Adapter present in system. HCA Card 1 : 0x0002c90300001368 Port 1 GUID : 0x0002c90300001369 Port 2 GUID : 0x0002c9030000136a SRP Targets : SRP IOC Profile : FVIC in Chassis 0x00066a00db00001e, Slot 4, Ioc 1 SRP IOC GUID : 0x00066a01dd000058 SRP IU SIZE : 320 SRP IU SG SIZE: 15 SRP IO CLASS: 0xff00 service 0 : name SRP.T10:0000000000000001 id 0x0000494353535250 service 1 : name SRP.T10:0000000000000002 id 0x0000494353535250 service 2 : name SRP.T10:0000000000000003 id 0x0000494353535250 service 3 : name SRP.T10:0000000000000004 id 0x0000494353535250 HCA 0 Port 1 0x0002c90300001369 -> Target Port GID 0xfe8000000000000000066a11dd000058 SRP IOC Profile : FVIC in Chassis 0x00066a00db00001e, Slot 4, Ioc 2 SRP IOC GUID : 0x00066a02dd000058 SRP IU SIZE : 320 SRP IU SG SIZE: 15 SRP IO CLASS: 0xff00 service 0 : name SRP.T10:0000000000000001 id 0x0000494353535250 service 1 : name SRP.T10:0000000000000002 id 0x0000494353535250 service 2 : name SRP.T10:0000000000000003 id 0x0000494353535250 service 3 : name SRP.T10:0000000000000004 id 0x0000494353535250 HCA 0 Port 1 0x0002c90300001369 -> Target Port GID 0xfe8000000000000000066a21dd000058 SRP IOC Profile : Mellanox OFED SRP target SRP IOC GUID : 0x00066a0098007069 SRP IU SIZE : 996 SRP IU SG SIZE: 58 SRP IO CLASS: 0x0100 service 0 : name SRP.T10:00066a0098007069 id 0x00066a0098007069 HCA 0 Port 1 0x0002c90300001369 -> Target Port GID 0xfe8000000000000000066a00a0007069 SRP IOC Profile : Mellanox OFED SRP target SRP IOC GUID : 0x00066a0098007069 SRP IU SIZE : 996 SRP IU SG SIZE: 58 SRP IO CLASS: 0x0100 service 0 : name SRP.T10:00066a0098007069 id 0x00066a0098007069 HCA 0 Port 1 0x0002c90300001369 -> Target Port GID 0xfe8000000000000000066a01a0007069 NOTE: The output has the following minor differences as compared to the InfiniServ SRP /proc/driver/ ics_srp/driver - ib_qlgc_srp_query now reports the SRP IO CLASS. - Under Target Path(s) ib_qlgc_srp_query now reports Target Port GID instead of Target Port GUID. Using ib_qlgc_srp_cfg: ib_qlgc_srp_cfg [-v #] [-f <srp_cfg_file_name.cfg>] Used by ib_qlgc_srp_build_cfg Using ib_qlgc_srp_build_cfg: ib_qlgc_srp_build_cfg [option flags] ib_qlgc_srp_build_cfg creates a configuration file based on discovered target devices. By default, the information is send to stdout. In order to create a configuration file, output should be redirected to a disk file. Type "ib_qlgc_srp_build_cfg -h" for a list and description of the option flags. QLogic SRP Module on Linux for OpenFabrics and InfiniPath Readme File 2 of 6
The default configuration generated by ib_qlgc_srp_build_cfg for OFED is similar to the one generated for the QuickSilver Stack with the following differences: - For OFED, the configuration automatically includes IOClass. - For OFED, the configuration automatically includes SID. - For OFED, the configuration provides targetportgid instead of targetportguid. - For OFED, the configuration automatically includes targetiocprofileidstring. Displaying configured SRP targets and statistics: ib_qlgc_srp_stats [/sys/class/qlogic_infiniband_srp/<name of adapter>] The program by default will display all know configured adapters along with each configured session. By specifying an adapter name, the display will be restricted to only that individual adapter. Displaying statistics and controlling SRP targets: ib_qlgc_srp_control [-l [-i]] [-a <Adapter Name> [-i] [-x] ] [-a <Adapter Name> -s <Session Name> <[-i] [-x] [-u] [-d] [-r]>] [-h] -l - list all QLogic SRP Virtual Adapters and Sessions -i - include stats output for Adapters and Sessions -a - Specify SRP Virtual Adapter Name -s - Specify a SRP Virtual Adapter's Session Name -x - Remove SRP Virtual Adapter or Session -u - Start a SRP Virtual Adapter's Session -d - Stop a SRP Virtual Adapter's Session -r - Restart a SRP Virtual Adapter's Session -h - display this help This program by default is interactive and initially displays all known configured adapters. You can enter commands at this point to drill down and control or query an adapter or one of it's configured sessions. You can also use the switches above to control or query an individual adapter or one of it's sessions noninteractively from the command line. 4. Package Installation NOTE: When installing for InfiniPath, you must export the following environment variable: export IPATH_DISTRO=<as entered to install InfiniPath> As an example, for Red Hat Enterprise Linux 4 UD 4: export IPATH_DISTRO=2.6.9_U4 Installing the SRP driver, tools, and scripts To install the driver, tools, and scripts, you must be the root user. As the root user in the QLGC_SRP directory, enter the following command:./install.sh Install will ask you if you wish to start the driver at boot. The default is no. If you supply the -a option./ install.sh -a), the driver will be set to automatically load at system boot and you will not be prompted about load at system boot. Starting the driver manually when not loaded during system boot To start the driver, enter the following command: /etc/init.d/qlgc_srp start or service qlgc_srp start To restart the driver, enter the following command: /etc/init.d/qlgc_srp restart or service qlgc_srp restart In order to stop the driver, enter the following command: /etc/init.d/qlgc_srp stop or service qlgc_srp stop To uninstall the driver, tools and scripts you must be the root user. As the root user in the QLGC_SRP directory, enter the following command:./uninstall.sh QLogic SRP Module on Linux for OpenFabrics and InfiniPath Readme File 3 of 6
5. OFED Package Requirements Choose the appropriate package for the installed HCA ipath HCAs: libipathverbs libipathverbs-devel Mellanox HCAs: libmthca libmthca-devel Mellanox ConnectX HCAs: libmlx4 libmlx4-devel Other required packages: ib_verbs kernel-ib kernel-ib-devel libibcm libibcm-devel libibcommon libibcommon-devel libibmad libibmad-devel libibumad libibumad-devel libibverbs libibverbs-devel libibverbs-utils libopensm libopensm-devel libosmcomp libosmcomp-devel libosmvendor libosmvendor-devel librdmacm librdmacm-devel librdmacm-utils mstflint openib-diags perftest tvflash ofed-docs ofed-scripts ibutils NOTE: Hot swapping a Fibre Channel bridge card requires exchanging I/Os in order for the QLogic SRP package to detect that the bridge card has been swapped. (This is identical to the QuickSilver SRP driver's functionality.) 6. Sample Configuration File # qlgc_srp.cfg file generated by /usr/sbin/ib_qlgc_srp_build_cfg, version 0.0.0.0.0, on Tue Apr 15 10:49:09 EDT 2008 registeradaptersinorder: ON # ============================================================================ # IOC Name: FVIC in Chassis 0x00066a00db00001e, Slot 4, Ioc 1 # IOC GUID: 0x00066a01dd000058 SRP IU SIZE : 320 # service 0 : name SRP.T10:0000000000000001 id 0x0000494353535250 session card: 0 port: 1 #portguid: 0x0002c90300001369 initiatorextension: 1 targetiocguid: 0x00066a01dd000058 targetiocprofileidstring: "FVIC in Chassis 0x00066a00db00001e, Slot 4, Ioc 1" targetportgid: 0xfe8000000000000000066a11dd000058 targetextension: 0x0000000000000001 SID: 0x0000494353535250 IOClass: 0xff00 IUSize: 320 IUSGTableSize: 15 QLogic SRP Module on Linux for OpenFabrics and InfiniPath Readme File 4 of 6
end adapter adapteriodepth: 1000 luniodepth: 16 adaptermaxio: 128 adaptermaxluns: 512 adapternoconnecttimeout: 60 adapterdevicerequesttimeout: 2 # set to 1 if you do not want target connectivity verification noverify: 0 description: "SRP Virtual HBA 1" end # ============================================================================ # IOC Name: Mellanox OFED SRP target # IOC GUID: 0x00066a0098007069 SRP IU SIZE : 996 # service 0 : name SRP.T10:00066a0098007069 id 0x00066a0098007069 session card: 0 port: 1 #portguid: 0x0002c90300001369 initiatorextension: 1 targetiocguid: 0x00066a0098007069 targetiocprofileidstring: "Mellanox OFED SRP target" targetportgid: 0xfe8000000000000000066a00a0007069 targetextension: 0x00066a0098007069 SID: 0x00066a0098007069 IOClass: 0x0100 IUSize: 996 IUSGTableSize: 58 end adapter adapteriodepth: 1000 luniodepth: 16 adaptermaxio: 128 adaptermaxluns: 512 adapternoconnecttimeout: 60 adapterdevicerequesttimeout: 2 # set to 1 if you do not want target connectivity verification noverify: 0 description: "SRP Virtual HBA 2" end Sample ib_qlgc_srp_stats output: SCSI Host # : 54 Mode : FAILOVER Trgt Adapter Depth : 768 Verify Target : Yes Rqst Adapter Depth : 1000 Rqst LUN Depth : 16 Tot Adapter Depth : 768 Tot LUN Depth : 16 Act Adapter Depth : 766 Act LUN Depth : 16 Max LUNs Scanned : 512 Max IO : 131072 (128 KB) Max Sectors : 256 Max SG Depth : 32 Session Count : 1 No Connect T/O : 60 Second(s) Register In Order : ON Dev Reqst T/O : 2 Second(s) Description : SRP Virtual HBA 1 Session : Session 1 State : Active Source GID : 0xfe800000000000000002c90300001369 Destination GID : 0xfe8000000000000000066a11dd000058 SRP IOC Profile : FVIC in Chassis 0x00066a00db00001e, Slot 4, Ioc 1 SRP Target IOClass : 0xFF00 SRP Target SID : 0x0000494353535250 SRP IPI Guid : 0x0002c90300001369 SRP IPI Extnsn : 0x0000000000000001 SRP TPI Guid : 0x00066a01dd000058 SRP TPI Extnsn : 0x0000000000000001 Source LID : 0x001a Dest LID : 0x0009 Completed Sends : 0x0000000000000484 Send Errors : 0x0000000000000000 Completed Receives : 0x0000000000000484 Receive Errors : 0x0000000000000000 Connect Attempts : 0x0000000000000000 Test Attempts : 0x0000000000000000 Total SWUs : 0x0000000000000300 Available SWUs : 0x0000000000000300 Busy SWUs : 0x0000000000000000 SRP Req Limit : 0x0000000000000300 SRP Max ITIU : 0x0000000000000140 SRP Max TIIU : 0x0000000000000140 QLogic SRP Module on Linux for OpenFabrics and InfiniPath Readme File 5 of 6
SRP Max Send IU : 0x0000000000000140 SRP Max Cmd : 0x0000000000000244 Req IU SG Depth : 0x000000000000000f Act IU SG Depth : 0x000000000000000f Max SG Used : 0x000000000000000f Host Busys : 0x0000000000000000 SCSI Host # : 55 Mode : FAILOVER Trgt Adapter Depth : 128 Verify Target : Yes Rqst Adapter Depth : 1000 Rqst LUN Depth : 16 Tot Adapter Depth : 128 Tot LUN Depth : 16 Act Adapter Depth : 126 Act LUN Depth : 16 Max LUNs Scanned : 512 Max IO : 131072 (128 KB) Max Sectors : 256 Max SG Depth : 32 Session Count : 1 No Connect T/O : 60 Second(s) Register In Order : ON Dev Reqst T/O : 2 Second(s) Description : SRP Virtual HBA 2 Session : Session 2 State : Active Source GID : 0xfe800000000000000002c90300001369 Destination GID : 0xfe8000000000000000066a00a0007069 SRP IOC Profile : Mellanox OFED SRP target SRP Target IOClass : 0x0100 SRP Target SID : 0x00066a0098007069 SRP IPI Guid : 0x0002c90300001369 SRP IPI Extnsn : 0x0000000000000001 SRP TPI Guid : 0x00066a0098007069 SRP TPI Extnsn : 0x00066a0098007069 Source LID : 0x001a Dest LID : 0x000d Completed Sends : 0x000000000000f6b2 Send Errors : 0x0000000000000000 Completed Receives : 0x000000000000f6b2 Receive Errors : 0x0000000000000000 Connect Attempts : 0x0000000000000000 Test Attempts : 0x0000000000000000 Total SWUs : 0x0000000000000080 Available SWUs : 0x0000000000000080 Busy SWUs : 0x0000000000000000 SRP Req Limit : 0x000000000000007f SRP Max ITIU : 0x00000000000003e4 SRP Max TIIU : 0x00000000000003e4 SRP Max Send IU : 0x0000000000000244 SRP Max Cmd : 0x0000000000000244 Req IU SG Depth : 0x000000000000003a Act IU SG Depth : 0x0000000000000020 Max SG Used : 0x0000000000000020 Host Busys : 0x0000000000000000 7. Contacting Support Please feel free to contact your QLogic approved reseller or QLogic Technical Support at any phase of integration for assistance. QLogic Technical Support can be reached by the following methods: Web: http://support.qlogic.com North America Contact Information Email: support@qlogic.com Phone: (952) 932-4040 Support contact information for other regions of the world is available at the QLogic website: http://support.qlogic.com Go to Top Copyright 2008. All rights reserved worldwide. QLogic, the QLogic logo, and the Powered by QLogic logo are registered trademarks of QLogic Corporation. All other brand and product names are trademarks or registered trademarks of their respective owners. QLogic SRP Module on Linux for OpenFabrics and InfiniPath Readme File 6 of 6