System Administration for the Intel Xeon Phi Coprocessor

Transcription

1 White Paper System Administration for the Intel Xeon Phi Coprocessor MPSS 3.4 Linux Host

2 Preface The intent of this document is not as a replacement for the documentation which comes with each release of the Intel Manycore Software Stack (Intel MPSS) but as a supplement, providing advice, troubleshooting suggestions and pointers to other useful documents. It is written with the small scale system administrator in mind but also contains information of use to the administrator of clusters. 2

3 Table of Contents 1 Overview Introduction Requirements for a system administrator Linux Init Scripts Useful documentation Installing System Software for the Intel Xeon Phi Coprocessor System requirements Download Intel MPSS Base Intel MPSS Installation Installed Files States of the Coprocessor Steps to uninstall Intel MPSS for the Intel Xeon Phi Coprocessor Recompiling modules from the Intel MPSS release Intel MPSS Configuration Network Configuration Static Pair (Default) topology Internal Bridge Topology External Bridge Topology Resetting network settings User Configuration Adding users to the Intel Xeon Phi coprocessor file system Removing users from the Intel Xeon Phi coprocessor file system Updating a Users SSH Keys on the Intel Xeon Phi Coprocessor File System Adding Groups to the Intel Xeon Phi Coprocessor File System

4 3.3 SSH Access and Configuration for the Intel Xeon Phi Coprocessor Power Management and Timing Measurements Power Management Configuration Peer to Peer Swap space System Clock Verify Installation of Intel MPSS and Monitor Intel Xeon Phi Coprocessors Miccheck Micinfo Micsmc Additional Configuration LDAP Configuration on the Intel Xeon Phi Coprocessor File System NIS Configuration on the Intel Xeon Phi Coprocessor File System Micrasd and Reliability Monitor Support Micrasd Reliability Monitor Installing Intel MPSS with OFED Support Installing Intel MPSS with GANGLIA* support Configuring NFS mount on Intel Xeon Phi Coprocessors Adding an NFS mount Removing an NFS mount NFS mounting a host export Configuring Lustre on Intel Xeon Phi Coprocessors Troubleshooting Common Issues

5 6.1.1 Hardware or Intel MPSS installation issues User configuration issues NFS mounting issues for Intel Xeon Phi coprocessors Miscellaneous issues For the Inexperienced Linux System Administrator Glossary Appendix A: Supported versions Acknowledgements About the Authors

6 1 Overview 1.1 Introduction This paper provides a look at the Intel Xeon Phi coprocessor from the system administrator s point of view. A server powered by one or more Intel Xeon processors serves as the host for the coprocessor. An individual host may have one or more coprocessors. The MPSS numbers the coprocessors from 0 to (n-1) where n is the number of coprocessors in the system. When the installation uses peer to peer (coprocessor to coprocessor) communication, the number of coprocessors should be limited to 8. At present, typical systems contain one or two coprocessors per host. The host must run either a Linux* or a Windows* operating system. The coprocessor runs a Linux microkernel regardless of what operating system is being used on the host. This document covers administering an Intel Xeon Phi coprocessor on a Linux host. The Intel MPSS download page contains a Windows specific guide succinctly entitled, Intel Xeon Phi Coprocessor Intel Manycore Platform Software Stack (Intel MPSS) User's Guide (Windows*). Figure 1 Default installation with Linux host 6

7 The coprocessor contains flash and SMC (System Management and Configuration) memory which hold the BIOS, bootloader and non-linux pre-kernel for the coprocessor. However this memory does not hold the complete Linux kernel for the coprocessor. Upon initialization, the bootloader loads and launches a small pre-kernel from the coprocessor s flash; the host s mpssd driver instructs the pre-kernel to load the coprocessor s Linux kernel from the host and then execute it. The coprocessor does not have direct access to any permanent file systems. All file systems must be either a RAM file system residing in coprocessor memory, or a network file system residing on disks attached to the host or elsewhere on the network. For a RAM file system, Intel MPSS stores all files on the host until the coprocessor is booted. When the coprocessor is booted, host s mpssd directs the pre-kernel to create a temporary file system in the coprocessor s memory and populate it with the files copied from the host. The coprocessor s initial Linux boot uses a very small temporary root file system stored in RAM. This file system contains the executables and configuration files which the kernel uses to bring up certain basic services such as networking. To bring the coprocessor to a full run state, this initial Linux boot will either create a second RAM file system containing the full root file system, or NFS mount the full root file system. The default is a RAM file system. The Linux operating system then automatically switches over to the new root file system and removes the small temporary RAM file system. For a more complete description of the boot process, see the Intel Manycore Platform Software Stack (Intel MPSS) User s Guide available at Because the default location for the coprocessor s root file system is in RAM and because a RAM file system takes up space which might otherwise be used by running processes, MPSS keeps the root file system included with the Intel Manycore Platform Software Stack (Intel MPSS) as small as possible. This is achieved by limiting the commands and libraries included by default. The majority of Linux commands are supplied by TinyLogin and BusyBox ( two programs containing many of the standard Linux commands though some in simplified implementations. On the host, the device driver providing the basic PCIe interface to the coprocessor is a kernel module named mic.ko. The host also runs the mpssd daemon which handles communication to the coprocessor during boot. The MPSS system administration tools for the coprocessor also reside on the host and execute from there. On the coprocessor side, the coi_daemon (Coprocessor Offload Infrastructure daemon) runs along with the standard Linux daemons. The coi_daemon handles process and space management for programs written using the offload programming model. (See Using the Offload Compiler in the Intel Xeon Phi Coprocessor Developer s Quick Start Guide) 7

8 1.2 Requirements for a system administrator In order to administer the coprocessor, the administrator must have super-user privileges on the host system, either as the root user or as a user with sudo privileges. All commands in this paper are written as if run by the root user. If they are run by a non-root user, sudo must prefix the commands. Sudo is the preferred method for safety and traceability reasons. In either case, root s home directory on the host must have an.ssh directory with valid RSA/DSA keys. 1.3 Linux Init Scripts Red Hat* Enterprise Linux* 6 and SUSE* Linux* Enterprise Server 11 use the System V init system, while Red Hat* Enterprise Linux* 7 uses the systemd init system. The System V init system uses the service command, which has the form: $ service SCRIPT COMMAND [OPTIONS] where the SCRIPT parameter specifies a System V init script, and the supported values of COMMAND depend on the invoked script. Systemd uses the systemctl command, which has the form: $ systemctl [OPTIONS ] COMMAND [NAME ] where [NAME ] is zero or more parameters to the COMMAND. The systemctl command is also used on RHEL* 7 instead of the chkconfig command. Init commands in this document are in System V format. On host systems with RHEL* 7, those commands should be changed to system format as follows: RHEL* 6/SUSE 11 Command service mpss unload service SCRIPT COMMAND chkconfig NAME on chkconfig NAME off RHEL* 7 Command systemctl stop mpss modprobe -r mic systemctl COMMAND SCRIPT systemctl enable NAME systemctl disable NAME For example, the command: $ service mpss restart must be changed to: $ systemctl restart mpss 8

9 Similary, on RHEL 7.* $ chkconfig mpss on must be changed to: $ systemctl enable mpss 1.4 Useful documentation The core documents relating to the adminstration of a system with Intel Xeon Phi Coprocessors are: readme.txt included with each Intel MPSS release (available at Intel Manycore Platform Software Stack (Intel MPSS) User s Guide, provided with most Intel MPSS releases (available at Intel Xeon Phi Coprocessor Developer s Quick Start Guide ( Other useful documents focusing upon other aspects of coprocessor support and use are: Intel Xeon Phi Coprocessor System Software Developers Guide ( Configuring Intel Xeon Phi coprocessors inside a cluster ( (note: the concepts in this paper are still relevant, but are written for an older generation Intel MPSS release). Using the Intel MPI Library on Intel Xeon Phi Coprocessor Systems ( Debugging on Intel Xeon Phi Coprocessor Use Case Overview ( Intel and Third Party Tools and Libraries available with support for Intel Xeon Phi Coprocessor ( 9

10 Intel Xeon Phi Coprocessor Instruction Set Reference Manual (available at under Guides & Manuals.) MICRAS Log User Guide ( 10

11 2 Installing System Software for the Intel Xeon Phi Coprocessor The system software for the coprocessor, as well as the host software which supports the coprocessor, is collectively known as the Intel MPSS, the Intel Manycore Platform Software Stack. When installing a new Intel MPSS release, always consult the readme files and other documentation that accompany the software. This documentation provide step by step instructions as well as the latest information on currently supported host operating systems and any special configuration requirements for that release. 2.1 System requirements Supported hardware platform with at least one Intel Xeon Phi coprocessor installed Super-user privileges are required to install the Intel MPSS It may be necessary to modify the BIOS on your host system. Check with the vendor of your host system to see if these changes apply to your system. o The BIOS must allow the use of memory mapped I/O address ranges above 4GB The Intel MPSS distribution does not provide the operating system for the host. As the administrator, you will need to obtain and install one of the supported Linux distributions 1. See the Intel Manycore Platform Software Stack (Intel MPSS) article on the Intel Developer Zone. This requirement may change in a future MPSS release. o o Host OS kernel updates beyond the "officially supported" initial release versions specified in the table above, may prevent the pre-built Intel MPSS driver modules from loading. Most Linux distributions provide minor kernel updates and patches. To ensure compatibility, disable kernel updates on your host OS If you choose to install an unsupported Linux distribution, you will need to recompile the host s Intel Xeon Phi coprocessor kernel module, as well as the kernel modules for any optional MPSS software you install. Section Recompiling Modules from the Intel MPSS Release explains steps to recompile Intel MPSS modules and the section Recompiling Modules and RPMs from the Intel MPSS Release in the Intel MPSS User s Guide explains steps to recompile RPMs for optional MPSS software. 1 See Appendix A: Supported versions for additional details. 11

12 o List of current supported Linux* host operating system (default configuration) Supported Host OS Versions Kernel Version Red Hat* Enterprise Linux* 64-bit Red Hat* Enterprise Linux* 64-bit Red Hat* Enterprise Linux* 64-bit Red Hat* Enterprise Linux* 64-bit SUSE* Linux* Enterprise Server 11 SP2 SUSE* Linux* Enterprise Server 11 SP default default When installing the host operating system, some software included with the operating system is optional. o It is recommended that you install a complete GNU build environment (gcc and related tools) on the host. It may be necessary to disable certain security features before installing the MPSS. Failing to do this can result in unexplained Connection refused messages when you attempt to log into the coprocessor. o o If you have SELinux enabled on the host, disable it and reboot the host before installing the Intel MPSS release. If you are running the iptables service (i.e. the Linux firewall) on the host, disable it (service iptables stop) before installing the Intel MPSS release. 12

13 2.2 Download Intel MPSS The Intel MPSS is available at Intel MPSS is delivered as several compressed tar files containing: mpss-<mpss-version>-linux.tar : It contains a complete set of runnable MPSS along with supporting utilities and documents for Intel MPSS. It includes Standard driver, Cross compiler, Out of the box µos. mpss-<mpss-version>-k1om.tar: It contains native tools for µos and other supported optional software for Intel Xeon Phi Coprocessor. mpss-src-<mpss-version>.tar: It contains sources for the complete µos and tools. It also contains sources for gcc, binutils, OFED, Ganglia, coprocessor native GDB, etc. 2.3 Base Intel MPSS Installation The general procedure to install Intel MPSS once you have the host operating system installed and booted is: To check if you have any previously installed version of Intel MPSS. It is required that you do not upgrade an existing version of MPSS. $ rpm -qa grep -e intel-mic -e mpss Note - Both yum and zypper support software upgrades and downgrades. However, it is necessary that Intel MPSS upgrades and downgrades be carried out by complete uninstallation of existing software followed by a clean installation of the replacement software. Please refer section 2.6 Steps to uninstall Intel MPSS for the Intel Xeon Phi coprocessors to first uninstall the Intel MPSS and then continue with Intel MPSS installation process. Untar the Intel MPSS package $ tar xvf mpss-<mpss-version>-linux.tar At this point a modules directory is created containing all kernel specific modules. Check if MPSS modules specific to your kernel are already available $ ls mpss-<mpss-version>/modules/*`uname r`*.rpm Note - If MPSS modules specific to your kernel are not present then refer section 2.7 Recompiling Modules for Intel MPSS release to re-build the MPSS modules for the updated kernel version prior to installing Intel MPSS. You will also need to refer Intel MPSS User Guide for your 13

14 release of Intel MPSS to recompile RPMs for any optional software you intend to use. For example, If you are using Infiniband* as interconnect, refer to the section Recompiling the Intel MPSS RPM specifically for OFED in the Intel MPSS User Guide for your release of Intel MPSS. Install the Intel MPSS package $ cd mpss-<mpss-version> $ cp./modules/*`uname r`*.rpm. $ yum install *.rpm #Red Hat* Enterprise Linux* $ zypper install *.rpm #SUSE* Linux* Enterprise Server Note - yum and zypper are the recommended tools for installing MPSS RPMs. - MPSS packages are not GPG signed. If local package GPG check (localpkg_gpgcheck) is enabled in yum.conf, then an additional --nogpgcheck option must be used as follows: $ yum install --nogpgcheck *.rpm Load the mic.ko driver, and then initialize MPSS Default Settings. Note - If using SUSE* Linux on the host machine, you must set allow_unsupported_modules to 1 in the file /etc/modprobe.d/unsupported-modules. $ modprobe mic $ micctrl --initdefaults Note - Above commands assumes the use of default MPSS configuration. Optionally you can change the MPSS configuration by using micctrl commands or by modifying.conf files. The Intel MPSS User Guide for your release of Intel MPSS explains in detail how to modify Intel MPSS configurations files. Flash & SMC consideration Note - These steps will not work if the flash files (ending in.rom.smc) are moved to a location other than the default install path - Current running version of Flash must be >= 375. If not, contact your Intel support representative - Please refer to the following document for Flash Issues & Remedies o Current version of flash can be checked as follows: 14

15 $ /usr/bin/micflash -getversion -d all o Check the status of the coprocessor(s): $ micctrl -s It is important that all of the coprocessor(s) are in ready state. If not, set the coprocessor(s) to ready state as follows: $ micctrl -rw o Update the flash using micflash utility Note - If using C0 stepping, or 5110P B1 SKUs with a TA of G or higher, or if the SMC boot loader version is 1.8 use: $ /usr/bin/micflash -update -device all Otherwise an additional option of smcbootloader should be added to the above command as follows: $ /usr/bin/micflash -update -device all smcbootloader o Reboot the host system for all flash and SMC changes to take effect $ reboot Start Intel MPSS by using the Linux* init script 1 : $ service mpss start o To configure the Intel MPSS service to start when the host OS boots 1 : $ chkconfig mpss on 15

16 2.4 Installed Files Intel MPSS files are all installed on the Host. Host side o Configuration files (/etc/mpss) o Driver (/lib/modules/ /mic.ko) o Tools (/usr/ ) o User space libraries like SCIF and COI (/usr/lib64/) Coprocessor o Kernel file for µos (/usr/share/mpss/ ) o Standard initrd for µos (/usr/share/mpss/ ) 2.5 States of the Coprocessor If the mpssd is running, you can use the command micctrl -s on the host to query the state of the coprocessors. resetting The coprocessor is undergoing reset. A firmware program is testing and initializing the hardware. This process must complete before the mpssd can boot the microkernel. ready The coprocessor has undergone reset. This is comparable to a PC waiting at the boot select screen. At this point, the coprocessor is ready for the mpssd to transfer and boot the Linux microkernel. booting The coprocessor is booting the Linux microkernel. A fresh reset can interrupt this process. online The coprocessor is ready for use. The Linux microkernel finished booting. Access to the coprocessor via ssh or minicom should be possible. 16

17 2.6 Steps to uninstall Intel MPSS for the Intel Xeon Phi Coprocessor The general procedure to uninstall Intel MPSS is stop, unload and uninstall all the drivers installed during Intel MPSS installation process. Note - The uninstall script will not uninstall rebuilt drivers. You must remove them manually before running the uninstall.sh script $ yum remove <name of rebuilt package rpm> #RHEL* $ zypper remove <name of rebuilt package rpm> #SUSE* Linux Note - If you have optional mic-related services installed then they must be stopped before unloading Intel MPSS. For Example, if you have Intel MPSS installed with OFED support then the services must be stopped in the following order before stopping Intel MPSS 1. $ service mpxyd stop $ service ofed-mic stop $ service opensmd stop $ service openibd stop Once all the optional mic-related services are stopped the Intel MPSS driver can be unloaded as follows: To uninstall 3.x-based builds: - Unload the MPSS driver using 1 : $ service mpss unload To uninstall 2.x-based builds: $ yum remove intel-mic\* #Red Hat* Enterprise Linux* $ zypper remove intel-mic\* #SUSE* Linux* Enterprise Server 2.7 Recompiling modules from the Intel MPSS release If the host system must run with an updated kernel, use the following steps to re-build the MPSS modules for the updated kernel version prior to installing MPSS. If using Infiniband as an interconnect, refer to section 9.1 of the MPSS User's guide, "Recompiling the Intel MPSS RPM specifically for OFED". Steps to recompile the MPSS host kernel modules: 17

18 Ensure the prerequisites are installed: $ yum install kernel-headers kernel-devel #Red Hat* Enterprise Linux $ zypper install kernel-default-devel rpm #SUSE* Linux Enterprise Server Regenerate the Intel MPSS driver module package: $ cd mpss-<mpss_version>/src/ $ rpmbuild --rebuild mpss-modules*.src.rpm The newly built mpss-modules and mpss-modules-dev rpms will be located at: ~/rpmbuild/rpms/x86_64/ /usr/src/packages/rpms/x86_64/ # Red Hat* Enterprise Linux # SUSE* Linux Enterprise Server Copy the re-built mpss-modules and mpss-modules-dev RPMs from the respective directory specified in above step into the /modules directory of the extracted MPSS tar file. Once the required RPMs are copied, proceed to section 2.3 Base Intel MPSS Installation to continue MPSS installation. 18

19 3 Intel MPSS Configuration 3.1 Network Configuration Network access to the Intel Xeon Phi coprocessors is a complicated process depending on the network topology to be supported. The supported coprocessor topologies have been categorized into static pair, internal bridge and external bridge topologies. Using a combination of the Bridge and Network configuration parameters allows a diverse and robust network setup. A virtual TCP/IP network connection between the host and the Intel Xeon Phi coprocessor is created over the PCIe bus Static Pair (Default) topology Note - Static Pair topology can only be used in single host installations and is unusable in a cluster. For cluster oriented network setup please refer to Section External Bridge Topology In the static pair topology, every Intel Xeon Phi coprocessor is assigned to a separate subnet known only to the host. The IP address for the Intel Xeon Phi coprocessor and host ends of the virtual network connection must be a fully qualified IP address and the first three quads of the addresses must match. The micctrl --initdefaults or micctrl --resetconfig commands assign a default value, , to the top two quads. The third quad indicates the card number. The Intel Xeon Phi coprocessor end of the virtual connection is assigned 1 for the last quad and the host end is assigned 254. An example of the network addresses that would be created is Subnet: Host side address of first coprocessor: IP address of first coprocessor: Host side address of second coprocessor: IP address of second coprocessor: The command syntax for setting up a network with static pair topology is as follows: micctrl --network=static [--ip=<ip>] [--mtu=<mtu>] [--modhost=<yes no>] \ [--modcard=<yes no>] [mic card list] The --network option set to static without specifying a bridge name provides an easy method for changing the Network configuration parameter to a static pair type. 19

20 Note - If no IP address is included, then the network interfaces for the listed cards is set to the default values provided by the original micctrl --initdefaults command. - If the IP address specified is the first two quads, then micctrl will finish the address by setting the third quad of each address to the card s ID + 1, and the fourth quad to 1 on the card and 254 on the host. - If the IP argument is set with the format cardip,hostip:cardip,hostip. Each cardip - hostip pair specifies a card's values. For example, if there are two cards in the system and the entry is , : , then mic0 will be assigned the card IP address of and host of Mic1 will be assigned and The default size of MTU is set to max IPV4 size of 64k. This value can be changed by specifying the MTU value. - The modhost parameter determines if the system administrator wants to modify the host /etc/hosts file or not. - The modcard determines if the system administrator wants the configuration process to create the /etc/hosts on the card Static Pair topology Example: $ micctrl --network=static --ip= modhost=yes --modcard=yes Figure 2-Static Pair network topology Figure 2 illustrates an example of static pair network configuration. On a two card system specifying an IP value of creates mic0 values of and and mic1 values of and Internal Bridge Topology The terminology internal bridge, in the context of Intel Xeon Phi coprocessor configuration, refers to the process of bridging together multiple Intel Xeon Phi coprocessor virtual network interfaces, on the same host. In this configuration, each Intel Xeon Phi coprocessor can communicate with the host and with the other Intel Xeon Phi coprocessors in the platform. 20

21 The internal bridging network topology connects the Intel Xeon Phi coprocessors in a system together with one IP address for the host. This is accomplished by first creating the bridge interface on the host and then connecting the virtual network interfaces to it. Such a network configuration could, for example, be used to support communication between the ranks of an MPI application that is distributed across the Intel Xeon Phi coprocessors and host. The command syntax for setting up a network with internal bridge topology is as follows: micctrl --addbridge=<brname> --type=internal --ip=<ip> [--netbits=<bits>] [--mtu=<mtu>] micctrl --network=static --bridge=<name> [--ip=<ip>] [--modhost=<yes no>] \ [--modcard=<yes no>] [mic card list] micctrl creates bridge interfaces with the --addbridge option. Note - The name of the bridge must be specified and generally on Linux* is br0 or br1. - Setting the type to internal will cause micctrl to always create the correct network configuration files for the host. - You may specify the netbits value and the mtu value but it has no real effect. Each of the virtual Ethernet interfaces added to the bridge will inherit these values from the bridge specification. Virtual network interfaces are added to the bridge with the micctrl --network command. Note - The bridge argument must be present - The IP argument must specify IP addresses with the first 3 quads matching those of the bridge IP address - The IP address must be a fully qualified dot notated address. If more than one card is specified, each card will get the same IP address with the cards number added to the fourth quad. For example, if the address is specified, the mic0 will receive and mic The modhost parameter determines if the system administrator wants to modify the host /etc/hosts file or not. - The modcard determines if the system administrator wants the configuration process to create the /etc/hosts on the card 21

22 Internal Bridge Example Figure 3-Internal Bridge network topology Figure 3 illustrates the internal bridged network topology. In this example the host and the cards can all communicate through the subnet. The host can communicate outside through the subnet but the cards cannot. The series of commands to create this topology would be: $ micctrl --addbridge=br0 --type=internal --ip= $ micctrl --network=static --bridge=br0 --ip= Note - Network service restart may be required after executing above commands in case one Intel Xeon Phi coprocessor cannot access the other coprocessors on the host External Bridge Topology The external bridge configuration bridges Intel Xeon Phi coprocessors to an external network. This is the typical configuration required when Intel Xeon Phi coprocessors are deployed in a cluster to enable remote access. The command syntax for setting up a network with external bridge topology is as follows: micctrl --addbridge=<brname> --type=external --ip=<ip> [--netbits=<bits>] [--mtu=<mtu>] micctrl --network=static --bridge=<name> [--ip=<ip>] [--modhost=<yes no>] \ [--modcard=<yes no>] [mic card list] The external bridge type definition differs from internal bridge only in that micctrl does not create the network configuration files for the host. It assumes that the bridge is already connected to a physical Ethernet device. It is a task of the system administrator to modify the configuration file for the physical Ethernet port to attach to the bridge. 22

23 External bridges may also be declared as dhcp instead of static. During boot, the Intel Xeon Phi coprocessor Linux* OS will attempt to retrieve an IP address from a DHCP server. micctrl --network=dhcp --bridge=<name> [--modhost=<yes no>] [--modcard=<yes no>] \ [mic card list] External Bridge Example Figure 4-External Bridge network topology Figure 4 depicts a cluster in which the Intel Xeon Phi coprocessors on each host node are bridged to the external network. This topology diagram shows how individual hosts and their Intel Xeon Phi coprocessors become a part of the larger subnet. The series of steps to implement this topology would be: On the host node0 the commands to configure the virtual network interfaces are: $ micctrl --addbridge=br0 --type=external --ip= $ micctrl --network=static --bridge=br0 --ip= And on the host node1 the commands are: $ micctrl --addbridge=br0 --type=external --ip= $ micctrl --network=static --bridge=br0 --ip=

24 Normally, the system administrator will have configured the bridge br0 and tied the eth0 interface to it before these configuration calls. If they did not, then micctrl will create the /etc/sysconfig/network-scripts/ifcfg-br0. It will not, however, create the configuration file for the eth0 physical network interface; the system administrator will need to do this step. An example of modified /etc/sysconfig/network-scripts/ifcfg-eth0 file is: DEVICE=eth0 NM_CONTROLLED=no TYPE=Ethernet ONBOOT=yes BRIDGE=br0 When this is completed, perform a network restart as follows 1 : $ service network restart In the process of configuring an external bridge, micctrl assumes that the network packet MTU size must be set to the Ethernet standard 1500 bytes. This value significantly slows down data transfer across the virtual Ethernet interfaces. If the physical network hardware allows, it would be better to increase the MTU size to the biggest possible value. Typically most hardware will support at least 9000 byte mtu sizes. This could have been set by the original micctrl --addbridge command with the addition of the --mtu=9000 argument. If this was not done and the system administrator wishes to increase the value, this may be done with the command: $ micctrl --modbridge=bridge_id --mtu= Resetting network settings The --network option set to default restores the network setting for the list of coprocessors to the value originally created by the micctrl --initdefaults command. $ micctrl --network=default [mic card list] Also if the bridge is no longer needed, the delbrigde option can be used to remove it from the Intel Xeon Phi coprocessor configuration. If the bridge is not external, it will also remove the corresponding host network configuration file. $ micctrl --delbridge=bridge_id 24

25 3.2 User Configuration Adding users to the Intel Xeon Phi coprocessor file system The --useradd option adds the specified user name to the /etc/passwd and /etc/shadow files on the Intel Xeon Phi coprocessor file system. The system administrator must specify the correct user and group IDs for the user that is being added. micctrl --useradd=<name> --uid=<uid> --gid=<gid> [--home=<dir>] [--comment=<string>] \ [--app=<exec>] [--sshkeys=<keydir>] [--nocreate] [--non-unique] [mic card list] Note - The behavior of micctrl --useradd differs from Linux* useradd. It does not check for conflicting uid or gid for other users on the ldap server. System administrators should not expect identical behavior between micctrl --useradd and Linux* useradd - The --home argument specifies the user s home directory in the card file system, and will cause the directory to be created. The default home directory for user <name> is /home/<name> - The --comment argument specifies a comment string to be added to the comment field in /etc/passwd. The default comment string is the user names - The --app argument specifies the default application executed by the user. The default app is /bin/sh - The --sshkeys argument specifies the host directory in which the user s secure shell key files are to be found. The default is /home/<name>/.ssh - The --nocreate option specifies to not create the home directory for the user and is intended to be used when the users home directories are remote mounted - The --non-unique option will allow the user to be added to the card s /etc/passwd and /etc/shadow files with the specified uid even if a user with that uid already exists - In addition, a default.profile file will be added for the user - The latest implementation will also add the user to the currently running coprocessor if it is in the online state Removing users from the Intel Xeon Phi coprocessor file system The --userdel option removes the specified user from the card s /etc/passwd and /etc/shadow files. It also removes the directory stored in the home field of the /etc/passwd file. micctrl --userdel=<name> [--remove] [mic card list] Note - The current implementation also removes the user from the /etc/passd and /etc/shadow files on the currently running coprocessor if it is in the online state. By default, it does not remove the user s home directory and is intended to prevent the removal of a user s remote mounted home directory. System administrators can force the removal of the user s home directory by including the --remove option 25

26 3.2.3 Updating a Users SSH Keys on the Intel Xeon Phi Coprocessor File System The --sshkeys option allows a user's SSH keys to be updated on the coprocessor file system. micctrl --sshkeys=<name> [--dir=<dir>] [mic card list] The directory is the location of all the keys required. This command copies the key files with correct owner and permissions to the cards file system. It also create entries in the authorized_keys file for any file(s) with a.pub extension if it does not already exist Adding Groups to the Intel Xeon Phi Coprocessor File System The --groupadd option adds the specified group name and ID to the card s /etc/group file. micctrl --groupadd=<name> --gid=<gid> [mic card list] Note - In the current implementation the group will also be added to the running coprocessor s /etc/group file if it is in the online state 3.3 SSH Access and Configuration for the Intel Xeon Phi Coprocessor Note - On both RHEL* and SLSE*, issues were encountered in configuring the virtual network interfaces for the Intel Xeon Phi coprocessors when "NetworkManager" is used. It is strongly recommended to use the "network daemon" instead. - To switch to the network daemon 1 : $ chkconfig NetworkManager off $ chkconfig network on $ service NetworkManager stop $ service network start Communication with the coprocessor is provided by a standard network interface. The interface uses a virtual network driver over the PCIe bus. Standard networking tools such as SSH are supported. The coprocessor Linux* OS supports network access using SSH keys or password authentication, provided that valid credentials exist on the coprocessor. The initial configuration phase of the Intel MPSS creates users for each coprocessor based on the current user IDs in the host /var/mpss/mic0/etc/passwd file. For each user in 26

27 /var/mpss/mic0/etc/passwd (including root), if SSH key files are found in the user's ".ssh" directory, those keys are also populated to the Intel Xeon Phi coprocessor's file system. The default user credentialing behavior on the coprocessor can be customized with the "micctrl --userupdate" option. Administrators who prefer to only provide a limited set of user credentials (root, sshd, micuser, nobody and nfsnobody) can run the following commands after the initial configuration to prevent all users from being propagated: - RHEL 6.x and SUSE $ service mpss stop $ micctrl --userupdate=none $ service mpss start - RHEL 7.x $ systemctl disable mpss $ micctrl --userupdate=none $ systemctl enable mpss In the case where a user did not have a valid SSH key present during the initial coprocessor configuration process, they can be added as necessary on a per-user basis. To generate the SSH key, each user must execute $ ssh-keygen Then, the "micctrl --sshkeys" option is used to allow Intel MPSS to pick up the new keys for a specific user. For example, the following updates the coprocessor configuration for mic0 to include root's SSH keys: - RHEL 6.x and SUSE $ service mpss stop $ micctrl --sshkeys=root mic0 $ service mpss start - RHEL 7.x $ systemctl disable mpss $ micctrl --sshkeys=root mic0 $ systemctl enable mpss Note - In case if the user did not exist during the initial coprocessor configuration process, they can be added as necessary on a per-user basis. For example, to add the user `foo` to /var/mpss/mic0/etc/passwd execute: $ micctrl --adduser=foo 27

28 3.4 Power Management and Timing Measurements Power Management Configuration Package C3 (PC3), and Package C6 (PC6) are power management states that are entered when the coprocessor is idle for a period of time. Power management states are enabled by default in Intel MPSS. Under normal circumstances, the host sends traffic to awaken the coprocessor and resume execution. The different power states are described in the Intel Xeon Phi Coprocessor System Software Developers Guide. When a program running on the coprocessor executes a sleep() function call, the coprocessor will appear idle to the power management software. This can result in the coprocessor entering a Deep Package State C3. When this happens, the program will never receive the timer signal and the coprocessor will hang. If this problem occurs, disable power management and restart the coprocessor. Command syntax to configure power management can be stated as follows: micctrl --pm micctrl --pm=<default off set> [--corec6=<on off>] [--pc3=<on off>] \ [--pc6=<on off>] [--cpufreq=<on off>] [mic card list] Note - Setting the pm argument to default resets the values of power management for the list of cards to the default values. - In the current release, all power management parameters other than cpufreq are set to off. This will change in a future release - Setting the pm argument to off will also set all parameters to off other than cpufreq - Setting the pm argument to set enables the use of the optional parameters corec6, pc3, pc6, and cpufreq. Each optional parameter can be individually enabled or disabled by setting the on or off values Peer to Peer In order to transfer data from one coprocessor directly to memory on a second coprocessor on the same host, peer to peer support must be enabled in the /etc/mocprobe.d/mic.conf file. P2P is enabled by default for Intel MPSS. To disable peer to peer, add p2p=0 to the end of the options line for the Intel Xeon Phi coprocessor kernel module and restart 1 the module using $ service mpss unload $ service mpss start 28

29 When a large number (>8) of coprocessors are installed in a host, the time required for a coprocessor to enable peer to peer to all other coprocessors can result in a timeout error. However, disabling peer to peer will break any code using SCIF (Symmetric Communications Interface) routines which rely on that type of access. This tradeoff should be kept in mind when deciding how many coprocessors you want to install on a given host Swap space Because there are no directly attached disks on the coprocessor, there is no natural location to create swap space for copying memory pages out. It is possible to set up a form of networked swap space using a virtio block device on the host. Directions for doing this can be found in the readme file that comes with the MPSS. However, there is a major performance penalty involved in moving memory pages between the host and coprocessor, particularly if the memory begins to thrash. Because of this, swapping should be enabled only when truly necessary System Clock The initial time on the coprocessor is set on boot from the host system. After that, time can be measured either by referring to the Elapsed Time Counter (ETC) or the Time Stamp Counter (TSC). The TSC is used as the default clock. There is one ETC on the coprocessor. Access time to the ETC is affected by ring bus latencies and contention between cores. A call to a routine such as gettimeofday() can be 100 times slower when ETC is used than when TSC is used. Each core on the coprocessor has its own TSC. In the past, power management sleep states affected the reliability of the TSC. Now, the power management software compensates for this difference. To change the clock source without rebooting, log onto the coprocessor as root, find the names of the available clock sources $ cat /sys/devices/system/clocksource/clocksource0/ available_clocksource And change the value of clock source in current_clocksource. For example: $ echo tsc > /sys/devices/system/clocksource/clocksource0/current_clocksource 29

30 4 Verify Installation of Intel MPSS and Monitor Intel Xeon Phi Coprocessors 4.1 Miccheck The miccheck utility is used to verify the configuration and current status of the Intel Xeon Phi coprocessor software stack. It performs sanity checks on a host system with Intel Xeon Phi coprocessor(s) installed, by running a suite of diagnostic tests. The default behavior is to run all enabled tests on the host system first, and then on each Intel Xeon Phi coprocessor in turn. There are two forms of the miccheck utility in Linux*: A Python program, callable directly as miccheck.py. A binary program, executable as miccheck. Both versions behave exactly the same and will produce the same output. Sample output for miccheck utility is as follows: 4.2 Micinfo Micinfo provides information about system configuration. It includes information about current Intel Xeon Phi coprocessor hardware and the driver. Sample output for micinfo utility is as follows: 30

31 4.3 Micsmc Micsmc is the binary executable for the Intel Xeon Phi coprocessor Platform Status Panel. The micsmc tool can function in two modes: GUI mode and command-line (CLI) mode. GUI mode provides real-time monitoring of all detected Intel Xeon Phi coprocessors installed in the system. The CLI mode produces a snap-shot view of the status, which allows CLI mode to be used in cluster scripting applications. The micsmc tool monitors core utilization, temperature, memory usage, power usage statistics, and error logs, among other features. 31

32 5 Additional Configuration 5.1 LDAP Configuration on the Intel Xeon Phi Coprocessor File System In order to successfully configure LDAP on the Intel Xeon Phi Coprocessor file system, it is important that you have configured the network as an external bridge so that LDAP server can be reached from the coprocessor. In order to setup an external bridge for Intel Xeon Phi coprocessors, please refer section External Bridge Topology. Note - To enable LDAP with micctrl it is important that k1om rpm directory is extracted and specified before configuring LDAP. This will allow micctrl to find the required LDAP_PAM and LDAP_NSS rpm files. If it is not already done, it can be completed as follows: $ micctrl --rpmdir=</path/to/k1om> Specify the LDAP server as follows: $ micctrl --ldap=<ldap server IP address> --base=<domain> [mic card list] Note - If the server argument is set to remove then the use of LDAP on the card will be disabled Please refer Intel MPSS User Guide corresponding to your MPSS version on the Intel MPSS download page for further details about Customization and Manual configuration of LDAP on the Intel Xeon Phi Coprocessor file system. Additionally the following article can be referenced on setup and troubleshooting. 5.2 NIS Configuration on the Intel Xeon Phi Coprocessor File System In order to successfully configure NIS on the Intel Xeon Phi Coprocessor file system, it is important that you have configured the network as an external bridge so that LDAP server can be reached from the coprocessor. In order to setup an external bridge for Intel Xeon Phi coprocessors, please refer section External Bridge Topology. 32

33 Note - To enable NIS with micctrl it is important that k1om rpm directory is extracted and specified before configuring NIS. This will allow micctrl to find the required NIS_YPTOOLS, NIS_YPBIND, NIS_NSS and NIS_RPC rpm files. If it is not already done, it can be completed as follows: $ micctrl --rpmdir=</path/to/k1om> Specify the NIS server as follows: $ micctrl --nis=<nis/yp server IP address> --base=<domain> [mic card list] Note - If the server argument is set to remove then the use of NIS on the card will be disabled Please refer Intel MPSS User Guide corresponding to your MPSS version on the Intel MPSS download page for further details about Customization and Manual configuration of NIS on the Intel Xeon Phi Coprocessor file system 5.3 Micrasd and Reliability Monitor Support Micrasd Micrasd is the application running on the Host to handle and log the hardware errors reported by Intel Xeon Phi Coprocessors. It is a daemon that is started by the micras service. Using the "-daemon" option will run micrasd in daemon mode. In this case, micrasd will run in background and handle/log errors silently. In daemon mode, micrasd log messages are logged in the /var/log/micras.log file. Using the "-maint" option will enable Maintenance mode for error test and repair. If micrasd is executed with no arguments, it runs at the console prompt, connects to devices, and waits for errors. Use Ctrl-C to exit micrasd and return to the console prompt. Note - Micras service has dependency on the mpss service. - The Micras service must be started after the mpss service and stopped prior to stopping the mpss service Micrasd can be started and stopped as a Linux* system service as follows 1 : $ service micras start $ service micras stop 33

34 5.3.2 Reliability Monitor Reliability Monitor is designed to monitor overall health of compute nodes on cluster level. It is running on head node, or management node. Reliability Monitor works closely with RAS agent running on each service node. Any uncorrectable error or crash symptoms will be reported to Reliability Monitor Please refer User Guide corresponding to your MPSS version on the Intel MPSS download page for further details about installing and configuring Intel MPSS with Reliability Monitor Support 5.4 Installing Intel MPSS with OFED Support Detailed instructions to install Intel MPSS with OFED support can be found in the Intel MPSS User Guide corresponding to your MPSS version on the Intel MPSS download page. 5.5 Installing Intel MPSS with GANGLIA* support Detailed instructions to install Intel MPSS with GANGLIA* support can be found in the Intel MPSS User Guide corresponding to your MPSS version on the Intel MPSS download page. Additionally for system administrators managing a cluster with Intel Xeon Phi coprocessors, reading the following technical blog on Monitoring Intel Xeon Phi Clusters Using Ganglia is strongly recommended. 5.6 Configuring NFS mount on Intel Xeon Phi Coprocessors Adding an NFS mount micctrl --addnfs=<nfs export> --dir=<mount dir> [--server=<server>] \ [--options=<option>[,option, ]] [mic card list] Miccctrl --addnfs option adds an NFS mount entry to the Intel Xeon Phi coprocessor s /etc/fstab file. Note - NFS export specifies the NFS export and the mount directory in the traditional <server>:<export> format. The server parameter may still be used for backward compatibility with previous 34

35 versions of micctrl and will prepend its argument to the specified export. If the server is not specified, it places the IP address of the host in the server field. - The --options argument allows the adding of the standard NFS mount options. Check NFS documentation for more information. The string supplied is directly placed into the options field in the card s /etc/fstab file - For SUSE* based host systems if the NFS file system mounts have been added and the chckconfig utility has been used to indicate starting the MPSS stack at host boot time, SUSE* does not ensure the NFS server is started before the coprocessors are booted. To ensure the NFS server is available, edit the /etc/init.d/mpss file and change the # Required-Start: line to read # Required-Start: nfsserver Removing an NFS mount micctrl --remnfs=<mount dir> [mic card list] The --remnfs option searches the /etc/fstab for the Intel Xeon Phi coprocessor for the specified mount point and removes the mount point from the file NFS mounting a host export Like any Linux* based system, NFS mounting a host export on an Intel Xeon Phi coprocessor requires that the NFS server is visible to the coprocessor and NFS export policy is set to allow the coprocessor to mount it. Note - For NFS to work, the host firewall or iptables may need to be configured to allow the following ports: tcp/udp port RPC 4.0 portmapper tcp/udp port nfs server For illustration purpose, let us assume that we intend to provide access from the /micnfs directory to the first Intel Xeon Phi coprocessor. Two of the known cases are: Case 1: NFS server is running on the same host to which coprocessor is attached. In this case the coprocessor would have access to the NFS server using static pair network topology. If we assume static pair was set with default configuration then the network addresses would be as follows: IP Address Role NFS Server & Intel Xeon Phi Coprocessor attached host system NFS Client (Intel Xeon Phi Coprocessor) 35

36 Case 2: NFS server is running on a different remote server (in cluster environment) and not on the same host to which coprocessor is attached. In this case the coprocessor would have access to the NFS server using external bridge network topology. If we assume external bridge was set as explained in section External Bridge Topology then the network addresses (as an example) would be as follows: IP Address Role NFS Server Intel Xeon Phi Coprocessor attached host system NFS Client (Intel Xeon Phi Coprocessor) In both the cases steps for NFS mounting a host export can be listed as follows: Ensure that the /micnfs directory you want to export exists on the NFS server if not, create it as follows: $ mkdir /micnfs Append /etc/exports on the NFS server with the line /micnfs <NFS Client IP Address or Range of IP Address>(rw, no_root_squash) Add in /etc/hosts.allow on the NFS server ALL: <NFS Client IP Address or Range of IP Address> Let NFS know the files have changed by executing $ exportfs -a Add an NFS mount entry to the Intel Xeon Phi coprocessor s /etc/fstab file. Execute this command on Intel Xeon Phi Coprocessor attached host system. $ micctrl --addnfs=<nfs Host IP Address>:/micNfs --dir=/micnfs Restart MPSS 1 $ service mpss restart Execute the following command on the Intel Xeon Phi coprocessor, i.e. NFS client, to verify if the mount was successful. $ mount 36

37 Note - If you perform a service mpss restart and find the NFS links unoperational, execute the following command on the coprocessor mount a - For SUSE*, to auto-start the NFS server on reboots, change the Intel MPSS script (/etc/init.d/mpss) line as follows below: From: # Required-Start: To: # Required-Start: nfsserver - For RHEL*, to auto-start the NFS server on reboots, ensure the NFS server priority is higher than that of Intel MPSS Please refer the Intel MPSS User Guide corresponding to your MPSS version on the Intel MPSS download page for help on other NFS related topics such as: How to Enable NFS Auto Mount with NIS/YP on the Intel Xeon Phi Coprocessor Mounting a share using NFS v4 Root is an NFS export 5.7 Configuring Lustre on Intel Xeon Phi Coprocessors For detailed instructions to use Lustre* on Intel Xeon Phi Coprocessors please refer the following blog on Compiling, Configuring and running Lustre on Intel Xeon Phi Coprocessor. 37

38 6 Troubleshooting 6.1 Common Issues Hardware or Intel MPSS installation issues Intel Xeon Phi coprocessor is not detected or Intel MPSS is not functioning Please refer to the available Intel MPSS troubleshooting flowchart for debugging hardware detection and PCIe link training issues. Host will not boot with the coprocessor card installed Does the host BIOS support the Intel Xeon Phi coprocessor? Check with your system supplier User configuration issues User does not show up in /etc/passwd file on the coprocessor when the file is autogenerated Is the user in the /etc/passwd file on the host? The autogenerated /etc/passwd file for the coprocessor is based on the host s /etc/passwd file Is the user s id number on the host in the range specified by the UserAuthentication option in the /etc/sysconfig/mic/default.conf or /etc/sysconfig/mic/mic<n>.conf files? A user entry is copied from the host s /etc/passwd file to the coprocessor s /etc/passwd file only if UserAuthentication is set to Local and the user id falls within the range specified. Does the /etc/sysconfig/mic/mic<n>.conf file you are looking at correspond to the coprocessor, mic<n>, where the problem is? UserAuthentication must be specified in the configuration for each coprocessor. Have you run micctrl --resetconfig since you added the user to the host s /etc/passwd file? The --resetconfig pushes changes to the files under /etc/sysconfig/mic out to the /opt/intel/mic/filesystem files. This includes rebuilding the /etc/passwd file for the coprocessor. User does not show up in the /etc/passwd file on the coprocessor when an externally created file is used Are you replacing the default /etc/passwd file created by micctrl with your own file? There are several ways to do this. You can use an Overlay entry to automatically copy your password file over the automatically generated password file in the /opt/intel/mic/filesystem/mic<n>.image file. You can manually copy your password file 38

39 over the /opt/intel/mic/filesystem/mic<n>/etc/passwd file before booting the coprocessor. You can also copy your password file over /etc/passwd on the coprocessor after booting it. Have you run micctrl --resetconfig since copying your password file over /opt/intel/mic/filesystem/mic<n>/etc/passwd? Some options to micctrl, such as -- resetconfig, will overwrite any changes you have made to files under the /opt/intel/mic/filesystem/mic<n> directory. Have you rebooted the coprocessor either by using micctrl --boot or by restarting the mpss service since you made your changes? Changes you make to /opt/intel/mic/filesystem/mic<n>/etc/passwd will not show up on the coprocessor until you reboot it. On the other hand, changes you make to /etc/passwd on the coprocessor itself will be overwritten by the file in /opt/intel/mic/filesystem/mic<n>.image when the coprocessor is rebooted. User does not show up in the /etc/passwd file on the coprocessor when you have added them using the micctrl --useradd? Did you add the user to the coprocessor you are looking at? You must either specify the specific coprocessor number you want to add the user to or you must specify no coprocessor number to add the user to all coprocessors on the system. Did you add the user after this coprocessor was installed? If you have added a new coprocessor since you added the user, the user will need to be specifically added to that coprocessor even if the user was originally added to all coprocessors. Have you rebooted the coprocessor since adding the adding the user? Changes to the password file made with micctrl --useradd are not pushed out to the coprocessor until it is rebooted. Have you run micctrl --resetconfig since adding the user? Changes made to the password file with micctrl --useradd are overwritten with the automatically generated password file when the --resetconfig option is used. User cannot log in Does the user show up in the /etc/passwd file on the coprocessor RAM file system? The user must exist on the coprocessor before they are able to log in Is the user using ssh to log in? The user must use a secure login to log into the coprocessor. Does the user have a valid.ssh directory in their home directory on the host? If you are using the automatically generated /etc/passwd file and home directories on the coprocessor, 39

40 the.ssh directory will have been copied from the host home directory to the default home directory on the coprocessor. The key in that directory must be a valid RSA key. Did the user set a passphrase when they generated their RSA key? Depending on the version of Linux you are using, there can be differences in the encryption of the passphrase. If the user created the key using a passphrase they should remove the key and generate a new one without a passphrase. If the user s home directory is mounted from a network file system, does that home directory contain a valid.ssh directory? If the home directory being used on the coprocessor is not the default one created by micctrl, you must make sure the.ssh directory for each user finds its way to that alternate home directory. Did the user receive a warning message WARNING: REMOTE HOST IDENTIFICATION HAS CHANGED? When the address associated with a system name has change since the last time ssh was used to connect to that system, SSH will refuse to continue until the user has resolved the problem. This can be done by editing the known_hosts file in the user s.ssh directory to remove the line for that system or by simply removing the known_hosts file. The next time ssh is used to connect to that system, the user will be asked if the system should be added to the known hosts; the user should reply yes. Note: if the user s home directory is a network mounted file system, the user should always use the complete name for the coprocessor, for example, `hostname`-mic0 rather than mic0. This will prevent confusion between that coprocessor and the mic0 coprocessors found on other hosts. Are you running SELinux on the host? Secure Linux uses different file permissions for some directories than those expected by ssh on the coprocessor. If you install the Intel MPSS with SELinux enabled, uninstall the Intel MPSS, disable SELinux and reinstall the Intel MPSS. Has the user defined StrictHostKeyChecking in the ssh options? If yes, does their.ssh/known_hosts file contain the correct key? User s home directory on the coprocessor is empty. Is the home directory for the user the default directory created by micctrl? The default home directory created by micctrl contains only the.ssh directory and a.profile file. Is the home directory for the user located on the coprocessor itself? The only file space directly on the coprocessor is on the RAM disk in the coprocessor s memory. Files the user leave in that directory will be lost when the coprocessor is rebooted. Is the home directory mounted from a network file system? Make sure the file system is mounted correctly and that the home directory listed in the /etc/passwd file matches the name of the home directory that has been mounted over the network. 40

41 6.1.3 NFS mounting issues for Intel Xeon Phi coprocessors NFS is not mounting a network file system Is the remote system allowed to export the files? The /etc/hosts.allow file on the remote system must allow the coprocessor to have NFS access. The /etc/exports file must list the directories that can be exported to the coprocessor. Have you run exportfs a? Any NFS daemons which are running when you change /etc/hosts.allow and /etc/exports won t see the changes until you run the exportfs command. Are the proper deamons running on the remote system? The daemons required to export files differ slightly between versions of Linux. Check the documentation for the remote host s operating system to be sure the proper NFS daemons are running. Are there firewalls between the coprocessor and remote system? Make sure ports 111 (portmapper) and 2049 (the nfs server) are not being blocked by any firewalls. Has the coprocessor attempted to mount the file system? You must either have a line in the /etc/fstab file on the coprocessor that provides mounting directions at boot time or you must execute a mount command, either by hand or by running a startup script. Do you have a mount point for that file system on the coprocessor? Before you can mount a file system over the network, you must have a directory on the coprocessor which can serve as the parent directory for the files that are being mounted. Make sure you have created that directory in the filelist file used to construct the disk image for the coprocessor. Can you ping the coprocessor from the file server and the file server from the coprocessor? You must have a network connection in order to mount network file systems. Check to be sure your IP addresses, netmask and subnet are correct and that, if the coprocessor requires a bridge to reach the remote host, the bridge is properly configured. Do the MTU sizes along the path between the coprocessor and the file server match? If the MTU sizes between the coprocessor and file server do not match, NFS will be unable to mount the file system even if ping succeeds. You can log onto the file server and use the traceroute -mtu command to determine the mtu sizes along the path. Alternately you can use the ifconfig command on each system to see how each interface is configured Miscellaneous issues Linux command returns not found on the coprocessor. Is the command provided on the coprocessor? To save space the coprocessor contains only a subset of the usual Linux commands. Check to see if the command is installed on the coprocessor. 41

42 Is the command a link to BusyBox? Many of the commands on the coprocessor are provided as links to the file /bin/busybox. Make sure the file /bin/busybox exists and the command you are trying to use points to the correct file. Is the command in your path? Check your PATH variable to be sure it points to the directory where the command is located. System administration tool on the host will not run Some tools provided in the Intel MPSS package only work if the coprocessor is booted. Some tools provided in the Intel MPSS package only work if the coprocessor is in the ready state and the mpssd daemon is running. $ micctrl --boot Some tools provided in the Intel MPSS package only work when the coprocessor is in ready state and the mpssd daemon is not running. $ micctrl --initdefaults $ micctrl --resetconfig $ micctrl --resetdefaults $ micctrl --cleanconfig The coprocessor appears to be hung Does micctrl -status say that the coprocessor is online? Try connecting to the coprocessor using a virtual console. Does miccheck say that the coprocessor is detected? If it is not detected, there may be a hardware problem. Check to be sure the coprocessor is properly seated in the PCIe connector. Review any recent installs that have been done to be sure the proper version of flash was installed. If the flash that was installed was for a different version of the coprocessor, contact the supplier for your coprocessor card. The problem may be unrecoverable. Does miccheck say that the host OS can talk to the coprocessor? Does miccheck say that the network connections are working? Try connecting to the coprocessor using a virtual console and using the standard Linux networking command to be sure the network is configured properly. 42

43 The flash has been updated but the flash version is still showing as the old value Did you reboot the host after updating the flash? Changes to flash show up only after the coprocessor card has been fully reinitiallized. Are you using the coprocessor from a virtual machine on the host? A software reboot of the virtual machine does not reinitiallize the coprocessor card. You must power cycle the host before changes in the flash take effect. Changes made to files in /etc/sysconfig/mic have disappeared Have you run micctrl --resetdefaults? This option sets your configuration files to the current default values. If the contents of the configuration files is different from the actual configuration of the Intel MPSS, those changes can disappear. Have you run micctrl --cleanconfig? This will backup your configuration files and remove the actual files. Look for the backup files. Flash update fails Have you followed the update directions from the readme file? Reread the directions and retry the update Have you tried the flash trouble shooting directions? Follow the troubleshooting directions in the forum post: Flash Issues & Remedies 6.2 For the Inexperienced Linux System Administrator How do I find out the version of Linux running on the host system? $ cat /etc/system-release Is it better to log in as root or precede commands with sudo? The sudo command leaves a better record of who did what and prevents you from accidentally doing something as root that you don t intent to (such as rm rf *). How do I find out what kernel modules are installed? $ lsmod 43

44 How do I find out if a specific module is installed? $ lsmod grep <name_of_the_module_i_am_interested_in> How do I findout which file systems are mounted? $ mount How do I find out which network interfaces are configured on a system? $ ifconfig How do I find out if I have a network connection to another system? $ ping <ip_address_of_target_system> How do I find out which path a network connection followed? $ traceroute <ip_address_of_target_system> Where can I go to find out more about a particular command? If you installed the man pages (manual pages) when you installed your host operating system: $ man <command_name_i_am_interested_in> 6.3 Glossary BIOS - Basic Input/Output System, the interface on the host which executes firmware from the system s ROM to initialize and test the system hardware at boot time, then manage and monitor the hardware after boot COI Coprocessor Offload Infrastructure, library routines and a daemon used when work is being offloaded from the host to the coprocessor; responsible for coordinating the moving of data, executables and libraries between the host and coprocessor and for managing process setup and teardown on the coprocessor coi-daemon the daemon which runs on the coprocessor and controls setup and teardown of processes that are offloaded from the host to the coprocessor flash ROM memory on the coprocessor containing the software used to initialize the coprocessor 44

45 Ganglia a tool providing web based system monitoring for clusters and grids; see GPL the General Public License (the Gnu license), which requires anyone providing executables built with GPL licensed code to also make the source code available hence the gpl directory in the Intel MPSS release grub - GRand Unified Bootloader, the Linux utility responsible for selecting the desired version of the operating system at boot time host a computer system containing one or more coprocessor cards kernel module a section of executable code which can be added to or removed from the operating system kernel without needing to reboot the system; kernel modules are often used to add drivers to the kernel, as in the case of the coprocessor kernel modules which provides the driver for accessing the Intel Xeon Phi coprocessor card Linux service a program or set of programs which run in the background, waiting to perform some action for the users; Linux services are controlled from scripts in /etc/rc.d/init.d which set up a known environment in which the service is executed; in the case of the mpss service, the mpss script starts the mpssd only after ensuring that the coprocessor kernel module is installed and checking on the status of the coprocessor cards using the micctrl command; the script also provides the ability to stop or restart the mpssd and to remove the coprocessor kernel module from the host s kernel Intel MPSS - the Manycore Platform Software Stack, the collection of software, including firmware, daemons, kernel modules, administrative tools, Linux uos and Linux commands, required by the Intel Xeon Phi coprocessor mpssd the daemon which runs on the host and is responsible for communicating with the coprocessor OFED - the OpenFabrics Enterprise Distribution (OFED), provides support for performing RDMA (remote direct memory access, see below) between computer systems; see PCIe - Peripheral Component Interconnect Express; a high speed serial expansion bus standard; Intel Xeon Phi coprocessor cards are connected to their host system via PCIe connections RAM FS a file system which is stored in the machines memory rather than on a disk or other permanent medium RAS reliability, availability and serviceability; a measure of a systems proper functioning, usable uptime and maintainability 45

46 RDMA Remote Direct Memory Access, the ability to read or write to another system s memory without the involvement of that system s CPU; RDMA minimizes overhead in the communication between network systems thereby optimizing bandwidth and latency rpm a software package manager that can build, install, update and uninstall software packages known as rpm files RSA keys a pair of keys, one publicly know and one known only to its owner, generated using the RSA algorithm; the SSH (secure shell) routines use this pair of keys to encrypt and decrypt information sent between two computer systems SCI (formerly SCIF) Symmetric Communications Interface, a mechanism for internode communication used to communicate between the host and coprocessor over the PCIe connection SMC - System Management and Configuration, an interface on coprocessor which executes firmware from ROM to configure, manage and monitor the coprocessor s hardware sudo a command which allows you to execute another command as if you were a different user, often used to allow a limited set of users to execute a specified set of commands with superuser (root) privileges without needing to provide them with the root password uos micro-os, a small operating system; uos can refer to the small boot kernel which is stored in flash on the coprocessor and is used to bring the coprocessor to the ready state; it can also refer to the small Linux kernel which is copied from the host to the coprocessor at boot time 46

47 Appendix A: Supported versions The list of supported versions of Linux and Microsoft Windows found in the afore mentioned article, Intel Manycore Platform Software Stack (MPSS), only mean that Intel actively supports and validates MPSS against those OSs. MPSS has been ported to many other OSs and versions, for example, Cento OS and Ubuntu, as well as kernels but the validation and support of these unsupported kernels is left up to the customer. Nevertheless, the Intel Xeon Phi Coprocessor technical forum will answer questions and provide whatever help they can to all users, including those with unsupported kernels and OSs. 47

48 Acknowledgements I would like to thank all the individuals who provided information, documentation and feedback in an effort to keep me honest about how an Intel Xeon Phi coprocessor is administered, especially, Michael Hebenstreit, Charles Congdon, Loc Nguyen, and Sumedh Naik. Of course, any mistakes I manage to slip past them are purely my responsibility. About the Authors Frances Roth received a Bachelor s degree in Math from MIT, a Master s degree in Geoscience from the University of Arizona and did graduate work in Computer Science at the University of Maryland, Baltimore County. She is currently a software engineer with Intel Corporation's Software and Services Group. Her interests include parallel processing and Fortran. Sunny Gogar received a Bachelor s degree in Electronics and Telecommunications from the University of Mumbai, India and a Master s degree in Electrical and Computer Engineering from the University of Florida, Gainesville. He is currently a software engineer with Intel Corporation's Software and Services Group. His interests include parallel programming and optimization for Multi-core and Many-core Processor Architectures. Taylor Kidd received his Ph.D. in Electrical Engineering from the University of California at San Diego. He is currently a software engineer with Intel Corporation's Software and Services Group. His interests include parallel programming and optimization for Multicore and Many-core Processor Architectures. He is an Intel Black Belt developer and writes blogs on a variety of topics. 48

49 Notices INFORMATION IN THIS DOCUMENT IS PROVIDED IN CONNECTION WITH INTEL PRODUCTS. NO LICENSE, EXPRESS OR IMPLIED, BY ESTOPPEL OR OTHERWISE, TO ANY INTELLECTUAL PROPERTY RIGHTS IS GRANTED BY THIS DOCUMENT. EXCEPT AS PROVIDED IN INTEL'S TERMS AND CONDITIONS OF SALE FOR SUCH PRODUCTS, INTEL ASSUMES NO LIABILITY WHATSOEVER AND INTEL DISCLAIMS ANY EXPRESS OR IMPLIED WARRANTY, RELATING TO SALE AND/OR USE OF INTEL PRODUCTS INCLUDING LIABILITY OR WARRANTIES RELATING TO FITNESS FOR A PARTICULAR PURPOSE, MERCHANTABILITY, OR INFRINGEMENT OF ANY PATENT, COPYRIGHT OR OTHER INTELLECTUAL PROPERTY RIGHT. A "Mission Critical Application" is any application in which failure of the Intel Product could result, directly or indirectly, in personal injury or death. SHOULD YOU PURCHASE OR USE INTEL'S PRODUCTS FOR ANY SUCH MISSION CRITICAL APPLICATION, YOU SHALL INDEMNIFY AND HOLD INTEL AND ITS SUBSIDIARIES, SUBCONTRACTORS AND AFFILIATES, AND THE DIRECTORS, OFFICERS, AND EMPLOYEES OF EACH, HARMLESS AGAINST ALL CLAIMS COSTS, DAMAGES, AND EXPENSES AND REASONABLE ATTORNEYS' FEES ARISING OUT OF, DIRECTLY OR INDIRECTLY, ANY CLAIM OF PRODUCT LIABILITY, PERSONAL INJURY, OR DEATH ARISING IN ANY WAY OUT OF SUCH MISSION CRITICAL APPLICATION, WHETHER OR NOT INTEL OR ITS SUBCONTRACTOR WAS NEGLIGENT IN THE DESIGN, MANUFACTURE, OR WARNING OF THE INTEL PRODUCT OR ANY OF ITS PARTS. Intel may make changes to specifications and product descriptions at any time, without notice. Designers must not rely on the absence or characteristics of any features or instructions marked "reserved" or "undefined". Intel reserves these for future definition and shall have no responsibility whatsoever for conflicts or incompatibilities arising from future changes to them. The information here is subject to change without notice. Do not finalize a design with this information. The Intel Xeon Phi Coprocessors described in this document may contain design defects or errors known as errata which may cause the product to deviate from published specifications. Current characterized errata are available on request. Contact your local Intel sales office or your distributor to obtain the latest specifications and before placing your product order. Copies of documents which have an order number and are referenced in this document, or other Intel literature, may be obtained by calling , or go to: Intel, the Intel logo, Intel Xeon Phi, Intel VTune Amplifier XE and Xeon are trademarks of Intel Corporation in the U.S. and other countries. *Other names and brands may be claimed as the property of others Copyright 2014 Intel Corporation. All rights reserved. 49

50 Optimization Notice Intel's compilers may or may not optimize to the same degree for non-intel microprocessors for optimizations that are not unique to Intel microprocessors. These optimizations include SSE2, SSE3, and SSE3 instruction sets and other optimizations. Intel does not guarantee the availability, functionality, or effectiveness of any optimization on microprocessors not manufactured by Intel. Microprocessor-dependent optimizations in this product are intended for use with Intel microprocessors. Certain optimizations not specific to Intel microarchitecture are reserved for Intel microprocessors. Please refer to the applicable product User and Reference Guides for more information regarding the specific instruction sets covered by this notice. Notice revision #