HPC system startup manual (version 1.30) Document change log Issue Date Change 1 12/1/2012 New document 2 10/22/2013 Added the information of supported OS 3 10/22/2013 Changed the example 1 for data download 4 3/4/2014 Added the chapter of Torque/Maui batch job system 5 3/4/2014 Changed text font 6 10/2/2014 Changed the example 1 for data download Modified pages Notes 1 Overview The high performance computing (HPC) system in the SACLA facility consists of 80 compute nodes, with two 6-core processors per node, for a total of 960 cores. The theoretical peak compute performance is 13 TFLOPS. The system supports a 170TB shared storage system, managed by the Lustre file system. Nodes are interconnected with Infiniband with a 40 Gbps bandwidth. This manual introduces the startup from account creation to the launch of software installed in the SACLA HPC system. 2 Account creation After your experiment proposal accepted, a responsible researcher of the SACLA is assigned to your proposal. Contact him by e-mail and request the use of the SACLA HPC system. He will create your account and give the username and password. Note that /home directory is limited by quota to 100 GB per user. When you are going to use more storage area for a large amount of experimental data and the result data of the analysis, work directory can be used. Please talk with your contact researcher for the use of /work directory. 3 How to login SACLA HPC system 1 / 5
The access of the SACLA HPC system is limited to be within SPring-8 local network. First of all, you have to connect your computer to the SPring-8 local network by using VPN service. After that, you can login the HPC server through SSH protocol. [ step 1. Connect the SPring-8 network with VPN service. ] Access the web page: https://hpc.spring8.or.jp with a web browser. Login the web page of VPN installation service. (Enter your username and password given in Sec. 2) Download and install VPN connection software into your computer. Supported OS for VPN service is as follows. [Windows] Windows 7 Professional(64bit) Windows 7 Home Premium(64bit) Windows Vista Home Premium(32bit) [Mac] Mac OS X Lion (10.7) [Linux] Debian GNU/Linux 6.0 (x86_64) Redhat Enterprise Linux 6.2 (x86_64) [ step 2. Login SACLA HPC system. ] Type ssh -X [username]@xhpcfep. The username is given in Sec.2. Enter your password given in Sec.2. 4 Torque/Maui batch job system The queuing system on SACLA HPC system is Torque/Maui. You need three main commands to use this system: showq, qsub, and qdel. [showq] showq displays information about active, eligible and blocked jobs. [qsub] qsub submits a job. In its simple form you would run qsub myscript, where myscript is a shell script that runs your job. For those who don't want to write scripts, you can do an interactive qsub with the -I option. You can also use -X option for enabling X11 forwarding (e.g. qsub I X q smp ). [qdel] 2 / 5
qdel job_number deletes your job. You can get information on job_number with showq. 5 Data handling in SACLA HPC system [ Ex. 1 Download your experimental data to your storage space in the HPC. ] This example downloads an HDF5 file with the data of run number 28930 in the tag range from 74121900 to 74121912 to the path of /home/[username]/data/test/sample.h5 using interactive qsub. Type qsub -I in the terminal window. Type mkdir -p ~/data/test in the terminal window. Type DataConvert3 -dir ~/data/test -o sample.h5 -each -mode 20 -r 28930 -starttag 74121900 - endtag 74121960 in the terminal window. DataConvert3 command can be used at all compute nodes. [ Ex. 2 Open your experimental data. ] This example shows you can open the HDF5 file of sample.h5 generated in Ex. 1 by using software of HDFView. After procedure listed below, the 2 dimensional data is displayed as an image on the HDFView GUI. Type hdfview in the terminal window. Open a file open dialog by clicking File -> Open in the menu bar of the HDFView. Specify a filename to be /home/[username]/data/test/sample.h5 in the file open dialog. Extract the HDF5 tree displayed in the HDFView left sidebar by clicking run_[run number] - > detector_2d_[detector ID] -> tag_[tag number] Invert and right click a data object of detector_data and select Open as in the context menu Select Display As Image and click OK in the dataset selection dialog. [ Ex. 3 Data transfer to your local computer. ] This example is a way to transfer your data from the SACLA HPC to your local computer through SSH protocol. Type scp [username]@xhpcfep:/home/[username]/data/test/sample.h5 [favorite path] on your local terminal window. [ Ex. 4 Browse SACLA Database. ] The beamline information is recorded to a MySQL server. You can browse and plot the recorded information by using web service on the SACLA HPC network. The example written below is a 3 / 5
procedure to display the trend graph of a signal on the web page. Type firefox in the terminal window. Access the web page: http://xqaccdaq01.daq.xfel.cntl.local/syncdaq/xdaq.html. Register a signal (in this example, we use beam monitor 1) to the left top sub-window by clicking links xq-bl3-fetc -> xfel_bl_3_tc_bm_1_pd/charge in the left bottom sub-window. Select the registered signal of xfel_bl_3_tc_bm_1_pd/charge to be left Y-axis by inverting the signal name in list box of Left Axis in the left top sub-window. Select Time to be X-axis in the left top sub-window. Specify BeignTime and EndTime to the example time period: 2012-05-15 23:29:29, 2012-05- 15 23:29:50 in the left top sub-window. Click plot button [ Ex. 5 Export data of SACLA Database to a Text file. ] You can export the beamline information registered in the MySQL server to an text file by using a command: syncdaq_get. Type syncdaq_get l200 -b"2012/05/15 23:29:29" -e"2012/05/15 23:29:50" xfel_bl_3_tc_bm_1_pd/charge > bm_data.txt in the terminal window. 6 Installed software List and how to use The available software is listed below. [ Hdfview ] Type hdfview in the terminal window. [ MATLAB] Type qsub -I -vdisplay in the terminal window. Type matlab in the terminal window. Note: The concurrent client number is restricted to be the number of license key RIKEN possess. You MUST close the MATLAB after finish your analysis. 4 / 5
[ ImageJ ] Type qsub -I -X in the terminal window. Type ImageJ in the terminal window. [ ROOT ] Type export ROOTSYS=/home/software/root/pro in the terminal window. Type export PATH="$PATH:$ROOTSYS/bin in the terminal window. Type export LD_LIBRARY_PATH="$LD_LIBRARY_PATH:$ROOTSYS/lib/root" in the terminal window. Type root in the terminal window. [ FireFox ] Type firefox in the terminal window. 5 / 5