Australian Synchrotron, Storage Gateway

Transcription

1 Australian Synchrotron, Storage Gateway User Help Manual Version 1.2

2 Storage Gateway User Help Manual 2 REVISION HISTORY Date Version Description Author 2 May Document creation Chris Myers 13 May Document Updated Chris Myers May Minor edits Mika Tsoi Purpose of the Storage Gateway... 3 Getting Started... 4 Requirements... 4 Logging In... 5 VBL Menu... 6 Storage Gateway Selection Menu... 7 Transfer Methods... 8 GridFTP Transfer... 8 After Uploading Your Proxy Certificate... 9 irods SLCS Certificate Transfer irods User/Password Status Requirements Using the Service SRB User/Password Transfer Transferring Files Transfer Setup Page Transfer File Windows Set the Transfer Target Advanced GridFTP Settings TRANSFER FILES NOW TRANSFER FILES LATER Troubleshooting Downloading Files to Your Computer SSL Transfer... 20

3 Storage Gateway User Help Manual 3 Purpose of the Storage Gateway The Storage Gateway is intended to enable users of the Australian Synchrotron to transfer files from Synchrotron storage to storage at their home institution. The Storage Gateway is intended to interact with existing storage solutions at home institutions, where the existing storage solutions implement irods, GridFTP or SRB. The Storage Gateway also allows a user to download files to their desktop via SSL. The storage at the Australian Synchrotron is intended to be short-term storage, for storing raw data from instruments. As such, methods are needed to transfer raw files to the home institutions of researchers. Instead of needing to know all the details of the Australian Synchrotron storage, the interface allows users to use the details of their home institution s storage to transfer files. To find out how long files are kept on Australian Synchrotron storage, see your Administrator. The Synchrotron is only for short-term storage. Long-term storage should be at your home institution or some other storage solution. The Storage Gateway allows you to transfer files from the short-term storage to your long term storage. The Storage Gateway allows both third party transfer and downloading of files. GridFTP, irods and SRB are all third party transfer methods. SSL transfer, despite the name, is a method of downloading files to the user s desktop. The Storage Gateway in not intended to replace the use of portable hard drives, but to complement their use.

4 Storage Gateway User Help Manual 4 Getting Started Requirements To use the Storage Gateway you will require an account at the Australian Synchrotron. To request an account: Call the Australian Synchrotron User Office on The Storage Gateway has been tested with Firefox and Internet Explorer. We recommend using Firefox with all VBL system pages. The VBL is using self-signed certificates at this time. If you are using Firefox or Safari, please click yes to accept the certificate and create an exception. If you are using Internet Explorer 7, click to continue. Your computer must be connected to the internet or the Australian Synchrotron local area network (LAN).

5 Storage Gateway User Help Manual 5 Logging In Open you web browser and connect to the following web site: You will then see: To login: Enter your username in the Username box. Enter your password in the Password box. Click on the Login button.

6 Storage Gateway User Help Manual 6 VBL Menu You will then see: This tasking area is the main menu for the VBL. You can move to all service areas and also change your password from this page. Please note: your username is displayed in the top right corner. To access the storage gateway: Click on the Storage Gateway link.

7 Storage Gateway User Help Manual 7 Storage Gateway Selection Menu You will then see: This page has a drop down menu that displays all the experiments that you have access to. Click on the experiment you wish to access. Below this is a drop down menu that gives you access to different methods of transfer. Click on the transfer method you wish to use. Click Submit. A range of methods can be used to transfer files to your home institution. The method you choose to use will be limited by the connection mechanisms available at the home for the data. The various transfer methods will be detailed following: GridFTP, irods, SRB, and SSL.

8 Storage Gateway User Help Manual 8 Transfer Methods GridFTP Transfer GridFTP is a fast secure file transfer protocol for grid applications. It is part of the Globus Toolkit and uses Grid Security Infrastructure for authentication and encryption. Used with the Storage Gateway, it is a third party transfer protocol. You can transfer files between Australian Synchrotron storage and storage at your home institution, but not directly to your desktop. To use GridFTP transfer, a GridFTP node must be set up at your home institution. To see if you have GridFTP set up at your home institution, ask your institution s grid or storage administrator. To use GridFTP transfer, upload your Grid Proxy File: Click on the User Preferences Area. Click Browse to locate your Grid Proxy File. Click Open. Click SUBMIT. Managing your Proxy Your proxy must have permission to both connect and write data to your storage. This can involve a number of factors including account permissions, group membership, virtual organisation membership (VOMS proxies). Contact your storage or Grid administrator to ensure you have the correct permissions. Please Note: Proxy files expire after a set period of time (default is usually 12 hours). If the default proxy expiry period is insufficient for data transfer you may need to create a proxy a with longer expiry time. After expiry you will need to upload a new proxy file before further GridFTP transfers. If you upload a new proxy while a transfer is going on, transfers can continue until that proxy expires.

9 Storage Gateway User Help Manual 9 After Uploading Your Proxy Certificate The Storage Gateway runs a test on the current proxy certificate and returns either VALID (proxy is good) or INVALID (proxy is bad, revoked or expired). If you have not uploaded a Grid Proxy File, the Storage Gateway will display an error. Please upload a Proxy. If your proxy is valid, the time left until the proxy expires is displayed. This information is important if your data transfer will take a long time to process or if you request for the transfer is to be scheduled. Please note: your proxy must be valid for the duration of your transfer, otherwise the transfer will fail. To transfer your files: Click the Start button to move to File Transfer. see the Transferring Files section on page 15.

10 Storage Gateway User Help Manual 10 irods SLCS Certificate Transfer irods is a middleware architecture that provides a virtualized file system interface for users, abstracted away from a physical file system. In terms of the Storage Gateway, it is a third party transfer protocol, allowing you to transfer files from the Australian Synchrotron storage to your home institution s storage, but not directly to your desktop. To use irods you will need to have irods set up at your home institution and know the details of that installation including your username and password, and your IdP. The difference between irods SLCS Certificate and irods User/Password, apart from the IdP, is the actual user/password pairing. irods User/Password will log you into that irods installation. irods SLCS Certificate will log you into an openidp, which will then give you access to irods. The two should be different. When you have this information, to use irods to transfer your data from the Australian Synchrotron to your institutional storage: Select your experiment from the drop down list. Select irods SLCS Certificate as your transfer method. Click Select. Enter your irods username. Enter your irods password. Select your IdP. Select your Home Domain. If you want to enter your irods host, port, zone and SLCS Server, do so under Advanced. Click Login.

11 Storage Gateway User Help Manual 11 Transfer files as per Transfer Files section on page 15.

12 Storage Gateway User Help Manual 12 irods User/Password irods is a middleware architecture that provides a virtualized file system interface for users, abstracted away from a physical file system. In terms of the Storage Gateway, it is a third party transfer protocol, allowing you to transfer files from the Australian Synchrotron storage to your home institution s storage, not directly to your desktop. The difference between irods SLCS Certificate and irods User/Password, apart from the IdP, is the actual user/password pairing. irods User/Password will log you into that irods installation. irods SLCS Certificate will log you into an openidp, which will then give you access to irods. The two should be different. Status This service is operational Requirements You need: An account for the Storage Gateway. Access to the experiment files to transfer. Be registered at The standard home domain is ARCS. Have irods set up at your home institution. Know the details of that irods installation.

13 Storage Gateway User Help Manual 13 Using the Service Select your experiment from the drop down list. Select irods Username/Password as your transfer method. Click Select. Enter your irods username. Enter your irods password. Select your home domain. If you want to enter your irods host, port, zone and Slcs Server, do so under Advanced. Click Login. Transfer files as per Transfer Files section on page 15.

14 Storage Gateway User Help Manual 14 SRB User/Password Transfer SRB provides a uniform interface to heterogeneous storage over a network. SRB provides a method of accessing files so they look the same to the end user, regardless of the actual physical location of the files. The Storage Gateway will interact with your home institution s SRB installation, once provided with details. SRB User/Password Transfer acts as a third party transfer service, allowing you to transfer files from the Australian Synchrotron storage to your home institution s SRB installation, but not directly to your desktop. Contact your home institution s grid or storage administrator to find out if your home institution has SRB installed and the settings for this installation including your username and password. When you have this information, to transfer your data from the Australian Synchrotron to your institutional storage using SRB: Select your experiment. Select SRB User/Password as your transfer method. Click Select. Enter your SRB username. Enter your SRB password. Select your Home Domain. If you want to enter your Mdas Collection Name, Collection Home, Domain Home, SRB Host, Port, or MCAT Zone, do so under Advanced. Click Login. Transfer files as per Transfer Files section on page 15.

15 Storage Gateway User Help Manual 15 Transferring Files Transfer Setup Page This page allows you to select which files to transfer. To select files to transfer: Drag a file from the left directory tree over to the right directory tree. Drag a directory from the left directory tree over to the right directory tree. Click the >> button which will move all the files across. To view the contents of a directory: Expand it using the + tree icon, or Click Expand All. To collapse a directory: Click the + tree icon next to an expanded directory, or Click Collapse All. To remove a file or all files: Click Remove, or Click Remove All.

16 Storage Gateway User Help Manual 16 Transfer File Windows Your transfer file windows should look like the screen above. When you have decided which files to transfer you can proceed with the rest of the process.

17 Storage Gateway User Help Manual 17 Set the Transfer Target To set the final location for the transferred files: Click on the drop down menu and select your site, if it is there. If not, then type the host name or IP of the server that is your transfer target into the drop down menu. To specify a target directory: In the box on the right, type in the directory you wish to place your files in. Advanced GridFTP Settings If you are using GridFTP as your transfer method, you can see Advanced GridFTP settings by doing the following: Click on View Transfer Options. The above settings can be modified for your transfer. Please note: if you don t know what these settings are or what they do, use the default settings.

18 Storage Gateway User Help Manual 18 TRANSFER FILES NOW Once the previous step is complete, you can proceed with the actual file transfer by clicking the TRANSFER FILES NOW button. You will see the page below, updating you on the progress of your transfer. At the end of the transfer either Done will be displayed or any errors will shown. TRANSFER FILES LATER If you click the TRANSFER FILES LATER button, you will be shown a calendar where you can select the date and time for this transfer to occur. To schedule a transfer: Select a date and time to transfer, by clicking on the calendar. Click the SCHEDULE button.

19 Storage Gateway User Help Manual 19 You will then be shown the record of the transfer. Troubleshooting If your file transfer fails, determine if any of the following situations have occurred: Problem: Not having files linked properly. Solution: Contact your Beamline Scientist. Problem: Your home institution s storage is full. Solution: Contact your institution s grid or storage administrator. Problem: Your transfer is interrupted. Solution: Start your transfer again.

20 Storage Gateway User Help Manual 20 Downloading Files to Your Computer SSL Transfer SSL Transfer is a relatively low speed web-based HTTPS download. SSL Transfer gives end-to-end encryption to give authentication and confidentiality on transfers. It acts the same as downloading a file from a web site. Using SSL Transfer, you can transfer files from the Australian Synchrotron storage to your desktop. The speed of the transfer is limited to X or the speed of the internet connection to your computer, whichever is the lowest. Unlike the other transfer methods you do not need to have an institutional storage solution, simple enough space on your computer to hold the files you wish to transfer. To proceed with this method of transfer: Select your experiment from the drop down list. Select SSL Transfer as your transfer method. Click Select. You will then see: This page provides many ways you can download your files as have been outlined previously. You need to select which files you wish to download to your computer: Double click a file on the left directory tree and this will download the file, Use the >> button to move all files to the right screen and then use the Download Selection button, Click the Download All button to download all files, or

21 Storage Gateway User Help Manual 21 Drag a file or directory from the left directory tree to the right directory tree and then click the Download Selection button. To expand and collapse directories: Use the Expand All and Collapse All buttons below the directory trees. Please note: When you download more than one file the tar function will be run on them, essentially creating a single file to download. (Like creating a.zip file.) When you have downloaded your files, untar them to separate the file. If you do not have the programs to do this, free utilities to do this are available to download from the internet for Windows, Mac and Linux.