BioHPC Cloud Storage. portal.biohpc.swmed.edu Updated for

Transcription

1 BioHPC Cloud Storage [web] [ ] portal.biohpc.swmed.edu 1 Updated for

2 Overview What is the cloud service/storage? Services we provide to BioHPC user? BioHPC cluster BioHPC Lamella Storage System BioHPC File Exchange How can I use BioHPC cloud storage? Basic File Management File Synchronization File Transfer File Sharing BioHPC manager account 2/32

3 Cloud Service ( Cloud Computing and Cloud Storage ) Cloud computing refers to the delivery of computing resources over the network. Cloud storage (one of the fundamental components of cloud computing) is a service model in which data is maintained, managed and backed up remotely and made available to users over a network (typically the Internet). Image retrieved from 3/32

4 How to Evaluate a Cloud Storage Service Security Reliability Speed Ease-of-use Storage space Image retrieved from 4/32

5 7 Best Cloud Storage Service 2015 Ranking retrieved from by Martyn Casserly, 09 February 15 5/32

6 Why We Need BioHPC Cloud Service Security UTSouthwestern General Cloud Policy: UTSW information must never be stored in unapproved online file storage systems. Storage space Dropbox: 2~18 GB (5 $75/month) Google Drive: 15 GB (30 $299.99/month) BioHPC Cloud: 150GB + access to BioHPC storage system (~ 1 PBs) Other Limitations of Web-based Storage System Max upload/download speed ~ 10 MB/s Web browser do not support folder uploading Upload/download limit for single file (size) 6/32

7 BioHPC Cloud Storage As a BioHPC user, you will have access to: BioHPC Cluster /home2/username /work/department/group/username /project/department/group/username 50 GB quota per user 5 TB quota per group 5 TB quota per group BioHPC File Exchange (web-interface) 50 GB quota per user BioHPC Lamella Cloud Storage (web-interface), on campus only, private cloud GB quota per user Gateway to BioHPC Cluster (via FTP, SAMBA or WebDAV) 7/32

8 Initial Setup on Lamella * Mount External Storage Setting up Lamella to access BioHPC cluster (home, project, and work space) For home leave blank For private project space: department/lab/user For lab shared project space: department/lab/shared BioHPC Endosome/Lysosome lysosome username password home project 8/32

9 What can we do with BioHPC cloud storage system Basic File Management view, move, rename, download, upload, delete your files and etc. File Synchronization owncloud client File Transfer Cluster Samba FTP WebDAV (not recommended) File Sharing Shared Folder BioHPC File Exchange ( BioHPC Lamella Cloud Storage ( BioHPC manager account A lab manager can share data to non-biohpc users 9/32

10 Basic File Management Navigation bar: Allows navigation between different parts New/Upload button: Allows you to create new files or upload existing ones from your device you can also drop files from Explorer or Finder onto the file view. With New button, you can create a text file, folder or download a file from the provided URL. Search/Settings: Search allows you to look for files and directories New/Upload button: Clicking on small thumbnails or icons of the files/directory Select multiple files can e deleted or downloaded as ZIP Viewing Files Can display text files, PDFs, and image files If displaying of a file is not possible, a download process will start Moving files Dragging and dropping files/directory 9/32

11 File Synchronization Lamella Cloud Storage > 100 GB < 100 GB BioHPC Cluster (work or project folder) 11/32

12 File Synchronization * File Synchronization Rule of thumb : Your data size should always smaller than your container Before setting up file synchronization: 1) Estimate the total size of your data if the total size of your data is always < 100GB, you may direct synchronize your data to lamella, otherwise use your BioHPC project directory. 2) Estimate the total free space on your local disk if the total size of your data is always < free space on your local disk, you may synchronize everything from lamella, otherwise chose what to sync. 12/32

13 File Synchronization * File Synchronization download and install owncloud client from connect to lamella.biohpc.swmed.edu -> username/password 13/32

14 File Synchronization * File Synchronization Chose What to Sync total data size < local disk space total data size > local disk space Total data size = actual file stored in lamella locally (up to 100 GB) + mounted folders (up to 10 TB) Anything your deleted from your PC will be removed from BioHPC cloud storage server as well. 14/32

15 File Transfer Lamella Cloud Storage Lysosome 15/32

16 File Transfer Accessing BioHPC Storage Directly from thin client, workstation If your are using a BioHPC thin client or workstation you have direct access to your BioHPC storage. Separate training session July 14th. 16/32

17 File Transfer Accessing BioHPC Storage Directly via SSH Download and install PuTTY from ontent/software/ Host Name: nucleus.biohpc.swmed.edu Port: 22 17/32

18 File Transfer Accessing BioHPC Storage Directly from Windows via Samba Computer -> Map Network Drive Folder is: \\lamella.biohpc.swmed.edu\username home \\lamella.biohpc.swmed.edu\project\department\group project \\lamella.biohpc.swmed.edu\work\department\group work Check Connect using different credentials Enter your BioHPC username and password when prompted. 18/32

19 File Transfer Accessing BioHPC Storage Directly from Mac OSX via Samba Finder -> Go -> Connect to Server Folder is: smb://lamella.biohpc.swmed.edu/username home smb://lamella.biohpc.swmed.edu/project/department/group project smb://lamella.biohpc.swmed.edu/work/department/group work Enter your BioHPC username and password when prompted. 19/32

20 File Transfer Transfer data to/from BioHPC cloud storage via FTP Host: Username: Password: Port: lamella.biohpc.swmed.edu Your BioHPC username Your BioHPC password Empty 20/32

21 File Transfer Accessing your Files from Windows via WebDAV Not recommended but still useful when you need direct access to the local storage of BioHPC Lamella or File Exchange system from your PC 1) Increase the maximum size of WebDav downloads to 4GB By default Windows limits the maximum size of WebDav downloads to ~50 MB regedit -> HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\WebClient\Parameters\FileSizelimitinBytes 21/32

22 File Transfer Accessing your Files from Windows via WebDAV 2) Mount your lamella storage to PC Start -> Computer -> Map Network Drive * You could mount BioHPC File Exchange storage to your PC in a similar way. (https//cloud.biohpc.swmed.edu/remote.php/webdav) 22/32

23 File Transfer Accessing your Files from Mac OSX via WebDAV Cyberduck A Client support FTP, WebDAV for both Mac and Windows Download and install Cyberduck from Once installed into your 'Applications' folder, start Cyberduck, choose 'Open Connection' and configure as in the image below: 23/32

24 File Transfer Uploading / Downloading Speed Lamella Cloud Storage (Samba) BioHPC File-Exchange (webdav) Lamella Cloud Storage (webdav) Lamella Cloud Storage (FTP ) Max Uploading and Downloading Speed* 400MB/s 10 MB /s 100 MB /s 300 MB ~ 1.4 GB/s Upload and download speed depends on your actual network connection We don t set up a limitation on max concurrent sessions 24/32

25 File Sharing If you have a workstation connected to our BioHPC cloud or you know how to use command line to connect to BioHPC cloud, you can share your data on cluster directly. Shared folders had already been set up at different levels Sharing data with all BioHPC user /work/shared /project/shared Sharing data inside your department /work/department_name/shared /project/department_name/shared Sharing data inside your group /work/department_name/group_name/shared /project/department_name/group_name/shared 25/32

26 File Sharing Sharing data with UTSW collaborators BioHPC Lamella Cloud Storage Share with link Share with BioHPC user or group ( notification) 26/32

27 File Sharing Sharing data with non UTSW collaborators BioHPC File Exchange Share with link ( notification, setup password & expiration date) 27/32

28 A lab manager ( BioHPC manager account ) sharing data to non-biohpc users Lab Manager Lab manger account : only provide storage space under /project/department/group/manager_name,and this is the home directory of manager account non-biohpc user A images directly saved to manager s home directory non-biohpc user B non-biohpc user C Lab Facilities, e.g.: scanning EM Lysosome 28/32

29 A lab manager ( BioHPC user ) sharing data to non-biohpc users Lab Manager non-biohpc user A Share data for user to download Lamella Cloud Storage non-biohpc user B non-biohpc user C Lysosome 29/32

30 A lab manager sharing data to non-biohpc users Step 1: mount manager s home directory to your lab facility to save data Support PC, Mac and Linux/Unix 30/32

31 A lab manager sharing data to non-biohpc users Step 2: Sharing data with non BioHPC users with a link password protection (recommended) set up an expiration date (recommended) notification 31/32

32 A lab manager sharing data to non-biohpc users Step 3: non BioHPC users download data by providing correct password 32/32

33 For More Information Read related articles from Lamella - Internal Cloud Storage BioHPC File Exchange - External Cloud Storage the ticket system: biohpc-help@utsouthwestern.edu