Using BioHPC Cloud Storage

Transcription

1 Using BioHPC Cloud Storage [web] [ ] portal.biohpc.swmed.edu 1 Updated for

2 Overview What is the cloud service (cloud computing)? What is the cloud storage? What services we provide for BioHPC cloud storage? How can I use BioHPC cloud storage? 2

3 Cloud Service ( Cloud Computing ) Cloud computing refers to the delivery of computing resources over the network. Image retrieved from 3

4 Benefits and Challenges Image retrieved from 4

5 Cloud Service Cloud Providers offer services that can be grouped into three categories: Software as a Service Platform as a Service Infrastructure as a Service Image retrieved from 5

6 Storage as a Service SaaS PaaS IaaS dsaas (data Storage as a Service) Cloud storage (one of the fundamental components of cloud computing) is a service model in which data is maintained, managed and backed up remotely and made available to users over a network (typically the Internet). Image retrieved from 6

7 How to Evaluate a Cloud Storage Service Security Reliability Speed Ease-of-use Storage space 7

8 7 Best Cloud Storage Service 2015 Ranking retrieved from by Martyn Casserly, 09 February 15 8

9 Why We Need BioHPC Cloud Service UTSouthwestern General Cloud Policies UT Southwestern Medical Center information must never be stored in unapproved online file storage systems. Image retrieved from 9

10 Compare with the Best Cloud Storage Services Free /Max* Storage Server-side Encryption Keep Deleted Files 2 GB / 5 TB 256-bit Unlimited 2~18 GB / 1 TB 256-bit 30 days 7 GB / 1 TB 128-bit 30 days 5 GB / 1 TB 256-bit No 15 GB / 30 TB 30 TB: $ / month 256-bit 30 days 5 GB / 1 TB 256-bit No File Exchange Lamella Cloud Storage Lysosome 50 GB / 5 TB 256-bit No 100 GB / 25 TB 256-bit 30 days 50 GB/ 1PB No * The max storage space may various as long as you pay/request for it Data source: 10

11 BioHPC Cloud Storage As a BioHPC user, you will have access to the BioHPC cloud storage: BioHPC Lysosome /home2/username 50 GB quota per user, long term /work/department/group/username 5 TB quota per group, short term /project/department/group/username 5 TB quota per group, long term, by PI request only BioHPC File Exchange (web-interface) 50 GB quota per user BioHPC Lamellar Cloud Storage (web-interface), on campus only GB quota per user 11

12 What can we do with BioHPC cloud storage system Basic File Management: view, move, rename, download, upload, delete your files and etc. More: Sharing your Files Accessing your Files via web interface Accessing your Files via webdav Mount External Storage File Synchronization What is WebDAV? Briefly: WebDAV stands for "Web-based Distributed Authoring and Versioning". It is a set of extensions to the HTTP protocol which allows users to collaboratively edit and manage files on remote web servers. 12

13 Sharing data on Lysosome If you have a workstation connected to our BioHPC cloud or you know how to use command line to connect to BioHPC cloud, you can share your data on Lysosome directly. Shared folders had already been set up at different levels Sharing data with all BioHPC user /work/shared /project/shared Sharing data inside your department /work/department_name/shared /project/department_name/shared Sharing data inside your group /work/department_name/group_name/shared /project/department_name/group_name/shared 13

14 Examples of how to use BioHPC web-based cloud storage system Case 1: Sharing data with non UTSW collaborators -- Sharing your Files -- Accessing your Files via web interface Case 2: Transfer data to/from Lysosome via Lamella -- Accessing your Files via web interface -- Accessing your Files via webdav -- Mount External Storage Case 3: Synchronize your data with Lamella cloud -- Accessing your Files via web interface -- Mount External Storage -- File Synchronization Case 4: A lab manager ( BioHPC user ) sharing data to non-biohpc users -- Sharing your Files -- Accessing your Files via web interface -- Mount External Storage 14

15 15 Case 1: Sharing data with non UTSW collaborators

16 Case 1: Sharing data with non UTSW collaborators BioHPC File Exchange (web-interface) 16

17 Case 1: Sharing data with non UTSW collaborators Navigation bar: Allows navigation between different parts New/Upload button: Allows you to create new files or upload existing ones from your device you can also drop files from Explorer or Finder onto the file view. With New button, you can create a text file, folder or download a file from the provided URL. Search/Settings: Search allows you to look for files and directories.

18 Case 1: Sharing data with non UTSW collaborators Selecting files clicking on small thumbnails or icons of the files/directory select multiple files can be deleted or downloaded as ZIP Viewing files can display text files, PDFs, and image files If displaying of a file is not possible, a download process will start. Moving files dragging and dropping files/directory into a directory. Sharing files Any file/folder can be shared click Share 18

19 Case 2: Transfer data to/from Lysosome via Lamella Lamella Cloud Storage Lysosome 19

20 Case 2: Transfer data to/from Lysosome via Lamella * Mount External Storage Setting up Lamella to access your BioHPC home, project, and work space For home leave blank For private project space: department/lab/user For lab shared project space: department/lab/shared BioHPC Endosome/Lysosome lysosome username password home project * most web browser do not support folder uploading 20

21 Case 2: Transfer data to/from Lysosome via Lamella * Accessing your Files via webdav Mount lamella storage to PC 1) Increase the maximum size of WebDav downloads to 4GB By default Windows limits the maximum size of WebDav downloads to ~50 MB regedit -> HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\WebClient\Parameters\FileSizelimitinBytes 21

22 Case 2: Transfer data to/from Lysosome via Lamella * Accessing your Files via webdav Mount lamella storage to PC 2) Mount your lamella storage to PC Start -> Computer -> Map Network Drive If your are using a BioHPC client or workstation you have direct access to your BioHPC storage and shouldn t try to mount lamella. You could mount file-exchange storage to your PC in a similar way. (URL: https//cloud.biohpc.swmed.edu/remote.php/webdav) 22

23 Case 2: Transfer data to/from Lysosome via Lamella * Accessing your Files via webdav Mount lamella storage to Mac Cyberduck An Alternative Method Download and install Cyberduck from Once installed into your 'Applications' folder, start Cyberduck, choose 'Open Connection' and configure as in the image below: 23

24 Lamella cloud storage upload limitation Upload limit for web-based system ( single file ) 64-bit versions of Google Chrome 64-bit versions of Internet Explorer, Mozilla Firefox 32-bit operating systems or browsers 24

25 Lamella cloud storage upload limitation : solution Transfer data to BioHPC cloud storage with FTP (Filezilla) Host: Username: Password: Port: lysosome.biohpc.swmed.edu Your BioHPC username Your BioHPC password Empty 25

26 Uploading / Downloading Speed Max Uploading and Downloading Speed* Effective Concurrent Sessions BioHPC File-Exchange 10 MB /s 10 Lamella Cloud Storage 100 MB /s 20 FTP (Filezilla) 300 MB ~ 1.4 GB/s 20 Upload and download speed depends on your actual network connection We don t set up a limitation on max concurrent sessions 26

27 Case 3: Synchronize your data with Lamella cloud Lamella Cloud Storage > 100 GB < 100 GB Lysosome 27

28 Case 3: Synchronize your data with Lamella cloud * File Synchronization Rule of thumb : Your data size should always smaller than your container Before setting up file synchronization: 1) Estimate the total size of your data if the total size of your data is always < 100GB, you may direct synchronize your data to lamella, otherwise use your BioHPC project directory. 2) Estimate the total free space on your local disk if the total size of your data is always < free space on your local disk, you may synchronize everything from lamella, otherwise chose what to sync. 28

29 Case 3: Synchronize your data with Lamella cloud * File Synchronization download and install owncloud client from connect to lamella.biohpc.swmed.edu -> username/password 29

30 Case 3: Synchronize your data with Lamella cloud * File Synchronization Chose What to Sync total data size < local disk space total data size > local disk space total data size = actual file stored in lamella (up to 100 GB) + mounted folders (up to 10 TB) 30

31 Case 4: A lab manager ( BioHPC manager account ) sharing data to non-biohpc users Lab Manager Lab manger account : only provide storage space under /project/department/group/manager_name,and this is the home directory of manager account non-biohpc user A images directly saved to manager s home directory non-biohpc user B non-biohpc user C Lab Facilities, e.g.: scanning EM Lysosome 31

32 Case 4: A lab manager ( BioHPC user ) sharing data to non-biohpc users Lab Manager non-biohpc user A Share data for user to download Lamella Cloud Storage non-biohpc user B non-biohpc user C Lysosome 32

33 Case 4: A file manager sharing data to non-biohpc users Step 1: mount manager s home directory to your lab facility to save data Support PC, Mac and Linux/Unix 33

34 Case 4: A file manager sharing data to non-biohpc users Step 2: Sharing data with non BioHPC users with a link password protection (recommended) set up an expiration date (recommended) notification 34

35 Case 4: A file manager sharing data to non-biohpc users Step 3: non BioHPC users download data by providing correct password 35

36 For More Information Read related articles from Lamella - Internal Cloud Storage BioHPC File Exchange - External Cloud Storage the ticket system: biohpc-help@utsouthwestern.edu 36