Data management on HPC platforms

Transcription

1 Data management on HPC platforms Transferring data and handling code with Git scitas.epfl.ch September 10,

2 What kind of data Categorizing data to define a strategy Based on size? Based on format? Based on purpose? 1 / 22

3 What kind of data Categorizing data to define a strategy Based on size? some kilo, multiple mega, a tera,... Based on format? Based on purpose? 1 / 22

4 What kind of data Categorizing data to define a strategy Based on size? some kilo, multiple mega, a tera,... Based on format? binary/ascii, etc... Based on purpose? 1 / 22

5 What kind of data Categorizing data to define a strategy Based on size? some kilo, multiple mega, a tera,... Based on format? binary/ascii, etc... Based on purpose? code/input data/output data, etc... 1 / 22

6 Show me your data, I tell you what you do Different types of data considered Simulation input Simulation output Processed results Simulations code, pre/post processing scripts,... 2 / 22

7 Show me your data, I tell you what you do Different types of data considered Simulation input Simulation output Processed results Simulations code, pre/post processing scripts,... Type of tools Big data type of tools Versioning type of tools 2 / 22

8 Clusters folder structure /home User configurations, codes, input files, submission scripts /scratch Output files from running jobs /work Long term output files storage /tmp Node local space if needed 3 / 22

9 Clusters folder structure /home User configurations, codes, input files, submission scripts /scratch Output files from running jobs /work Long term output files storage /tmp Node local space if needed home visibly all clusters / all nodes size global 100TB, quota 100GB per user location /home/<username> backup backup on tapes, snapshots easy access $HOME 3 / 22

10 Clusters folder structure /home User configurations, codes, input files, submission scripts /scratch Output files from running jobs /work Long term output files storage /tmp Node local space if needed scratch visibly per cluster / all nodes size aries 87TB, bellatrix 200TB, castor 22TB, deneb 350TB location /scratch/<group>/<username> backup no backup, no snapshots, data removed if needed easy access $SCRATCH in jobs 3 / 22

11 Clusters folder structure /home User configurations, codes, input files, submission scripts /scratch Output files from running jobs /work Long term output files storage /tmp Node local space if needed work visibly all cluster / all nodes size global 100TB, quota 50GB per group (+1TB/300.- for 3years) location /work/<group> backup backup on demand (to pay), snapshots easy access $WORK in jobs 3 / 22

12 Clusters folder structure /home User configurations, codes, input files, submission scripts /scratch Output files from running jobs tmp /work Long term output files storage /tmp Node local space if needed visibly per node size node dependent location /tmp/${slurm JOB ID} only during job backup no backup, no snapshots, removed at job end easy access $TMPDIR in jobs 3 / 22

13 Exercise 1: Simple connection Questions: Connect to your favorite front node Check the different folders /home /scratch /work Ok this exercise is just to be sure you can connect to the cluster 4 / 22

14 Big data SSH based file transfer SSH: Secure SHell Different ways: scp secure copy sftp secure file transfer rsync remote synchronization sshfs ssh file system This data should be on your /scratch or /work 5 / 22

15 Move your Big data with scp scp works fine for Linux/MacOS, preinstalled Could be complicated on Windows: pscp.exe from a cmd.exe download.html 6 / 22

16 Exercise 2: Using scp Questions: Copy a file from your machine to the cluster. Retrieve a file from the cluster to your machine 7 / 22

17 Move your Big data with sftp sftp tones of GUIs, for example: Filezilla ( all OSes) Cyberduck (Windows and Mac) Tunnelier (Windows) TUIs also exists like lftp 8 / 22

18 Exercise 3: Using sftp Questions: Try downloading one of these tools and connect to a cluster 9 / 22

19 Move your Big data with rsync sshfs On Linux type file location in your file browser sshfs://<remote>/<path> Note: seams to have ways to be used on MacOS or Windows rsync Synchronizes two folders, folders could be remote Could/should be used instead of mv GUIs exist for all OSes 10 / 22

20 Exercise 4: Using rsync Questions: Create a temporary tmp/ folder in your home folder Copy the folder /ssoft/sources/cmake to this tmp/ folder Create a backup/ folder in the tmp/ folder Use rsync to copy the cmake folder in your tmp/backup folder Modify the README file in the code of cmake in tmp/cmake Re-synchronize the files 11 / 22

21 Versionning What version control means Keep track of the changes Merge changes from multiple sources Strategies Local version control (RCS) Remote on a central server (CVS, Subversion) Distributed version control (Git, Mercurial, Bazaar) 12 / 22

22 Versionning : with Git Git: the stupid content tracker Distributed revision control Originally developed by Linus Torvald Named after the egotistical bastard Linus Remote server File versions DB Version 3 Version 2 Version 1 Computer 1 File File versions DB Version 3 Version 2 Version 1 Computer 2 File File versions DB Version 3 Version 2 Version 1 13 / 22

23 Git basics working copy staging space.git directory remote server 14 / 22

24 Git basics: clone repository working copy staging space.git directory remote server clone Command git clone <repo url> 14 / 22

25 Git basics: add file working copy staging space.git directory remote server add Command git add <files> / 22

26 Git basics: commit modifications working copy staging space.git directory remote server commit Command git commit -m "comment" 14 / 22

27 Git basics: pull modifications working copy staging space.git directory remote server pull Command git pull 14 / 22

28 Git basics: push to server working copy staging space.git directory remote server push Command git push 14 / 22

29 Git basics working copy staging space.git directory remote server clone add commit pull push 14 / 22

30 Exercise 5: First step with Git Questions: If you do not have git installed, get it from Clone the repository Create a file, use your username in the name of the file Check the state of your working copy Add the file to the repository Commit your modifications Pull the potential modifications from the server Push your changes to the server 15 / 22

31 Resolve conflicts working copy staging space.git directory remote server modify file text 16 / 22

32 Resolve conflicts working copy staging space.git directory remote server add commit 16 / 22

33 Resolve conflicts working copy staging space.git directory remote server pull remote modifications on text 16 / 22

34 Resolve conflicts working copy staging space.git directory remote server Correct conflit 16 / 22

35 Resolve conflicts working copy staging space.git directory remote server commit -a 16 / 22

36 Resolve conflicts working copy staging space.git directory remote server push 16 / 22

37 Resolve conflicts working copy staging space.git directory remote server modify file text add commit pull remote modifications on text Correct conflit commit -a push 16 / 22

38 Exercise 6: Generate and solve conflicts Questions: Clone the scitas-test repository in a different folder Modify the file created in the previous exercise in both clones Commit this both modifications Pull and push in one of the clone Pull in the second clone, You should get a conflict <<<<<<<<<< One version ========== Other version >>>>>>>>>> Check the local status Correct the conflict and commit using git commit -a Push the modifications 17 / 22

39 Multiple servers for one project Remote server 1 File versions DB Version 3 Version 2 Version 1 18 / 22

40 Multiple servers for one project Remote server 1 File versions DB Version 3 Version 2 Version 1 Computer Command On Computer: git clone <remote url 1> File File versions DB Version 3 Version 2 Version 1 18 / 22

41 Multiple servers for one project Remote server 1 File versions DB Version 3 Remote server 2 File versions DB Version 2 Version 1 origin Computer File File versions DB Version 3 Version 2 Version 1 Command On Remote server 2: git init --bare 18 / 22

42 Multiple servers for one project Remote server 1 File versions DB Version 3 Remote server 2 File versions DB Version 2 Version 1 origin Computer File server2 Command On Computer: git remote server2 \ <remote url 2> File versions DB Version 3 Version 2 Version 1 18 / 22

43 Multiple servers for one project Remote server 1 File versions DB Version 3 Version 2 Version 1 Remote server 2 File versions DB Version 3 Version 2 Version 1 origin server2 Command Computer File File versions DB Version 3 Version 2 Version 1 On Computer: git push server2 18 / 22

44 Exercise 7: Handle remotes Questions: Connect on the front node of your favorite cluster Create a new folder that will contain your server In this folder initialize a new git server In one of the former clone of scitas-test add the new remote URL <cluster name>:<path to repo> List the remotes to see if everything looks correct Push the local content to the new server On the cluster clone this new server URL <path to repo> Note: The access permission on this new server are based on the file system permissions 19 / 22

45 Branching - Merging version 1 20 / 22

46 Branching - Merging Command git commit... version 2 version 1 20 / 22

47 Branching - Merging version 3 Command git branch branch name git checkout branch name or git checkout -b branch name version 2 version 1 20 / 22

48 Branching - Merging version 3 version 3 Command git checkout master git commit... version 2 version 1 20 / 22

49 Branching - Merging version 4 Command git commit... version 3 version 3 version 2 version 1 20 / 22

50 Branching - Merging version 4 version 3 version 4 version 3 Command git checkout branch name git commit... git checkout branch name git commit... version 2 version 1 20 / 22

51 Branching - Merging version 4 merge 1 version 4 Command git merge master version 3 version 3 version 2 version 1 20 / 22

52 Branching - Merging version 5 version 4 merge 1 version 4 Command git commit... version 3 version 3 version 2 version 1 20 / 22

53 Branching - Merging version 5 version 5 version 4 version 3 merge 1 version 4 version 3 Command git checkout master git commit... version 2 version 1 20 / 22

54 Branching - Merging merge 2 version 5 version 5 version 4 merge 1 version 4 Command git merge branch name version 3 version 3 version 2 version 1 20 / 22

55 Branching - Merging version 6 merge 2 version 5 version 5 version 4 merge 1 version 4 Command git commit... version 3 version 3 version 2 version 1 20 / 22

56 Exercise 8: Handle remotes Questions: Create a branch Modify a file and commit the changes Checkout the master branch Modify a file and commit the changes Merge the branch previously created in the master branch List all branches Print the logs of the different modifications Delete the merged branch 21 / 22

57 Sources and extra infos Sources Wikipedia Manpages: rsync, git Learn more Optimizing your research data management Git with a game: Handling big files with git: 22 / 22