Data management on HPC platforms

Save this PDF as:
 WORD  PNG  TXT  JPG

Size: px
Start display at page:

Download "Data management on HPC platforms"

Transcription

1 Data management on HPC platforms Transferring data and handling code with Git scitas.epfl.ch September 10,

2 What kind of data Categorizing data to define a strategy Based on size? Based on format? Based on purpose? 1 / 22

3 What kind of data Categorizing data to define a strategy Based on size? some kilo, multiple mega, a tera,... Based on format? Based on purpose? 1 / 22

4 What kind of data Categorizing data to define a strategy Based on size? some kilo, multiple mega, a tera,... Based on format? binary/ascii, etc... Based on purpose? 1 / 22

5 What kind of data Categorizing data to define a strategy Based on size? some kilo, multiple mega, a tera,... Based on format? binary/ascii, etc... Based on purpose? code/input data/output data, etc... 1 / 22

6 Show me your data, I tell you what you do Different types of data considered Simulation input Simulation output Processed results Simulations code, pre/post processing scripts,... 2 / 22

7 Show me your data, I tell you what you do Different types of data considered Simulation input Simulation output Processed results Simulations code, pre/post processing scripts,... Type of tools Big data type of tools Versioning type of tools 2 / 22

8 Clusters folder structure /home User configurations, codes, input files, submission scripts /scratch Output files from running jobs /work Long term output files storage /tmp Node local space if needed 3 / 22

9 Clusters folder structure /home User configurations, codes, input files, submission scripts /scratch Output files from running jobs /work Long term output files storage /tmp Node local space if needed home visibly all clusters / all nodes size global 100TB, quota 100GB per user location /home/<username> backup backup on tapes, snapshots easy access $HOME 3 / 22

10 Clusters folder structure /home User configurations, codes, input files, submission scripts /scratch Output files from running jobs /work Long term output files storage /tmp Node local space if needed scratch visibly per cluster / all nodes size aries 87TB, bellatrix 200TB, castor 22TB, deneb 350TB location /scratch/<group>/<username> backup no backup, no snapshots, data removed if needed easy access $SCRATCH in jobs 3 / 22

11 Clusters folder structure /home User configurations, codes, input files, submission scripts /scratch Output files from running jobs /work Long term output files storage /tmp Node local space if needed work visibly all cluster / all nodes size global 100TB, quota 50GB per group (+1TB/300.- for 3years) location /work/<group> backup backup on demand (to pay), snapshots easy access $WORK in jobs 3 / 22

12 Clusters folder structure /home User configurations, codes, input files, submission scripts /scratch Output files from running jobs tmp /work Long term output files storage /tmp Node local space if needed visibly per node size node dependent location /tmp/${slurm JOB ID} only during job backup no backup, no snapshots, removed at job end easy access $TMPDIR in jobs 3 / 22

13 Exercise 1: Simple connection Questions: Connect to your favorite front node Check the different folders /home /scratch /work Ok this exercise is just to be sure you can connect to the cluster 4 / 22

14 Big data SSH based file transfer SSH: Secure SHell Different ways: scp secure copy sftp secure file transfer rsync remote synchronization sshfs ssh file system This data should be on your /scratch or /work 5 / 22

15 Move your Big data with scp scp works fine for Linux/MacOS, preinstalled Could be complicated on Windows: pscp.exe from a cmd.exe download.html 6 / 22

16 Exercise 2: Using scp Questions: Copy a file from your machine to the cluster. Retrieve a file from the cluster to your machine 7 / 22

17 Move your Big data with sftp sftp tones of GUIs, for example: Filezilla ( all OSes) Cyberduck (Windows and Mac) Tunnelier (Windows) TUIs also exists like lftp 8 / 22

18 Exercise 3: Using sftp Questions: Try downloading one of these tools and connect to a cluster 9 / 22

19 Move your Big data with rsync sshfs On Linux type file location in your file browser sshfs://<remote>/<path> Note: seams to have ways to be used on MacOS or Windows rsync Synchronizes two folders, folders could be remote Could/should be used instead of mv GUIs exist for all OSes 10 / 22

20 Exercise 4: Using rsync Questions: Create a temporary tmp/ folder in your home folder Copy the folder /ssoft/sources/cmake to this tmp/ folder Create a backup/ folder in the tmp/ folder Use rsync to copy the cmake folder in your tmp/backup folder Modify the README file in the code of cmake in tmp/cmake Re-synchronize the files 11 / 22

21 Versionning What version control means Keep track of the changes Merge changes from multiple sources Strategies Local version control (RCS) Remote on a central server (CVS, Subversion) Distributed version control (Git, Mercurial, Bazaar) 12 / 22

22 Versionning : with Git Git: the stupid content tracker Distributed revision control Originally developed by Linus Torvald Named after the egotistical bastard Linus Remote server File versions DB Version 3 Version 2 Version 1 Computer 1 File File versions DB Version 3 Version 2 Version 1 Computer 2 File File versions DB Version 3 Version 2 Version 1 13 / 22

23 Git basics working copy staging space.git directory remote server 14 / 22

24 Git basics: clone repository working copy staging space.git directory remote server clone Command git clone <repo url> 14 / 22

25 Git basics: add file working copy staging space.git directory remote server add Command git add <files> / 22

26 Git basics: commit modifications working copy staging space.git directory remote server commit Command git commit -m "comment" 14 / 22

27 Git basics: pull modifications working copy staging space.git directory remote server pull Command git pull 14 / 22

28 Git basics: push to server working copy staging space.git directory remote server push Command git push 14 / 22

29 Git basics working copy staging space.git directory remote server clone add commit pull push 14 / 22

30 Exercise 5: First step with Git Questions: If you do not have git installed, get it from Clone the repository Create a file, use your username in the name of the file Check the state of your working copy Add the file to the repository Commit your modifications Pull the potential modifications from the server Push your changes to the server 15 / 22

31 Resolve conflicts working copy staging space.git directory remote server modify file text 16 / 22

32 Resolve conflicts working copy staging space.git directory remote server add commit 16 / 22

33 Resolve conflicts working copy staging space.git directory remote server pull remote modifications on text 16 / 22

34 Resolve conflicts working copy staging space.git directory remote server Correct conflit 16 / 22

35 Resolve conflicts working copy staging space.git directory remote server commit -a 16 / 22

36 Resolve conflicts working copy staging space.git directory remote server push 16 / 22

37 Resolve conflicts working copy staging space.git directory remote server modify file text add commit pull remote modifications on text Correct conflit commit -a push 16 / 22

38 Exercise 6: Generate and solve conflicts Questions: Clone the scitas-test repository in a different folder Modify the file created in the previous exercise in both clones Commit this both modifications Pull and push in one of the clone Pull in the second clone, You should get a conflict <<<<<<<<<< One version ========== Other version >>>>>>>>>> Check the local status Correct the conflict and commit using git commit -a Push the modifications 17 / 22

39 Multiple servers for one project Remote server 1 File versions DB Version 3 Version 2 Version 1 18 / 22

40 Multiple servers for one project Remote server 1 File versions DB Version 3 Version 2 Version 1 Computer Command On Computer: git clone <remote url 1> File File versions DB Version 3 Version 2 Version 1 18 / 22

41 Multiple servers for one project Remote server 1 File versions DB Version 3 Remote server 2 File versions DB Version 2 Version 1 origin Computer File File versions DB Version 3 Version 2 Version 1 Command On Remote server 2: git init --bare 18 / 22

42 Multiple servers for one project Remote server 1 File versions DB Version 3 Remote server 2 File versions DB Version 2 Version 1 origin Computer File server2 Command On Computer: git remote server2 \ <remote url 2> File versions DB Version 3 Version 2 Version 1 18 / 22

43 Multiple servers for one project Remote server 1 File versions DB Version 3 Version 2 Version 1 Remote server 2 File versions DB Version 3 Version 2 Version 1 origin server2 Command Computer File File versions DB Version 3 Version 2 Version 1 On Computer: git push server2 18 / 22

44 Exercise 7: Handle remotes Questions: Connect on the front node of your favorite cluster Create a new folder that will contain your server In this folder initialize a new git server In one of the former clone of scitas-test add the new remote URL <cluster name>:<path to repo> List the remotes to see if everything looks correct Push the local content to the new server On the cluster clone this new server URL <path to repo> Note: The access permission on this new server are based on the file system permissions 19 / 22

45 Branching - Merging version 1 20 / 22

46 Branching - Merging Command git commit... version 2 version 1 20 / 22

47 Branching - Merging version 3 Command git branch branch name git checkout branch name or git checkout -b branch name version 2 version 1 20 / 22

48 Branching - Merging version 3 version 3 Command git checkout master git commit... version 2 version 1 20 / 22

49 Branching - Merging version 4 Command git commit... version 3 version 3 version 2 version 1 20 / 22

50 Branching - Merging version 4 version 3 version 4 version 3 Command git checkout branch name git commit... git checkout branch name git commit... version 2 version 1 20 / 22

51 Branching - Merging version 4 merge 1 version 4 Command git merge master version 3 version 3 version 2 version 1 20 / 22

52 Branching - Merging version 5 version 4 merge 1 version 4 Command git commit... version 3 version 3 version 2 version 1 20 / 22

53 Branching - Merging version 5 version 5 version 4 version 3 merge 1 version 4 version 3 Command git checkout master git commit... version 2 version 1 20 / 22

54 Branching - Merging merge 2 version 5 version 5 version 4 merge 1 version 4 Command git merge branch name version 3 version 3 version 2 version 1 20 / 22

55 Branching - Merging version 6 merge 2 version 5 version 5 version 4 merge 1 version 4 Command git commit... version 3 version 3 version 2 version 1 20 / 22

56 Exercise 8: Handle remotes Questions: Create a branch Modify a file and commit the changes Checkout the master branch Modify a file and commit the changes Merge the branch previously created in the master branch List all branches Print the logs of the different modifications Delete the merged branch 21 / 22

57 Sources and extra infos Sources Wikipedia Manpages: rsync, git Learn more Optimizing your research data management Git with a game: Handling big files with git: 22 / 22

Version Control with Svn, Git and git-svn. Kate Hedstrom ARSC, UAF

Version Control with Svn, Git and git-svn. Kate Hedstrom ARSC, UAF 1 Version Control with Svn, Git and git-svn Kate Hedstrom ARSC, UAF 2 Version Control Software System for managing source files For groups of people working on the same code When you need to get back last

More information

Version Control with Git. Linux Users Group UT Arlington. Rohit Rawat rohitrawat@gmail.com

Version Control with Git. Linux Users Group UT Arlington. Rohit Rawat rohitrawat@gmail.com Version Control with Git Linux Users Group UT Arlington Rohit Rawat rohitrawat@gmail.com Need for Version Control Better than manually storing backups of older versions Easier to keep everyone updated

More information

MATLAB & Git Versioning: The Very Basics

MATLAB & Git Versioning: The Very Basics 1 MATLAB & Git Versioning: The Very Basics basic guide for using git (command line) in the development of MATLAB code (windows) The information for this small guide was taken from the following websites:

More information

Version Control with Git. Kate Hedstrom ARSC, UAF

Version Control with Git. Kate Hedstrom ARSC, UAF 1 Version Control with Git Kate Hedstrom ARSC, UAF Linus Torvalds 3 Version Control Software System for managing source files For groups of people working on the same code When you need to get back last

More information

Version Control using Git and Github. Joseph Rivera

Version Control using Git and Github. Joseph Rivera Version Control using Git and Github Joseph Rivera 1 What is Version Control? Powerful development tool! Management of additions, deletions, and modifications to software/source code or more generally

More information

Version Control! Scenarios, Working with Git!

Version Control! Scenarios, Working with Git! Version Control! Scenarios, Working with Git!! Scenario 1! You finished the assignment at home! VC 2 Scenario 1b! You finished the assignment at home! You get to York to submit and realize you did not

More information

Advanced Computing Tools for Applied Research Chapter 4. Version control

Advanced Computing Tools for Applied Research Chapter 4. Version control Advanced Computing Tools for Applied Research Jaime Boal Martín-Larrauri Rafael Palacios Hielscher Academic year 2014/2015 1 Version control fundamentals 2 What you probably do now Manually save copies

More information

Version Control Tutorial using TortoiseSVN and. TortoiseGit

Version Control Tutorial using TortoiseSVN and. TortoiseGit Version Control Tutorial using TortoiseSVN and TortoiseGit Christopher J. Roy, Associate Professor Virginia Tech, cjroy@vt.edu This tutorial can be found at: www.aoe.vt.edu/people/webpages/cjroy/software-resources/tortoise-svn-git-tutorial.pdf

More information

Version Control with Git. Dylan Nugent

Version Control with Git. Dylan Nugent Version Control with Git Dylan Nugent Agenda What is Version Control? (and why use it?) What is Git? (And why Git?) How Git Works (in theory) Setting up Git (surviving the CLI) The basics of Git (Just

More information

Using Git for Project Management with µvision

Using Git for Project Management with µvision MDK Version 5 Tutorial AN279, Spring 2015, V 1.0 Abstract Teamwork is the basis of many modern microcontroller development projects. Often teams are distributed all over the world and over various time

More information

Version Control with Subversion

Version Control with Subversion Version Control with Subversion Introduction Wouldn t you like to have a time machine? Software developers already have one! it is called version control Version control (aka Revision Control System or

More information

Introduction to Version Control

Introduction to Version Control Research Institute for Symbolic Computation Johannes Kepler University Linz, Austria Winter semester 2014 Outline General Remarks about Version Control 1 General Remarks about Version Control 2 Outline

More information

Version control. with git and GitHub. Karl Broman. Biostatistics & Medical Informatics, UW Madison

Version control. with git and GitHub. Karl Broman. Biostatistics & Medical Informatics, UW Madison Version control with git and GitHub Karl Broman Biostatistics & Medical Informatics, UW Madison kbroman.org github.com/kbroman @kwbroman Course web: kbroman.org/tools4rr Slides prepared with Sam Younkin

More information

Version Control with Git

Version Control with Git Version Control with Git Claudius Coenen License: CC-BY-4.0 What We're Not Talking About Conceived by Linus Torvalds around 2005 Distributed Version Control vs. Central Version Control Why git is better

More information

Version Control with Git

Version Control with Git Version Control with Git Ben Wasserman (benjamin@cmu.edu) 15-441 Computer Networks Recitation 3 1/28 What is version control? Revisit previous code versions Backup projects Work with others Find where

More information

Revision control systems (RCS) and

Revision control systems (RCS) and Revision control systems (RCS) and Subversion Problem area Software projects with multiple developers need to coordinate and synchronize the source code Approaches to version control Work on same computer

More information

Distributed Version Control with Mercurial and git

Distributed Version Control with Mercurial and git OpenStax-CNX module: m37404 1 Distributed Version Control with Mercurial and git Hannes Hirzel This work is produced by OpenStax-CNX and licensed under the Creative Commons Attribution License 3.0 Abstract

More information

Git - Working with Remote Repositories

Git - Working with Remote Repositories Git - Working with Remote Repositories Handout New Concepts Working with remote Git repositories including setting up remote repositories, cloning remote repositories, and keeping local repositories in-sync

More information

Source Control Systems

Source Control Systems Source Control Systems SVN, Git, GitHub SoftUni Team Technical Trainers Software University http://softuni.bg Table of Contents 1. Software Configuration Management (SCM) 2. Version Control Systems: Philosophy

More information

Version Control. Version Control

Version Control. Version Control Version Control CS440 Introduction to Software Engineering 2013, 2015 John Bell Based on slides prepared by Jason Leigh for CS 340 University of Illinois at Chicago Version Control Incredibly important

More information

Git Basics. Christopher Simpkins chris.simpkins@gatech.edu. Chris Simpkins (Georgia Tech) CS 2340 Objects and Design CS 1331 1 / 22

Git Basics. Christopher Simpkins chris.simpkins@gatech.edu. Chris Simpkins (Georgia Tech) CS 2340 Objects and Design CS 1331 1 / 22 Git Basics Christopher Simpkins chris.simpkins@gatech.edu Chris Simpkins (Georgia Tech) CS 2340 Objects and Design CS 1331 1 / 22 Version Control Systems Records changes to files over time Allows you to

More information

An Introduction to Subversion

An Introduction to Subversion An Introduction to Subversion Stéphane Marchesin stephane.marchesin@gmail.com 20 août 2009 Stéphane Marchesin (stephane.marchesin@gmail.com) An Introduction to Subversion 20 août 2009 1 / 26 1 What is

More information

Version Control Systems

Version Control Systems Version Control Systems ESA 2015/2016 Adam Belloum a.s.z.belloum@uva.nl Material Prepared by Eelco Schatborn Today IntroducGon to Version Control Systems Centralized Version Control Systems RCS CVS SVN

More information

Using GitHub for Rally Apps (Mac Version)

Using GitHub for Rally Apps (Mac Version) Using GitHub for Rally Apps (Mac Version) SOURCE DOCUMENT (must have a rallydev.com email address to access and edit) Introduction Rally has a working relationship with GitHub to enable customer collaboration

More information

Version Control with. Ben Morgan

Version Control with. Ben Morgan Version Control with Ben Morgan Developer Workflow Log what we did: Add foo support Edit Sources Add Files Compile and Test Logbook ======= 1. Initial version Logbook ======= 1. Initial version 2. Remove

More information

Introduc)on to Version Control with Git. Pradeep Sivakumar, PhD Sr. Computa5onal Specialist Research Compu5ng, NUIT

Introduc)on to Version Control with Git. Pradeep Sivakumar, PhD Sr. Computa5onal Specialist Research Compu5ng, NUIT Introduc)on to Version Control with Git Pradeep Sivakumar, PhD Sr. Computa5onal Specialist Research Compu5ng, NUIT Contents 1. What is Version Control? 2. Why use Version control? 3. What is Git? 4. Create

More information

Version Control for Computational Economists: An Introduction

Version Control for Computational Economists: An Introduction Version Control for Computational Economists: An Introduction Jake C. Torcasso April 3, 2014 Starting Point A collection of files on your computer Changes to files and new files over time Interested in

More information

Version control. HEAD is the name of the latest revision in the repository. It can be used in subversion rather than the latest revision number.

Version control. HEAD is the name of the latest revision in the repository. It can be used in subversion rather than the latest revision number. Version control Version control is a powerful tool for many kinds of work done over a period of time, including writing papers and theses as well as writing code. This session gives a introduction to a

More information

FEEG6002 - Applied Programming 3 - Version Control and Git II

FEEG6002 - Applied Programming 3 - Version Control and Git II FEEG6002 - Applied Programming 3 - Version Control and Git II Sam Sinayoko 2015-10-16 1 / 26 Outline Learning outcomes Working with a single repository (review) Working with multiple versions of a repository

More information

Annoyances with our current source control Can it get more comfortable? Git Appendix. Git vs Subversion. Andrey Kotlarski 13.XII.

Annoyances with our current source control Can it get more comfortable? Git Appendix. Git vs Subversion. Andrey Kotlarski 13.XII. Git vs Subversion Andrey Kotlarski 13.XII.2011 Outline Annoyances with our current source control Can it get more comfortable? Git Appendix Rant Network traffic Hopefully we have good repository backup

More information

CSE 374 Programming Concepts & Tools. Laura Campbell (Thanks to Hal Perkins) Winter 2014 Lecture 16 Version control and svn

CSE 374 Programming Concepts & Tools. Laura Campbell (Thanks to Hal Perkins) Winter 2014 Lecture 16 Version control and svn CSE 374 Programming Concepts & Tools Laura Campbell (Thanks to Hal Perkins) Winter 2014 Lecture 16 Version control and svn Where we are Learning tools and concepts relevant to multi-file, multi-person,

More information

TortoiseGit with GitHub and TFS. Author: Ian Zhang Date: 2013

TortoiseGit with GitHub and TFS. Author: Ian Zhang Date: 2013 TortoiseGit 101 -- with GitHub and TFS Author: Ian Zhang Date: 2013 1 Background Git is developed by Linus Torvald around 2005 for Linux. Git has strong Unix culture. Most of the power GIT commands are

More information

Version control with GIT

Version control with GIT AGV, IIT Kharagpur September 13, 2012 Outline 1 Version control system What is version control Why version control 2 Introducing GIT What is GIT? 3 Using GIT Using GIT for AGV at IIT KGP Help and Tips

More information

Lab Exercise Part II: Git: A distributed version control system

Lab Exercise Part II: Git: A distributed version control system Lunds tekniska högskola Datavetenskap, Nov 25, 2013 EDA260 Programvaruutveckling i grupp projekt Labb 2 (part II: Git): Labbhandledning Checked on Git versions: 1.8.1.2 Lab Exercise Part II: Git: A distributed

More information

Introduction to Git. Markus Kötter koetter@rrzn.uni-hannover.de. Notes. Leinelab Workshop July 28, 2015

Introduction to Git. Markus Kötter koetter@rrzn.uni-hannover.de. Notes. Leinelab Workshop July 28, 2015 Introduction to Git Markus Kötter koetter@rrzn.uni-hannover.de Leinelab Workshop July 28, 2015 Motivation - Why use version control? Versions in file names: does this look familiar? $ ls file file.2 file.

More information

Derived from Chris Cannam's original at, https://code.soundsoftware.ac.uk/projects/easyhg/wiki/sc2012bootcamppl an.

Derived from Chris Cannam's original at, https://code.soundsoftware.ac.uk/projects/easyhg/wiki/sc2012bootcamppl an. Version Control Key Points ========================== Mike Jackson, The Software Sustainability Institute. This work is licensed under the Creative Commons Attribution License. Copyright (c) Software Carpentry

More information

CS 2112 Lab: Version Control

CS 2112 Lab: Version Control 29 September 1 October, 2014 Version Control What is Version Control? You re emailing your project back and forth with your partner. An hour before the deadline, you and your partner both find different

More information

Subversion. Nadir SOUALEM. Linux Users subversion client svn 1.6.5 or higher. Windows users subversion client Tortoise 1.6.

Subversion. Nadir SOUALEM. Linux Users subversion client svn 1.6.5 or higher. Windows users subversion client Tortoise 1.6. Subversion Nadir SOUALEM 1 Requirements Linux Users subversion client svn 1.6.5 or higher Windows users subversion client Tortoise 1.6.6 or higher 2 What is Subversion? Source control or version control

More information

An Introduction to Mercurial Version Control Software

An Introduction to Mercurial Version Control Software An Introduction to Mercurial Version Control Software CS595, IIT [Doc Updated by H. Zhang] Oct, 2010 Satish Balay balay@mcs.anl.gov Outline Why use version control? Simple example of revisioning Mercurial

More information

UNIVERSITY OF CALIFORNIA Department of Electrical Engineering and Computer Sciences Computer Science Division. P. N. Hilfinger

UNIVERSITY OF CALIFORNIA Department of Electrical Engineering and Computer Sciences Computer Science Division. P. N. Hilfinger UNIVERSITY OF CALIFORNIA Department of Electrical Engineering and Computer Sciences Computer Science Division CS 61B Fall 2012 P. N. Hilfinger Version Control and Homework Submission 1 Introduction Your

More information

CPSC 491. Today: Source code control. Source Code (Version) Control. Exercise: g., no git, subversion, cvs, etc.)

CPSC 491. Today: Source code control. Source Code (Version) Control. Exercise: g., no git, subversion, cvs, etc.) Today: Source code control CPSC 491 Source Code (Version) Control Exercise: 1. Pretend like you don t have a version control system (e. g., no git, subversion, cvs, etc.) 2. How would you manage your source

More information

Version control with Subversion

Version control with Subversion Version control with Subversion Davor Cubranic Grad Seminar October 6, 2011 With searching comes loss And the presence of absence: My Thesis not found. Version Control A tool for managing changes to a

More information

Introducing Xcode Source Control

Introducing Xcode Source Control APPENDIX A Introducing Xcode Source Control What You ll Learn in This Appendix: u The source control features offered in Xcode u The language of source control systems u How to connect to remote Subversion

More information

Setting up a local working copy with SVN, MAMP and rsync. Agentic - 2009

Setting up a local working copy with SVN, MAMP and rsync. Agentic - 2009 Setting up a local working copy with SVN, MAMP and rsync Agentic - 2009 Get MAMP You can download MAMP for MAC at this address : http://www.mamp.info/en/downloads/index.html Install MAMP in your APPLICATION

More information

Version control systems. Lecture 2

Version control systems. Lecture 2 Version control systems Lecture 2 VCS Many people s version- control method of choice is to copy files into another directory (e.g. a @me- stamped directory). But this approach is error prone. Easy to

More information

Software Configuration Management. Slides derived from Dr. Sara Stoecklin s notes and various web sources.

Software Configuration Management. Slides derived from Dr. Sara Stoecklin s notes and various web sources. Software Configuration Management Slides derived from Dr. Sara Stoecklin s notes and various web sources. What is SCM? SCM goals Manage the changes to documents, programs, files, etc. Track history Identify

More information

Continuous Integration

Continuous Integration Continuous Integration Collaborative development issues Checkout of a shared version of software ( mainline ) Creation of personal working copies of developers Software development: modification of personal

More information

Connecting to the School of Computing Servers and Transferring Files

Connecting to the School of Computing Servers and Transferring Files Connecting to the School of Computing Servers and Transferring Files Connecting This document will provide instructions on how to connect to the School of Computing s server. Connect Using a Mac or Linux

More information

The Hitchhiker s Guide to Github: SAS Programming Goes Social Jiangtang Hu d-wise Technologies, Inc., Morrisville, NC

The Hitchhiker s Guide to Github: SAS Programming Goes Social Jiangtang Hu d-wise Technologies, Inc., Morrisville, NC Paper PA-04 The Hitchhiker s Guide to Github: SAS Programming Goes Social Jiangtang Hu d-wise Technologies, Inc., Morrisville, NC ABSTRACT Don't Panic! Github is a fantastic way to host, share, and collaborate

More information

Introduction to the Git Version Control System

Introduction to the Git Version Control System Introduction to the Sebastian Rockel rockel@informatik.uni-hamburg.de University of Hamburg Faculty of Mathematics, Informatics and Natural Sciences Department of Informatics Technical Aspects of Multimodal

More information

Introduction to Version Control with Git

Introduction to Version Control with Git Introduction to Version Control with Git Dark Cosmology Centre Niels Bohr Institute License Most images adapted from Pro Git by Scott Chacon and released under license Creative Commons BY-NC-SA 3.0. See

More information

MATLAB @ Work. MATLAB Source Control Using Git

MATLAB @ Work. MATLAB Source Control Using Git MATLAB @ Work MATLAB Source Control Using Git Richard Johnson Using source control is a key practice for professional programmers. If you have ever broken a program with a lot of editing changes, you can

More information

Version Control Script

Version Control Script Version Control Script Mike Jackson, The Software Sustainability Institute Things you should do are written in bold. Suggested dialog is in normal text. Command- line excerpts and code fragments are in

More information

Mercurial. Why version control (Single users)

Mercurial. Why version control (Single users) Mercurial Author: Hans Fangohr Date: 2008-05-21 Version: 033c85b22987 Id: talk.txt,v 033c85b22987 2008/05/21 08:42:42 fangohr Series: SESA2006 2008, last lecture Hint Adjust font-size

More information

CISC 275: Introduction to Software Engineering. Lab 5: Introduction to Revision Control with. Charlie Greenbacker University of Delaware Fall 2011

CISC 275: Introduction to Software Engineering. Lab 5: Introduction to Revision Control with. Charlie Greenbacker University of Delaware Fall 2011 CISC 275: Introduction to Software Engineering Lab 5: Introduction to Revision Control with Charlie Greenbacker University of Delaware Fall 2011 Overview Revision Control Systems in general Subversion

More information

Fachpraktikum Grafik-Programmierung WS2010 Introduction to Version Control

Fachpraktikum Grafik-Programmierung WS2010 Introduction to Version Control Fachpraktikum Grafik-Programmierung WS2010 Introduction to Version Control Alexandros Panagiotidis, Daniel Kauker Institut für Visualisierung und Interaktive Systeme Universität Stuttgart Advantages of

More information

Introduction to Version Control with Git

Introduction to Version Control with Git Introduction to Version Control with Git Dark Cosmology Centre Niels Bohr Institute License All images adapted from Pro Git by Scott Chacon and released under license Creative Commons BY-NC-SA 3.0. See

More information

Principles of Software Construction: Objects, Design, and Concurrency

Principles of Software Construction: Objects, Design, and Concurrency Principles of Software Construction: Objects, Design, and Concurrency Version Control Jonathan Aldrich Charlie Garrod School of Computer Science Based on slides by Christian Kästner 1 Administrivia Extra

More information

BioHPC Cloud Storage. portal.biohpc.swmed.edu biohpc-help@utsouthwestern.edu. Updated for 2015-06-10

BioHPC Cloud Storage. portal.biohpc.swmed.edu biohpc-help@utsouthwestern.edu. Updated for 2015-06-10 BioHPC Cloud Storage [web] [email] portal.biohpc.swmed.edu biohpc-help@utsouthwestern.edu 1 Updated for 2015-06-10 Overview What is the cloud service/storage? Services we provide to BioHPC user? BioHPC

More information

Version Control with Mercurial and SSH

Version Control with Mercurial and SSH Version Control with Mercurial and SSH Lasse Kliemann lki@informatik.uni-kiel.de Vorkurs Informatik 2010 Wishlist While working on a project, it is nice to... be able to switch back to older versions of

More information

Version Control Using Subversion. Version Control Using Subversion 1 / 27

Version Control Using Subversion. Version Control Using Subversion 1 / 27 Version Control Using Subversion Version Control Using Subversion 1 / 27 What Is Version Control? Version control is also known as revision control. Version control is provided by a version control system

More information

Git, GitHub & Web Hosting Workshop

Git, GitHub & Web Hosting Workshop Git, GitHub & Web Hosting Workshop WTM Hamburg Git, GitHub & Web Hosting Documentation During our Workshops we re going to develop parts of our WTM Hamburg Website together. At this point, we ll continue

More information

Developer Workshop 2015. Marc Dumontier McMaster/OSCAR-EMR

Developer Workshop 2015. Marc Dumontier McMaster/OSCAR-EMR Developer Workshop 2015 Marc Dumontier McMaster/OSCAR-EMR Agenda Code Submission 101 Infrastructure Tools Developing OSCAR Code Submission: Process OSCAR EMR Sourceforge http://www.sourceforge.net/projects/oscarmcmaster

More information

BlueJ Teamwork Tutorial

BlueJ Teamwork Tutorial BlueJ Teamwork Tutorial Version 2.0 for BlueJ Version 2.5.0 (and 2.2.x) Bruce Quig, Davin McCall School of Engineering & IT, Deakin University Contents 1 OVERVIEW... 3 2 SETTING UP A REPOSITORY... 3 3

More information

Manual. CollabNet Subversion Connector to HP Quality Center. Version 1.2

Manual. CollabNet Subversion Connector to HP Quality Center. Version 1.2 Manual CollabNet Subversion Connector to HP Quality Center Version 1.2 A BOUT THE CONNECTOR About the Connector The CollabNet Subversion Connector to HP Quality Center enables Quality Center users to

More information

Source Control Guide: Git

Source Control Guide: Git MadCap Software Source Control Guide: Git Flare 11.1 Copyright 2015 MadCap Software. All rights reserved. Information in this document is subject to change without notice. The software described in this

More information

HPC system startup manual (version 1.30)

HPC system startup manual (version 1.30) HPC system startup manual (version 1.30) Document change log Issue Date Change 1 12/1/2012 New document 2 10/22/2013 Added the information of supported OS 3 10/22/2013 Changed the example 1 for data download

More information

Version Control Systems: SVN and GIT. How do VCS support SW development teams?

Version Control Systems: SVN and GIT. How do VCS support SW development teams? Version Control Systems: SVN and GIT How do VCS support SW development teams? CS 435/535 The College of William and Mary Agile manifesto We are uncovering better ways of developing software by doing it

More information

BACKUP YOUR SENSITIVE DATA WITH BACKUP- MANAGER

BACKUP YOUR SENSITIVE DATA WITH BACKUP- MANAGER Training course 2007 BACKUP YOUR SENSITIVE DATA WITH BACKUP- MANAGER Nicolas FUNKE PS2 ID : 45722 This document represents my internships technical report. I worked for the Biarritz's Town Hall during

More information

Modeling and Simulating Social Systems with MATLAB

Modeling and Simulating Social Systems with MATLAB Modeling and Simulating Social Systems with MATLAB Lecture 3 Git Olivia Woolley, Tobias Kuhn, Dario Biasini, Dirk Helbing Chair of Sociology, in particular of Modeling and Simulation ETH Zürich 2014-03-03

More information

Continuous Integration. CSC 440: Software Engineering Slide #1

Continuous Integration. CSC 440: Software Engineering Slide #1 Continuous Integration CSC 440: Software Engineering Slide #1 Topics 1. Continuous integration 2. Configuration management 3. Types of version control 1. None 2. Lock-Modify-Unlock 3. Copy-Modify-Merge

More information

Two Best Practices for Scientific Computing

Two Best Practices for Scientific Computing Two Best Practices for Scientific Computing Version Control Systems & Automated Code Testing David Love Software Interest Group University of Arizona February 18, 2013 How This Talk Happened Applied alumnus,

More information

Drupalcamp Vienna 2009

Drupalcamp Vienna 2009 Drupalcamp Vienna 2009 Development workflow and deployment at Klaus Purer 2009-11-28 http://klausi.fsinf.at Who am I? Student at the Vienna University of Technology Software Engineering & Internet Computing

More information

Memopol Documentation

Memopol Documentation Memopol Documentation Release 1.0.0 Laurent Peuch, Mindiell, Arnaud Fabre January 26, 2016 Contents 1 User guide 3 1.1 Authentication in the admin backend.................................. 3 1.2 Managing

More information

How to Backup XenServer VM with VirtualIQ

How to Backup XenServer VM with VirtualIQ How to Backup XenServer VM with VirtualIQ 1. Using Live Backup of VM option: Live Backup: This option can be used, if user does not want to power off the VM during the backup operation. This approach takes

More information

Version Uncontrolled! : How to Manage Your Version Control

Version Uncontrolled! : How to Manage Your Version Control Version Uncontrolled! : How to Manage Your Version Control Harold Dost III, Raastech ABSTRACT Are you constantly wondering what is in your production environment? Do you have any doubts about what code

More information

Software configuration management

Software configuration management Software Engineering Theory Software configuration management Lena Buffoni/ Kristian Sandahl Department of Computer and Information Science 2015-09-30 2 Maintenance Requirements System Design (Architecture,

More information

Software Configuration Management AE6382

Software Configuration Management AE6382 Software Configuration Management Software Configuration Management What is it? Software Configuration Management is the process of tracking changes to software. Why use it? Maintain multiple branches

More information

Redmine: A project management software tool. January, 2013

Redmine: A project management software tool. January, 2013 Redmine: A project management software tool January, 2013 Outline Introduction to Redmine. Important concepts of Redmine. How to use Redmine. 1 Introduction: What is Redmine? Redmine is a project management

More information

MOOSE-Based Application Development on GitLab

MOOSE-Based Application Development on GitLab MOOSE-Based Application Development on GitLab MOOSE Team Idaho National Laboratory September 9, 2014 Introduction The intended audience for this talk is developers of INL-hosted, MOOSE-based applications.

More information

COSC345 2013 Software Engineering. Lecture 7: Version Control

COSC345 2013 Software Engineering. Lecture 7: Version Control COSC345 2013 Software Engineering Lecture 7: Version Control Some Problems Communications File system problems Version control Basic principles and use Outline When to use version control Examples SCCS

More information

WinSCP Tutorial 01/28/09: Y. Liow (yliow@ccis.edu)

WinSCP Tutorial 01/28/09: Y. Liow (yliow@ccis.edu) WinSCP Tutorial 01/28/09: (yliow@ccis.edu) What is WinSCP? WinSCP is an open source SFTP client for Windows. Its main function is the secure file transfer between a local and a remote computer. Beyond

More information

Accessing the FTP Server - User Manual

Accessing the FTP Server - User Manual CENTRAL BANK OF CYPRUS Accessing the FTP Server - User Manual IT Department, CENTRAL BANK OF CYPRUS TABLE OF CONTENTS 1 EXECUTIVE SUMMARY... 1 1.1 AUDIENCE... 1 1.2 SCOPE... 1 2 CHANGES FROM THE OLD FTP

More information

The Bazaar Version Control System. Michael Hudson, Canonical Ltd michael.hudson@canonical.com

The Bazaar Version Control System. Michael Hudson, Canonical Ltd michael.hudson@canonical.com The Bazaar Version Control System Michael Hudson, Canonical Ltd michael.hudson@canonical.com What is Bazaar? Bazaar is a Distributed Version Control System (DVCS) You probably know what a VCS is by now:

More information

Hodor and Bran HPC Tutorial File Transfer

Hodor and Bran HPC Tutorial File Transfer Hodor and Bran HPC Tutorial File Transfer UND Computational Research Center Please read the following information for instructions on how to transfer files to Hodor and Bran, the UND HPC Linux clusters.

More information

HADOOP CLUSTER SETUP GUIDE:

HADOOP CLUSTER SETUP GUIDE: HADOOP CLUSTER SETUP GUIDE: Passwordless SSH Sessions: Before we start our installation, we have to ensure that passwordless SSH Login is possible to any of the Linux machines of CS120. In order to do

More information

Software Configuration Management and Continuous Integration

Software Configuration Management and Continuous Integration 1 Chapter 1 Software Configuration Management and Continuous Integration Matthias Molitor, 1856389 Reaching and maintaining a high quality level is essential for each today s software project. To accomplish

More information

Export & Backup Guide

Export & Backup Guide Eport & Backup Guide Welcome to the WebOffice and WorkSpace eport and backup guide. This guide provides an overview and requirements of the tools available to etract data from your WebOffice or WorkSpace

More information

Extending Remote Desktop for Large Installations. Distributed Package Installs

Extending Remote Desktop for Large Installations. Distributed Package Installs Extending Remote Desktop for Large Installations This article describes four ways Remote Desktop can be extended for large installations. The four ways are: Distributed Package Installs, List Sharing,

More information

Source code management systems

Source code management systems Source code management systems SVN, Git, Mercurial, Bazaar,... for managing large projects with multiple people work locally or across a network store and retrieve all versions of all directories and files

More information

CONFIGURING ECLIPSE FOR AWS EMR DEVELOPMENT

CONFIGURING ECLIPSE FOR AWS EMR DEVELOPMENT CONFIGURING ECLIPSE FOR AWS EMR DEVELOPMENT With this post we thought of sharing a tutorial for configuring Eclipse IDE (Intergrated Development Environment) for Amazon AWS EMR scripting and development.

More information

Reflection DBR USER GUIDE. Reflection DBR User Guide. 995 Old Eagle School Road Suite 315 Wayne, PA 19087 USA 610.964.8000 www.evolveip.

Reflection DBR USER GUIDE. Reflection DBR User Guide. 995 Old Eagle School Road Suite 315 Wayne, PA 19087 USA 610.964.8000 www.evolveip. Reflection DBR USER GUIDE 995 Old Eagle School Road Suite 315 Wayne, PA 19087 USA 610.964.8000 www.evolveip.net Page 1 of 1 Table of Contents Overview 3 Reflection DBR Client and Console Installation 4

More information

Using sftp in Informatica PowerCenter

Using sftp in Informatica PowerCenter Using sftp in Informatica PowerCenter Applies to: Informatica PowerCenter Summary This article briefs about how to push/pull files using SFTP program in Informatica PowerCenter. Author Bio Author(s): Sukumar

More information

PARALLELS SERVER 4 BARE METAL README

PARALLELS SERVER 4 BARE METAL README PARALLELS SERVER 4 BARE METAL README This document provides the first-priority information on Parallels Server 4 Bare Metal and supplements the included documentation. TABLE OF CONTENTS 1 About Parallels

More information

1. History 2. Structure 3. Git Comparison 4. File Storage 5. File Tracking 6. Staging 7. Queues (MQ) 8. Merge Tools 9. Interfaces

1. History 2. Structure 3. Git Comparison 4. File Storage 5. File Tracking 6. Staging 7. Queues (MQ) 8. Merge Tools 9. Interfaces 1 Hg 1. History 2. Structure 3. Git Comparison 4. File Storage 5. File Tracking 6. Staging 7. Queues (MQ) 8. Merge Tools 9. Interfaces 2 Mercurial / Git History Bitmover's BitKeeper Proprietary distributed

More information

GENERAL FILE TRANSFER GUIDELINES

GENERAL FILE TRANSFER GUIDELINES GENERAL FILE TRANSFER GUIDELINES EMSTARS extracts (the XML files containing patient care records) will be transferred to the FDOH Secure Server at intervals defined in the EMSTARS Business Rules (available

More information

Miguel A. Figueroa Villanueva Xabriel J. Collazo Mojica. ICOM 5047 Capstone Miguel A. Figueroa Villanueva University of Puerto Rico Mayagüez Campus

Miguel A. Figueroa Villanueva Xabriel J. Collazo Mojica. ICOM 5047 Capstone Miguel A. Figueroa Villanueva University of Puerto Rico Mayagüez Campus Document and Information Management: A Software Developer s Perspective Xabriel J. Collazo Mojica Outline Introduction Why should I (you) care? Document management CMS Wiki Aigaion Code and Document Repositories

More information

Backing Up TestTrack Native Project Databases

Backing Up TestTrack Native Project Databases Backing Up TestTrack Native Project Databases TestTrack projects should be backed up regularly. You can use the TestTrack Native Database Backup Command Line Utility to back up TestTrack 2012 and later

More information

Assignment 1. Setting up your tools. CS102 Programming 1 LIU 1 of 8. 6 September 2012

Assignment 1. Setting up your tools. CS102 Programming 1 LIU 1 of 8. 6 September 2012 CS102 Programming 1 LIU 1 of 8 Assignment 1 6 September 2012 e purpose of this assignment is to help you set up some tools that we will use for assignments the rest of the semester. Due Monday 10 September

More information

SFTP Server User Login Instructions. Open Internet explorer and enter the following url: https://sftp.sae.org

SFTP Server User Login Instructions. Open Internet explorer and enter the following url: https://sftp.sae.org SFTP Server User Login Instructions Open Internet explorer and enter the following url: https://sftp.sae.org You will be prompted for a user id and password as such. Please enter your account id and password.

More information