Programme SURF Research Boot Camp 21 April 2016 Registration at

Transcription

1 Programme SURF Research Boot Camp 21 April 2016 Registration at h PROGRAMMING DATA MANAGEMENT COMPUTE MISCELLANEOUS Introduction to UNIX Jeroen Engelberts SURFsara Data cleaning with OpenRefine Mateusz Kuzak Netherlands escience Center Introduction to compute infrastructures Jan Bot SURFsara Good practices for research data sharing and transfer Niek Bosch SURFsara, Paul van Dijk SURFnet BIG DATA part 1 Scalable data analysis with Apache Spark and Hadoop Mathijs Kattenberg, Jeroen Schot & Machiel Jansen SURFsara 12:30 14:30 h PROGRAMMING DATA MANAGEMENT part 1 COMPUTE MISCELLANEOUS Python crash course Ad Thiers & Maurice Verheesen AT Computing PID and irods web tools to manage and share research data Ton Smeele Utrecht University Christine Staiger SURFsara High performance computing in the cloud Markus van Dijk & Ander Astudillo SURFsara Local and remote data visualisation Paul Melis & Casper van Leeuwen SURFsara BIG DATA part 2 see BIG DATA part 1 15:00 17:00 h PROGRAMMING DATA MANAGEMENT part 2 COMPUTE MISCELLANEOUS Scientific computing with Python Ad Thiers & Maurice Verheesen AT Computing PID and irods command line tools to securely store and manage research data Ton Smeele Utrecht University Christine Staiger SURFsara Cluster computing Jeroen Engelberts SURFsara Version control with Git Carlos Martinez Ortiz Netherlands escience Center BIG DATA part 3 see BIG DATA part 1

2 PROGRAMMING TRACK # Introduction to UNIX Time: h Jeroen Engelberts SURFsara In this part of the Programming track, you will learn the history of UNIX and get acquainted with some basic commands you will need to start using data, compute, and network facilities. work with a UNIX terminal/shell use some basic UNIX commands login to a UNIX cluster follow the Introduction to Cluster Computing and Introduction to HPC Cloud from the Compute track. Some basic knowledge working with a computer. Bring along your own laptop. Make sure that you have installed Firefox on your laptop. Please install the FireSSH plugin, which can be obtained free of charge here: US/firefox/addon/firessh/ You will receive a username and password to login to a UNIX system. # Python crash course Time: 12:30 14:30 h Ad Thiers & Maurice Verheesen AT Computing In this session you will learn the basics of the Python programming language. In last couple of years, Python received much attention, specifically in the realm of technical computing. Its benefits superb integration with existing (C, C++ and Fortran) code, and a very simple, but powerful syntax obviously contribute to this popularity. This workshop will cover basic language constructs like loops, if then else statements and exceptions, and it will briefly deal with data types.

3 read and understand existing Python code create a simple Python script determine whether Python should be part of their 'research toolbox' This workshop assumes some programming / scripting familiarity. Bring your own laptop. A basic Python installation is required. This can be downloaded from A more complete installation for scientific computing with Python is Anaconda available from Be sure to install the Python 3 versions. # Scientific computing with Python Time: 15:00 17:00 h Ad Thiers & Maurice Verheesen AT Computing You will be introduced into the world of scientific Python. Plain Python due to its versatility is not well suited for number crunching. There is, however, a wealth of Python software available modules and packages allowing you to perform computationally intensive jobs. All these packages are built around NumPy and matplotlib. Their basic functionality working with n dimensional arrays and visualizing data will be covered in this workshop to some level. You will also get an overview of the major modules in SciPy an enormous toolkit for scientific calculations. At the end of the track, you are be able to: create a simple NumPy script and manipulate arrays create plots with matplotlib find their way in the SciPy / NumPy toolkit This workshop assumes a basic knowledge of the Python programming language Bring your own laptop. An installation for scientific computing with Python, with NumPy, SciPy and MatPlotLib is required. The Anaconda distribution from contains all prerequisites (and more). Be sure to install the Python 3 versions.

4 DATA MANAGEMENT TRACK # Data Cleaning with OpenRefine Time: h Mateusz Kuzak Netherlands escience Center OpenRefine is a powerful tool for working with messy data e.g. to: clean data transform data from one format into another extending it with web services and external data With OpenRefine you will get a better picture of your dataset. You will learn how to use faceting, clustering and filtering features and to correct errors in a dataset. You will also learn how to find and remove whitespace errors and how to split columns. In addition, you will learn how to move forward and backward on the timeline of changes you applied to the data set and how to script changes for future reuse. At the end, you will export a clean dataset to a new file. See also: for introduction videos on OpenRefine import tabular data to OpenRefine find and correct errors in the dataset find and clean whitespace errors for whole columns script cleaning steps This workshop assumes only familiarity with tabular data formats, like comma or tab separated. Bring your own laptop. You will need Firefox Web Browser installed and the OpenRefine browser plugin. Here is the guide to install OpenRefine: Instructions # Data management part 1: PID and irods web tools to manage and share research data Time: 12:30 14:30 h Ton Smeele Utrecht University Christine Staiger SURFsara

5 The workshop explains the European common data infrastructure EUDAT services including persistent identifiers. You will use web tools such as B2FIND to discover research data sets, B2SHARE to publish your own research data and the Handle/EPIC websites to inspect persistent identifiers. In addition, you securely store and manage your data using web tools to work with an irods data grid. At the end of the workshop, you will be able to: understand the common EUDAT B2 suite data services know how to use web tools to find and share research data sets across Europe know how to use web tools to store and manage research data understand the concept of persistent identifiers for research data and know how to use them This is an introductory workshop open to all disciplines. No specific software has to be installed. Bring your own laptop. # Data management part 2: PID and irods command line tools to securely store and manage research data Time: 15:00 17:00 h Ton Smeele Utrecht University Christine Staiger SURFsara During the workshop you create persistent identifiers for your data. Also the workshop introduces the key functions of a data grid and allows you to experiment with an existing irods grid: save files to the grid and retrieve them again, use advanced search techniques to find your files based on their metadata context, build a pipeline to automate (post)processing of data files. At the end of the workshop, you will be able to: understand how to create EPIC persistent identifiers using Python programs understand the benefits of using data grids for storing research data understand the architecture of data grids know the basic set of commands to interact with a data grid know how to automate pipelines using a data grid This workshop assumes some basic familiarity with terminal command line (Linux or DOS).

6 It also assumes the participant is familiar with general concept of persistent identifiers such as DOI and EPIC (these concepts are introduced in the workshop Data Management Part 1: PID and irods web tools to manage and share research data ). While knowledge of Python programming language is an advantage it is not a prerequisite. Bring your own laptop. SSH/Putty tools is required in order to access the data grid server. o Linux and Mac users don t have to install anything; an SSH client is installed. o Windows users: download and install Putty or SSH US/firefox/addon/firessh/ COMPUTE TRACK # Introduction to compute infrastructures Time: h Jan Bot SURFsara We will provide you with a basic understanding of the different compute infrastructures that are available and whether you should consider using one. This module is a prerequisite for the cluster and cloud compute hands ons. We will explain in which situations these infrastructures are useful and provide you with enough background knowledge to decide which infrastructure is best suited for your research. choose between the different computational infrastructures have a basic understanding of how cluster computing systems work have a basic understanding of how cloud computing works None None

7 # High performance computing in the cloud Time: 12:30 14:30 h Ander Astudillo & Markus van Dijk SURFsara Computing in the cloud allows you flexible and easy access to computing and data resources that you would otherwise have to host yourself. SURFsara runs the HPC Cloud providing an Infrastructure as a Service (IaaS) model (as will be explained in the workshop Introduction to High Performance Computing ). This workshop provides a general introduction to cloud computing, teaches HPC Cloud characteristics and how to use it hands on. At the end of the workshop, your are able to: use the HPC Cloud understand and apply different scaling models for parallel computing build (clusters of) Virtual Machines This workshop assumes familiarity with the Unix command line and SSH (can be learned from Introduction to UNIX, in the first hour of the Programming track). Bring your own laptop with a browser (Chrome or Firefox will do fine) and a SSH client: o Linux and Mac users don t have to install anything; an SSH client is installed. o Windows users: download and install git for windows : for windows.github.io # Cluster computing Time: 15:00 17:00 h Jeroen Engelberts SURFsara In this part of the Compute track, you will learn how the national cluster Lisa and the national supercomputer Cartesius are setup. This presentation will be followed up by a hands on with some small and easy to follow examples on both systems. login to a UNIX cluster prepare, submit and analyze a batch job on the national cluster Lisa / supercomputer Cartesius

8 Some basic knowledge of the UNIX operating system (can be learned from Introduction to UNIX, in the first hour of the Programming track). Bring your laptop. Make sure that you have installed Firefox on your laptop. Please install the FireSSH plugin, which can be obtained free of charge here: US/firefox/addon/firessh/ MISCELLANEOUS TRACK # Good practices for research data sharing and transfer Time : h Niek Bosch SURFsara Paul van DIjk SURFnet Description : You will obtain some basic knowledge of tooling and protocols that will help you to share your data fast, secure and easy! Transferring data to colleagues world wide can sometimes be a real hassle. Whether you are dealing with portable hard disks or transfers via the internet, slow transfer times often occur when using suboptimal protocols or solutions. By following some simple guides and tricks, these problems soon belong to the past. We will also touch topics like legal requirements, encryption of data and the prevention of research data disasters. At the end of the track, you are able to: assess which tools meet with legal guidelines choose the most suitable file systems and protocols to transfer data select suitable data sharing tools and services and learn to know their cons and pros use basic encryption tools like PGP None Software needed: No specific software has to be installed. Bring your own laptop.

9 # Local and remote data visualisation Time: 12:30 14:30 h Paul Melis & Casper van Leeuwen SURFsara Visualisation can play an important role in research, but also in communicating research and results to stakeholders. We will give a practical introduction to both scientific visualisation and information visualisation using open source tools (ParaView, Jupyter, matplotlib). Have an overview of different visualisation domains Create basic scientific visualisations using ParaView Perform basic data visualisation with matplotlib and Jupyter No specific prior knowledge is needed. Bring your own laptop, with a web browser (preferably Chrome). ParaView (version 5.0) needs to installed in advance to work on the exercises. It is available for Windows, MacOS X and Linux. See: # Version control with Git Time: 15:00 17:00 h Carlos Martinez Ortiz Netherlands escience Center Version control is extremely useful to manage collaborative work. Nothing committed to version control (git) will ever be lost, it is always possible to go back in time to see exactly who wrote what on a particular day. When several people collaborate in the same project, git automatically notifies users in case of a conflict between changes made by two people. Lone researchers can also benefit immensely of keeping a record of what was changed, when, and why. Git is extremely useful for all researchers if they ever need to come back to the project later on (e.g., a year later, when memory has faded). set up git for tracking changes in a project use git for working in parallel on the same set of files

10 go back to previous versions of a document. Familiarity with working in the command line (either in Windows, OS X or Linux) is recommended. Installation of git client will be required: scm.com/downloads or for windows.github.io/ Creation of a github account would also be advisable. BIG DATA TRACK # Scalable data analysis with Apache Spark and Hadoop Time: :00 h Mathijs Kattenberg, Jeroen Schot & Machiel Jansen SURFsara You are introduced to the Apache Hadoop and Spark frameworks for processing big data. These frameworks offer a novel way for creating data analysis applications that easily scale over hundreds to thousands of machines. This data parallel approach has been pioneered in industry by tech companies such as Google and Facebook, and is very applicable to many scientific workloads in general. We introduce you to the key concepts and features of the Apache Hadoop and Spark stacks. In addition, you will work on hands on Spark exercises in a Jupyter notebook environment. The presentations, exercises and demos will provide a basic understanding of Hadoop and Spark and teach you about fundamental concepts in big data processing. Understand Spark and Hadoop concepts and fundamentals Understand requirements for scalable applications Run and create basic Spark code in a notebook environment This workshop is for anyone who would like to get started with Apache Spark and Hadoop to build robust and scalable applications. You should be familiar with the basics of programming (preferably Python) and the Unix command line. Most scientific programmers and technically minded researchers will feel right at home. No specific software has to be installed. Bring your own laptop.