Cyberinfrastructure Education and Hands-on Training Using the CH3D-GTM Virtual Appliance on SURAGrid Renato Figueiredo http://grid-appliance.org J. Davis, J. Fortes, P. Sheng, V. Paramygin, B. Tutak, D. Wolinsky Electrical and Computer Engineering Civil and Coastal Engineering University of Florida
The CI-TEAM Virtual Appliance Goals: Education, training and outreach in Cyberinfrastructure Facilitate use/deployment of coastal/estuarine simulation models Easily tap into computational power of local and remote resources Approach: A self-contained plug and play package A Grid appliance Virtualization and zero-configuration techniques Advanced Computing and Information Systems laboratory 2
Grid appliance in a nutshell Easy-to-install software with a preconfigured Grid environment Virtual machine appliance; open-source software running on Linux, Windows, Mac Hands-on examples, bootstrap Grid infrastructure, and zero-configuration software you re off to a quick start 15-30 minutes from download to job submission Advanced Computing and Information Systems laboratory 3
Grid appliance in a nutshell Quick start and examples you become familiar with capabilities and limitations You can continue to use the bootstrap Grid infrastructure Or create a Grid of your own Creating an equivalent Grid on your own network is also very easy Copy the same appliance to clusters, PC labs Simple management easy on your system administrator, even you can manage a small Grid Advanced Computing and Information Systems laboratory 4
Context Facilitating dissemination of simulation models Key challenges: Modelers want controlled software environment E.g. certain version of Linux, perl, libraries, But users don t have that exact environment Windows, incompatible libraries, Porting models? Burden on modelers Adapting environments? Burden on users Advanced Computing and Information Systems laboratory 5
Virtual machine appliances Can configure, test an entire system from scratch, just like a computer Can be easily disseminated as virtual disks: CDs, downloads Easy to install and run side-by-side with existing host system Vehicle for hands-on education and training Guest (e.g. CH3D GTM appliance) Host (e.g. Windows, Linux) Advanced Computing and Information Systems laboratory 6
What is a virtual machine (VM)? Functionally: same as physical computer Microprocessor, memory, disk, network, VM does not have resources of its own It is bound to resources of a physical machine (e.g. a fraction of the memory) when it runs Flexible and efficient VMs are defined by files easy to move, copy VMs run at speeds close to physical (1-10% overhead) Advanced Computing and Information Systems laboratory 7
Hosted VMs Advanced Computing and Information Systems laboratory 8
Virtual Machine technologies Renaissance in contemporary x86 systems: Several software vendors Hardware support from major vendors (AMD, Intel) VMware: player and server are free VirtualBox, KVM, Xen: open-source Windows, Linux, MacOS Advanced Computing and Information Systems laboratory 9
What is a virtual appliance? VM and pre-configured software Customized for an application and for easy dissemination TiVo analogy Computer + hard disk + software/interface packaged and customized for a function (TV record/playback) Advanced Computing and Information Systems laboratory 10
Example: the CH3D-GTM appliance NERR UF: develop, customize Deploy on Grid resources Two simple steps: -Install VMware -Unzip appliance Disseminate: download, CD SJRWMD Advanced Computing and Information Systems laboratory 11
Example: the SCOOP appliance Currently under development Environment for coastal modelers to run retrospective analysis, model comparisons SCOOP models, access to catalog, archive Condor workflows Advanced Computing and Information Systems laboratory 12
Use-case example (1) Ann is a researcher; she sometimes needs computing cycles for several long-running simulation experiments, but can make cycles from her computer available to others in her research community when idle. She downloads and runs a Grid appliance, creating a virtual machine (VM) in her own desktop that is now part of the appliance community Grid appliance Ann Advanced Computing and Information Systems laboratory 13
Use-case example (2) Bob, Carol, Diego, Esther, are other users belonging to the same community and with similar needs; they also download and run the appliance Advanced Computing and Information Systems laboratory 14
Use-case example (3) Although they are distributed and on different computers, all the appliances are configured in the same way, and are connected by a virtual network, so applications that Ann tests on her own appliance can run on other appliances Now Ann can submit simulation jobs to the pool of community resources; they will make use of idle computing cycles from other users Submit Advanced Computing and Information Systems laboratory 15
Use-case example (4) When Ann s appliance is idle, it is made available for others in the community to execute their own jobs Because jobs run in a separate, virtual environment, Ann s computer is isolated from unintended or malicious behavior arising from programs that run within the appliance Execute Advanced Computing and Information Systems laboratory 16
One appliance, multiple uses The same appliance software: Works in an open wide-area environment E.g. our global bootstrap infrastructure Works in a local-area environment E.g. your department, lab, college Works in a private wide-area environment A virtual organization Strong public key infrastructure for security; can connect resources across a restricted set of institutions Advanced Computing and Information Systems laboratory 17
Usage example local cluster The virtual appliance can be deployed on servers and clusters to support execution of models from multiple users Can present a Web-based portal interface to users Linux cluster Advanced Computing and Information Systems laboratory 18
Usage example local desktop pool A job scheduler within the appliance can be used to scavenge idle CPU cycles queue jobs Advanced Computing and Information Systems laboratory 19
What s inside the Grid appliance It is a virtual machine with its own operating system and software Linux and Grid middleware It connects to other appliances in a virtual network Like a VPN, but P2P (U. Florida s IPOP) It runs a robust Grid middleware scheduler to manage user jobs U. Wisconsin s Condor Advanced Computing and Information Systems laboratory 20
What do you need to run it? Intel/AMD PC, laptop, server running Linux, Windows or MacOS Virtual machine (VM) software Free: VMware Player/Server Open-source: VirtualBox, KVM Your appliance runs like any other program in your computer You don t need to go through lengthy process of configuring your computer to be a Grid node; all you need is to install the VM Advanced Computing and Information Systems laboratory 21
Why does it use Condor? High-Throughput Computing (HTC) Goal: effective management and exploitation of available resources Key technology: Condor Developed at University of Wisconsin; part of the NSF Middleware Initiative (NMI) Key features: Distributed ownership of resources Harvesting of idle compute cycles Robustness Advanced Computing and Information Systems laboratory 22
How is it related to other systems? SETI@Home, Folding@Home, and other voluntary computing efforts The Grid appliance allows its users to both contribute and use resources The Grid appliance is not specific to an application; it can run your existing programs You can deploy your own Grid pools on resources at your institution (or with your collaborators across the Internet) Advanced Computing and Information Systems laboratory 23
Education and training Our target user could benefit from Grid computing but doesn t know how to start I have simulation programs which take long to execute And I have several parameters to simulate Running on a single desktop takes too long I want to exploit the computational capability of a Grid But I don t want to change your programs I want a plug-and-play solution that runs on my desktop Advanced Computing and Information Systems laboratory 24
Hands-on tutorial preview Goals: Demonstrate key features of virtual machines, and the Grid appliance specifically Installation, interfaces User interaction; moving data from/to appliances Virtual networking: LAN and WAN Illustrate usage scenario based on Monte- Carlo simulations in Grid/cluster computing Self-learning, easy to install; realistic environment with robust software Advanced Computing and Information Systems laboratory 25
Tutorial steps Warm up to the appliance and Condor Hands-on experience with the VM appliance; simple Condor commands; simple Monte Carlo job submission Install VMM (e.g. VMware Player) Copy the appliance image to your desktop Follow the Grid Appliance introduction tutorial from http://grid-appliance.org, up to job submission Advanced Computing and Information Systems laboratory 26
For more information Additional documents, papers, presentations, videos in the Grid appliance web page Condor manuals and tutorials: http://www.cs.wisc.edu/~condor NSF Office of Cyberinfrastructure http://www.nsf.gov/dir/index.jsp?org=oci Projects using the Grid appliance http://www.nanohub.org http://ci-team.acis.ufl.edu http://archer.acis.ufl.edu Advanced Computing and Information Systems laboratory 27
Credits and acknowledgments IPOP/Grid appliance development team http://grid-appliance.org CI-TEAM at U. Florida http://ci-team.acis.ufl.edu Condor team at U. Wisconsin http://www.cs.wisc.edu Advanced Computing and Information Systems laboratory 28
Sponsors National Science Foundation Under grants SCI-0537455, SCI-0438246, ANI-0300118 Advanced Computing and Information Systems laboratory 29
Sponsors SURA Coastal Ocean Observing and Prediction (SCOOP) Program Initiative of the Southeastern Universities Research Association (SURA). Office of Naval Research, Award #N00014-04-1-0721 NOAA Ocean Service, Award # NA04NOS4730254. Any opinions, findings and conclusions or recommendations expressed in this material are those of the authors and do not necessarily reflect the views of the sponsors. Advanced Computing and Information Systems laboratory 30