Cyberinfrastructure Education and Hands-on Training Using the CH3D-GTM Virtual Appliance on SURAGrid

Transcription

1 Cyberinfrastructure Education and Hands-on Training Using the CH3D-GTM Virtual Appliance on SURAGrid Renato Figueiredo J. Davis, J. Fortes, P. Sheng, V. Paramygin, B. Tutak, D. Wolinsky Electrical and Computer Engineering Civil and Coastal Engineering University of Florida

2 The CI-TEAM Virtual Appliance Goals: Education, training and outreach in Cyberinfrastructure Facilitate use/deployment of coastal/estuarine simulation models Easily tap into computational power of local and remote resources Approach: A self-contained plug and play package A Grid appliance Virtualization and zero-configuration techniques Advanced Computing and Information Systems laboratory 2

3 Grid appliance in a nutshell Easy-to-install software with a preconfigured Grid environment Virtual machine appliance; open-source software running on Linux, Windows, Mac Hands-on examples, bootstrap Grid infrastructure, and zero-configuration software you re off to a quick start minutes from download to job submission Advanced Computing and Information Systems laboratory 3

4 Grid appliance in a nutshell Quick start and examples you become familiar with capabilities and limitations You can continue to use the bootstrap Grid infrastructure Or create a Grid of your own Creating an equivalent Grid on your own network is also very easy Copy the same appliance to clusters, PC labs Simple management easy on your system administrator, even you can manage a small Grid Advanced Computing and Information Systems laboratory 4

5 Context Facilitating dissemination of simulation models Key challenges: Modelers want controlled software environment E.g. certain version of Linux, perl, libraries, But users don t have that exact environment Windows, incompatible libraries, Porting models? Burden on modelers Adapting environments? Burden on users Advanced Computing and Information Systems laboratory 5

6 Virtual machine appliances Can configure, test an entire system from scratch, just like a computer Can be easily disseminated as virtual disks: CDs, downloads Easy to install and run side-by-side with existing host system Vehicle for hands-on education and training Guest (e.g. CH3D GTM appliance) Host (e.g. Windows, Linux) Advanced Computing and Information Systems laboratory 6

7 What is a virtual machine (VM)? Functionally: same as physical computer Microprocessor, memory, disk, network, VM does not have resources of its own It is bound to resources of a physical machine (e.g. a fraction of the memory) when it runs Flexible and efficient VMs are defined by files easy to move, copy VMs run at speeds close to physical (1-10% overhead) Advanced Computing and Information Systems laboratory 7

8 Hosted VMs Advanced Computing and Information Systems laboratory 8

9 Virtual Machine technologies Renaissance in contemporary x86 systems: Several software vendors Hardware support from major vendors (AMD, Intel) VMware: player and server are free VirtualBox, KVM, Xen: open-source Windows, Linux, MacOS Advanced Computing and Information Systems laboratory 9

10 What is a virtual appliance? VM and pre-configured software Customized for an application and for easy dissemination TiVo analogy Computer + hard disk + software/interface packaged and customized for a function (TV record/playback) Advanced Computing and Information Systems laboratory 10

11 Example: the CH3D-GTM appliance NERR UF: develop, customize Deploy on Grid resources Two simple steps: -Install VMware -Unzip appliance Disseminate: download, CD SJRWMD Advanced Computing and Information Systems laboratory 11

12 Example: the SCOOP appliance Currently under development Environment for coastal modelers to run retrospective analysis, model comparisons SCOOP models, access to catalog, archive Condor workflows Advanced Computing and Information Systems laboratory 12

13 Use-case example (1) Ann is a researcher; she sometimes needs computing cycles for several long-running simulation experiments, but can make cycles from her computer available to others in her research community when idle. She downloads and runs a Grid appliance, creating a virtual machine (VM) in her own desktop that is now part of the appliance community Grid appliance Ann Advanced Computing and Information Systems laboratory 13

14 Use-case example (2) Bob, Carol, Diego, Esther, are other users belonging to the same community and with similar needs; they also download and run the appliance Advanced Computing and Information Systems laboratory 14

15 Use-case example (3) Although they are distributed and on different computers, all the appliances are configured in the same way, and are connected by a virtual network, so applications that Ann tests on her own appliance can run on other appliances Now Ann can submit simulation jobs to the pool of community resources; they will make use of idle computing cycles from other users Submit Advanced Computing and Information Systems laboratory 15

16 Use-case example (4) When Ann s appliance is idle, it is made available for others in the community to execute their own jobs Because jobs run in a separate, virtual environment, Ann s computer is isolated from unintended or malicious behavior arising from programs that run within the appliance Execute Advanced Computing and Information Systems laboratory 16

17 One appliance, multiple uses The same appliance software: Works in an open wide-area environment E.g. our global bootstrap infrastructure Works in a local-area environment E.g. your department, lab, college Works in a private wide-area environment A virtual organization Strong public key infrastructure for security; can connect resources across a restricted set of institutions Advanced Computing and Information Systems laboratory 17

18 Usage example local cluster The virtual appliance can be deployed on servers and clusters to support execution of models from multiple users Can present a Web-based portal interface to users Linux cluster Advanced Computing and Information Systems laboratory 18

19 Usage example local desktop pool A job scheduler within the appliance can be used to scavenge idle CPU cycles queue jobs Advanced Computing and Information Systems laboratory 19

20 What s inside the Grid appliance It is a virtual machine with its own operating system and software Linux and Grid middleware It connects to other appliances in a virtual network Like a VPN, but P2P (U. Florida s IPOP) It runs a robust Grid middleware scheduler to manage user jobs U. Wisconsin s Condor Advanced Computing and Information Systems laboratory 20

21 What do you need to run it? Intel/AMD PC, laptop, server running Linux, Windows or MacOS Virtual machine (VM) software Free: VMware Player/Server Open-source: VirtualBox, KVM Your appliance runs like any other program in your computer You don t need to go through lengthy process of configuring your computer to be a Grid node; all you need is to install the VM Advanced Computing and Information Systems laboratory 21

22 Why does it use Condor? High-Throughput Computing (HTC) Goal: effective management and exploitation of available resources Key technology: Condor Developed at University of Wisconsin; part of the NSF Middleware Initiative (NMI) Key features: Distributed ownership of resources Harvesting of idle compute cycles Robustness Advanced Computing and Information Systems laboratory 22

23 How is it related to other systems? and other voluntary computing efforts The Grid appliance allows its users to both contribute and use resources The Grid appliance is not specific to an application; it can run your existing programs You can deploy your own Grid pools on resources at your institution (or with your collaborators across the Internet) Advanced Computing and Information Systems laboratory 23

24 Education and training Our target user could benefit from Grid computing but doesn t know how to start I have simulation programs which take long to execute And I have several parameters to simulate Running on a single desktop takes too long I want to exploit the computational capability of a Grid But I don t want to change your programs I want a plug-and-play solution that runs on my desktop Advanced Computing and Information Systems laboratory 24

25 Hands-on tutorial preview Goals: Demonstrate key features of virtual machines, and the Grid appliance specifically Installation, interfaces User interaction; moving data from/to appliances Virtual networking: LAN and WAN Illustrate usage scenario based on Monte- Carlo simulations in Grid/cluster computing Self-learning, easy to install; realistic environment with robust software Advanced Computing and Information Systems laboratory 25

26 Tutorial steps Warm up to the appliance and Condor Hands-on experience with the VM appliance; simple Condor commands; simple Monte Carlo job submission Install VMM (e.g. VMware Player) Copy the appliance image to your desktop Follow the Grid Appliance introduction tutorial from up to job submission Advanced Computing and Information Systems laboratory 26

27 For more information Additional documents, papers, presentations, videos in the Grid appliance web page Condor manuals and tutorials: NSF Office of Cyberinfrastructure Projects using the Grid appliance Advanced Computing and Information Systems laboratory 27

28 Credits and acknowledgments IPOP/Grid appliance development team CI-TEAM at U. Florida Condor team at U. Wisconsin Advanced Computing and Information Systems laboratory 28

29 Sponsors National Science Foundation Under grants SCI , SCI , ANI Advanced Computing and Information Systems laboratory 29

30 Sponsors SURA Coastal Ocean Observing and Prediction (SCOOP) Program Initiative of the Southeastern Universities Research Association (SURA). Office of Naval Research, Award #N NOAA Ocean Service, Award # NA04NOS Any opinions, findings and conclusions or recommendations expressed in this material are those of the authors and do not necessarily reflect the views of the sponsors. Advanced Computing and Information Systems laboratory 30