ISPASS-2009 Tutorial Proposal Archer: Zero-configuration Virtual Appliances for Architecture Simulation

Transcription

1 ISPASS-2009 Tutorial Proposal Archer: Zero-configuration Virtual Appliances for Architecture Simulation Tutorial audience and goals: This tutorial targets computer architecture researchers and students interested in: accessing high-throughput computing resources to run architecture simulations; better utilizing their own resources within a group; and/or sharing computer architecture applications and datasets. The tutorial will use hands-on activities to show users how they can gain access and use a shared distributed computing infrastructure (the Archer system) and how they can leverage its virtual machine appliances to manage local resource pools. (More information about Archer can be found at and in the attached flier). Organizers: The main organizer of this tutorial is Renato Figueiredo (University of Florida), in collaboration with the Archer project PIs: - Jih-Kwon Peir, Jose A. B. Fortes, Tao Li, P. Oscar Boykin (U. of Florida) - Gary Tyson (Florida State University) - Lizy K. John (University of Texas at Austin) - David Kaeli (Northeastern University) - David Lilja (University of Minnesota) - Gokhan Memik (Northwestern University) - Sally McKee (Chalmers) - Alain Roy (University of Wisconsin) Bio: Renato Figueiredo is an Associate Professor with the Advanced Computing and Information Systems (ACIS) Laboratory and the Center for Autonomic Computing at the University of Florida. He received the B.S. and M.S. degrees in Electrical Engineering from the University of Campinas in 1994 and 1995, respectively, and the Ph.D. degree in Electrical and Computer Engineering from Purdue University in From 2001 until 2002 he was with the faculty of the School of Electrical and Computer Engineering, Northwestern University. In 2002 he joined the Department of Electrical and Computer Engineering of the University of Florida. Figueiredo is the PI of the Archer project, a virtualized shared wide-area infrastructure for computer architecture research and education. In addition to Archer, he has participated in various research projects which apply machine, network and file system virtualization in support of high-performance and high-throughput computing in wide-area distributed systems. His broad research interests are in the areas of resource virtualization, computer architecture, distributed systems, network overlays, and autonomic computing. Presenters: The main presenter of the tutorial will be Renato Figueiredo. He will be assisted by one or two Archer staff or graduate students for questions/answers and troubleshooting during hands-on activities.

2 Abstract: Modern computer architecture research is driven by quantitative analysis. Leading-edge research requires detailed, cycle-accurate evaluation of many benchmark applications with several simulated configurations and is thus tightly dependent on the availability of high-throughput computing (HTC) systems. However, many research groups are hindered in their ability to perform research because of lack of access to computational resources, tools and datasets. This tutorial presents Archer ( a community-based computing resource for computer architecture research and education. Archer integrates technologies for resource virtualization, batch job schedulers, and multi-institution collaboration, in order to create a collaborative wide-area infrastructure with the following features. (1) Scalability: Archer starts from a seed set of hundreds of processor cores across distributed cluster resources. Subsequently, each new user joining Archer with one or more desktops or servers seamlessly contributes to its aggregate capacity. The system uses Wisconsin s Condor ( a robust batch scheduler that scales to large resource pools (2) Ease of use: Archer leverages packaging and distribution of open-source software environments as self-configuring virtual networks of virtual appliances, which can easily be installed by individual users in their own resources using contemporary, freely available VM software. (3) Support for unmodified applications and community sharing: Archer runs unmodified Linux-based full-fledged open-source and commercial software simulation environments (such as Simics, SESC, SimpleScalar, Fes2, PTLsim) and features a Wiki and plug-and-play NFS distributed file systems so users can easily share application executables, support scripts, input and output data sets, tutorials, and manuals. In doing so, Archer facilitates the dissemination of useful tools and data sets, and foster creation of reproducible simulation experiments. Duration: This is a half-day tutorial, consisting of a brief technical introduction to the Archer cyberinfrastructure and hands-on, interactive access to the Archer infrastructure by attendees using virtual appliances distributed during the tutorial, as described below. Topics: - Presentation (20 minutes): o Brief overview of core technologies in Archer: Contemporary virtual machine monitors, virtual appliances Virtual private networks Condor job scheduler o Putting it all together: introduction to the Archer distributed system - Hands-on session (30 minutes): o Archer appliance installation o Basic user interaction with the appliance User interface Data transfers Hello world job submission

3 - Presentation (20 minutes): o Software installation and data sharing: Virtual disk organization Software installation; creation of custom appliance modules Data sharing using NFS (Network File System) Case study: Simics - Hands-on session (30 minutes): o Running Simics in interactive mode o Making datasets available via NFS o Running Simics in batch mode - Presentation (20 minutes): o Managing jobs with Condor Vanilla and standard jobs Job configuration files Specifying job requirements Introduction to workflows Job monitoring, priorities - Hands-on session (30 minutes) o Compiling SESC for Condor o Creation and submission of parameter-sweep SESC batch o Creation and submission of workflow job o Job monitoring and management - Presentation (20 minutes): o Deployment on clusters and desktop pools Joining the global pool Private/group pools; security Opportunistic scheduling on desktop resources - Hands-on session (30 minutes) o Deploying a private resource pool Tutorial requirements: This tutorial requires broadband wireless Internet access in the room for all attendees. Tutorial participants are required to bring a laptop if they wish to follow the hands-on component of the tutorial: - x86-based processor - 512MB RAM minimum, 1GB or more suggested - Windows, MacOS or Linux - VirtualBox or VMware VMs and the Archer virtual appliance image instructions will be given on the tutorial Web site for those who want to preinstall the software; we will also bring all software in CDs and USB sticks.

4 Previous tutorials: Hands-on tutorials covering the virtual appliance-based Condor infrastructure underlying Archer (but without emphasis on computer architecture applications) have been given in the following venues: - CollaborateCom 2008 (half-day tutorial; ~10 attendees) - SuperComputing 2007 (hands-on tutorial part of a half-day education program session; ~15 attendees) - TeraGrid Conference 2007 (hands-on tutorial part of a half-day session; ~10 attendees) - St. John s River Water Management District, 2007 (half-day tutorial, ~10 attendees) References: [1] R. Figueiredo, P. O. Boykin, J. Fortes, T. Li, J-K. Peir, D. Wolinsky, L. John, D. Kaeli, D. Lilja, S. McKee, G. Memik, A. Roy, G. Tyson, 'Archer: A Community Distributed Computing Infrastructure for Computer Architecture Research and Education'. In 4th International Conference on Collaborative Computing (CollaborateCom 2008), 11/2008 [2] D. Wolinsky, R. Figueiredo, Simplifying Resource Sharing in Voluntary Grid Computing with the Grid Appliance. In 2nd Workshop on Desktop Grids and Volunteer Computing Systems (PCGrid 2008).

5 Archer: Information a shared Technology cyberinfrastructure Solutions for computer architecture research and education VIRTUAL CLUSTERS Archer builds upon virtualized resources virtual machines The Archer project is building a distributed, open community resource for simulationbased computer architecture research and education. Computer architecture researchers and students world-wide are invited to participate in this cyber-infrastructure, gaining access to high-throughput computing pools with easy-to-deploy architecture tools, and sharing their own idle computer cycles, applications, data sets and educational material. and overlay virtual networks to create scalable virtual clusters over wide-area Internet resources. VM appliances and selfconfiguring IP-over-P2P virtual network software enables Archer nodes to be easily deployed by end users. CONDOR Archer virtual clusters run the Condor batch job scheduler, a robust Grid middleware that supports typical computer architecture simulation Free pre-packaged Archer Virtual appliances - run on free VMMs (VMware, VirtualBox, KVM) Portal and Wiki: Community-contributed content: applications, datasets, tutorials 2. Boot appliance: 1: Download automatically appliance joins Archer pool Archer Global Virtual Network 3. Run architecture simulation jobs on the Archer pool through Condor Simulators: System software: Simics, SESC, Condor scheduler Simplescalar NFS file systems Archer seed resources 300+ cores Fall 2008 workloads high-throughput execution of independent jobs, such as parametric sweeps. COLLABORATION TOOLS Archer provides a Wiki to share documents and tutorials, auto-configured NFS file systems to share datasets and tools, and user groups for questions and support. Contribute to the community by sharing your own research and education content such as tools, scripts, tutorials, class assignments and datasets. archer-project.org Flexible deployment options: Archer Express: For first time users. Quickly gain access to a small-scale pool of resources within ~ 30 minutes to test-drive Archer capabilities on your own. Archer Global: For academic research and education use. Gain access to a large-scale world-wide pool of CPU resources, shared tools and datasets. Effortlessly contribute with your own resources to the community when idle. Archer Local: For deployment of private Archer pools on your own localarea resources (clusters, servers, desktops). Easy steps to get started: 1. Install a free VM monitor (VMware Player, VirtualBox, or KVM) in your desktop - x86-based Windows, Linux, or MacOS. 2. Go to archer-project.org, download and run the Archer Express Grid appliance image 3. Follow the quick start tutorial to submit your first simulation run This work is sponsored by the National Science Foundation under CRI collaborative awards , , , , , , , and

6 Archer: a shared cyberinfrastructure for computer architecture research and education PARTICIPATING INSTITUTIONS Archer is a collaborative effort led by University of Florida, Florida State University, Northeastern University, University of Texas at Austin, Northwestern University, Cornell University, University of Minnesota and University of Wisconsin-Madison. APPLICATIONS Simulation tools widely-used in the community are available in Archer, including SESC, Simics, SimpleScalar, and PTLSim. In the typical case, Linux 32-bit x86 binaries will work on Archer, unmodified. MANAGEMENT AND USER SUPPORT Management staff develops and maintains the Archer virtual machine appliance image so Archer users can focus on the science rather than the infrastructure. Management staff also provides support through user mailing lists. Advanced Computing and Information Systems Lab A community cyberinfrastructure: The Archer technology targets simple deployment and use: Shared distributed computers: Archer makes high-throughput computing Pre-packaged VM appliances: Archer accessible to the computer architecture packages all the software needed to run community at large. Seeded with hundreds architecture simulation jobs conveniently. It of CPUs across a distributed system (to be uses freely available virtual machine deployed over three years), the software to create self-configuring, plugand-play virtual appliances and the robust infrastructure grows as a community effort because users can effortlessly contribute Condor batch job scheduler. Archer VM with resources of their own to the appliances enable functionality similar to community when they are idle. systems such as SETI@Home, but support unmodified applications and provide Shared architecture content: Archer sandboxing of remote jobs through the use facilitates the sharing of tools, datasets, of virtualization. self-contained experimental setups, and educational materials. Sharing of tools and The Archer software makes it very easy for experiments is facilitated by providing a computer architects to deploy their own consistent virtual environment to all Archer Linux Condor pools on their own resources users and automatically-configured NFS (for example, instructional lab PCs). A file systems. Sharing and dissemination of small local pool can be brought up in a content is facilitated by the Archer Wiki. matter of hours. Frequently Asked Questions What are the software licenses used in Archer? The core of the Archer middleware is open-source (GPL and Apache2 licenses). Applications include both opensource packages (e.g. SESC) and simulators with academic licenses available to Archer users (e.g. Simics). Who can use Archer? Is it free? Archer is available and free for use by computer architecture researchers and educations within the US and abroad. Usage of the Archer Global pool is limited to academic use only; Archer middleware may be used to deploy Archer Local pools for non-academic purposes. Can I use Archer to create an isolated, private Condor pool in my own resources? Yes, you can use the VM appliances to deploy Archer Local pools, which are independent from the shared pool. However, Archer staff will not be able to assist with Archer Local management. What security mechanisms are provided? All jobs run within a virtual machine sandbox. Access control is mediated by Condor, which means users do not have direct access (e.g. ssh) to a VM, except through Condor. Remote jobs run as UNIX user "nobody" within a VM sandbox. Archer jobs can only send packets to other virtual machines within the Archer virtual network. Packets sent to other Internet hosts are blocked by a firewall. Archer confines its traffic to the virtual network through firewalls to avoid the use of Archer to launch denial-of-service attacks. All virtual network IP traffic between Archer VMs is authenticated and encrypted.