QosCosGrid Grid Technologies and Complex System Modelling

Transcription

1 QosCosGrid Grid Technologies and Complex System Modelling Pamela Burrage Krzysztof Kurowski Institute for Molecular Bioscience, University of Queensland, Australia

2 Vision, objectives Complex systems (motivations) Very broad application class with widely varying requirements QosCosGrid grid technologies (motivations) requirements, advanced capabilities, prototype Examples: complex systems from molecular bioscience - use cases 3 and 5 Overview and main characteristics Protein interactions, lipid rafts Metabolic pathways Design and development Overview

3 EU 6th Framework Programme STREP Project 2.5 years, ends in 03/ Euro Strong QCG Consortium: 11 partners (2 private companies) from 10 countries Technical Manager DIISR Australian funding, 2 years, ends 06/2009 QCG in numbers

4 Gap and Vision Gap = Vision Reality Still wide range of demanding applications and complex systems run only on supercomputers and/or local clusters QosCosGrid vision To address & (make first step towards closing) this gap by developing a quasi-opportunistic supercomputer based on advanced grid middleware and new programming and execution environments Quasi-opportunistic supercomputer (QoS) Quasi-opportunistic = not really opportunistic Qos uses grid technologies to deliver supercomputer-like performance Qos facilitates execution of demanding parallel and distributed applications in grids through key technologies bridging the visionreality gap of the grid

5 QCG Objectives 1. To develop tools for end users and complex system developers Fault-tolerant cluster-to-cluster message passing libraries based on Open MPI (C/C++/Python) and ProActive (Java) Remote complex system steering and control capabilities User and admin easy-to-use web interfaces based on GridSphere/Vine toolkits 2. To develop advanced grid middleware - Dynamic resource brokering (for complex systems simulations) Reservation and orchestration of resources, communication, synchronization and routing as known from massively parallel processors computers 3. Integrate and evaluate QCG concept with various types of complex systems (9 example use cases) and running simulations on a real prototype testbed

6 Complex Systems gridification Developers/Users 1. Real problem 2. Problem decomposition (including algorithm and communication structure design) 3. Agglomeration 4. Mapping 5. Execution and Control QCG grid middleware

7 Complex Systems categorization EGEE or TeraGrid middleware T0: No communications T1: Explicitly defined, static comm. graph T2: Explicitly defined, dynamic comm. graph T3: Cellular automata T4: Distance-dependent communication T5: Unknown (random) communication QCG grid middleware

8 RECV Does it matter how it goes? It took around 0.3 sec and the average data transfer was 0.1 Mb/s GEANT2 SEND AARnet

9 it is important not for use case developers but for the QCG grid middleware, (fully transparent for CSS developers, using same well known APIs based on MPI or ProActive/RPC) * Use case users using the QCG grid middleware have to provide only a list of requirements for their CSS (number of processes, network topologies, networks speed, hardware architectures, stage-in/out data, etc.) and then the QCG grid middleware will take care of: security (sensitive data, identity/authentication, authorization and accounting with different administrative domains) monitoring of computing, storage and network resources in our international testbed load balancing, advanced reservation and co-allocation of computing resources required for multi domain experiments parallel and distributed application control and steering * It is partially true

10 QosCosGrid features usability (e.g. user interface) web based interfaces for scientific users AND administrators command line tools also provided for advanced users and administrators a set of tutorials, guides, template solutions and best practices available for application developers security and trust: improved authentication and single sign-on mechanisms via web for end users improved authorization, policy control and enforcement mechanisms via web for administrators Performance, deployment Australians have been already added to QCG!

11 Use Case 5 (Barcelona) goals Simulates a genetic regulatory network. Involves sets of highly-coupled DEs. Need to find globally optimal sets of parameters for a given model. ByoDyn* is a computational package aimed at integrating different types of DEs, through its interface with several publicly available packages Python, BLAS, LAPACK, gnuplot, Uses QosCosGrid environment for optimization of parameters through the use of different techniques. More research in this area is conducted in the BioBridge project: * ByoDyn is using Python

12 Use Case 3 (UQ) goals Prototype lattice of size 250 x 378 voxels: 1µm 1.5µm 1 time step ( 4 µs ) allows each molecule to move. Simulation for T = 1000 is only 0.004s real time. 600nm x 600nm, 25% rafts, fences, 2000 proteins, some obstacles: 2 mins compute-time, 0.004s real time. 600nm x 600nm, 50% rafts, proteins, FRAP: approx. 2 weeks on a PC, for 4s real time. To model a membrane 100 times as large, up to several real time seconds.

13 Parallel Implementation Master-slave implementation. Split membrane into vertical strips, one per slave. Track proteins as they move between slaves (unique ID). Provide report data and visualisation capability via the master.

14 Implementation contd. Visualisation capability Slaves send data to master (one large file) Front-end security via certificate signing Animation or snapshots at certain times

15 Implementation Issues Message passing via master more robust than slave-slave. Master outputs all files. Each slave has LH and RH overlap of neighbour s membrane. How much overlap? How often communicate? System integrity maintaining system dynamics.

16 Performance Compare timings for multi-processor computer for local clusters for remote clusters Speed-up vs frequency of communications vs volume of communication (size of membrane overlap)

17 Timings Examples

18 Technical Support Krzysztof Kurowski Original Development of Model Dan Nicolau (UQ, Oxford) John Hancock (UQ, Texas) Kevin Burrage (UQ, Oxford) Project Support Prof. Mark Ragan Other Programming Support Martin Swain (Ulster) Michal Lorenc (Hamburg) Use Case Discussions Jordi Villa i Freixa (Barcelona) George Kampis (Budapest) Acknowledgements

19 QCG Testbed prototype