Building PRAGMA Grid

Transcription

1 Building PRAGMA Inter-Cloud Infrastructure by Virtual Cluster Technologies Cindy Zheng (+ Yoshio Tanaka) For Resources Working Group

2 Status and Future Direction of GEO Grid National Institute of Advanced Industrial Science and Technology AIST

3

4 TOC Why Grid to Cloud Heterogeneous Cloud needs and solutions Distributed/Common file system Overall strategies Key technologies /VC sharing SDN Building infrastructure for scientists Lifemapper

5 PRAGMA Grid UZH Switzerland UoHyd India LZU China NECTEC KU Thailand MIMOS USM Malaysia CNIC China HKU HongKong IHPC/NGO NTU Singapore JLU China KISTI KMU Korea MU Australia ASGC NCHC Taiwan ASTI Philippines AIST OsakaU UTsukuba Japan HCMUT HUT IOIT-Hanoi IOIT-HCM Vietnam BESTGrid New Zealand SDSC USA CICESE UNAM Mexico CeNAT-ITCR Costa Rica UValle Columbia UChile Chile UPRM Puerto Rico 28 institutions in 17 countries/regions, 22 compute sites (+ 7 site in preparation)

6 Shifting Focus to Cloud Basic goals unchanged Distributed resources Data collection/storage, expertise, computing, Needs of sharing and collaborations Try to make easier for scientists to collaborate and use distributed resources Grid is still hard to use Main difficulty: provide software environments for users Cloud can be easier to use allow customize/maintain user environment But needs work How can users deploy s/vcs easily in a heterogeneous global cloud

7 GridFMO - The First Principle Calculations on the Grid (2007) Empowered by Grid technology, large scale fragment molecular orbital (FMO) calculations were conducted making use of computational resources on the Grid. The resources were collected interoperably from PRAGMA and OSG domains, via ordinary queue systems on each machine. Without special arrangements, a series of the calculations were carried out for about 3 months using maximum of 240 CPUs. Full electron calculation of bio molecules by using FMO method Long-term calculations supported by fault-tolerance Utilize vast computational resources on the Grid Flexible resource management to exploit ad-hoc available resources

8 Grid experiments were exciting and successful, but Using heterogeneous and distributed resources was not easy. Heterogeneity did exist in various places OS, Library version in more details of the system configuration Configuration of queuing systems e.g. max wall clock time disk quota limit Firewall configuration We need to install and test applications at each site one by one. We should not expect end users to do the same!

9 Shifting Focus to Cloud Basic goals unchanged Distributed resources Data collection/storage, expertise, computing, Needs of sharing and collaborations Try to make easier for scientists to collaborate and use distributed resources Grid is still hard to use Main difficulty: provide software environments for users Cloud can be easier to use allow customize/maintain user environment But needs work How can users deploy s/vcs easily in a heterogeneous global cloud

10 PRAGMA Grid/Cloud UZH Switzerland UoHyd India LZU China KU Thailand CNIC China HKU HongKong JLU China ASGC NCHC Taiwan ASTI Philippines AIST OsakaU UTsukuba Japan UCSD USA KU USA IndianaU USA UFL USA MIMOS UTM Malaysia NTU Singapore MU Australia BESTGrid New Zealand UChile Chile 23 institutions in 14 countries/regions, 18 compute sites, 13 Cloud sites March 4, 2013

11 PRAGMA Cloud Resources

12 PRAGMA Heterogeneous Cloud OpenNebula K OpenNebula K Rocks Xen Rocks K Rocks Xen Rocks Xen Share and VC vmdb.txt AIST Geogrid + Bloss Able to deploy to heterogenous cloud sites OpenNebula Xen NCHC FMotif LZU Bio1 Rocks Xen AIST Nyouga UCSD BioApp5 Need a common file system AIST QuickQuake Gfarm Rocks Xen Rocks Xen Rocks K Eucalyptus xen Rocks K OpenStack K

13 Gfarm Open source Distributed file system Developer University of Tsukuba Used by PRAGMA Grid and some regional networks GSI authentication Meta-servers File servers Clients

14 PRAGMA Cloud Storage Gfarm metaserver File server Client Gfarm

15 Overall Strategy Goals Easy to use Distributed computing Distributed services Use distributed data Secure Private network span globally Technologies Virtualization Sharing /VC images among heterogeneous clouds Moving big data is slow Running application where the data is SDN Secure envelopes for projects How Pilot team leads experiments Build infrastructure for scientists Work with science team first Biodiversity

16 /VC Sharing Experiments Methedology Pilot team (UCSD, AIST, NCHC) Extended testing team (All PRAGMA Cloud sites) Experiments Manually create s - pilot Manually deploy s - pilot Between K and Xen Automate deployment pilot then PRAGMA Cloud Manually create VCs - pilot Manually deploy VCs - pilot Automate VC deployment pilot then PRAGMA Cloud sites Results Pilot team designed /VC sharing scheme and interface standards Pilot team developed software and share with all sites Successfully auto-deploy /VC among many PRAGMA Cloud sites

17 Define Roles and Processes /VC authors Create /VC with applications If a /VC includes any network-sensitive service/application and needs adjustments when the /VC is deployed in a new network environment, the /VC author should provide script /root/reconfig or /root/vcreconfig to reconfigure the and VC Deposit /VC images into Gfarm Provide information about the /VC /VC users Request resources for /VC deployment Deploy /VC use scripts provided by resources providers Run /root/reconfig or /root/vcreconfig if provided Manage the /VC and run applications Resources providers Provide resources and access to users Provide easy means for users to deploy and manage /VCs

18 Implement Easy /VC deployment Automate deployment Implementation per virtual environment Virtualization engine (K, Xen, ) Virtualization manager (Rocks, OpenNebula, ) Share among sites with the same virtual environment Standard interface Transparency for users Enable easy implementation among different virtual environment

19 Design Interface Standards /VC image depository /VC datebase VC deployment input xml file VC deployment output xml file /VC deployment script command-line

20 /VC Images Depository Standards /VC images Gfarm depository directory structure /vm-images/ vmdb.txt vcdb.txt <author institution>/ images VC directory Frontend image, compute node image, xml file Example $ gfls /vm-images/sdsc/calit rw-r--r-- 1 zhengc gfarmadm Jan 24 10:12 calit img.gz -rw-r--r-- 1 zhengc gfarmadm 1148 Feb 21 13:35 calit xml -rw-r--r-- 1 zhengc gfarmadm Jan 24 10:20 hosted-vm img.gz

21 /VC Database Standards database $ gfexport /vm-images/vmdb.txt lzu-bio1,kvm-xen,lzu/lzu-bio1.img.gz,418f8b472dc7578bbfd661a71c bioapp5,xen-kvm,sdsc/bioapp5.img.gz,fd555578ddd725c07d3ee378452d8147 fmotif,kvm-xen,nchc/fmotif.hda.gz,06fff5f211750a38fd0883e8c97102ff geobloss,kvm-xen,aist/geobloss.hda.gz,fcf9b1e45098f2e8053afc1e0e15b6a5 nyouga,xen-kvm,aist/nyouga.img.gz,e4342a37740b9998c781fe2210fa8989 quiquake,xen-kvm,aist/quiquake.img.gz,db64fc7fcaced46ef3b7baf7e worker,xen-kvm,aist/worker_qq.img.gz,83113dddf665e86368a15d0682a3a532 -name boot-order image path checksum VC database $ gfexport /vm-images/vcdb.txt calit ,sdsc/calit /calit xml VC-name XML file path All VC files are in the same directory images, xml, checksum

22 VC Deployment Input VC-in.xml template Cluster topology K or Xen Architecture, H? Memory size, number of CPUs Network interface name, type How to boot

23 VC Deployment Output VC-out.xml template Cluster topology K or Xen Architecture, H? Number of compute nodes Each compute node name, MAC address, IP address

24 VC Deployment Command-line Interface $ pragma_boot vc-name [number of compute nodes] [local VC image depository path] "pragma_boot" is the script name, should be run in a normal user account "vc-name" is the name of the VC to be deployed. It's a required argument and should match the first field of an entry in vcdb.txt "number of compute node" is an optional argument. Default is 1. "local VC image depository path" is an optional argument. Default is standard gfarm path. Hide heterogeneity from users Needs minimum one implementation per virtual environment (virtualization engine, virtualization manager)

25 The Process vcdb.txt Gfarm F C F C Users vmdb.txt xml Authors Hosting server frontend vm-container-1-0 vm-container-1-0 vm-container-1-0 vm-container-1-0 vm-container-0-0 vm-container-0-0 vm-container-0-0 vm-container-0-0 C Pragma _boot Resources Providers

26 Automated /VC Deployment Solutions Deployment Script Rocks/Xen (UCSD), installed and ran on 7 sites OpenNebula/K (AIST) VC Deployment Script demo tomorrow Rocks/K (UCSD) OpenNebula/K (AIST) Next phase Expand usage to more Rocks/K-Xen and OpenNebula/K sites More implementation in other virtual environment

27 VC Deployment Script for OpenNebula/K Yoshio Tanaka, Akihiko Ota (AIST)

28 Demo Environments & Prerequisites Demo Environment OpenNebula Version 3.6 (built from source bundles) 1 front node + 4 compute nodes Each node has 500GB SSD Prerequisites The user is able to register/delete images to/from OpenNebula The following network resources are available in OpenNebula pragmapub (public network addresses) pragmapriv (private network addresses) vc-in.xml and image files are available in the same directory (e.g. /gfarm/vm-images/sdsc)

29 Step 1/4 1. Embed CONTEXT script $ sudo./pragma-makeover.pl /gfarm/vmimages/sdsc/calit xml CONTEXT script will set the following parameters and output vc-out.xml IP addresses / netmasks default gateway /etc/resolve.conf /etc/hosts hostname sshd

30 Step 2/4 2. Register the image to OpenNebula $./pragma-register.pl /gfarm/vmimages/sdsc/calit xml This may take 3 minutes (on SSD) or minutes (on HDD). Check the status by oneimage command $ oneimage top

31 Step 3/4 3. Boot a VC $./pragma-boot.pl c 3 /gfarm/vmimages/sdsc/calit xml First, boot a front node, then boot compute nodes. Booting the front node may take 3 minutes (on SSD) or minutes (on HDD). Check the status by onevm command

32 4. Login and test the VC Step 4/4 $ ssh root@ $ ssh hosted-vm We haven t provided scripts for shutdown the VC and delete images. Need to use OpenNebula commands. ex: $ onevm shutdown 157 # shutdown compute node $ onevm shutdown 154 # shutdown front node $ oneimage delete 20 # delete image for compute node $ oneimage delete 21 # delete image for front node

33 Network Overlay Experiment OpenFlow ViNe

34 Virtual network slice A Openflow network environment Virtual network slice B Lead by OsakaU, Pilot + AIST + UCSD Motivation Eliminate reconfigure network Better access control AIST GR E Open vswitch Open vswitch Open vswitch GRE Osaka Univ. Openflow Controller Trema (Sliceable routing switch) Open vswitch GRE GRE Open vswitch GRE Open vswitch UCSD Open vswitch GRE Open vswitch Open vswitch

35 Site Setup Example For Software OpenFlow

36 OpenFlow Experiments In PRAGMA Cloud First experiment by pilot team (OsakaU, AIST, UCSD) Successfully setup a OpenFlow subnet Easy deployment of Geogrid s as condor workers Successfully ran Geogrid applications OpenFlow activities fourishing in PRAGMA Cloud OsakaU leads expansion and further research on performance (demo) UCSD getting ready to test hardware OpenFlow (OpenFlow switch) functionalities and performance NTU, KU-Thailand setup OpenVswitch at their sites and connecting to PRAGMA Cloud OpenFlow network LZU built a Openflow switch test environment use NetFPGA at their site

37 Another Network Overlay ViNe Experiment Lead by UFL First experiment, +AIST Setup ViNe subnet Successfully tested live migration Second experiment, +UCSD+IU Setup ViNe subnets Successfully tested live migration Successfully deploy VC and run condor job (demo)

38 Building Infrastructure For Scientists First case - Lifemapper (1) Mount Kinabalu specimen data storage (2) (3) Predicted Habitat GeoPortal at UTM LifeMapper WebService at UKansas (4) Metadata (5) (1) A script queries GeoPortal instance at UTM (Malaysia )to retrieve raster data (2) Retrieves raster data from stored dataset (3) Submit dataset to LifeMapper as occurrences/experiments I. Prepare occurrence dataset II. Post occurrence set III. Post experiment (4) LifeMapper outputs prediction result as metadata (EML) (5) Catalogs LifeMapper predicted output metadata in GeoPortal at IU (US) GeoPortal at IU PRAGMA node Source: Uma Pavalanthan

39 Infrastructure for Lifemapper Needs Private network among biodiversity sites UFL, KU, IU (USA), UTM (Malaysia) Distribute data services Currently only KU, UTM Distribute computing Currently only KU Solutions Overlay network Setup and tested ViNe subnet among UFL, IU and UCSD Nest step, add other sites into the subnet Virtualize Lifemapper components First virtualize Lifemapper compute component

40 Build a Virtual Cluster for Lifemapper Team: Kansas University (KU), UCSD KU: modulate compute component UCSD: build VC, install software Team: Testing Successfully run lifemapper jobs on VC (demo) Next step: deploy the VC to other sites, use automated VC deployment solutions

41 Welcome Everyone To Get Involved