Implementing and using IaaS cloud within the Flexible Services for the Support of Research project

Transcription

1 Implementing and using IaaS cloud within the Flexible Services for the Support of Research project Dr David Wallom, Associate Director - Innovation (Oxford e-research Centre) Technical Director (UK NGS) VP-Community (OGF) 1

2 Outline Infrastructure as a Service Cloud Computing Eucalyptus FleSSR project, its core and pilots Conclusions

3 Models of Cloud Computing: SaaS, PaaS, IaaS SaaS: Software as a Service > Google Apps, Force.com, Facebook; PaaS: Platform as a Service > Google App Engine, Azure Platform; IaaS: Infrastructure as a Service > Amazon Web Services. Defn: If a user speaks to a person to get access to resources, its virtualisation, if the user gets access through a computational interface it s a Cloud! instantiated OS image IaaS provider

4 Using an IaaS Users retains (full) control on: operating system: create, modify or use existing OS images; VM instantiation and management (start, stop, #VMs); networking: elastic IP, virtual firewalls, isolation (security groups); data: create and manage EBS devices; snapshotting. Great flexibility vs. extra effort

5 Eucalyptus Elastic Utility Computing Architecture Linking Your Programs To Useful Systems Open source and Commercial Amazon Web Services API compatible EC2, storage - S3, Elastic Block Store (EBS) Easy to install Xen and KVM hypervisors Commercial version supports others (inc. VMWare) Packaged version distributed as Ubuntu Enterprise Cloud

6 Client Tools Hybridfox Command Line Interface HybridFox RightScale Gems RightAws

7 The Project Flexible Services for the Support of Research JISC funded, 10 months 6 Partners, 12 staff members: academic and industrial; 3 cloud infrastructures; 3 solutions providers. Goals: building an integrated Public-Private hybrid cloud infrastructure; Developing accounting tools for cloud pilot 1, Multi Platform Software Development; pilot 2, On demand Research data storage.

8 Aims Develop Hybrid Public-Private cloud within HEI and trusted sector provider Implement stand-alone policy driven management layer allowing user abstraction Implement open standards (OCCI) interface within Canonical UEC Develop two trial use-cases as exemplar of developing management interfaces of swarms of VM with different policies

9 Private/Public Multiple Clouds NGS cloud Amazon cloud OeRC cloud Users Eduserv cloud Globally distributed; different resources/cost; different applications; EGI cloud non standardised: different AAA and UI.

10 Mediated Private/Public Multiple Clouds Amazon cloud NGS cloud Users Management Interface OeRC cloud Automation; load balancing; costs reduction; usability. EGI cloud Eduserv cloud

11 Install the Clouds Eduserv Oxford Reading

12 Integrate Management and Accounting Eduserv Oxford Reading STFC/NGS Accounting Database ezeel Management ezeel Management

13 Develop our Pilot case studies Eduserv Oxford Reading STFC/NGS Accounting Database ezeel Management ezeel Management Software Development Framework Storage Manager Software Development Framework Storage Manager

14 Accounting the Cloud Requirements & Implementation Requirements No modification of Eucalyptus code base; Accounting based on local accounts with VM, EBS, Snapshot usage for each registered user; Non-destructive treatment and permanent recording of accounting data; Open standard publishing allowing for multiple types of consumption. Implementation Data sources: Eucalyptus logs (cc.log, output-cloud.log, debug-cloud.log); 4 stage design: Aggregation: logs are parsed. Records are date and time stamped, checked for duplication and saved in daily archive files; Parsing, records are parsed and stored into an accounting database (MySql 5.1); Mining, database is mined to create OGF standard usage records; Uploading, Usage Records are uploaded to the RUS server.

15 The not so simple Accounting Database Schema

16 Accounting the Cloud Advantages & Challenges Advantages: Full regression chain from account record to original Eucalyptus logs; Agnostic storage and representation of the accounting records (SQL); Multiple, concurrent access of accounting records from different types of accounting clients: RUS + local web interface; Multiple representation of the accounting logs (SQL DB + aggregated files) allows for consistency tests embedded in the parser; Challenges: High volume of daily logs: up to 500Mb daily for just 3 file types; Highly redundant log information -> expensive duplicates checking; Non uniform formatting of log records, lack or date stamping for 2/3 of them.

17 Pilot 1: Multi-Platform Software Development SSH IDE Instance Configuration Manager (Software Upload, OS, Libraries etc.) Development Instance CVS/Subversi on

18 Pilot 1: Multi-Platform Software Development Instance Configuration Manager (Software Upload, OS, Libraries etc.) CVS/Subve rsion Build Manager Build Instance 1 Build Instance 2 Build Instance 3 Build Instance 4

19 Pilot 1: Multi-Platform Software Development Select Project Select dependant codes Select build platforms, OS and architecture Select project source-code Start Instances Based on select platform list start instances On running instances D/load and install dependencies and source-code Start build process Stream stdout/err from building instance Compile and report outputs Pack complete products Report errors and ship complete builds Allow access to error plagued builds directly on instances

20 Pilot 2: On Demand Research Data Service Data Management Controller (Choice of volume, Interface type and policies on lifetime) Data Interface Type 1 Instance Virtual Storage Volume EBS Controller

21 Pilot 2: On Demand Research Data Service Data Management Controller (Choice of volume, Interface type and policies on lifetime) Cloud 1 EBS Controller Virtual Storage Volume Cloud 2 EBS Controller

22 Pilot 2: On Demand Research Data Service Data Management Controller (Choice of volume, Interface type and policies on lifetime) Cloud 1 EBS Controller Data Interface Type 1 Instance Cloud 2 Virtual Storage Volume EBS Controller

23 Pilot 2: On Demand Research Data Service Request Storage Resource Volume Size Time Period Public/Private State Cost limitations Access Storage Resource Select storage interface, scp, WebDAV, FTP, etc. User Feedback Available resources endpoint returned Running Interfaces

24 Status of the Project and Conclusions Instances installed at Oxford, Reading & Eduserv Design of both pilot systems finalised Software development environment based on an extended Hudson Software Configuration Manager Design prototype developed Data management environment web based interface and connection to ezeel finalised Full internal workflow completed Initial design Stage 1 of the Multi Platform Software development system has been completed Stage 1 of the Flexible Storage Services pilot will be complete within 2 weeks Buisness models work to be delivered, over to Matt.