National-level einfrastructure development: from networks and Grids to clouds

Dr. Ognjen Prnjat European and Regional einfrastructure Management Greek Research and Technology Network on behalf of GRNET Technical department http://www.grnet.gr National-level einfrastructure development: from networks and Grids to clouds TICAL 2012 Conference, Lima

Why this talk? CLARA expanding from networking into services Includes computational services Greek national model could present a useful case study Very much appreciated long-term cooperation with our LA friends!

Outline GRNET overview GRNET network Computing: Grid and High-Performance Computing Cloud developments: R&E and egovernement Regional integrations Pan-European integrations Worldwide integrations

Outline GRNET overview GRNET network Computing: Grid and High-Performance Computing Cloud developments: R&E and egovernement Regional integrations Pan-European integrations Worldwide integrations

GRNET mission GRNET is a state-owned company operating under the supervision of the Ministry of Development (General Secretariat of Research & Technology); non-profit Main mission is to provide high-quality electronic infrastructure services to the Greek academic and research institutions: National and international connectivity services Distributed Computing Infrastructure services (computing, storage, visualisation) Supporting activity is the promotion and dissemination of the use of ICT in the public and private sector towards an e-government, e- Learning and e-business environment Main sources of funding are the Operational Programme for the Information Society, Ministry of Economy and Finance and EC projects GRNET has been certified by ISO 9001:2000 in project management

Pan-EU e-infrastructures vision The Research Network infrastructure provides fast interconnection and advanced services among Research and Education institutes of different countries The Research Distributed Computing Infrastructure provides a distributed environment for sharing computing power, storage, instruments and databases through the appropriate software (middleware) in order to solve complex application problems This integrated networking & DCI environment is called electronic infrastructure (einfrastructure) allowing new methods of global collaborative research - often referred to as electronic science (escience) The creation of the einfrastructure is a key objective of the European Research Area e-science Collaborations Distributed Computing Infrastructure Network Infrastructure

National e-infrastructures vision GRNET was one of the first NRENs in Europe to expand its services to grid and computing in general Being infrastructure-oriented and application-neutral serving all user e- Science communities Adopting the EU e-infrastructure initiative: Integrated Networking & Distributed Computing Infrastructures; serviceoriented Source: European Commission DG INFSOM Unit F3: Research Infrastructures

Outline GRNET overview GRNET network Computing: Grid and High-Performance Computing Cloud developments: R&E and egovernement Regional integrations Pan-European integrations Worldwide integrations

GRNET main networking tasks Interconnects universities, research centers, academic organizations (>150), primary and secondary schools (15000) 500.000 end-users Continuously upgrades the national backbone (Dark fiber backbone (Nx10 Gbps), institutions access (1 10 Gbps per institution) and international backbone (currently at 4*10 Gbps) Operates the GR Internet Exchange (GR-IX), peering of Greek commercial Internet Service Providers at 10Gbps each Cooperates with Greek and international research and academic institutions for the development of innovative networking services

GRNET network evolution: from GRNET2 to GRNET3 GRNET2: 2,5Gbps leased lambdas, 20M, 2000-2005 GRNET3: dark fibre based, 10Gbps capable, 30M, 2005-2008

GRNET network evolution: GRNET3 >50 PoPs 9000km fibers (IRU) MANs Attiki & Thessaloniki DF loops 33 cities Single-mode fiber pair 15-years IRUs Availability > 99% 1GbE interconnection over DF to the closest IP router 10GbE links for the Power Users 11

GRNET network evolution: from GRNET3 to GRNET4 GRNET3: dark fibres, 10Gbps capable, 30M, 2005-08 GRNET4: Equipment upgrades, 7.5M, 2012-2015 Service Oriented Design Optical Services Layer: physical layer connectivity Carrier Services Layer: Carrier Ethernet (MPLS) interconnection IP Services Layer: IP interconnection among GRNET customers and the rest of Internet 40/100Gbps wavelengths based on PM-QPSK modulation

GRNET NOC Manages the network devices and servers Maintains services (network, computing, voice & video) Monitors the performance and the use of network resources Resolves problems that occur in the networking and computational equipment Establishes security rules for devices and services in the backbone network and to handle security incidents Develops tools and additional services Monitor the adaptation of the network's acceptable use policy and procedures affecting it Produce documentations and reports Designs new network services Evaluates software or technologies of interest 13

Outline GRNET overview GRNET network Computing: Grid and High-Performance Computing Cloud developments: R&E and egovernement Regional integrations Pan-European integrations Worldwide integrations

HELLASGRID (HG) infrastructure http://www.hellasgrid.gr/infrastructure HG-01 cluster (pilot phase): @Demokritos - Athens 64 CPU, 10TB FC SAN, 12TB Tape Library, glite middleware HG02-HG07 clusters (HG project): Athens (NDC/EKT,Min EDU, IASA), Thessaloniki (AUTH), Crete (ICS-FORTH).Patras (CTI) ~1200 Cores ~40 TBytes total raw SAN storage capacity ~80TBytes Tape Library 6 extra Sites offered by Greek Universities/Research institutes ~600 Cores and 200 TB of Storage Total funding ~2Me 15

HELLASGRID applications HECTOR: Enabling Microarray Experiments over the Hellenic Grid Infrastructure GRISSOM Platform: Grids for In Silico Systems Biology and Medicine Evaluating the impact of climate change on European air quality Density Functional Theory Calculations on Atmospheric Degradation Reactions of Fluorinated Propenes Investigating the nature of explosive percolation transition First-principles studies on traditional and emerging materials 16

HPC infrastructure Invited by the Ministry of Education to submit a proposal which will support Greece s participation in European HPC trends Goal is the development of a national HPC infrastructure that will join PRACE s Tier-1 European infrastructure Budget 3.5MEuro Procurement and installation of HPC infrastructure Operation and provision of support services

HPC: Technical specs and target applications Aiming for at least a ~150 Tflops system Petabyte level storage Hosted in GRNET s existing Datacenter Support for a wide range of scientific disciplines: Biomedicince, Bioinformatics, Computational Engineering, Physics, Meteorology, Climatology, Seismology, Computational Chemistry etc. Based on the results of a feasibility study National survey among 29 academic and research institutes Collected requirements from 200 scientific applications developed by 162 research teams from various scientific domains

Outline GRNET overview GRNET network Computing: Grid and High-Performance Computing Cloud developments: R&E and egovernement Regional integrations Pan-European integrations Worldwide integrations

Cloud for R&E: the process Vision: flexible, production-quality cloud services for Greek R&E community Rationale: Step beyond Grid in terms of flexibility and availability Economies of scale for the R&E community; solving understaffing problems, poor service, low maintenance Minimizes the investment in equipment and support contracts Policy background: existing MoU in place for Grid computing, expanding for HPC as well Methodology: Technical workshops and requirements capture meetings Gradual offering of services, starting with storage, moving to VM on demand, IaaS, and then PaaS&SaaS Paving the way to public sector Funding: 2.2ME Data Centres; 4.5ME s/w and services

okeanos service: snapshot ~okeanos is set to deliver Infrastructure Compute (Virtual Machines) Network (Virtual Networks, L2 private Ethernets or public IPv4/6) and Storage (Virtual Disks) as a Service Target group: GRNET s customers direct: IT depts of connected institutions indirect: university students, researchers in academia Alpha2: March 2012-900 VMs - 700 alpha users

okeanos service http://www.grnet.gr Virtual Compute Machines Virtual Network Ethernets Virtual Storage Disks Virtual Security Firewalls

5x 8x okeanos: your own virtual infrastructure 2x 1x http://www.grnet.gr

okeanos: building blocks Compute: Cyclades Files: Pithos+ Images: Plankton Identity: Astakos Volumes: Archipelago Accounting/Billing: Aquarium

okeanos: requirements Need for easy, secure access to GRNET s datacenters User friendliness, simplicity Scalable to the thousands #VMs, TBs, users Running within GRNET s Identity Federation Resell or build your own?

okeanos: design (1/3) Commercial IaaS vs own IaaS Commercial IaaS Amazon EC2 not an end-user service Need to develop custom UI, AAI layers Vendor lock-in Unsuitable for IT depts persistent, long-term servers, custom networking requirements Gain know-how, build on own IaaS reuse for own services (HelixNebula as a commercial alternative Telefonica)

okeanos: design (2/3) Working with opensource Eucalyptus, OpenNebula, OpenStack Need a mature opensource core to build around Taking into account the issues of extensibility, flexibility, upgradeability, maintainability.

okeanos: design (3/3) Reuse existing components Build on Google Ganeti for low-level VM management operations Target commodity hardware Release to the community as opensource Design principles Modular development, reusability, iterative development, small team

okeanos: Software Stack REST API Multiple users, multiple resources Synnefo Multiple VMs on cluster Single VM Ganeti KVM

okeanos: Platform design user@home Web Client CLI Client Web Client 2 admin@home GRNET datacenter OpenStack Compute API v1.1 Synnefo cloud management software GRNET Proprietary Google Ganeti KVM Debian Virtual Hardware Direct Out of Band Access

IaaS Compute (1) Virtual Machines Powered by KVM (performing full system virtualisation) Google Ganeti for VM cluster management (creation, migration, handling downtimes, etc) Accessible by the end-user over Web or programmatically (OpenStack Compute v1.1)

IaaS Compute (2) User has full control over own VMs Create Select # CPUs, RAM, System Disk OS selection from pre-defined Images, or custom Images Popular Linux distributions (Fedora, Debian, Ubuntu) and Windows On Debian hosts Start, Shutdown, Reboot, Destroy Out-of-Band console for troubleshooting

IaaS Compute (3) REST API for VM management OpenStack Compute v1.1 compatible Python & Django implementation Supports 3rd party tools and client libraries User InterfaceAll UI operations happen over the API Just another API client, all on client side in JS/jQuery

Virtual Machine Actions My_Windows_desktop Start Console Reboot Shutdown Destroy

cloud storage - Pithos Online storage for all Greek academic and research community 100 GB/user; files, groups Access by web browsers, native test apps, iphone, Android Open source implementation of OpenStack object storage (REST API) Also used in ~oceanos: Stores custom and user-created images Plankton provides image registry over pithos

http://www.grnet.gr

http://www.grnet.gr

http://www.grnet.gr

http://www.grnet.gr

http://www.grnet.gr

http://www.grnet.gr

http://www.grnet.gr

http://www.grnet.gr

http://www.grnet.gr

http://www.grnet.gr

http://www.grnet.gr

http://www.grnet.gr

http://www.grnet.gr

http://www.grnet.gr

http://www.grnet.gr

http://www.grnet.gr

http://www.grnet.gr

http://www.grnet.gr

http://www.grnet.gr

http://www.grnet.gr

http://www.grnet.gr

http://www.grnet.gr

http://www.grnet.gr

http://www.grnet.gr

http://www.grnet.gr

http://www.grnet.gr

http://www.grnet.gr

http://www.grnet.gr

http://www.grnet.gr

http://www.grnet.gr

http://www.grnet.gr

http://www.grnet.gr

http://www.grnet.gr

http://www.grnet.gr

http://www.grnet.gr

http://www.grnet.gr

http://www.grnet.gr

http://www.grnet.gr

http://www.grnet.gr

http://www.grnet.gr

http://www.grnet.gr

http://www.grnet.gr

http://www.grnet.gr

http://www.grnet.gr

http://www.grnet.gr

700 alpha users ~oceanos: stats 900VMs (windows, ubuntu, debian in order of usage) Storage: 20% disk usage. Average 20G/VM 1 minute approx startup time for VM Scalable to thousands (Pithos to 10k) Per-VM or per-gb billing to be implemented

Cloud Infrastructure: Data Centers Data Center 1 (NDC) Status Production operation Location National Documentation Center, Athens, Greece Purpose Central PoP of GRNET, Connection to GEANT, GR-IX, Cloud&Grid. Mainly networking Number of Racks - 16 Total Power 110KW with N+1 redundancy Data Center 2 (MINEDU) Status Production operation Location Ministry of Education Athens, Greece Purpose Cloud/HPC services Number of Racks - 28 Total Power 450KW with N+1 redundancy Data Center 3 (LOUROS) Status Design phase Location Hydroelectric Plant of Louros Purpose GRNET disaster recovery for cloud/hpc services, GREEN Data Center, PUE<1.3 Number of Racks - 14 Total Power 250KW with N+1 redundancy

Cloud MINEDU snapshots

Cloud for R&E: Why? STUDENT- It gives me the opportunity to test different kinds of software on a machine that I no longer need after my work is done PROFESSOR - It makes it possible for me to deploy PC labs without having to worry about specific hardware or physical space. It makes me capable of providing machines to my students for a scheduled amount of time. It gives me storage space to upload content and to share data with my students or access them through my virtual hardware RESEARCHER- It enables me to run experiments in many different environments and network topologies which I can provision easily, quickly and dynamically. I can have persistent or volatile machines according to my needs. I can also upload (besides my files) my own Images and launch Virtual Machines from them Next step: Cloud for egovernement?

Current government IT status Multiple small-to-medium Information Systems scattered in Public Sector Many computer rooms / server rooms Under-used Devalued very fast Need specialized personnel (administration) High energy consumption 300m for server hardware during Operational Program Information Society Redundant computing power cannot be utilized Since there is no money for new equipment New Applications cannot be delivered to Public Sector No New Applications for Citizens

G-cloud development strategy (1) Phase 1: Virtualize everything Old physical systems are converted to VMs New system needs are covered from current physical machines Machine 1 VM 1 VM 2 VM 3 Needed Machine #1 Needed Machine #2 same hardware

G-cloud development strategy (2) Phase 2: Build few but BIG G-Data Centers Utilize virtualisation Since old systems are already VMs (1 st Step) are easily migrated to G-Data Centers New systems are created as VMs from the beginning Benefits: Reduced administration needs/costs Easier migration and disaster recovery procedures Improved energy efficiency Rapid development of new services Low services downtime

G-cloud development strategy (2) VM 1 VM 1 New VM VM 2 VM 3 OLD SYSTEM Needed New System VM 2 VM 3 SPACE FOR MANY VIRTUAL MACHINES SPACE FOR MANY VIRTUAL MACHINES SPACE FOR MANY VIRTUAL MACHINES NEW G-DATA CENTER

G-cloud development strategy (3) Phase 3: Share software, licenses, storage Sharing Software (eg. 2 Apps share same DBMS for non-critical systems) Sharing Licenses (eg. 2 Apps share same DBMS license) Sharing Storage (eg. Apps utilize space from common storage array)

G-cloud: policy background A MoU has been signed among all players (GRNET, Infosoc SA, Ktimatologio, Management Organization Unit, General Secretariat of Information Systems, and others) to define the national policy on government cloud computing. This team led by the Cabinet Office (included representatives from over 10 Government departments) set out to define how the public sector could utilize the Cloud Computing approach to ICT delivery and explore what benefits and challenges this approach would create. Open call from Information Society Operational Program of 15ME InfoSoc to build G-cloud

Outline GRNET overview GRNET network Computing: Grid and High-Performance Computing Cloud developments: R&E and egovernement Regional integrations Pan-European integrations Worldwide integrations

SEE Model: Converged communication & service infrastructure for the region Seismology, Meteorology, Environment Comp physics, Comp chem, Life sciences HP- SEE GEANT & SEE- LIGHT

Network: SEELIGHT SEEREN projects set up regional NREN connectivity and GEANT links SEE-LIGHT: South-East European Lambda Network Facility for R&E Deployment of an advanced regional network infrastructure, fibres and equipment Under the Hellenic Plan for the Economic Reconstruction of the Balkans HiPERB (80-20) Serbia implementation stage, Bulgaria tender stage, Romania on own funds, FYR of Macedonia ongoing SEENet: a management body for SEELIGHT

Grid: the SEE-GRID series

Grid: the SEE-GRID series Regional infrastructure and operations built through 3 projects User community buy-in secured National structuring via NGIs All countries in European Grid Initiative Key to success: distributing operations and supporting cross-border communities; joint lobbying strategies Before SEE-GRID-SCI During SEE-GRID-SCI

HPC: HP-SEE 120 Tflops aggregate 2 BlueGene machines Larger procurements coming up Greece and Serbia 26 applications in 3 VRCs Envisaged as bridge to PRACE Joint operations centre studied and assessed Balance of national & EC funds!

Outline GRNET overview GRNET network Computing: Grid and High-Performance Computing Cloud developments: R&E and egovernement Regional integrations Pan-European integrations Worldwide integrations

Network: GEANT 2x10 Gbps IP uplink 10 Gbps for GEANT+ circuits Circuits to Vienna, Milan, Sofia Advanced networking services Strong role in GN3 project and GEANT community

Grid: European Grid Infrastructure 30/05/2012 Project Presenta[on May 2012

Grid: European Grid Infrastructure Metrics Installed Capacity Logical CPUs EGI- InSPIRE and EGI Council members Including integrated RPs Value (Yearly increase) 270,800 (+30.7% ) 399,300 HEP- SPEC 06 EGI- InSPIRE and EGI Council members and integrated RPs 2.96 Million (+49.5% ) Storage Disk (PB) 139 PB (+31.4% ) Tape (PB) 134.3 PB (+50%) Resource Centres EGI- InSPIRE and EGI Council members 326 Including integrated RPs 352 Suppor[ng MPI 90 Countries EGI- InSPIRE and Council members 42 Including integrated RPs Performance Monthly Availability/Reliability 94.50%/95.42% U[liza[on HEP- SPEC 06 Hours 10.5 Billion (+52.91% ) Jobs 54 492.5 Million Jobs /year 1.35 Million Jobs/day (+46.42%)

HPC: PRACE 24 countries 5 Tier0 systems currently deployed ~9 Pflops total power (Tier-0) 15+ Tier 1 systems PRACE PP + phases of implementation projects, total EC investment ~ 70MEuro Access by European peer review Tier- 0 European centres Tier- 1 Na:onal centres Tier- 2 Regional/University centres

Pan-European Cloud initiatives participation GRNET has strong collaboration and leadership role in nascent pan-european cloud initiatives No core initiative available yet StartusLab GRNET leads operations, IaaS EGI cloud activity providing virtualised Grid sites on cloud GRNET cloud to be accessible via CHAIN gateway

Outline GRNET overview GRNET network Computing: Grid and High-Performance Computing Cloud developments: R&E and egovernement Regional integrations Pan-European integrations Worldwide integrations

CHAIN and CHAIN-REDS CHAIN 2010-12; CHAIN-REDS 2012-2015 Partners: 1) INFN (Italy - Coordinator) 2) CESNET (Czech Rep.) 3) CIEMAT (Spain) 4) GRNET (Greece) 5) IHEP (China) 6) UBUNTUNET (Africa) 7) CLARA (Latin America) 8) PSA (India) 9) ASREN (Med./Middle East/Gulf)

CHAIN Strategic Vision Interoperable World-wide Distributed Computing Infrastructure Virtual Research Communities can transparently access different kind of resources Sharing resources ubiquitously across different administrative domains CHAIN to promote and validate a proof-of-concept CNGrid NKN & Garuda EUAsiaGrid SAGrid & SANREN

Conclusion layered developments GRNET as Case study for national-level einfrastructure developments Regional integrations very important In LA CLARA as centre of gravity of the operations, but also community Path to worldwide infrastructures and integrations

Conclusion: Cloud for R&E - advantages for Latin America (1) Economies of scale for the community Minimizes the investment in equipment and support contracts Infrastructure, staff Solving understaffing problems, problems with QoS, maintenance

Conclusion: Cloud for R&E - advantages for Latin America (2) Grid resources sharing built the community and the culture CLARA as the centre of gravity Keeps supporting the regional collaborations and know-how Grid operations a very good step towards computing for research, should be supported further (MoU signed in Mexico City this Friday! CLARA support needed) Potential cloud services as a step ahead in terms of flexibility, manageability Greek national model comparable to CLARA potential model

Thank you!