Open Source based C-DAC Cloud Infrastructure & Services

Open Source based C-DAC Cloud Infrastructure & Services Dr. Prahlada Rao B.B Joint Director 19 Mar 2014 NISG DeitY Industrial Cloud Workshop New Delhi Centre for Development of Advanced Computing (C-DAC) Bangalore 1

C-DAC CDAC was launched as India s initiative to Design & Develop High Performance Computers Indigenously in 1988 Premier R&D organization under the Department of Electronics and Information Technology (DeitY), Govt. of India C-DAC is engaged in R&D and Development of PARAM Series of High Performance Machines and also Specialized in : Cloud/Grid Computing Technologies E-Governance Applications Multilingual, Ubiquitous Computing Health Informatics Embedded, Mobile computing GIS Educational Research: ACTS Centre for Development of Advanced Computing : http://www.cdac.in 2

Presentation Plan About C-DAC C-DAC Cloud Computing Activities o C-DAC Cloud Infrastructure o Scientific Cloud Computing o SuMegha-Cloud Lab Kit Service Delivery Model Cloud Disseminations Conclusions 3

Supercomputing Evolution @ C-DAC 1 Gbps NKN connectivity 2012 Param 8000 Param 9000 Param 10000 ParamPadma GARUDA Grid Comp Param Yuva II Cloud Comp PetaFlop Comp 4

70 + Partners 10000 CPUs 550TF 1500 + Certificates 220TB Storage EGI, CHAIN- REDS NKN 5

Focus Areas PAN C-DAC Cloud Infrastructure C-DAC Cloud Stack Enablement of multi-centric applications on C-DAC Cloud Cloud Interoperability 6

Highlights Infrastructure as a service Cloud Stacks : Meghdooth SuMegha (Scientific Cloud) Storage as a Service Enablement of Multi centric Applications 7

PAN C-DAC Cloud Initiatives: Overview Mumbai: Disaster Recovery Mohali: Courses Hyderabad: E-learning e-shikshak Pune: Machine Translation MANTRA Bangalore: HPC Applications PSE SFM PSE NGS Noida: HMS Deployment Kolkatta: NE Courses TVM: Healthcare Applications HPC as a Service: MPI/Hadoop Virtual Cluster(VC) Problem Solving Environments IaaS SaaS StaaS Chennai: Meghdooth Cloud Stack Bangalore: SuMegha: Scientific Cloud Stack Bangalore: PAN C-DAC Cloud Infrastructure 8

Pan C-DAC Cloud Computing C-DAC local afforts to setup cloud with limited infrastructure. Cloud Stacks implemented Based on open source. Eucalyptus, Nimbus, OpenStack as the core engine. Many extensions to meet production requirement. Monitoring, metering and billing supported. Elasticity based on runtime parameters. Scientific Cloud Implemented (IaaS, SaaS, PaaS). Scientific Cloud Stack-SuMegha Released. Cloud Storage as a Service (CloudVault) Developed SuMegha_Cloud LabKit (to Build Privite Clouds) Developed MeghaSikshak Cloud based elearning Tool Developed Currently Applications or Enabled on local infrastructure. 19 Mar 2014 C-DAC Cloud Services 2 EGov 9

Pan C-DAC Cloud Infrastructure Physical Cloud infrastructure Set up Host Cloud Middleware Stacks: Meghdoot, Sumegha-Scientific Cloud Availability of the Cloud Infrastructure for applications. Based on Meghdoot/SuMegha Cloud Stacks. Physical Security, Power, Air Conditioning Available. C-DAC Centres able to access resources for building and experimenting with cloud applications. Repository: for Application Snapshots, Tools, and common tools. Provide tools for repositories, cloud management. Disaster Recovery, backup for projects. 10

PAN C-DAC Cloud: Applications Pan C-DAC Cloud Computing Supports: Pan C-DAC Cloud infrastructure Operating with: Meghdoot / Sumegha Stacks. Cloud Enable Applications of Pan C-DAC Cloud Project o o o o o o Hospital Information System(HIS) C-DAC, Noida HIS- Health Care Applications C-DAC, Thiruvananthapuram Learning Management System (e-sikshak) C-DAC, Hyderabad MANTRA (MAchiNe assisted TRAnslation tool) C-DAC Pune Disaster Recovery as a Service (DRaaS) C-DAC Mumbai esanjeevani (A web based health care solution) C-DAC Mohali 19 Mar 2014 C-DAC Cloud Services 2 EGov 11

Public network (Internet) over VPN Private C-DAC network 12

Highlights Infrastructure as a service Cloud Stacks : Meghdooth SuMegha (Scientific Cloud) Storage as a Service Enablement of Multi centric Applications 13

C-DAC Scientific Cloud 14

C-DAC Scientific Cloud Scientific Cloud is a model for on-demand access to a shared pool of HPC resources (e.g.,networks, servers, storage, applications) that can be easily provisioned as and when needed by the researchers and scientists. Benefits of Scientific Cloud Easy access to large infrastructures On demand access to HPC resources Virtual ownership of cloud resources Ease of deployment Scientific Cloud provide an alternative for HPC outreach 15

SuMegha: Scientific Cloud www.sumegha.in Provides on demand access to Compute, Storage resources and enables parallel environments. Services Supported : Infrastructure as a service(iaas) Storage as a service(staas) Software as a service(saas) Stakeholders Faculty and Students from engineering colleges Researchers 16

SuMegha: Scientific Cloud www.sumegha.in Cluster as an Infrastructure Service (IaaS) Provisioning of on demand HPC clusters Provisioning of Map reduce clusters to process large data sets Support for MPI as a tightly-coupled applications category. Design methodology for providing/supporting installation of MPI libraries in multiple machines simultaneously Storage as a Service (StaaS): CloudVault SaaS: Job Submission Portal (C_JSP), Cloud Access Portals Monitoring tools PaaS: Program development and analysis Tools Applications as Service (AaaS) PSEs PSE for Climate Modelling PSE s for Bio Informatics Types of applications supported Sequential & Concurrent Applications Data Parallel Applications - Map Reduce Parallel Programs: OpenMP,Pthreads, MPI 19 Mar 2014 C-DAC Cloud Services 2 EGov 17

SuMegha Stack A U T H & S E C U R I T Y ) Scientific Cloud portal and Problem Solving Environments SaaS (Job submissi on portal) SCIENTIIFC APPLICATIONS (Bioinformatics, Climate modeling) PaaS (HADOOP, MPI with High Perf IO) IaaS (VM, Virtual Clusters, storage) Cluster Virtualization (Contextualization) Cloud Middleware (NIMBUS, Openstack Swift) Operating System (RHEL) Hypervisors (XEN) Computational and Storage resources Interconnect ( Ethernet & infiniband) StaaS (Storage as a Service) Managemen t SLA & Policy Management Monitori ng (NAGIOS + Ganglia) Accounti ng, Metering & Billing 18

SciCloud Service Offerings Virtual Machine (OS, Time, Type) Small Medium Large GUI IaaS PaaS SaaS Cloud Portal ( Selects the Resource) PSEs for Bioinformatics Climate Modelling Virtual Cluster (No. of nodes, type, Time) Cloud Vault (StaaS) Web interface MPI Cluster (No. of nodes, Time) Hadoop Cluster (No. of nodes, Time) Job Submission Portal 19 Mar 2014 C-DAC Cloud Services 2 EGov 19

Infrastructure as a Service(IaaS) Accessible using Self service portal http://scientificcloud.cdacb.in/ Infrastructure as a Service : Provisioning of virtual machines(vm) & virtual clusters (VC) VM Specs Small, medium, large images Access to cloud storage(cloud vault) Access to additional elastic storage, through CIFS gateway VC Specs Upto 8 nodes in a cluster Each node: Small, medium, Large Type of cluster : MPI, HADOOP Access to cloud storage(cloud vault) Access to additional elastic storage, through CIFS gateway Secure Access :Secure ssh based access to the VMs & VCs through internet using secure key VM Size Details Size Detail Size No. Of Virtual CPU RAM (GB) Small 1 1 Medium 2 2 Large 4 4 20

Job Submission Portal for Virtual Clusters Web based interface to provide an easy & user friendly mechanism for job submission to MPI based virtual clusters. Enables users to run Sequential & Parallel jobs Features Job submission through Torque job scheduler. Job monitoring to know the Status of submitted jobs Output/Error files can be Viewed/Downloaded from the Portal User Management on Virtual Cluster 21

Cloud Vault Cloud Vault is your personal hard drive in the cloud. Store any kind of file like Documents, Music, Videos, Photos, and Virtual Machine Images on CDAC Cloud Vault secure servers. End-User Desktop Application Web Application Mobile Application Swift OpenStack Object Storage Storage and Data Center Physical Storage 22

Storage as a Service : Cloud Vault Cloud Vault provides access to storage space for storing and accessing data from anywhere, anytime http://cloudvault.cdacb.in:8084 Features Easy access to Storage : Supports easy access mechanisms such as Web interface, Linux clients File and directory operations : list, upload, download, delete Data Sharing: Enables users to share their data with the other registered users of cloud vault in a controlled way Interfacing Clusters with the Virtual Machines & Virtual Reliability & Availability: Provides 99.5% up-time guarantee and 3-way redundancy for data. Huge file support : Allows users to upload files from range of MBs, GBs (upto 40GB) Data isolation and security: Data of one user is isolated & secured from other 23

Web based problem solving environment to run seasonal forecast model on MPI based virtual cluster provisioned by SuMegha Enables on line visualization of the results Facility is provided to run GrADS commands from the browser, which visualize the necessary forecast parameters. PSE for SFM 24

PSE-SFM: JOB SUBMISSION: 25

PSE-SFM: JOB MONITORING 26

PSE-SFM: VISUALIZATION 27

PSE for NGS Problem Solving Environment for Next Generation Sequencing (PSE for NGS) Salient Features: Executes the complete workflow Preprocessing(Quality of input data),indexing, Alignment, post processing, Visualization Run-Time Logs for debugging and Directory Trees to navigate easily among the projects or different output files/directories. Huge Datasets uploading via Internet (More than 10GB supported). Common View window for visualisations and other textual outputs. Notification Centre for the user to know which step is going on at any given time and other information like which project is active and which is pending. 28

Next Generation Sequencing 29

SuMegha Cloud Kit Packaged software kit to enable the automated deployment of private scientific clouds on the hardware. Suitable for academic institutes & research organizations who want to setup cloud labs. Benefits Automated deployment of cloud setup for Infrastructure as a service Provides web based tools that supports easy and quick access to the Virtual machines and Virtual HPC clusters Simple and quick installation of all the cloud components. Provides Pre-built OS images with complex parallel environments like MPI and Map Reduce to cater the HPC needs. Provides Golden images with preloaded HPC applications like Seasonal Forecast Model,Cloud Next generation pipeline with the required visualization tools and job submission portal for easy parallel job submission Enables the users to setup cloud lab without having expertise of system administration. 30

SuMegha Software Stack & Supported Images Components : Hypervisor Xen Cloud Middleware Nimbus, Openstack Swift Storage Glusterfs Portal SuMegha portal Job Submission Portal Cloud Vault Portal PSE PSE for SFM PSE for NGS Pipeline Supported Images: Virtual Machines CentOS (small, large, medium) Job Submission Portal Virtual Clusters MPICH enabled with job submission portal HADOOP enabled PSE enabled images SFM NGS Pipeline 31

Enabling Applications on Meghraj using SuMegha Cloud Lab Kit 32

SuMegha Cloud Lab kit Sophisticated Packaged software Kit to enable deployment of private clouds on the hardware. the automated Highlights Open source components: Xen Hypervisor, Nimbus(cloud middleware) and GlusterFS Cloud tools & Dashboard: Cloud Portal, Job Submission Portal and Problem Solving Environment (PSE), Cloud monitoring. Automated Installer: Installation and configuration of private cloud on the recommended hardware to setup Cloud Lab without needing expert knowledge of System Administration. Secured Access to VMs through SSH using private keys Golden Images: VM s of different sizes - small, medium, large with CentOS operating system Parallel Programming environments with MPI libraries and HADOOP framework. Advantages Provides a hand holding mechanism for moving existing applications & building new applications on cloud Lower cost to application developers :Application developer need not pay for the cloud resources at the application development & testing phase Enhanced cloud adoption in several domains 33

Phases of Cloud Application Enablement using SuMegha Lab kit Enabling Existing Application Setup cloud testing environment using SuMegha lab kit auto installer Test application on local cloud setup using VMs Deploy application on Meghraj Cloud Developing New Application Developing Application on SuMegha Development Environment (SDE) using Cloud APIs(openstack, eucalyptus,etc) Deploy & Test Application on local setup Deploy Application on Meghraj cloud 34

Elastic Framework for Cloud Applications Elasticity Framework allows cloud resources (VMs, Storage) to dynamically scale-up or down as required by the application at run-time Building Blocks User defined application configurations. Cloud monitoring. Cloud Scalar. 35

4 Developer 1 3 Private Cloud (Development Environment) SuMegha Lab kit TENANT 1 APP STOR E 2 SuMegha Elasticity Framework Clou d API 4 Production Cloud A 3 2 Production Cloud B Monitoring Configuration Deployment Configuration and Scripts 36

Steps to Cloud Application Enablement using SuMegha Lab kit Developer Of cloud Application :- 1. Create his application 2. Provide Deployment configuration and scripts to the Elasticity frame work 3. Test on private cloud Created Using SuMegha Cloud Labkit 4. List that in AppStore User of Cloud Application 1. Choose the cloud application and provide the monitoring threshold listed in app store 2. The deployment configuration will be taken from Elasticity Framework. 3. Setting the threshold in the motoring component of Elasticity Framework. 4. Deployed on the cloud chosen by user. 37

Points to Ponder Licensing for commercial cloud platforms supported by SuMegha Billing & Pricing based on cloud platform chosen by the users 38

Cloud Publications/Patents Provisional patent on A CLOUD STORAGE SERVICE FOR CLOUD COMPUTING, Indian Patent Office, Provisional Patent Application No. 1392/CHE/2013, Patent Filed on: 28 March 2013. Shivay Veer Sharma, Anil Bhargava, Payal Saluja, Prahlada Rao B. B., Accessing E- Infrastructures using CDAC Scientific Cloud (CSC) Services, 2 nd IEEE CCEM 2013, Bangalore (Poster) Deepanshu S, Aman A, Ramesh Naidu Laveti, Arunachalam B, Vineeth Simon Arackal, Prahlada Rao B B; Problem Solving Environment for Seasonal Forecast Model on CDAC Scientific Cloud; ACC 2013, Bangalore AmanArora, Pavan Kumar, Vineeth Simon Arackal, Janaki Chintalapati, Arunachalam B,Prahlada Rao B B, Cloud NGS Tool: An Online Pipeline for Next Generation Sequencing Data; ACC 2013, Bangalore Vineeth Simon Arackal, AmanArora, Deepanshu Saxena, Arunachalam B, Prahlada Rao B B; SciInterface: A Web-Based Job Submission Mechanism for Scientific Cloud Computing; IEEE CCEM 2013, Bangalore Payal Saluja, Prahlada Rao BB, Ankit Mittal, Rameez Ahmad, C-DAC Scientific Cloud: On demand provisioning of HPC resources for Scientific Applications, paper published & presented at 18th International conference of parallel and distributed processing techniques and applications(pdpta 2012) 6th-19th July 2012 at Las Vegas, Nevada, USA Prahlada Rao B. B, Payal Saluja, Neetu Sharma, Ankit Mittal, Shivay Veer Sharma, Cloud Computing for Internet of Things & Sensing Based Applications poster paper published and presented at 6th International conference on Sensing Technology (ICST 2012), 18-21st Dec 2012, Kolkata, India 39

Dissemination: Workshops/Trainings Workshops Conducted: (3) o Indo_Taiwan International Workshop on Cloud Computing & Its Applications, 7-8 th Nov 2013, Bangalore o Prahlada Rao B. B., National Workshop on Cloud Computing, conducted at CDAC Knowledge Park, Bangalore, 21 st May 2011.(80+ participants from Industry, Academia) o National Work Shop on Virtualization, Feb 2011, C-DAC Bangalore ((70+ participants from Industry, Academia) Tutorials o Prahlada Rao B.B., Payal Saluja, Vineeth Simon Arackle, Half-Day Tutorial Open Source Platforms for Scientific Clouds CSI s Annual Conference on Advances in Cloud Computing (ACC2012), 26th July 2012, Bangalore.India (60+ participants from Industry, Academia) o Prahlada Rao, Vineeth Simon Arackal, Payal Saluja Tutorial on Cloud Computing for HPC C-DAC s National Conference on Parallel Computing Technologies (PARCOMPTECH 2013) 21-23 Feb, Bangalore, India o Prahlada Rao B.B, Tutorial on Scientific Clouds International Conference on Advances in Computing, Communications and Informatics (ICACCI-2013), Mysore o Prahlada Rao B.B, Invited Talk on Trends in Cloud Computing International conference on Advances in Cloud Computing (ACC 2013), Bangalore, 19-20 Oct 2013, India o Cloud Computing Talks: 15+ Invited /KeyNote Talks in Intl Conferences/ Workshops, and TEQiP programs 40

Dissemination: Technical Talks Given Prahlada Rao B.B., Resent Trends in Cloud Computing Technology day Talk C-DAC Bangalore 10 th May 2013. Prahlada Rao B.B., CDAC Cloud Initiatives Workshop on National Cloud Innovation Council, C- DAC Bangalore 29 th April 2013. Prahlada Rao B.B., HPC for BigData International Workshop on Data Analytics and Applications (IWDAA) 2013, BITS, Goa Campus, 1 st Mar2013. Prahlada Rao B.B, Payal Saluja, Neetu Sharma, Ankit Mittal, Shivay Veer Sharma, Cloud Computing for IOT & Sensing Based Applicationss, in IEEE International Conference on Sensing Technologies, ICST 2012, Kollkata, India, Dec 2012. Prahlada Rao B.B., Tutorial On Cloud Computing, International Conference on Trends in Knowledge and Information Dynamics, ICTK 2012, organized by ISI, NIMHANS Conv Centre, Bangalore, India, 10 July 2012.(150 participants from Industry, Academia) Payal Saluja, Prahlada Rao, Ankit Mittal, Rameez Ahmad, CDAC Scientific Cloud: On Demand Provisioning of Resources for Scientific Applications, in Proceedings of 18 th International Conference PDPTA 2012, Las vegas, Nevada, USA, 16-19 July 2012. Dr. Prahlada Rao BB, Cloud Computing for Telemedicine, National Workshop on Telemedicine Today & Tomorrow, C-DAC Mohali, An Invited Talk, 6 th July 2012. Vineeth Simon Arackal, Arunachalam B, Payal Saluja, Prahlada Rao B B, Co-existence of Cloud and Grid: A Case Study in Service Oriented Grid GARUDA. The International Conference on Information Technology- New Generations (ITNG 2011), Las Vegas, USA. 11-13April 2011. Vineeth Simon Arackel, Coexistence of Grid and Cloud at "Grid To Cloud (G2C)- Challenges and Applications" Workshop at VIT, Vellore, 2nd April 2011. 41

Conclusions Middleware Development for Cloud, Grid & HPC:C-DAC Strength C-DAC Cloud Activities o Open Source based C-DAC Cloud Testbed o Cloud Applications of National Importance o Scientific Cloud Computing Research o SuMegha- Cloud LabKit for Privite Cloud Building o Cloud Interoperability Research o Elastic Cloud Applications Development C-DAC Founding member of IntercloudTestbed-IEEE Active member CCICI (Cloud Computing Innovation Council for India) 42

Thank You 43