Research Collaboration in the Cloud: - the NeCTAR Research Cloud

Transcription

1 Research Collaboration in the Cloud: - the NeCTAR Research Cloud National eresearch Collaboration Tools and Resources nectar.org.au NeCTAR is an initiative of the Australian Government being conducted as part of the Super Science Initiative and financed from the Education Investment Fund, and the National Collaborative Research Infrastructure Strategy. The University of Melbourne has been appointed the lead agent. Objectives: to enhance research collaboration through the development of eresearch infrastructure.

2 Australian eresearch Infrastructure Super Science eresearch Investments : Shared Data: Australian National Data Service (ANDS) Research Apps, Collaboration, Cloud NeCTAR Data Storage Research Data Storage Infrastructure (RDSI) High Performance Computing National Computational Initiative (NCI) Pawsey Centre Networks National Research Network (NRN) AU$48M AU$47M AU$50M AU$50M AU$80M AU$37M

3 Do computational modeling, complete data analysis, visualize results NCI Pawsey Use new tools, apps, work remotely and collaborate in the cloud NeCTAR eresearch Infrastructure Keep data and observations, describe, collect, share, find, and re-use them ANDS RDSI Understand mechanisms impossible to observe or experiment with directly Undertake novel research studies more extensive than ever before Generate new theories using data at scales previously inconceivable

4 Australian eresearch Backbone Cloud Data Data HPC HPC Network Network 50,000 concurrent tasks PB > 1 Pf/s 100 Tf/s Nx100 Gb/s 10 Gb/s 40 Software Tools and Virtual Laboratory Projects 250 Data improvement all institutions Researcher Single Signon available to the entire infrastructure

5 NeCTAR - Research Cloud Hosting research applications in the cloud A robust, scalable platform for research apps Any Researcher, Any Discipline, Any Where Supporting cross-institution collaborative services including shared research workflows across institutional and national boundaries Computational resource complementing HPC A computational infrastructure for many computation needs Cost effective and scalable for many classes of computation Freeing up resources on HPC facilities High throughput computing, Bag of Tasks, Parameter sweep, Optimisation A platform for innovation in research software and services Rapid deployment and wide sharing of research apps and services. successful services can scale to demand. 5

6 The NeCTAR Research Cloud A world first The NeCTAR Research Cloud is a partnership between 8 institutions and research organisation s who are deploying and operating Australia s first federated research cloud. University of Melbourne National Computation Infrastructure (NCI) Monash University Queensland CyberInfrastructure Foundation (QCIF) eresearch SA (ersa) University of Tasmania Intersect, NSW ivec, WA

7 A single, federated infrastructure An OpenStack based cloud infrastructure A single cloud deployed across 8 host organisations Implemented using OpenStack Cells

8 NeCTAR Research Cloud Providing research compute capacity Operating since January 2012 Based on the initial node at the University of Melbourne Over 20,000 computing cores now available to researchers Up to 30,000 by end 2014 Usage closely follows deployment and supply Each newly commissioned node brings new cohorts of users CPU Usage by Cloud Node

9 NeCTAR Research Cloud Supporting Australian researcher needs Sustained user growth Over 5,000 research users since January 2012 Up to 250 new users per month Low barriers to access: Dashboard login with University username and password Australian Access Federation Small resource allocations available on demand. Any researcher, anywhere

10 Research Cloud Usage Supporting the breadth of Australian research

11 Plant Energy Biology On the Cloud Plant Energy Biology Centre of Excellence Building collaboration on the Cloud. Researchers study how plants capture energy from sunlight, how they store that energy, and how they use that energy to grow and develop. Researchers are hosting collaborations with the Max Planck Institute and the Beijing Genomics Institute on the NeCTAR Research Cloud. Lead: University of Western Australia NeCTAR makes it much easier, much faster. It means more collaborations projects that would have just been too hard to go ahead. Professor Ian Small, Director, Laureate Fellow, West Australian Scientist of the Year.

12 Examples - Research Software as a Service Building a Matlab distributed compute service Matlab is a popular desktop scientific data tool Matlab in the cloud can parallellise your code over many machines Integrate Matlab with OpenStack Heat to elastically provision workers Also developing ipython Notebook as a Service, etc University of Melbourne

13 Why Should Researchers use the Cloud? How does NeCTAR support Australian researchers to bring their research to the cloud? We built it and they came!! Along with low barriers to access, many research communities supported their own migration to the Research Cloud. Early adopter training and workshops A series of dojo style workshops around Australia. Software Carpentry programs for researchers. Software Application Migration Programs Each Research Cloud node was funded to run a program to migrate research applications to the cloud. diversity in approach. NeCTAR Software Infrastructure Programs A $27M call for proposals to build research software platforms on the Research Cloud Virtual Laboratories and Research Tools.

14 Research Software Infrastructure Software is eating the research world too Research Software Tools, Applications and Services Embodies the understanding and knowledge of the Researcher Models and Simulation Embody scientific and human knowledge Data Analysis Tools and Algorithms To understand and interpret the Data Deluge Need for ongoing rapid refinement and innovation To remain competitive at the bleeding edge Understand and solve complex problems Wicked Problems advancing human knowledge Improving Research Software quality, reliability and sustainability Partnerships between researchers and software development expertise Research Communities and eresearch Communities

15 NeCTAR Virtual Laboratories Formed around engaged Research Communities Collaboratively creating collaborative infrastructure Approaching national in scale Delivering a transformative research impact Exemplars for sector adoption of capability Building on existing research capabilities Instruments and Data, Compute and Tools, Modelling, Analytics Integrating access to national and institutional research infrastructure:» Research Facilities, Instruments, Laboratories, Collections, Applications, Sensor networks, Repositories, Data, Computing, Remote Access, Research Workflows» NeCTAR Research Cloud, Single Signon, Data Storage, High Performance Computing Supporting research workflows Automating and sharing research methodologies Across institutional and discipline boundaries

16 Marine Virtual Laboratory - MARVL Ocean observations and modelling to improve planning for marine and coastal environments. Ian Coghlan is studying coastal erosion. MARVL saves him 3 months effort to access local data, wave model simulations and computing resources. MARVL enables you to start thinking about your problem sooner. Dr Roger Proctor, Director e-marine Information Infrastructure Facility. Lead: University of Tasmania

17 Genomics Virtual Laboratory - GVL Easy access to Genomics tools and resources for Australian biologists. The Peter MacCallum Cancer Centre is using the GVL in the NeCTAR Cloud, allowing researchers to collaborate easily and to access their data no matter where they are. Lead: University of Queensland This is the best exemplar of this kind of platform in the world Genomics capability for the masses. Associate Professor Andrew Lonie, Head of the Life Sciences Computation Centre at VLSCI.

18 Virtual Geophysics Laboratory Easy access to geophysics workflows, simulations and datasets. Lead: CSIRO The speed at which we carry out geophysical inversions was not possible before. Now the VGL does the cropping and reprojecting the data on the fly. We can complete our work in a matter of hours, instead of months. Dr. Carina Kemp, GeoScience Australia

19 Biodiversity and Climate Change VL Simplifies biodiversity-climate change modelling. The Virtual Laboratory accelerates biodiversity modelling, allowing researchers to integrate, analyse and model across large disparate datasets quickly and easily. BCCVL decreases the time to complete biodiversity and climate analysis from 2 months to 5 minutes, supporting new applications in research, government and industry. Professor Brendan Mackay, Director, Griffith Climate Change Response Program Lead: Griffith University

20 Endocrine Genomics Virtual Laboratory Statistical power for clinical research. EndoVL contains a registry with information on more than 6,000 adrenal tumour cases. Giving endocrinologists statistical power to improve clinical research into endocrine disorders. EndoVL enabled the investigators to learn from the data in ways they had not envisaged at the beginning of the study. Associate Professor Maria Craig, Australasia Diabetes Data Network. Lead: University of Melbourne

21 All Sky Virtual Observatory Theoretical and observational astronomy data, simulations and tools accessible from your desktop. ASVO provides access to cosmological simulations, galaxy formation models, and a comprehensive environment for analysis and exploration of the SkyMapper Southern-Sky Survey. Lead: Astronomy Australia Limited I was able to gain access to some of the most advanced simulations of our Universe ever created. Dr. Alan Duffy, Swinburne University The stars we are finding number one in a million the ANU SkyMapper telescope is unique. Professor Mike Bessell, Australian National University

22 Humanities Networked Infrastructure - HuNI Unlocking and uniting 30 of Australia s most important cultural datasets One million authoritative records relating to people, organisations, works, places, concepts and events. Researchers can combine, collect, connect and collaborate using the HuNI platform. Not only does HuNI make searching millions of records simple, it will revolutionise the way humanities researchers work collaboratively and cumulatively. Dr Michelle Smith, The Centre for Memory, Imagination and Invention, Deakin University Lead: Deakin University

23 More details at: and more Virtual Laboratories Climate and Weather Science Laboratory Bureau of Meteorology Integrated environment for climate and weather science modelling and data Endocrine Genomics Virtual Laboratory - Melbourne University Statistical power to improve clinical research into endocrine disorders Characterisation Virtual Laboratory Monash University Research environments for exploring inner space Human Communications Science University of Western Sydney Above and beyond Speech, Language and Music: A national collaborative environment for HCS research. Industrial Ecology Virtual Laboratory Sydney University Supporting comprehensive environmental carbon footprinting and sustainability assessments and 16 Research Tool Software Projects across broad research disciplines

24 Supporting Research Collaboration Research Cloud offers a platform for collaborative access to loosely coupled tools and services Virtualisation, Workflows, Research in the Cloud The Virtual Laboratory program succeeded in large part due to the Research Cloud. The market place or meeting point effect Bringing together Observation (data) and Modelling (simulation) communities Geoscience, Marine, Astronomy, Climate, Aggregating divergent research tools and data sets on the cloud.

25 Research Cloud Next Steps Moving beyond Infrastructure as a Service (IaaS) Ride the Wave of the Global OpenStack Community Ensure required capabilities are delivered through partnership with international OpenStack community Accelerate deployment of the OpenStack EcoSystem of services: Orchestration OpenStack Heat Automating deployment of complex, robust research services on the cloud. Database as a Service OpenStack Trove Platform as a Service Cloud Foundry

26 Research Cloud Next Steps Supporting interoperability and international collaboration Federating with international research cloud infrastructures NeCTAR has been a pioneer, but others are moving Eg. Federate with the EU funded EGI Federated Cloud. Interest from other regions Japan, South Africa, New Zealand, Building the Partnership NeCTAR works with and is active in the OpenStack community Seek to strengthen mutually beneficial relationships with industry partners Support access to commercial cloud platforms Future capacity growth may be in partnership with industry partners

27 Thank you nectar.org.au

28

29 Alveo Human Communication Science Studying speech, language, text, and music on a larger scale. Human communication scientists can now study speech, language, text, and music. Alveo brings together data collections, analysis tools, and workflow in a common environment. All of it the workflow and analysis is nicely automated in Alveo, and the results are reproducible. This is really exciting for me. Professor Denis Burnham, Lead Chief Investigator

30 NeCTAR through 2015 National Collaborative Research Infrastructure Scheme (NCRIS) NeCTAR has an additional $9.4M to: Scale up central operations of the Research Cloud Improve interoperability with emerging international cloud infrastructures Operate support services for the Virtual Laboratories Improving sustainability Support greater uptake and re-use of research software infrastructure