Unidata s Vision for Transforming Geoscience by Moving Data Services and Software to the Cloud

Size: px

Start display at page:

Download "Unidata s Vision for Transforming Geoscience by Moving Data Services and Software to the Cloud"

Theodore Barber
8 years ago
Views:

1 Unidata s Vision for Transforming Geoscience by Moving Data Services and Software to the Cloud The WWOSC August 2014 Montreal, CA Dr. Mohan Ramamurthy Unidata Program Center University Corporation for Atmospheric Research 0

2 Unidata Niché: Providing comprehensive data services for Earth System sciences. Data: Over 30 data streams provided in real-time Data collection, cataloging, and distribution Both push and pull technologies are used User Support & Training: Direct support Community mailing lists Annual Training Workshops, Triennial Users Workshops, and Regional Workshops as needed. Software: Data Distribution: LDM Remote Data Access: THREDDS, ADDE, and RAMADDA Data Management: netcdf, UDUNITS, and Rosetta Analysis and Visualization: GEMPAK, McIDAS, IDV, and AWIPS II GIS support via TDS (WCS, WMS) and KML and Shapefiles Community: Equipment Awards to universities; Seminars; Information Commons; Advocacy;

support Community mailing lists Annual Training Workshops, Triennial Users Workshops, and Regional Workshops as needed.

3 The Era of Data Intensive Science Data is the lifeblood of science, but we need to move from creating data to discovering knowledge. 2

4 Sea of Data Given the large data volumes, we must bring the processing to the data data proximate analysis/visualization

5 The Cloud Public clouds are where you will find things like Netflix and Dropbox. Private clouds exist within an organization and are often more specialized than a public cloud.

6 Cloud Computing The cloud provides elasticity, virtualization, and on demand computational, storage, networking, and software services via the Internet. Many types of services running in the cloud Any Resource Any Time Any Device Any Place

networking, and software services via the Internet.

7 Principal Drivers Diminishing IT infrastructure Maturity of Cloud computing Era of Big Data BYOD: Bring Your Own Device Increasing Student Expectations

8 Goals Provide services to our users using one or more public clouds. Provide software and services to our community which they can run on their own cloud, regardless of the type. Develop and provide dataproximate processing, analysis and visualization.

9 Unidata Cloud Projects There are several cloud related projects currently underway. Radar and satellite Product generation in the cloud Migrate Unidata remote data services to a public cloud. Bring the Integrated Data Viewer (IDV) visualization tool to tablets and post PC devices via Application Streaming. Deploy AWIPS II data servers in the Cloud and provide analysis and visualization on AWIPS II thin clients. Provide and maintain a preconfigured computing environment, Unidata in a Box.

Bring the Integrated Data Viewer (IDV) visualization tool to tablets and post PC devices via Application Streaming.

10 Product Generation in the Cloud We are now streaming some data streams to Amazon Web Services EC2 and Microsoft Azure cloud instances We generate products there (radar and satellite composites) And we stream the resulting products back to Unidata for broader distribution.

instances We generate products there (radar and satellite

11 Remote Data Access Services Unidata currently provides "Remote Data" services from a single, monolithic server: Motherlode. THREDD S ADDE RAMADDA Web Motherlode

12 Remote Data Access Services Unidata is working on providing remote data access services via the cloud, using an Amazon Web Services (AWS) and Microsoft Azure instances. Web RAMADDA TDS ADDE

13 Remote Data Access Services By moving to the cloud, we will be able to scale the size and power of the resources needed, leveraging the inherent elasticity of cloud computing. RAMADDA Web Web RAMADDA RAMADDA Web TDS TDS AWI PS II AWI PS II AWI PS II TDS TDS AWI PS II AWI PS II AWI PS II AWI PS II AWI PS II AWI PS II

14 Bringing the IDV to New The Integrated Data Viewer is used for data analysis and visualization. It would take a significant resource commitment to bring the IDV to tablet devices. We are able to avoid this altogether with cloudbased application streaming. Devices

It would take a significant resource commitment to bring the IDV

15 Bringing the IDV to New The interface is streamed to the client device. All processing is performed in the cloud. In the future, we plan to bring AWIPS II to the cloud by deploying AWIPS II servers on AWS and Azure. Devices

16 We are working to provide a Virtual Machine image containing preconfigured Unidata software that is portable and ready to run. Unidata in a Box

17 Unidata in a Box By coupling this with cloud software and services, we are able to remove almost all of the technical overhead of distributing and running these services. This also scales nicely, allowing us to reach an arbitrarily large number of users.

overhead of distributing and running these services.

18 Internal Projects Unidata is also using the cloud internally. Software Development Software Testing Software Deployment Benefits: Reduced hardware requirements Rapid deployment Increased security More efficient workflow

19 Summary The Unidata community has expressed strong support for us to pursue projects which emphasis cloud functionality and capabilities. We are making concrete strides on several cloudrelated projects that will benefit our users. Future Unidata cloud efforts will benefit from what we learn from these initial projects. Acknowledgement NSF AGS

We are making concrete strides on several cloudrelated projects that will benefit our

Environmental Data Management:

Environmental Data Management: Challenges & Opportunities Mohan Ramamurthy, Unidata University Corporation for Atmospheric Research Boulder CO 25 May 2010 Environmental Data Management Workshop Silver