Unidata s Vision for Transforming Geoscience by Moving Data Services and Software to the Cloud The WWOSC 2014 17 August 2014 Montreal, CA Dr. Mohan Ramamurthy Unidata Program Center University Corporation for Atmospheric Research 0
Unidata Niché: Providing comprehensive data services for Earth System sciences. Data: Over 30 data streams provided in real-time Data collection, cataloging, and distribution Both push and pull technologies are used User Support & Training: Direct email support Community mailing lists Annual Training Workshops, Triennial Users Workshops, and Regional Workshops as needed. Software: Data Distribution: LDM Remote Data Access: THREDDS, ADDE, and RAMADDA Data Management: netcdf, UDUNITS, and Rosetta Analysis and Visualization: GEMPAK, McIDAS, IDV, and AWIPS II GIS support via TDS (WCS, WMS) and KML and Shapefiles Community: Equipment Awards to universities; Seminars; Information Commons; Advocacy;
The Era of Data Intensive Science Data is the lifeblood of science, but we need to move from creating data to discovering knowledge. 2
Sea of Data Given the large data volumes, we must bring the processing to the data data proximate analysis/visualization
The Cloud Public clouds are where you will find things like Netflix and Dropbox. Private clouds exist within an organization and are often more specialized than a public cloud.
Cloud Computing The cloud provides elasticity, virtualization, and on demand computational, storage, networking, and software services via the Internet. Many types of services running in the cloud Any Resource Any Time Any Device Any Place
Principal Drivers Diminishing IT infrastructure Maturity of Cloud computing Era of Big Data BYOD: Bring Your Own Device Increasing Student Expectations
Goals Provide services to our users using one or more public clouds. Provide software and services to our community which they can run on their own cloud, regardless of the type. Develop and provide dataproximate processing, analysis and visualization.
Unidata Cloud Projects There are several cloud related projects currently underway. Radar and satellite Product generation in the cloud Migrate Unidata remote data services to a public cloud. Bring the Integrated Data Viewer (IDV) visualization tool to tablets and post PC devices via Application Streaming. Deploy AWIPS II data servers in the Cloud and provide analysis and visualization on AWIPS II thin clients. Provide and maintain a preconfigured computing environment, Unidata in a Box.
Product Generation in the Cloud We are now streaming some data streams to Amazon Web Services EC2 and Microsoft Azure cloud instances We generate products there (radar and satellite composites) And we stream the resulting products back to Unidata for broader distribution.
Remote Data Access Services Unidata currently provides "Remote Data" services from a single, monolithic server: Motherlode. THREDD S ADDE RAMADDA Web Motherlode
Remote Data Access Services Unidata is working on providing remote data access services via the cloud, using an Amazon Web Services (AWS) and Microsoft Azure instances. Web RAMADDA TDS ADDE
Remote Data Access Services By moving to the cloud, we will be able to scale the size and power of the resources needed, leveraging the inherent elasticity of cloud computing. RAMADDA Web Web RAMADDA RAMADDA Web TDS TDS AWI PS II AWI PS II AWI PS II TDS TDS AWI PS II AWI PS II AWI PS II AWI PS II AWI PS II AWI PS II
Bringing the IDV to New The Integrated Data Viewer is used for data analysis and visualization. It would take a significant resource commitment to bring the IDV to tablet devices. We are able to avoid this altogether with cloudbased application streaming. Devices
Bringing the IDV to New The interface is streamed to the client device. All processing is performed in the cloud. In the future, we plan to bring AWIPS II to the cloud by deploying AWIPS II servers on AWS and Azure. Devices
We are working to provide a Virtual Machine image containing preconfigured Unidata software that is portable and ready to run. Unidata in a Box
Unidata in a Box By coupling this with cloud software and services, we are able to remove almost all of the technical overhead of distributing and running these services. This also scales nicely, allowing us to reach an arbitrarily large number of users.
Internal Projects Unidata is also using the cloud internally. Software Development Software Testing Software Deployment Benefits: Reduced hardware requirements Rapid deployment Increased security More efficient workflow
Summary The Unidata community has expressed strong support for us to pursue projects which emphasis cloud functionality and capabilities. We are making concrete strides on several cloudrelated projects that will benefit our users. Future Unidata cloud efforts will benefit from what we learn from these initial projects. Acknowledgement NSF AGS 1344155