Strategic Plan. Scientific Computing Division. December National Center for Atmospheric Research Boulder, Colorado, USA

Transcription

1 Scientific Computing Division Strategic Plan to serve the computing, research, and data management needs of atmospheric and related sciences National Center for Atmospheric Research Boulder, Colorado, USA December 2004 SCD Strategic Plan 1

2 2 SCD Strategic Plan

3 Table of Contents Executive Summary 1 Introduction 3 The Challenges 5 Science Driving Demand for Scientific Computing Data Storage, Management, and Analysis Physical Infrastructure An Evolving Cyberinfrastructure Human Resources Financial Environment The Plan 9 High-End Computing Petascale Data Management Infrastructure Data Archival Data Storage and Management Infrastructure A Grid Capability at NCAR Networking Securing UCAR s Cyberinfrastructure Physical Infrastructure Research Data Stewardship and Curation Research, Development, and Community Software Technology Tracking Research in Applied Mathematics and Computer Science Frameworks for Modeling the Earth System Software Engineering and Lifecycle Management A Knowledge Environment for the Geosciences Data and Knowledge Systems Terascale Data Analysis and Visualization Advancing Collaborative Environments for Research Evolution of User Services Education and Outreach People: Valuing SCD s Workforce Conclusion 27 SCD Strategic Plan 3

4 4 SCD Strategic Plan

5 For over 40 years, the National Center for Atmospheric Research (NCAR) and its broader community of university and multi-agency researchers have relied on the facilities of the Scientific Computing Division (SCD) to conduct research in the atmospheric and related sciences. Since NCAR created its Computing Facility in 1964, the mission of providing highperformance computing has been an important mandate. SCD will continue to provide a complete, end-to-end suite of computational services to meet the needs of UCAR s scientific research constituents. SCD operates within a challenging environment characterized by rapidly changing complex technology and a substantial scientific demand for high-end computing. Over the next five years, SCD envisions a future of combined stable services utilizing proven technologies in the areas of high-end computing and data archival and access, while pursuing promising new technologies. Collectively, these components enhance an organizational cyberinfrastructure to create and maintain a broadly articulated knowledge-development environment. Such an environment encompasses all of the computational and analytical tools, systems, processes, infrastructures, and people needed to create and maintain it in partnership with a broad scientific community. Executive Summary The Scientific Computing Division (SCD) is part of the National Center for Atmospheric Research (NCAR) in Boulder, Colorado. NCAR is managed by the University Corporation for Atmospheric Research (UCAR) and sponsored by the National Science Foundation (NSF). Integrated with the NCAR Strategic Plan, the SCD Strategic Plan is focused on goals for the next five years. It lays out the challenges ahead in addressing the needs of the broad research community while enhancing our constituents ability to conduct scientific investigations. SCD intends to meet the challenges of the future and provide leadership for NCAR and the larger atmospheric research community by providing an integrated cyberinfrastructure for atmospheric science research. These 12 strategic areas define SCD s collective vision for scientific computing at NCAR during the next five years: High-End Computing By 2009, SCD plans to achieve a 25-fold increase over current sustained computing capacity. Petascale Data Management Infrastructure Commensurate with the plan for high-end computing, SCD will enhance its data storage and archiving facilities to unify and advance data management and scientific workflow efficiency. A Grid Capability at NCAR SCD will develop and deploy a Grid-enabled environment to support the community s evolving cyberinfrastructure. SCD Strategic Plan 1

6 Networking SCD will provide and evolve the network infrastructure to support advances in UCAR science, technology, and education. Securing UCAR s Cyberinfrastructure SCD will secure and protect UCAR s information technology infrastructure from illicit activities. Physical Infrastructure SCD will expand the computing facility to accommodate NCAR s future computational and data storage infrastructure. Research Data Stewardship and Curation SCD will expand data stewardship and improve access to its growing, internationally recognized research data archive. Research, Development, and Community Software SCD will track and deploy new technology, perform applied mathematics and computer science research, and develop new software infrastructure. SCD s research and development objective is to provide new solutions that can be applied to the atmospheric and related sciences. A Knowledge Environment for the Geosciences Working in concert with other community efforts, SCD will advance a portfolio of projects aimed at building future collaborative knowledge environments that span data management, access, analysis, visualization, and collaboration technology. Evolution of User Services SCD will provide a balanced set of services to enable researchers to easily and effectively utilize community resources. Education and Outreach SCD will participate in education and outreach activities that engage diverse audiences and inform constituents of SCD s role in advancing scientific research. People: Valuing SCD s Workforce SCD will provide opportunities for staff to work together to pursue rewarding careers in a changing technological environment. 2 SCD Strategic Plan

7 The Scientific Computing Division (SCD) is a core facility of the National Center for Atmospheric Research (NCAR). SCD provides a computational environment that balances computing resources with associated services and research activities that are dedicated to the needs of the scientific research community. One of the compelling reasons for establishing a national center over 40 years ago was the need for technological resources and services beyond what is generally available at individual universities; SCD is charged with filling this need. SCD s mission is to serve the computing, data management, and research needs of the atmospheric and related sciences. Consistent with this mission, SCD supports a robust, advanced computational infrastructure that allows the scientific community to address national and international research agendas. Introduction One of five founding goals of NCAR was... to provide, or arrange for provision of, facilities for the scientific community as a whole whose initial cost and upkeep lie beyond the capacity of individual universities or research groups. NCAR Blue Book In anticipation of future needs, SCD also conducts applied research in partnership with the scientific community in the areas of visualization, networking, computer science, numerical methods, and computational science. SCD s current high-performance computing, data service, networking, and operations environment includes: Six supercomputers with an aggregate capacity of 12.1 peak teraflops and a sustained computational capacity of 0.55 teraflops available in a batch computing environment Nearly 100 peak gigaflops of computational capacity dedicated to data analysis, post-processing, and scientific visualization A 55-terabyte research data archive that is recognized worldwide as a high-quality collection of observations and analyzed data developed over nearly 40 years through expert stewardship, careful curation practices, and provision to the research community A Mass Storage System with a capacity of six petabytes that currently holds over 22 million files, totaling more than two petabytes of computational and observational data High-speed, reliable, and secure network connectivity including five campuses, requiring the support of over 150 logical networks, 180 monitored network devices, and more than 4,700 network-attached devices, plus management of and access to additional municipal and wide-area networks SCD Strategic Plan 3

8 A multifaceted physical infrastructure for all the supercomputing facility equipment, along with a professional operations staff that provides support 24 hours a day, 365 days a year In addition, SCD maintains a complementary and appropriately balanced set of service and research activities that include: User support for development, maintenance, usability, and performance optimization of community research codes Data analysis and visualization capabilities that enable scientific discovery Software infrastructure research and development projects aimed at creating new frameworks and updated functionality for use in the geosciences Mathematical research aimed at developing software for novel, accurate, efficient, and scalable numerical dynamical cores for climate, weather, and turbulence applications Development and construction of progressive networking and web services that target data and information access in an effective and efficient manner SCD is a premier scientific computing facility serving a disciplinary community of over 1,100 university, UCAR, and NCAR scientists and other researchers around the world. NCAR Strategic Plan: NCAR as an Integrator A Vision for the Atmospheric Sciences and Geosciences, October 2001 SCD remains committed to providing viable services to users, which includes a diverse university research community, UCAR and NCAR research scientists, and scientists participating in the multi-agency Climate Simulation Laboratory. SCD continually evaluates the scientific computing needs of all users; their input and suggestions for improving services and enhancing computing resources are extremely important. SCD shares the values stated in the NCAR Strategic Plan creativity, innovation, diversity, and excellence in science. SCD acts as an essential integrator in the search for new insights into the interrelated systems of weather, climate, solar, and ocean phenomena. This five-year plan is informed by and responsive to SCD s divisional NSF review, the NCAR Strategic Plan, NCAR s 2004 reorganization, and input from university and NCAR users. 4 SCD Strategic Plan

9 SCD and NCAR face an increasingly challenging environment for highend computing dedicated to research in the atmospheric and related sciences. This rapidly changing environment is comprised of a complex mixture of increasing demands for scientific computing and data storage capacity, new demands for security and resource integrity, deployment of compute and data grids, and the challenge of maintaining the appropriate professional skill set used by SCD s workforce. These challenges are further exacerbated by budgetary and infrastructural limitations. The Challenges Science Driving Demand for Scientific Computing The atmospheric sciences continue to demonstrate an insatiable appetite for high-end computing and data storage resources. For example, the 2003 Community Climate System Model (CCSM) Business Plan envisions a five-year climate science goal, summarized in Figure 1, that requires a 144-fold increase in computational capacity. Figure 1 Estimated increase in computer allocation by the CCSM project as a factor of 2003 usage. The dramatically increased need for computing resources is due to the combination of increases in horizontal and vertical grid discretization, increases in complexity of physical process parameterization, and increases in the number of processes and cycles simulated that are observed in nature but until now were not considered significant enough to affect previous simulations that relied on cruder models. In a fashion similar to many scientific disciplines, the computational needs in the coming years of the CCSM plan will be driven by the need to generate crucial ensemble simulations. This growth rate in scientific simulation substantially exceeds the 18-month doubling rate in computational capability that characterizes the rapid advancement of the computer industry. See management/busplan pdf SCD Strategic Plan 5

10 Data Storage, Management, and Analysis Developed and maintained by SCD, the Mass Storage System (MSS) has grown in size and stature over the past 25 years to become an integral component of the NCAR computing environment. MSS data growth continues at an exponential rate, doubling in size every 18 months. Following Moore s Law, computing technology will continue to accelerate the production of model-generated data, while new observing platforms will produce unprecedented amounts of data that scientists will want to store and analyze. Indeed, the ability to analyze these data is itself a major challenge, requiring new tools, broad collaborative efforts, and extensive use of metadata cataloging services. Finally, scientists will continue to use other supercomputing centers, which may not have the capability to store the resulting data, and will move their data (via Grid and robust network infrastructures) back to the MSS. In its purest form, the challenge is this: How can this potential flood of data, crucial to the success of NCAR s research community, be managed effectively in ways that ensure the continued provision of consistent, high-quality, secure data and analysis services? Expanding centers include: the Australian Bureau of Meteorology, European Center for Medium Range Forecasts, Deutsches Klimarechenzentrum, Meteorological Service of Canada, National Energy Research Scientific Computing Center (NERSC), Oak Ridge National Laboratory, San Diego Supercomputer Center, UK Meteorological Office. Physical Infrastructure The NCAR computing facility is approaching 30 years of age. The power and cooling demands of contemporary high-end hardware are rapidly outpacing the capability and capacity of the existing utilities infrastructure. Meanwhile, the physical capacity of the machine room to hold additional computing and storage hardware is at its limit. Many other centers have either recently expanded their facilities or are planning an expansion in the near future. SCD and NCAR must start now to plan for an expansion and/or relocation of the computing facility. Whatever strategy is adopted, facility planning must begin at least two years ahead of implementation, making this a problem of paramount and immediate importance for SCD and NCAR. An Evolving Cyberinfrastructure Increasingly, scientists want to work in a ubiquitous, interoperable computing environment that transcends organizational boundaries and facilitates collaborations. For this reason, SCD must begin to engage in the development of a fully integrated, secure, Grid-enabled computing and data management environment. In practical terms, this means building partnerships with key scientific computing centers and vendors, devel- 6 SCD Strategic Plan

11 oping new software infrastructure where necessary, and enhancing inhouse expertise in key areas. Deploying Grid technologies is a phased approach. This endeavor begins with integrating existing SCD services, must then extend across the UCAR campus, and ultimately expand to the entire university research community. Human Resources Consistent with changes in computing, storage, networking technology, fundamental research, and provision of services, SCD must continuously modernize and realign the skill sets of the staff. SCD must identify and embrace innovative ways to integrate the strategic goals for the division with meaningful professional growth opportunities for its staff. Financial Environment The core budgets for SCD facilities have not kept pace with inflation over the past eight years, and early indications of the prospects for the future have been characterized as uncertain. Table 1 shows the budget numbers and annual percentage change. SCD s NSF Funding ($million) FY97 FY98 FY99 FY00 FY01 FY02 FY03 FY Table 1 SCD funding, FY1997 FY2004. % Increase SCD has met this fiscal challenge by retooling, rethinking, streamlining, and re-prioritizing many activities. Also, some success in the external funding arena has partially addressed the challenge. Nevertheless, SCD has been cautioned by its NSF review panel to monitor the level of external funding, and to carefully examine the appropriate balance with core funding to assure that external funding is consistent with SCD s strategic objectives. Current levels of service cannot be sustained indefinitely in light of these financial pressures, and the issue of sustaining critical core infrastructure from external sources is problematic. SCD will continue to evaluate the quality of services provided and prioritize services and other (potentially externally funded) projects within these constraints.... computing will become far more complex, but also far more useful, as computational and data grids emerge. NCAR must meet these challenges head-on: it is a responsibility that it bears on behalf of the entire computational atmospheric sciences community. NSF Review of the Scientific Computing Division, Fall 2001 SCD Strategic Plan 7

12 8 SCD Strategic Plan Dealing proactively and imaginatively with these fundamental challenges will enable SCD to continue its leadership role for NCAR and the atmospheric research community. By implementing the strategies discussed in this plan, SCD will enhance the quality and expand the capacity of research, services, and facilities it provides to the university and NCAR research communities.

13 SCD s vision of the future direction of scientific computing at NCAR is defined by the following strategic goals. These goals characterize major themes that recognize the continued provision of critical services while anticipating the future needs of NCAR s research constituencies. The Plan High-End Computing By 2009, SCD plans to achieve a 25-fold increase over current sustained computing capacity. SCD s computing capacity goals are driven by scientific needs and requirements, both of which have grown dramatically over recent years. In the last five years, in response to this challenge, SCD returned the NCAR computing facility to prominence among supercomputing centers through an aggressive investment in IBM cluster systems. In June 1999, NCAR ranked 142nd on the Top500 list and delivered a sustained computing capacity of 20 gigaflops. In early 2004, the available sustained computing capacity had been increased to approximately 500 gigaflops. As shown in Figure 1, the science requirements for computing are projected to continue to outpace Moore s Law. Therefore, SCD plans to continue an aggressive growth rate in sustained computing capacity over the next five years. SCD believes this growth can be accomplished via the accrual of nominal price performance benefits through Moore s Law and by learning to exploit cost-effective systems more efficiently. In early 2004, the NCAR/SCD facility had a peak capacity of nearly 11 Teraflops, and was ranked number 13 on the Top500 ranking of supercomputer centers in the world (see for details). Figure 2 illustrates the historical increase in sustained computing capacity at NCAR. It clearly demonstrates SCD s success in augmenting the computing capacity for scientific research over the past five years. Figure 2 Historical increase in sustained computing capacity at NCAR, SCD Strategic Plan 9

14 SCD plans to vigorously pursue, within the current funding envelope, a strategy that will go beyond Moore s Law to achieve a 25-fold increase in sustained computational capacity (Figure 3). This target will provide more than 12 teraflops of sustained computing capacity to SCD s user communities by Figure 3 Target increase in sustained computing capacity at NCAR, The goal will drive a revitalized program of evaluation and deployment of new computational systems and related technologies. This will include selection of technologies suitable for key applications and integration of major equipment acquisitions with focused efforts to achieve and deploy algorithmic advances. In making new equipment acquisitions, SCD recognizes the importance of evaluating the trade-off between changing architectures to gain computational speed and the work required to harness that speed. SCD will continue to help application scientists optimize codes, and will investigate the usefulness of parallel software that can make scientists more productive. Further, SCD will pursue opportunities for substantial additional funding for computational equipment and infrastructure to support the realization of demanding institutional science objectives. 10 SCD Strategic Plan

15 Petascale Data Management Infrastructure Commensurate with the plan for high-end computing, SCD will enhance its data storage and archiving facilities to unify and advance data management and scientific workflow efficiency. SCD will enhance scientific discovery by providing integrated systems for efficient data management. Data from high-end computing at SCD, modern Earth observing systems, and projects with collaborating centers have taken data management to the petascale level. The data growth rate (see Figure 4), currently at 25 bytes per million sustained floating point operations (~1 terabyte per day), will continue as new computing systems are deployed, and will accelerate as data are ingested from collaborations with other scientific computing centers and new observational facilities. Figure 4 Growth of data holdings in the NCAR Mass Storage System. An enhanced data management infrastructure, built on an existing scalable storage system, is needed to ensure smooth workflow for the scientific community. SCD s response will be to deploy a two-pronged plan. First, to ensure data preservation, SCD will implement a manageable archival growth plan. Second, SCD will develop and support an enhanced data flow system that provides high-performance, low-latency data access to meet the needs of iterative scientific discovery on data collections. Data Archival SCD will preserve critical data from NCAR and its associated research community. SCD will prepare a sustainable MSS growth plan that scales to future needs for Grid-based data management activities, increased SCD Strategic Plan 11

16 local computing capacity, broadened science programs, data derived from modern Earth observing systems, and archival needs for data computed outside of NCAR. To achieve manageable archive growth, the plan will include: Equitable data storage policies that recognize user requirements, yet constrain growth within SCD budget levels Outreach efforts to educate and inform users of policies and procedures Decision support tools for users and MSS administrators New, reliable, cost-effective storage technologies Incorporation of data management systems, closely integrated with high-end computing, that promote data reduction prior to archival storage Techniques to reduce storage while preserving the required numerical precision Data Storage and Management Infrastructure To address a number of data management, distribution, and analysis challenges, SCD will initiate a project to integrate and unify its computational, data management, sharing, analysis, and access facilities. By building on past experience, SCD will combine and enhance its storage resources to provide a data storage and analysis environment that enables more efficient and productive scientific workflow. Integration of SCD s data storage and management infrastructure will be progressive, from evaluation of new system components to their subsequent integration into the operational environment. Emphasis will be placed on the coordination and advancement of all necessary components to provide tightly coupled, scalable, and reliable systems. Essential system components of this data storage environment are: Large-capacity high-performance file systems shared between heterogeneous computers used for modeling, data analysis, visualization, and access High-speed network connectivity Data archive infrastructure support 12 SCD Strategic Plan

17 Figure 5 illustrates this proposed integrated data storage and management infrastructure. It will resemble cache-based memory hierarchies on SCD s supercomputers, attempting to bring the most frequently used datasets closer to the computations. In concert with this data storage infrastructure, metadata cataloging services and data management decision tools will be implemented to facilitate data discovery and management activities such as defining semantic data content and dataset lifetime, locality, accessibility, and replication. High-end Computing Data Analysis Visualization Data Management Figure 5 A schematic view illustrating a high-performance shared file system that brings frequently used data closer to the user and complements MSS archiving. MSS high-performance shared filesystems All sources, from in-house development, commercial procurement, and/ or university collaborations will be evaluated as SCD develops the infrastructure to support these activities. A Grid Capability at NCAR SCD will develop and deploy a Grid-enabled environment to support the community s evolving cyberinfrastructure. SCD intends to support collaborative research through the development and deployment of an integrated environment that is commensurate with its high-end computing and petascale data management goals. This environment includes computing, communications, data analysis, visualization, storage, service, and research elements that will serve users within NCAR, UCAR, as well as various university, national, and international collaborative communities. Specifically, over the next five years SCD will: Build in-house Grid technology, expertise and infrastructure Participate in appropriate partnerships with peer institutions to advance SCD Strategic Plan 13

18 national Grid-based services that provide a capable, high-capacity virtual environment for atmospheric science research Identify a functional Grid-based capability that complements NCAR s discipline-specific orientation Enter the evolving Grid economy as a participant consistent with NCAR s role as a national research center The emerging vision is to use cyberinfrastructure to build more ubiquitous, comprehensive digital environments that become interactive and functionally complete for research communities in terms of people, data, information, tools, and instruments that operate at unprecedented levels of computational, storage and data transfer capabilities. Revolutionizing Science and Engineering Through Cyberinfrastructure (the Atkins Report), January 2003 SCD has already taken several important steps into the Grid arena. As a participant in the DOE Earth System Grid, developers in SCD have demonstrated the infrastructure that allows scientists to access data via the Community Data Portal to enhance access to the high-end computing environment at NCAR. SCD is testing a common batch subsystem facility that allows job dispatch and control in a heterogeneous vendor environment and extends support for Grid environments. To further NCAR s Grid capability, it is strategically important to deploy Grid technologies for data that are interoperable with computational Grid technologies. As a development activity, SCD will pursue the evaluation, selection, and deployment of these technologies to provide high-performance, high-capacity disk storage accessible from all computational and data analysis systems. Networking SCD will provide and evolve the network infrastructure to support advances in UCAR science, technology, and education. The spectrum of strategic directions within UCAR require substantial local, metropolitan, and wide-area network capabilities. These network capabilities include: Calculations on distributed supercomputers and storage, assimilation, and analysis of multi-terabyte datasets Web-based education and outreach, distributed visualization Remote collaboration among UCAR s scientific researchers SCD will continue to evaluate the applicability of emerging high-bandwidth networking capabilities to meet these demands. Additional networking objectives include operational maintainability, adaptability of new protocols and application requirements, anticipation of future growth 14 SCD Strategic Plan

19 and standards, cost-effectiveness, and research into high-performance networking technologies. SCD will assess and monitor Grid technology as it relates to networking, including participating as appropriate in such viable projects as the Earth System Grid and the TeraGrid. A complementary Strategic Plan for UCAR Networking contains a complete summary of specific actions to be taken in support of SCD s goals during the next five years. That plan is designed to implement local, metropolitan, and wide-area networking to support the high-end computing, data management, and collaboration in the SCD Strategic Plan. See docs/reports/stratplan2004/ To this end, fiber that UCAR owns and on which it installs its own hardware to activate services (as opposed to buying services from a provider) has been installed between all UCAR campuses. This UCAR-owned fiber enables the flexibility to support both the current 1 gigabit/second (Gbps) local area-network while anticipating expansion to 10 Gbps and beyond as required. Fiber has also been acquired from Boulder to the Front Range GigaPoP (FRGP) in Denver; this also provides valuable flexibility to increase speed and provide dedicated bandwidth as required. Currently, the FRGP provides access to and from the commodity Internet (>2 Gbps aggregate) and Internet2 Abilene (at 622 megabits/second soon to be 1 Gbps). FRGP will provide access to and from the National LambdaRail (at 10 Gbps) to UCAR and its user community. These services will be expanded and enhanced as required by available national options and as needed by UCAR user applications and services. Networking technology is vital to NCAR and UCAR s ability to function and prosper as an integrator in a rapidly evolving technological environment. SCD will balance operational and research components to provide the best network infrastructure to all users. Securing UCAR s Cyberinfrastructure SCD will secure and protect UCAR s information technology infrastructure from illicit activities. SCD manages a diverse computational and data storage environment encompassing high-end computers, mass storage subsystems, data archives, visualization, , domain name service, authentication, web servers, and networks, including Internet Protocol telephony. Not only are SCD Strategic Plan 15

20 these systems valuable monetarily, they comprise vital scientific research tools and business continuation systems used by the UCAR organization and university communities. When they are at risk, the mission of the institution is at risk. Supercomputing centers are facing increasing challenges in the area of security. Accordingly, SCD will establish, in concert with NCAR divisions, UCAR programs, and the UCAR Computer Security Advisory Committee, an organization-wide computer security policy, implementing procedures that balance the competing needs of an open, collaborative research environment and the security and integrity of the institution s resources. Additionally, SCD will: Place increased importance on computer and network security when acquiring and configuring new equipment (computers, storage, network routers, etc.) Engage in collaborative efforts with peer centers to share cybersecurity information, best practices, and incident notification Ensure that system administration staff (as well as all staff) are appropriately trained and informed of institutional policies and procedures, as well as computer industry best practices Augment existing Intrusion Detection System (IDS) software and hardware, to provide early detection of possible security compromises Physical Infrastructure SCD will expand the computing facility to accommodate NCAR s future computational and data storage infrastructure. Increases in computational power have returned NCAR to a leading position among atmospheric research institutions. They have also pushed an aging Mesa Lab facility beyond its original design limits. Projections for supercomputer technology, coupled with the availability of lower-cost commodity computer systems, indicate a continued increase in power and cooling requirements as clock speeds increase. Thus, within existing funding profiles and the timeframe of this plan, one or more of the power, cooling, or space limitations for the Mesa Lab facility will be exceeded. The infrastructure restrictions of the Mesa Lab site will have a detrimental impact on scientific progress by seriously 16 SCD Strategic Plan

21 constraining SCD s ability to purchase and deploy additional computing capacity. Either a new and/or augmented computational facility is required. SCD is working within the UCAR Facilities planning process to implement the most cost-effective solution for expanding physical infrastructure and to employ best practices for managing and operating its computational and data storage facility. Research Data Stewardship and Curation SCD will expand data stewardship and improve access to its growing, internationally recognized research data archive. The research data required by the general science community are becoming more diverse, require larger amounts of storage, and are more complexly organized than ever before. SCD will adapt to these trends and provide the important data that supports scientific research. SCD currently offers a 55-terabyte research data archive that is recognized worldwide as a high-quality collection of observations and analyzed data. It is critical for science to develop data archives under expert stewardship and data management systems that facilitate access, use, and interpretation. SCD will continue to build the archive and strengthen data discovery and access to it by: Adding to the archive content through systematic data collection Applying data stewardship practices to enhance data quality, improve the associated metadata, and make the archive more useful Preserving irreplaceable data by ensuring secure backup and successful archive migration through media and software evolution Expanding access and discovery for a broader user community by using metadata and data standards, as well as data portal technologies Federating with other national and international centers to form costsaving partnerships for managing large data archives, create increased data availability, and foster open access data policies such as those articulated by the World Meteorological Organization Authorizing the archive content and future growth through proactive user surveys and review from a representative scientific data advisory group SCD Strategic Plan 17

22 Research, Development, and Community Software SCD will track and deploy new technology, perform applied mathematics and computer science research, and develop new software infrastructure. SCD s research and development objective is to provide new solutions that can be applied to the atmospheric and related sciences. SCD s research and development agenda is established and pursued through close collaboration with SCD s partners in the scientific community, in industry, and in SCD s peer centers. The overall goal is to ensure that the end-to-end scientific simulation environment envisioned by the NCAR Strategic Plan for High Performance Simulation is realized in a rapidly changing technological landscape. To pursue this objective requires SCD to engage in research and development activities on several broad fronts. The Panel directs NCAR to develop a plan for increasing their investment in the Computational Sciences Section, emphasizing those areas of applied mathematics and computer science research [and] that the size of the Computational Sciences Section be at least doubled before the next NSF review. NSF Review of the Scientific Computing Division, Fall 2001 Technology Tracking Since supercomputing services are increasingly integrated, SCD will create a coherent and well-integrated technology tracking, evaluation, and deployment program to provide the best facilities possible to the UCAR scientific community. This evaluation and deployment program will be multifaceted, involving technologists, computer scientists, and application experts. It will require SCD to invest in staff dedicated to system evaluation and integration, as well as to acquire experimental hardware and software systems. It will also require a certain tolerance for risk. Research in Applied Mathematics and Computer Science Since there is a relationship between algorithms and computer architecture, SCD will pursue applied numerical research. The goal of this research will be to improve the fidelity, performance, and capability of scientific simulations, and to do this through a combination of novel techniques and efficient implementations. SCD will work in partnership with the scientific community and computational scientists to ensure that research investments are made in areas where grand challenges are perceived and where real progress can be made. At present these include, but are not limited to development in the following areas: Efficient dynamical cores for atmospheric simulations Scalable, mass-conserving advection schemes Adaptive mesh technologies for climate simulation Techniques for simulating multi-scale phenomena 18 SCD Strategic Plan

23 Frameworks for Modeling the Earth System Developing and deploying modeling frameworks is of strategic importance to SCD, as they offer solutions to the problems of software redundancy, consistency, robustness, and performance portability that arise in atmospheric and Earth system modeling. Frameworks offer widespread interoperability and software reuse when they are adopted throughout the community. Thus, a strong partnership and comprehensive support plan with the scientific community is required. The atmospheric science community will benefit most by finding a productive balance between the promotion of community frameworks and the initiation of new, innovative infrastructure projects. In addition, the continued development of new frameworks and alternative implementations of established libraries and frameworks is essential to technical innovation. The Earth System Modeling Framework (ESMF), which is being implemented by a team in SCD, is emerging as the leading national effort to develop geophysical modeling infrastructure. This project, which receives funding from multiple agencies, has become the cornerstone for the strategic plans and development of modeling programs in the Department of Defense, NASA, NOAA, and other organizations. The ESMF team is actively working toward achieving technical compatibility and shared standards for model metadata with its sister European effort, the PRogramme for Integrated earth System Modeling (PRISM). Over the next few years, a broad group of ESMF collaborators will pursue a shared vision for the evolution of the ESMF into an end-to-end modeling environment, the Earth System Modeling Environment (ESME). The ESME will provide a database back end and a run-time environment for the ESMF, allowing users to: Assemble and run applications composed of ESMF components Archive, retrieve, and compare model configurations and output Facilitate scientific queries across published studies, datasets, and models Permit models to be easily configured when switching platforms By linking universities, operational centers, and research centers under a common computational framework, ESMF and ESME will enable the scientific advances of the researcher to be transitioned quickly to key forecasting, prediction, and simulation applications. SCD Strategic Plan 19

24 Software Engineering and Lifecycle Management Most of the areas described above represent significant investments in software. As a major owner and investor in large software systems, and as a partner with other supercomputing sites and NCAR divisions sharing common scientific and software interests, SCD will substantially elevate the priority placed on formal software engineering practices. This effort will emphasize requirements for the long-term viability of a software system, risk assessment and analysis, and software lifecycle management. SCD will use matrix and project management techniques in the development and deployment of new and updated software systems. SCD faces challenges similar to other high-performance computing organizations in dealing with legacy software that fulfills a critical need. SCD will develop an ongoing process for the evaluation of software, including managing or prioritizing its current software infrastructure, along with what is needed over the next decade. Maintaining legacy software involves expenses. Thus, a cost-benefit analysis will be a crucial part of the process. Over time, this will enable SCD to enhance its end-to-end computing environment while maximizing the effective utilization of its software engineering staff. A Knowledge Environment for the Geosciences Working in concert with other community efforts, SCD will advance a portfolio of projects aimed at building future collaborative knowledge environments that span data management, access, analysis, visualization, and collaboration technology. It is becoming increasingly difficult to explore the many interesting scientific questions that hinge on developing new models and understanding data generated by terascale computing coupled with modern Earth observing systems. Addressing the critical path from experiment to discovery and publication is a key thrust area in both the SCD Strategic Plan and the NCAR Strategic Plan. An integrated knowledge development environment spans modeling activities, data and knowledge management systems, data sharing, mining, analysis, visualization, and collaboration. SCD has rapidly increased its participation in these areas over the past several years it represents a key strategic investment, particularly as it reflects NCAR s broad priorities in Information Technology (IT). SCD will develop and deploy critical software infrastructure and related technologies to improve the means by which people from many disci- 20 SCD Strategic Plan

25 plines and different locations can work together, convert data into scientific knowledge, and disseminate knowledge to drive scientific progress. Figure 6 illustrates a knowledge development environment consisting of complex, cyclical processes that span all of the cyberinfrastructure elements that enable scientific discovery. Figure 6 A logical knowledge development environment to support scientific discovery at NCAR. Using a graphical interface, scientists will be able to seamlessly research and access model components, configure selected components into an application, execute the application, archive data locally and remotely, locate and access observational data and results from other applications for comparison, analyze the data, and visualize results. Each of these capabilities already exists in some form the challenge is integrating the number of large, distributed, multi-agency, softwareintensive projects involving data and Grid services. This includes programs for model comparison and evaluation, as well as modeling infrastructure. SCD will collaborate with these various efforts to create a more cohesive and functional knowledge development environment. Data and Knowledge Systems A significant aspect of creating a usable knowledge development environment is advancing the ability to manage, share, and broadly access rapidly growing archives of scientific data. This is a challenge for the SCD Strategic Plan 21

26 larger scientific community and is particularly relevant to the geosciences community. SCD will build on successes with the Community Data Portal and Earth System Grid projects. SCD will offer the institution a cohesive framework to be a world-class scientific data provider for researchers, educators, and the public. Specifically, SCD will: Establish a web and Grid-based data presence that is scalable, costeffective for the institution, and designed to interoperate efficiently with emerging data services worldwide Contribute to community efforts to develop effective authoring, metadata schema, and data management methods Provide common techniques and shared infrastructure to increase data publication by community researchers and NCAR/UCAR Lead the establishment of a global Grid for Earth system science SCD s strategic emphasis in this area reflects an evolution from managing local data files to managing semantic information in the context of next-generation distributed and/or federated knowledge systems. Terascale Data Analysis and Visualization SCD s research community is faced with large volumes of data that are growing rapidly in size, complexity, and diversity. As an integral part of the knowledge development environment, software available for analyzing and visualizing these data is critically important. Unfortunately, the software needed is not keeping pace with modeling and observational activities already underway. Building on a well-established and successful program of providing the community with integrated visualization tools (e.g., the Python interface to the underlying NCAR Command Language, NCL), SCD will address important needs for NCAR s scientific community. Spanning software, outreach, and educational realms, the primary objectives are as follows: Define requirements and understand community needs Develop a modern, open source software framework for analysis and visualization that can effectively handle complex terascale data, couple quantitative and qualitative 2D/3D capabilities, and reduce the level of redundant effort that is occurring across organizations and agencies Develop new methods and algorithms for understanding and visualizing very large, time-varying scientific datasets 22 SCD Strategic Plan

27 Operate and develop a state-of-the-art Visualization Lab and Access Grid collaboration facility, exploring the intersection of these technologies Leverage visualization projects in support of educational and outreach activities Advancing Collaborative Environments for Research Since SCD introduced NCAR s AccessGrid, it has become an operational fabric for conducting science and business at NCAR. Staff members are engaged in collaborative activities with DOE and others to develop and deploy new generations of this technology for the atmospheric sciences community. This strengthens SCD s ability to interact and work with both internal and external collaborators. Evolution of User Services SCD will provide a balanced set of services to enable researchers to easily and effectively utilize community resources. On a regular basis, SCD will review and evaluate all of its user services to ensure they match user needs. New services are essential in an evolving cyberinfrastructure. As they become available, they will be prioritized and offered so researchers can easily, securely, and efficiently use SCD s resources. Input for the evolution of SCD s user and enterprise services will come from a variety of sources, including: SCD user forums, visits to universities, biennial user conferences, user committees, direct contact with principal investigators, and one-onone contact with users via phone or Key SCD User Services: Consulting Documentation Web sites Training DNS Secure access SCD s Advisory Panel Computing in Atmospheric Sciences Workshop, organized by SCD UCAR s Information Technology Committee and its advisory committees Collaborations and information exchanges with peer centers NSF reviews, NSF Geoscience strategic plans, other NSF-sponsored review committees, and nationally recognized reports Birds-of-a-feather sessions at relevant national and international conferences and workshops SCD Strategic Plan 23

28 SCD has established processes and procedures for tracking supercomputing service levels such as reliability, maintainability, serviceability, and system-wide and application-specific performance. To ensure a best practices approach, SCD will enhance these procedures and establish service-level targets, in concert with appropriate committees. Because SCD foresees a continuing evolution in NCAR s computational environment, SCD will guide users through these changes by implementing an outreach program to identify the needs of model development groups using NCAR high-performance computers as well as researchers using NCAR community models at remote locations. To deliver these services effectively, SCD will: Participate and collaborate in model-specific programmer and software engineering meetings Create training programs to increase user knowledge and skill Support major scientific computing and data campaigns Education and Outreach SCD will participate in education and outreach activities that engage diverse audiences and inform constituents of SCD s role in advancing scientific research. SCD and its facilities, particularly its Visualization Laboratory, serve a large and varied set of audiences. These include, among many others, university collaborators, political representatives, policy makers, scientific visitors, and student guests with the NCAR Public Visitor Program. The visibility of SCD s role at NCAR dictates that we inform these audiences via outreach and educational programs, and thereby increase their awareness of SCD s contribution to NCAR and the atmospheric sciences. SCD will collaborate with the UCAR Education and Outreach Office (E&O), the UCAR Office of Communications, and the media to promote its innovations, services, capabilities, and collaborations. Elements of these collaborations will consist of: Publicizing SCD s activities through press releases, journal submissions, web content, technical online news services, exhibit handouts, and visualization material suitable for broadcast media needs 24 SCD Strategic Plan