What are the major goals of the project?

Transcription

1 The Need for Sustainable High Performance Computational and Data Science in the Extreme Scale Era: A Report to NSF from the Community Brainstorming HPCD Workshops What are the major goals of the project? The BrainstormHPCD workshops were designed to gather information, opinion and expert advice concerning the direction NSF should take to support and further the use of High Performance Computing and Data infrastructure (HPCDi) in the quest for improved scientific discovery and insights for national competiveness in science, engineering, research and commerce. The first workshop the Requirements and Needs Workshop - concentrated on gathering comprehensive information about the needs of the NSF sponsored science and engineering research community for advanced capacity through capability computing, including data- intensive and compute- intensive. The workshop focused on the research community needs at the high end of spectrum and attempted to articulate the value, contributions and impacts of at scale computing for the advancement of science, engineering and other research. The second workshop Implementation Alternatives Workshop - concentrated on alternative operational models for providing computing capability, including the role of private industry, NSF centers, support of science workflows, and data management. This covered both how resources are provided to scientists and how they are managed. Each workshop had six breakout groups that did detailed analysis of the questions above. What was accomplished under these goals (you must provide information for at least one of the 4 categories below)? Major Activities: In all, 57 nationally recognized scientific, engineering and industry experts from 40 different institutions attended one or both of the workshops. 43 white papers were submitted to accelerate discussions and support the conclusions. A repository of all the workshop information and activities can be found at and should be considered as part of this report. Specific Objectives: Several activities in the Federal government are defining the science and research computational and data science efforts and activities for the US national research cyber infrastructure. In particular, in 2014, the NSF initiated a National Research Council report entitled Future Directions for NSF Advanced Computing Infrastructure to Support U.S. Science and Engineering in to anticipate priorities and associated tradeoffs for advanced computing. The committee issued an interim report in fall 2014 and is due to issue the final report. Specifically, the 1

2 committee was asked to report back on a) the contribution of high- end computing to U.S. leadership and competiveness in basic science and engineering and the role that NSF should play in sustaining this leadership; b) the expected future national- scale computing needs; c) complementarities and trade- offs that arise among investments in supporting advanced computing ecosystems; d) the range of operational models for delivering computational infrastructure; and e) the expected technical challenges to affordably delivering the capabilities needed for world- leading scientific and engineering research. Simultaneously, the Office of Science and Technology Policy was developing a government wide plan that would define a government wide strategy for high performance computing, which was announced as the National Strategic Computing Initiative in July The NSCI has five strategic objectives: 1 Accelerating delivery of a capable exascale computing system that integrates hardware and software capability to deliver approximately 100 times the performance of current 10 petaflop systems across a range of applications representing government needs. 2 Increasing coherence between the technology base used for modeling and simulation and that used for data analytic computing. 3 Establishing, over the next 15 years, a viable path forward for future HPC systems even after the limits of current semiconductor technology are reached (the "post- Moore's Law era"). 4 Increasing the capacity and capability of an enduring national HPC ecosystem by employing a holistic approach that addresses relevant factors such as networking technology, workflow, downward scaling, foundational algorithms and software, accessibility, and workforce development. 5 Developing an enduring public- private collaboration to ensure that the benefits of the research and development advances are, to the greatest extent, shared between the United States Government and industrial and academic sectors. It also empowers NSF as one of three co- leading agencies for the NSCI with particular focus in scientific discovery advances, the broader HPC ecosystem for scientific discovery, and workforce development. Significant Results: In the Spring 2015, community BrainstormHPCD workshops were held to identify the on- going requirements among open science, engineering and research communities for future high performance computational and data science (HPCD) resources and services. These workshops identified a persistent, broad and deep need for high performance resources and services to enable leading- edge science, engineering and research investigations as well as to maintain a competitive national economy and research enterprise. The workshops provided insight into the changing requirements of pioneering investigations and explored alternative methods to provide resources to the research communities. 2

3 Key Outcomes of other Achievements: The consensus view of the workshop attendees is captured in the following high- level statement: Because NSF has the broader, national mission of advancing science and engineering and ensuring the scientific and economic competitiveness of the nation, it is vitally important NSF provide a series of HPCDi Track 1 and Track 2 systems and services over time to meet the increasing demands of high spectrum digital science with increased funding and coherence. Outcomes and a summary of the workshop white papers and findings were submitted to the NSCI Request for Information and these summarize important results and conclusions of the two BrainstormHPCD workshops, entitled The Need for Sustainable High Performance Computational and Data Science in the Extreme Scale Era: A Report from the Community Brainstorming HPCD Complete information and detailed white papers are found at What opportunities for training and professional development has the project provided? The workshops offered the opportunity for a diverse set of computational and data analytics scientists to learn about other disciplines and methods being used by other teams. It provided a forum for academic and industry leaders to understand their needs and requirements. How have the results been disseminated to communities of interest? The materials developed for the workshops are available at: Other Conference Presentations / Papers Summaries of the workshop have been presented at meetings and other workshops. Examples include an extended presentation and discussion at the NITRD High End Computing (HEC) Interagency Working Group (IWG) (March) prepared and submitted but presentation cancelled, 2015 Blue Waters Symposium (May), NCSA Private Sector Program Workshop (May), 2015 NCSA/Kisti Joint Laboratory Meeting (July), and other forums. Other Products The workshop report and information was submitted as a white paper to the NSCI RFI call. 3

4 Other Publications All whitepapers and materials are available on the BrainstormHPCD web site at and should be considered as part of this report. Websites BrainstormHPCD web site is active and maintained. Workshop materials are collected and updated at Participants/Organizations Workshop Contributors and Attendees Stuart Anderson, LIGO/Caltech; Richard Arthur, GE Global Research; Dinshaw Balsara, University of Notre Dame; Bill Barth, TACC; Klaus Bartschat, Drake University; Greg Bauer, NCSA/University of Illinois; Cristina Beldica, NCSA; Jerzy Bernholc, North Carolina State University at Raleigh ; Martin Berzins, SCI Institute University of Utah; Wes Bethel, LBNL; Tom Cheatham, University of Utah; David Dixon, University of Alabama; Thom Dunning, Univ of Washington; Said Elghobashi, University of California, Irvine; Jim Fonseca, Purdue University; Steven Gottlieb, Indiana University; Rama Govindaraju, Google; Bruce Harmon, Iowa State University; Anna Hasenfratz, University of Colorado Boulder; Thomas Hauser, University of Colorado Boulder; Victor Hazlewood, University of Tennessee JICS; Curtis Hillegas, Princeton; Peter Kasson, University of Virginia; Fatemeh Khalili- Araghi, University of Illinois at Chicago; Kevin Kissel, Google; William Kramer, NCSA/University of Illinois; John Levesque, Cray Inc; Mike Levine, PSC; David Lifka, Cornell University; Honggao Liu, Louisiana State University; Paul Mackenzie, Fermilab; Pieter Maris, Iowa State University; Michael Norman, University of California San Diego; Steve Oberlin, NVIDIA; Nikolai Pogorelov, University of Alabama in Huntsville; Thomas Quinn, University of Washington; Jagnnathan Ramanujam, Center for Computation and Tech LSU ; Ralph Roskies, Pittsburgh Supercomputing Center; Mark Scheel, California Institute of Technology; Barry Schneider, NIST; Ed Seidel, NCSA/University of Illinois; Todd Simons, Rolls- Royce Corporation; Dan Stanzione, University of Texas at Austin; John Towns, NCSA/University of Illinois; Frank Tsung, UCLA; Jorge Vinals, Minnesota Supercomputing Institute; Shaowen Wang, NCSA/University of Illinois; Liqiang Wang, University of Wyoming; Nancy Wilkins- Diehr, San Diego Supercomputing Center; Paul Woodward, University of Minnesota; Steve Wolff, Internet2; Donald Wuebbles, University of Illinois; Pui- kuen (P.K) Yeung, Georgia Institute of Technology; Shiwei Zhang, College of William & Mary What individuals have worked on the project? Workshop Organizers William Kramer, Ed Seidel, Cristina Beldica, John Towns and Gregory Bauer Workshop Funders National Science Foundation, National Center for Supercomputing Applications 4

5 Editor and Contact Author William Kramer National Center for Supercomputing Applications University of Illinois 1205 W Clark Street Urbana, Illinois wtkramer@illinois.edu In addition to the participants listed above, Michelle Chappel handled logistical arrangements. Susan Vinson arranged travel and reimbursements for attendees. NCSA supports the BrainstormHPCD website. Impacts What is the impact on the development of the principal discipline(s) of the project? The workshops highlighted the pressing need and motivation for NSF to provide a continuity in high performance computing and data analysis resources in the future in order to continue the nation s science and economic competitiveness. What is the impact on other disciplines? The workshops highlighted the future science and research High Performance requirements across a range of disciplines including physics, astronomy, bio- medicine, chemistry, materials and others. What is the impact on physical resources that form infrastructure? The report findings are useful to NSF and others for planning future resource investments and services. Public Outcomes: The consensus view of the workshop attendees is Because NSF has the broader, national mission of advancing science and engineering and ensuring the scientific and economic competitiveness of the nation, it is vitally important NSF provide a series of High Performance Computing and Data (HPCD) Track 1 and Track 2 systems and services over time to meet the increasing demands of high spectrum digital science with increased funding and coherence. The reasons for this conclusion include the following workshop findings. 1. Past sustained funding for HPCD has led to tremendous advances in many fields and NSF cannot let that progress and momentum stop. 2. The success and pervasiveness of NSF s HPCD investments created an environment in which access to such HPCD resources are now assumed by the entire research enterprise. 3. There is compelling evidence that computational and data intensive computing is now implicitly intertwined with all areas of discovery and competitiveness. 5

6 4. The demand for high performance computing and data science resources is growing and will continue to grow over the next decades to enable productive and timely science and research progress. 5. NSF and other agencies have succeeded in increasing the use and demand for HPCD resources by expanding the impact of traditional communities and broadening the engagements with new communities. Further, workforce and research development efforts have enabled more researchers to make efficient use of high performance resources. 6. Scientific needs increasingly require high performance capability for both modeling and simulation and for data science and science, engineering and other types of research. There is an increasing symbiotic relationship between high performance modeling and simulation and high performance experimental/observational data science. Today, the most challenging experiments and instruments cannot fully understand their data without modeling and simulation guidance and verification. Likewise, modeling and simulation projects have intimate engagements with experimental and observed data. 7. The NSF National Cyber Infrastructure must include systems of the scale and capability that match the best systems the major national laboratories are deploying. 8. NSF should evaluate and develop funding methods and/or programs for advanced HPCD that recognizes the need for coherence and continuity for suppliers, providers and consumers of these capabilities in order to enhance the productivity of the scientific and engineering research endeavors. 9. There is a need for a variety of platforms and service models to be efficiently addressing the breadth and depth of the national open- science research endeavors. While diversity is important, there is a point where dropping below a minimum size/capability of deployed national resources is counter productive and inefficient for the national science communities, the funders and the stakeholders. 10. While there are, and will be, cases for cloud based resources for pursuing research, the workshop conclusion, which included commercial cloud architects, is that cloud computing methods are limited and cannot address the breadth nor the depth of the national open research portfolio. Even if there was a sufficient business case for commercial cloud HPC providers, there was consensus that those services would be limited to applications that use, at best, no more than a few thousand to a few tens of thousands standard cores of computing for the foreseeable future. 11. Many frontier science applications fundamentally need much more scale to the order 100,000 s to 1,000,000 s of cores to make frontier and best of breed problem solutions feasible. Additionally, these problems require systems with robust and performant interconnects, storage, I/O services, and large amounts of memory in balanced architecture. 12. There are emerging, curated petabyte scale data repositories with thousands of people accessing them for both observational data and simulated data.. The need for sharing and reanalysis of data will increase and NSF should invest in 6

7 resources to support community based, curated large data repositories with analysis capability to be used by 100 s to 1,000 s of scientists. 13. An increasingly important requirement for new and existing research communities is to enable complex workflows that are converging to be simultaneously computation- intensive and data intensive in nature. 14. There exists a compelling need to invest in software development and improvement programs. In particular, NSF should be investing in HPCD university programs related to the computational science curriculum in addition to funding specific applications. Some driving reasoning for the need for SW improvement include: 15. Industrial usage of HPCD resources to perform science and engineering work at scale has important and competitive impacts. NSF should pioneer allocation pathways for non- discovery use for high- end systems is critical for economic impacts. 7