PRACE: World Class HPC Services for Science F. Berberich, Forschungszentrum Jülich, February 2012, PRACE Workshop on HPC Approaches on Life Sciences and Chemistry, Sofia, Bulgaria
Overview PRACE AS A RESEARCH INFRASTRUCTURE PRACE AS A EUROPEAN PROJECT PEER REVIEW AND ACCESS USER SERVICES TRAINING
Why do we need PRACE? Money: No one system fits all problems Quality: Scientific Excellence: European Peer Review Tier-0: HPCsystems of the highest performance class Access: guarantee independent access for all computational scientists and European industry Sustainability: EU and national governments are motivated through PRACE to establish robust and persistent funding schemes 3
PRACE AS A RESEARCH INFRASTRUCTURE
PRACE History An Ongoing Success Story HPC part of the ESFRI Roadmap; creation of a vision involving 15 European countries Creation of the Scientific Case HPCEUR 2004 2005 Signature of the MoU HET 2006 PRACE Initiative 2007 Creation of the PRACE Research Infrastructure PRACE RI 2008 2009 2010 2011 2012 PRACE Preparatory PRACE-3IP PRACE-1IP Phase Project PRACE-2IP 5
Realizing the ESFRI Vision for a HPC RI Tier-0: European Centres for Petaflop/s Tier-1: National Centres Tier-2: Regional/University Centres Creation of a European HPC ecosystem HPC service providers on all tiers Scientific and industrial user communities The European HPC hard- and software industry Other e-infrastructures Tier-0 European centres Tier-1 National centres Tier-2 Regional/University centres # of systems capability European HPC-facilities at the top of an HPC provisioning pyramid
April, 23rd 2010 creation of the legal entity (AISBL) PRACE with seat location in Brussels, Belgium 24 PRACE Members 68+ Million from EC FP7 for preparatory and implementation Grants INFSO-RI-211528, 261557, and 283493 Complemented by ~ 50 Million from PRACE members Interest by Denmark, Israel,
Three pillars of the PRACE Mission HPC for Industry HPC for Science HPC by Vendors Guaranteeing in Europe the independent access to HPCcompetence for industry Implement the ESFRI vision of a European HPC-service at the top of an HPC provisioning pyramid Helping European (hard& software) vendors to foster their technology and HPCcompetence
PRACE Tier-0 Capability and Support Accumulated Tier-0 performance > 15 Pflop/s in 2013/14 PRACE includes 18 Tier-1 systems with accumulated capability of > 2 PF (building on DEISA / DECI) PRACE provides capability support competence centres over several sites 9
Provision of Capacity and Access Binding commitments: France, Germany, Italy, Spain GENCI GCS CINECA BSC Each: 100 Mio over 5 years in terms of cycles Contribution accounted as TCO Access strictly by peer review at a European level Calls: 5/2010, 9/2010, 2/2011... incl. DECI call Start of provision: 8/2010 10
Tier-0 PRACE is building the top of the pyramid... Tier-1 Tier-2 First production system available: 1 Petaflop/s IBM BlueGene/P (JUGENE) at GCS (Gauss Centre for Supercomputing) partner FZJ (Forschungszentrum Jülich) Italy and Spain expect to deploy their own Tier-0 systems from 2012. Second production system available: Bull Bullx CURIE at GENCI partner CEA. Full capacity of 1.8 Petaflop/s reached by late 2011. Fourth production system available by mid 2012: 3 Petaflop/s IBM (SuperMUC) at GCS partner LRZ (Leibniz-Rechenzentrum). Third production system available by the end of 2011: 1 Petaflop/s Cray (HERMIT) at GCS partner HLRS (High Performance Computing Center Stuttgart). Upgrade to 45 Petaflop/s planned in 2013.
Organization Council Chair: Achim Bachem
Scientific Communities in the Driver Seat Scientific Steering Committee Gives opinions on all matters of a scientific and technical nature Proposes the members of the Access Committee Members (Feb. 2012): Richard Kenway (Chair), José M. Baldasano, Kurt Binder, Paolo Carloni Giovanni Ciccotti, Sylvie Joussaume, Ben Moore, Gernot Muenster, Risto Nieminen, Modesto Orozco, Maurizio Ottaviani, Michele Parrinello, Olivier Pironneau, Thierry Poinsot, Simon Portegies Zwart, Kenneth Ruud, Wolfgang Schroeder, Christof Schütte, Luis Silva, Alfonso Valencia.
Scientific Communities in the Driver Seat Access committee Gives advice on the scientific use of the Tier-0 Infrastructure Provides recommendations on the allocation of PRACE resources based on the Peer Review process Members (Feb. 2012): Kenneth Ruud (Chair), Roberto Capuzzo Dolcetta, Peter Nielaba, Manuel Peitsch, Andreas Schaefer, JeanClaude Andre and Hester Bijl. 14
Scientific Communities in the Driver Seat User Forum Open to all scientific and industrial user communities Main communication channel between HPC users and PRACE AISBL Interaction with members of the PRACE AISBL Discussion and issuing recommendations to PRACE AISBL Promoting HPC usage Fostering collaborations between user communities Turlough Downes PRACE User Forum - Dublin at the Bedford Hall in Dublin Castle March 14, 2012
PRACE AS A EUROPEAN PROJECT
Preparatory Project 16 Partners (14 countries), funding 10 Million Coordination: FZJ/JSC 1.1.2008 30.6.2010, successfully finished Review October 1st, 2010 in Brussels 17
Work Packages WP1 Management WP2 Organizational concept Statutes WP3 Dissemination, outreach and training WP4 Distributed computing WP5 Deployment of prototype systems WP6 Software enabling for prototype systems WP7 Petaflop/s systems for 2009/2010 WP8 Future Peta to Exaflop/s technologies
Example: Categorisation of Applications (2009) Other 5.8 Plasma Physics 3.3 Surveys of PRACE Computational partners HPC systems Engineering 3.7 and major applications Life Sciences 5.3 Astronomy & 24 systems and 69 Cosmology 5.8 applications Earth & Climate 7.8 Quantitative basis for selecting representative CFD 8.6 applications Particle Physics 23.5 Computational Chemistry 22.1 Condensed Matter Physics 14.2 19
1st Implementation Project 21 partners (20 countries), funding 20 Million by EC Coordination: FZJ/JSC 1.7.2010 30.6.2012, running Strong focus on application scaling in cooperation with user groups 20
2nd Implementation Project 22 partners (21 countries), funding 18 Million Preparation/Coordination: FZJ/JSC/PRACE PMO 1.9.2011 31.8.2013, running Scaling of important user codes Tier-1 Integration (DEISA PRACE) 21
3nd Implementation Project Funding 20 Million Under Preparation: FZJ/JSC/PRACE PMO Mid 2012 Mid 2014 Planned: Pre-commercial procurement exercise Planned: Industry application focus 22
Installed Petaflop Prototypes in Preparatory Project IBM BlueGene/P (FZJ) 01-2008 (MPP) IBM Power6 (SARA) 07-2008 Cray XT5 (CSC) 11-2008 IBM Cell/Power (BSC ) 12-2008 (CEA/FZJ): Intel Nehalem/Xeon installation date April 2009 IBM Cell/Power (BSC ) 12-2008 23
Prototyping is Mandatory for PRACE Identification of User Requirements Assessment of emerging Technologies Recommendations for: - Procurements - Deployment of mature Software Technologies Technology Watch Joint Developments with Vendors Recommendations for: further Developments Prototyping is a mandatory step in the selection and deployment of new technologies Prototyping is a vehicle for cooperation with technology providers 24
Prototypes: Final Selection Nr. Title Partners 1 Exascale I/O 2 Novel MPP-Exascale system I/O concepts Interconnect Virtualization for Scalable Heterogeneous Platforms CEA, CINES, BSC, STFC, JUELICH, HLRS JUELICH, ETHZ, (IBM) 3 ETHZ, CaSToRC, JUELICH, EPSRC, HLRS, UYBHM, CINECA 4 NUMA-CiC: Numascale Cachecoherent Inter-Connect for Exascale Clusters 5 Benchmarking Energy-to-Solution on LRZ, JKU, BSC, PSNC, future hardware platforms KTH Total UiO, GRNET, CSC, JUELICH, CaSToRC, PSNC Total Costs 700 000 472 000 (+ 528 000 ) 60 000 400 000 (+ 250 000 ) 800 550 2 432 550 25
PEER REVIEW AND ACCESS
PRINCIPLES Transparency Prioritisation Expert assessment Managing interests Confidentiality No parallel assessment Right to reply with national organizations Ensure fairness to the science proposed
PRACE - Peer Review 28
Types of Access Preparatory access only technical peer review prepare for project access, optionally with PRACE support Project access both technical and scientific peer review 12 month allocations, 2 calls per year: watch www.prace-ri.eu Programme access (to come) both technical and scientific peer review
Types of Access Preparatory Project Programme Technical Assessment Yes Yes Yes Scientific Assessment No Yes Yes Additional Assessment No No Potentially Mid-term Review No No Potentially Duration 6 months 12 months 24 months Final Report Technical General General 30
Preparatory Access Types Type A Code scalability testing to obtain scalability data Assessment of applications using a light-weight application procedure Maximum allocation for type A e.g. 100,000 core hours on JUGENE and 50,000 core hours on CURIE Maximum allocation time: 2 months Type B Code development and optimisation by the applicant using their own personnel resources Applicants need to describe the planning for development in detail Maximum allocation e.g. 250,000 core hours on JUGENE and 200,000 core hours on CURIE Maximum allocation time: 6 months Type C Code development with support from experts from PRACE Maximum allocation e.g. 250,000 core hours on JUGENE and 200,000 core hours on CURIE Maximum allocation time: 6 months 31
Received Proposals Early Access Call,1st-3rd regular call: 223 proposals 32
60 Awarded Proposals 33
Success Ratio 34
Peer Review: Next Calls 5th regular call April 17, 2012 Call Opened May 30, 2012 Call closed November 1, 2012 Tier-0 access for successful applicants Preparatory Access Ongoing process. It is continuously open, with a cut-off every three months. 35
USER SERVICES
User Services a Project activity Evolution of PRACE Common Production Environment (PCPE) Helpdesk User Documentation Applications Monitoring Advanced Application Production Assistance 37
PRACE Services PRACE Common Production Environment Common module environment deployed at all PRACE sites First iteration based on DEISA DCPE Standard module environment for all PRACE users regardless of architecture 38
PRACE Services - Helpdesk Central PRACE Helpdesk instance Central helpdesk for all issues Based on very successful DEISA model and tool Primary user interface via web-interface Secondary interface via email support@prace-ri.eu Site specific email also available e.g. cea-support@prace-ri.eu Queues configured for each Tier-0 and Tier-1 site Additional internal queues for PRACE RI services WP6 provides resources to staff the Helpdesk Helpdesk on duty rotates through partners Similar process as used for DEISA 39
40
PRACE Services Applications Monitoring Central service for monitoring status of services Status of services on both Tier-0 and Tier-1 sites Includes monitoring of PCPE Based on very successful DEISA INCA model Single instance for both Tier-1 and Tier-0 Concentrate development effort on one excellent central monitoring service Central monitoring infrastructure hosted at LRZ Local components at host sites 41
PRACE Services User Documentation Each subtask leader owns User Docs for that area Network, Monitoring, Data, Compute, AAA, User Services, Generic Advanced Application Production Assistance Support for new Tier-1 Users Similar concept as was used for DEISA DECI Site visits to new users Direct assistance to 42
TRAINING
Training in PRACE 44
PRACE-2IP T4.1 Carry out Training local infrastructure. Survey to identify requirements AND existing T4.2 Establishment of PRACE (PATCs). T4.3 Continue the series of EU/US (Dublin, June 2012). T4.4 Organise training events (4 seasonal schools, 2 community-targeted workshops). T4.5 Additional material & build on the Training Advanced Training Centres Summer Schools Portal. 45
Training Surveys (users & trainers) 416 users (330 complete). 205 trainers (166 complete). Identified areas of HPC training to be prioritised (e.g. performance analysis, optimisation, debugging techniques). PRACE is in an ideal position to complement local training activities (through PATCs). Established database of HPC trainers who may be available to implement PRACE training. 46
PRACE Advanced Training Centres Establishment of 6 PRACE Advanced Training Centres (PATCs) in 2012. Barcelona Supercomputing Center (Spain) CINECA - Consorzio Interuniversitario (Italy) CSC - IT Center for Science Ltd (Finland) EPCC at the University of Edinburgh (UK) Gauss Centre for Supercomputing (Germany) Maison de la Simulation (France) 47
PRACE Advanced Training Centres All centres responsible for developing and delivering a coordinated programme of training events (designed annually). Available to researchers and scientists from all over Europe free of charge. Some events targeted at specific communities; some targeted at developing HPC trainers. Build up a repository of material for the PRACE Training Portal. 48
PRACE Seasonal Schools Feb 6-10, Bologna, Italy May 16-18, Cracow, Poland http://www.prace-ri.eu/prace-training-events 49
EU-US Summer School, Dublin 2012 Date: 24-28 June Application deadline: 18 March Open to all European-based scientists, acceptance to school is competitive. PRACE covers accommodation & meal expenses. European-based applicants to source travel funding from home institution. Collaboration between PRACE & XSEDE (35 EU students, 25 US students). https://www.xsede.org/web/summerschool12/home 50
PRACE Training Portal www.training.prae-ri.eu 51
PRACE Training Portal 52
30.8.2011 Thomas Lippert - FZJ 53
54