Where is Ireland in the Global HPC Arena? and what are we doing there? Dr. Brett Becker Irish Supercomputer List College of Computing Technology Dublin, Ireland
Outline The Irish Supercomputer List Ireland on the Top500 HPC in Ireland Today Comparisons & Trends Hardware & Applications (Case Study: Trinity Centre For High Performance Computing) Other adventures in Irish HPC
The Irish Supercomputer List www.irishsupercomputerlist.org @IrishHPCList Started in June 2013 First list November 2013 List updated in June and Nov of each year Roughly one week after Top500 releases Currently on 3 rd List (Nov 2014)
The Irish Supercomputer List Maintenance Committee Kevin Casey (DCU, former UCD, TCD) John Regan (University of Helsinki, former TCD) Mike Salter-Townsend (Oxford, former TCD) Brett Becker (former UCD) Advisory Committee Kostas Katrinis (IBM Ireland) Rolf Riesen (Intel) David Power (Boston Ltd) Simon Appleby (SGI) Dimitrios Nikolopoulos (Queen s University Belfast)
Missions 1. To form a central point for HPC installation operators and users in Ireland across all sectors, public and private, including (but not limited to) academic, research, industry and government. 2. To improve the awareness of HPC amongst the Irish public. 3. To represent and enhance the profile of the HPC landscape domestically and internationally. 4. To serve as a globally-facing resource representing Irish HPC, in keeping with both global and national conventions. 5. To identify current and emerging trends in Irish HPC, which can then be compared to those identified in global lists. 6. To provide a list whose statistics are representative of the Irish HPC landscape, for use by anyone for any purpose.
The Irish Supercomputer List to-date Divided into two lists High Rmax List ( 10 TFlops/s) currently 10 machines Low Rmax List (<10 TFlops/s) currently 15 machines Industry, Research, Academia The list has included two Top500 machines ICHEC Fionn (#358 Nov 13, #476 Jun 14) Anonymous Software Company (#434 Nov 13) Five former Top500 machines ICHEC (Stokes) Anonymous (Web, Gaming x3)
Ireland Global Performance Small number of Top500 machines Makes comparisons with other Top500 countries highly variable Sometimes great, sometimes not great Nov 2013: 2 x Top500 machines 5 th in world for Top500 machines per capita! Nov 2014: 0 Top500 machines Fionn lasted one year #358 Nov 13 > #476 Jun 14 > N/A Nov 14
Ireland Top500 History
Ireland Top500 History
Ireland Top500 History
11-03 05-04 11-04 05-05 11-05 05-06 11-06 05-07 11-07 05-08 11-08 05-09 11-09 05-10 11-10 05-11 11-11 05-12 11-12 05-13 11-13 05-14 11-14 Cores TFlops/s Ireland Top500 History 35000 Ireland Top500 Performance 300 30000 25000 20000 15000 10000 5000 250 200 150 100 50 0 0 Cores Performance (Tflops/s)
India Brazil Russia Taiwan Spain China Poland Italy Belgium Korea, South Saudi Arabia Austria Netherlands Denmark Australia Japan Israel Germany Hong Kong Canada France UK Finland Ireland Sweden Norway Switzerland USA Ireland Top500 History 0.9 0.8 0.7 0.6 0.5 0.4 0.3 0.2 0.1 0 November 2013: Ireland was 5 th globally in number of Top500 supercomputers per capita! Number of Top500 Supercomputers per million Inhabitants
Irish Supercomputer List Nov 14 3 rd iteration 25 machines 13 Bodies (Academia, Industry, Research/Gov t) All-Island #1 ICHEC (Fionn) 140.4 TFlops/s Combined >75,000 CPU cores Combined Performance ~648 TFlops/s
Number of Systems Irish Supercomputer List Nov 14 18 16 14 12 10 8 6 4 2 0 Segment Share? Academia Industry Research/Gov't
Irish Supercomputer List Nov 14 8 7 6 5 4 3 2 1 0 Vendor Share Nov-13 Jun-14 Nov-14
Irish Supercomputer List Nov 14 Many machines are anonymous Mostly industry Submitted to Top500 anonymously (often by vendors) Submitted to ISL anonymously (citing competition, etc.)
Irish Supercomputer List Nov 14 Accelerator use is increasing 28% (7/25) machines are utilising accelerators Currently >106,000 accelerator cores across 7 sites (Phi, Nvidia)
# Systems Irish Supercomputer List Nov 14 8 Accelerator Use 7 6 5 4 3 Phi Tesla Total 2 1 0 Nov-13 May-14 Nov-14
Irish Supercomputer List June 2014 Total CPU cores: almost 70,000 Total Accelerator cores: over 100,000 Total RAM (15 reporting): 73TB Total Storage (10 reporting): 1172TB Most machines (6/23): Trinity College Dublin Top Vendor: Dell 29% (6/21 reporting) Most popular chip: Intel (20/23) Number of machines per sector: Academia(12), Industry(7), Other(4) Total performance: 676TFlops/s Average performance: 30TFlops/s
Ireland June 2014 Top500 List Position Largest Research Computer (EEA Countries on Top500)
Ireland June 2014 RPeak/Capita Largest Research Computer (EEA Countries on Top500)
Ireland June 2014 RMax/Capita Largest Research Computer (EEA Countries on Top500)
Ireland June 2014 Cores/Capita, Largest Research Computer (EEA Countries on Top500)
Where is Ireland Today? Nov 2014 Total CPU cores: > 75,000 Total Accelerator cores: over 106,000 Total RAM (17 reporting): 68TB Total Storage (11 reporting): 1106TB Most machines (6/23): Trinity College Dublin Top Vendor: Dell 28% (7/25 reporting) Most popular chip: Intel (21/23) Number of machines per sector: Academia(16), Industry(7), Other(2) Total performance: 648TFlops/s Average performance: 26TFlops/s
Where is Ireland Today? Nov 2014 Number of Cores in Largest Research Computer By Country (EEA Countries on Top500 + Ireland)
Where is Ireland Today? Nov 2014 Cores/Capita of Largest Research Computer By Country (EEA Countries on Top500 + Ireland)
Where is Ireland Today? Nov 2014 RPeak/Capita of Largest Research Computer By Country (EEA Countries on Top500 + Ireland)
Where is Ireland Today? Nov 2014 RMax/Capita of Largest Research Computer By Country (EEA Countries on Top500 + Ireland)
Where is Ireland Today? Nov 2014 140 Irish Supercomputer List Jun '14 vs Nov '14 120 100 80 60 40 Jun-14 Nov-14 20 0 Cores (x10^3) Accelerator Cores (x10^3) Ram (RB) Storage (TB x10^1) Total Average Performance Performance (TFlops/s (TFlops/s) x10^1) Some metrics see decline due to decommissioning of Stoney (ICHEC)
Comparisons with Russia, India Two other known national lists Top Supercomputers India http://topsupercomputers-india.iisc.ernet.in/ Top 50 Supercomputers Russia http://supercomputers.ru/ China had one at some point
Comparisons with Russia, India 1000 100 10 Ireland India Russia 1 Years in Existence Number of Systems #1 Rmax (Tflops/s) Min Rmax (Tflops/s)
Case Study: TCHPC Trinity Centre for High End Computing Currently 5 machines on Irish SC List Boyle, #8 Parsons, #9 Kelvin, #10 Lonsdale, #11 Crusher, #22
Xeon 6 core (160 cores total) 20 x Phi 2Ghz, 4GB 20 TB Storage 720 GB Ram Infiniband QDR 16.59 TFlops/s Boyle (#8)
Xeon 4 core (1,232 cores total) 3.2 TB Ram Infiniband DDR 8.9 TFlops/s Parsons (#9)
Xeon 6 core (1,200 cores total) 2.4 TB Ram Qlogic Infiniband DDR 9 TFlops/s Kelvin (#10)
Opteron 6 core (1,200 cores total) 2.4 TB Ram Infiniband DDR 9 TFlops/s Lonsdale (#11)
TCHPC Applications Computational Mechanobiology 3D Solar Mass Ejections High Energy Physics Computational Material Science Electron Transport in Electronic Devices and Biological Sensors Molecular Dynamics Acoustic Shielding of Novel Aircraft Configurations High Temperature Superconducting
Other Irish HPC Happenings Amazon EC2 Amazon-built supercomputers Supercomputing in the cloud Cycle Computing Nekomata Naga Megarun
1064 Cc2.8xl instances in US-East (Virginia) Installed in Ireland [1] #1 on ISL is 140.4TFlop s/s [1] http://insidehpc.com/2012/06/amazon-installs-top500-hpc-cluster-in-ireland/
Nekomata (Cycle Computing) 3,809 eight-core (c1.xlarge) AWS instances = 30,472 cores 27TB RAM 2PB storage $1,279/hr or 4c/core Molecular modelling for big pharma weeks on dedicated (existing cluster) ~10.9 core-years Embarrassingly Parallel Nekomata: 7hrs and $9,000
Nekomata ~240Tflop/s? Three AWS regions US West (California) US East (Virginia) EU West (Dublin)
Nekomata EU-West (Dublin) 500 servers 4,000 cores 3.5TB RAM ~31TFlop/s (7th on current ISL is 37.83)
Naga Schrödinger Software/Nimbus Discovery High Resolution Protein Docking 7 million ligands, 21 million conformations
Naga 400 core in-house cluster would require 275 hours (11.5 days) 6,742 EC2 instances 51,132 cores Result ~408TFlops/s 3 hours $4,828/hour Estimated cost of 50,000 core in-house cluster (capital, not running): 10-15 million.
Naga Irish Contribution: 469 servers, 2,360 cores ~19 Tflops/s (#8 on ISL is ~17TFlops/s)
Megarun Identifying organic compounds for photovoltaics, Mark Thompson, University of Southern California 205,000 compounds investigated 156,314 cores, 16,788 instances, 8 regions 18 hours (264 core-years), $33,000 On internal 300 core cluster: 10 months, $132,000
Megarun 1.21PFlops/s ~29 th if ranked on Top500 Irish share ~113TFlops/s (ISL#1 is 140)
To where from here? EU-West (Dublin) is Amazon s second-largest data centre. Together with US-East (Virginia) these two account for ¾ of all AWS usage [2] [2] http://www.datacenterknowledge.com/archives/2013/06/04/how-big-is-aws-newnetcraft-numbers-show-insight/
To where from here? By some counts, EU-West (Ireland) accounts for 17% of AWS servers
To where from here? Nobody really knows how many instances you can get at any one time from any one [Amazon] region. Dave Powers (Cycle Computing) Amazon to Build Data Centres in 'Every Large Country' in Cloud Push (November 2014) [3] What will this bring to (what does this mean for) Ireland as Cloud HPC ramps up? [3] http://gadgets.ndtv.com/internet/news/amazon-to-build-data-centres-in-every-largecountry-in-cloud-push-620181
To where from here? 1. ICHEC, Industry and cloud are the only hopes for Top500-class machines in Ireland.??? Power consumption is too great for individual academic institution 2. Increased accelerator use 3. Sort-term (CPU) core counts are actually going down while performance is going up Good chips: Faster clock speeds, more ops/cycle, larger caches Power a major consideration Accelerators playing larger part
To where from here? What about leveraging the cloud more for HPC a-la Cycle Computing? What are Microsoft, Google and other large data-centre holders doing in Ireland? We at least know a little about Amazon
To where from here? Google Compute Engines are probably in Ireland and Oklahoma [4] [4] https://gigaom.com/2014/03/02/5-things-you-probably-dont-know-about-googlecloud/
Do you know of more systems? University of Ulster? Galway? Cork? Industry? Or more about Google? Amazon? Microsoft? Others? Tell us! info@irishsupercomputerlist.org http://www.irishsupercomputerlist.org/participate/submissions/