Scaling out compute resources LOCAL (INTER)NATIONAL

Transcription

1 Scaling out compute resources LOCAL (INTER)NATIONAL Irene Nooren, Patrick Kemmeren (UMC Utrecht) Life Science e-infrastructure workshop, Nov 13, 2014

2 Agenda Why scaling out Scaling out models Work & interest group HPC

3 Scaling out: why? Local (scale up) - deal with (shorter) peak capacity needs - less overhead per group - expertise accessible for embedded bioinformaticians (Inter)National - deal with longer, project specific, peak capacity needs - deal with very specific, not frequently used, hardware requirements - (inter)national consortia

4 Scaling out : how? PC è high-end PC è local cluster è What to do / who to go to? - Build / expand local cluster: scale up - Scale out to clusters in the area or with collaborators - Scale out to (inter)national services, e.g. SURFsara, Grid (SURFsara, Nikhef, RUG-CIT), EGI

5 Scaling out : how? Models to scale out to other services - Requires compatibility between systems, e.g. batch scheduler - Requires good (network) infrastructure - Requires secure network connectivity - Requires agreement on management of resources & operational maintenance, e.g. accounting - Requires knowledge exchange, support

6 Multilayer approach Expertise Collaboration Experiment Application Software services Platform Hosting web & DB services Infrastructure Compute, storage, network Scalability requires - Scalable support - Scalable software - Scalable platform - Scalable infrastructure E.g. PC -> high-end PC -> PC cluster -> high-end compute Integration of service functional level September 23, 2013 DTL Focus Meeting

7 Example: UMC Utrecht HPC UMC Utrecht/UBC - Nineteen research groups from UMC Utrecht, Utrecht University, Hubrecht Institute - Large enough for basic needs - Facilitates/coordinates HPC usage for non-expert users Scale-out to SURFsara (pilot) - Project MinE, 25,000 whole-genome sequences ALS patients - Temporarily (1/2 year) large compute power for recalibration/ processing BAM files - Scale out/collaboration between HPC@UMC Utrecht and SURFsara - Transparent for end-user/project

8 Combine national and local resources Example LISA What maintenance handle requests & access accounting Who National (SURFsara) Local PI (VU) National (SURFsara) support Local PI (VU functional, national ) (SURFsara technical) financed Local (VU)

9 Scale out models model status findings Federated cloud Proof of Concept ongoing In pilot phase Grid infrastructure Production Hard to use; requires training Hybrid cluster-cloud: cluster in cloud Initiation To be piloted With federation of identity management: SURFconext, Terena, Moonshot Available: Life Science Grid

10 Scaling out: cluster in cloud VM is equivalent to cluster OS Cluster storage mounted on VM Can be made completely transparent for the user

11 Scaling out: Cloud federation Moving compute to data On demand virtualized resources Ad-hoc distributed VM provisioning (?) Secure local data mounts

12 Scaling out: Life Science Grid WHAT IT IS Several clusters at academic medical centers and universities. Open to all life scientists based in the Netherlands. Clusters can be used separately or together by using Grid middleware. Currently more than 3700 job slots for concurrent processing. Distributive storage capabilities and automatic replication of data. Connected by lighthpaths to allow fast and efficient data transfers Maintained and supported by SURFsara.

13 Workgroup HPC Initially: NFU onderzoeksdatainfrastructuur WP7 : Integrated and coherent research HPC infrastructuur Morris Swertz (UMCG), Irene Nooren (SURFsara) Choose models based on requirements, possibly different models for different use cases Share expertise Pilots for the selected scale-out models Start an interest group

14 Interest Group HPC Share knowledge - Best practices - Documentation Team up (inter)nationally when possible - Grants - Specific use-cases/pilots Soon to start: DTL/SURF Interest Group High-Performance Compute resources for Life Science research