Grid and Cloud Computing at LRZ Dr. Helmut Heller, Group Leader Distributed Resources Group

Transcription

1 Grid and Cloud Computing at LRZ Dr. Helmut Heller, Group Leader Distributed Resources Group

2 Overview Grid: What is Grid computing? Advantages of Grid computing (why you should use it) Certificates (and how to get them) Globus at LRZ data transfer with Globus Transfer (part of Globus Online) and GridFTP login with GSISSH-Term Cloud: What is Cloud computing? Advantages of Cloud computing (why you want to use it) The LRZ Compute Cloud How to get access How to use it 2

3 Motivation Common problems of HPC-center users: A single user computes on more than one HPC computer Authentication is cumbersome: usernames and passwords have to be entered over and over again (and they differ from site to site!), security! Data have to be transferred manually from one site to another (which can be very confusing) The user has to master different interfaces, e.g., file systems (quota, policies, etc.), queuing systems (NQS, LoadLeveler, etc.) are different at each site How to get access to distributed resources, e.g., EGI, PRACE, LCG GRID-computing tries to solve these problems so that the user can concentrate on science and is not distracted by computer idiosyncrasies 3

4 What Is Grid Computing? User at home Middleware: Globus, UNICORE, glite,... Resources, world-wide

5 Advantages of Grid Computing (Why You Should Use It) GRID computing simplifies the usage of distributed resources: No more cryptic user names like di98azf Single Sign-on! GRID computing is secure: certificates are used instead of username/password GRID computing provides a uniform, simple, and easy-to-learn interface to shared, distributed, coupled resources GRID-computing introduces a new layer of abstraction: The user no longer has to fight with the idiosyncrasies of each computer system The user interacts now with the GRID as a whole and not with individual components Grid tools are tuned for high performance (HPC)! Can be by anyone with a Linux Cluster or SuperMUC account 5

6 Certificates: How They Work An X.509 certificate is your personal ID card and key to the Grid world authentication: you can prove who you are public and private key; certification authorities are trust roots to boost security, only short-lived (12h) copies (proxies) are passed on 6

7 Certificates: How to Get Them LRZ MyProxy-CA: myproxy.lrz.de fully integrated with Globus Online all you need to generate a certificate is an LRZ SIM account (TUM/LMU ok) downside: these certs are only trusted by LRZ Short Lived Credential Service SLCS of DFN: all you need to generate a certificate is an LRZ SIM account (TUM/LMU ok) downside: only valid for up to 7 days Real (long-lived, world-wide valid) certificates through LRZ RA: you need to come by LRZ and show your passport and prove your affiliation free of charge valid for 1 year 7

8 Grid Data Transfer with Globus Transfer Example: you want to transfer a (big) file from the LRZ Linux Cluster to your laptop we will use the LRZ myproxy-ca Setup (only needs to be done ONCE and then never again) install Globus Connect Personal on your laptop: follow instructions at connect your LRZ account with your certificate follow instructions at basically enter your distinguished name into LRZ SIM 8

9 Grid Data Transfer with Globus Transfer Actual transfer: Select the pre-defined endpoint at LRZ#lxlogin1 Select your local endpoint Globus-userID#endpointname Select the file(s) to transfer and the target directory Click on transfer! 9

10 Grid Data Transfer with Globus Transfer Advantages: With Globus Transfer you get high performance data transfer higher throughput than scp! Even if the transfer is interrupted, it will be continued automatically once the connection returns Fire and forget: you can now close your browser and the transfer will continue Once it is done (or if there is a non-recoverable problem) you will receive an notification Single Sign-On: no username or password needs to be given world-wide access to SuperMUC $HOME through gridmuc.lrz.de, port 2811with GridFTP/Globus Transfer fully scriptable Command line version: GridFTP (globus-url-copy) from Globus Toolkit 10

11 Login with GSISSH-Term Instructions at Works on Mac, Linux, Windows Java based (you need Java SE 8 installed!) Automatic software updates Click me Java WebStart feature: just click it! (Mac Safari users must double-click downloaded jws.jnlp) 11

12 Login with GSISSH-Term Advantages: Single Sign-On: no passwords, no account names multi-platform: Mac, Linux, Windows secure world-wide access to SuperMUC through gridmuc.lrz.de on port 2222 Command line version: gsissh from Globus Toolkit fully scriptable packages already included in most Linux distributions if not, look at 12

13 Grid Computing Contacts To contact our registration authority (RA), e.g., to make an appointment for a personal visit, send an to LRZ-GRID-RA grid-ra@lrz.de In case of trouble, you can always contact us: open an LRZ trouble ticket through our service desk: send us an LRZ Grid Support grid-support@lrz.de 13

14 What Is Cloud Computing? Basically, like a rain cloud obstructs your view, in Cloud computing you don t know which resources you are really using they are hidden in a cloud Formal Cloud definition by NIST: On-demand self-service Broad network access Resource pooling (multi-tenant model) Rapid elasticity Measured Service (pay-as-you-go) All of the above in combination Use of virtualization 14

15 IaaS, PaaS, SaaS Dropbox Amazon, Azure, Google, SoftLayer 15

16 The LRZ Compute Cloud: What Is Behind It In total: 640 cores, 7 TB RAM, 300 TB disk space Intel Xeon E5540, 2.53GHz, 8MB cache Per VM: up to 8 cores, up to 64 GB RAM Infrastructure as a Service (IaaS) with Cloud middleware OpenNebula 16

17 Advantages of Cloud Computing (Why You Want to Use It) Flexibility! run your own VM images (e.g., prepared by other researchers) use whatever operating system you want (Ubuntu, Debian, Red Hat, ) self service: start a VM quickly whenever you want startup time: 2 minutes, also on Saturday at 11pm use the resources (CPUs, RAM, disk) that you need not more, not less elasticity: add more resources exactly when you need them Inbound and outbound network traffic free of charge Backup to TSM Data sharing within the LRZ Cloud is possible Added security through 4 security zones internet (no firewall!), MWN, your VMs, no network 17

18 Advantages of Cloud Computing (Why You Want to Use It) Close integration with other LRZ services: LAMBS, TSM, Hosted in Germany by trusted German entity (LRZ) respecting German data protection laws no data to NSA; US patriot act not applicable Tutorials, training classes, IT support available Domain support for astrophysics, life sciences, geo sciences, and energy research Friendly user phase March to August 2015 service is free of charge afterwards: competitive pricing 18

19 Advantages of Cloud Computing (Why You Want to Use It) Typical usage scenarios: using operating systems not supported by LRZ quick (turn around in minutes, not days) tests that require a clean slate and root access usage of ready-made software packages in the form of virtual machine images platform for students to have their own system, yet no need for physical hardware (diploma thesis, practicals, classes, teaching) platform for Big Data applications (Hadoop as PaaS pre-installed) Cloud bursting: using a lot of resources for a short time (poster deadline!) 19

20 Caveats of Cloud Computing YOU are responsible for the security of your VMs install and maintain your own firewalls keep your OS patched and secure change all passwords (do not use default PWs!) We will do port scans of your VMs! If you get hacked and you were negligent, we will kick you off the Cloud Virtualization adds an overhead, so VMs are slower than bare metal No guaranteed service quality: best available effort 20

21 The LRZ Compute Cloud: How to Get Access You need to have an LRZ account: tell us your LRZ ID! Send to and describe what you want to do in the Cloud (science) why you can t use other LRZ compute services instead what your technical background is (admin, user, novice) what resources you intend to use (# CPUs, CPU-h, RAM, disk, ) who is your Master User who will pay for your Cloud usage (we will contact them!) We will enable your account and give you a budget ( pre-paid model) 21

22 The LRZ Compute Cloud: How to Use It Read the manual at Work through the tutorial Cloud for Dummies - a Short Tutorial at Go to to work with the Cloud Shut down your VMs once you don t need them any more 22

23 Cloud Computing Contacts Access the Cloud: Read the manual: In case of trouble, if you have questions, or if you want to start using the Cloud, you can always contact us: open an LRZ trouble ticket through our service desk: send us an LRZ Cloud Support 23