- from Clusters and Grids to Cloud computing

Transcription

1 Lecture - 1 Moving Ahead - from Clusters and Grids to Cloud computing Salman Toor salman.toor@it.uu.se

2 Basic questions Why Cloud computing? What are the previous technologies? What was missing in the previous technologies? Will previous technologies be substituted? Can legacy applications run on Cloud platforms? 2

3 Were supercomputers the only source of large scale computing before Clouds? ANSWER: NO 3

4 Cluster Computing Accessible via Local Area Network (LAN) Grid Computing Based on Wide Area Network (WAN) Cloud Computing Next generation computing model Distributed Computing Infrastructures (DCI) Desktop Computing Utility Computing P2P Computing Pervasive Computing Ubiquitous Computing Mobile Computing 4

5 Contribution of large scale computing Areas in which the role of large scale computing is inevitable: Particle Physics Bioinformatics Computational Mathematics Quantum Chemistry 5

6 Computing model Most of the large scale applications both from academia and industry were designed for batch processing Batch Processing: A complete set of batch or group of instructions together with the required input data to accomplish a given task (often known as job). No user interaction is possible during the execution. 6

7 Cluster computing

8 Cluster computing A cluster is a type of parallel or distributed computer system, which consists of a collection of interconnected stand-alone computers working together as a single integrated computing resource First realised in 60 s but gained real momentum in mid 80 s The aim is to move away from the specialised supercomputing platform and build more general purpose computing environment based on commodity hardware

9 Cluster computing The concept of building computing clusters materialised with tremendous growth in computer hardware In a typical scenario (worker/slave/compute) cluster nodes are dedicated resources with no external peripherals attached Specifically designed for batch processing Cluster Types: Supercomputing clusters Commodity hardware based clusters 9

10 Known Softwares of Cluster computing: Cluster computing HTCondor Portable Batch System (PBS) Load Sharing Facility (LSF) Simple Linux Utility for Resource Management (SLRM) Rocks.. 10

11 Cluster computing Uniform access to available resources Load balancing Various job scheduling techniques Cluster management tools User interfaces single job submission complex workflows management Fundamental level security (in typical cases) Production quality softwares are available Advantages 11

12 Cluster computing Disadvantages Applications need to adopt the way underlying infrastructure is designed Cluster softwares are non-coherent Steep learning curve Less secure (improved significantly over the years) Tightly coupled with the underlying resources Difficult to port new applications Applications need to stick with the available tools and libraries Non standard interfaces 12

13 Cluster computing Current status Cluster computing is one of the most established way of accessing limited amount of interconnected computational resources For example, hundreds of organisations in industry, government, and academia have used HTCondor Extension like Directed Acyclic Graph Manager (DagMAN) in HTCondor are still in use to define complex workflows

14 Cluster computing Short falls Uniform access to large number of resources System that can handle complex and large workloads Possible next steps Explore ways to find more resources Uniform access to distributed computational resources A bigger system for batch processing 14

15 Definition - 1 : (Computational Grid) Definition - 2 : (Computational Power Grids) Grid computing Grid is a type of parallel and distributed system that enables the sharing, selection, and aggregation of geographically distributed autonomous resources dynamically at runtime depending on their availability, capability, performance, cost, and users' quality-of-service requirements. The computational power grid is analogous to electric power grid and it allows to couple geographically distributed resources and offer a consistent and inexpensive access to resources irrespective of their physical location or access point. The anatomy of the grid: Enabling scalable virtual organizations The Grid 2: Blueprint for a new computing infrastructure 15

16 Grid computing Vision 16

17 Grid computing Actual picture 17

18 Grid computing System components Application execution tools Multi-level scheduling Resource discovery Reliability Quality of Services (QoS) Resource allocation Metadata management Information extraction Runtime environments Security Data management Interoperability Virtual Organisation Management System (VOMS).. 18

19 Virtual Organisation Grid computing Virtual Organisation Management System (VOMS) An abstract entity grouping Users, Institutions and Resources in a same administrative domain. Virtual Organisation Management System VOMS is a system for managing authorisation data within multi-institutional collaborations. VOMS provides a database of user roles and capabilities and a set of tools for accessing and manipulating the database and using the database contents to generate Grid credentials for users when needed. Article: From gridmap-file to VOMS: managing authorization in a Grid environment 19

20 Large Hadron Collider Grid (LCG) 20

21 Grid Computing Basic Workflow voms-proxy-init Job Submit Event UI JDL Job Query Input sandbox Output sandbox Job Status Job Status Job Status Resource Broker Expanded JDL Job Submission Service DataSets info Input sandbox + Broker Info Globus RSL SE & CE info Output sandbox Job workflow in glite middleware: Computing Element Information Service Publish Storage Element 21

22 Grid computing at CERN Large Hadron Collider (LHC) experiment at European Organisation for Nuclear Research (CERN) The Grid runs more than two million jobs per day Till 2013, system had 100PB of data and its growing 27PB per year Expected to generate 400PB of data till

23 Grid computing Advantages Seamless access to geographically distributed resources Provide means to accelerate collaborative science The concept of virtual organisations (VO) evolved with Grids Each site in the Grid system is fully autonomous Transparent access to the heterogeneous resources Allows large scale batch processing capabilities 23

24 Complex system architecture Grid Computing Disadvantages Steep learning curve for the end user Only allow batch processing, zero level interactivity Difficult to attach a comprehensive economic model The sites are autonomous but the softwares are tightly connected with the underlying hardware Mostly available for academic and research activities Lack of standard interface Static availability of resources 24

25 European Middleware Initiative (EMI) Grid computing Current status Compute Resources: glite Middleware Advanced Resource connector (ARC) Unicore Storage Resources DCache Castor DPM 25

26 Grid computing Current status Advanced Resource connector (ARC) 26

27 Nordic Data Grid Facility (NDGF) Grid computing Current status Storage/data grid based on DCache software stack Data is distributed over many computing centres across Scandinavia Secure data access using variety of protocols 27

28 Tight coupling with hardware resources User interfaces Limited user community Weak monitoring and billing system Limited user level access Complex software stack Grid computing Short falls Security model users and project management system Possible next steps A system that can address these limitations 28

29 Cloud computing NIST definition Cloud computing is a model for enabling ubiquitous, convenient, on-demand network access to a shared pool of configurable computing resources (e.g., networks, servers, storage, applications, and services) that can be rapidly provisioned and released with minimal management effort or service provider interaction. 29

30 Example from SoVware Engineering Waterfall Model Spiral Model Grid Computing Cloud Computing Unified Modeling Language (UML) 30

31 Strength of cloud computing Cloud compu\ng reduces the gap between the concept and the implementa\on by defining roles and responsibili\es that allows: level of abstrac\on Service Level Agreements (SLA) paradigm shiv from servers to *-as-a-service possibility to a_ach economic model on-demand resource availability 31

32 Cloud computing Roles and responsibilities Infrastructure provider Platform provider Software provider Network provider 32

33 Cloud computing Why Cloud Computing? 33

34 Cloud computing A well-defined economic model Driving force behind Cloud concept Public Clouds Amazon HP Helion Cloud Intel Cloud Private or Community Clouds Smog epouta UberCloud 34

35 Cloud computing Complete isolation, direct access and full control of allocated resources 35

36 Cloud computing On demand resource allocation No job queues! No need of specialised static worker nodes 36

37 Cloud computing Loose coupling with the underlying resources Live or block based VM migration 37

38 Cloud computing Standard interface to interact with the cloud resources Amazon EC2 and S3 APIs could be used to connect to OpenStack based Cloud RestAPIs based communication 38

39 Cloud computing Orchestration of scalable services Amazon EC2 (Compute) Amazon S3 (Storage) Amazon Elastic MapReduce OpenStack Sahara (virtual Hadoop cluster) OpenStack Trove (Database) 39

40 Cloud computing Minimal interaction with service providers 40

41 Cloud computing Are legacy applications portable to Clouds? ANSWER: Yes 41

42 Cloud computing Computing model Together with batch processing, Cloud computing model provides interactive processing of complex applications Wikipedia: Interactive computing refers to software which accepts input from humans for example, data or commands. Frameworks like; IPython or Jupyter notebooks extend web technologies for interactive computing 42

43 Cloud Computing 43

44 Introduction to SNIC Cloud (IaaS) 44

45 Security on SNIC Cloud 45

46 Resources Compute Storage Network What SNIC Cloud will provide? Users will have complete control over the allocated resources. Power comes with the responsibility! 46

47 Important Users can login as supper-user root, can install or uninstall whatever they want. Question: What if for connivance I will create a user account on my VM with the name XXX and password XXX123 Can I? The answer is YES, you can. But it may have serious consequences!!!! 47

48 Consequences Since the VMs will be available via Internet and with weak password or sometimes even with strong passwords, systems can get hacked. The attacker can do varies things: Destroy the data available on the VM Corrupt the VM so it will not be usable Generate an attack using your VM Or even much more 48

49 Don t use password based logins! What should we do? The convention is to use SSH key-pair login mechanism. For this course it is required that all the students always use SSH keys to access resources. 49

50 What is SSH key-pair? A public key based authentication system used to identify users on SSH enabled servers based on pair of keys private key (user s personal key) public key (world readable key) User can generated RSA or DSA based keys RSA (Rivest-Shamir-Adleman) keys have a minimum key length of 768 bits and the default length is 2048 The key length of DSA (Digital Signature Algorithm) is always

51 Key-Pair generation OpenStack based key generation interface Command-line interface $ ssh-keygen or $ ssh-keygen -t rsa -b