Sriram Krishnan, Ph.D.

Transcription

1 Sriram Krishnan, Ph.D.

2 (Re-)Introduction to cloud computing Introduction to the MapReduce and Hadoop Distributed File System Programming model Examples of MapReduce Where/how to run MapReduce code On the cloud (AWS) Traditional HPC systems (myhadoop)

3 Cloud computing is a model for enabling convenient, on-demand network access to a shared pool of configurable computing resources (e.g., networks, servers, storage, applications, and services) that can be rapidly provisioned and released with minimal management effort or service provider interaction

4 A consumer can unilaterally provision computing capabilities, such as server time and network storage As needed Automatically Without requiring human interaction from service provider

5 Capabilities are available over the network Accessed through standard mechanisms Heterogeneous thin or thick client platforms (e.g., mobile phones, laptops, and PDAs). Typically via light-weight Web service APIs

6 The provider s computing resources are pooled to serve all consumers using a multi-tenant model Different physical and virtual resources dynamically assigned and reassigned according to consumer demand The customer generally has no control or knowledge over the exact location of the provided resources But may be able to specify location at a higher level of abstraction (e.g., country, state, or datacenter) Examples of resources include storage, processing, memory, network bandwidth, and virtual machines

7 Capabilities can be rapidly and elastically provisioned to quickly scale up And rapidly released to quickly scale down To the consumer, the capabilities available for provisioning often appear to be infinite Can be purchased in any quantity at any time

8 Automatic control and optimization of resource use by leveraging a metering capability At the level of abstraction appropriate to the type of service (e.g., storage, processing, bandwidth, and active user accounts) Resource usage can be monitored, controlled, and reported providing transparency for both the provider and consumer of the utilized service

9 Enterprises incur no infrastructure capital costs, just operational costs on a pay-per-use basis Elimination of up-front costs Architecture specifics are abstracted Capacity can be scaled up or down dynamically, and immediately Illusion of infinite resources, on-demand The underlying hardware can be anywhere geographically

10 * Source: McKinsey & Co

11 Infrastructure as a Service (IaaS) Platform as a Service (PaaS) Software as a Service (SaaS)

12 The capability provided to the consumer is to provision processing, storage, networks, and other fundamental computing resources Consumer is able to deploy and run arbitrary software, which can include operating systems and applications The consumer does not manage or control the underlying cloud infrastructure But has control over operating systems, storage, deployed applications, and possibly select networking components (e.g., firewalls, load balancers) Examples: Amazon EC2 (compute), S3 (storage)

13 The capability provided to the consumer is to deploy onto the cloud infrastructure consumer-created applications using programming languages and tools supported by the provider (e.g., Java, Python,.Net) The consumer does not manage or control the underlying cloud infrastructure, network, servers, operating systems, or storage But the consumer has control over the deployed applications and possibly application hosting environment configurations Examples: Google AppEngine, Amazon Elastic MapReduce (EMR), Microsoft Azure

14 The capability provided to the consumer is to use the provider s applications running on a cloud infrastructure Accessible from various client devices through a thin client interface such as a Web browser (e.g., web-based ) The consumer does not manage or control the underlying cloud infrastructure With the possible exception of limited user-specific application configuration settings Examples: Google Maps, Facebook, Animoto

15 Cloud computing necessitates a different approach for large scale data processing Use of non-specialized commodity hardware and low latency interconnect between nodes Not suitable for implementing traditional databases or HPC systems which require specialized hardware configurations Traditional databases also not known to scale very well to thousands of nodes which may be available on the cloud

16 Programming environment for very large scale data processing on commodity resources Originally developed by Google Managing task executions and data transfers in a shared-nothing environment Distributed data repository ( file system ): Google File System (GFS) Round-robin partitioning of data More info: MapReduce: Infrastructure to support data scatter / gather Pushing the computation to the data More info:

17 Open Source Implementation of Google s MapReduce ecosystem Hadoop Distributed File System (HDFS): Based on the Google File System Hadoop MapReduce: Based on Google s MapReduce

18 Shared-nothing (MapReduce-style) Architectures ETHERNET COMPUTE/DATA CLUSTER WITH LOCAL STOARGE

19 *Source:

20 The computation takes a set of input key/ value pairs, and Produces a set of output key/value pairs. The MapReduce library expresses the computation as two functions Map and Reduce All data resides in a distributed file system

21 map (k1,v1) list(k2,v2) Map function transforms input (key, value) pairs into a list of intermediate (key, value) pairs reduce (k2,list(v2)) list(v2) Reduce function transforms a key and a set of its values to a final list of values (outputs)

22 map(string key, String value): // key: document name // value: document contents for each word w in value: EmitIntermediate(w, "1"); reduce(string key, Iterator values): // key: a word // values: a list of counts int result = 0; for each v in values: result += ParseInt(v); Emit(AsString(result))

23 Distributed grep Pattern matching in parallel Count of URL access frequency (URL, total count) from web logs Reverse Web-Link Graph (target, list(source)) for URLs

24 *Source:

25 Scalable and conducive to data-intensive and data-parallel applications Fault-tolerant by design workers can be restarted on failures Ability to run on non-specialized commodity hardware And cloud resources such as AWS (EC2 MapReduce service)

26 Easy to use complexity is hidden by the library A large variety of problems are easily expressible in this model Caveat: Not all algorithms lend themselves very well to be implemented using this model Implementation scales to 1000s of machines Restricting the programming model makes it easy to parallelize and deal with failures

27

28 * Credit: "Cloud computing and the DNA data race", M. C. Schatz, B. Langmead, S. L Salzberg. Nature Biotechnology 28, (2010)

29 Full-featured DEM Bare earth DEM

30

31 Dedicated Hadoop Cluster On the cloud (e.g. Amazon Elastic MapReduce) On HPC resources

32 Dedicated Hadoop cluster Set of commodity nodes configured to run Hadoop in distributed mode

33 4-node Hadoop cluster: Masters: cat $HADOOP_HOME/conf/masters Slaves: cat $HADOOP_HOME/conf/slaves Other configs: ls al $HADOOP_HOME/conf Starting up Hadoop: $HADOOP_HOME/bin/start-all.sh Shutting down Hadoop: $HADOOP_HOME/bin/stop-all.sh

34 Job Tracker: Task Tracker:

35 HDFS commands: Copy data into HDFS: hadoop dfs -copyfromlocal Data input-data HDFS directory listing: hadoop dfs -ls HDFS file cat : hadoop dfs -cat input-data/gutenberg/ txt

36 Running MapReduce jobs: hadoop jar /opt/hadoop/hadoop / hadoop examples.jar wordcount inputdata/gutenberg output Results: List: hadoop dfs ls output View: hadoop dfs cat output/part-r-00000

37 On the cloud Variable number of nodes uses Amazon s Elastic MapReduce

38

39

40

41

42 On traditional HPC systems Need scripts to configure Hadoop on-demand via traditional batch systems HOD: myhadoop:

43 An open source tool for running Hadoop jobs on HPC resources Easy to configure and use for the end-user Play nice with existing batch systems on HPC resources Why do we need such a tool? End-users: I already have Hadoop code and I only have access to regular HPC-style resources Computer Scientists: I want to study the implications of using Hadoop on HPC resources And I don t have root access to these resources

44 Shared-nothing (Hadoop) versus HPCstyle architectures In terms of philosophies and implementation Control and co-existence of Hadoop and HPC batch systems Typically both Hadoop and HPC batch systems (viz., SGE, PBS) need completely control over the resources for scheduling purposes

45 BATCH PROCESSING SYSTEM (PBS, SGE) [1] [2, 3] COMPUTE NODES PERSISTENT MODE [4(b)] HADOOP DAEMONS NON PERSISTENT MODE [4(a)] PARALLEL FILE SYSTEM

46 BOOTSTRAP TEARDOWN

47 Write a PBS/SGE script to request N nodes from the scheduler

48 Configure Hadoop on allocated nodes via myhadoop

49 Run your Hadoop job then clean up after execution

50 myhadoop an open source tool for running Hadoop jobs on HPC resources Without need for root-level access Co-exists with traditional batch systems Allows persistent and non-persistent modes to save HDFS state across runs Tested on SDSC Triton, TeraGrid and UC Grid resources More information Software: SDSC Tech Report:

51 Hadoop: MapReduce Tutorial: HDFS Users Guide: Hadoop Streaming: Use of custom scripts as mappers and reducers No Java code required Amazon Elastic MapReduce: Disco (MapReduce in Python):

52 Feel free to me

53

54 Private Cloud Operated solely for an organization May be managed by the organization or a third party and may exist on premise or off premise Community Cloud Shared by several organizations and supports a specific community that has shared concerns May be managed by the organizations or a third party and may exist on premise or off premise

55 Public Cloud Made available to the general public or a large industry group Owned by an organization selling cloud services Hybrid Cloud A composition of two or more clouds (private, community, or public) Bound together by standardized or proprietary technology that enables data and application portability

56 Access to HPC resources is typically via batch systems viz. PBS, SGE, Condor, etc These systems have complete control over the compute resources Users typically can t log in directly to the compute nodes (via ssh) to start various daemons Hadoop manages its resources using its own set of daemons NameNode & DataNode for Hadoop Distributed File System (HDFS) JobTracker & TaskTracker for MapReduce jobs Hadoop daemons and batch systems can t co-exist seamlessly Will interfere with each other s scheduling algorithms

57 1. Enabling execution of Hadoop jobs on shared HPC resources via traditional batch systems a) Working with a variety of batch systems (PBS, SGE, etc) 2. Allowing users to run Hadoop jobs without needing rootlevel access 3. Enabling multiple users to simultaneously execute Hadoop jobs on the shared resource 4. Allowing users to either run a fresh Hadoop instance each time (a), or store HDFS state for future runs (b)

58