CIS 4930/6930 Spring 2014 Introduction to Data Science Data Intensive Computing. University of Florida, CISE Department Prof.

Transcription

1 CIS 4930/6930 Spring 2014 Introduction to Data Science Data Intensive Computing University of Florida, CISE Department Prof. Daisy Zhe Wang

2 Cloud Computing and Amazon Web Services Cloud Computing Amazon Web Services: EC2, S3 Amazon EMR: Apache Hadoop AMT: Amazon Mechanical Turk

3 Cloud Computing: Latest IT Infrastructure for Business Apps Typical business application architecture Desktop-centric installation (dominates 1980s) Client/server solutions (1990s) Loosely coupled web services and serviceoriented architecture (SOA) (2000s) Virtualization, Parallel Computing, HPC, SaaS, data analytics applications Cloud Computing (2010s, started around 2005) 3

4 What is Cloud Computing Cloud computing is a style of computing where scalable and elastic IT-enabled capabilities are delivered as a (pay-asyou-go) service to external customers using Internet technologies. -- Gartner IT Glossary Cloud Computing is a new term for a long-held dream of computing as a utility -- Above the Clouds,

5 Cloud Computing = Cloud + SaaS Cloud computing refers to both: Cloud: The hardware and system software in the datacenters that provide those services, and SaaS: The software and applications delivered as services over the Internet. Public Cloud (Utility Computing) vs. Private Cloud 5

6 Users and Providers of Cloud Computing SaaS providers can also be SaaS users. For example, a mash-up provider of rental maps might be a user of the Craigslist and Google maps services. 6

7 SaaS Benefits Service provider Minimize customer support reduce cost Greatly simplified software installation, maintenance and centralized control over versioning. End User No up-front cost, pay-as-you-go reduce cost Access to service anytime, anywhere Share data and collaborate more easily Cloud computing provide SaaS without provisioning a datacenter. 7

8 SaaS Problems Dependency on good connectivity Data security issues May not be good for high-throughput and real-time apps (i.e., high data velocity, highly responsive) 8

9 Cloud Computing Benefits The illusion of infinite computing resources available on demand (no provisioning) The elimination of an up-front commitment by Cloud users (start small) The ability to pay for use of computing resources on a short-term (e.g., storage by day, processor by hour) basis as needed and release as needed, reward conservation Elasticity: scaling up and down High availability: data center in many locations with data/sw/hw replication 9

10 Cloud Computing Problems Shared resources unrepeatable performance Virtualization shield underlying physical hardware important for OS/HPC research to understand physical hardware connectivity, and data security 10

11 Key Enablers of Cloud Computing Early 2000 s The construction and operation of extremely large-scale, commodity-computer datacenters (Amazon, Google) Combined with software that can efficiently use multiple computers via parallelization (Google: Map-Reduce) 2005: Visualization, Web Services (Amazon) 11

12 Key Enablers of Cloud Computing (Cont.) Technology Push Virtual Machines: allows customers to choose software stack without disrupting each other sharing the same hardware Slice and dice hardware Virtualization over clusters of commodity computers Application Pull Mobile interactive apps: requires high availability, large datasets (e.g., mashups) Parallel batch processing: rely on cloud s cost associativity Rise of big data science & analytics 12

13 Cloud Computing Providers Distinguished based on the level of abstraction presented to the programmer and level of management of resources Amazon EC2: hardware virtual machines Google App Engine: application frameworks MS Azure: intermediate Amazon EC2 Pros and Cons Users can have full control to entire software stack Difficulty to offer automatic management of scalability and failover 13

14 When Public Cloud? When Private Datacenter? 1. Demand for a service varies with time 2. Demand is unknown in advance 3. Organizations that perform large batch analytics 4. Should Facebook use Cloud? 5. Should a online music start-up use Cloud? 14

15 Amazon and Cloud Computing 2005: building and managing the large-scale, reliable, and efficient IT infrastructure that powered one of the world s largest online retail platforms. 2006: launched AWS to provide other organizations with the same infrastructure that runs Amazon s large-scale, distributed, transactional IT infrastructure 2009: Above the Clouds: A Berkeley view of Cloud Computing 15

16 Example use of AWS A large enterprise quickly and economically deploys new internal apps, such as HR, Payroll, online training, etc. An e-commerce website with accommodation to sudden demands A research institution execution large-scale simulation or big data analysis Media companies serve unlimited music, video to worldwide customers Support Big Data/Data Science courses 16

17 AWS Cloud Platform Compute: EC2, EMR/Hadoop (HDFS, MapReduce, Hive, Pig) Storage: S3 Database: DynamoDB, Redshift, ElastiCache, Kinesis Crowdsourcing: Amazon Mechanical Turk More Advanced: Spark, Shark, Mahout (will be covered in the next course: Advanced Topics in Data Science) 17

18 Amazon EC2 (Elastic Compute Cloud) A Web service that provides resizable compute capacity in the cloud. Designed to make Web-scale computing easier for developers. A simple Web service interface that provides high-degree of control of your computing resources

19 Amazon EC2 Benefits Reduces the time required to obtain and boot new server instances to minutes Quickly scales capacity, both up and down, as your computing requirements change Changes the economics of computing: No start-up, monthly, or fixed costs Pay only for capacity that you actually use a + bc becomes just bc

20 Pricing Models On-Demand Instances On-Demand Instances let you pay for compute capacity by the hour with no long-term commitments. Reserved Instances Reserved Instances give you the option to make a low, one-time payment for each instance you want to reserve and in turn receive a significant discount on the hourly charge for that instance. Spot Instances Spot Instances allow customers to bid on unused Amazon EC2 capacity and run those instances for as long as their bid exceeds the current Spot Price. 20