Expanding the Horizons of Cloud Computing Beyond the Data Center. Jon Weissman Department of CS&E University of Minnesota

Transcription

1 Expanding the Horizons of Cloud Computing Beyond the Data Center Jon Weissman Department of CS&E University of Minnesota Open Cirrus October 2011

2 Key Trends Client technology devices: smart phones, tablets, sensors Big data distributed and dynamic Privacy/trust small circles Multiple DCs global services

3 Minnesota Cloud Research Eye towards cloud evolution Projects Nebula Mobile cloud DMapReduce Proxy cloud Active cloud storage Virtualization

4 Big Data Trend Big data is distributed earth science: weather data, seismic data life science: GenBank, PubMed health science: GoogleEarth + CDC pandemic data web 2.0: user multimedia blogs everyone is a sensor Cost in moving data to the cloud

5 Big Data Trend: Nebulas Process data close by fully and/or on-route to the central cloud cost: save time and money privacy (think: patient records, local google doc) Close by network distance trusted peers

6 Example: Dispersed-Data-Intensive Services Data is geographically distributed Costly, inefficient to move to central location

7 Example Instance: Blog Analysis blog1 blog2 blog3

8 Nebula Decentralized, less-managed cloud dispersed storage/compute resources low user cost

9 Nebula: A New Cloud Model Make the cloud more distributed exploit the rich collection of edge computers volunteers commercial (CDNs) enormous computing potential, network diversity lower latency: on demand, native client sandboxing Nebula Central

10 Example: Blog Analysis blog1 blog2 blog3

11 Blog Results Amazon emulator Nebula testbed Time taken (sec) # Blogs

12 Current Status Prototype running on PlanetLab Chrome browser clients + native client Distributed data-store service Network dashboard service

13 Nebula Going Forward Organize Nebulas around trusted peers social groups communities of interest local resources Nebula + commercial cloud use the edge opportunistically

14 Nebula Going Forward Hybrid paradigms early discard fan-in

15 Mobility Trend: Mobile Cloud

16 Mobility Trend: Mobile Cloud Mobile users/applications: s-phones, tablets resource limited: power, CPU, memory applications are becoming sophisticated Improve mobile user experience performance, reliability, fidelity tap into the cloud dynamically based on current resource state, preferences, interests, privacy

17 Cloud Mobile Opportunity Dynamic outsourcing move computation, data to the cloud dynamically User context exploit user behavior to pre-fetch, pre-compute, cache Multi-user sharing discover implicit cloud sharing based on interests, social ties

18 Outsourcing Partitioning across (Mobile <-> Cloud) local data capture + cloud processing images/video, speech, digital design, aug. reality Server Server Server Server. Server Dalvik cloud end (EC-2) Code repository Proxy Outsourcing Client mobile end (Android). Application Profiler Outsourcing Controller

19 Experimental Results -Image Response time both WIFI & 3G up to 27 speedup Sharpening Avg. Time Power consumption save up to 9 times 19 Avg. Power

20 Current Work Optimizations User patterns => speculation, aggregation, data reduction Cloud-side Cross-user patterns => sharing: VM provisioning, data placement, data re-use

21 Big Data Trend + Multi-DCs: DMapReduce Data Push Map Reduce Input Data Output Data 21

22 Wide-Area MapReduce Big data is distributed earth science: weather data/seismic data life science: GenBank/PubMed health science: GoogleEarth /pandemic data web 2.0: user multimedia blogs Data in different data-centers Run MapReduce across them Data-flow spanning wide-area networks

23 Option 1: Local MapReduce MapReduce Job Final Result Data Push (Fast) 23 Data Source (US) Data Center (US) Data Center (EU) Data Push (Slow) Data Source (EU)

24 Option 2: Global MapReduce MapReduce Job Final Result Data Push (Fast) Data Push (Fast) 24 Data Source (US) Data Center (US) Data Push (Slow) Data Center (EU) Data Push (Slow) Data Source (EU)

25 Option 3: Distributed MapReduce Combine Results Final Result Data Push (Fast) MapReduce Job MapReduce Job Data Push (Fast) Data Source (US) Data Center (US) Data Center (EU) Data Source (EU) 25

26 PlanetLab: WordCount (text data) Time in seconds Local MR Global MR Distributed MR Push US Push EU Map Reduce Result Combine Total Distributed MR works best in presence of data aggregation 26

27 PlanetLab: WordCount (random data) 900 Time in seconds Local MR Global MR Distributed MR Push US Push EU Map Reduce Result Combine Total Local MR works best in presence of data ballooning 27

28 EC-2 Results Time in seconds Local MR WordCount (Text) Distributed MR Push US Push EU Map Reduce Result Combine 300 Total Time in Seconds Sort WordCount (Random) Local MR Distributed MR Push US Push EU Map Reduce Result Combine Total Time in Seconds Local MR Distributed MR 28 0 Push US Push EU Map Reduce Result Combine Total

29 Intelligent Data Placement for Global Services HDFS push local node, same rack, random rack Resource Topology Application Characteristics /DC i /rack A /node X Data placement Scheduling Data expansion factors input->intermediate, α Intermediate->output, β

30 Experimental Results (Word count) Smart HDFS HDFS

31 Cloud Evolution Summary mobile users, big data, privacy/trust, global services Our Vision of the Cloud locality of users, data, other clouds/data centers, usercentric behavior