Archiving and Sharing Big Data Digital Repositories, Libraries, Cloud Storage

Transcription

1 Archiving and Sharing Big Data Digital Repositories, Libraries, Cloud Storage Cyrus Shahabi, Ph.D. Professor of Computer Science & Electrical Engineering Director, Integrated Media Systems Center (IMSC) Director, VSoE Informatics Viterbi School of Engineering University of Southern California Los Angeles, CA

2 OUTLINE Some Backgrounds Cloud CompuBng An Example IMSC s TransDec A Proposal USC DataLab 2

3 OUTLINE Some Backgrounds Cloud CompuBng An Example IMSC s TransDec A Proposal USC DataLab 3

4 Cloud CompuBng Cloud compubng is the delivery of compubng and storage resources as a service across Internet to mulbple external customers through massive scale data centers. Some stabsbcs 51% of all global workloads in 2014 were processed in cloud versus tradibonal IT space 1. IBM Big Blue cloud project generated $7 billion revenue in 2014, up 75% from the previous year 2. By 2020, it is esbmated that 80% of small businesses in US will use cloud compubng, up from 37% in [1] Cisco, hyp://newsroom.cisco.com/release/ [2] hyp://talkincloud.com/cloud- compubng- funding- and- finance/ /ibm- q4- earnings- cloud- revenues- hit- 7b

5 Advantages ü Reduced Cost Cloud CompuBng eliminabon of the investment in stand- alone so_ware or servers ü Scalability and ElasBcity providing on- demand resources instantaneously ü Availability downbme is very small throughout year ü Quick deployment minimum effort in integrabng applicabon ü Environment friendly less cooling cost per server, more ublizabon 5

6 Cloud CompuBng Disadvantages ü Security and Privacy by leveraging a remote cloud based infrastructure, a company essenbally gives away private data and informabon ü Dependency and Vendor lock- in implicit dependency on the provider ü Limited Flexibility since the applicabons and services run on remote, third party virtual environments, users have limited control over the hardware and so_ware ü Increased Vulnerability since cloud based solubons are exposed on the public internet and are thus a more vulnerable target for malicious users and hackers. 6

7 Cloud CompuBng Market shares of big players 7

8 Cloud CompuBng - Pricing Virtual Machines (Servers) Servers are grouped into certain categories such as disk- op(mized, memory- op(mized, CPU- op(mized, GPU. Each server group consists of mulbple servers Note: smallest means the server with the lowest configurabon in that group Group Amazon Microso8 Google price ($/hour) price ($/hour) price ($/hour) smallest largest smallest largest smallest largest General purpose Compute op<mized Memory op<mized Disk op<mized Micro GPU

9 OUTLINE Some Backgrounds Cloud CompuBng An Example IMSC s TransDec A Proposal USC DataLab 9

10 Traffic Data Lifecycle: Data Aggregator An Exclusive Contract w LA- Metro (2010) Data Type Sample XML File Size (in KB) Variety (gps, video, loop Cycle Duration (in seconds) Minute (in KB) Hourly (in KB) Daily (in KB) Annual (in KB) 3 Years (in KB) sensor, events) bus_mta_inv2.xml , , bus_mta_rt2.xml , , ,882, ,646, cctv_inv.xml , , cms_inv.xml , , cms_rt.xml , , ,183, ,549, event_d7.xml , ,625, ,875, rail_mta_inv.xml , rail_rt.xml , ,204, ,614, rms_inv.xml , , rms_rt.xml , ,423, ,713, ,559,139, signal_inv.xml , , ,294, signal_rt.xml , , ,061, ,847,308, ,541,926, tt_d7_inv.xml , , tt_d7_rt.xml , , ,891, ,673, vds_art_d7_inv.xml , , Velocity vds_art_d7_rt.xml , , ,652, ,956, vds_art_ladot_inv.xml , , ,779, vds_art_ladot_rt.xml , ,395, ,306, ,527,919, vds_fr_d7_inv.xml , ,047, vds_fr_d7_rt.xml , ,039, ,483, ,138,449, Total KB from XML data , , ,057, ,670,968, ,012,906, Volume 10

11 TransDec: Big data acquisibon, storage & access Input Traffic Data Data Processing Storage Retrieval, Analysis &VisualizaBon Sensor 4 46 MB/min MB/min TB/Year Sensor 3 Sensor 2 Highway (4313) Arterial (4780) Real- <me Queries & Bus & Rail (2000) Data Cleansing Ramp meter Events & CMS (800/day) Spa<otemporal Indexing (Oracle Award, IEEE CloudCom Best paper) Sensor 1 Event LocaBon E.g., Accident impact analysis & predic<on (ICDM 12 & 13) 11

12 OUTLINE Some Backgrounds Cloud CompuBng An Example IMSC s TransDec A Proposal USC DataLab 12

13 Berkeley Data AnalyBcs Stack - BDAS- BDAS: BDAS is an open source so8ware stack that integrates open- source so_ware components to make sense of Big Data. A High Level overview of BDAS Components Data Processing Data Management Resource Management 13

14 Berkeley Data AnalyBcs Stack BDAS More in Depth - BDAS- Numerous available open source packages for: - Machine Learning (MLlib) - Graph analysis (GraphX) - Real- <me Analysis (Streaming) BigData applica<ons for various domains Flexible intercommunica<on between layers Unlimited expansion Many more projects to come 14

15 USC DataLab Create a shared repository of USC data & code for research (on BDAS) Example: Security- related Datasets CCTV videos from DPS Mobile videos from any individuals Sensor Readings from Buildings from Facility Management Crime Reports from DPS ShuYle bus routes/locabons from USC TransportaBon Security patrol cars/ambassadors locabons from DPS Events from various sources Crowdsourced data from USC community Shared So_ware (for data analysis such as object recognibon) from USC community 15

16 Backup Slides 16

17 Cloud CompuBng - Amazon Virtual Machines 28 different types of servers Big Data analysis for both offline and stream data. Services: ElasBc MapReduce, Kinesis, RedShi_ Scalable NoSQL databases Service: DynamoDB TradiBonal relabonal databases Service: RDS File and Object Storage Service: S3 17

18 Cloud CompuBng - Microso_ Virtual Machines 18 different types of servers Big Data analysis for both offline and stream data. Services: HDInsight Scalable NoSQL databases Service: Windows Azure Table TradiBonal relabonal databases Service: SQL server File and Object Storage Service: Windows Azure 18

19 Cloud CompuBng - Google Virtual Machines 15 different types of servers Big Data analysis for both offline and stream data. Services: Big Query, Hadoop Scalable NoSQL databases Service: Cloud Datastore TradiBonal relabonal databases Service: Google Cloud SQL File and Object Storage Service: Google Cloud Storage 19

20 Berkeley Data AnalyBcs Stack - BDAS- BDAS Important Components Mesos A cluster management layer Resource management and scheduling across enbre datacenters and cloud environments Spark An in- memory, distributed, fault- tolerant processing framework Data sharing enabled compared to Map Reduce In- memory solu<on, extremely faster for tasks that boyleneck on disk I/O in MapReduce MulBple running packages on top of Spark Core (Spark SQL, SPARK MLlib, SPARK Streaming) Tachyon Fault- tolerant, memory- centric distributed file system Tachyon caches working set files in memory Avoids going to disk to load datasets that are frequently read Provides memory level response <mes for frequently accessed data 20

21 InstallaBon Dependencies Berkeley Data AnalyBcs Stack - BDAS- BDAS can be installed on any cloud provider with Amazon Cloud Google Cloud Microso_ Azure Cloud Private Cloud HPC So8ware Requirements Runs on both Windows and Unix- like systems (Centos, RHEL, Mac OS) Produc<on Requirements Memory per machine good behavior documented from 8GB to hundred CPU cores per machine provide at least 8-16 cores 21