High Performance Compu2ng and Big Data. High Performance compu2ng Curriculum UvA- SARA h>p://

Transcription

1 High Performance Compu2ng and Big Data High Performance compu2ng Curriculum UvA- SARA h>p://

2 Big data was big news in 2012 and probably in 2013 too. The Harvard Business Review talks about it as The Management Revolu0on. The Wall Street Journal says Meet the New Big Data, and Big Data is on the Rise, Bringing Big Ques0ons.

3 Big- Data popularity

4 Where Big Data Comes From? Big Data is not Specific applica*on type, but rather a trend or even a collec2on of Trends- napping mul2ple applica2on types Data growing in mul2ple ways More data (volume of data ) More Type of data (variety of data) Faster Ingest of data (velocity of data) More Accessibility of data (internet, instruments, ) Data Growth and availability exceeds organiza2on ability to make intelligent decision based on it Addison Snell CEO. Intersect360, Research

5 Data is Big If It is Measured in MW A good sweet spot for a data center is 15 MW Facebook s leased data centers are typically between 2.5 MW and 6.0 MW. Facebook s Pineville data center is 30 MW Google s compu2ng infrastructure uses 260 MW Robert Grossman, Collin BenneC University of Chicago Open Data Group

6 Jim Gray Vision We have to do be>er at producing tools to support the whole research cycle from data capture and data cura*on to data analysis and data visualiza*on. Today, the tools for capturing data both at the mega- scale and at the milli- scale are just dreadful. Aber you have captured the data, you need to curate it before you can start doing any kind of data analysis, and we lack good tools for both data cura*on and data analysis. Then comes the publica*on of the results of your research, and the published literature is just the 2p of the data iceberg. By this I mean that people collect a lot of data and then reduce this down to some number of column inches in Science or Nature or 10 pages if it is a computer science person wri2ng. So what I mean by data iceberg is that there is a lot of data that is collected but not curated or published in any systema*c way. Based on the transcript of a talk given by Jim Gray to the NRC- CSTB1 in Mountain View, CA, on January 11, 2007

7 Advice From Jim Gray 1. Analysing Big data requires scale- out solu2ons not scale- up solu2ons (GrayWulf) 2. Move the analysis to the data. 3. Work with scien2sts to find the most common 20 queries and make them fast. 4. Go from working to working. Robert Grossman, Collin BenneC University of Chicago Open Data Group

8 What is the Fundamental Trade Off? Robert Grossman, Collin BenneC University of Chicago Open Data Group

9 How much data? Google processes 20 PB a day (2008) Wayback Machine has 3 PB TB/month (3/2009) Facebook has 2.5 PB of user data + 15 TB/day (4/2009) ebay has 6.5 PB of user data + 50 TB/day (5/2009) CERN s Large Hydron Collider (LHC) generates 15 PB a year 640K ought to be enough for anybody. h>p://econsultancy.com/nl/blog/ big- data- shising- the- bell- curve- to- the- long- tail 9 BigData.pptx - Computer Science

10 BigData is the new hype By the end of 2011 the world had an es*mated 1.8 zetabytes (trillion gigabytes) of data and it is es*mated to 10 grow to 35 zetabytes in the next few years

11 content Defining BigData Enablers of BigData BigData is good when Where to store the data? Moving big Data Scien2fic e- infrastructure some challenges to overcome a wish list

12 How We Define Big Data Big in Big Data refers to: Big size is the primary defini2on. Big complexity rather than big volume. it can be small and not all large datasets are big data size ma>ers... but so does accessibility, interoperability and reusability. define Big Data using 3 Vs; namely: volume, variety, velocity Big Data - Back to mine 12

13 volume, variety, and velocity Aggrega2on that used to be measured in petabytes (PB) is now referenced by a term: zetabytes (ZB). A zetabyte is a trillion gigabytes (GB) or a billion terabytes in 2010, we crossed the 1ZB marker, and at the end of 2011 that number was es2mated to be 1.8ZB

14 volume, variety, and velocity The variety characteris2c of Big Data is really about trying to capture all of the data that pertains to our decision- making process. Making sense out of unstructured data, such as opinion, or analysing images.

15 Type of Data Rela2onal Data (Tables/Transac2on/Legacy Data) Text Data (Web) Semi- structured Data (XML) Graph Data Social Network, Seman2c Web (RDF), Streaming Data You can only scan the data once

16 volume, variety, and velocity velocity is the rate at which data arrives at the enterprise and is processed or well understood In other terms How long does it take you to do something about it or know it has even arrived?

17 volume, variety, velocity, and veracity Veracity refers to the quality or trustworthiness of the data. A common complica2on is that the data is saturated with both useful signals and lots of noise (data that can t be trusted)

18

19

20 content Defining BigData Enablers of BigData BigData is good when Where to store the data? Moving big Data Scien2fic e- infrastructure some challenges to overcome a wish list

21 Enabler: Data storage

22 Enabler: computa2on capacity

23 Enabler: Data storage

24 Type of available data

25 content Defining BigData Enablers of BigData BigData is good when Where to store the data? Moving big Data Scien2fic e- infrastructure some challenges to overcome a wish list

26 When to Consider a Big Data Solu2on User point of view You re limited by your current plaworm or environment because you can t process the amount of data that you want to process You want to involve new sources of data in the analy2cs, but you can t, because it doesn t fit into schema- defined rows and columns without sacrificing fidelity or the richness of the data You need to ingest data as quickly as possible and need to work with a schema- on- demand

27 When to Consider a Big Data Solu2on You re forced into a schema- on- write approach (the schema must be created before data is loaded), but you need to ingest data quickly, or perhaps in a discovery process, and want the cost benefits of a schema- on- read approach (data is simply copied to the file store, and no special transforma2on is needed) un2l you know that you ve got something that s ready for analysis? The data arriving too fast at your organiza2on s doorstep for the current analy2cs platorm to handle

28 When to Consider a Big Data Solu2on You want to analyse not just raw structured data, but also semi- structured and unstructured data from a wide variety of sources you re not sa2sfied with the effec2veness of your algorithms or models when all, or most, of the data needs to be analysed or when a sampling of the data isn t going to be nearly as effec2ve

29 When to Consider a Big Data Solu2on you aren t completely sure where the inves2ga2on will take you, and you want elas*city of compute, storage, and the types of analy2cs that will be pursued all of these became useful as we added more sources and new methods If your answers to any of these ques*ons are yes, you need to consider a Big Data solu*on.

30 content Defining BigData Enablers of BigData BigData is good when Where to store the data? Moving big Data Scien2fic e- infrastructure some challenges to overcome a wish list

31 What are the choices to store the Big data Pa>ern 1: Put the metadata in a database and point to files in a file system Pa>ern 2: Put the data into a distributed file system. Pa>ern 3: Put the data into a NoSQL applica2on Robert Grossman University of Chicago Open Data Group, November 14, 2011

32 What are the choices to store the Big data Pa>ern 1: Put the metadata in a database and point to files in a file system User Rela2onal DB Search is only done on the meta- data Robert Grossman University of Chicago Open Data Group, November 14, 2011

33 What are the choices to store the Big data Pa>ern 2: Put the data into a distributed file system. Google Distributed File System (GDFS). Hadoop Distributed File Systems (HDFS) Sector Distributed File System (SDFS) Robert Grossman University of Chicago Open Data Group, November 14, 2011

34 What are the choices to store the Big data Pa>ern 3: Put the data into a NoSQL database Google s BigTable Amazon s Dynamo Facebook s Cassandra Robert Grossman University of Chicago Open Data Group, November 14, 2011

35 Bill Howe, UW

36 MapReduce vs. Databases A. Pavlo, et al. "A comparison of approaches to large- scale data analysis," in SIGMOD 09: Proceedings of the 35th SIGMOD interna=onal conference on Management of data, New York, NY, USA, 2009, pp Conclusions: at the scale of the experiments we conducted, both parallel database systems displayed a significant performance advantage over Hadoop MR in execu2ng a variety of data intensive analysis benchmarks.

37 content Defining BigData Enablers of BigData BigData is good when Where to store the data? Moving big Data Scien2fic e- infrastructure some challenges to overcome a wish list

38 The problem TCP Was never designed to move large datasets over wide area high Performance Networks. For loading a webpage, TCP is great. For sustained data transfer, it is far from ideal. Most of the 2me even though the connec*on itself is good (let say 45Mbps), transfers are much slower. There are two reason for a slow transfer over fast connec2ons: Latency and packet loss bring TCP- based file transfer to a crawl. Robert Grossman University of Chicago Open Data Group, November 14, 2011

39 TCP Throughput vs RTT and Packet Loss LAN US US-EU US-ASIA Throughput (Mb/s) % 0.5% 0.1% % 0.05% Round Trip Time (ms) Source: Yunhong Gu, 2007, experiments over wide area 1G.

40 The solu2ons Use parallel TCP streams GridFTP Use specialized network protocols UDT, FAST, etc. Use RAID to stripe data across disks to improve throughput when reading These techniques are well understood in HEP, astronomy, but not yet in biology Robert Grossman University of Chicago Open Data Group, November 14, 2011

41 Moving 113GB of Bio- mirror Data Site RTT TCP UDT TCP/UDT Km Site RTT TCP UDT TCP/UDT Km NCSA NCSA Purdue ORNL ,200 TACC ,000 SDSC ,300 CSTNET ,000 Purdue ORNL ,200 TACC ,000 SDSC ,300 CSTNET ,000 GridFTP TCP and UDT transfer 2mes for 113 GB from gridip.bio- - - mirror.net/biomirror/ blast/ (Indiana USA). All TCP and UDT 2mes in minutes. Source: h>p://gridip.bio- mirror.net/biomirror/ Robert Grossman University of Chicago Open Data Group, November 14, 2011

42 Case study: CGI 60 genomes Trace by Complete Genomics showing performance of moving 60 complete human genomes from Mountain View to Chicago using the open source Sector/UDT. Approximately 18 TB at about 0.5 Mbs on 1G link. Robert Grossman University of Chicago Open Data Group, November 14, 2011

43 content Defining BigData Enablers of BigData BigData is good when Where to store the data? Moving big Data Scien2fic e- infrastructure some challenges to overcome a wish list

44 Scien2fic e- infrastructure some challenges to overcome Collec2on How can we make sure that data are collected together with the informa*on necessary to re- use them? Trust How can we make informed judgements about whether certain data are authen*c and can be trusted? How can we judge which repositories we can trust? How can appropriate access and use of resources be granted or controlled Riding the wave, How Europe can gain from the rising 2de of scien2fic data

45 Scien2fic e- infrastructure some challenges to overcome Usability How can we move to a situa2on where non- specialists can overcome the barriers and be able to start sensible work on unfamiliar data Interoperability How can we implement interoperability within disciplines and move to an overarching mul2- disciplinary way of understanding and using data? How can we find unfamiliar but relevant data resources beyond simple keyword searches, but involving a deeper probing into the data How can automated tools find the informa2on needed to tackle data Riding the wave, How Europe can gain from the rising 2de of scien2fic data

46 Scien2fic e- infrastructure some challenges to overcome Diversity How do we overcome the problems of diversity heterogeneity of data, but also of backgrounds and data- sharing cultures in the scien2fic community? How do we deal with the diversity of data repositories and access rules within or between disciplines, and within or across na2onal borders? Security How can we guarantee data integrity? How can we avoid data poisoning by individuals or groups intending to bias them in their interest? Riding the wave, How Europe can gain from the rising 2de of scien2fic data

47 content Defining BigData Enablers of BigData BigData is good when Where to store the data? Moving big Data Scien2fic e- infrastructure some challenges to overcome a wish list

48 Scien2fic e- infrastructure a wish list Open deposit, allowing user- community centres to store data easily Bit- stream preserva*on, ensuring that data authen2city will be guaranteed for a specified number of years Format and content migra*on, execu2ng CPU- intensive transforma2ons on large data sets at the command of the communi2es Riding the wave, How Europe can gain from the rising 2de of scien2fic data

49 Scien2fic e- infrastructure a wish list Persistent iden*fica*on, allowing data centres to register a huge amount of markers to track the origins and characteris2cs of the informa2on Metadata support to allow effec2ve management, use and understanding Maintaining proper access rights as the basis of all trust A variety of access and cura*on services that will vary between scien2fic disciplines and over 2me Riding the wave, How Europe can gain from the rising 2de of scien2fic data

50 Scien2fic e- infrastructure a wish list Execu*on services that allow a large group of researchers to operate on the stored date High reliability, so researchers can count on its availability Regular quality assessment to ensure adherence to all agreements Distributed and collabora*ve authen2ca2on, authorisa2on and accoun2ng A high degree of interoperability at format and seman2c level Riding the wave, How Europe can gain from the rising 2de of scien2fic data

51

52 Google BigQuery Google BigQuery is a web service that lets you do interac2ve analysis of massive datasets up to billions of rows. Scalable and easy to use, BigQuery lets developers and businesses tap into powerful data analy2cs on demand h>p:// v=p78t_zdwqyk

53 IBM BigInsights BigInsights = analy2cal platorm for persistent big data Based on open sources & IBM technologies Dis2nguishing characteris2cs Built- in Analy2cs Big Data: Frequently Asked Ques2ons for IBM InfoSphere BigInsights h>p://

54 References T. Hey, S. Tansley, and K. Tolle, The Fourth Paradigm: Data- Intensive Scien2fic Discovery, T. Hey, S. Tansley, and K. Tolle, Eds. Microsob, Available: h>p://research.microsob.com/en- us/collabora2on/ fourthparadigm/ h>p:// knowledge- crea2on- data- driven- science h>p://royalsociety.org/uploadedfiles/ Royal_Society_Content/policy/projects/sape/ SAOE.pdf h>p:// item/sobware- sustainability- ins2tute- blog- how- big- big- data

55 Real2me Analy2cs for Big Data: A Facebook Case Study h>p://