Of all the data in recorded human history, 90 percent has been created in the last two years. - Mark van Rijmenam, Think Bigger, 2014

Transcription

1 What is Big Data?

2 Of all the data in recorded human history, 90 percent has been created in the last two years. - Mark van Rijmenam, Think Bigger, 2014

3 Data in the Twentieth Century and before

4 In 1663, John Graunt recorded information about mortality rolls in London He wanted to gain an understanding of the ongoing bubonic plague

5

6

7 1943: Colossus Searched for patterns in intercepted Nazi messages at 5000 charters per second. Reduced time from weeks to hours.

8 1952: Harry Truman created the National Security Agency (NSA), one of the largest US intelligence organizations ever created, to decipher cold war communications.

9 1965: First US data centre to store more than 742 million tax returns and 175 million sets of fingerprints. Represented beginning of electronic storage era.

10 He wanted to facilitate the sharing of information through a hypertext system. 1989: Creation of the World Wide Web by Tim Berners- Lee, software engineer at CERN in Geneva, Switzerland.

11 - We are developing a super-computer that will do more calculating in a second than a person with a hand-held calculator can do in 30,000 years. Democratic presidential nomination acceptance speech, Bill Clinton, 1995

12 Data in the Twenty-First Century

13 2005: Roger Mougalas of O Reilly Media coins the term Big Data The term referred to a large set of data that is almost impossible to manage and process using traditional business intelligence tools. 2005: Yahoo! creates Hadoop on top of Google s MapReduce It s goal was to index the entire WWW. Today open-source Hadoop is used around the world to crunch massive data sets.

14 2009: The Indian government took an iris scan, fingerprint and photograph of all its 1.2 billion inhabitants to store in the world s largest biometric database. 2010: Eric Schmidt, Executive chairman of Google: Every two days now we create as much information as we did from the dawn of civilization up until 2003 That s something like five exabytes of data

15 2011: By 2018, the US will face a shortage of k data scientists, as well as 1.5m data managers.

16 - Big Data = Big Storage In fact, the amount of data is growing at such an explosive rate that we have gone past the decimal system. Mark van Rijmenam, Think Bigger, 2014

17

18

19 What is Big Data?

20 Everything that is digital is data. Existing hardware and software cannot deal with the vast amounts of data being created at such high speeds. Big Data is too complex, too dynamic, and too vast for us to process, store, analyze and manage with traditional data tools.

21

22

23 Big Data can be explained according to 3 Vs: Velocity, Variety, Volume. However, three more (+1) Vs are also pondered: Veracity, Variability, Visualization, and Value

24 The magical number 3 plus or minus 3 more

25 Velocity The speed at which data is created, stored, analyzed and visualized. Every minute we upload 100 hours of video to YouTube Over 200 million s are sent 20 million photos are viewed 30,000 photos are uploaded to Flickr 300,000 tweets are sent 2.5 million queries are performed on Google The velocity challenge of Big Data is a real-time challenge.

26 Variety The structure of data. Over 90% of data today is unstructured Data now comes in structured, semistructured, unstructured and complex structured formats Each data type requires a different type of analysis and tools to interpret. Social media: Twitter and Facebook Sensor data The variety of data types are vast and rapidly expanding.

27

28

29

30

31

32 Volume The sheer amount of data that is created. The amount of data we create online will double every two years In 2011, we created 1.8 zettabytes of data By 2020, IDC estimates 50 times that amount The largest contributor to the ever expanding digital universe is the Internet of Things The variety of data types are vast and rapidly expanding.

33 Volume continued Airplanes generate approximately 2.5 billion terabytes of data each year from the sensors in their engines. John Deere uses sensor data to monitor machine optimization. Shell s sensors find additional oil in wells. The SQAT (Square Kilometer Array Telescope is estimated to generate 1 exabyte of data per day.

34 Veracity Commitment to the correctness of the data, and accuracy of the analysis Variability Data that does not commit to a fixed pattern or context: the definition is changing. (problem in sentiment analysis) Visualization Making vast amount of data comprehensible in a manner easy to read and understand

35 Value Of course, data in itself is not valuable at all. The value is in the analyses done on that data and how the data is turned into information and, eventually, knowledge and wisdom. the value is in how organizations use that data to create information-centric companies that base their decision making on insights derived from data analyses. - Mark van Rijmenam, Think Bigger, 2014

36 How can we do this? Turn to Moore s and Metcalf s Laws Decreasing storage costs Faster networking of computers Better storage solutions (Hadoop) Algorithms to create meaning from all this data

37 Eight Big Data Realities 1. Big Data requires a different culture 2. People are the drivers behind Big Data 3. Big Data is everywhere 4. Big Data engineers and analysts will be scarce 5. Big Data does require big security measures 6. Debates about privacy are inevitable 7. Governments are increasing their Big Data efforts 8. Big Data is not all about the quantity of data

38 Big Data Tools in this course Excel 2013 HortonWorks Hadoop Google BigQuery Tableau Public RapidMiner Studio Gapminder Big Data marketplaces