Surfing the Data Tsunami: A New Paradigm for Big Data Processing and Analytics

Transcription

1 Surfing the Data Tsunami: A New Paradigm for Big Data Processing and Analytics Dr. Liangxiu Han Future Networks and Distributed Systems Group (FUNDS) School of Computing, Mathematics and Digital Technology, Manchester Metropolitan University June, 2014

2 Outline Data tsunami What is big data? Value of big data Challenges of big data Technologies for big data Data exploration for future

3 Data tsunami Increased capability of generating and capturing data (e.g. Petascale simulations, experimental devices, the Internet, sensors, etc.) 300m photos, 2.5m contents shared per day cabig: 4.7+millions for cancers

4 Data tsunami Gene expression data in GEO and ArrayExpress: over 1 millions Climate data from NASA: 32 Petabytes (2 50 ) SKA(The Square Kilometre Array): The data collected by the SKA in a single day would take nearly two million years to playback on an ipod.

5 Slide Credit: Intel

6 Data tsunami Data intensive era -- big data/data rich/datacentric/data-driven era 40 Data Volumes

7 What is big data? Data size representation Binary digit (bit) Byte(B): 8 bits Kilobyte (KB): 210 bytes Megabyte (MB): 220 bytes Gigabyte (GB): 230 bytes Terabyte (TB): 240 bytes; Petabyte (PB): 250 bytes Exabyte (EB): 260 bytes; Zettabyte (ZB): 270 bytes Yottabyte (YB): 280 bytes

8 What is big data? Big Data... and the Next Wave of InfraStress John R. Mashey Chief Scientist, SGI Timeline Technology Waves: NOT technology for technology s sake IT S WHAT YOU DO WITH IT But if you don t understand the trends IT S WHAT IT WILL DO TO YOU OK! Uh oh! 4/25/98 page the origins of big data D Data Management: Controlling Data Volume,Velocity, and Variety by Doug Laney widespread in the Economist Gartner, IBM, Cisco, Microsoft, etc Data, data everywhere A special report on managing information

9 What is big data? A relative term ( don t define it in terms of size being larger than a certain number of terabytes or petabytes) Larger, more complex and hard to access, organise and analysis beyond the capability of the existing tools (varying on sectors) The data volume, velocity or variety/complexity (3 V) limits the ability to perform effective analysis using traditional approaches

10 What is big data? Big data is about pushing limits!

11 What is big data? Volume (Data at rest) The size and scale of the data By 2015,it will reach 8 Zettabytes (IDC)

12 What is big data? Velocity (data in motion) Real time capture and analytics/streaming processing and analytics Stock exchange, fraud analysis/customer churn predictions

13 What is big data? Variety/complexity (data in many forms) Various formats, types and structures Structured data, e.g. data defined by schema, relational databases, or semi-structured (xml) Unstructured data, e.g. free form text, s, logs, images, audio, video, social media data (e.g. graph)

14 What is big data? Two more Vs Value: business value to be derived Veracity( data in doubt): the quality and understandability of the data!$#,-"))!"#$%&'()!"#$ %&'&$!*#,")!"+*%&'()!*+&"'()

15 Value of big data Next frontier for innovation, competition and productivity: Commerce and economy Science discovery in all most every science and engineering discipline for addressing societal challenges ( health, food, energy, environment, etc)

16 Value of big data Source: wikibon

17 Value of big data New paradigm Big Data leads science discovery

18 Source: samsung Challenges of big data Bottleneck in Technology: IT infrastructures

19 Source: eskills Challenges of big data Bottleneck in technical skills: professionals to handle big data!"#$%&'&$$ ())*+',-"./0$

20 Technologies for big data What kind of big data technologies in your mind? Cloud computing?...

21 Technologies for big data Big data processing and analytics Architectures for efficiently processing big data Data analytics for filtering, analysing and generating actionable insights 2+-3!"#"$6'078)9$ 6"&+)#19$ ('8%0):+#19$!"#"$%&'()**+,- $",.$/,"01*+* 4'5 4'5!"#"$6"07) 2+-3

22 Architectures Traditional approaches, for example, OLTP( online transaction processing) OLAP(online analytical processing): data warehouse Operations OLTP Business strategy Business process OLAP Informations Data analytics Decision making Business datawarehouse

23 Architectures Issues Relational databases (RDBMS), dealing with structured data only doesn t support complex analytics lacks scalability and performance

24 Architectures Current solutions Apache Hadoop: an open source for storage and large-scale processing of data-sets (both structured and unstructured data (nosql)); major components: HDFS, MapReduce, HCommon, HYarn Google File System and MapReduce Apache Spark: combine SQL, streaming, and complex analytics and in-memory computing

25 Architectures Parallel and distributed computing for data processing!"#$%"&'$()**!"#"$$%$&&

26 Architectures Parallelisation: a sought after solution for speeding up an application, particularly for data intensive applications Three considerations: How to distribute workloads or decompose an algorithm into parts How to map the tasks onto various computing nodes and execute subtasks in parallel How to coordinate and communicate subtasks on those computing nodes.

27 Architectures Data parallelism: workload are distributed into different computing nodes and the same task can be executed on different subsets of the data simultaneously Task parallelism: tasks are independent and can be executed purely in parallel Pipelining: an iteration of a task consisting of many stages, where each stage in the task is chained and executed in order and the output of one stage is the input of the next one. Pipelining can be implemented with streaming and without using streaming

28 Architectures Programming models for parallel and distributed computing (e.g. MPI, MapReduce, POSIX Threads, OpenMP, etc) Bridging the gap between the underlying hardware and the supporting layers of software available to applications Independent of programming languages and API

29 Architectures MapReduce: a programming model for processing large scale datasets Implementations(e.g. Google DFS, Apache Hadoop)

30 Architectures Map and Reduce functions Map: perform a function on an individual value of a data set and return a new list of values Given a dataset: A={1, 2, 3}, Map function: Square = X*X. After Map process, it returns {2,4,9}

31 Architectures Reduce: performa a function by combining values in a data set Given a dataset: B={2,4,9}, Reduce function: sum = X1+X2+X3. After Reduce process, it returns 15!!"#! "#$%&! '#(#!!!! $$%&'()& )*(+*(,-&./!0#1*&2!!

32 MapReduce Architectures <Hello, 4> <big data, 5> <Hello, 2> <big data, 3> <Hello, 6> <big data, 8> Map Reduce The question to count the words called hello and big data from big data

33 Architectures Comparison of RDBMS-based approaches, spark MapReduce

34 Data analytics Data analytics: discovery of useful, possibly unexpected, patterns in data, automation of data exploration and analysis Statistics analysis Machine learning/data mining, for example Classification, Clustering, Association rule, Regression, graph mining,...

35 Data analytics Classification AMD Diabetic Retinopathy

36 !!"#$%&'(')*+',-$.)&%.'+/'0,+)-123'$."2#'456&12',-$.)&%"2#'1-#+%")76' ' Data analytics! Clustering! Clustering of the fish industry in UK ''''''''''''''''''!"#$%&'8''!".7"2#'"23$.)%9''!!

37 Data analytics Graph mining... and so on Community detection in Facebook friends Source:

38 Big data technology = Architectures for data processing and management +Data analytics

39 Technologies for big data Future development: programming abstractions need to be developed to support and facilitate big data processing and analytics Apps Programming abstractions to support big data processing and analytics RDBMS NoSQL DFS...

40 Data exploration for future We focus on: ----both fundamental and applied research in large-scale data processing and analytics --- Intelligent management and optimisation of largescale networked distributed systems ( challenges: reliability, scalability, security, resilience, autonomy and self-adaptation)

41 Data exploration for future Food Health Future Energy People sustainability society Planning Manufacturing

42 Acknowledgement & Collaboration BBSRC EPSRC-DHPA Sustainability Society Network+ Amazon MMU Optos Fera MRC HGU

43 Acknowledgement & Collaboration Outside: MRC HGU, University of Edinburgh University of Manchester University of StrathClyde Heriot Watt University Loughborough University University of Glasgow... Optos Fera... University of Melbourne

44 Acknowledgement & Collaboration Inside: School of Science and Environment Business School School of Engineering Department of Sociology...

45 Thank You!