Technologies and algorithms to

Transcription

1 Big Data: Technologies and algorithms to deal with challenges Francisco Herrera Research Group on Soft Computing and Information Intelligent Systems (SCI 2 S) Dept. of Computer Science and A.I. University of Granada, Spain [email protected]

2 Big Data 2

3 Outline Big Data. Big Data Science Why Big Data? Technologies: MapReduce Paradigm. Hadoop Ecosystem Big Data Analytics. Algorithms Real Cases Big Data at SCI 2 S Final Comments. Challenges

5 What is Big Data? 3 Vs of Big Data volume Ej. Genomics Astronomy Transactions Gartner, Doug Laney, 2001

6 What is Big Data? 3 Vs of Big Data velocity Gartner, Doug Laney, 2001

7 What is Big Data? 3 Vs of Big Data variety Gartner, Doug Laney, 2001

8 What is Big Data? 3 Vs of Big Data Some Make it 4V s: Veracity

9 What is Big Data?

10 What is Big Data? No single standard definition Big data is a collection of data sets so large and complex that it becomes difficult to process using on-hand database management tools or traditional data processing applications. Big Data is data whose scale, diversity, and complexity require new architectures, techniques, algorithms, and analytics to manage it and extract value and hidden knowledge from it

11 What is Big Data? (in short) Big data refers to any problem characteristic that represents a challenge to process it with traditional applications

12 (Big) Data Science Data Science combines the traditional scientific method with the ability to explore, learn and gain deep insighti for (Big) Data It is not just about finding patterns in data it is mainly about explaining those patterns

13 Data Science Process Data Prepr rocess sing Clean Sample Aggregate Imperfect data: missing, noise, Reduce dim.... > 70% time! rocess sing Data P Explore data Represent data Link data Learn from data Deliver insight Analy ytics Data Clustering Classification Regressionession Network analysis Visual analytics Association

15 Why Big Data? Scalability to large data volumes: Scan 100 TB on 1 50 MB/sec = 23 days Scan on 1000-node cluster = 33 minutes Divide-And-Conquer (i.e., data partitioning) A i l hi t l l f d t A single machine can not manage large volumes of data efficiently

16 Why Big Data? Scalability to large data volumes: Scan 100 TB on 1 50 MB/sec = 23 days Scan on 1000-node cluster = 33 minutes Divide-And-Conquer (i.e., data partitioning) How must we process 1000 TB or TB?

17 Why Big Data? a MapReduce Scalability to large data volumes: Scan 100 TB on 1 50 MB/sec = 23 days Scan on 1000-node cluster = 33 minutes Divide-And-Conquer (i.e., data partitioning) MapReduce Overview: Data-parallel programming model An associated parallel and distributed implementation for commodity clusters Pioneered by Google Processes 20 PB of data per day Popularized by open-source Hadoop project Used by Yahoo!, Facebook, Amazon, and the list is growing

18 MapReduce MapReduce is a popular approach to deal with Big Data Data Fragmentation with Parallel Processing + Model fusion

19 MapReduce MapReduce workflow Blocks/ Intermediary Output fragments Files Files The key of a MapReduce data fragmentation approach is usually on the reduce phase J. Dean, S. Ghemawat, MapReduce: Simplified data processing on large clusters, Communications of the ACM 51 (1) (2008)

20 MapReduce Experience Runs on large commodity clusters: 10s to 10,000s of machines Processes many terabytes of data Easy to use since run-time complexity hidden from the users Cost-efficiency: Commodity nodes (cheap, but unreliable) Commodity network Automatic fault-tolerance (fewer administrators) Easy to use (fewer programmers)

21 MapReduce. Hadoop Hadoop is an open source implementation of MapReduce computational paradigm Created by Doug Cutting (chairman of board of directors of the Apache Software Foundation, 2010)

22 MapReduce. Hadoop birth July Hadoop Wins Terabyte Sort Benchmark One of Yahoo's Hadoop clusters sorted 1 terabyte of data in 209 seconds, which beat the previous record of 297 seconds in the annual al general purpose pose (Daytona) terabyte short bechmark. This is the first time that either a Java or an open source program has won.

23 MapReduce: Limitations If all you have is a hammer, then everything looks like a nail. The following types of algorithms are examples where MapReduce: Pregel (Google) Iterative Graph Algorithms: PageRank Gradient Descent Expectation Maximization

24 MapReduce: Limitations More than applications in Google Enrique Alfonseca Google Research Zurich 24

25 Hadoop On the limitations of Hadoop. New platforms GIRAPH (APACHE Project) ( Procesamiento iterativo de grafos Twister (Indiana University) Clusters propios GPS - A Graph Processing System, (Stanford) para Amazon's EC2 Distributed GraphLab (Carnegie Mellon Univ.) Amazon's EC2 PrIter (University of Massachusetts Amherst, Northeastern University-China) Cluster propios y Amazon EC2 cloud HaLoop (University of Washington) cs edu/node/14 Amazon s EC2 Spark (UC Berkeley) (100 times more efficient i than GPU based platforms Mars Grex Hadoop, including iterative algorithms, according to creators)

26 Hadoop Ecosystem The project GIRAPH (APACHE Project) ( Iterative Graphs Recently: Apache Spark Spark (UC Berkeley) (100 times more efficient than Hadoop, including iterative algorithms, according to creators)

27 Spark Hadoop: In Memory oy Hadoop Evolution InMemory HDFS Hadoop + SPARK Ecosystem Apache Spark Future version of Mahout for Spark Databricks, Groupon, ebay inc., Amazon, Hitachi, Nokia, Yahoo!,

28 Spark Hadoop. Spark birth October 10, 2014 Using Spark on 206 EC2 nodes, we completed the benchmark in 23 minutes. This means that t Spark sorted the same data 3X faster using 10X fewer machines. All the sorting took place on disk (HDFS), without using Spark s inmemory cache.

29 Spark Hadoop. Spark birth

30 Why Big Data? MapReduce Paradigm. Hadoop Ecosystem (A Snapshot) Big Data is high-volume, high-velocity and high-variety information assets that demand cost-effective, innovative forms of information processing for enhanced insight and decision making. There are two main issues related to Big Data: 1. Database/storage t frameworks: to write, read, and manage data. 2. Computational models: to process and analyze data. Recently, there are the following Big Data frameworks: 1. Storage frameworks: Google File System (GFS), Hadoop Distributed File Systems (HDFS). 2. Computational models: MapReduce (Apache Hadoop), Resilient Distributed Datasets (RDD by Apache Spark).

32 Big Data Analytics Potential scenarios: Clustering Classification Real Time Analytics/ Big Data Streams Association Recommendation Systems Social Media Mining Social Big Data

33 Big Data Analytics: Tools Generation 1st 2nd Generation 3nd Generation Generation Examples SAS, R, Weka, Mahout, Pentaho, Spark, Haloop, GraphLab, SPSS, KEEL Cascading Pregel, Giraph, ML over Storm Scalability Vertical Horizontal (over Horizontal (Beyond Hadoop) Hadoop) Algorithms Available Algorithms Not Available Fault- Tolerance Huge collection of algorithms Practically nothing Single point of failure Small subset: sequential logistic regression, linear SVMs, Stochastic Gradient Decendent, k- means clustsering, Random forest, etc. Vast no.: Kernel SVMs, Multivariate Logistic Regression, Conjugate Gradient Descendent, ALS, etc. Most tools are FT, as they are built on top of Hadoop Much wider: CGD, ALS, collaborative filtering, kernel SVM, matrix factorization, Gibbs sampling, etc. Multivariate logistic regression in general form, k-means clustering, etc. Work in progress to expand the set of available algorithms FT: HaLoop, Spark Not FT: Pregel, GraphLab, Giraph

34 Big Data Analytics: Tools Mahout MLlib Version 1.4.1

35 Big Data Analytics: Tools MLlib

36 Big Data Analytics Case of Study: Random Forest for KddCup 99 The RF Mahout Partial implementation: is an algorithm that builds multiple trees for different portions of the data. Two phases: Building phase Classification phase 36

37 Big Data Analytics Case of Study: Random Forest for KddCup 99 Class Instance Number normal DOS PRB R2L U2R 52

38 Big Data Analytics Case of Study: Random Forest for KddCup 99 Class Instance Number normal DOS PRB R2L U2R 52 Cluster ATLAS: 16 nodes -Microprocessors: p 2 x Intel E (6 cores/12 threads, 2 GHz) - RAM 64 GB DDR3 ECC 1600MHz -Mahout version 0.8

39 Big Data Analytics:Two last comments Image Credit: Shutterstock Without Analytics, Big Data is Just Noise Guest post by Eric Schwartzman, founder and CEO of Comply Socially

40 Big Data Analytics:Two last comments Mahout Lack of big data preprocessing algorithms MLlib Version 1.4.1

41 Big Data Analytics Data Preprocessing Big Data Data Science Model building Predictive and descriptive Analytics Lack of big data preprocessing algorithms

42 Outline Big Data. Big Data Science Why Big Data? Technologies: MapReduce Paradigm. Hadoop Ecosystem Big Data Analytics. Algorithms Real Cases. Health, Social Media and People shopping mall identification Big Data at SCI 2 S Final Comments. Challenges

43 Real Cases. Health HEALTHCARE ORGANIZATIONS HAVE COLLECTED EXABYTES OF DATA LAST YEARS Big data can facilitate action on the modifiable risk factors that contribute to a large fraction of the chronic disease burden, such as physical activity, diet, tobacco use, and exposure to pollution. Big Data, Vol.1, 3, 2013

44 Real Cases. Health and Social Media. Detect pandemic risk in real time Google Flu Trends (2008) ( is a web service operated by Google. They could say, as the prevention and control centers, where the flu had spread but almost in real time, not one or two weeks. It provides estimates of influenza activity for more than 25 countries. ti By aggregating Google search queries (45 different terms) J. Ginsberg, M.H. Mohebbi, R.S. Patel, L. Brammer, M.S. Smolinski, L. Brilliant. Detecting influenza epidemics using search engine query data. Nature 475 (2009)

45 Real Cases. Health and Social Media. Detect pandemic risk in real time Google Flu J. Ginsberg, M.H. Mohebbi, R.S. Patel, L. Brammer, M.S. Smolinski, L. Brilliant. Detecting influenza epidemics using search engine query data. Nature 475 (2009)

46 Real Cases. Health and Social Media. Detect pandemic risk in real time Google Flu In 2013 he overestimated the levels of influenza (prevention and control centers estimate x2). The overestimation may be due to extensive media coverage of flu can modify behaviors search SCIENCE VOL MARCH 2014

47 Real Cases. Health and Social Media. Detect pandemic risk in real time Google Flu The models are updated annually 2014 model update applied to prior years The model has been extended to Dengue

48 Real Cases. Health and Social Media. Detect pandemic risk in real time Wikipedia is better than Google at tracking flu trends? Wikipedia traffic could be used to provide real time tracking of flu cases, according to the study published by John Brownstein. Wikipedia Usage Estimates Prevalence of Influenza-Like Illness in the United States in Near Real-Time David J. McIver, John S. Brownstein, Harvard Medical School, Boston Children s Hospital, Plos Computational Biology, 2014.

49 Real Cases. Health and Social Media You Are What You Tweet: Analyzing Twitter for Public Health Discovering Health Topics in Social Media Using Topic Models Michael J. Paul, Mark Dredze, Johns Hopkins University, Plos One, 2014 Analyzing user messages in social media can measure different population characteristics, including public health measures. A system filtering Twitter data can automatically infer health topics in 144 million Twitter messages from 2011 to ATAM discovered 13 coherent clusters of Twitter messages, some of which correlate with seasonal influenza (r = 0.689) and allergies (r = 0.810) temporal surveillance data, as well as exercise (r =.534) and obesity (r =2.631) related geographic survey data in the United States. 49

50 Real Cases. Shopping mall identification. 3 MONTHS OF CREDIT CARD RECORDS 11 MILLIONS OF PEOPLE SCIENCE VOL 347 January 30, 2015, pp

52 Big Data at SCI 2 S - UGR Bird's eye view SCI 2 S website

53 Big Data at SCI 2 S - UGR Our approaches: Big Data with Fuzzy Models Fuzzy Rule Based System for classification Fuzzy Rule Based System with cost sensitive for imbalanced data sets

54 Big Data at SCI 2 S - UGR Our approaches: Imbalanced Big Data Preprocessing: Random undersampling, oversampling Cost sensitive Bioinformatic Applications

55 Big Data at SCI 2 S - UGR Our approaches: Big Data Preprocessing Evolutionary data reduction Feature selection and discretization

56 ECBDL 14 Big Data Competition Vancouver, 2014 ECBDL 14 Big Data Competition 2014: Self-deployment track Objective: Contact map prediction Details: 32 million instances 631 attributes (539 real & 92 nominal values) 2 classes 98% of negative examples About 56.7GB of disk space Evaluation: True positive rate True negative rate TPR TNR J. Bacardit et al, Contact map prediction using a large-scale ensemble of rule sets and the fusion of multiple predicted structural features, Bioinformatics 28 (19) (2012)

57 Evolutionary Computation for Big Data and Big Learning Workshop ECBDL 14 Big Data Competition 2014: Self-deployment track The challenge: Very large size of the training set Does not fit all together in memory. Even large for the test set (5.1GB, 2.9 million instances) Relatively high dimensional data. Low ratio (<2%) of true contacts. t Imbalance rate: >49 Imbalanced problem! Imbalanced Big Data Classification

58 Imbalanced Big Data Classification A MapReduce Approach 32 million instances, 98% of negative examples Low ratio of true contacts (<2%). Imbalance rate: > 49. Imbalanced problem! Previous study on extremely imbalanced big data: S. Río, V. López, J.M. Benítez, F. Herrera, On the use of MapReduce for Imbalanced Big Data using Random Forest. Information Sciences 285 (2014) Over-Sampling Random Focused Under-Sampling Random Focused Cost Modifying i (cost-sensitive) Boosting/Bagging approaches (with preprocessing)

59 ECBDL 14 Big Data Competition Our approach: ECBDL 14 Big Data Competition Balance the original training data Random Oversampling (As first idea, it was extended) 2 Learning a model 2. Learning a model. Random Forest

60 ECBDL 14 Big Data Competition Oversampling rate: {100%} RandomForest

61 ECBDL 14 Big Data Competition

62 ECBDL 14 Big Data Competition Our approach: ECBDL 14 Big Data Competition Balance the original training data Random Oversampling incresing the ROS percentage (As first idea, it was extended) 2. Learning a model. Random Forest 3. Detect relevant features. es 1. Evolutionary Feature Weighting Classifying test set.

63 ECBDL 14 Big Data Competition How to increase the performance? Third component: MapReduce Approach for Feature Weighting for getting a major performance over classes

64 ECBDL 14 Big Data Competition Last decision: We investigated to increase ROS until 180% with 64 mappers 64 mappers Algorithms TNR*TPR Training TPR TNR TNR*TPR Test ROS+ RF (130%+ FW 90+25f+200t) ROS+ RF (140%+ FW 90+25f+200t) ROS+ RF (150%+ FW 90+25f+200t) ROS+ RF (160%+ FW 90+25f+200t) 0, ROS+ RF (170%+ FW 90+25f+200t) ROS+ RF (180%+ FW 90+25f+200t) 0, To increase ROS and reduce the mappers number lead us to get a trade-off with good results ROS replications of the minority instances

65 ECBDL 14 Big Data Competition Evolutionary Computation for Big Data and Big Learning Workshop Results of the competition: Contact map prediction TPR Team Name TPR TNR Acc TNR Efdamis ICOS UNSW HyperEns PUC-Rio_ICA Test EmeraldLogic EFDAMIS team ranked first in the ECBDL 14 big data competition LidiaGroup

66 ECBDL 14 Big Data Competition Our algorithm: ROSEFW-RF MapReduce Process Iterative MapReduce Process I. Triguero, S. del Río, V. López, J. Bacardit, J.M. Benítez, F. Herrera. ROSEFW-RF: The winner algorithm for the ECBDL'14 Big Data Competition: An extremely imbalanced big data bioinformatics problem imbalanced big data bioinformatics problem. Knowledge-Based Systems, Volume 87, October 2015, Pages

68 Final Comments Parallelization of machine learning algorithms with datapartitioning approaches can be successfully performed with MapReduce. Partitioning and applying the machine learning algorithm to each part. Focus on the combination phase (reduce). The combination of models is the challenge for each algorithm. Data Mining, Machine learning and data preprocessing: Huge collection of algorithms against big data analytics algorithms

69 Final Comments Data Mining, i Machine learning and data preprocessing: Huge collection of algorithms Big Data Analytics Big Data: A small subset of algorithms Big Data Preprocessing: A few methods for preprocessing in Big Data analytics. Deep learning: Neural networks based approach for signal/image processing in big data.

70 Final Comments Big data and analytics: a large challenge offering great opportunities A small subset of algorithms. It is necessary to redesign new algorithms Computing Model Accuracy and Approximation Efficiency requirements for Algorithm Quality data for quality models in big data analytics Quality decisions must be based on quality big data! Big Data Preprocessing Noise in data distorts Need automatic methods for cleaning the data Missing values management Big Data Reduction

71 Final Comments Where we are going: 3BigDatastages By Gregory Piatetsky, Dec 8, Big Data 3.0: Intelligent Google Now, Watson (IBM)... Big Data 3.0 would be a combination of data, with huge knowledge bases and a very large collection of algorithms, perhaps reaching the level of true Artificial Intelligence (Singularity?).

72 Final Comments

73 Thanks!!! Big Data: Technologies Big Data: Technologies and Applications