Obtaining Value from Big Data

Transcription

1 Obtaining Value from Big Data Course Notes in Transparency Format technology basics for data scientists Spring Jordi Torres, UPC - BSC

2 Data deluge, is it enough? 2

3 Data = Information? 3

4 Prediction using data models The information is non actionable knowledge 4

5 Obtaining value from data World is becoming instrumented and interconnected and we can take advantage of it if we can process it in real time. - Data + Data cannot be taken at face value Value + Information Knowledge Volume - The information is non actionable knowledge 5

6 Why Learn? Machine learning is programming computers to optimize a performance criterion using example data or past experience. There is no need to learn to calculate payroll Learning is used when: Human expertise does not exist, Humans are unable to explain their expertise Solution changes in time Solution needs to be adapted to particular cases Source: Lecture Notes for E Alpaydın 2010 Introduction to Machine Learning 2e The MIT Press (V1.0) 6

7 What We Talk About When We Talk About Learning Learning general models from a data of particular examples Data is cheap and abundant (data warehouses, ); knowledge is expensive and scarce. Example in retail: Customer transactions to consumer behavior: People who bought A also bought B ( Build a model that is a good and useful approximation to the data. Source: Lecture Notes for E Alpaydın 2010 Introduction to Machine Learning 2e The MIT Press (V1.0) 7

8 Where can be appried? Retail: Market basket analysis, Customer relationship management (CRM) Finance: Credit scoring, fraud detection Manufacturing: Control, robotics, troubleshooting Medicine: Medical diagnosis Telecommunications: Spam filters, intrusion detection Bioinformatics: Motifs, alignment Web mining: Search engines SmartCities: City planning And... dozens and dozens Source: Lecture Notes for E Alpaydın 2010 Introduction to Machine Learning 2e The MIT Press (V1.0) 8

9 Obtaining value from data In my opinion A Big Challenge is (Important research area) The majority of algorithms function well in thousands of registers, however at the moment they are impractical for thousands of milions. 9

10 What is Machine Learning? Optimize a performance criterion using example data or past experience. Statistics vs Computer science? Role of Statistics: Inference from a sample Role of Computer science: Efficient algorithms to Solve the optimization problem Representing and evaluating the model for inference Source: Lecture Notes for E Alpaydın 2010 Introduction to Machine Learning 2e The MIT Press (V1.0) 10

11 Example: Learning Associations Basket analysis: P (Y X ) probability that somebody who buys X also buys Y where X and Y are products/services. Example: P ( chips beer ) = 0.7 Source: Lecture Notes for E Alpaydın 2010 Introduction to Machine Learning 2e The MIT Press (V1.0) 11 1

12 Example: Classification Example: Credit scoring Differentiating between low-risk and high-risk customers from their income and savings Discriminant: IF income > θ 1 AND savings > θ 2 THEN low-risk ELSE high-risk Source: Lecture Notes for E Alpaydın 2010 Introduction to Machine Learning 2e The MIT Press (V1.0) 12 1

13 Example: Classification Applications Also know as Pattern recognition Face recognition: Pose, lighting, occlusion (glasses, beard), make-up, hair style Character recognition: Different handwriting styles. Speech recognition: Temporal dependency. Medical diagnosis: From symptoms to illnesses Biometrics: Recognition/authentication using physical and/or behavioral characteristics: Face, iris, signature, etc... Source: Lecture Notes for E Alpaydın 2010 Introduction to Machine Learning 2e The MIT Press (V1.0) 13

14 Example: Regression Example: Price of a used car x : car attributes y : price y = wx+w 0 Source: Lecture Notes for E Alpaydın 2010 Introduction to Machine Learning 2e The MIT Press (V1.0) 14

15 Machine Learning Usefulness Supervised Learning: Uses Prediction of future cases: Use the rule to predict the output for future inputs Knowledge extraction: The rule is easy to understand Compression: The rule is simpler than the data it explains Outlier detection: Exceptions that are not covered by the rule, e.g., fraud Source: Lecture Notes for E Alpaydın 2010 Introduction to Machine Learning 2e The MIT Press (V1.0) 15

16 Machine Learning: an impressive world! 16

17 Machine Learning: an impressive world! 17

18 Decision Trees (case study bigml) 18

19 Right questions? Tech problem of business problem? what to look for in the data? how to model the data? where to start??? Effective analysis depends more on asking the right question or designing a good experiment than on tools and techniques. 19

20 DATA vs MODEL Large datasets provide the opportunity to take advantage of.effective results from coupling large datasets with relatively simply algorithms imm_mid=09b70d&cmp=em-strata-newsletters-nov14-direct#more