Data Analytics Applied

Transcription

1 Data Analytics Applied A case study from the utilities sector Bram Steurtewagen - bram.steurtewagen@ugent.be - 1

2 Outline 1. Who are we? 2. Toolkit: R and PySpark 3. The Case Study 4. The Modular Solution I. Clustering II. Classification III. Evaluation 5. Challenges 6. Conclusions 2

3 Outline 1. Who are we? 2. Toolkit: R and PySpark 3. The Case Study 4. The Modular Solution I. Clustering II. Classification III. Evaluation 5. Challenges 6. Conclusions 3

4 Who are we? The Department of Marketing Analytics at Ghent University. (don't forget the WWW, its Welocity, Wolume, Wariety) Transversal team of data scientists combining: - Economics background - Analytical thinking - IT-oriented mindset Bram Steurtewagen - bram.steurtewagen@ugent.be - 4

5 Who am I? Bram Steurtewagen - bram.steurtewagen@ugent.be - 5

6 Outline 1. Who are we? 2. Toolkit: R and PySpark 3. The Case Study 4. The Modular Solution I. Clustering II. Classification III. Evaluation 5. Challenges 6. Conclusions 6

7 R and R-Studio The 'bread-and-butter' of Data Science Some pros: - CRAN: Lots of packages - Huge installed base - Picked up by MS (Microsoft R Open) - Only basic programming skills needed 7

8 R and R-Studio The 'bread-and-butter' of Data Science Some cons: - No coding standards - Limited scaling - R code "in production" is rare - However: MS integrating it into Microsoft SQL is huge for "production" code. 8

9 CRAN: R Packages FACULTY OF ECONOMICS data.table: intelligent data.frames ggplot2: the grammar of graphics caret: machine learning made easy domc: multicore support for caret 9

10 Apache Spark on Jupyter FACULTY OF ECONOMICS The new kid on the big data block Some pros: - Keeps on scaling - Coding standards - Huge momentum going into v2.0 - Outsources problems to DevOps 10

11 Apache Spark on Jupyter FACULTY OF ECONOMICS The new kid on the big data block Some cons: - Packages still under development - Doesn't tackle all problems equally - Spark code in production is "rare" - Requires programming background - Outsources problems to DevOps 11

12 Jupyter Interactive notebooks On the fly combining of Spark and Python Multi-language support Brings some of R's ease-of-use to the Spark environment 12

13 Spark modules and a Python package spark-csv: csv to Spark dataframes spark-ts: time series in Spark sparkml: machine learning pipelines matplotlib: honorable mention 13

14 Outline 1. Who are we? 2. Toolkit: R and PySpark 3. The Case Study 4. The Modular Solution I. Clustering II. Classification III. Evaluation 5. Challenges 6. Conclusions 14

15 The Case Study FACULTY OF ECONOMICS 15

16 The Case Study FACULTY OF ECONOMICS Load Profile Power Consumption Time of Day 16

17 The Case Study FACULTY OF ECONOMICS 17

18 The Case Study FACULTY OF ECONOMICS Smart Metering Infrastructure - Limited to a pilot project - Interesting 'Typical Daily Profiles' No further deployment of Smart Meters was planned on the Belgian market No way to generalize these 'Typical Daily Profiles' 18

19 Outline 1. Who are we? 2. Toolkit: R and PySpark 3. The Case Study 4. The Modular Solution I. Clustering II. Classification III. Evaluation 5. Challenges 6. Conclusions 19

20 The Modular Solution FACULTY OF ECONOMICS Unsupervised Component - We don't have labels for 'representative' patterns, so we're going to apply this to our load profiles (= cluster centers) Supervised Component - Link the external dataset to the cluster profile labels so we can then estimate new customers probably profiles 20

21 Outline 1. Who are we? 2. Toolkit: R and PySpark 3. The Case Study 4. The Modular Solution I. Clustering II. Classification III. Evaluation 5. Challenges 6. Conclusions 21

22 Clustering Algorithms: FACULTY OF ECONOMICS Unsupervised learning: - K-Means - DBSCAN - Spectral Clustering - Partition over Medians - Birch Going with K-Means for this exercise 22

23 FACULTY OF ECONOMICS K-Means Algorithm: iteration 0 23

24 FACULTY OF ECONOMICS K-Means Algorithm: iteration 1 24

25 FACULTY OF ECONOMICS K-Means Algorithm: iteration 2 25

26 FACULTY OF ECONOMICS K-Means Algorithm: solution x 26

27 K-Means applied to our Data (theory) Load Profile Power Consumption Time of Day 27

28 K-Means applied to our Data (theory) Companies Household Load Profile Power Consumption Time of Day 28

29 K-Means applied to our Data (practice) Energy Consumption data: 1. Read-in 2. Discover 3. Make "Typical Daily Profile" (mean per user) 4. Scale energy consumption (divide by user maximum) 5. Cluster 6. Evaluate results 7. Re-iterate if not satisfactory 29

30 Outline 1. Who are we? 2. Toolkit: R and PySpark 3. The Case Study 4. The Modular Solution I. Clustering II. Classification III. Evaluation 5. Challenges 6. Conclusions 30

31 Classification Algorithms: FACULTY OF ECONOMICS Supervised Learning: - Adaptive Boosting - Random Forests - Decision Trees - Logistic regression - Support Vector Machines - Neural Networks - Nearest Neighbours Going with Random Forest for this exercise 31

32 Random Forests FACULTY OF ECONOMICS source: scirp.org 32

33 Random Forests on our Data (theory) LotSize Region Thermal avgincome Label Client Ghent-Industry Cluster 1 Client Antwerp-Industry Cluster 1 Client 3 60 Nazareth Cluster 2 Client Antwerp Cluster 2 Client Ostend Cluster 2 source: Fictitious example 33

34 Random Forests on our Data (practice) Customer profile data: 1. Read-in 2. Split into training and test set 3. Train and fit model 34

35 Outline 1. Who are we? 2. Toolkit: R and PySpark 3. The Case Study 4. The Modular Solution I. Clustering II. Classification III. Evaluation 5. Challenges 6. Conclusions 35

36 Evaluation Metrics FACULTY OF ECONOMICS 1 Confusion Matrix - cutoff dependent 2 AUC (Multi-class or Binary) - cutoff independent 36

37 Evaluation Metrics: Confusion Matrix True Positives, True Negatives, False Positives and False Negatives If you also report accuracy with this grid, you have a good overview of what the model can do Can you permit a higher chance of being wrong when you say 'no' or when you say 'yes'? source: StackExchange 37

38 Evaluation Metrics: AUC (AUROC) Area under the Receiver-Operating Curve Sensitivity(TPR) vs 1 - Specificity(TNR) The probability that you rank a actual positive case higher than a negative case 50% = tossing a coin source: StackExchange 38

39 Metrics on our Data (practice) Calculating the AUC in RStudio 1. Load the package (proc) 2. Use the model on the test set to predict labels 3. Assess the model performance 39

40 Metrics on our Data (practice) Calculating the accuracy in PySpark 1. Use the model on the test set to predict labels 2. Calculate the accuracy to assess the final performance 40

41 Outline 1. Who are we? 2. Toolkit: R and PySpark 3. The Case Study 4. The Modular Solution I. Clustering II. Classification III. Evaluation 5. Challenges 6. Conclusions 41

42 Challenges Deploying the model Aligning model outcomes to business needs (and vice versa) Mining the customer profiles: - Web Scraping - Using Open API's - Both from commercial and non-commercial sources 42

43 Outline 1. Who are we? 2. Toolkit: R and PySpark 3. The Case Study 4. The Modular Solution I. Clustering II. Classification III. Evaluation 5. Challenges 6. Conclusions 43

44 Conclusions Successfully identifying key "Typical Daily Profiles" Successfully clustering key "Typical Daily Profiles" Successfully classifying new clients into those profiles Our models had adequate performance in a real-life context 44

45 Conclusions Data Analytics is not hard to get in to, but it is a very multidisciplinary field - It's not all about the business - It's not all about the algorithms - It's not all about the IT Every project in a new field is a new start PLAN-DO-CHECK-ACT Always look at your data before and after you model 45

46 Questions? Bram Steurtewagen - bram.steurtewagen@ugent.be

47 Full research paper: VERCAMER D., STEURTEWAGEN B., VAN DEN POEL D. & VERMEULEN F. (2016), Predicting Consumer Load Profiles Using Commercial and Open Data, Forthcoming in IEEE Transactions on Power Systems. Bram Steurtewagen - bram.steurtewagen@ugent.be