Fortgeschrittene Computerintensive Methoden

Transcription

1 Fortgeschrittene Computerintensive Methoden Einheit 3: mlr - Machine Learning in R Bernd Bischl Matthias Schmid, Manuel Eugster, Bettina Grün, Friedrich Leisch Institut für Statistik LMU München SoSe 2014

2 Introduction + Motivation No unifying interface for machine learning in R Experiments require lengthy, tedious and error-prone code Machine learning is (also) experimental science: We need powerful and flexible tools! mlr now exists for 2-3 years, grown quite large Still heavily in development, but official releases are stable Was used for nearly all of my papers We cannot cover everything today, short intro + overview Focus: Resampling / model selection / benchmarking! Bernd Bischl c SoSe 2014 Fortgeschrittene Computerintensive Methoden 3 1 / 17

3 Package + Documentation Main project page on Github URL: Contains further links, tutorial, issue tracker. Official versions are released to CRAN. How to install install.packages("mlr") install_github("mlr", username = "berndbischl") Documentation Tutorial on project page (still growing) R docs and examples in HTML on project page (and in package) Bernd Bischl c SoSe 2014 Fortgeschrittene Computerintensive Methoden 3 2 / 17

4 Features I Clear S3 / object oriented interface Easy extension mechanism through S3 inheritance Abstract description of learners and data by properties Description of data and task Many convenience methods, generic building blocks for own experiments Resampling like bootstrapping, cross-validation and subsampling Easy tuning of hyperparameters Variable selection Benchmark experiments with 2 levels of resampling Bernd Bischl c SoSe 2014 Fortgeschrittene Computerintensive Methoden 3 3 / 17

5 Features II Growing tutorial / examples Extensive unit-testing (testthat) Extensive argument checks to help user with errors Parallelization through parallelmap (local, socket, MPI and BatchJobs modes) Parallelize experiments without touching code Job granularity can be changed, e.g., so jobs don t complete too early Bernd Bischl c SoSe 2014 Fortgeschrittene Computerintensive Methoden 3 4 / 17

6 Remarks on S3 Not much needed (for usage) Makes extension process easy Extension is explained in tutorial If you simply use the package, you don t really need to care! Constructors / Accessors # c o n s t r u c t o r s task = makeclassiftask ( data = i r i s, t a r g e t = " Species " ) l r n = makelearner ( " c l a s s i f. lda " ) # accessors task$task. desc$target gettaskdata ( task ) Bernd Bischl c SoSe 2014 Fortgeschrittene Computerintensive Methoden 3 5 / 17

7 Overview of Implemented Learners Classification LDA, QDA, RDA, MDA Logistic / multinomial regression k-nearest-neighbours Naive Bayes Decision trees (many variants) Random forests Boosting (different variants) SVMs (different variants) Neural Networks... Regression Linear model Penalized LM: lasso and ridge Boosting k-nearest-neighbours Decision trees, random forests SVMs (different variants) Neural / RBF networks... Bernd Bischl c SoSe 2014 Fortgeschrittene Computerintensive Methoden 3 6 / 17

8 Examples ex1.r: Training and prediction ex2.r: Probabilities and ROC Bernd Bischl c SoSe 2014 Fortgeschrittene Computerintensive Methoden 3 7 / 17

9 Resampling Hold-Out Cross-validation Bootstrap Subsampling and quite a few extensions Bernd Bischl c SoSe 2014 Fortgeschrittene Computerintensive Methoden 3 8 / 17

10 Example 3 ex3.r: Resampling + Comparison Bernd Bischl c SoSe 2014 Fortgeschrittene Computerintensive Methoden 3 9 / 17

11 Remarks on model selection I Basic machine learning Fit parameters of model to predict new data Generalisation error commonly estimated by resampling, e.g. 10-fold cross-validation 2nd, 3rd,... level of inference Comparing inducers or hyperparameters is harder Feature selection either in 2nd level or adds a 3rd one... Statistical comparisons on the 2nd stage are non-trivial Still active research Very likely that high performance computing is needed Bernd Bischl c SoSe 2014 Fortgeschrittene Computerintensive Methoden 3 10 / 17

12 Tuning / Grid Search Used to find best hyperparameters for a method in a data-dependend way Most basic method Exhaustively try all combinations of finite grid Inefficient Searches large, irrelevant areas Combinatorial explosion Reasonable for continuous parameters? Still often default method Bernd Bischl c SoSe 2014 Fortgeschrittene Computerintensive Methoden 3 11 / 17

13 Example 4 ex4.r: Basic tuning: grid search Bernd Bischl c SoSe 2014 Fortgeschrittene Computerintensive Methoden 3 12 / 17

14 Remarks on Model Selection II Salzberg (1997): On comparing classifiers: Pitfalls to avoid and a recommended approach Many articles do not contain sound statistical methodology Compare against enough reasonable algorithms Do not just report mean performance values Do not cheat through repeated tuning Use double cross-validation for tuning and evaluation Apply the correct test (assumptions?) Adapt for multiple testing Think about independence and distributions Do not rely solely on UCI We might overfit on it Bernd Bischl c SoSe 2014 Fortgeschrittene Computerintensive Methoden 3 13 / 17

15 Nested Resampling Bernd Bischl c SoSe 2014 Fortgeschrittene Computerintensive Methoden 3 14 / 17

16 Example 5 ex5.r: Tuning + nested resampling via wrappers Bernd Bischl c SoSe 2014 Fortgeschrittene Computerintensive Methoden 3 15 / 17

17 Outlook / Not Shown Today I Available Smarter tuning algorithms CMA-ES, iterated F-facing, model-based optimization... Variable selection Filters: Simply imported available R methods Wrappers: Forward, backward, stochastic search, GAs Bagging for arbitary base learners Generic imputation for missing values Wrapping / tuning of preprocessing Regular cost-sensitive learning (class-specific costs)... Bernd Bischl c SoSe 2014 Fortgeschrittene Computerintensive Methoden 3 16 / 17

18 Outlook / Not Shown Today II Current work, most of it nearly done Survival analysis: Tasks, learners and measures Cost-sensitive learning (example-dependent costs) Over / Undersampling for unbalanced class sizes Multi-critera optimization... Bernd Bischl c SoSe 2014 Fortgeschrittene Computerintensive Methoden 3 17 / 17