R YOU READY FOR PYTHON? Sunday 19th April, 2015

Size: px

Start display at page:

Download "R YOU READY FOR PYTHON? Sunday 19th April, 2015"

Gerald Tyler
8 years ago
Views:

1 R YOU READY FOR PYTHON? Sunday 19th April, 2015

2 THIS IS NOT A PYTHON VS R TALK credits -

3 WHO ARE WE? Danilo Maurizio Advanced Analytics Division data must be attended to give them confidence Gianluca Emireni Advanced Analytics Senior a life spent between good (statistics) and evil (IT) "@horsa.it, ".join(["danilo.maurizio","gianluca.emireni",""])[:-2]

4 WHO ARE WE? :-)

5 THE MOST COMPLETE REFERENCE :-) Data scientist Average programmer looking for a job that pays as much as what a top programmer would get. Sometimes also goes by the name data analyst. Statistician Mathematician who can t program. Correlation does not imply causation We looked at the wrong data set and can t draw any conclusions from it. Often represented in a graph to create the illusion of adding value. Machine Learning Statistical technique used by the sales and marketing department of big data vendors to secure their yearly bonus. (also see Our company is big data ready ) Hadoop Open-source software used for distributed computing. Data Scientists seem to have a quota to drop the name every two sentences when talking big data, but most only know the logo is a yellow elephant. There is a significant effect but Sentence-start used by data scientists or statisticians when they ve put weeks of work into their analysis, the results look fishy and not as expected, and there is no time to redo the analysis.

6 spurious correlation - ON TORTURING DATA

7 WE PREFER THIS: DRAWING INFERENCES FROM THE DATA Data scientist is sometimes used as an excuse for ignorance, as in I don t understand probability and all that stuff, but I don t need to because I m a data scientist, not a statistician Data science could be a useful umbrella term for statistics, machine learning, decision theory, etc. Also, the title data scientist is rightfully associated with people who have better computational skills than statisticians typically have John D. Cook:

8 WHAT WE DO We draw inference from data using: R python legacy stats suite (SAS, SPSS, SAP predictive stack, STATA)

9 COMFORTABLE WITH OPEN SOURCE project python pandas R RStudio Spark estimated effort (COCOMO model) $ 15 Mio. $ 2.5 Mio. $ 12 Mio. $ 6 Mio. $ 5.3 Mio.

10 more and more often we turn around the central question: how to balance and mix the best of R and python?

11 WHEN PYTHON We slightly and slowly moved all of our data management towards python (etl and data movement)

12 AND WHEN R while being tied to R for statistical learning

13 PURSUING THE BEST BALANCE

14 RANDOM FORESTS THROUGH SCIKIT- LEARN ON REVERSE LOGISTIC Fashion e-commerce has a huge problem with return rate - most of us have wives and credit cards :-) Purchase history and carts have enough information to train a model? (we know that features design/selection is the most time consuming activity) We tried hard and succeeded :-) Most important features: fit index, cart entropy, past shoppers attitude, transaction value, product quantity, Now deployed real time as a web service with milliseconds response time using: flask, circus, scikit-learn used to dynamically set shipping price

15 WHY PYTHON? We love caret library, an R counterpart of scikit-learn, but is much easier web serving this kind of model on python stack. 150 lines of code are enough to deliver json document with shopping bags return probability.

16 WHY R? In some contexts, CRAN shop (R libraries repository) offer very mature packages able to solve the whole class of statistical problems. For example, time series forecasting and statistical matching (causal inference) are kind of problems where R outperforms python in terms of completeness, documentation, deepness, The forecast library (Rob J. Hyndman) is the best in class package for time series modeling, not only for code quality but also for its theoretical and methodological support. The MatchIt and CEM library (Iacus, King, Porro) offer a full range of techniques to perform statistical matching.

17 flickr CC - BETTER TOGETHER

18 WHY TOGETHER? The wide range of problems we are called to face let us use both languages together: sometimes python leads the analysis, other times is R. R and python glued together:

19 flickr CC - WE ARE IN GOOD COMPANY

20 DO YOU KNOW STACKOVERFLOW? How many times, googling for help, you have been led to stackoverflow? Did you notice that R users and pydata users meet together in a huge number of threads?

21 STACKOVERFLOW - #PYDATA

22 Disclaimer: not done with matplotlib YET ANOTHER TAG CLOUD #1/2

23 STACKOVERFLOW - #RSTATS

24 Disclaimer: not done with ggplot2 YET ANOTHER TAG CLOUD #2/2

25 QUESTIONS How wide is the area of stackoverflow users active in both python and R Q&A? Is there any difference among the behavior of polyglots and their purists colleagues? Are they finally any smarter?

26 DATA CAPTURE Thanks to StackExchange data explorer* we designed a bunch of query to harvest information about questions and answers related to tags associated with python data stack (scipy, numpy, pandas, scikit-learn) and R (r, rstat, r-faq). On top of this data we built a user registry, labeling each user according to their joint (in)activity in these two different domains. (*)

27 A STEADY GROWTH

28 GROUPS OVERLAP users Pythonistas

29 POLYGLOTS Stackoverflow users you can find in both groups

30 AND THEIR ACTIVITY users Pythonistas Q&A volumes

31 AVERAGE INTERACTIONS PER USER

32 WHO OWNS THE KNOWLEDGE? Pareto point of view for Pythonistas

33 WHO OWNS THE KNOWLEDGE? Pareto point of view for Pythonistas _Pythonistats are users active on stackoverflow for #pydata related #tags _80% of total amount of answers is given by (core) users _more than users have never answered to a question

34 WHO OWNS THE KNOWLEDGE? Pareto point of view for users

35 WHO OWNS THE KNOWLEDGE? Pareto point of view for users _users are users active on stackoverflow for #rstats related #tags _80% of total amount of answers is given by less than (core) users _more than users have never answered to a question

36 POLYGLOTS ARE PROBLEM SOLVERS? We tried to explain the probability of a Question being successfully closed with an accepted Answer, using some regressors: year and month of the question view counts, score, favorites and comments summed by post number of answers given by polyglots and number of answers given by Pythonistas/useRs plus some tags used as dummies The main question is: are polyglots smarter than R and python purists?

37 POLYGLOTS VS PYTHONISTAS While Pythonistas answers are likely to reduce the probability of the post being closed, the effect of polyglots contributions to posts is statistically significantly greater than zero, having a positive effect.

38 POLYGLOTS vs users Also R based posts benefit of polyglots interventions, showing a positive effect on the probability of successful post closing.

39 TOWARDS A LESS TAUTOLOGICAL QUESTION ﬂickr cc -

40 HOW LONG DOES IT TAKES TO ANSWER A QUESTION? How Pythonistas, users and Polyglots presence affects questions lifetime?

41 YOUR #PYDATA QUESTIONS WILL BE ANSWERED IN 200* MINUTES OR NEVER Questions are solved mostly immediately, 50% closed within 40 minutes *3rd quartile

42 PYTHON QUESTIONS LIFETIME Polyglots presence contribute on reducing question resolution Kaplan-Meier survival estimates

43 YOUR #RSTATS QUESTIONS WILL BE ANSWERED IN 136 MINUTES OR NEVER Questions are solved mostly immediately, 50% closed within 32 minutes *3rd quartile

44 R QUESTIONS LIFETIME users and Polyglots presence ensure the lowest response time Kaplan-Meier survival estimates

45 PYTAGS QUESTIONS LIFETIME 1/3

46 PYTAGS QUESTIONS LIFETIME 2/3

47 PYTAGS QUESTIONS LIFETIME 3/3

48 from greeting import thankyou

DATA SCIENCE CURRICULUM WEEK 1 ONLINE PRE-WORK INSTALLING PACKAGES COMMAND LINE CODE EDITOR PYTHON STATISTICS PROJECT O5 PROJECT O3 PROJECT O2

DATA SCIENCE CURRICULUM WEEK 1 ONLINE PRE-WORK INSTALLING PACKAGES COMMAND LINE CODE EDITOR PYTHON STATISTICS PROJECT O5 PROJECT O3 PROJECT O2 DATA SCIENCE CURRICULUM Before class even begins, students start an at-home pre-work phase. When they convene in class, students spend the first eight weeks doing iterative, project-centered skill acquisition.