Introducing open source statistical and data science tools to business analytics students and professionals

Transcription

1 Detroit ASA January 2015 Introducing open source statistical and data science tools to business analytics students and professionals Mark Isken Assoc. Prof. of MIS School of Business Administration Oakland University

2 Abstract Tools such as Excel, SQL databases, SPSS and SAS have long been staples of the quantitative side of business education and professional practice. Recently, this community has seen a surge in popularity of open source tools of a more computational nature. In response, in the spring of 2014, I developed and delivered a course entitled "Practical Computing for Business Analytics" within the School of Business at Oakland University. This course relied entirely on open source software for course development, delivery and student work. Specifically we used the Linux OS along with R and Python. R Markdown documents and IPython notebooks were the primary teaching tools. I will describe my teaching methods, discuss how the course went, and share plans for the continued dissemination of this material in the business analytics community.

3 Mark Isken BSE, MSE, Ph.D. in Industrial and Operations Engineering from University of Michigan Operations analyst for William Beaumont Hospital and Henry Ford Health System and some small consulting companies (~10 years) Joined OU Fall 1999 as full-time faculty member of Dept. of Decision and Information Sciences I m a techie love working with computers and mathematical models to help solve business problems Teach/taught business analytics, statistics, computer simulation, intro MIS courses and healthcare operations mgt I remember when INFORMS was ORSA and TIMS and the controversy that ensued when merger proposed. 3

4 X X X Management science Business analytics Machine learning Decision science Data science Operations research Analytics Data visualization OLAP Statistics Business intelligence Data mining Data warehousing Knowledge discovery in databases Big data

5 Healthcare Operations Analysis Internal business analysis / decision support consultant Simulation modeling Critical care tower, emergency departments Pneumatic tube systems, outpatient clinics, pharmacy robots Staffing and scheduling models people, cases, tests, etc. Queueing, simulation, optimization Database and analytical tool development using Access, Excel, VBA and other software Various statistical and operations analysis studies Short term census forecasting 45 Postpartum Staffing Needs Nurses Sun 12 am Sun 06 am Sun 12 pm Sun 06 pm Mon 12 am Mon 06 am Mon 12 pm Mon 06 pm Tue 12 am Tue 06 am Tue 12 pm Tue 06 pm W ed 12 am W ed 06 am W ed 12 pm W ed 06 pm Thu 12 am Thu 06 am Thu 12 pm Thu 06 pm Fri 12 am Fri 06 am Fri 12 pm Fri 06 pm Sat 12 am Sat 06 am Sat 12 pm Sat 06 pm Introduction to BAM 5

6 My view of business analytics Programming & databases Math & stats Domain knowledge Art and Craft of modeling Communication, visualization, story-telling

7 Spreadsheet Based Modeling & Decision Support (taught since 2001) Management science modeling Simulation Optimization Data analysis Database EDA/data viz Statistics OLAP/DW Data Mining Application Development System design User Forms Automation Environment customization, error prevention & handling Basic foundation Modeling Spreadsheet Modeling/engineering VBA Introduction to BAM 7

8 Getting started with Free and Open Source Software (FOSS) PhD days using FORTRAN based network flow algorithms for scheduling problems in addition to commercial tools like IBM's OSL and CPLEX Clearly saw FOSS allowed me to learn by code exploration It allowed me to create decision support apps with sophisticated code built in that didn't force end user organizations (hospitals with no extra $$$) to buy expensive commercial software that I could extend this software to solve my specific problem better Wrote dissertation in LaTex My research experience along with several years as a practicing industrial engineer with two large healthcare systems and a few years of university teaching launched my real plunge into the world of FOSS

9 FOSS for analytics in practice A smallish healthcare analytics firm run by a good friend of mine from grad school days Very Microsoft-centric place and client base (SQL Server,.Net apps, Excel, Access, PPT) I've been introducing and helping people get up to speed with things like R and Python to overcome common limitations of their current Excel centric analytical workflow practices ad hoc and non-reproducible data cleaning, transforming lots of pointing and clicking for repetitive tasks sketchy documentation of analytical workflow ease of doing things with R (via apply family or plyr and ggplot2) and Python (via pandas and matplotlib) such as percentile calculations within pivot style or group by analysis small multiples both of the above are hideous to do in Excel and even the first is tough in specialized tools like Tableau. I've got beginner level tutorials on these things on hselab.org both in R and Python

10 No fun to do in Excel Small multiples Percentiles by group

11 hselab.org This is my primary outlet for sharing tutorials, teaching materials, FOSS and other analytics related things Tutorials and guides Blog posts Links to my FOSS projects Working on Shiny apps Open courses Science, engineering, research are all evolving in response to calls for reproducibility, open access to data and results, changes to publishing models and the possibilities offered by FOSS along with internet infrastructure that facilitates organic evolution of social and technical ecosystems Got me thinking we really needed a course on this stuff within the School of Business...

12 MIS 480/680 - Practical Computing for Business Analytics Hey MBAs! Microsoft isn't the only game in town. If you really want to do analytics in the business world, you better learn to do some programming!

13 Structure 14 3hr sessions 202EH Computer Teaching Lab First half of the semester Session 1: Intro to analytics Second half of the semester 9/10: Intro to Python 2: Intro to R and R Studio 11: Data analysis and plotting in Python 3: Exploratory data analysis with R 12: Data acquisition, prep and more analysis 4: Group by analysis and more stats - R 13: Time series, datetime analysis in Python 5: Linear models in R 6: Data mining in R (knn, cluster, Rattle) 7/8 Text files, regex, Linux tools (e.g. shell, grep, basic scripting) 14: Overview of Hadoop & MapReduce

14 Open Source Tools

15 Our computing appliance - pcba Computer running Windows, Mac OS, or Linux Programs MS Office Notepad Browser VirtualBox Documents Spreadsheets, Word documents, text files,pdf Virtual machines VM running Lubuntu Linux Programs R, R Studio, R packages Python (Anaconda) Geany OpenOffice Browser File Manager Shell Documents R scripts, Python programs OpenOffice documents Text files, pdf pcba

16 No one book really fit

17 Why R and Python? Both R and Python are widely used in the data science and business analytics worlds A quote from Enterprise Data Analysis and Visualization: An Interview Study on the growing need for technically adept analysts: When discussing recruitment, one Chief Scientist said analysts that can t program are disenfranchised here Both support a combination of interactive use via tools like R Studio and IPython along with programmatic use via text scripting Huge communities and ecosystems supporting R and Python for analytics work Both facilitate reproducible analysis Some things that are simply hideously difficult to do in tools like Excel or a database, are simple in R and/or Python Group By or Pivoting type analysis for operations such as percentiles Small multiples and other complex graphing/charting/plotting Documenting and reproducing complex series of data cleaning and transformations

18 Flow of a typical class Guided exploration of topics via interactive use of R Markdown documents or IPython Notebooks In class assignment where I act as roving consultant Open lab time for homework and project work Collaborate with classmates Get questions answered by me

19 R Markdown documents Mixture of markdown (simple plain text formatting) and executable R code chunks Facilitates authoring informative and reproducible analysis documents Can generate output in numerous forms including PDF, HTML, MS Word Can publish resultant HTML directly to RPubs Used in PCBA as an interactive session delivery, exploration and note taking method, homework submissions, and project deliverables IPython notebooks facilitate interactive Python computing in a browser based environment Mixture of markdown and Python Inline plotting Magic commands for interacting with the OS

20 IPython Notebooks nbviewer Notebooks are just json text files Gallery of interesting notebooks Fernando Perez

21 Homework assignments HW0 - Intro to PCBA - guided exploration of pcba virtual appliance - overview reading from DDS and exploration of links on course website HW1 - Intro to R - use R Studio, create an RMarkdown document - data importing and exporting - view and modify dataframes (change data types, add cols) - answer questions about some R lists, vectors, arrays, matrices - generate html from Rmd file HW2 - EDA with R - EDA: summary stats, group by, plots - data reshaping HW3 - Predictive modeling with R - regression models to predict MLB winning percentage - try out a few predictive modeling techniques for the Kaggle Titanic Challenge - feature engineering HW4 - Simulating the Monty Hall 3-Door Problem with Python - skeleton code and comments provided in IPython notebook

22 Final Projects Options 1. Analyze dataset of interest 2. Research into techniques and/or tools 3. Compete in active Kaggle competition A few of the resulting projects - Neural nets for the Kaggle bike share competition - financial portfolio analysis with Python and tkinter (for GUI) - exploration of R packages for financial analysis - blackjack simulator in Python for exploring different playing strategies - a tutorial for creating a basic R Shiny app - maps of website use based on Apache logs using Python, pandas, matplotlib - using EDA, knn, decision trees to explore factors affecting vehicle fuel economy

23 Student mix R/Python Analytics Summer I 2014 MBA 6 BS-MIS 3 MS-STA 1 MSITM 16 Post-Bac 1 Spreadsheet based Business Analytics BS-FIN 1 BS-MIS 2 BS-POM 1 MACC 7 MBA 12 MSITM 13 Fall 2014

24 Our analytical profiles I'll give everyone an index card and on it you'll profile yourself (on a relative scale) with respect to the following dimensions Computer programming Math Statistics Data visualization Machine learning / data mining Modeling Domain expertise Communication and presentation skills Click the picture

25 Why learn to use Linux for analytics? Linux widely used in the data science and analytics world Linux shell FAR superior to Windows command line application Powerful shell scripting language Tab completion Command line is often way more efficient than GUI Linux is free (as in freedom and beer) and open source Sets you apart from other business analysts who only know Windows and Microsoft applications

26 The Geek Factor Using and creating FOSS earns you geek cred It's fun to use tools like R and Python and Linux Do an import this and then an import antigravity an IPython notebook or any Python shell Do you think MS inspires this kind of thing? :) FOSS facilitates users becoming more tech savvy Real geeks use Linux; but seriously, command line use github installing software getting your hands dirty leveraging the Unix philosophy of small focused tools that you can put together to do amazing things $wc -l *.pdb sort -g head -1 here's the overview presentation I use as part of the hands-on session to introduce B-school students to Linux

27 About Practical Computing for Business Analytics introduced B-school students to non-microsoft world that exists Linux shell scripting, Linux OS, and world of FOSS for Linux R and Python created a Lubuntu based computing appliance distributed as.ova exported from VirtualBox free! I could get it totally configured by installing and setting up the software as I wanted students didn't waste time trying to get myriad of tools working on their systems minimized hassle on our OU IT staff entire course was created and delivered with FOSS "I wouldn't use Windows at all if not for Excel An open version of the course website is available from my hselab site in the courses section In Summer I 2015, course will be offered again (and every Summer I) MIS 447 Practical computing for data analytics