Agenda. R Does Pivot Tables Sparklines (Edward Tufte) Misc. Graphics Questions Jim Holtman

Transcription

1 Agenda 10 Minute Workshop R Does Pivot Tables Sparklines (Edward Tufte) Misc. Graphics Questions Jim Holtman

2 What is Open source language and environment for statistical processing Based on the S language developed at Bell Labs by John Chambers in the early 1980s John won the 1998 ACM Award for the development of the S Language I have been using it for the last 25 years Over 85 books available on R/S-Plus Many recent statistics graduates have a background in R. Many new statistical procedures use R as their infrastructure. Gene research is heavy into the use of R. Google nyt r for the recent New York Times article on R. Data Analysts Captivated by R s Power Jim Holtman

3 What is Object oriented Everything in R is an object R is written in R Base routine are written in FORTRAN/C Interpreted, but fast Functional language Interactive development of scripts cut/paste from a text editor to develop an R script Batch processing similar to UNIX shell files Complete programming environment with a learning curve similar to C/Java Jim Holtman

4 What is External interfaces text files (.csv,.txt, ) Relational databases (ODBC) Oracle, Informix, DB2, MySQL, Access, Statistical systems SAS, SPSS, Stata, Systat, Minitab, High quality output sweave package: generate LaTeX output that combines text, computations and graphics Jim Holtman

5 What is Where to find it google r Download binaries for Windows, MacOS & Linux Compile from source; complete build system provided Basic documentation delivered with the package 100 page Introduction to R provides the basic information that you need to start developing R programs. on-line help has examples of each command that you can run to see how they work Jim Holtman

6 Frequency Generate & Summarize 1M Random Numbers Histogram of x x Jim Holtman

7 R Objects The basic object is a vector. This can hold zero, or more, values of the same type; e.g.: character vector of character strings of varying lengths numeric real (floating point) numbers integer signed integers (typically 32-bits) logical TRUE/FALSE complex complex numbers list a vector of R objects (similar to struct in C/Java) matrix 2 dimensional array of objects of the same type array - n-dimensional array of objects dataframe 2 dimensional object, where each column can be a different type (think of an EXCEL spreadsheet) POSIXct date/time class user defined create your own objects and methods to work on them Jim Holtman

8 R Language R is a complete programming environment with a number of operators, control structures and functions Operators: +,-,*,/,<,<=,>,>=,==,!=,&,,...and more... Control structures: if (condition) true else false for (variable in sequence) statement while (condition) statement repeat statement break Creating function: func <- function(parameters) statement Jim Holtman

9 Vectors Vectors are one dimensional objects with each element being the same type Operations are carried out on all elements of the vector The power of R lies in its ability to perform vectorized operations; you do not have to code an explicit for loop Jim Holtman

10 Question on R? See me at break for any followup If you have some data that you would like to see how it can be processed in R, please feel free to send it to me and I will send back a quick script to do some basic stuff. Attend my workshop at CMG Jim Holtman

11 Pivot Tables & More John Van Wagenen s CMG2008 paper Pivot Tables/Charts Magic Beans Without Living in a Fairly Tale. Pivot tables are a nice way to slice/dice/aggregate data. I had been doing similar things in R, so it motivated me to write a paper on another way to get the same information. I have used his data to illustrate how to do these techniques in R. Now walk through some examples Jim Holtman

12 Excel Spreadsheet CSV File Exported from above (10,696 data lines) Jim Holtman

13 Excel Pivot Table Generated from the Data Read John s paper for the procedure for generating the pivot table in Excel Jim Holtman

14 Jim Holtman

15 This is what the data objects in R look like Jim Holtman

16 Casting New Data From the same melt data, I can create a daily summary and add an indicator for PRIME time: Jim Holtman

17 Excel Spreadsheet (24,560 data points) Pivot Table Chart Jim Holtman

18 R Script 0.6 seconds to read in 24,560 lines of data, summarize by shift and create the pie chart. Breakdown by Shifts WEEKEND HOLIDAY PERIOD2 PERIOD3 PRIME Jim Holtman

19 batch Data Object in R Jim Holtman

20 Frequency Frequency Frequency EDA on the batch Data Histogram of batch$cpu.hrs Histogram of batch$cpu.hrs[batch$cpu.hrs < 0.03] Histogram of batch$cpu.hrs[batch$cpu.hrs < 0.005] batch$cpu.hrs batch$cpu.hrs[batch$cpu.hrs < 0.03] batch$cpu.hrs[batch$cpu.hrs < 0.005] Jim Holtman

21 5/1/2007 6/1/2007 7/1/2007 8/1/2007 9/1/ /1/2 11/1/2 12/1/2 1/1/2008 2/1/2008 3/1/2008 4/1/2008 5/1/2008 6/1/2008 cpu seconds Summarize by Prod & Dev (3 rd character) Excel Spreadsheet Pivot Table Chart From Pivot Table DEV PROD Jim Holtman

22 Summarize by Prod & Dev Using R Jim Holtman

23 Total CPU Seconds Chart from R DEV PROD Jim Holtman

24 Pivot Table Summary R & Excel (and other products) can produce summaries that are equivalent to pivot tables In R it is easy to automate the scripts and run through a set of files and quickly produce output in various formats: PDF, PNG for web pages, WMF for inclusion in WORD/PowerPoint documents, The interactive nature of R makes it easy to do EDA (exploratory data analysis) on your data Jim Holtman

25 Sparklines Invented by Edward Tufte, well known expert on data visualization for more examples Jim Holtman

26 Sparklines from vmstat data Script on production systems log the vmstat data to a file every 30 seconds. This is used to create the daily and monthly utilization charts for a system. Data used to create sparklines of 19 variables in the log file below Jim Holtman

27 Jim Holtman

28 Monthly Data Have used levelplot to show 3D data day of the month on the y- axis, time of day on the x-axis and color to represent the value of the z-axis, which would be the CPU utilization. Sparklines for the month s performance of the system were plotted next to the levelplot for comparison. Both presentation methods allow you to look for patterns. Which do you find the easiest to see patterns in? Sparklines would make an interesting presentation of yearly data. The example just duplicates the monthly data to provide an idea of what it might look like Jim Holtman

29 levelplot and sparklines of the same monthly utilization data.

30 Jim Holtman

31 Transaction Data Consolidated ~79K transactions into 10 transaction groups and 10 user pools to make the reports easier to see. Data has the user, transaction name, start and end time. Response was calculated. Look at this data with some stacked barcharts and mosaic plots. Pivot Table of User/Transaction Counts Jim Holtman

32 Total Transactions Stacked Bar Chart of Transaction Count by User Tran.01 Tran.02 Tran.03 Tran.04 Tran.05 Tran.06 Tran.07 Tran.08 Tran.09 Tran.10 User.01 User.02 User.03 User.04 User.05 User.06 User.07 User.08 User.09 User Jim Holtman

33 Stacked Bar Chart/Mosaic Chart Lets you see who the busy users are in terms of number of transactions. A mosaic chart sows the same data, but the area of the boxes is proportional to the counts. y-axis range is the same for all data elements. Sometimes easier to the ratios (mix) between the use of transactions for a user; may denote a different role for that user. User.06 (lowest count) has on Trans.06, Trans.09 and Trans.10 a higher ratio than User.08 (highest count) Jim Holtman

34 Tran User.01 User.02 User.03 User.04 User.05 User.06 User.07 User.08 User.09 User.10 Mosaic Plot of the Number of Transactions by User - Area Proportional to Count Trans.01 Trans.02 Trans.03 Trans.04 Trans.05 Trans.06 Trans.07 Trans.08 Trans.09 Trans.10 User Jim Holtman

35 References [1] J. Van Wagenen, Pivot Tables/Charts Magic Beans Without Living in a Fairy Tale, CMG 2008 [2] Ron Kaminski, Automating Process Pathology Detection Rule Engine Design Hints, CMG 2008 [3] R Development Core Team, R: A Language and Environment for Statistical Computing, {ISBN} , [4] J. Holtman, Using R for System Performance Analysis, CMG 2004 [5] J. Holtman, Visualization Techniques for Analyzing Patterns in System Performance Data, CMG 2005 [6] N. J. Gunther, Guerrilla Capacity Planning, Springer-Verlag, Heidelberg, Germany, 2007 [7] H. Wickham, Reshaping data with the reshape package, Journal of Statistical Software, 21(12), 2007 [8] Venables, W. N. and Ripley, B. D. Modern Applied Statistics with S. Fourth Edition. Springer, 2002, ISBN [9] Tufte, Edward Beautiful Evidence Graphic Press 2006 [10] Spector, Phil Data Manipulation with R (Use R) Springer, ISBN Jim Holtman

36 Questions? Jim Holtman