Data Visualization Richard T. Watson
Data Visualization Richard T. Watson, 2012 i
Chapter 1 Data Visualization The commonality between science and art is in trying to see profoundly to develop strategies of seeing and showing. Edward Tufte
Learning objectives Students completing this chapter will: Understand the principles of the grammar of graphics; Be able to use ggplot2 to create common business graphics; Be able to use a PHP template to depict locations on a Google map; Know how to select data from a database for graphic creation. Visual processing Humans are particularly skilled at processing visual information because it is an innate capability compared to reading which is a learned skill. When we evolved on the Savannah of Africa, we had to be adept at processing visual information (e.g., recognizing a danger) and deciding on the right course of action (fight or flight). Our ancestors are those who were efficient visual processors and quickly detected threats and used this information to make effective decisions. They selected actions that led to survival. Those who were inefficient visual information processors did not survive to reproduce. Even those with good visual skills failed to propagate if they made poor decisions relative to detected dangers. Consequently, we should seek opportunities to present information visually and play to our strengths. As people vary in their preference for visual and textual information, it often makes sense to support both types of reporting. In order to learn how to visualize data, you need to become familiar with the grammar of graphics, R (a statistical package), and ggplot2 (an R extension for graphics) written by Hadley Wickham. In line with the learning of data modeling and SQL, we will take an intertwined spiral approach. First we will tackle the grammar of graphics (the abstract) and then move to ggplot2 (the concrete). You will also learn how to take the output of an SQL query and feed it directly into ggplot2. The end result will be a comprehensive set of practical skills for data visualization. Figure 1.1 Charles Joseph Minard s graphic of Napoleon s Russian expedition in 1812 (en.wikipedia.org/wiki/file:minard.png) 3
The grammar of graphics A grammar is a system of rules for generating valid statements in a language. A grammar makes it possible for people to communicate accurately and concisely. English has a rather complex grammar, as do most languages. In contrast, computer-related languages, such as SQL, have a relatively simple grammar because they are carefully designed for expressive power, consistency, and ease of use. Each of the dialects of data modeling also has a grammar, and each of these grammars is quite similar in that they all have the same foundational elements: entities, attributes, identifiers, and relationships. A specific grammar for graphics has been designed for creating graphics to enhance their expressiveness and comprehensiveness. From a mathematical perspective, a graph is a set of points. A graphic is a physical representation of a graph. Thus, a graph can have many physical representations, and one of the skills you need to gain is to be able to judge what is a meaningful graphic for your clients. A grammar for graphics provides you with many ways of creating a graphic, just as the grammar of English gives you many ways of writing a sentence. Of course, we differ in our ability to convey information in English. Similarly, we also differ in our skills in representing a graph in a visual format. The aesthetic attributes of a graph determine its ability to convey information. For a graphic, aesthetics are specified by elements such as size and color. One of the most applauded graphics is Minard s drawing in 1861 of Napoleon s advance on and retreat from Russia during 1812. The dominating aspect of the graphic is the dwindling size of the French army as it battled winter, disease, and the Russian army. The graph shows the size of the army, its location, and the direction of its movement. The temperature during the retreat is drawn at the bottom of graphic. Wilkinson s grammar of graphics is based on six elements: 1. Data: a set of data operations that creates variables from datasets, 2. Trans: variable transformations, 3. Scale: scale transformations, 4. Coord: a coordinate system, 5. Element: a graph and its aesthetic attributes, 6. Guide: one or more guides. Before learning how to use this grammar, we need to develop some skills in R, our platform for deploying the grammar. R R is a language and environment for statistical computing and graphics. In our case, we are mainly interested in its graphics facilities. However, you should be aware that R supports a wide variety of statistical techniques. R is open source and cross platform. It is also highly extensible and statisticians are continually adding new features and enhancing existing tools. R has a graphical user interface (GUI) that supports a limited number of features of R. However, because it is extensible by the statistical community, it is difficult to support a GUI across all extensions. As a result, many users rely on the command line interface Figure 1.2 R Studio 4
for executing commands to access the full power of R. Rstudio was developed to facilitate the use of R s command line interface. It provides four windows, which are very useful for seeing various aspects of the problem on which you are currently working (e.g., R command, R text and graphic output, and the input dataset), as shown in the following screen capture. We will use RStudio as the interface to R. Data Most structured data, which is what we require for graphing, are typically stored in spreadsheets or databases. It is highly likely that you are familiar with a spreadsheet, so we will first take that approach to preparing a dataset. Next, we will learn how to use MySQL with R. The spreadsheet approach Select your favorite spreadsheet application and enter the data for the following table. After saving the data in the spreadsheet s standard format, select Save As or Export to convert the spreadsheet into comma-delimited format (CSV), an input format that R can read. Table 1.1 CO 2 parts per million (ppm) of the earth s atmosphere Year Here are the commands to read the table into R and list its contents. Note that comment lines are preceded by #. Alternatively, use RStudio to import the file (use Import Dataset). # read a file with a header row and value separated by commas carbon <- read.table(file.choose(), header=true, sep=",") # list the contents carbon # for long files, just show the first few rows head(carbon) CO2 2000 369.40 2001 371.07 2002 373.17 2003 375.78 2004 377.52 2005 379.76 2006 381.85 2007 383.71 2008 385.57 2009 384.78 Year CO2 1 2000 369.40 2 2001 371.07 3 2002 373.17 4 2003 375.78 5 2004 377.52 6 2005 379.76 The database approach RJDBC is an R package for providing access to a relational database. You need to download and install it before use. You must know the path to the JDBC driver for the database you are using. The example shows access to a MySQL database. library(rjdbc) # Load the driver drv <- JDBC("com.mysql.jdbc.Driver", "SSD250/Library/Java/Extensions/mysql-connector-java-3.1.18-bin.jar") # connect to the database conn <- dbconnect(drv, "jdbc:mysql://localhost/classicmodels", "user", "pwd") # Query the database and create file p p <- dbgetquery(conn,"select DISTINCT productline from Products;") Transformation A transformation converts data into a format suitable for the intended visualization. In this case, we want to depict the relative change in carbon levels since pre-industrial periods, when the value was 280 ppm. Here are the R commands. # computer a new column in carbon containing the relative change in CO2 carbon$relco2 = (carbon$co2-280)/280 Notice how we define a column s name by first identifying the table (carbon), adding $ as a separator, and then specifying the column name (relco2). There are many ways that you might want to transform data. The preceding example just illustrates the general nature of a transformation. 5
You can also think of SQL as a transformation process as it selects columns from a database. Coord A coordinate system describes where things are located. A geopositioning system (GPS ) reading of latitude and longitude describes where you are on the surface of the globe. It is typically layered onto a map so you can see where you are relative to your neighborhood. Most graphs are plotted on a two-dimensional (2d) grid with x (horizontal) and y (vertical) coordinates. ggplot2 currently supports six 2D coordinate systems, as shown in the following table. The default coordinate system is Element An element is a graph and its aesthetic attributes. Let s start with a simple scatterplot of year against CO2 emissions. We do this in two steps applying the ggplot approach of building a graphic by adding layers. The foundation is the ggplot function, which identifies the source of the data and what is to be plotted using the aes function. Geoms, or geometric objects, describe the type of plot. For example, Table 1.2 Coordinate systems Name cartesian equal flip trans map Description Cartesian coordinates Equal scale Cartesian coordinates Flipped Cartesian coordinates Transformed Cartesian coordinates Map projections Figure 1.3 Atmospheric CO 2 (ppm) geom_point() specifies a scatterplot of points. Notice how you specify the color to use for showing the points. Cartesian. polar Polar coordinates As you will see later, a bar chart (horizontal bars) is simply a histogram (vertical bars), rotated clockwise 90º. This means you can create a histogram and then add one command coord_flip() to flip the coordinates and make it a bar chart. # load the library, which needs to be done each time you start R library(ggplot2) # Select Year(x) and CO2(y) to create a x-y point plot ggplot(carbon,aes(year,co2)) + geom_point() # Specify red points, as you find that aesthetically pleasing ggplot(carbon,aes(year,co2)) + geom_point(color='red') # Add some axes labels # Notice how + is used for commands that extend over one line ggplot(carbon,aes(year,co2)) + geom_point(color='red') + xlab('year') + ylab('co2 ppm of the atmosphere') # Save the graphic as a png ggsave(filename="carbon.png", width=6, height = 3) 6
Scale Scales control the visualization of data. It is usually a good idea to have a zero point for the y axis so you don t distort the slope of the line. ggplot(carbon,aes(year,co2)) + geom_point(color='red') + xlab('year') + ylab('co2 ppm of the atmosphere') + ylim(0,400) Figure 1.4 Relative change of atmospheric CO2 The change is less discernible when the zero point is in place, but let s repeat the graph with the relative change. # computer a new column in carbon containing the relative change in CO2 carbon $relco2 = (carbon $CO2-280)/280 ggplot(carbon,aes(year,relco2)) + geom_point(color='red') + xlab('year') + ylab('relative change of atmospheric CO2') + ylim(0,.5) As the prior graphics show, the present level of CO 2 in the atmosphere is about 35 percent higher than the pre-industrial period and is continuing to grow. Legend Axis title Axi tick mark and label Figure 1.5 Elements of a graphic Figure 1.6 Product lines as a histogram 7
Guides Axes and legends are both forms of guides, which help the reader to understand a graphic. In ggplot2, legends and axes are generated automatically based on the parameters specified in the ggplot command. You have the capability to override the defaults for axes, but the legend is quite fixed in its format. In the following graphic, for each axis there is a label and there are tick marks, with values so that you convert a point on the graphic to its x and y values. The legend enables you to determine which color, in this case, matches each year. A legend could also use shape (e.g., a diamond) and shape size to aid matching. Some recipes Learning the full power of ggplot2 is quite an undertaking, so here are a few recipes for visualizing data extracted from a database. Histogram Histograms are useful for showing a single column of data, which in statistical terms is termed univariate data. A histogram has a vertical orientation. The number of occurrences of a particular value in a column are automatically counted by the ggplot function. Figure 1.7 Product line as a bar chart # Assume connection as been made as discussed previously d <- dbgetquery(conn,"select productline from Products;") # Plot the number of product lines by specifying the appropriate column name ggplot(d,aes(x=productline)) + # Internal fill color is red geom_histogram(fill='red') Notice how the titles for various categories overlap. If this is the case, it is a good idea to convert to a bar chart, which is just a case of flipping the coordinates. Figure 1.8 Value by month Bar chart A bar chart can be thought of as a histogram turned on its side with some reorientation of the labels. Here are the same data as a bar chart, with a different fill color and font. # Assume connection as been made as discussed previously 8
d <- dbgetquery(conn,"select productline from Products;") # Plot the number of product lines by specifying the appropriate column name ggplot(d,aes(x=productline)) + geom_histogram(fill='gold') + # Reorient to horizontal coord_flip() # Save the graphic as a png ggsave(filename="carbon.png", width=6, height = 3) Scatterplot A scatterplot shows points on an x-y grid, which means you need to have an x and y with numeric values. d <- dbgetquery(conn,"select YEAR(orderDate) AS orderyear, MONTH(order- Date) AS Month, sum((quantityordered*priceeach)) AS Value FROM Orders, OrderDetails WHERE Orders.orderNumber = OrderDetails.orderNumber GROUP BY orderyear, Month;") Figure 1.10 A fluctuation plot Figure 1.9 Value by month and year # Assume connection as been made as discussed previously d <- dbgetquery(conn,"select MONTH(orderDate) AS ordermonth, sum((quantityordered*priceeach)) AS ordervalue FROM Orders, OrderDetails WHERE Orders.orderNumber = OrderDetails.orderNumber GROUP BY ordermonth;") # Plot data orders by month # Show the points and the line ggplot(d,aes(x=ordermonth,y=ordervalue)) + geom_point(color='red') + geom_line(color='blue') It is sometimes helpful to show multiple scatterplots on the one grid. In ggplot2, you can create groups of points for plotting. Let s examine grouping by year. # Plot data orders by month and grouped by year # ggplot expects grouping variables to be character, so convert d$year <- as.character(d$orderyear) ggplot(d,aes(x=month,y=value,group=year)) + geom_line(aes(color=year)) + # Format as dollars scale_y_continuous(formatter = "dollar") + # Specify font for labels opts(theme_text(family='georgia')) Fluctuation plots A fluctuation plot visualizes tabular data. It is useful when you have two categorical variables cross tabulated. Consider the case for the ClassicModels database where we want to get an idea of the different # Assume connection as been made as discussed previously 9
show points on a Google map. Some of these are one time steps, and others you repeat each time you want to create a new map. Find a database containing geographic data The Offices table in the Classic Models database includes the location of each office in officelocation, which has a datatype of POINT. Get a developer key You need to sign up for a Google maps API key before you can embed a map on your Web page. Usually your instructor will have done this for the class if you are creating a page on your university's Web site. If not, follow the procedures on the sign up page. Figure 1.11 A heat map model scales in each product line. We can get a quick picture with the following code. # Assume connection as been made as discussed previously d <- dbgetquery(conn,"select * from Products;") # Plot product lines ggfluctuation(table(d$productline,d$productscale)) + xlab("scale") + ylab("line") + # Specify font for labels opts(theme_text(family='arial')) # Save the graphic as a png ggsave(filename="carbon.png", width=6, height = 3) We can get a color, rather than size, based plot with # Assume connection as been made as discussed previously # Get the product data d <- dbgetquery(conn,"select * from Products;") # Plot product lines ggfluctuation(table(d$productline,d$productscale),type="color") + xlab("scale") + ylab("line") Geographic data R supports the mapping of geographic data, but Google offers a much richer environment because of the data density of its maps (e.g., roads and satellite views). However, there are a few steps you need to take to Copy the skeleton code into a text editor 11 The following is a mix of HTML, JavaScript, and PHP code that you can execute on any server that supports PHP. You might consider running a version of LAMP, MAMP, or WAMP on your personal computer as this will provide you with both a MySQL and PHP server. Thus, you will avoid the need to get an account on a server to execute the PHP code. If you are new to coding, then don t worry about the fine detail. Just change the bits in blue to match your situation. <!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN" "http://www.w3.org/tr/xhtml1/dtd/xhtml1-transitional.dtd"> <head> <title>classic Models Offices</title> <?php // connect to the database $db = mysql_connect("server","userid","password"); mysql_select_db("database", $db);?> <!-- Google maps key --> <script src="http://maps.google.com/maps?file=api&v=1&key=enteryourkeyhere"></ script> <script type="text/javascript"> // mark a location function createmarker(point,html) { var marker = new GMarker(point); return marker; } // set up the map function initialize() { if (GBrowserIsCompatible()) { 10
var map = new GMap2(document.getElementById("map_canvas"),{ size: new GSize(580,400) } ); map.removemaptype(g_hybrid_map); map.setcenter(new GLatLng(-3.703250,40.416741,0), 1); var mapcontrol = new GMapTypeControl(); map.addcontrol(mapcontrol); map.addcontrol(new GLargeMapControl()); <?php $query="select x(officelocation) AS x, y(officelocation) AS y FROM Offices"; $result1 = mysql_query($query, $db)or die(mysql_error()); // loop to display the markers while(list($x,$y) = mysql_fetch_row($result1)){ echo "\n var point = new GLatLng(".$x.",".$y.");\n"; echo "var marker = createmarker(point,'');\n"; echo "map.addoverlay(marker);\n"; echo "\n"; }?> } } </script> </head> <body onload="initialize()" onunload="gunload()"> <div id="map_canvas" style="width: 580px; height: 300px"></div> </body> </html> Execute the code Read the instructions for the (LMW)AMP package you installed as to where you store the PHP code and how you execute it using a browser. For example, with MAMP you do the following: 1. Store the code as index.php in Applications/MAMP/htdocs 2. Load MAMP 3. Enter http://www.localhost:8888/index.php as the URL in your browser Summary Because of evolutionary pressure, humans are strong visual processors. Consequently, graphics can be very effective for presenting information. The grammar of graphics provides a logical foundation for the construction of a graphic by adding successive layers of information. ggplot2 is a package implementing the grammar of graphics in R, the open source statistics platform. Data can be extracted from a MySQL database for processing and graphical presentation in R. Spatial data can be selected from a database and displayed on a Google map. Figure 1.12 A Google map Key terms and concepts Grammar of graphics Graph Graphic References Kabacoff, R. I. (2009). R in action: data analysis and graphics with R. Greenwich, CT: Manning Publications. Tufte, E. (1983). The visual display of quantitative information. Cheshire, CT: Graphics Press. 11
Wickham, H. (2009). ggplot2: elegant graphics for data analysis. New York: Springer-Verlag New York Inc. Wilkinson, L., & Wills, G. (2005). The grammar of graphics (2nd ed.). New York: Springer. Exercises 1. Visualize in blue the number of items for each product scale. 2. Prepare a line plot with appropriate labels for total payments for each month in 2004. 3. Create a histogram with appropriate labels for the value of orders received from the Nordic countries (Denmark, Finland, Norway, Sweden). 4. Create a heatmap for product lines and Norwegian cities. 5. Show on a Google map the customers in France. 6. Show on a Google map the customers in the US who have never placed an order. 12
Classic Models database A sample database <http://www.eclipse.org/birt/phoenix/db/>that is useful for teaching SQL. The MySQL code to create the database is available from <http://richardtwatson.com/dm5e/reader/sql/classicmodels.sql>. Related Glossary Terms Drag related terms here Index Find Term Chapter 1 - Data Visualization
CSV A comma-separated values (CSV) file is a simple text file in which each row of a spreadsheet or record of a database table is one line. Fields are separated by commas. Related Glossary Terms Drag related terms here Index Find Term Chapter 1 - Data Visualization
Ggplot2 An R extension for graphics <http://had.co.nz/ggplot2/>. See Wickham, H. (2009). ggplot2: elegant graphics for data analysis: Springer-Verlag New York Inc. Related Glossary Terms Drag related terms here Index Find Term Chapter 1 - Data Visualization
Google maps API An application programming interface (API) for embedding Google Maps into applications. See <http://code.google.com/apis/maps/index.html> Related Glossary Terms Drag related terms here Index Find Term Chapter 1 - Data Visualization
Grammar for graphics Wilkinson, L., & Wills, G. (2005). The grammar of graphics (2nd ed.). New York: Springer. Related Glossary Terms Drag related terms here Index Find Term Chapter 1 - Data Visualization
R An open source statistical package. See <http://www.r-project.org/>. For a list of colors you can use in R, see <http://www.stat.columbia.edu/~tzheng/files/rcolor.pdf> Related Glossary Terms Drag related terms here Index Find Term Chapter 1 - Data Visualization
Regular expression Lorem ipsum dolor sit amet, consectetur adipisicing elit, sed do eiusmod tempor incididunt ut labore et dolore magna aliqua. Ut enim ad minim veniam, quis nostrud exercitation ullamco laboris nisi ut aliquip ex ea commodo consequat. Related Glossary Terms Drag related terms here Index Find Term
Rstudio A graphical user interface (GUI) for R. See rstudio.org. Related Glossary Terms Drag related terms here Index Find Term Chapter 1 - Data Visualization