STATISTICAL LABORATORY, USING R FOR BASIC STATISTICAL ANALYSIS

Transcription

1 STATISTICAL LABORATORY, USING R FOR BASIC STATISTICAL ANALYSIS Manuela Cattelan 1 ABC OF R 1.1 ARITHMETIC AND LOGICAL OPERATORS. VARIABLES AND AS- SIGNMENT OPERATOR R works just as a pocket calculator, when performing elementary computations. The arithmetic operators of addition, subtraction, multiplication, division and power are ADDITION SUBTRACTION MULTIPLICATION DIVISION POWER + - * / ^ When necessary, (round) parentheses can be used to force a given order of the computations. > /3 [1] 4 > ( )/3 [1] 2 > 2 * (10-20)^3 [1] > (0.2 * * * 7)/2 [1] 4.55 In most cases it is convenient to store constants or intermediate results into the computer memory for later use, in the same session. The corresponding areas of the computer memory are identified by names chosen by the user. The assignment operator <- is obtained by entering the < and - keys on the computer keyboard. > a <- 100 > b <- a * (5-2)/5 > b [1] 60 The operators for binary comparisons are The logical operators! (NOT ), & (AND) and (OR) are used to form complex logical expressions. Within R, the logical constants TRUE and FALSE are used to show that a given logical expression is true or false. Based on material prepared by Prof. Mario Romanazzi 1

2 1 ABC OF R 2 LOWER LOWER OR EQUAL GREATER GREATER OR EQUAL EQUAL NOT EQUAL < <= > >= ==! = > neg <- -1 > pos <- 10 > neg <= 0 [1] TRUE > neg * pos < 0 [1] TRUE > neg < 0 & pos > 0 [1] TRUE > pos >= 0 pos < 5 [1] TRUE 1.2 MATHEMATICAL FUNCTIONS The usual mathematical functions are available. Some common functions are listed below. FUNCTION Absolute value Square root Logarithm (natural, base e) Logarithm (base 10) Exponential Trigonometric: sine, cosine, tangent Factorial Binomial coefficient R NAME abs sqrt log log10 exp sin, cos, tan factorial choose The following results reflect the very definitions of the functions. > sqrt(100) [1] 10 > log10(10000) [1] 4 > 10^(log10(10000)) [1] > sin(pi/2) [1] 1 > cos(pi/2) [1] e-17 > factorial(5)

3 1 ABC OF R 3 [1] 120 > factorial(10) [1] > choose(10, 2) [1] USER DEFINED FUNCTIONS The user can define specific functions. For example, the following code defines two functions to compute the length of the circumference and the area of a circle from the length of the radius. > circ_l <- function(x) 2 * pi * x > circ_a <- function(x) pi * x^2 > circ_l(1) [1] > circ_a(0.5) [1] DATA STRUCTURES: VECTORS The most important data structure is the vector, an ordered collection of n 1 items of the same type (numerical, alphanumeric, logical). Note that a scalar is a numerical vector with just one element. There are several ways to define vectors. The most general one is through the c function (c means concatenate). The length function gives the size (number of components) of a vector. > itp_age <- c(71, 74, 63, 71, 66, 63, 82, 58, 74, 79, 81) > itp_name <- c("e. De Nicola", "L. Einaudi", "G. Gronchi", "A. Segni", + "G. Saragat", "G. Leone", "S. Pertini", "F. Cossiga", "O. L. Scalfaro", + "C. A. Ciampi", "G. Napolitano") > usp_age <- c(61, 63, 44, 55, 56, 61, 53, 70, 65, 47, 55, 47) > usp_name <- c("h. S. Truman", "D. D. Eisenhower", "J. F. Kennedy", + "L. B. Johnson", "R. Nixon", "G. Ford", "J. Carter", "R. Reagan", + "G. Bush", "B. Clinton", "G. W. Bush", "B. Obama") > length(itp_age) [1] 11 > length(itp_age) == length(itp_name) [1] TRUE Let x denote the vector (x 1, x 2,..., x n ). Special vectors can be defined by using the (colon) : operator (x i+1 x i = 1, i = 1,..., n 1) or the seq (x i+1 x i = c, i = 1,..., n 1, where c is a constant) and the rep (x i = c, i = 1,..., n, where c is a constant) functions (seq and rep are abbreviations of sequence and repeat, respectively). > -2:10 [1] > -2.5:10

4 1 ABC OF R 4 [1] > seq(1, 20, 2) [1] > seq(-5, 5, 0.5) [1] [16] > rep(0, 5) [1] > c(-0.5, 1:4, rep(5, 3)) [1] VECTOR OPERATIONS Vector components or subvectors are obtained by giving either their positions within the vector or a property they satisfy. In both cases, the subsetting operator [] is used. > itp_name[1] [1] "E. De Nicola" > itp_name[1:3] [1] "E. De Nicola" "L. Einaudi" "G. Gronchi" > itp_name[c(1, length(itp_name))] [1] "E. De Nicola" "G. Napolitano" > itp_name[itp_age < 60] [1] "F. Cossiga" > usp_name[usp_age > 40 & usp_age < 50] [1] "J. F. Kennedy" "B. Clinton" "B. Obama" In general, transformation of vectors by mathematical functions is performed component wise. Moreover, arithmetic operations involving several vectors require the vectors to have the same length. > v1 <- -2:5 > v2 <- 6:13 > abs(v1) [1] > v1 + v2 [1] > 2 * v1 - v2 [1]

5 2 BASIC STATISTICS 5 > 2 * v1-1 [1] To arrange the vector components from minimum to maximum, the sort function can be used. The functions min, max, which.min, which.max produce the minimum and maximum entries and the corresponding positions within the vector. > sort(itp_age) [1] > c(min(itp_age), max(itp_age)) [1] > c(which.min(itp_age), which.max(itp_age)) [1] 8 7 > sort(c("carla", "Francesco", "Paola", "Matteo", "Maria")) [1] "Carla" "Francesco" "Maria" "Matteo" "Paola" 2 BASIC STATISTICS We use the Presidents age data to illustrate how to produce a statistical report with R. We list in the following table some basic functions (both analytical and graphical). Note that they all have as basic argument the data vector. STATISTICAL FUNCTION Sample size Frequency table Stem-and-leaf Order statistic Basic location statistics Quantile Median Mean Variance (unbiased version) Standard deviation (unbiased version) Box-plot Histogram R NAME length table stem sort summary quantile median mean var sd boxplot hist The stem-and-leaf display shows the general features of the distribution: range, location, dispersion, shape, possible outliers. It is mainly useful with small sample sizes, as in the present case. > stem(itp_age, scale = 0.5) The decimal point is 1 digit(s) to the right of the > stem(usp_age)

6 2 BASIC STATISTICS 6 The decimal point is 1 digit(s) to the right of the The summary function, when applied to a numerical vector, gives basic statistics to evaluate location: minimum and maximum values, quartiles and mean. In computing empirical (sample) quantiles, R employs a more refined interpolation algorithm that that described in the textbook (to be used with hand computations). With small samples, discrepancies are observed. The range and the interquartile range are easily evaluated from these results. Another dispersion statistic is the standard deviation, to be used together with the sample mean. > summary(itp_age) Min. 1st Qu. Median Mean 3rd Qu. Max > summary(usp_age) Min. 1st Qu. Median Mean 3rd Qu. Max > sd(itp_age) [1] > sd(usp_age) [1] A very useful graphical comparison of the two samples is given by the paired box-plot. Whereas the stem-and-leaf displays the full order statistic, the box-plot displays only the quartiles and the extreme statistics. > boxplot(itp_age, usp_age, horizontal = TRUE, xlab = "Presidents' age (years)", + names = c("italy", "US"), col = "lavender", main = "Italy vs US, ")

7 3 EXPLORATION OF DATA WITH FREQUENCY TABLES AND DISTRIBUTIONAL PLOTS 7 Italy vs US, Italy US Presidents' age (years) Here, the most important feature is a location shift on the right of Italy with respect to US. The statistical tendency is that the italian Presidents are older than US Presidents: the difference between the medians is about 15.5 years. The dispersion does not seem very different (e.g, compare the standard deviations and the IQRs). 3 EXPLORATION OF DATA WITH FREQUENCY TABLES AND DISTRIBUTIONAL PLOTS The file regioni.txt contains the required information and can be used to simulate the collection of the data. First of all we input the file into the R system. > regio <- read.table(" + header = TRUE, na.strings = "NA") > str(regio) 'data.frame': 20 obs. of 7 variables: $ name : Factor w/ 20 levels "Abruzzo","Basilicata",..: $ area : Factor w/ 3 levels "C","N","S": $ capo : Factor w/ 20 levels "Ancona","Aosta",..: $ totsurf: int $ coast : int NA NA 346 NA NA NA... $ res09 : int $ car : num

8 3 EXPLORATION OF DATA WITH FREQUENCY TABLES AND DISTRIBUTIONAL PLOTS 8 > regio name area capo totsurf coast res09 car 1 Piemonte N Torino NA Valle d'aosta N Aosta 3263 NA Liguria N Genova Lombardia N Milano NA Trentino-Alto Adige N Trento NA Veneto N Venezia Friuli-Venezia Giulia N Trieste Emilia-Romagna N Bologna Toscana C Firenze Umbria C Perugia 8456 NA Marche C Ancona Lazio C Roma Abruzzo S L'Aquila Molise S Campobasso Campania S Napoli Puglia S Bari Basilicata S Potenza Calabria S Reggio Calabria Sicilia S Palermo Sardegna S Cagliari The structure is a data table with N = 20 rows (the regions, statistical units) and 7 columns (the variables) 1. name, name of region (identifier) 2. area, geographical area (stratification v.) 3. maint, region capital (categorical v.) 4. totsurf, region total surface area (numerical v., km 2 ) 5. coast, region total coast length (numerical v., km) 6. res09, total number of residents, as of 1/1/2009 (numerical v., count) 7. car, number of cars for 1000 residents, (numerical v.) We start data exploration with the study of the stratification variable, geographical area. The frequency distribution is obtained by the R function table and the corresponding graphical display with barplot. > N <- dim(regio)[1] > abs_f <- table(regio$area) > abs_f C N S > rel_f <- 100 * (table(regio$area)/n) > rel_f C N S

9 3 EXPLORATION OF DATA WITH FREQUENCY TABLES AND DISTRIBUTIONAL PLOTS 9 > barplot(rel_f, xlab = "Geographical Area", ylab = "Frequency (%)", + main = "Distribution of Regions According to Geographical Area", + col = "lavender") Distribution of Regions According to Geographical Area Frequency (%) C N S Geographical Area The density of the residents is the ratio of the total number of residents in a geographical area and the corresponding surface measure. In the present situation we can obtain the population density of each region dividing res09 by totsurf. > dens <- regio$res09/regio$totsurf > dens [1] [8] [15] > stem(dens) The decimal point is 2 digit(s) to the right of the

10 4 THE UNIFORM DISTRIBUTION 10 Note that the decimal point does not appear in the stem-and-leaf, but the legend supplies the necessary information. In the present case, only the hundred and ten digits are retained and the data are rounded accordingly. The stems are the classes (0, 100), [100, 200), etc. The distribution is skewed on the left and unimodal with the highest frequency in the second class. To obtain the joint distribution of geographical area and density, we use the table function again. Note that variable dens is previously divided into classes with the function cut to avoid uninformative profileration of entries in the frequency table. > table(regio$area, cut(dens, breaks = c(0, 150, 300, 450), include.lowest = TRUE)) [0,150] (150,300] (300,450] C N S The result suggests that northern and central regions concentrate in the middle density class, southern regions in the lowest class. 4 THE UNIFORM DISTRIBUTION Is there any pattern in the statististical distribution of decimal digits of real numbers? As an example we use the first 49 decimal digits of and π e The appropriate tools are the frequency distribution or the stem and leaf plot of the data. > dig_pi <- " " > dig_pi <- unlist(strsplit(dig_pi, split = ""))[-c(1, 2)] > dig_pi [1] "1" "4" "1" "5" "9" "2" "6" "5" "3" "5" "8" "9" "7" "9" "3" "2" "3" "8" "4" [20] "6" "2" "6" "4" "3" "3" "8" "3" "2" "7" "9" "5" "0" "2" "8" "8" "4" "1" "9" [39] "7" "1" "6" "9" "3" "9" "9" "3" "7" "5" "1" > table(as.numeric(dig_pi)) > stem(as.numeric(dig_pi)) The decimal point is at the

11 5 ITALIAN VS NON ITALIAN AGE DISTRIBUTION 11 > dig_e <- " " > dig_e <- unlist(strsplit(dig_e, split = ""))[-c(1, 2)] > dig_e [1] "7" "1" "8" "2" "8" "1" "8" "2" "8" "4" "5" "9" "0" "4" "5" "2" "3" "5" "3" [20] "6" "0" "2" "8" "7" "4" "7" "1" "3" "5" "2" "6" "6" "2" "4" "9" "7" "7" "5" [39] "7" "2" "4" "7" "0" "9" "3" "7" "0" "0" "0" > table(as.numeric(dig_e)) > stem(as.numeric(dig_e)) The decimal point is at the In both cases the distribution does not exhibit a clear mode, as in the unimodal situation, nor a clear increasing or decreasing pattern. A theoretical model could be a uniform distribution on the integers 0, 1, 2,..., 9 giving equal weight 1/10 to all digits. The discrepancies of the observed distribution from this model should depend on the low sample size. 5 ITALIAN VS NON ITALIAN AGE DISTRIBUTION The age classes have different widths, hence density must be used, not frequency. We show the results in the table below. Italian Residents Non Italian Residents Veneto Italy Veneto Italy Age Class Width Freq. % Dens. % Freq. % Dens. % Freq. % Dens. % Freq. % Dens. % (0, 15) [15, 30) [30, 45) [45, 65) [65, 80) [80, 110) As expected, there are no very important differences between Veneto and Italy, both for italian and non italian residents. The age distribution of italian residents is unimodal, with the density peak in the class [30, 45). In contrast, the highest frequency is in the class [45, 65).

12 5 ITALIAN VS NON ITALIAN AGE DISTRIBUTION 12 The age distribution of non italian residents is still unimodal, with the density peak in the class [30, 45), but here the concentration of the data in the modal class is much higher (the density is 2.58 against 1.60). The statistical tendency is clear: non italian residents are younger. With reference to Italy, 19.7% of italian residents have 65 years or more, against 2.1% of non italian residents. What is the explanation of this finding? Do you expect to be a permanent character or to change in the future? The four histograms are shown in the figure. > layout(matrix(1:4, 2, 2)) > plot(0, 0, type = "p", xlab = "Age (Years)", ylab = "Density (%)", + main = "Veneto (Italian Res.)", xlim = c(0, 115), ylim = c(0, + 2.6)) > rect(c(0, 15, 30, 45, 65, 80), c(0, 0, 0, 0, 0, 0), c(15, 30, + 45, 65, 80, 110), c(0.93, 1.05, 1.7, 1.29, 0.94, 0.17), col = "lavender") > plot(0, 0, type = "p", xlab = "Age (Years)", ylab = "Density (%)", + main = "Italy (Italian Res.)", xlim = c(0, 115), ylim = c(0, + 2.6)) > rect(c(0, 15, 30, 45, 65, 80), c(0, 0, 0, 0, 0, 0), c(15, 30, + 45, 65, 80, 110), c(0.94, 1.12, 1.6, 1.27, 0.97, 0.17), col = "lavender") > plot(0, 0, type = "p", xlab = "Age (Years)", ylab = "Density (%)", + main = "Veneto (Non Italian Res.)", xlim = c(0, 115), ylim = c(0, + 2.6)) > rect(c(0, 15, 30, 45, 65, 80), c(0, 0, 0, 0, 0, 0), c(15, 30, + 45, 65, 80, 110), c(1.4, 1.77, 2.56, 0.64, 0.08, 0), col = "lavender") > plot(0, 0, type = "p", xlab = "Age (Years)", ylab = "Density (%)", + main = "Italy (Non Italian Res.)", xlim = c(0, 115), ylim = c(0, + 2.6)) > rect(c(0, 15, 30, 45, 65, 80), c(0, 0, 0, 0, 0, 0), c(15, 30, + 45, 65, 80, 110), c(1.27, 1.68, 2.58, 0.75, 0.12, 0), col = "lavender")

13 5 ITALIAN VS NON ITALIAN AGE DISTRIBUTION 13 Veneto (Italian Res.) Veneto (Non Italian Res.) Density (%) Density (%) Age (Years) Age (Years) Italy (Italian Res.) Italy (Non Italian Res.) Density (%) Density (%) Age (Years) Age (Years)