Statistics Data Presentation, Frequency Distribution, Histogram, Sample Variance, etc. Instructor: Daisuke Nagakura

Transcription

1 Statistics Data Presentation, Frequency Distribution, Histogram, Sample Variance, etc. Instructor: Daisuke Nagakura 1

2 Frequency Distribution and Histogram Frequency distribution table and histogram are convenient ways to organize and summarize one dimensional data. Frequency distribution table Frequency distribution table is made by classifying observations into some groups or categories called classes (or class intervals) according to their values, counting the number of observations in each class, which is called frequency, and displaying those numbers 2

3 Frequency Distribution and Histogram A class is determined by its lower and upper limits. The average of lower and upper limits is called the class value. A relative frequency is a proportion (percentage) of observations in each class, which is obtained by dividing each frequency by the total number of observations A cumulative sum of frequencies is called a cumulative frequency, and cumulative sum of relative frequencies is called a cumulative relative frequency. 3

4 Frequency Distribution and Histogram Making a frequency distribution table (1) Determine the number of classes, intervals of classes (usually the same for all classes). (2) Compute the class values. (3) Count the frequencies. (4) Do other necessary calculations such as relative frequencies. 4

5 Frequency Distribution and Histogram Sturges formula One convenient way to determine the number of classes is Sturges formula, which is given as Number of classes = log 10 (Total number of observations). In an example below, the total number of observations is 47, and so we have log Therefore, we may set the number of classes to 6 or 7 (6 is chosen below). 5

6 Frequency Distribution and Histogram Open-end class As in the example below, a class without upper or lower limit is called an open-end class. In this case, the class value is the average of all observations belonging to the class. 6

7 Frequency Distribution and Histogram Example (frequency distribution table) The data below is the number of live births in 47 prefectures of Japan in 2004, which is taken from the vital statistics of Ministry of Health, Labor and Welfare, arranged from lowest to highest (unit:a thousand) 5, 6, 6, 6, 7, 8, 8, 8, 8, 9, 9, 10, 10, 10, 11, 11, 12, 12, 12, 12, 13, 14, 15, 16, 16, 16, 18, 18, 18, 18, 18, 19, 20, 20, 22, 26, 26, 34, 44, 45, 50, 53, 62, 70, 79, 80, 99 Let s make a frequency distribution table of the data. 7

8 Low - high Frequency Distribution and Histogram Class Value frequ ency cumulative frequency Relative frequency The class value of open-end class at the bottom is the rounded-off value. Cumulative relative frequency

9 Frequency Distribution and Histogram Histogram Histogram is a bar graph drawn so that the areas of bars are proportional to the frequencies (or relative frequencies), where the widths of bars (vertical rectangles) are equal to the intervals of classes and heights of bars are equal to the frequencies or relative frequencies, the middle points of widths are located at the class values. For an open-end class, the width of bar is set to twice the distance between the upper limit of the next class and the class value of the open-end class, and then the height is adjusted so that the are of bar is proportional to frequencies (or relative frequencies). Histogram visually presents the distribution of the data. 9

10 Frequency Distribution and Histogram Example (Histogram) The histogram of the above frequency distribution table is as follows

11 Measuring the variability of data Variance Mean, median, and mode are statistics for measuring the center of a given data set. However, important characteristics of data is not only the position of the center. For example, the variability or dispersion of data is one of the most important characteristics of data. A variance is a statistic for measuring the variability of data. Here, we review the definition of variance and its properties. 11

12 Measuring the variability of data Variance as a descriptive statistic Suppose that we have a data set consisting of n observations: {x 1, x 2,, x n }. Suppose further that these are the whole sample of a population of interest. Then, the variance as a descriptive statistic is defined as 2 1 n i1 ( x i x) This is sometimes called population variance. n 2. 12

13 Measuring the variability of data Variance as an estimate Again, suppose that we have a data set consisting of n observations: { x 1, x 2,, x n }. This time, suppose that these observations are of a sample of a population of interest. In this case, we estimate the variance of population, or population variance, with: s 2 1 n 1 n i1 ( x i x) This is called sample variance or sample unbiased variance

14 14 14 Measuring the variability of data Exercise 1 Show that σ 2 and s 2 can be expressed as: and, n i i x n x n n i i x n x n s

15 Measuring the variability of data Difference between two variances The difference between σ 2 and s 2 is that σ 2 divides the sum by n, whereas s 2 divides the sum by n 1. (The details will be given later, but) we can show that, in a statistical point of view, s 2 has a desirable property as an estimator of σ 2, that is called unbiasedness. We will see this later in the semester. 15

16 Measuring the variability of data Standard deviation Population standard deviation is defined as the square root of population variance, while sample standard deviation is defined as the square root of sample variance. They are denoted by σ, and s, respectively. The meaning of standard deviation is almost the same as variance, namely, measuring the variability of data. Standard deviation is more often used than variance because its unit is the same as the data itself. 16

17 Measuring the variability of data Example of variance and standard deviation Let s compare the variance and standard deviation of the following two data set (the number of observations are both 30). (Data set 1) {26, 32, 4, 28, 12, 31, 27, 15, 26, 18, 27, 13, 29, 13, 45, 39, 18, 23, 35, 19, 33, 26, 21, 37, 21, 36, 23, 23, 24, 26} (Date set 2) {4, 12, 27, 43, 23, 14, 26, 35, 15, 17, 38, 19, 22, 25, 49, 7, 42, 31, 23, 46, 25, 28, 36, 24, 1, 15, 33, 8, 27, 35} 17

18 Measuring the variability of data The means of the two data sets are both 25. Below are the histograms of the two data sets (Histogram of Data set 1) (Histogram of Data set 2) Which data set do you think is more dispersed? (i.e., which data set has more observations that are far from the mean?)

19 Measuring the variability of data The (population) variance and standard deviation of Data set 1 are: Variance Standard deviation 8.8 The (population) variance and standard deviation of Data set 2 are: Variance Standard deviation Variance and standard deviation increases as the data becomes more dispersed, or the variability of data increases 19

20 Measuring the variability of data Properties of variance and standard deviation There are two important properties of variance and standard deviation (these properties hold for both population and sample variances and standard deviations). 1. Adding (and subtracting) a constant to (from) every observation does not change the values of variance and standard deviation. 2. Multiplying every observation by k, then the value of variance is multiplied by k 2, and the value of standard deviation is multiplied by k. 20

21 Standardization and standard score Standardization Standardization is to transform the data so that the transformed data has mean of 0 and standard deviation of 1 (the variance is also 1). Standard score Consider a data set transformed so that it has mean of 50 and standard deviation of 10. Then, standard score of an observation (in the original data set) is the value in the transformed data that calculated from that observation. 21

22 Standardization and standard score Standardization When a data set has mean of and variance of 2 x, then the transformation of x i such as is called standardization (i.e., subtracting the mean and dividing by the standard deviation). x xi x zi, i 1,..., n x It is not difficult to show that z i standard deviation of 1. has mean of 0, and 22

23 Standardization and standard score Calculation of standard score Let y i be the standard score of x i. (1) Standardize x i. Let z i be standardized value of x i. (2) Then, y i is calculated with z i by y i 50 10zi. It is not difficult to see that the mean and standard deviation of y i is 50 and 10, respectively. (standard scores are the data transformed so that they have mean of 50 and standard deviation of 10.) 23

24 Standardization and standard score The meaning of Standardization and standard score Given a data set, we can see each value s position relative to other values by comparing the value to the mean and variance. However, if we are given two data sets whose means and variances are different, it is hard to compare the relative positions of values in the two data sets. Standardization and standard score is helpful for that purpose. 24

25 Standardization and standard score Exercise 2 Suppose that we have two data sets. The mean and variance of Data set 1 is 5 and 4, whereas the mean and variance of Data set 2 is 6 and 9. Both data sets contain the value of 9. (1) Calculate the standardized value of 9 for each data set. (2) Which 9 (in the two data sets) is considered to be relatively large value compared to other values in each data set? 25

26 Analysis of bivariate data Mean and variance are statistics that are useful for summarizing features of univariate data. However, in a more advanced analysis, we need to deal with multivariate data and analyze their interrelationships. In what follows, we consider statistics that can summarize the features of bivariate data. 26

27 Analysis of bivariate data Scatter plot Suppose that we are given n pairs of observations: { (x 1, y 1 ), (x 2, y 2 ),, (x n, y n ) } A scatter plot is a graph that displays a two dimensional data set. 27

28 Analysis of bivariate data Example (Scatter plot) Table below reports the results of 18 terms in J league (a soccer league in Japan) of a year* Team Wins GF GA Team Wins GF GA Statistics Morimune, K., Terui, N., Nakagawa, M., Nishino, H., and Kurozumi, E., 2008, Yuhikaku 28

29 Analysis of bivariate data Example (Scatter plot) Scatter plot of {(x, y)} = {( GF(Goals for), Wins )} 25 Scatter Plot of GF and Win 20 Wins GF 29

30 Analysis of bivariate data Example (Scatter plot) Scatter plot of {(x, y)} = {( GA(Goals against, Wins)} 25 Scatter Plot of GA and Wins 20 Wins GA 30

31 Analysis of bivariate data Example (Scatter plot) Scatter plot of {(x, y)} = {(GF, GA)} Scatter Plot of GF and GA GA GF 31

32 Analysis of bivariate data Covariance For n pairs of observations: {(x 1, y 1 ), (x 2, y 2 ),, (x n, y n )}, the population covariance is defined as x xy y 1 n n i1 ( x i x)( y y), where and are the sample means of x i and y i. Similarly, the sample covariance is defined by replacing n with n 1. i 32

33 Analysis of bivariate data Interpretation of covariance The sign of covariance indicates the sign of a linear relationship between those two variables, i.e., positive or negative. Problem of covariance Covariance tells us about the sign of a linear relationship between two variables, but does not give any information about the magnitude of the relationship between them. 33

34 Analysis of bivariate data Correlation coefficient Correlation coefficient compensates this drawback of covariance and can measure the strength of a linear relationship between two variables. Correlation coefficient is denoted by r xy, and defined as r xy n i1 n ( x i1 i ( x i x) 2 x)( y i n i1 y) ( y i y) 2 r xy between two variables is equal to the covariance between the standardized variables of those two variables. 34

35 Analysis of bivariate data Interpretation of correlation coefficient When r xy is positive (negative), the two variables are said to have a positive (negative) correlation, which implies that one variable tends to be positively (negatively) proportional to the other variable. As the value of correlation is closer to positive (negative) 1, the positive (negative) linear relationship between the two variables becomes stronger. In the J league examples, r xy of GF and wins is r xy of GA and wins is 0.71 r xy of GF and GA is

36 Analysis of bivariate data Properties of correlation coefficient (1) Correlation coefficient is nothing but an index to measure the strength of a linear relationship. When x and y have a very strong relationship, but if it is non-linear, then the value of correlation does not reflect that relationship. (Strong relationship, but 0 correlation)

37 Analysis of bivariate data (2) Correlation coefficient takes values between 1 and 1 ( 1 r xy 1). When r xy = 1, the two variables are said to be perfectly negatively correlated, and when r xy = 1, they are said to be perfectly positively correlated. (perfect negative correlation) 10 (perfect positive correlation)

38 Analysis of bivariate data Properties of correlation coefficient (3) It is an index of not causality but correlation relationship. Causality is a relationship such that one is a cause of the other. For example, a taller person tends to weigh more, but being fatter does not imply begin taller. So, in this case, the direction of causality is body height body weight, but not the other way around. 38

39 Analysis of bivariate data Exercise 3 (1) For n pairs of observations {x 1,, x n } and {y 1,, y n }, Suppose that their correlation is 0.2. Define a new variable z i by z i = y i +2, for i =1,,n. What is the value of correlation coefficient between {x 1,, x n } and {z 1,, z n }? (2) Suppose that the correlation coefficient between {x 1,, x n } and {y 1,, y n } is 0.1. What is the value of correlation coefficient between {w 1,,w n } and {x 1,, x n }, where w i = 2y i, for i =1,,n? 39