Descriptive Statistics

Transcription

1 2 Descriptive Statistics Cosider a batchof data, cosistig of observatios o a sigle variable. We represet the data by the array z 1 ;:::;z. We are ot cocered, at this stage, with the way the data have bee gathered. The data may be the result of a cesus or complete eumeratio of a populatio, or may be sample data. The term batch is used to emphasize the fact that, for the momet, o commitmet to a statistical model is beig made. Our rst step i the aalysis is to look at the data to see what they seem to say. This is called exploratory data aalysis, ad plays a fudametal role i statistical aalysis. The purpose of data aalysis is to orgaize ad reduce the data ito forms that are useful ad meaigful from the poit of view of the problem at had. This is doe by usig both umerical ad graphical methods. Exploratory data aalysis is ofte the rst step i a statistical aalysis, for it helps uderstadig the mai features of the data that a aalyst is usig. Itelliget descriptios or summaries of the data may sometimes be su±ciet to ful ll the purposes for which the data were gathered. E ective summaries ca also poit to \bad" data or uexpected aspects thatmight go uoticed if data are blidly cruched by computers. Further, exploratory data aalysis suggest possible statistical models for the data ad helps uderstadig the populatio features that a good model ought to be able to reproduce. I this ote we itroduce a few simple techiques. For more details o data aalysis, see J.W. Tukey (1977), Exploratory Data Aalysis, Addiso-Wesley, Readig, MA. 2.1 FREQUENCY DISTRIBUTIONS Whe the data may be classi ed i a small umber of categories, a useful piece of iformatio is how the observatios are distributed amog the various categories. Thigs are straightforward whe the data are discrete valued or categorical. I geeral, however, results may be sesitive to the particular way i which the data are classi ed. Suppose that the observatios are classi edi J categories or measuremet classes or cells. The frequecy of group j is de ed as f j = o. of observatios i category j; j = 1;:::;J:

2 6 The relative frequecy or proportio of category j is de ed as p j = f j ; 1;:::;J: If the categories ca be ordered i a atural way (for example, the variable cosidered is umerical), the oe may also cosider the cumulative relative frequecy o. of observatios i category j or lower F j = total o. of observatios f h = h j = h j p h ; j = 1;:::;J: Thus, F 1 = p 1 ad with F J = 1. F j = F j 1 + p j ; j = 2;:::;J; FREQUENCY TABLES A frequecy table presets i a tabular form the frequecy of the various categories i which the data have bee classi ed. Ofte data take very may values. I such cases, prelimiary groupig the data ito a small umber of categories or cells becomes essetial. Example 2.1 The followig set of data refers to the o er price of codomiiums (codos) i the Los Ageles Westside area. It is draw from the Westside Real Estate Guide, March 27, 1988, p. 16. Data are i thousads dollars. 2; 950 1; :6 169:5 159:5 A frequecy table ca be costructed to represet the distributio of the o er price of codos. Ulike the case of discrete data, there is ow o atural way of classifyig the data ito categories. Oe possibility is show i Table 1. It follows from this table that 64 percet of the codo o er prices are less tha 500 thousad dollars, 86 percet are less tha 1 millio dollars, etc. Usig the table, oe ca also compute the proportio of codo o er prices betwee 500 thousads ad 2 millio dollars, which is equal to p 2 + p 3 + p 4 = :22 + : = :29; or the proportio of the codo o er prices above 1 millio dollars, which is equal to Notice that: p 3 + p 4 + p 5 + p 6 = : :07 = :14:

3 DESCRIPTIVE STATISTICS 7 Table 1 Frequecy table of the codo price data. cell cell cell tally absolute relative cumulative boudaries midpoit freq. freq. relative freq. (f j ) (p j ) (F j ) 1 0{ xxxxxxxxx {1, xxx ,000{1,500 1,250 x ,500{2,000 1, ,000{2,500 2, ,500{3,000 2,750 x Table 2 Alterative frequecy table of the codo price data. cell cell cell tally absolute relative cumulative boudaries midpoit freq. freq. relative freq. (f j ) (p j ) (F j ) 1 0{ xxxx { xxxxx , xxx ,000{2,000 1,500 x ,000{3,000 2,500 x ² The choice of the umber of cells ad cell boudaries is arbitrary. ² The cell midpoit is a measure of where each cell is cetered, but eed ot correspod to ay speci c value appearig i tha data. The choice of cells i Table 1 may ot be etirely satisfactory, because it provides too little detail i the price rage from 0 to 500 thousads, whereas the detail provided for higher prices is oly apparet because two of the cells are empty. A di eret picture is obtaied by choosig the cells di eretly, as is doe i Table 2. Notice that: ² The width of the various cells is ow di eret. ² This secod represetatio captures oe feature of the data that the rst represetatio was uable to reveal, amely the relative cocetratio of the data i the rage from 250 to 500. ² The ewrepresetatiois otwithout problems itself, because itobscures the fact that oe value (2,950) is clearly quite distat from the rest of the data. This value may be termed a outlier. 2

4 8 Figure 1 Pie charts of the codo price data. 64% % % % % % % % % GRAPHICAL DISPLAYS The relative frequecies i Tables 1 ad 2 may also be represeted graphically, by meas of pie charts ad bar graphs or histograms. The pie charts correspodig to Tables 1 ad 2 are preseted i Figure 1. For histograms, the height of a bar may be equal to either the absolute frequecy of a cell (frequecy histogram, as i Figure 2) or its relative frequecy (relative frequecy histogram, as i Figure 3). A third alterative is to make the area of a bar equal to the relative frequecy of a cell. This last case has the advatage that the total area below the histogram is equal to oe. Notice that: ² For the choice of cells i Table 1, where each cell has the same width, it does ot matter whether the height of a bar is equal to the frequecy or the relative frequecy, or whether the area of a bar is equal to the relative frequecy. ² For the choice of cells i Table 2, where cells have di eret width, the appearace of a histogram chages somewhat depedig o whether the relative frequecy is measured by the the height of a bar or by its area. To summarize, costructig frequecy tables ad plottig relative or cumulative frequecies is very helpful. Some care is eeded, however, i the choice of the umber of cells ad the cell boudaries. Bad choices may obscure or eve distort what the data have to say. 2.2 THE EMPIRICAL DISTRIBUTION FUNCTION Suppose that the variable cosidered is umerical. If all observatios i a batch of data are distict, the it does ot make sese to compute relative frequecies uless the data have bee prelimiarly grouped i a small umber of categories. This is because the relative frequecy of a sigle observatio is just equal to 1=. Oe may however rak the observatios upward, from lowest to highest, ad the compute the cumulative relative frequecy of these ordered values. The ordered data values are also called the order statistics of the data. The graph of the cumulative

5 DESCRIPTIVE STATISTICS 9 Figure 2 Frequecy histogram of the codo price data. 10 frequecy offer price Figure 3 Relative frequecy histogram of the codo price data..6 relative frequecy offer price

6 10 Table 3 Frequecy table of o er price of codos i the Los Ageles-Westside area, March upward price dowward depth value of rak rak the edf / / / / / / / / / / / / , / , /14 relative frequecies agaist the order statistics is called the empirical distributio fuctio (edf). Notice that the cumulative relative frequecies are just the upward raks divided by the umber of observatios. To costruct the edf i the case of Example 2.1, we rst rak the observatios as is doe i Table 3. Oce observatios have bee raked, it becomes clear that for 1/14 percet of the codos the o er price is less or equal to 159.5, for 2/14 percet the o er price is less or equal to 169.5, etc. The edf is just the collectio of these percetages. The edf is usually represeted as a step fuctio with jumps at each of the distict data values (Figure 4). The size of each jump is equal to the relative frequecy of each data value. If the data values are all distict, the all jumps are of the same size 1=. If a data value is repeated m times, the the size of the jump at that value is equal to m=. The edf gives a detaileddescriptio of the geeral patters of the data. I particular, it shows quite clearly the existece of distict groupigs ad of outliers, that is, arbitrarily large or small data values. The plot of the edf also eables oe to compute easily ad rapidly certai importat summaries of the data. By this we mea a few easily foud umbers to tell somethig about the data as a whole. Fially, the edf provides a way of comparig two di eret batches of data, perhaps of di eret size. 2.3 MEASURES OF CENTER I the remaider of this ote we brie y discuss ways of summarizig three features of the distributio of a batch z 1 ;:::;z of umerical data: its ceter or locatio, its

7 DESCRIPTIVE STATISTICS 11 Figure4 Empirical distributio fuctio of the codo price data. 1 relative frequecy offer price spread or dispersio, ad its shape. All three cocepts are deliberately kept vague THE MEDIAN We have see that a detaileddescriptioof the data cabe obtaied by orderig them as doe i Table 3. These ordered data cotai all the relevat iformatio ad lead to a very atural measure of ceter or locatio. After rakig the observatios both upward ad dowward, let us de e the depth of a observatio as the least rak, either up or dow, that ca be give to such a observatio. The depth of a extreme is always equal to oe. We may de e the \middle value" as the oe that has the highest depth. If the batch has a odd umber of values, the there is a sigle value i the middle by cout. For example, if the are 13 values i all, the the middle value is the oe whose depth is equal to 7. If there are 15, the middle value is the oe whose depth is equal to 8. However, whe the batch has a eve umber of values, as i the case of our data, the middle value is ot uique, but cosists of a pair. This leads to the followig de itio. De itio 2.1 The media ^z is the middle value if the batch has a odd umber of values, ad is covetioally de ed as the average of the two middle values if the batch has a eve umber of values. 2 Example 2.2 I the case of our data, sice = 14 is eve, the media is ^z = = 435:

8 THE MEAN Aother commo middle value is the mea or arithmetic average, de ed as ¹z = z z = 1 z i : Notice that the mea value eed ot be oe of the actual values i the batch. If data are discrete, ad the jth value z j occurs with relative frequecy p j, the ¹z = j z j p j : The mea is very easy to compute, because it requires o orderig of the data. The use of the mea is partly justi ed by two importat properties. First, give two batches of data of equal size, x 1 ;:::;x ad y 1 ;:::;y, if z i = a + bx i + cy i, where a, b ad c are arbitrary costats, the the mea of z 1 ;:::;z is ¹z = 1 z i = 1 (a + bx i + cy i ) = a + b Ã 1 = a + b¹x + c¹y: x i! + c Ã! 1 y i (2.1) This property is ot shared by the media. What the media has i commo with the mea is oly the fact that if z i = a + bx i, where b > 0, the ^z i = a + b ^ i. Secod, suppose that i additio to the rst batch of data z 1 ;:::;z, for which ¹z is the mea, a secod batch z +1 ;:::;z +m of m data poits becomes available. Let ¹z be the mea of this secod batch. The mea of the pooled set of + m data poits is ¹z + = z z + z z +m + m = z z + m = + m ¹z + + z z +m + m m + m ¹z : (2.2) Thus, the overall mea is just a weighted sum of the meas computed for each batch, with weights that are oegative, add up to oe, ad are equal to the fractio of data from each batch o the total. This property also is ot shared by the media. If the distributioofthe data is symmetric, thethe meaad the media coicide. If the data are skewed to the right (left), the the mea is higher (lower) tha the media. If there are outliers or the distributio of the data is very asymmetric, the the mea ad the media ca be far apart from each other.

9 DESCRIPTIVE STATISTICS 13 Example 2.3 I the case of our data, the mea is ¹z = 8; 766:5 14 = 626:2; which is 44 percet greater tha the media. 2 To see why the the mea may be cosiderably greater tha the media, otice that the former is a weighted sum, ad that each term i the sum receives the same weight equal to 1=. As a cosequece, the mea is very sesitive to addig ad droppig observatios. Cosider for example addig a sigle value z to the available batch. By result (2.2), the ew mea becomes ¹z = ¹z + z + 1 ; ad so the di erece with respect to the old mea is ¹z ¹z = ¹z + z + 1 ¹z = z ¹z + 1 : This shows that the mea is very sesitive to the presece of just a sigle outlier. Viewed as a fuctio of z, the di erece ¹z ¹z is called the i uece fuctio of the mea. It is just a straight lie with slope equal to 1=( + 1). By a similar argumet, oe ca show that the mea is very sesitive to droppig a sigle value from the available batch. If ¹z (i) deotes the value of the mea for the batch of size 1 obtaied by droppig the ith data poit, the oe ca show that ¹z (i) ¹z = ¹z z i 1 ; i = 1;:::;: Because a sigle outlier i a batch of data poits is eough to completely spoil the mea, we say that the mea has breakdow poit equal to 1=. Example 2.4 I the case of our data, the mea is very sesitive to the iclusio of the sigle value 2,950. This may be cosidered a very i uetial data poit, sice droppig this observatio gives ¹z = 5; 816:5=13 = 447:4, which is ot far from the value of the media. 2 Besides lookig for the presece of outliers or i uetial data poits, oe should always try to uderstad why these poits occured. I the case of our data, for example, the value 2,950 may be a geuiely high o er price or it may istead be a recordig error, due to the additio of a digit TRIMMED MEANS To reduce the i uece of extreme observatios, oe may cosider droppig them from the average. A -percet symmetrically trimmed mea is obtaied by symmetrically trimmig percet of the data o either extreme ad the takig the average of the remaiig values.

10 14 Example 2.5 I our data, droppig the largest ad the smallest observatio ad the computig the average of the remaiig 12 values gives ~z = 169: ; = 5; 657:1 12 = 471:4; whichis much closer to the media. The proportio of data trimmed o either extreme is = 1=14. I order to completely spoil this trimmed mea, we ow eed two large outliers, that is, ~Z has a breakdow poit of 2= percet. Droppig the largest ad the smallest pair of observatios ad the computig the average of the remaiig 10 values gives ~z = 169: = 4; 406:6 10 = 440:7: The proportio of data trimmed o either extreme is ow = 2=14 = 1=7. I order to completely spoil this trimmed mea, we ow eed three large outliers, that is, the breakdow poit of ~z is ow 3= percet. 2 By icreasig the proportio trimmed o either extreme, the the media is evetually obtaied. Ideed, the media may be viewed as a symmetrically trimmed mea with trimmig proportio equal to = :5( 1)= if is odd ad = :5( 2)= if is eve. Thus, the media is essetially a 50 percet symmetrically trimmed mea. It is ot hard to see that, because of this, the breakdow poit of the media is early 50 percet THE MODE The mode is de ed as the data value that occurs most frequetly. The mode provides a particularly coveiet summary whe the data are grouped i categories. I this case, the modal category is the category that occurs most ofte. Notice that, ulike the mea ad the media, the mode is well de ed also for categorical data. 2.4 MEASURES OF SPREAD We cosider two di eret types of measure of spread for umerical data. The rst type is based o the cocept of rage. The secod type is based o the cocept of mea deviatio from some cetral value RANGES The di erece betwee the two extremes provides a rough measure of the spread of the data, called the rage. Example 2.6 I our data, rage = 2, = 2, We may also cosider how spread out is the \bulk" of the data. Oe possible way of de ig the bulk of data may is to cout half a way from each extreme to the media. The values thus obtaied are called higes by Tukey ad upper ad lower

11 DESCRIPTIVE STATISTICS 15 quartiles by other authors. Notice that the iterval betwee the two higes covers roughly 50 percet of the data. As a measure of spread oe may cosider the legth of such iterval, called the iterquartile rage (IQR) IQR = upper quartile - lower quartile: Example 2.7 I our data, there are seve values betwee each extreme ad the media. The two higes are therefore the 4th ad the 11th order statistic, that is, 175 ad 659, ad IQR = = Sectio 2.5 shows how the extremes, the media, the IQR ad the higes ca be summarized by a sigle graph MEAN ABSOLUTE AND MEAN SQUARED DEVIATIONS Other measures of spread ca be obtaied by cosiderig various kids of mea deviatio from some cetral value. If the mea deviatio is small, the the data are ot very spread out, if it is large, the the data are very spread out. Suppose rst that the cetral value is the mea ¹z. The sum of the deviatios of each observatio z i from ¹z satis es (z i ¹z) = z i ¹z = ¹z ¹z = 0: Hece, the mea average deviatio from the mea is always equal to zero. Therefore, if we wat a measure of average deviatio from the mea, we must cosider either the absolute deviatios jz i ¹zj or the squared deviatios (z i ¹z) 2. This leads to, respectively, the mea absolute deviatio from the mea MAD = 1 jz i ¹zj ad the mea squared deviatio from the mea MSD = 1 (z i ¹z) 2 = 1 (zi 2 2z i ¹z + ¹z 2 ) = 1 zi 2 ¹z2 : If data are discrete, ad the jth value z j occurs with relative frequecy p j, the MSD = j (z j ¹z) 2 p j = j z 2 j p j ¹z 2 :

12 16 As we shall see later, it may sometimes be preferable to use, istead of the MSD, the variace s 2 = 1 (z i ¹z) 2 = 1 1 MSD: Notice that squarig chages the scale of the observatios. Thus, for compariso with the MAD, it is more appropriate to cosider the (positive) square root of MSD or s 2. The (positive) square root of s 2 is deoted by s ad called the stadard deviatio. The MSD (ad therefore s 2 ) ejoys the followig ice property. If a variable Y is related to the variable Z through the relatioship y i = a + bz i, the the MSD of Y is MSD Y = 1 (y i ¹y) 2 = 1 (a + bz i a b¹z) 2 = b 2 Ã 1 = b 2 MSD Z ; (z i ¹z) 2! where MSD Z is the MSD of Z. Notice that all measures of spread cosidered so far, the rage, the IQR, the MAD ad MSD, are equal to zero if the data show o variability, that is, if z i is costat for all i. Other measures of spread ca easily be costructed by replacig the mea, as the middle value, with the media. Further, istead of cosiderig the average (absolute or squared) deviatio from some middle value, oe may cosider the media (absolute or squared) deviatio. 2.5 SUMMARIZING THE SHAPE Oe measure of ceter ad oe measure of spread are ofte all oe eeds to cocisely summarize the data. Just a pair of summary statistics, however, does ot provide a accurate descriptio of the data, i the sese that arbitrarily di eret batches of data may result i exactly the same descriptio. J.W. Tukey suggested the use of a box-ad-whisker plot or box plot, a graphical procedure with the followig features: (i) it combies a measure of locatio (the media) ad a measure of spread (the IQR), (ii) it shows the presece of possible outliers, ad (iii) it provides some idicatio about the shape of the distributio of the data i terms of their symmetry or skewess. Costructio of a box plot (Figure 5) proceeds as follows: 1. Vertical lies are draw at the media ad the upper ad lower quartiles are joied by horizotal lies to produce a box that cotais roughly 50 percet of the data. 2. A horizotal lie is draw up from the upper quartile to the most extreme data poit that is withi a distace of 1.5 IQR from the upper quartile.

13 DESCRIPTIVE STATISTICS 17 Figure 5 Box plot of the codo price data offer price 2950 A similarly de ed horizotal lie is draw dow from the lower quartile. Short vertical lies are added to mark the eds of these horizotal lies or \whiskers". The distace betwee the ed of the two whiskers is 4 times the IQR. 3. Each data poit beyod the ed of the whiskers is marked with a asterix or a dot. Symmetry or asymmetry is revealed by the locatio of the media relative to the upper ad lower quartiles. 2.6 SCATTERPLOTS The methods reviewed so far oly provide descriptios of a sigle variable. I may cases, we may be iterested i the relatioship betwee two variables, say ad Y. If x 1 ;:::;x ad y 1 ;:::;y are the observed values of two quatitative variables ad Y, the a scatterplot is a graph of the ordered pairs (x 1 ;y 1 );:::;(x ;y ). A scatterplot reveals immediately the presece of a relatioship betwee the two variables, or the lack therefor (Figure 6). I particular, a scatterplot reveals whether they ted to vary i the same directio (positive correlatio) or i the opposite directio (egative correlatio).

14 18 Figure 6 Scatterplots showig the types of correlatio betwee two variables, ady. positive egative 5 0 Y -5 o correlatio