Descriptive Statistics. Understanding Data: Categorical Variables. Descriptive Statistics. Dataset: Shellfish Contamination

Size: px

Start display at page:

Download "Descriptive Statistics. Understanding Data: Categorical Variables. Descriptive Statistics. Dataset: Shellfish Contamination"

Candice Bond
7 years ago
Views:

1 Descriptive Statistics Understanding Data: Dataset: Shellfish Contamination Location Year Species Species2 Method Metals Cadmium (mg kg - ) Chromium (mg kg - ) Copper (mg kg - ) Lead (mg kg - ) Mercury (mg kg - ) Zinc (mg kg - ) Understand the distribution of the response variable Look for trends with the explanatory variables Consider the need for transformation Look for potentially influential observations Find errors that have occurred during data entry Test the assumptions of the statistical models that you intend to employ 2 Descriptive Statistics Categorical Variables Numerical size, middle and spread Size: n Graphical: A picture can save a thousand numbers. summary: proportions 3 4

for transformation Look for potentially influential observations Find errors that have occurred during data entry Test the assumptions of the statistical models that you

2 Continuous Variables Size: n middle : n x x, Median n i i Quantiles Sort the data into ascending order to obtain a sequence of order statistics x, x,, x n 2 The p'th quantile q p is the +(n-)p'th order statistic x +(n-)p (or an average of neigbouring values if +(n-)p is not integer). Spread: Standard deviation s q.2 =lower quartile, q. =median, q.7 = upper quartile Range, R=max(x)-min(x), E.g. n=, median = +()(.)=6 th order statistic 6 Unlike the arithmetic mean, the median is not at all influenced by the exact value of the largest objects and so provides a resistant measure of the central location. Graphical Summaries A picture can save a thousand numbers. 7 8

2 =lower quartile, q. =median, q.7 = upper quartile Range, R=max(x)-min(x), E.g. n=, median = +()(.

3 Boxplot (box-and-whiskers plot) The boxplot is a useful way of plotting the quantiles q, q.2,q., q.7 and q of the data. The ends of the whiskers show the position of the minimum and maximum of the data whereas the edges and line in centre of the box show the upper and lower quartiles and the median. The whiskers show at a glance the behaviour of the extreme outliers, whereas the box edges and mid-line summarize the sample in a resistant manner. Strong asymmetry in the box mid-line and whiskers suggests that the data is not symmetric. Histogram The range of values is divided up into a finite set of class intervals (bins). The number of objects in each bin is then counted and divided by the sample size to obtain the frequency of occurrence and then these are plotted as vertical bars of varying height. The histogram quickly reveals the location, spread, and shape of the distribution. The shape of the distribution can be unimodal (one hump), multimodal (many humps) or skewed (fatter tail to left or right). 9 Time Series plots Probability Distributions Useful way of seeing if there is any trend in a continuous variable across time. Scatter plots Useful way of seeing if there is any relationship between pairs of continuous variables. Models for population variability Provide simple descriptions Used as basis for statistical inference Many different models discrete: categories, counts continuous: standard measurements 2

The whiskers show at a glance the behaviour of the extreme outliers, whereas the box edges and mid-line summarize the sample in a resistant manner.

4 Discrete Distributions Described by probabilities.2. Chart of Mean( C2 ) vs C Example: Binomial distribution % of fish with high pcb levels How many contaminated fish in a group of size n? Mean of C C 3 4 Continuous Distributions For variables measured to an arbitrary precision on some scale No probability associated with specific values Histograms provide a useful lead-in Equal-width intervals: Draw boxes of height equal to frequency for each interval Frequency Area of each bar is proportional to frequency and relative frequency 9 bwt 9 6

5 Probability Densities Illustration Population histogram, bins Population histogram, bins For a population histogram, as you increase the number of histogram cells, and decrease the interval width The histogram approaches a smooth curve (conceptually). Density. e+. e- 2. e- pop Population histogram, bins Density. e+. e- 2. e- pop Population histogram, bins This is called a probability density function, or simply a density. Density. e+. e- 2. e- Density. e+. e- 2. e- pop pop 7 8 Probability Models Histogram with superimposed normal probability model This smooth density curve gives us a probability model for the population Take (simple) mathematical forms for these Allow probability calculations for the population (areas under the density curve) Can be compared with the distribution of the sample given by a histogram x Good Agreement! 9

6 Normal Distribution Model for continuous measurements Bell-shaped curve that approximates a density histogram for many types of observations Single mode Symmetric Parameters: mean standard deviation (variance 2 ) = 2 =74 Effects of and (a) Changing (b) Increasing shifts the curve along the axis increases the spread and flattens the curve = 6 = 2 = 6 2 = = 2 = Understanding the standard deviation Histogram with normal curve (c) Probabilities and numbers of standard deviations Shaded area =.683 Shaded area =.94 Shaded area = Histogram of Cadmium Normal Mean.2687 StDev.633 N 68 2 Frequency 68% chance of falling between and 9% chance of falling between and 99.7% chance of falling between and Cadmium

2 =7 2 22 Understanding the standard deviation Histogram with normal curve (c) Probabilities and numbers of standard deviations Shaded area =.683 Shaded area =.94 Shaded area =.

7 Probability Plot Probability Plot Normality Probability Plot of Cadmium Normal - 9% CI Probability Plot of C22 Normal - 9% CI Mean.2687 StDev.633 N 68 AD 4.39 P-Value < Mean.447 StDev.68 N AD.94 P-Value.89 Percent 8 7 Percent Cadmium C Percent Plotting by groups Probability Plot of Cadmium Normal - 9% CI M O M Mean.66 StDev.872 N 89 AD 3.46 P-Value <. O Mean.3848 StDev. N 79 AD.242 P-Value.763 Skewness Measured by skewness coefficient Negative left skewed (tail to left) Zero symmetric Positive right skewed (tail to right) Environmental data is frequently positive and skewed to the right mean > median...2. Panel variable: SpeciesGroup.7. Cadmium Variable Skewness Cadmium

66 StDev.872 N 89 AD 3.46 P-Value <. O Mean.3848 StDev. N 79 AD.242 P-Value.

8 Outliers Histogram of Copper Points which are outside the general pattern of the data recording errors Measurement failures Rogue values Greater variability Unsuspected factors Identify, assess impact, delete? Frequency M Panel variable: SpeciesGroup Copper O 29 Measures of location Mean highly sensitive to outliers, skewness Measurement scale dependent Median insensitive to outliers, distribution shape invariant to measurement scale Trimmed mean trim % from each tail; calculate mean of central part Median is % trimmed mean Measures of Spread Range = max min highly sensitive to outliers Standard deviation very sensitive to outliers, skewness Interquartile range length of central box of boxplot MAD median absolute deviation of data values from the median; robust 3 32

Frequency M 9 8 7 Panel variable: SpeciesGroup Copper O 29 Measures of location Mean highly sensitive to outliers, skewness Measurement scale dependent Median insensitive to outliers,

STATS8: Introduction to Biostatistics. Data Exploration. Babak Shahbaba Department of Statistics, UCI

STATS8: Introduction to Biostatistics Data Exploration Babak Shahbaba Department of Statistics, UCI Introduction After clearly defining the scientific problem, selecting a set of representative members