Week 1. Exploratory Data Analysis

Size: px

Start display at page:

Download "Week 1. Exploratory Data Analysis"

Horatio Laurence Brown
10 years ago
Views:

1 Week 1 Exploratory Data Analysis

2 Practicalities This course ST903 has students from both the MSc in Financial Mathematics and the MSc in Statistics. Two lectures and one seminar/tutorial per week. Exam (for the MSc in Financial Mathematics) in January, plus assessed coursework.

3 Aims and Objectives What s the course about? 1. Describing financial data 2. Modelling financial data 3. Making inferences about financial data

4 Samples and Populations Samples and Populations (Experimental) Unit the object on which measurements are made Population the set of all units about which information is wanted Sample the set of units about which information is available (Simple) random sample a sample such that units in the population have equal chance of inclusion, independent of the inclusion of any other unit Variable a measurable characteristic of a unit Statistic a measurable characteristic of a sample Parameter a measurable characteristic of a population

5 Variation Variation Natural Variation variation due to different units in the population having different values of the same variable Sampling Variation variation due to different samples containing different units and hence producing different values of the same statistic

6 Nature and Structure of Data Primary and Secondary Data Primary and Secondary Data Primary Data are collected specifically for the current study Observational e.g. survey data Intervention e.g. experimental data Secondary Data collected and/or compiled for another purpose can be limitations or problems with quality

7 Nature and Structure of Data Primary and Secondary Data Example: National Unemployment Data Suppose we want to know the UK unemployment figures in 5-year age bands to compare with similar figures from China, collected in Published data may be insufficient because only collected from major cities unemployment numbers presented in 10-year age bands compiled from a survey 10 years ago

8 Nature and Structure of Data Form of Data Form of Data - Samples Relationship between samples independent samples e.g. unemployment figures from two countries dependent samples e.g. social class of father and son Structure across samples unstructured e.g. unemployment figures from two countries structured e.g. 2 2 factorial experiment

9 Nature and Structure of Data Form of Data Form of Data - Variables Number of variables univariate, bivariate or multivariate Scales of measurement continuous e.g. age discrete e.g. sex (binary), ethnic origin (unordered categorical), social class (ordered categorical)

10 Stages of Data Analysis Stages of Data Analysis 1. Exploratory data analysis using descriptive statistics numerical summaries tabular summaries graphical summaries 2. Formal analysis using statistical techniques, often based on an assumed probability model 3. Presentation and evaluation of results

11 Numerical Summaries Numerical Summaries Numerical summaries help to describe and compare samples give information about corresponding parameters Qualitative data can be summarise by counts or percentages. Quantitative data can be summarised by measures of location, scale and shape.

12 Measures of Location Averages For observations x 1,..., x n, let x (j) denote the j th smallest observation (j th order statistic) Sample mean x = 1 n x i n i=1 Sample median x ( ) n+1 if n is odd 2 x M = [ ] 1 2 x ( ) n + x ( ) n if n is even = x ( ) n Sample mode the value which occurs most frequently in the sample

13 Measures of Location Averages - Advantages and Disadvantages Sample mean adv: conventional average; uses every value, convenient mathematically disadv: rarely corresponds to sample unit, influenced by outliers Sample median reverse adv/disadv of the sample mean Sample mode often not well defined; sample values are often poor values for populations

14 Measures of Location Quantiles Sample Lower Quartile x L = x ( ) n Sample Upper Quartile x U = x ( ) 3n pth Sample Percentile x 100p% = x ( ) pn Five Number Summary ( x(1), x L, x M, x U, x (n) )

15 Measures of Scale Measures of Scale Sample Variance V ar(x) = n j=1 (x j x) 2 n 1 Sample Standard Deviation (SD) V ar(x) Inter-Quartile Range(IQR) Sample Range x U x L x (n) x (1)

16 Measures of Scale Measures of Scale - Advantages and Disadvantages Variance similar adv/disadv to mean SD in the same units as the data - useful for interpretation IQR robust measure Sample Range sensitive to outliers, sampling variability and data errors

17 Measures of Shape Measures of Shape Modality number of peaks in the sample distribution Skewness a statistic measuring symmetry such that 0 symmetric sample distribution +ve skewed to the right (long right-hand tail) -ve skewed to the left (long left-hand tail) Kurtosis a statistic measuring peakedness such that 3 same peakedness as the Normal distribution (mesokurtic) > 3 more peaked - slim or long-tailed (leptokurtic) < 3 less peaked - flat, fat or short-tailed (platykurtic) Sometimes adjusted to give 0 for mesokurtic distributions.

18 Measures of Shape Skewness and Kurtosis f(x) x f(x) x

19 Measure of Linear Relation Between Two Variables Correlation Coefficient For observations x 1,..., x n ; y 1,..., y n of two variables X and Y Correlation Coefficient n i=1 r = (x i x)(y i ȳ) [ n i=1 (x i x) 2][ n i=1 (y i ȳ) 2] measure of linear relationship correlation does not imply cause may be linked via third variable

20 Tabular Summaries Tabular Summaries Provide succinct display of data set Emphasise the structure of the data Sometimes more powerful than a graph, or may provide record of graphed data Things to consider included data layout (dimensions, ordering, totals) representation of numbers (units, significant figures, percentages)

21 Tabular Summaries Example: Society of Business Economists Salary Survey Age (years) Per cent of responses Median salaries ( k)* & under Over Men Women *Including any London/regional allowance and self-employment income Source:

22 Graphical Summaries Graphical Summaries Graphical summaries are useful for providing an overall picture of the data exploring relationships e.g. comparing groups, exploring trends over time checking assumptions underlying methods of formal analysis checking for problems with the data, e.g. outliers

23 Graphical Summaries for Qualitative Data Graphical Summaries for Qualitative Data Pie Charts area of slices proportional to frequency - misleading to compare pie charts of different area or based on different sample sizes limited accuracy - rounding can be misleading hard to read with large number of segments Bar Charts height of bars proportional to frequency - more intuitive bars can be segmented to show component parts

24 Graphical Summaries for Qualitative Data Example: Shares of National Income Source: Survey of Current Business (2006) 86(1),

25 Graphical Summaries for Qualitative Data Example: Shares of National Income Other Taxes on production & imports Net interest & misc. payments Corporate profits Rental income of persons Proprietors' income Supplements to wages & salaries Wages and salary accruals

26 Graphical Summaries for Quantitative Data Stem-and-Leaf Plots Tallies data in bins, using values themselves for display E.g. Times (in hours) to first failure of air-conditioning unit on Boeing 720, different transformations hours 10 hours log10 hours

27 Graphical Summaries for Quantitative Data Stem-and-Leaf Plots E.g. Carbon-dating fragments of a pre-historic artefact, different scales 1000 years 100 years 100 years, split (* = 0-4,. = 5-9) * * *

28 Graphical Summaries for Quantitative Data Box Plots Represent five number summaries diagrammatically. Most software produce truncated box plots, which exclude outliers - these are usually plotted as isolated points E.g. Inflation rates over 20 year period for five countries USA UK Japan Germany France

29 Graphical Summaries for Quantitative Data Histogram Equivalent of barchart for binned continuous data Area of bars proportional to frequency in each bin - usually choose equal bin widths so height proportional to frequency E.g. GDP per capita for 26 countries Frequency GDP per capita ($)

30 Graphical Summaries for Quantitative Data Graphical Summaries of Distribution for Quantitative Data Stem-and-leaf adv: good for small data sets - shows all of the data disadv: choice of bins affects display Box plot adv: simple, can split by group, almost any sample size will do disadv: can be too simple, e.g. no good for multi-modal data Histogram adv: good for large data sets, shows all characteristics of distribution disadv: choice of bins affects display

31 Graphical Summaries for Quantitative Data Scatterplot Plot of data points in 2-D or 3-D space with variables as axes Useful for exploring relationships between variables E.g. Standard & Poor (S&P) company s index of 500 common stock prices against the Consumer Price Index (CPI) for SP500 Index CPI

32 Graphical Summaries for Quantitative Data Time Series Plot of data against time Look for seasonality, unusual events, etc E.g. Quarterly personal consumption expenditure (PCE) from (AUS$) PCE Time

Exploratory data analysis (Chapter 2) Fall 2011

Exploratory data analysis (Chapter 2) Fall 2011 Data Examples Example 1: Survey Data 1 Data collected from a Stat 371 class in Fall 2005 2 They answered questions about their: gender, major, year in school,