# Econometrics Simple Linear Regression

Size: px
Start display at page:

Transcription

1 Econometrics Simple Linear Regression Burcu Eke UC3M

2 Linear equations with one variable Recall what a linear equation is: y = b 0 + b 1 x is a linear equation with one variable, or equivalently, a straight line. linear on x, we can think this as linear on its unknown parameter, i.e., y = x b 0 and b 1 are constants, b 0 is the y-intercept and b 1 is the slope of the line, y is the dependent variable, and x is the independent variable slope of a line being b 1 means for every 1 unit horizontal increase there is a b 1 unit vertical increase/decrease depending on the sign of b 1. 2

3 Linear equations with one variable A linear equation is of deterministic nature outcomes are precisely determined without any random variation A given input will always produce the same output perfect relationship In real life data, it is almost impossible to have such a prefect relationship between two variables. We almost always rely on rough predictions. One of our tools to do so is regression. 3

4 Empirical and Theoretical Relationships Economists are interested in the relationship between two or more economic variables at least bivariate populations The economic theory in general suggests relationships in functional forms (recall economic models). These relations are deterministic, such as Y = f(x), or Y = f (X 1, X 2,...X k ) A given input will always produce the same output perfect relationship 4

5 Example 1: Study time and score Let s consider the following example. The table below shows the data for total hours studied for a calculus test, x, and the test scores, y. time score x y

6 Example 1, cont d Consider the scatter plot based on the data of example 1. Does it look like a perfect relationship? 95 Study time vs Exam score

7 Example 2: Savings and income Relationship between savings (Y) and income (X) (Goldberger, 1991) Data from 1027 families, between , in the USA The joint distribution of savings and income are presented in the next table P (Y, X) For discrete case, P (Y, X) P (Y = y and X = x). 7

8 Example 2, cont d Table: Joint Distribution Y, savings, and X, income: P (Y, X) X (in 1000) Y P (Y ) P (X)

9 Example 2, cont d Among the information we can obtain from a joint probability table, there are two that are of interest to us: whether we can have a deterministic relationship, i.e., Y = f(x) whether savings and income are independent Can we have a deterministic relationship between savings and income based on the previous table? No. In order to have a deterministic relationship, we need to have a unique savings for each level of income, in other words we need to have a functional relationship. In terms of probability, for each income level, we need to have only one savings level with positive probability! 9

10 Example 2, cont d But we do have a relationship between income level and savings As income increases, savings level increases. To further investigate this, lets calculate the conditional distribution: P (Y, X) P (Y X) = P (X) ˆµ Y X=x = yp (Y = y X = x) y 10

11 Example 2, cont d Table: Conditional Distribution of Y, savings, given X, income: P (Y X) X (in 1000) Y Column sum ˆµ Y X What is the relationship between the conditional mean and income level? 11

12 Empirical and Theoretical Relationships The empirical relationships between economic variables not deterministic, but stochastic To combine theory and data, one must interpret the economic theory in a different way When the economic theory postulates that Y is a function of X, Y = f(x), it implies that the expected value of Y is a function of X, E[Y ] = f(x) Y = f(x) deterministic, E[Y ] = f(x) stochastic 12

13 Prediction When we are in a stochastic setting, we are in general interested in prediction, but how do we form our prediction? One way would be just to guess a number, but do you think it will be good? How can we assess that it is good? We are interested in finding the best way of predicting 13

14 Prediction Suppose we want to predict Y, using the information on X. Let c(x) be a prediction Then, U = Y c(x) will be our prediction error. We want the expected value of the square of our prediction, E[U 2 ] error to be as small as possible Why E[U 2 ]? Choose a prediction so that E[U 2 ] = (y c(x)) 2 P (y) y Y is minimized 14

15 Prediction: The best constant prediction We do not always have bivariate, or multivariate case, sometimes we only have information on Y In these cases, our prediction does not depend on other variables, so our prediction can be denoted as c Then we choose a prediction such that we minimize E[U 2 ] = y Y (y c) 2 P (y) c = E[Y ] = µ Y Average of Y is the best constant predictor 15

16 Exercise 2, cont d Y (Saving rate) P (Y ) Then the best constant predictor for the saving rate is: E[Y ] = = = 9.85% 16

17 Prediction: The best linear prediction In the case that we have bivariate data (Y, X), say savings rate and income levels, then we can make use of the relationship between them to predict Y Linear prediction implies that our c(x) is linear, i.e., c(x) = c 0 + c 1 X So, we need to find c 0 and c 1 such that E[U 2 ] = (y c(x)) 2 P (y) is minimized y Y 17

18 Prediction: The best linear prediction By replacing c(x) by c 0 + c 1 X, and solving for the minimization problem we get c 0 = α 0 = E(Y ) α 1 E(X) = µ Y α 1 µ X c 1 = α 1 = C(X, Y ) V (X) = σ XY σ 2 X The function α 0 + α 1 X is the linear projection (or the best linear prediction) of Y given X L(Y X) = α 0 + α 1 X 18

19 Example 2, cont d To get the best linear prediction for savings rate, we need to calculate: E[X], E[Y ], E[XY ], C(X, Y ), E[X 2 ], V (X) For E[XY ]: XY P(XY) XY P(XY) XY P(XY) XY P(XY) E[XY ] = X i Y j P (XY = X i Y j ) = i=1 j=1 19

20 Example 2, cont d E[X] = = C(X, Y ) = E[XY ] E[X]E[Y ] = = E[X 2 ] = = V (X) = E[X 2 ] (E[X]) 2 = = c 0 = α 0 = E(Y ) α 1 E(X) = c 1 = α 1 = C(X, Y ) = V (X) 20

21 Example 2, cont d L(Y X) = X The for discrete values of X, we get = if X = = if X = 4.9 L(Y X) = = if X = = if X =

22 Best Prediction So far we considered best constant predictor, and best linear predictor Let s relax the linearity restriction on c(x), i.e., c(x) can be any function that minimizes E[U 2 ] The best predictor of Y then becomes the conditional expected value of Y, E[Y X] If the E[Y X] is linear, then E[Y X] and L(Y X) are the same. The reverse is NOT true!!! If the E[Y X] is NOT linear, then L(Y X) is the best linear approximation to E[Y X] 22

23 Example 2, cont d X (income) c L(Y X) E[Y X] (in \$1000s) BCP BLP BP

24 Prediction Whenever L(Y X) E[Y X], L(Y X) provides a good approximation to E[Y X], hence can be used in some circumstances However, E[Y X] characterizes conditional mean of Y given X, the L(Y X) does not E[Y X] can have causal interpretation, the L(Y X) can not 24

25 SIMPLE LINEAR REGRESSION (SLR) 25

26 Simple Linear Regression Our big goal to analyze and study the relationship between two variables One approach to achieve this is simple linear regression, i.e, Y = β 0 + β 1 X + ε While answering our question, a simple linear regression model addresses some issues: 1. How to deal with the factors other than X that effects Y 2. How to formulate the relationship between X and Y 3. Whether our model captures a ceteris paribus relationship between X and Y 26

27 Simple Linear Regression SLR Model: Y = β 0 + β 1 X + ε Y Dependent variable, endogenous variable, response variable, regressand... X Independent variable, exogenous variable, control variable, regressor... β = (β 0, β 1 ) Parameter vector, population parameters ε Error term 27

28 Simple Linear Regression: Assumptions A1: Linear in parameters It implies that a unit change in X has the same effect on Y, independently of the initial value of X. SLR is linear in parameters: The following are linear in parameters: Y = β 0 + β 1 X + ε, Y = β 0 X 2 Z + β 1 logz + ε The following is NOT linear in parameters: Y = β 1 X β2 ε Violations of this assumption are form of a specification errors, such as nonlinearity (when the relationship is not linear in parameters) We can overcome the limitations this assumption imposes 28

29 Simple Linear Regression: Assumptions A2: E[ε X] = 0 For the all values of X, the mean error rate is same and equal to 0. E[ε X] = 0 implies for any function h(), C(h(X), ε) = 0. Hence, ε is not correlated with any function of X In particular, C(X, ε) = 0 C(X, ε) = E[Xε] E[X]E[ε] E[Xε] = E[E[Xε X]] = E[XE[ε X]] = 0 E(X)E(ε) = 0 29

30 Simple Linear Regression: Assumptions E[Y X] = β 0 + β 1 X, the conditional expectation function is linear, hence C(Y, X) C(Y, X) = C (E[Y X], X) = β 1 V (X) β 1 = V (X) E[Y ] = E [E[Y X]] = β 0 + β 1 E[X] β 0 = E[Y ] β 1 E[X] E[Y X] = L(Y X) = β β 1 X, where L(.) is the linear projection 30

31 Simple Linear Regression: Assumptions A3: Conditional Homoscedasticity V (ε X) = σ 2 X This implies V (ε) = σ 2 V (ε X) = E[ε 2 X] E [(E[ε X])] = E[ε 2 X] = σ 2 E[ε 2 ] = E [ E[ε 2 X] ] = σ 2 V (ε) = E[ε 2 ] (E[ε]) 2 = σ 2 This also implies V (Y X) = σ 2 31

32 Simple Linear Regression: Interpretation of Coefficients From A1, we have Y = β 0 + β 1 X + ε, if we take the conditional expectation of both sides we get E[Y X] = E [β 0 + β 1 X + ε] = β 0 + β 1 E[X X] + E[ε X](by A2) = β 0 + β 1 X E[Y X] This implies: β 1 =. That is, When X increases X by one unit, Y, on average, increases by β 1 units. 32

33 Simple Linear Regression: Interpretation of Coefficients E[Y X = 0] = E [β 0 + β 1 X + ε X = 0] = β 0 β 0 is the mean value of Y, when X = 0 β 0 is the intercept term In the case that X = 0 is not included in the range of X in our data set, β 0 has no interpretation 33

34 Simple Linear Regression: Interpretation of Coefficients Notice that A2 also implies E[Y X] = β 0 + β 1 X = L(Y X), that is, the best prediction is the best linear prediction If A2 is not satisfied, then E[Y X] L(Y X). This implies: β 0 and β 1 are only the parameters of a linear projection, NOT of a conditional expectation β 0 and β 1 do not have causal interpretations 34

35 Simple Linear Regression: Example 3 Scatterplot - Two Variables salary age 35

36 Simple Linear Regression: Example 3 The regression equation: Y = X (Assume that all assumptions are met) What is the interpretation of 13.10? Can we predict mean salary of a 0 years old person? Why? Why not? What is the interpretation of ? 36

37 SIMPLE LINEAR REGRESSION: THE ESTIMATION 37

38 Simple Linear Regression: Estimation Our Objective: Estimate the population parameters, i.e., β = (β 0, β 1 ) and σ 2 Our Model: Y = β 0 + β 1 X + ε, where E[ε X] = 0 and V (ε X) = σ 2 Our Data: Pairs of (y i, x i ) i = 1, 2,..., n are the sample realizations of population data (Y i, X i ): y x y 1 x 1 y 2 x y n x n 38

39 Simple Linear Regression: Estimation We need a method to estimate our population parameters β = (β 0, β 1 ) and σ 2, using our model Y i = β 0 + β 1 X i + ε i for all i = 1, 2,..., n We need to distinguish between population parameters β = (β 0, β 1 ) and σ 2 and the estimated values, denoted as ˆβ = ( ˆβ 0, ˆβ 1 ) and ˆσ 2 39

40 Estimation: The Analogy Principle An approach for estimation Use the corresponding sample quantity as an estimate of the population quantity Use sample mean Ȳ = mean n i=1 Use sample variance s 2 = population variance y i n i=1 to estimate the population ( n yi Ȳ ) 2 to estimate the n 1 Method of moments: Let µ i = E[X i ] be the i th moment, the by method of moments, we estimate µ i by n (x k ) i corresponding sample moment m i = n k=1 40

41 Estimation: The Analogy Principle Recall the bets prediction and the best linear prediction. Under our assumptions: E[Y X] = L(Y X) = β 0 + β 1 X β 0 = E(Y ) β 1 E(X) β 1 = C(X, Y ) V (X) 41

42 Estimation: The Analogy Principle then, by using the analogy principle we have: ˆβ 0 = Ȳ ˆβ 1 X ˆβ 1 = n i (X i X)(Y i Ȳ ) n i (X i X) 2 = 1 n n i (X i X)Y i n i (X i X) 2 1 n = S XY S 2 X 42

43 Estimation: The Ordinary Least Squares (OLS) The idea of the (OLS) principle is to choose parameter estimates that minimize the squared error term, i.e., the distance between the data and the model min E[ε 2 ], or equivalently, min E [(Y β 0 + β 1 X) 2] ε i = Y i E[Y i X i ] 43

44 The Ordinary Least Squares (OLS) For the sample data, we define the error term as ˆε i = Y i Ŷi = Y i ˆβ 0 ˆβ 1 X i Then the sample equivalent of min E[ε 2 ] is 1 min β 0,β 1 n n i=1 ˆε 2 i 44

45 The Ordinary Least Squares (OLS) FOC are: n i=1 ˆε = 0 n i=1 ˆεX i = 0 These FOC implies: ˆβ 0 = Ȳ ˆβ 1 X ˆβ 1 = n i=1 x iy i n i=1 x2 i c i = x i n i=1 x2 i n = c i Y i where x i = (X i X) and i=1 45

46 The Ordinary Least Squares (OLS) Compare Least square estimators and analogy principle estimators. are the similar? Predicted values: Ŷi = ˆβ 0 ˆβ 1 X i Predicted values are orthogonal to the predicted error term i Ŷiˆε i = 0 46

47 Estimation Terminology Estimator is a RULE for calculating an estimate of a parameter based on observed data, i.e., ˆβ 0 = Ȳ ˆβ 1 X Estimate is a number that is an outcome of an estimation process, i.e, ˆβ 1 =

48 The Properties of The OLS Estimators P1 Linear in Y ˆβ0 = Ȳ ˆβ 1 X ˆβ1 = i c iy i 48

49 The Properties of The OLS Estimators P2 Unbiasedness: Expected value of the estimator is equal to the parameter itself We achieve this property by A1 and A2 E[ ˆβ 0 ] = β 0 E[ ˆβ 1 ] = β 1 49

50 The Properties of The OLS Estimators The unbiasedness property indicates that if we have infinitely many samples of size-n from the same population and estimate the same model with each of the samples: We will have a distribution of the estimator of β l, with a different realization for each sample The mean of that distribution will be the the population parameter β l 50

51 The Properties of The OLS Estimators E[ ˆβ 1 ] = β 1 ˆβ 1 = i c i Y i = c i (β 0 + β 1 X i + ε i ) i = β 0 c i + β 1 c i X i + c i ε i i i i = β 1 c i X i + ( c i ε i because ) c i = 0 i i i = β 1 + ( c i ε i because ) c i X i = 1 i i 51

52 The Properties of The OLS Estimators E[ ˆβ 1 X] = E [ β 1 + i c i ε i X ] = β 1 + i = β 1 E[β 1 ] = E [E[β 1 X]] = E[β 1 ] = β 1 c i E[ε i X] } {{ } 0 52

53 The Properties of The OLS Estimators The Variance From A1, A2, and A3, we have V ( ˆβ 0 ) = i X 2 i n V ( ˆβ 1 ) [ ] V ( ˆβ 1 ) = σ2 1 n E SX 2 53

54 The Properties of The OLS Estimators Let s show that V ( ˆβ 1 ) = σ2 n E [ ( V ( ˆβ ) ] 2 1 ) = E ˆβ1 β 1 [ 1 S 2 X ] ( = E i ) 2 c i ε i = i [ = E E [ c 2 i ε 2 i X ]] [ = E c 2 i E [ ε 2 i X ]] = σ 2 E = σ 2 E = σ2 n E i [ i [ i [ 1 S 2 X c 2 i ] ( xi ] (by A3) i x2 i ) 2 ] 54 i = σ 2 E [ i 1 n 1 n ( i x2 i )2 E[c 2 i ε 2 i ] ]

55 The Properties of The OLS Estimators: Gauss-Markov Theorem Under assumptions A1 to A3, the Gauss-Markov Theorem states that the ˆβ 0 and ˆβ 0 are the efficient estimators among all linear unbiased estimators ˆβ0 and ˆβ 0 are the most efficient estimators among all the linear unbiased estimators Best Linear Unbiased Estimator 55

56 The Properties of The OLS Estimators: Gauss-Markov Theorem What does it mean to be a consistent estimator? ˆβ 0 and ˆβ 1 are consistent if in other words, p lim ˆβ j = β j, j = 0, 1 n ( ) lim P ˆβj β j < δ = 1 δ > 0 n Consistency means the distributions of the estimators ˆβ 0 and ˆβ 1 become more and more concentrated near the true values β 0 and β 1 as sample size increases, so that the probability of ˆβ 0 and ˆβ 1 being arbitrarily close to β 0 and β 1 converges to one For these results, we need A1 to A3 to met!!! 56

57 The Variance of The OLS Estimators The variance of the OLS estimators, ˆβ 0 and ˆβ 1, depends on σ 2 = V (ε) = E[ε 2 ] The problem is, the errors ε = ε 1, ε 2,..., ε n are not observed, so we need to estimate ε ˆε = Y ˆβ 0 ˆβ 1 X = β 0 + β 1 X + ε ˆβ 0 ˆβ 1 X ( ) ( ) = ε ˆβ0 β 0 ˆβ1 β 1 X Even though E[ ˆβ j ] = β j for j = 0, 1, ˆε ε 57

58 The Variance of The OLS Estimators One attempt to estimate σ 2 would be to use (1\n) ε 2 instead of E[ε 2 ]. However, this is not feasible Another attempt to estimate σ 2 would be using sample values of the error, ˆε, and use this to estimate the sample variance: σ 2 = (1\n) ˆε 2. However, this is biased. The bias comes from unmatched degrees of freedom and denominator When we account for this bias, we get a consistent and unbiased estimate: ˆσ 2 = i ˆε 2 i n 2 58

59 The Variance of The OLS Estimators Note that both of ˆσ 2 and σ 2 are consistent and for larger samples, their value is very close to each other. [ ] Recall that V ( ˆβ 1 ) = σ2 1 n E SX 2 [ ] We estimate V ( ˆβ 1 ) by ˆσ 2 ˆV ( ˆβ 1 ) = ˆσ2 1 nsx 2, where E SX 2 is estimated by 1 SX 2 Recall V ( ˆβ 0 ) = i X 2 i n V ( ˆβ 1 ) We estimate V ( ˆβ 0 ) by ˆσ 2 ˆV ( ˆβ 0 ) = ˆσ2 i X2 i n 2 S 2 X 59

60 The Goodness of Fit Measures The goodness of fit measures are the measures of how well our model fits the data. Standard error of regression, ˆσ, is one intuitive way of approaching goodness of fit. Recall that in the standard simple linear regression, we have Y = β 0 + β 1 X + ε and V (ε X) = σ 2. One problem with this goodness of fit method is that its magnitude heavily depends on the units of Y 60

61 The Goodness of Fit Measures Coefficient of Determination, R 2, is a very common measure of goodness of fit R 2 = i ŷ2 i = 1 i ε2 i (1) i y2 i i y2 i where y i = Y i Ȳ, ŷ i = Ŷi Ȳ, and ε i = Y i Ŷi Notice that y i = ŷ i + ε i. This implies that 0 i ŷ2 i 1 If R 2 = 1, the model explains all of the variation (100%) in Y If R 2 = 0, the model explains none of the variation (0%) in Y i y2 i 61

62 The Goodness of Fit Measures A low R 2 implies: The model fails to explain a significant proportion of the variation of Y. A low R2 does not necessarily mean that the estimates are unhelpful or inappropriate. The R 2 can be used for comparing different models for the same dependent variable Y. Also, for the correlation coefficient between actual and estimated values of Y, ρ Y, Ŷ, we ahe the following ( ) 2 relationship: R 2 = ˆρ Y, Ŷ 62

### 2. Linear regression with multiple regressors

2. Linear regression with multiple regressors Aim of this section: Introduction of the multiple regression model OLS estimation in multiple regression Measures-of-fit in multiple regression Assumptions

### Lecture 3: Linear methods for classification

Lecture 3: Linear methods for classification Rafael A. Irizarry and Hector Corrada Bravo February, 2010 Today we describe four specific algorithms useful for classification problems: linear regression,

### Covariance and Correlation

Covariance and Correlation ( c Robert J. Serfling Not for reproduction or distribution) We have seen how to summarize a data-based relative frequency distribution by measures of location and spread, such

### Simple Linear Regression Inference

Simple Linear Regression Inference 1 Inference requirements The Normality assumption of the stochastic term e is needed for inference even if it is not a OLS requirement. Therefore we have: Interpretation

### IAPRI Quantitative Analysis Capacity Building Series. Multiple regression analysis & interpreting results

IAPRI Quantitative Analysis Capacity Building Series Multiple regression analysis & interpreting results How important is R-squared? R-squared Published in Agricultural Economics 0.45 Best article of the

### Introduction to Regression and Data Analysis

Statlab Workshop Introduction to Regression and Data Analysis with Dan Campbell and Sherlock Campbell October 28, 2008 I. The basics A. Types of variables Your variables may take several forms, and it

### Part 2: Analysis of Relationship Between Two Variables

Part 2: Analysis of Relationship Between Two Variables Linear Regression Linear correlation Significance Tests Multiple regression Linear Regression Y = a X + b Dependent Variable Independent Variable

### Outline. Topic 4 - Analysis of Variance Approach to Regression. Partitioning Sums of Squares. Total Sum of Squares. Partitioning sums of squares

Topic 4 - Analysis of Variance Approach to Regression Outline Partitioning sums of squares Degrees of freedom Expected mean squares General linear test - Fall 2013 R 2 and the coefficient of correlation

### Chapter 3: The Multiple Linear Regression Model

Chapter 3: The Multiple Linear Regression Model Advanced Econometrics - HEC Lausanne Christophe Hurlin University of Orléans November 23, 2013 Christophe Hurlin (University of Orléans) Advanced Econometrics

### 17. SIMPLE LINEAR REGRESSION II

17. SIMPLE LINEAR REGRESSION II The Model In linear regression analysis, we assume that the relationship between X and Y is linear. This does not mean, however, that Y can be perfectly predicted from X.

### Chapter 13 Introduction to Nonlinear Regression( 非 線 性 迴 歸 )

Chapter 13 Introduction to Nonlinear Regression( 非 線 性 迴 歸 ) and Neural Networks( 類 神 經 網 路 ) 許 湘 伶 Applied Linear Regression Models (Kutner, Nachtsheim, Neter, Li) hsuhl (NUK) LR Chap 10 1 / 35 13 Examples

### Joint Exam 1/P Sample Exam 1

Joint Exam 1/P Sample Exam 1 Take this practice exam under strict exam conditions: Set a timer for 3 hours; Do not stop the timer for restroom breaks; Do not look at your notes. If you believe a question

### Example: Boats and Manatees

Figure 9-6 Example: Boats and Manatees Slide 1 Given the sample data in Table 9-1, find the value of the linear correlation coefficient r, then refer to Table A-6 to determine whether there is a significant

### Overview of Violations of the Basic Assumptions in the Classical Normal Linear Regression Model

Overview of Violations of the Basic Assumptions in the Classical Normal Linear Regression Model 1 September 004 A. Introduction and assumptions The classical normal linear regression model can be written

### Solución del Examen Tipo: 1

Solución del Examen Tipo: 1 Universidad Carlos III de Madrid ECONOMETRICS Academic year 2009/10 FINAL EXAM May 17, 2010 DURATION: 2 HOURS 1. Assume that model (III) verifies the assumptions of the classical

### 2. Simple Linear Regression

Research methods - II 3 2. Simple Linear Regression Simple linear regression is a technique in parametric statistics that is commonly used for analyzing mean response of a variable Y which changes according

### MULTIVARIATE PROBABILITY DISTRIBUTIONS

MULTIVARIATE PROBABILITY DISTRIBUTIONS. PRELIMINARIES.. Example. Consider an experiment that consists of tossing a die and a coin at the same time. We can consider a number of random variables defined

### Summary of Formulas and Concepts. Descriptive Statistics (Ch. 1-4)

Summary of Formulas and Concepts Descriptive Statistics (Ch. 1-4) Definitions Population: The complete set of numerical information on a particular quantity in which an investigator is interested. We assume

### Module 3: Correlation and Covariance

Using Statistical Data to Make Decisions Module 3: Correlation and Covariance Tom Ilvento Dr. Mugdim Pašiƒ University of Delaware Sarajevo Graduate School of Business O ften our interest in data analysis

### Chapter 7: Simple linear regression Learning Objectives

Chapter 7: Simple linear regression Learning Objectives Reading: Section 7.1 of OpenIntro Statistics Video: Correlation vs. causation, YouTube (2:19) Video: Intro to Linear Regression, YouTube (5:18) -

### CORRELATION ANALYSIS

CORRELATION ANALYSIS Learning Objectives Understand how correlation can be used to demonstrate a relationship between two factors. Know how to perform a correlation analysis and calculate the coefficient

### Estimation of σ 2, the variance of ɛ

Estimation of σ 2, the variance of ɛ The variance of the errors σ 2 indicates how much observations deviate from the fitted surface. If σ 2 is small, parameters β 0, β 1,..., β k will be reliably estimated

### Univariate Regression

Univariate Regression Correlation and Regression The regression line summarizes the linear relationship between 2 variables Correlation coefficient, r, measures strength of relationship: the closer r is

### Basic Statistics and Data Analysis for Health Researchers from Foreign Countries

Basic Statistics and Data Analysis for Health Researchers from Foreign Countries Volkert Siersma siersma@sund.ku.dk The Research Unit for General Practice in Copenhagen Dias 1 Content Quantifying association

### Correlation in Random Variables

Correlation in Random Variables Lecture 11 Spring 2002 Correlation in Random Variables Suppose that an experiment produces two random variables, X and Y. What can we say about the relationship between

### Correlation key concepts:

CORRELATION Correlation key concepts: Types of correlation Methods of studying correlation a) Scatter diagram b) Karl pearson s coefficient of correlation c) Spearman s Rank correlation coefficient d)

### Stochastic Inventory Control

Chapter 3 Stochastic Inventory Control 1 In this chapter, we consider in much greater details certain dynamic inventory control problems of the type already encountered in section 1.3. In addition to the

### Clustering in the Linear Model

Short Guides to Microeconometrics Fall 2014 Kurt Schmidheiny Universität Basel Clustering in the Linear Model 2 1 Introduction Clustering in the Linear Model This handout extends the handout on The Multiple

### NCSS Statistical Software Principal Components Regression. In ordinary least squares, the regression coefficients are estimated using the formula ( )

Chapter 340 Principal Components Regression Introduction is a technique for analyzing multiple regression data that suffer from multicollinearity. When multicollinearity occurs, least squares estimates

### ECON 142 SKETCH OF SOLUTIONS FOR APPLIED EXERCISE #2

University of California, Berkeley Prof. Ken Chay Department of Economics Fall Semester, 005 ECON 14 SKETCH OF SOLUTIONS FOR APPLIED EXERCISE # Question 1: a. Below are the scatter plots of hourly wages

### Simple linear regression

Simple linear regression Introduction Simple linear regression is a statistical method for obtaining a formula to predict values of one variable from another where there is a causal relationship between

### Hypothesis testing - Steps

Hypothesis testing - Steps Steps to do a two-tailed test of the hypothesis that β 1 0: 1. Set up the hypotheses: H 0 : β 1 = 0 H a : β 1 0. 2. Compute the test statistic: t = b 1 0 Std. error of b 1 =

### Simple Regression Theory II 2010 Samuel L. Baker

SIMPLE REGRESSION THEORY II 1 Simple Regression Theory II 2010 Samuel L. Baker Assessing how good the regression equation is likely to be Assignment 1A gets into drawing inferences about how close the

### 5. Linear Regression

5. Linear Regression Outline.................................................................... 2 Simple linear regression 3 Linear model............................................................. 4

### The Method of Least Squares

Hervé Abdi 1 1 Introduction The least square methods (LSM) is probably the most popular technique in statistics. This is due to several factors. First, most common estimators can be casted within this

### Chapter 10. Key Ideas Correlation, Correlation Coefficient (r),

Chapter 0 Key Ideas Correlation, Correlation Coefficient (r), Section 0-: Overview We have already explored the basics of describing single variable data sets. However, when two quantitative variables

### Sales forecasting # 2

Sales forecasting # 2 Arthur Charpentier arthur.charpentier@univ-rennes1.fr 1 Agenda Qualitative and quantitative methods, a very general introduction Series decomposition Short versus long term forecasting

### CHAPTER 13 SIMPLE LINEAR REGRESSION. Opening Example. Simple Regression. Linear Regression

Opening Example CHAPTER 13 SIMPLE LINEAR REGREION SIMPLE LINEAR REGREION! Simple Regression! Linear Regression Simple Regression Definition A regression model is a mathematical equation that descries the

### Answer: C. The strength of a correlation does not change if units change by a linear transformation such as: Fahrenheit = 32 + (5/9) * Centigrade

Statistics Quiz Correlation and Regression -- ANSWERS 1. Temperature and air pollution are known to be correlated. We collect data from two laboratories, in Boston and Montreal. Boston makes their measurements

### Financial Risk Management Exam Sample Questions/Answers

Financial Risk Management Exam Sample Questions/Answers Prepared by Daniel HERLEMONT 1 2 3 4 5 6 Chapter 3 Fundamentals of Statistics FRM-99, Question 4 Random walk assumes that returns from one time period

### Variance Reduction. Pricing American Options. Monte Carlo Option Pricing. Delta and Common Random Numbers

Variance Reduction The statistical efficiency of Monte Carlo simulation can be measured by the variance of its output If this variance can be lowered without changing the expected value, fewer replications

### 1 Teaching notes on GMM 1.

Bent E. Sørensen January 23, 2007 1 Teaching notes on GMM 1. Generalized Method of Moment (GMM) estimation is one of two developments in econometrics in the 80ies that revolutionized empirical work in

### Statistical Machine Learning

Statistical Machine Learning UoC Stats 37700, Winter quarter Lecture 4: classical linear and quadratic discriminants. 1 / 25 Linear separation For two classes in R d : simple idea: separate the classes

### Chapter 4: Vector Autoregressive Models

Chapter 4: Vector Autoregressive Models 1 Contents: Lehrstuhl für Department Empirische of Wirtschaftsforschung Empirical Research and und Econometrics Ökonometrie IV.1 Vector Autoregressive Models (VAR)...

### Section 14 Simple Linear Regression: Introduction to Least Squares Regression

Slide 1 Section 14 Simple Linear Regression: Introduction to Least Squares Regression There are several different measures of statistical association used for understanding the quantitative relationship

### Efficiency and the Cramér-Rao Inequality

Chapter Efficiency and the Cramér-Rao Inequality Clearly we would like an unbiased estimator ˆφ (X of φ (θ to produce, in the long run, estimates which are fairly concentrated i.e. have high precision.

### Course Objective This course is designed to give you a basic understanding of how to run regressions in SPSS.

SPSS Regressions Social Science Research Lab American University, Washington, D.C. Web. www.american.edu/provost/ctrl/pclabs.cfm Tel. x3862 Email. SSRL@American.edu Course Objective This course is designed

### Multi-variable Calculus and Optimization

Multi-variable Calculus and Optimization Dudley Cooke Trinity College Dublin Dudley Cooke (Trinity College Dublin) Multi-variable Calculus and Optimization 1 / 51 EC2040 Topic 3 - Multi-variable Calculus

### IMPACT EVALUATION: INSTRUMENTAL VARIABLE METHOD

REPUBLIC OF SOUTH AFRICA GOVERNMENT-WIDE MONITORING & IMPACT EVALUATION SEMINAR IMPACT EVALUATION: INSTRUMENTAL VARIABLE METHOD SHAHID KHANDKER World Bank June 2006 ORGANIZED BY THE WORLD BANK AFRICA IMPACT

### problem arises when only a non-random sample is available differs from censored regression model in that x i is also unobserved

4 Data Issues 4.1 Truncated Regression population model y i = x i β + ε i, ε i N(0, σ 2 ) given a random sample, {y i, x i } N i=1, then OLS is consistent and efficient problem arises when only a non-random

### Introduction to General and Generalized Linear Models

Introduction to General and Generalized Linear Models General Linear Models - part I Henrik Madsen Poul Thyregod Informatics and Mathematical Modelling Technical University of Denmark DK-2800 Kgs. Lyngby

### Review Jeopardy. Blue vs. Orange. Review Jeopardy

Review Jeopardy Blue vs. Orange Review Jeopardy Jeopardy Round Lectures 0-3 Jeopardy Round \$200 How could I measure how far apart (i.e. how different) two observations, y 1 and y 2, are from each other?

### Chicago Booth BUSINESS STATISTICS 41000 Final Exam Fall 2011

Chicago Booth BUSINESS STATISTICS 41000 Final Exam Fall 2011 Name: Section: I pledge my honor that I have not violated the Honor Code Signature: This exam has 34 pages. You have 3 hours to complete this

### Principle of Data Reduction

Chapter 6 Principle of Data Reduction 6.1 Introduction An experimenter uses the information in a sample X 1,..., X n to make inferences about an unknown parameter θ. If the sample size n is large, then

### Quadratic forms Cochran s theorem, degrees of freedom, and all that

Quadratic forms Cochran s theorem, degrees of freedom, and all that Dr. Frank Wood Frank Wood, fwood@stat.columbia.edu Linear Regression Models Lecture 1, Slide 1 Why We Care Cochran s theorem tells us

### The Bivariate Normal Distribution

The Bivariate Normal Distribution This is Section 4.7 of the st edition (2002) of the book Introduction to Probability, by D. P. Bertsekas and J. N. Tsitsiklis. The material in this section was not included

### Data Mining and Data Warehousing. Henryk Maciejewski. Data Mining Predictive modelling: regression

Data Mining and Data Warehousing Henryk Maciejewski Data Mining Predictive modelling: regression Algorithms for Predictive Modelling Contents Regression Classification Auxiliary topics: Estimation of prediction

### Chapter 13 Introduction to Linear Regression and Correlation Analysis

Chapter 3 Student Lecture Notes 3- Chapter 3 Introduction to Linear Regression and Correlation Analsis Fall 2006 Fundamentals of Business Statistics Chapter Goals To understand the methods for displaing

### Multiple Linear Regression in Data Mining

Multiple Linear Regression in Data Mining Contents 2.1. A Review of Multiple Linear Regression 2.2. Illustration of the Regression Process 2.3. Subset Selection in Linear Regression 1 2 Chap. 2 Multiple

### 1. What is the critical value for this 95% confidence interval? CV = z.025 = invnorm(0.025) = 1.96

1 Final Review 2 Review 2.1 CI 1-propZint Scenario 1 A TV manufacturer claims in its warranty brochure that in the past not more than 10 percent of its TV sets needed any repair during the first two years

### 1 Prior Probability and Posterior Probability

Math 541: Statistical Theory II Bayesian Approach to Parameter Estimation Lecturer: Songfeng Zheng 1 Prior Probability and Posterior Probability Consider now a problem of statistical inference in which

### PS 271B: Quantitative Methods II. Lecture Notes

PS 271B: Quantitative Methods II Lecture Notes Langche Zeng zeng@ucsd.edu The Empirical Research Process; Fundamental Methodological Issues 2 Theory; Data; Models/model selection; Estimation; Inference.

### Note 2 to Computer class: Standard mis-specification tests

Note 2 to Computer class: Standard mis-specification tests Ragnar Nymoen September 2, 2013 1 Why mis-specification testing of econometric models? As econometricians we must relate to the fact that the

### Point Biserial Correlation Tests

Chapter 807 Point Biserial Correlation Tests Introduction The point biserial correlation coefficient (ρ in this chapter) is the product-moment correlation calculated between a continuous random variable

### Hypothesis Testing for Beginners

Hypothesis Testing for Beginners Michele Piffer LSE August, 2011 Michele Piffer (LSE) Hypothesis Testing for Beginners August, 2011 1 / 53 One year ago a friend asked me to put down some easy-to-read notes

### 1 Sufficient statistics

1 Sufficient statistics A statistic is a function T = rx 1, X 2,, X n of the random sample X 1, X 2,, X n. Examples are X n = 1 n s 2 = = X i, 1 n 1 the sample mean X i X n 2, the sample variance T 1 =

### X X X a) perfect linear correlation b) no correlation c) positive correlation (r = 1) (r = 0) (0 < r < 1)

CORRELATION AND REGRESSION / 47 CHAPTER EIGHT CORRELATION AND REGRESSION Correlation and regression are statistical methods that are commonly used in the medical literature to compare two or more variables.

### Introduction to Fixed Effects Methods

Introduction to Fixed Effects Methods 1 1.1 The Promise of Fixed Effects for Nonexperimental Research... 1 1.2 The Paired-Comparisons t-test as a Fixed Effects Method... 2 1.3 Costs and Benefits of Fixed

### Some probability and statistics

Appendix A Some probability and statistics A Probabilities, random variables and their distribution We summarize a few of the basic concepts of random variables, usually denoted by capital letters, X,Y,

### Section 3 Part 1. Relationships between two numerical variables

Section 3 Part 1 Relationships between two numerical variables 1 Relationship between two variables The summary statistics covered in the previous lessons are appropriate for describing a single variable.

### Coefficient of Determination

Coefficient of Determination The coefficient of determination R 2 (or sometimes r 2 ) is another measure of how well the least squares equation ŷ = b 0 + b 1 x performs as a predictor of y. R 2 is computed

### Department of Mathematics, Indian Institute of Technology, Kharagpur Assignment 2-3, Probability and Statistics, March 2015. Due:-March 25, 2015.

Department of Mathematics, Indian Institute of Technology, Kharagpur Assignment -3, Probability and Statistics, March 05. Due:-March 5, 05.. Show that the function 0 for x < x+ F (x) = 4 for x < for x

### Lecture 2. Marginal Functions, Average Functions, Elasticity, the Marginal Principle, and Constrained Optimization

Lecture 2. Marginal Functions, Average Functions, Elasticity, the Marginal Principle, and Constrained Optimization 2.1. Introduction Suppose that an economic relationship can be described by a real-valued

### APPLICATION OF LINEAR REGRESSION MODEL FOR POISSON DISTRIBUTION IN FORECASTING

APPLICATION OF LINEAR REGRESSION MODEL FOR POISSON DISTRIBUTION IN FORECASTING Sulaimon Mutiu O. Department of Statistics & Mathematics Moshood Abiola Polytechnic, Abeokuta, Ogun State, Nigeria. Abstract

### HYPOTHESIS TESTING: CONFIDENCE INTERVALS, T-TESTS, ANOVAS, AND REGRESSION

HYPOTHESIS TESTING: CONFIDENCE INTERVALS, T-TESTS, ANOVAS, AND REGRESSION HOD 2990 10 November 2010 Lecture Background This is a lightning speed summary of introductory statistical methods for senior undergraduate

### HYPOTHESIS TESTING: POWER OF THE TEST

HYPOTHESIS TESTING: POWER OF THE TEST The first 6 steps of the 9-step test of hypothesis are called "the test". These steps are not dependent on the observed data values. When planning a research project,

### Multiple Regression: What Is It?

Multiple Regression Multiple Regression: What Is It? Multiple regression is a collection of techniques in which there are multiple predictors of varying kinds and a single outcome We are interested in

### Week 5: Multiple Linear Regression

BUS41100 Applied Regression Analysis Week 5: Multiple Linear Regression Parameter estimation and inference, forecasting, diagnostics, dummy variables Robert B. Gramacy The University of Chicago Booth School

### 4. Simple regression. QBUS6840 Predictive Analytics. https://www.otexts.org/fpp/4

4. Simple regression QBUS6840 Predictive Analytics https://www.otexts.org/fpp/4 Outline The simple linear model Least squares estimation Forecasting with regression Non-linear functional forms Regression

### Correlation Coefficient The correlation coefficient is a summary statistic that describes the linear relationship between two numerical variables 2

Lesson 4 Part 1 Relationships between two numerical variables 1 Correlation Coefficient The correlation coefficient is a summary statistic that describes the linear relationship between two numerical variables

### Generating Random Numbers Variance Reduction Quasi-Monte Carlo. Simulation Methods. Leonid Kogan. MIT, Sloan. 15.450, Fall 2010

Simulation Methods Leonid Kogan MIT, Sloan 15.450, Fall 2010 c Leonid Kogan ( MIT, Sloan ) Simulation Methods 15.450, Fall 2010 1 / 35 Outline 1 Generating Random Numbers 2 Variance Reduction 3 Quasi-Monte

### 1 Simple Linear Regression I Least Squares Estimation

Simple Linear Regression I Least Squares Estimation Textbook Sections: 8. 8.3 Previously, we have worked with a random variable x that comes from a population that is normally distributed with mean µ and

### 1. Suppose that a score on a final exam depends upon attendance and unobserved factors that affect exam performance (such as student ability).

Examples of Questions on Regression Analysis: 1. Suppose that a score on a final exam depends upon attendance and unobserved factors that affect exam performance (such as student ability). Then,. When

### A Review of Cross Sectional Regression for Financial Data You should already know this material from previous study

A Review of Cross Sectional Regression for Financial Data You should already know this material from previous study But I will offer a review, with a focus on issues which arise in finance 1 TYPES OF FINANCIAL

### A Primer on Mathematical Statistics and Univariate Distributions; The Normal Distribution; The GLM with the Normal Distribution

A Primer on Mathematical Statistics and Univariate Distributions; The Normal Distribution; The GLM with the Normal Distribution PSYC 943 (930): Fundamentals of Multivariate Modeling Lecture 4: September

### 10.2 Series and Convergence

10.2 Series and Convergence Write sums using sigma notation Find the partial sums of series and determine convergence or divergence of infinite series Find the N th partial sums of geometric series and

### CURVE FITTING LEAST SQUARES APPROXIMATION

CURVE FITTING LEAST SQUARES APPROXIMATION Data analysis and curve fitting: Imagine that we are studying a physical system involving two quantities: x and y Also suppose that we expect a linear relationship

### Getting Correct Results from PROC REG

Getting Correct Results from PROC REG Nathaniel Derby, Statis Pro Data Analytics, Seattle, WA ABSTRACT PROC REG, SAS s implementation of linear regression, is often used to fit a line without checking

### Session 7 Bivariate Data and Analysis

Session 7 Bivariate Data and Analysis Key Terms for This Session Previously Introduced mean standard deviation New in This Session association bivariate analysis contingency table co-variation least squares

### Unit 31 A Hypothesis Test about Correlation and Slope in a Simple Linear Regression

Unit 31 A Hypothesis Test about Correlation and Slope in a Simple Linear Regression Objectives: To perform a hypothesis test concerning the slope of a least squares line To recognize that testing for a

### Week 3&4: Z tables and the Sampling Distribution of X

Week 3&4: Z tables and the Sampling Distribution of X 2 / 36 The Standard Normal Distribution, or Z Distribution, is the distribution of a random variable, Z N(0, 1 2 ). The distribution of any other normal

### Review of Fundamental Mathematics

Review of Fundamental Mathematics As explained in the Preface and in Chapter 1 of your textbook, managerial economics applies microeconomic theory to business decision making. The decision-making tools

### DEPARTMENT OF PSYCHOLOGY UNIVERSITY OF LANCASTER MSC IN PSYCHOLOGICAL RESEARCH METHODS ANALYSING AND INTERPRETING DATA 2 PART 1 WEEK 9

DEPARTMENT OF PSYCHOLOGY UNIVERSITY OF LANCASTER MSC IN PSYCHOLOGICAL RESEARCH METHODS ANALYSING AND INTERPRETING DATA 2 PART 1 WEEK 9 Analysis of covariance and multiple regression So far in this course,

### Maximum Likelihood Estimation

Math 541: Statistical Theory II Lecturer: Songfeng Zheng Maximum Likelihood Estimation 1 Maximum Likelihood Estimation Maximum likelihood is a relatively simple method of constructing an estimator for

### 1 Another method of estimation: least squares

1 Another method of estimation: least squares erm: -estim.tex, Dec8, 009: 6 p.m. (draft - typos/writos likely exist) Corrections, comments, suggestions welcome. 1.1 Least squares in general Assume Y i

### Least Squares Regression. Alan T. Arnholt Department of Mathematical Sciences Appalachian State University arnholt@math.appstate.

Least Squares Regression Alan T. Arnholt Department of Mathematical Sciences Appalachian State University arnholt@math.appstate.edu Spring 2006 R Notes 1 Copyright c 2006 Alan T. Arnholt 2 Least Squares

### Business Statistics. Successful completion of Introductory and/or Intermediate Algebra courses is recommended before taking Business Statistics.

Business Course Text Bowerman, Bruce L., Richard T. O'Connell, J. B. Orris, and Dawn C. Porter. Essentials of Business, 2nd edition, McGraw-Hill/Irwin, 2008, ISBN: 978-0-07-331988-9. Required Computing