# Econometrics Simple Linear Regression

Save this PDF as:

Size: px
Start display at page:

## Transcription

1 Econometrics Simple Linear Regression Burcu Eke UC3M

2 Linear equations with one variable Recall what a linear equation is: y = b 0 + b 1 x is a linear equation with one variable, or equivalently, a straight line. linear on x, we can think this as linear on its unknown parameter, i.e., y = x b 0 and b 1 are constants, b 0 is the y-intercept and b 1 is the slope of the line, y is the dependent variable, and x is the independent variable slope of a line being b 1 means for every 1 unit horizontal increase there is a b 1 unit vertical increase/decrease depending on the sign of b 1. 2

3 Linear equations with one variable A linear equation is of deterministic nature outcomes are precisely determined without any random variation A given input will always produce the same output perfect relationship In real life data, it is almost impossible to have such a prefect relationship between two variables. We almost always rely on rough predictions. One of our tools to do so is regression. 3

4 Empirical and Theoretical Relationships Economists are interested in the relationship between two or more economic variables at least bivariate populations The economic theory in general suggests relationships in functional forms (recall economic models). These relations are deterministic, such as Y = f(x), or Y = f (X 1, X 2,...X k ) A given input will always produce the same output perfect relationship 4

5 Example 1: Study time and score Let s consider the following example. The table below shows the data for total hours studied for a calculus test, x, and the test scores, y. time score x y

6 Example 1, cont d Consider the scatter plot based on the data of example 1. Does it look like a perfect relationship? 95 Study time vs Exam score

7 Example 2: Savings and income Relationship between savings (Y) and income (X) (Goldberger, 1991) Data from 1027 families, between , in the USA The joint distribution of savings and income are presented in the next table P (Y, X) For discrete case, P (Y, X) P (Y = y and X = x). 7

8 Example 2, cont d Table: Joint Distribution Y, savings, and X, income: P (Y, X) X (in 1000) Y P (Y ) P (X)

9 Example 2, cont d Among the information we can obtain from a joint probability table, there are two that are of interest to us: whether we can have a deterministic relationship, i.e., Y = f(x) whether savings and income are independent Can we have a deterministic relationship between savings and income based on the previous table? No. In order to have a deterministic relationship, we need to have a unique savings for each level of income, in other words we need to have a functional relationship. In terms of probability, for each income level, we need to have only one savings level with positive probability! 9

10 Example 2, cont d But we do have a relationship between income level and savings As income increases, savings level increases. To further investigate this, lets calculate the conditional distribution: P (Y, X) P (Y X) = P (X) ˆµ Y X=x = yp (Y = y X = x) y 10

11 Example 2, cont d Table: Conditional Distribution of Y, savings, given X, income: P (Y X) X (in 1000) Y Column sum ˆµ Y X What is the relationship between the conditional mean and income level? 11

12 Empirical and Theoretical Relationships The empirical relationships between economic variables not deterministic, but stochastic To combine theory and data, one must interpret the economic theory in a different way When the economic theory postulates that Y is a function of X, Y = f(x), it implies that the expected value of Y is a function of X, E[Y ] = f(x) Y = f(x) deterministic, E[Y ] = f(x) stochastic 12

13 Prediction When we are in a stochastic setting, we are in general interested in prediction, but how do we form our prediction? One way would be just to guess a number, but do you think it will be good? How can we assess that it is good? We are interested in finding the best way of predicting 13

14 Prediction Suppose we want to predict Y, using the information on X. Let c(x) be a prediction Then, U = Y c(x) will be our prediction error. We want the expected value of the square of our prediction, E[U 2 ] error to be as small as possible Why E[U 2 ]? Choose a prediction so that E[U 2 ] = (y c(x)) 2 P (y) y Y is minimized 14

15 Prediction: The best constant prediction We do not always have bivariate, or multivariate case, sometimes we only have information on Y In these cases, our prediction does not depend on other variables, so our prediction can be denoted as c Then we choose a prediction such that we minimize E[U 2 ] = y Y (y c) 2 P (y) c = E[Y ] = µ Y Average of Y is the best constant predictor 15

16 Exercise 2, cont d Y (Saving rate) P (Y ) Then the best constant predictor for the saving rate is: E[Y ] = = = 9.85% 16

17 Prediction: The best linear prediction In the case that we have bivariate data (Y, X), say savings rate and income levels, then we can make use of the relationship between them to predict Y Linear prediction implies that our c(x) is linear, i.e., c(x) = c 0 + c 1 X So, we need to find c 0 and c 1 such that E[U 2 ] = (y c(x)) 2 P (y) is minimized y Y 17

18 Prediction: The best linear prediction By replacing c(x) by c 0 + c 1 X, and solving for the minimization problem we get c 0 = α 0 = E(Y ) α 1 E(X) = µ Y α 1 µ X c 1 = α 1 = C(X, Y ) V (X) = σ XY σ 2 X The function α 0 + α 1 X is the linear projection (or the best linear prediction) of Y given X L(Y X) = α 0 + α 1 X 18

19 Example 2, cont d To get the best linear prediction for savings rate, we need to calculate: E[X], E[Y ], E[XY ], C(X, Y ), E[X 2 ], V (X) For E[XY ]: XY P(XY) XY P(XY) XY P(XY) XY P(XY) E[XY ] = X i Y j P (XY = X i Y j ) = i=1 j=1 19

20 Example 2, cont d E[X] = = C(X, Y ) = E[XY ] E[X]E[Y ] = = E[X 2 ] = = V (X) = E[X 2 ] (E[X]) 2 = = c 0 = α 0 = E(Y ) α 1 E(X) = c 1 = α 1 = C(X, Y ) = V (X) 20

21 Example 2, cont d L(Y X) = X The for discrete values of X, we get = if X = = if X = 4.9 L(Y X) = = if X = = if X =

22 Best Prediction So far we considered best constant predictor, and best linear predictor Let s relax the linearity restriction on c(x), i.e., c(x) can be any function that minimizes E[U 2 ] The best predictor of Y then becomes the conditional expected value of Y, E[Y X] If the E[Y X] is linear, then E[Y X] and L(Y X) are the same. The reverse is NOT true!!! If the E[Y X] is NOT linear, then L(Y X) is the best linear approximation to E[Y X] 22

23 Example 2, cont d X (income) c L(Y X) E[Y X] (in \$1000s) BCP BLP BP

24 Prediction Whenever L(Y X) E[Y X], L(Y X) provides a good approximation to E[Y X], hence can be used in some circumstances However, E[Y X] characterizes conditional mean of Y given X, the L(Y X) does not E[Y X] can have causal interpretation, the L(Y X) can not 24

25 SIMPLE LINEAR REGRESSION (SLR) 25

26 Simple Linear Regression Our big goal to analyze and study the relationship between two variables One approach to achieve this is simple linear regression, i.e, Y = β 0 + β 1 X + ε While answering our question, a simple linear regression model addresses some issues: 1. How to deal with the factors other than X that effects Y 2. How to formulate the relationship between X and Y 3. Whether our model captures a ceteris paribus relationship between X and Y 26

27 Simple Linear Regression SLR Model: Y = β 0 + β 1 X + ε Y Dependent variable, endogenous variable, response variable, regressand... X Independent variable, exogenous variable, control variable, regressor... β = (β 0, β 1 ) Parameter vector, population parameters ε Error term 27

28 Simple Linear Regression: Assumptions A1: Linear in parameters It implies that a unit change in X has the same effect on Y, independently of the initial value of X. SLR is linear in parameters: The following are linear in parameters: Y = β 0 + β 1 X + ε, Y = β 0 X 2 Z + β 1 logz + ε The following is NOT linear in parameters: Y = β 1 X β2 ε Violations of this assumption are form of a specification errors, such as nonlinearity (when the relationship is not linear in parameters) We can overcome the limitations this assumption imposes 28

29 Simple Linear Regression: Assumptions A2: E[ε X] = 0 For the all values of X, the mean error rate is same and equal to 0. E[ε X] = 0 implies for any function h(), C(h(X), ε) = 0. Hence, ε is not correlated with any function of X In particular, C(X, ε) = 0 C(X, ε) = E[Xε] E[X]E[ε] E[Xε] = E[E[Xε X]] = E[XE[ε X]] = 0 E(X)E(ε) = 0 29

30 Simple Linear Regression: Assumptions E[Y X] = β 0 + β 1 X, the conditional expectation function is linear, hence C(Y, X) C(Y, X) = C (E[Y X], X) = β 1 V (X) β 1 = V (X) E[Y ] = E [E[Y X]] = β 0 + β 1 E[X] β 0 = E[Y ] β 1 E[X] E[Y X] = L(Y X) = β β 1 X, where L(.) is the linear projection 30

31 Simple Linear Regression: Assumptions A3: Conditional Homoscedasticity V (ε X) = σ 2 X This implies V (ε) = σ 2 V (ε X) = E[ε 2 X] E [(E[ε X])] = E[ε 2 X] = σ 2 E[ε 2 ] = E [ E[ε 2 X] ] = σ 2 V (ε) = E[ε 2 ] (E[ε]) 2 = σ 2 This also implies V (Y X) = σ 2 31

32 Simple Linear Regression: Interpretation of Coefficients From A1, we have Y = β 0 + β 1 X + ε, if we take the conditional expectation of both sides we get E[Y X] = E [β 0 + β 1 X + ε] = β 0 + β 1 E[X X] + E[ε X](by A2) = β 0 + β 1 X E[Y X] This implies: β 1 =. That is, When X increases X by one unit, Y, on average, increases by β 1 units. 32

33 Simple Linear Regression: Interpretation of Coefficients E[Y X = 0] = E [β 0 + β 1 X + ε X = 0] = β 0 β 0 is the mean value of Y, when X = 0 β 0 is the intercept term In the case that X = 0 is not included in the range of X in our data set, β 0 has no interpretation 33

34 Simple Linear Regression: Interpretation of Coefficients Notice that A2 also implies E[Y X] = β 0 + β 1 X = L(Y X), that is, the best prediction is the best linear prediction If A2 is not satisfied, then E[Y X] L(Y X). This implies: β 0 and β 1 are only the parameters of a linear projection, NOT of a conditional expectation β 0 and β 1 do not have causal interpretations 34

35 Simple Linear Regression: Example 3 Scatterplot - Two Variables salary age 35

36 Simple Linear Regression: Example 3 The regression equation: Y = X (Assume that all assumptions are met) What is the interpretation of 13.10? Can we predict mean salary of a 0 years old person? Why? Why not? What is the interpretation of ? 36

37 SIMPLE LINEAR REGRESSION: THE ESTIMATION 37

38 Simple Linear Regression: Estimation Our Objective: Estimate the population parameters, i.e., β = (β 0, β 1 ) and σ 2 Our Model: Y = β 0 + β 1 X + ε, where E[ε X] = 0 and V (ε X) = σ 2 Our Data: Pairs of (y i, x i ) i = 1, 2,..., n are the sample realizations of population data (Y i, X i ): y x y 1 x 1 y 2 x y n x n 38

39 Simple Linear Regression: Estimation We need a method to estimate our population parameters β = (β 0, β 1 ) and σ 2, using our model Y i = β 0 + β 1 X i + ε i for all i = 1, 2,..., n We need to distinguish between population parameters β = (β 0, β 1 ) and σ 2 and the estimated values, denoted as ˆβ = ( ˆβ 0, ˆβ 1 ) and ˆσ 2 39

40 Estimation: The Analogy Principle An approach for estimation Use the corresponding sample quantity as an estimate of the population quantity Use sample mean Ȳ = mean n i=1 Use sample variance s 2 = population variance y i n i=1 to estimate the population ( n yi Ȳ ) 2 to estimate the n 1 Method of moments: Let µ i = E[X i ] be the i th moment, the by method of moments, we estimate µ i by n (x k ) i corresponding sample moment m i = n k=1 40

41 Estimation: The Analogy Principle Recall the bets prediction and the best linear prediction. Under our assumptions: E[Y X] = L(Y X) = β 0 + β 1 X β 0 = E(Y ) β 1 E(X) β 1 = C(X, Y ) V (X) 41

42 Estimation: The Analogy Principle then, by using the analogy principle we have: ˆβ 0 = Ȳ ˆβ 1 X ˆβ 1 = n i (X i X)(Y i Ȳ ) n i (X i X) 2 = 1 n n i (X i X)Y i n i (X i X) 2 1 n = S XY S 2 X 42

43 Estimation: The Ordinary Least Squares (OLS) The idea of the (OLS) principle is to choose parameter estimates that minimize the squared error term, i.e., the distance between the data and the model min E[ε 2 ], or equivalently, min E [(Y β 0 + β 1 X) 2] ε i = Y i E[Y i X i ] 43

44 The Ordinary Least Squares (OLS) For the sample data, we define the error term as ˆε i = Y i Ŷi = Y i ˆβ 0 ˆβ 1 X i Then the sample equivalent of min E[ε 2 ] is 1 min β 0,β 1 n n i=1 ˆε 2 i 44

45 The Ordinary Least Squares (OLS) FOC are: n i=1 ˆε = 0 n i=1 ˆεX i = 0 These FOC implies: ˆβ 0 = Ȳ ˆβ 1 X ˆβ 1 = n i=1 x iy i n i=1 x2 i c i = x i n i=1 x2 i n = c i Y i where x i = (X i X) and i=1 45

46 The Ordinary Least Squares (OLS) Compare Least square estimators and analogy principle estimators. are the similar? Predicted values: Ŷi = ˆβ 0 ˆβ 1 X i Predicted values are orthogonal to the predicted error term i Ŷiˆε i = 0 46

47 Estimation Terminology Estimator is a RULE for calculating an estimate of a parameter based on observed data, i.e., ˆβ 0 = Ȳ ˆβ 1 X Estimate is a number that is an outcome of an estimation process, i.e, ˆβ 1 =

48 The Properties of The OLS Estimators P1 Linear in Y ˆβ0 = Ȳ ˆβ 1 X ˆβ1 = i c iy i 48

49 The Properties of The OLS Estimators P2 Unbiasedness: Expected value of the estimator is equal to the parameter itself We achieve this property by A1 and A2 E[ ˆβ 0 ] = β 0 E[ ˆβ 1 ] = β 1 49

50 The Properties of The OLS Estimators The unbiasedness property indicates that if we have infinitely many samples of size-n from the same population and estimate the same model with each of the samples: We will have a distribution of the estimator of β l, with a different realization for each sample The mean of that distribution will be the the population parameter β l 50

51 The Properties of The OLS Estimators E[ ˆβ 1 ] = β 1 ˆβ 1 = i c i Y i = c i (β 0 + β 1 X i + ε i ) i = β 0 c i + β 1 c i X i + c i ε i i i i = β 1 c i X i + ( c i ε i because ) c i = 0 i i i = β 1 + ( c i ε i because ) c i X i = 1 i i 51

52 The Properties of The OLS Estimators E[ ˆβ 1 X] = E [ β 1 + i c i ε i X ] = β 1 + i = β 1 E[β 1 ] = E [E[β 1 X]] = E[β 1 ] = β 1 c i E[ε i X] } {{ } 0 52

53 The Properties of The OLS Estimators The Variance From A1, A2, and A3, we have V ( ˆβ 0 ) = i X 2 i n V ( ˆβ 1 ) [ ] V ( ˆβ 1 ) = σ2 1 n E SX 2 53

54 The Properties of The OLS Estimators Let s show that V ( ˆβ 1 ) = σ2 n E [ ( V ( ˆβ ) ] 2 1 ) = E ˆβ1 β 1 [ 1 S 2 X ] ( = E i ) 2 c i ε i = i [ = E E [ c 2 i ε 2 i X ]] [ = E c 2 i E [ ε 2 i X ]] = σ 2 E = σ 2 E = σ2 n E i [ i [ i [ 1 S 2 X c 2 i ] ( xi ] (by A3) i x2 i ) 2 ] 54 i = σ 2 E [ i 1 n 1 n ( i x2 i )2 E[c 2 i ε 2 i ] ]

55 The Properties of The OLS Estimators: Gauss-Markov Theorem Under assumptions A1 to A3, the Gauss-Markov Theorem states that the ˆβ 0 and ˆβ 0 are the efficient estimators among all linear unbiased estimators ˆβ0 and ˆβ 0 are the most efficient estimators among all the linear unbiased estimators Best Linear Unbiased Estimator 55

56 The Properties of The OLS Estimators: Gauss-Markov Theorem What does it mean to be a consistent estimator? ˆβ 0 and ˆβ 1 are consistent if in other words, p lim ˆβ j = β j, j = 0, 1 n ( ) lim P ˆβj β j < δ = 1 δ > 0 n Consistency means the distributions of the estimators ˆβ 0 and ˆβ 1 become more and more concentrated near the true values β 0 and β 1 as sample size increases, so that the probability of ˆβ 0 and ˆβ 1 being arbitrarily close to β 0 and β 1 converges to one For these results, we need A1 to A3 to met!!! 56

57 The Variance of The OLS Estimators The variance of the OLS estimators, ˆβ 0 and ˆβ 1, depends on σ 2 = V (ε) = E[ε 2 ] The problem is, the errors ε = ε 1, ε 2,..., ε n are not observed, so we need to estimate ε ˆε = Y ˆβ 0 ˆβ 1 X = β 0 + β 1 X + ε ˆβ 0 ˆβ 1 X ( ) ( ) = ε ˆβ0 β 0 ˆβ1 β 1 X Even though E[ ˆβ j ] = β j for j = 0, 1, ˆε ε 57

58 The Variance of The OLS Estimators One attempt to estimate σ 2 would be to use (1\n) ε 2 instead of E[ε 2 ]. However, this is not feasible Another attempt to estimate σ 2 would be using sample values of the error, ˆε, and use this to estimate the sample variance: σ 2 = (1\n) ˆε 2. However, this is biased. The bias comes from unmatched degrees of freedom and denominator When we account for this bias, we get a consistent and unbiased estimate: ˆσ 2 = i ˆε 2 i n 2 58

59 The Variance of The OLS Estimators Note that both of ˆσ 2 and σ 2 are consistent and for larger samples, their value is very close to each other. [ ] Recall that V ( ˆβ 1 ) = σ2 1 n E SX 2 [ ] We estimate V ( ˆβ 1 ) by ˆσ 2 ˆV ( ˆβ 1 ) = ˆσ2 1 nsx 2, where E SX 2 is estimated by 1 SX 2 Recall V ( ˆβ 0 ) = i X 2 i n V ( ˆβ 1 ) We estimate V ( ˆβ 0 ) by ˆσ 2 ˆV ( ˆβ 0 ) = ˆσ2 i X2 i n 2 S 2 X 59

60 The Goodness of Fit Measures The goodness of fit measures are the measures of how well our model fits the data. Standard error of regression, ˆσ, is one intuitive way of approaching goodness of fit. Recall that in the standard simple linear regression, we have Y = β 0 + β 1 X + ε and V (ε X) = σ 2. One problem with this goodness of fit method is that its magnitude heavily depends on the units of Y 60

61 The Goodness of Fit Measures Coefficient of Determination, R 2, is a very common measure of goodness of fit R 2 = i ŷ2 i = 1 i ε2 i (1) i y2 i i y2 i where y i = Y i Ȳ, ŷ i = Ŷi Ȳ, and ε i = Y i Ŷi Notice that y i = ŷ i + ε i. This implies that 0 i ŷ2 i 1 If R 2 = 1, the model explains all of the variation (100%) in Y If R 2 = 0, the model explains none of the variation (0%) in Y i y2 i 61

62 The Goodness of Fit Measures A low R 2 implies: The model fails to explain a significant proportion of the variation of Y. A low R2 does not necessarily mean that the estimates are unhelpful or inappropriate. The R 2 can be used for comparing different models for the same dependent variable Y. Also, for the correlation coefficient between actual and estimated values of Y, ρ Y, Ŷ, we ahe the following ( ) 2 relationship: R 2 = ˆρ Y, Ŷ 62

### e = random error, assumed to be normally distributed with mean 0 and standard deviation σ

1 Linear Regression 1.1 Simple Linear Regression Model The linear regression model is applied if we want to model a numeric response variable and its dependency on at least one numeric factor variable.

### 2. Linear regression with multiple regressors

2. Linear regression with multiple regressors Aim of this section: Introduction of the multiple regression model OLS estimation in multiple regression Measures-of-fit in multiple regression Assumptions

Lecture 5: Linear least-squares Regression III: Advanced Methods William G. Jacoby Department of Political Science Michigan State University http://polisci.msu.edu/jacoby/icpsr/regress3 Simple Linear Regression

### Simultaneous Equation Models As discussed last week, one important form of endogeneity is simultaneity. This arises when one or more of the

Simultaneous Equation Models As discussed last week, one important form of endogeneity is simultaneity. This arises when one or more of the explanatory variables is jointly determined with the dependent

### Heteroskedasticity and Weighted Least Squares

Econ 507. Econometric Analysis. Spring 2009 April 14, 2009 The Classical Linear Model: 1 Linearity: Y = Xβ + u. 2 Strict exogeneity: E(u) = 0 3 No Multicollinearity: ρ(x) = K. 4 No heteroskedasticity/

### Definition The covariance of X and Y, denoted by cov(x, Y ) is defined by. cov(x, Y ) = E(X µ 1 )(Y µ 2 ).

Correlation Regression Bivariate Normal Suppose that X and Y are r.v. s with joint density f(x y) and suppose that the means of X and Y are respectively µ 1 µ 2 and the variances are 1 2. Definition The

### Lecture 3: Linear methods for classification

Lecture 3: Linear methods for classification Rafael A. Irizarry and Hector Corrada Bravo February, 2010 Today we describe four specific algorithms useful for classification problems: linear regression,

### 17.0 Linear Regression

17.0 Linear Regression 1 Answer Questions Lines Correlation Regression 17.1 Lines The algebraic equation for a line is Y = β 0 + β 1 X 2 The use of coordinate axes to show functional relationships was

### Chapter 11: Two Variable Regression Analysis

Department of Mathematics Izmir University of Economics Week 14-15 2014-2015 In this chapter, we will focus on linear models and extend our analysis to relationships between variables, the definitions

### 1. The Classical Linear Regression Model: The Bivariate Case

Business School, Brunel University MSc. EC5501/5509 Modelling Financial Decisions and Markets/Introduction to Quantitative Methods Prof. Menelaos Karanasos (Room SS69, Tel. 018956584) Lecture Notes 3 1.

### Lecture 2: Simple Linear Regression

DMBA: Statistics Lecture 2: Simple Linear Regression Least Squares, SLR properties, Inference, and Forecasting Carlos Carvalho The University of Texas McCombs School of Business mccombs.utexas.edu/faculty/carlos.carvalho/teaching

### IAPRI Quantitative Analysis Capacity Building Series. Multiple regression analysis & interpreting results

IAPRI Quantitative Analysis Capacity Building Series Multiple regression analysis & interpreting results How important is R-squared? R-squared Published in Agricultural Economics 0.45 Best article of the

### The Scalar Algebra of Means, Covariances, and Correlations

3 The Scalar Algebra of Means, Covariances, and Correlations In this chapter, we review the definitions of some key statistical concepts: means, covariances, and correlations. We show how the means, variances,

### Simple Linear Regression Inference

Simple Linear Regression Inference 1 Inference requirements The Normality assumption of the stochastic term e is needed for inference even if it is not a OLS requirement. Therefore we have: Interpretation

### 4. Joint Distributions of Two Random Variables

4. Joint Distributions of Two Random Variables 4.1 Joint Distributions of Two Discrete Random Variables Suppose the discrete random variables X and Y have supports S X and S Y, respectively. The joint

### Covariance and Correlation

Covariance and Correlation ( c Robert J. Serfling Not for reproduction or distribution) We have seen how to summarize a data-based relative frequency distribution by measures of location and spread, such

### Outline. Topic 4 - Analysis of Variance Approach to Regression. Partitioning Sums of Squares. Total Sum of Squares. Partitioning sums of squares

Topic 4 - Analysis of Variance Approach to Regression Outline Partitioning sums of squares Degrees of freedom Expected mean squares General linear test - Fall 2013 R 2 and the coefficient of correlation

### Chapter 3: The Multiple Linear Regression Model

Chapter 3: The Multiple Linear Regression Model Advanced Econometrics - HEC Lausanne Christophe Hurlin University of Orléans November 23, 2013 Christophe Hurlin (University of Orléans) Advanced Econometrics

### Regression in SPSS. Workshop offered by the Mississippi Center for Supercomputing Research and the UM Office of Information Technology

Regression in SPSS Workshop offered by the Mississippi Center for Supercomputing Research and the UM Office of Information Technology John P. Bentley Department of Pharmacy Administration University of

### Introduction to Regression and Data Analysis

Statlab Workshop Introduction to Regression and Data Analysis with Dan Campbell and Sherlock Campbell October 28, 2008 I. The basics A. Types of variables Your variables may take several forms, and it

### Simple Linear Regression Chapter 11

Simple Linear Regression Chapter 11 Rationale Frequently decision-making situations require modeling of relationships among business variables. For instance, the amount of sale of a product may be related

### Numerical Summarization of Data OPRE 6301

Numerical Summarization of Data OPRE 6301 Motivation... In the previous session, we used graphical techniques to describe data. For example: While this histogram provides useful insight, other interesting

### DEPARTMENT OF ECONOMICS. Unit ECON 12122 Introduction to Econometrics. Notes 4 2. R and F tests

DEPARTMENT OF ECONOMICS Unit ECON 11 Introduction to Econometrics Notes 4 R and F tests These notes provide a summary of the lectures. They are not a complete account of the unit material. You should also

### , then the form of the model is given by: which comprises a deterministic component involving the three regression coefficients (

Multiple regression Introduction Multiple regression is a logical extension of the principles of simple linear regression to situations in which there are several predictor variables. For instance if we

### AMS7: WEEK 8. CLASS 1. Correlation Monday May 18th, 2015

AMS7: WEEK 8. CLASS 1 Correlation Monday May 18th, 2015 Type of Data and objectives of the analysis Paired sample data (Bivariate data) Determine whether there is an association between two variables This

### Chapter 13 Introduction to Nonlinear Regression( 非 線 性 迴 歸 )

Chapter 13 Introduction to Nonlinear Regression( 非 線 性 迴 歸 ) and Neural Networks( 類 神 經 網 路 ) 許 湘 伶 Applied Linear Regression Models (Kutner, Nachtsheim, Neter, Li) hsuhl (NUK) LR Chap 10 1 / 35 13 Examples

### Joint Exam 1/P Sample Exam 1

Joint Exam 1/P Sample Exam 1 Take this practice exam under strict exam conditions: Set a timer for 3 hours; Do not stop the timer for restroom breaks; Do not look at your notes. If you believe a question

### Elementary Statistics. Scatter Plot, Regression Line, Linear Correlation Coefficient, and Coefficient of Determination

Scatter Plot, Regression Line, Linear Correlation Coefficient, and Coefficient of Determination What is a Scatter Plot? A Scatter Plot is a plot of ordered pairs (x, y) where the horizontal axis is used

### Part 2: Analysis of Relationship Between Two Variables

Part 2: Analysis of Relationship Between Two Variables Linear Regression Linear correlation Significance Tests Multiple regression Linear Regression Y = a X + b Dependent Variable Independent Variable

### 17. SIMPLE LINEAR REGRESSION II

17. SIMPLE LINEAR REGRESSION II The Model In linear regression analysis, we assume that the relationship between X and Y is linear. This does not mean, however, that Y can be perfectly predicted from X.

### MULTIVARIATE PROBABILITY DISTRIBUTIONS

MULTIVARIATE PROBABILITY DISTRIBUTIONS. PRELIMINARIES.. Example. Consider an experiment that consists of tossing a die and a coin at the same time. We can consider a number of random variables defined

### Variance of OLS Estimators and Hypothesis Testing. Randomness in the model. GM assumptions. Notes. Notes. Notes. Charlie Gibbons ARE 212.

Variance of OLS Estimators and Hypothesis Testing Charlie Gibbons ARE 212 Spring 2011 Randomness in the model Considering the model what is random? Y = X β + ɛ, β is a parameter and not random, X may be

### Regression Analysis Prof. Soumen Maity Department of Mathematics Indian Institute of Technology, Kharagpur. Lecture - 2 Simple Linear Regression

Regression Analysis Prof. Soumen Maity Department of Mathematics Indian Institute of Technology, Kharagpur Lecture - 2 Simple Linear Regression Hi, this is my second lecture in module one and on simple

### Example: Boats and Manatees

Figure 9-6 Example: Boats and Manatees Slide 1 Given the sample data in Table 9-1, find the value of the linear correlation coefficient r, then refer to Table A-6 to determine whether there is a significant

### Bivariate Regression Analysis. The beginning of many types of regression

Bivariate Regression Analysis The beginning of many types of regression TOPICS Beyond Correlation Forecasting Two points to estimate the slope Meeting the BLUE criterion The OLS method Purpose of Regression

### Regression Analysis. Pekka Tolonen

Regression Analysis Pekka Tolonen Outline of Topics Simple linear regression: the form and estimation Hypothesis testing and statistical significance Empirical application: the capital asset pricing model

### SYSM 6304: Risk and Decision Analysis Lecture 3 Monte Carlo Simulation

SYSM 6304: Risk and Decision Analysis Lecture 3 Monte Carlo Simulation M. Vidyasagar Cecil & Ida Green Chair The University of Texas at Dallas Email: M.Vidyasagar@utdallas.edu September 19, 2015 Outline

### Covariance and Correlation. Consider the joint probability distribution f XY (x, y).

Chapter 5: JOINT PROBABILITY DISTRIBUTIONS Part 2: Section 5-2 Covariance and Correlation Consider the joint probability distribution f XY (x, y). Is there a relationship between X and Y? If so, what kind?

### Lecture 5: Correlation and Linear Regression

Lecture 5: Correlation and Linear Regression 3.5. (Pearson) correlation coefficient The correlation coefficient measures the strength of the linear relationship between two variables. The correlation is

### where b is the slope of the line and a is the intercept i.e. where the line cuts the y axis.

Least Squares Introduction We have mentioned that one should not always conclude that because two variables are correlated that one variable is causing the other to behave a certain way. However, sometimes

### Regression Analysis: Basic Concepts

The simple linear model Regression Analysis: Basic Concepts Allin Cottrell Represents the dependent variable, y i, as a linear function of one independent variable, x i, subject to a random disturbance

### The aspect of the data that we want to describe/measure is the degree of linear relationship between and The statistic r describes/measures the degree

PS 511: Advanced Statistics for Psychological and Behavioral Research 1 Both examine linear (straight line) relationships Correlation works with a pair of scores One score on each of two variables ( and

### 2. Simple Linear Regression

Research methods - II 3 2. Simple Linear Regression Simple linear regression is a technique in parametric statistics that is commonly used for analyzing mean response of a variable Y which changes according

### Module 3: Correlation and Covariance

Using Statistical Data to Make Decisions Module 3: Correlation and Covariance Tom Ilvento Dr. Mugdim Pašiƒ University of Delaware Sarajevo Graduate School of Business O ften our interest in data analysis

### Univariate Regression

Univariate Regression Correlation and Regression The regression line summarizes the linear relationship between 2 variables Correlation coefficient, r, measures strength of relationship: the closer r is

### 2. What are the theoretical and practical consequences of autocorrelation?

Lecture 10 Serial Correlation In this lecture, you will learn the following: 1. What is the nature of autocorrelation? 2. What are the theoretical and practical consequences of autocorrelation? 3. Since

### Chapter 7: Simple linear regression Learning Objectives

Chapter 7: Simple linear regression Learning Objectives Reading: Section 7.1 of OpenIntro Statistics Video: Correlation vs. causation, YouTube (2:19) Video: Intro to Linear Regression, YouTube (5:18) -

### 1 OLS under Measurement Error

ECON 370: Measurement Error 1 Violations of Gauss Markov Assumptions: Measurement Error Econometric Methods, ECON 370 1 OLS under Measurement Error We have found out that another source of endogeneity

### Chapter 9. Section Correlation

Chapter 9 Section 9.1 - Correlation Objectives: Introduce linear correlation, independent and dependent variables, and the types of correlation Find a correlation coefficient Test a population correlation

### 12/31/2016. PSY 512: Advanced Statistics for Psychological and Behavioral Research 2

PSY 512: Advanced Statistics for Psychological and Behavioral Research 2 Understand linear regression with a single predictor Understand how we assess the fit of a regression model Total Sum of Squares

### Simple linear regression

Simple linear regression Introduction Simple linear regression is a statistical method for obtaining a formula to predict values of one variable from another where there is a causal relationship between

### Overview of Violations of the Basic Assumptions in the Classical Normal Linear Regression Model

Overview of Violations of the Basic Assumptions in the Classical Normal Linear Regression Model 1 September 004 A. Introduction and assumptions The classical normal linear regression model can be written

### Clustering in the Linear Model

Short Guides to Microeconometrics Fall 2014 Kurt Schmidheiny Universität Basel Clustering in the Linear Model 2 1 Introduction Clustering in the Linear Model This handout extends the handout on The Multiple

### Stochastic Inventory Control

Chapter 3 Stochastic Inventory Control 1 In this chapter, we consider in much greater details certain dynamic inventory control problems of the type already encountered in section 1.3. In addition to the

### Basic Statistcs Formula Sheet

Basic Statistcs Formula Sheet Steven W. ydick May 5, 0 This document is only intended to review basic concepts/formulas from an introduction to statistics course. Only mean-based procedures are reviewed,

### Solución del Examen Tipo: 1

Solución del Examen Tipo: 1 Universidad Carlos III de Madrid ECONOMETRICS Academic year 2009/10 FINAL EXAM May 17, 2010 DURATION: 2 HOURS 1. Assume that model (III) verifies the assumptions of the classical

### Basic Statistics and Data Analysis for Health Researchers from Foreign Countries

Basic Statistics and Data Analysis for Health Researchers from Foreign Countries Volkert Siersma siersma@sund.ku.dk The Research Unit for General Practice in Copenhagen Dias 1 Content Quantifying association

### PASS Sample Size Software. Linear Regression

Chapter 855 Introduction Linear regression is a commonly used procedure in statistical analysis. One of the main objectives in linear regression analysis is to test hypotheses about the slope (sometimes

### Correlation in Random Variables

Correlation in Random Variables Lecture 11 Spring 2002 Correlation in Random Variables Suppose that an experiment produces two random variables, X and Y. What can we say about the relationship between

### NCSS Statistical Software Principal Components Regression. In ordinary least squares, the regression coefficients are estimated using the formula ( )

Chapter 340 Principal Components Regression Introduction is a technique for analyzing multiple regression data that suffer from multicollinearity. When multicollinearity occurs, least squares estimates

### Estimation of σ 2, the variance of ɛ

Estimation of σ 2, the variance of ɛ The variance of the errors σ 2 indicates how much observations deviate from the fitted surface. If σ 2 is small, parameters β 0, β 1,..., β k will be reliably estimated

### Financial Risk Management Exam Sample Questions/Answers

Financial Risk Management Exam Sample Questions/Answers Prepared by Daniel HERLEMONT 1 2 3 4 5 6 Chapter 3 Fundamentals of Statistics FRM-99, Question 4 Random walk assumes that returns from one time period

### Summary of Formulas and Concepts. Descriptive Statistics (Ch. 1-4)

Summary of Formulas and Concepts Descriptive Statistics (Ch. 1-4) Definitions Population: The complete set of numerical information on a particular quantity in which an investigator is interested. We assume

### 5. Linear Regression

5. Linear Regression Outline.................................................................... 2 Simple linear regression 3 Linear model............................................................. 4

### Regression Analysis Prof. Soumen Maity Department of Mathematics Indian Institute of Technology, Kharagpur

Regression Analysis Prof. Soumen Maity Department of Mathematics Indian Institute of Technology, Kharagpur Lecture - 7 Multiple Linear Regression (Contd.) This is my second lecture on Multiple Linear Regression

### Class 6: Chapter 12. Key Ideas. Explanatory Design. Correlational Designs

Class 6: Chapter 12 Correlational Designs l 1 Key Ideas Explanatory and predictor designs Characteristics of correlational research Scatterplots and calculating associations Steps in conducting a correlational

### Regression. Name: Class: Date: Multiple Choice Identify the choice that best completes the statement or answers the question.

Class: Date: Regression Multiple Choice Identify the choice that best completes the statement or answers the question. 1. Given the least squares regression line y8 = 5 2x: a. the relationship between

### Sales forecasting # 2

Sales forecasting # 2 Arthur Charpentier arthur.charpentier@univ-rennes1.fr 1 Agenda Qualitative and quantitative methods, a very general introduction Series decomposition Short versus long term forecasting

### An Introduction to Time Series Regression

An Introduction to Time Series Regression Henry Thompson Auburn University An economic model suggests examining the effect of exogenous x t on endogenous y t with an exogenous control variable z t. In

### OLS in Matrix Form. Let y be an n 1 vector of observations on the dependent variable.

OLS in Matrix Form 1 The True Model Let X be an n k matrix where we have observations on k independent variables for n observations Since our model will usually contain a constant term, one of the columns

### UNDERSTANDING MULTIPLE REGRESSION

UNDERSTANDING Multiple regression analysis (MRA) is any of several related statistical methods for evaluating the effects of more than one independent (or predictor) variable on a dependent (or outcome)

### CHAPTER 13 SIMPLE LINEAR REGRESSION. Opening Example. Simple Regression. Linear Regression

Opening Example CHAPTER 13 SIMPLE LINEAR REGREION SIMPLE LINEAR REGREION! Simple Regression! Linear Regression Simple Regression Definition A regression model is a mathematical equation that descries the

### ECON 142 SKETCH OF SOLUTIONS FOR APPLIED EXERCISE #2

University of California, Berkeley Prof. Ken Chay Department of Economics Fall Semester, 005 ECON 14 SKETCH OF SOLUTIONS FOR APPLIED EXERCISE # Question 1: a. Below are the scatter plots of hourly wages

### Hypothesis testing - Steps

Hypothesis testing - Steps Steps to do a two-tailed test of the hypothesis that β 1 0: 1. Set up the hypotheses: H 0 : β 1 = 0 H a : β 1 0. 2. Compute the test statistic: t = b 1 0 Std. error of b 1 =

### Simultaneous Equations

Simultaneous Equations 1 Motivation and Examples Now we relax the assumption that EX u 0 his wil require new techniques: 1 Instrumental variables 2 2- and 3-stage least squares 3 Limited LIML and full

### CORRELATION ANALYSIS

CORRELATION ANALYSIS Learning Objectives Understand how correlation can be used to demonstrate a relationship between two factors. Know how to perform a correlation analysis and calculate the coefficient

### Yiming Peng, Department of Statistics. February 12, 2013

Regression Analysis Using JMP Yiming Peng, Department of Statistics February 12, 2013 2 Presentation and Data http://www.lisa.stat.vt.edu Short Courses Regression Analysis Using JMP Download Data to Desktop

### Chapter 4: Vector Autoregressive Models

Chapter 4: Vector Autoregressive Models 1 Contents: Lehrstuhl für Department Empirische of Wirtschaftsforschung Empirical Research and und Econometrics Ökonometrie IV.1 Vector Autoregressive Models (VAR)...

### AN INTRODUCTION TO ECONOMETRICS. Oxbridge Economics; Mo Tanweer

AN INTRODUCTION TO ECONOMETRICS Oxbridge Economics; Mo Tanweer Mohammed.Tanweer@cantab.net Econometrics What is econometrics? Econometrics means economic measurement Economics + Statistics = Econometrics

### Topic 4: Multivariate random variables. Multiple random variables

Topic 4: Multivariate random variables Joint, marginal, and conditional pmf Joint, marginal, and conditional pdf and cdf Independence Expectation, covariance, correlation Conditional expectation Two jointly

### PS 271B: Quantitative Methods II. Lecture Notes

PS 271B: Quantitative Methods II Lecture Notes Langche Zeng zeng@ucsd.edu The Empirical Research Process; Fundamental Methodological Issues 2 Theory; Data; Models/model selection; Estimation; Inference.

### Simple Regression Theory II 2010 Samuel L. Baker

SIMPLE REGRESSION THEORY II 1 Simple Regression Theory II 2010 Samuel L. Baker Assessing how good the regression equation is likely to be Assignment 1A gets into drawing inferences about how close the

### Notes for STA 437/1005 Methods for Multivariate Data

Notes for STA 437/1005 Methods for Multivariate Data Radford M. Neal, 26 November 2010 Random Vectors Notation: Let X be a random vector with p elements, so that X = [X 1,..., X p ], where denotes transpose.

### The Method of Least Squares

Hervé Abdi 1 1 Introduction The least square methods (LSM) is probably the most popular technique in statistics. This is due to several factors. First, most common estimators can be casted within this

### Correlation key concepts:

CORRELATION Correlation key concepts: Types of correlation Methods of studying correlation a) Scatter diagram b) Karl pearson s coefficient of correlation c) Spearman s Rank correlation coefficient d)

### 1. What is the critical value for this 95% confidence interval? CV = z.025 = invnorm(0.025) = 1.96

1 Final Review 2 Review 2.1 CI 1-propZint Scenario 1 A TV manufacturer claims in its warranty brochure that in the past not more than 10 percent of its TV sets needed any repair during the first two years

### problem arises when only a non-random sample is available differs from censored regression model in that x i is also unobserved

4 Data Issues 4.1 Truncated Regression population model y i = x i β + ε i, ε i N(0, σ 2 ) given a random sample, {y i, x i } N i=1, then OLS is consistent and efficient problem arises when only a non-random

### Quadratic forms Cochran s theorem, degrees of freedom, and all that

Quadratic forms Cochran s theorem, degrees of freedom, and all that Dr. Frank Wood Frank Wood, fwood@stat.columbia.edu Linear Regression Models Lecture 1, Slide 1 Why We Care Cochran s theorem tells us

### Lecture 18 Linear Regression

Lecture 18 Statistics Unit Andrew Nunekpeku / Charles Jackson Fall 2011 Outline 1 1 Situation - used to model quantitative dependent variable using linear function of quantitative predictor(s). Situation

### Data Mining and Data Warehousing. Henryk Maciejewski. Data Mining Predictive modelling: regression

Data Mining and Data Warehousing Henryk Maciejewski Data Mining Predictive modelling: regression Algorithms for Predictive Modelling Contents Regression Classification Auxiliary topics: Estimation of prediction

### Variance Reduction. Pricing American Options. Monte Carlo Option Pricing. Delta and Common Random Numbers

Variance Reduction The statistical efficiency of Monte Carlo simulation can be measured by the variance of its output If this variance can be lowered without changing the expected value, fewer replications

### Structural Econometric Modeling in Industrial Organization Handout 1

Structural Econometric Modeling in Industrial Organization Handout 1 Professor Matthijs Wildenbeest 16 May 2011 1 Reading Peter C. Reiss and Frank A. Wolak A. Structural Econometric Modeling: Rationales

### Statistical Machine Learning

Statistical Machine Learning UoC Stats 37700, Winter quarter Lecture 4: classical linear and quadratic discriminants. 1 / 25 Linear separation For two classes in R d : simple idea: separate the classes

### Efficiency and the Cramér-Rao Inequality

Chapter Efficiency and the Cramér-Rao Inequality Clearly we would like an unbiased estimator ˆφ (X of φ (θ to produce, in the long run, estimates which are fairly concentrated i.e. have high precision.

### Course Objective This course is designed to give you a basic understanding of how to run regressions in SPSS.

SPSS Regressions Social Science Research Lab American University, Washington, D.C. Web. www.american.edu/provost/ctrl/pclabs.cfm Tel. x3862 Email. SSRL@American.edu Course Objective This course is designed

### Notes 5: More on regression and residuals ECO 231W - Undergraduate Econometrics

Notes 5: More on regression and residuals ECO 231W - Undergraduate Econometrics Prof. Carolina Caetano 1 Regression Method Let s review the method to calculate the regression line: 1. Find the point of

### 1. χ 2 minimization 2. Fits in case of of systematic errors

Data fitting Volker Blobel University of Hamburg March 2005 1. χ 2 minimization 2. Fits in case of of systematic errors Keys during display: enter = next page; = next page; = previous page; home = first

### SIMPLE REGRESSION ANALYSIS

SIMPLE REGRESSION ANALYSIS Introduction. Regression analysis is used when two or more variables are thought to be systematically connected by a linear relationship. In simple regression, we have only two