Desirable Statistical Properties of Estimators. 1. Two Categories of Statistical Properties

Transcription

1 Desirable Statistical Properties of Estimators 1. Two Categories of Statistical Properties There are two categories of statistical properties of estimators. (1) Small-sample, or finite-sample, properties of estimators The most fundamental desirable small-sample properties of an estimator are: S1. Unbiasedness S2. Minimum Variance S3. Efficiency (2) Large-sample, or asymptotic, properties of estimators The most important desirable large-sample property of an estimator is: L1. Consistency Both sets of statistical properties refer to the properties of the sampling distribution, or probability distribution, of the estimator for different sample sizes. β Page 1 of 14 pages

2 1.1 Small-Sample (Finite-Sample) Properties The small-sample, or finite-sample, properties of the estimator β refer to the properties of the sampling distribution of β for any sample of fixed size N, where N is a finite number (i.e., a number less than infinity) denoting the number of observations in the sample. N = number of sample observations, where N <. Definition: The sampling distribution of β for any finite sample size N < is called the small-sample, or finite-sample, distribution of the estimator β. In fact, there is a family of finite-sample distributions for the estimator β, one for each finite value of N. Page 2 of 14 pages

3 The sampling distribution of β is based on the concept of repeated sampling. Suppose a large number of samples of size N are randomly selected from some underlying population. Each of these samples contains N observations. Each of these samples in general contains different sample values of the observable random variables that enter the formula for the estimator. β For each of these samples of N observations, the formula for β is used to compute a numerical estimate of the population parameter β. Each sample yields a different numerical estimate of the unknown parameter β. Why? Because each sample typically contains different sample values of the observable random variables that enter the formula for the estimator β. If we tabulate or plot these different sample estimates of the parameter β for a very large number of samples of size N, we obtain the small-sample, or finite-sample, distribution of the estimator β. Page 3 of 14 pages

4 1.2 Large-Sample (Asymptotic) Properties The large-sample, or asymptotic, properties of the estimator β refer to the properties of the sampling distribution of β as the sample size N becomes indefinitely large, i.e., as sample size N approaches infinity (as N ). Definition: The probability distribution to which the sampling distribution of β converges as sample size N becomes indefinitely large (i.e., as N ) is called the ultimate distribution of the estimator β. Page 4 of 14 pages

5 Nature of Small-Sample Properties 2. Small-Sample Estimator Properties The small-sample, or finite-sample, distribution of the estimator β for any finite sample size N < has 1. a mean, or expectation, denoted as E( β ) Var( β ). 2. a variance denoted as, and The small-sample properties of the estimator are defined in terms of the mean E β and the variance Var( β ) of the finite-sample distribution of the estimator β for any finite sample size N <. β ( ) Page 5 of 14 pages

6 S1: Unbiasedness Definition of Unbiasedness: The estimator β is an unbiased estimator of the population parameter β if the mean or expectation of the finite-sample distribution of β is equal to the true β. That is, β is an unbiased estimator of β if E ( ) = β β for any given finite sample size N <. Definition of the Bias of an Estimator: The bias of the estimator β is defined as Bias ( ) = E( β ) β β = the mean of β minus the true value of β. The estimator β is an unbiased estimator of the population parameter β if the bias of β is equal to zero; i.e., if Bias ( β ) E( = β ) β = 0 E( ) = β β. Alternatively, the estimator β is a biased estimator of the population parameter β if the bias of β is nonzero; i.e., if Bias ( β ) E( = β ) β 0 E( ) β β. Page 6 of 14 pages

7 1. The estimator β is an upward biased (or positively biased) estimator the population parameter β if the bias of β is greater than zero; i.e., if Bias ( β ) E( = β ) β > 0 E( ) > β β. 2. The estimator β is a downward biased (or negatively biased) estimator of the population parameter β if the bias of β is less than zero; i.e., if Bias ( β ) E( = β ) β < 0 E( ) < β β. Page 7 of 14 pages

8 Meaning of the Unbiasedness Property The estimator β is an unbiased estimator of β if on average it equals the true parameter value β. This means that on average the estimator β is correct, even though any single estimate of β for a particular sample of data may not equal β. More technically, it means that the finite-sample distribution of the estimator β is centered on the value β, not on some other real value. The bias of an estimator is an inverse measure of its average accuracy. The smaller in absolute value is ( ) population parameter β. Bias β, the more accurate on average is the estimator β in estimating the Thus, an unbiased estimator for which Bias( β ) = 0 -- that is, for which ( ) perfectly accurate estimator of β. E β = β -- is on average a Given a choice between two estimators of the same population parameter β, of which one is biased and the other is unbiased, we prefer the unbiased estimator because it is more accurate on average than the biased estimator. Page 8 of 14 pages

9 S2: Minimum Variance Definition of Minimum Variance: The estimator β is a minimum-variance estimator of the population parameter β if the variance of the finite-sample distribution of β is less than or equal to the variance of the finite-sample distribution of β, where β is any other estimator of the population parameter β ; i.e., if Var ( ) Var( β ) β for all finite sample sizes N such that 0 < N < where Var Var ( ) [ ( )] 2 = E β E β ( ) = E[ β E( β )] 2 β = the variance of the estimator β ; β = the variance of any other estimator β. Note: Either or both of the estimators β and about whether the estimators are biased or unbiased. Meaning of the Minimum Variance Property β may be biased. The minimum variance property implies nothing The variance of an estimator is an inverse measure of its statistical precision, i.e., of its dispersion or spread around its mean. The smaller the variance of an estimator, the more statistically precise it is. A minimum variance estimator is therefore the statistically most precise estimator of an unknown population parameter, although it may be biased or unbiased. Page 9 of 14 pages

10 S3: Efficiency A Necessary Condition for Efficiency -- Unbiasedness The small-sample property of efficiency is defined only for unbiased estimators. Therefore, a necessary condition for efficiency of the estimator is that unbiased estimator of the population parameter β. β E( ) = β β, i.e., β must be an Definition of Efficiency: Efficiency = Unbiasedness + Minimum Variance Verbal Definition: If β and β are two unbiased estimators of the population parameter β, then the estimator β is efficient relative to the estimator β if the variance of β is smaller than the variance of β for any finite sample size N <. Formal Definition: Let β and β be two unbiased estimators of the population parameter β, such that ( E β ) = β and E( β ) = β. Then the estimator β is efficient relative to the estimator β if the variance of the finite-sample distribution of β is less than or at most equal to the variance of the finite-sample distribution of β ; i.e. if Var ( β ) Var( β ) for all finite N where ( E β ) = β and E( ) = β β. Note: Both the estimators β and β must be unbiased, since the efficiency property refers only to the variances of unbiased estimators. Page 10 of 14 pages

11 Meaning of the Efficiency Property Efficiency is a desirable statistical property because of two unbiased estimators of the same population parameter, we prefer the one that has the smaller variance, i.e., the one that is statistically more precise. In the above definition of efficiency, if β is any other unbiased estimator of the population parameter β, then the estimator β is the best unbiased, or minimum-variance unbiased, estimator of β. Page 11 of 14 pages

12 3. Large Sample Estimator Properties L1: Consistency the minimal requirement of any useful estimator Definition of Consistency Verbal Definition: Let β (N) denote an estimator of the population parameter β based on a sample of size N observations. The estimator β (N) is a consistent estimator of the population parameter β if its sampling distribution collapses on, or converges to, the value of the population parameter β as N. Formal Definition: The estimator β (N) is a consistent estimator of the population parameter β if the probability limit of β (N) is β, i.e., if plim β (N) = β or lim Pr ( β (N) β ε) = 1 β as N. or Pr( β ε) 1 The estimator β (N) is a consistent estimator of the population parameter β if the probability that β (N) is arbitrarily close to β approaches 1 as the sample size N or if the estimator β (N) converges in probability to the population parameter β. Page 12 of 14 pages

13 Intuitive Meaning of the Consistency Property As sample size N becomes larger and larger, the sampling distribution of β (N) becomes more and more concentrated around β. As sample size N becomes larger and larger, the value of β (N) is more and more likely to be very close to β. Page 13 of 14 pages

14 A Sufficient Condition for Consistency One way of determining if the estimator β (N) is consistent is to trace the behavior of the sampling distribution of β (N) as sample size N becomes larger and larger. If as N (sample size N approaches infinity) both the bias of β (N) and the variance of β (N) approach zero, then β (N) is a consistent estimator of the parameter β. Recall that the bias of (N) is defined as Thus, the bias of β (N) approaches zero as distribution of β (N) approaches β as β Bias( (N)) = E( β (N) ) β β. N if and only if the mean or expectation of the sampling N : lim Bias ( β (N)) = lim E( β (N)) β = 0 lim E β (N) ( ) = β. Result: A sufficient condition for consistency of the estimator β (N) is that lim Bias ( β (N)) = 0 lim E β (N) or ( ) = β and lim Var( β (N)) = 0. This condition states that if both the bias and variance of the estimator β (N) approach zero as sample size N then β (N) is a consistent estimator of β. Page 14 of 14 pages