Lecture Notes 5. For now, we focus on parametric models. Later we consider nonparametric models.

Transcription

1 Lecture Notes 5 Statistical Models (Chapter 6) A statistical model P is a collection of probability distributions (or a collection of densities) Examples of nonparametric models are { P = p : (p (x)) dx <, and P = {all distributions on R d A parametric model has the form { P = p(x; θ) : θ Θ where Θ R d An example is the set of Normal densities {p(x; θ) = σ π e (x µ) /(σ ) where θ = (µ, σ) For now, we focus on parametric models Later we consider nonparametric models Statistics Let X,, X n p(x; θ) Let X n (X,, X n ) Any function T = T (X,, X n ) is itself a random variable which we will call a statistic Some examples are: order statistics, X () X () X (n) sample mean: X = n i X i, 3 sample variance: S = n i (X i x), 4 sample median: middle value of ordered statistics, 5 sample minimum: X () 6 sample maximum: X (n) 3 Sufficiency We continue with parametric inference In this section we discuss data reduction as a formal concept

2 3 Sufficient Statistics Suppose that X,, X n p(x; θ) T is sufficient for θ if the conditional distribution of X,, X n T does not depend on θ Thus, p(x,, x n t; θ) = p(x,, x n t) Intuitively, this means that you can replace X,, X n with T (X,, X n ) without losing information (This is not quite true as we ll see later But for now, you can think of it this way) Notation: In what follows we use the following notation: X n (X,, X n ), x n (x,, x n ) Example X,, X n Poisson(θ) Let T = n i= X i Then, But p X n T (x n t) = P(X n = x n T (X n ) = t) = P (Xn = x n and T = t) P (T = t) P (X n = x n and T = t) = { 0 T (x,, x n ) t P (X = x,, X n = x n ) T (x,, x n ) = t Hence, P (X n = x n ) = Now, T (x n ) = x i = t and so n e θ θ x i i= x i! = e nθ θ x i (xi!) = e nθ θ t (xi!) Thus, P (T = t) = e nθ (nθ) t t! P (X n = x n ) P (T = t) = since T Poisson(nθ) t! ( x i )!n t which does not depend on θ So T = i X i is a sufficient statistic for θ Other sufficient statistics are: T = 37 i X i, T = ( i X i, X 4 ), and T (X,, X n ) = (X,, X n )

3 3 Sufficient Partitions It is better to describe sufficiency in terms of partitions of the sample space Example Let X, X, X 3 Bernoulli(θ) Let T = X i x n t p(x t) (0, 0, 0) t = 0 (0, 0, ) t = /3 (0,, 0) t = /3 (, 0, 0) t = /3 (0,, ) t = /3 (, 0, ) t = /3 (,, 0) t = /3 (,, ) t = 3 8 elements 4 elements A partition B,, B k is sufficient if f(x X B) does not depend on θ A statistic T induces a partition For each t, {x : T (x) = t is one element of the partition T is sufficient if and only if the partition is sufficient 3 Two statistics can generate the same partition: example: i X i and 3 i X i 4 If we split any element B i of a sufficient partition into smaller pieces, we get another sufficient partition Example 3 Let X, X, X 3 Bernoulli(θ) Then T = X is not sufficient Look at its partition: x n t p(x t) (0, 0, 0) t = 0 ( θ) (0, 0, ) t = 0 θ( θ) (0,, 0) t = 0 θ( θ) (0,, ) t = 0 θ (, 0, 0) t = ( θ) (, 0, ) t = θ( θ) (,, 0) t = θ( θ) (,, ) t = θ 8 elements elements 3

4 33 The Factorization Theorem Theorem 4 T (X n ) is sufficient for θ if the joint pdf/pmf of X n can be factored as p(x n ; θ) = h(x n ) g(t; θ) Example 5 Let X,, X n Poisson Then p(x n ; θ) = e nθ θ X i (xi!) = (xi!) e nθ θ i X i Example 6 X,, X n N(µ, σ ) Then p(x n ; µ, σ ) = ( ) n { exp πσ (xi x) + n(x µ) σ (a) If σ known: ( ) n { p(x n (xi x) ; µ) = exp πσ σ {{ h(x n ) { n(x µ) exp σ {{ g(t (x n ) µ) Thus, X is sufficient for µ (b) If (µ, σ ) unknown then T = (X, S ) is sufficient So is T = ( X i, X i ) 34 Minimal Sufficient Statistics (MSS) We want the greatest reduction in dimension Example 7 X,, X n N(0, σ ) Some sufficient statistics are: T (X,, X n ) = (X,, X n ) T (X,, X n ) = (X,, Xn) ( m n T (X,, X n ) = Xi, T (X,, X n ) = X i i= i=m+ X i ) 4

5 T is a Minimal Sufficient Statistic if the following two statements are true: T is sufficient and If U is any other sufficient statistic then T = g(u) for some function g In other words, T generates the coarsest sufficient partition Suppose U is sufficient Suppose T = H(U) is also sufficient T provides greater reduction than U unless H is a transformation, in which case T and U are equivalent Example 8 X N(0, σ ) X is sufficient X is sufficient X is MSS So are X, X 4, e X Example 9 Let X, X, X 3 Bernoulli(θ) Let T = X i x n t p(x t) u p(x u) (0, 0, 0) t = 0 u = 0 (0, 0, ) t = /3 u = /3 (0,, 0) t = /3 u = /3 (, 0, 0) t = /3 u = /3 (0,, ) t = /3 u = 73 / (, 0, ) t = /3 u = 73 / (,, 0) t = /3 u = 9 (,, ) t = 3 u = 03 Note that U and T are both sufficient but U is not minimal 35 How to find a Minimal Sufficient Statistic Theorem 0 Define Suppose that T has the following property: R(x n, y n ; θ) = p(yn ; θ) p(x n ; θ) R(x n, y n ; θ) does not depend on θ if and only if T (y n ) = T (x n ) Then T is a MSS 5

6 Example Y,, Y n iid Poisson (θ) p(y n ; θ) = e nθ θ y i yi, p(y n ; θ) p(x n ; θ) = θ yi x i yi!/ x i! which is independent of θ iff y i = x i This implies that T (Y n ) = Y i is a minimal sufficient statistic for θ The minimal sufficient statistic is not unique But, the minimal sufficient partition is unique Example Cauchy Then p(x; θ) = p(y n ; θ) p(x n ; θ) = The ratio is a constant function of θ if π( + (x θ) ) n { + (x i θ) n { + (y j θ) i= j= T (Y n ) = (Y (),, Y (n) ) It is technically harder to show that this is true only if T is the order statistics, but it could be done using theorems about polynomials Having shown this, one can conclude that the order statistics are the minimal sufficient statistics for θ 4 What Sufficiency Really Means If T is sufficient, then T contains all the information you need from the data to compute the likelihood function It does not contain all the information in the data We will define the likelihood function shortly 6