IEOR 65 Lecture 2 Method of Momets Estimatig Mea ad Variace Imagie a sceario i which we are buildig a egieerig model for a telephoe call ceter to hadle airlie reservatios For the purpose of aalysis, we eed to determie (i) the average time spet hadlig a sigle call, ad (ii) the variace of time spet hadlig a sigle call To determie these quatities, we decide to coduct a experimet i which we record the legth of time spet hadlig radomly chose calls over the time spa of oe moth The questio is how ca we determie the average ad variace usig this data? Abstract Model We ca abstract this sceario ito the followig mathematical settig: Suppose X, X 2,, X are idepedet ad idetically distributed (iid) radom variables from some ukow distributio with cdf F X (u) We should thik of the X i as idepedet measuremets from a sigle ukow distributio, ad this correspods to the legth of time spet hadlig the radomly chose calls i our sceario above The mathematical questio we are iterested i aswerig is how to determie µ(x) ad σ 2 (X)? 2 Law of Large Numbers Oe useful isight comes from the law of large umbers (ll) Roughly speakig, the ll states that if U,, U are iid radom variables, the the sample average of the U i coverges to to true mea µ(u) as goes to ifiity: U i µ(u) This is a useful isight because it meas that we ca use the sample average U i as a estimate of the true mea µ(u) 3 Mea ad Variace Estimates I fact, this isight provides a approach for estimatig both the mea ad variace of X I particular, observe that X i µ(x)
Thus, the sample average ˆµ = X i is a estimate of the mea of X (Note that we use a hat over variables to deote a estimated quatity, as opposed to the true quatity) Estimatig the variace requires a little more work First, ote that variace is defied as σ 2 (X) = E((X µ) 2 ) Performig algebra o this expressio, we have σ 2 (X) = E((X µ) 2 ) = E(X 2 2µX + µ 2 ) = E(X 2 ) 2µE(X) + µ 2 = where we have used the followig properties of expectatio: E(X + Y ) = E(X) + E(Y ); E(kX) = ke(x) Secod, recall that Xi 2 E(X 2 ) E(X 2 ) 2µ 2 + µ 2 = E(X 2 ) µ 2, This meas we ca use X2 i as a estimate of E(X 2 ), ad we have already chose X i as our estimate of the mea µ Combiig this, we coclude that a estimate of the variace is give by ˆσ 2 = Xi 2 ˆµ 2 = Xi 2 4 Example: Call Ceter Estimatio ( ) 2 X i Suppose the data from the experimets is X = 3, X 2 =, X 3 = 20, X 4 = 6, X 5 = 5 (I a real experimet, we would have more tha = 5 measuremets) The usig the above formulas, our estimate of the mea is ˆµ = X i = (3 + + 20 + 6 + 5) = 7, 5 ad our estimate of the variace is ˆσ 2 = Xi 2 ˆµ 2 = 5 (32 + 2 + 20 2 + 6 2 + 5 2 ) 7 2 = 452 2
2 Method of Momets I the above call ceter sceario, we assumed that the distributio for the time spet hadlig a sigle call was ukow However, it is the case that the distributio for the service time i a call ceter is ofte well represeted by a expoetial distributio A radom variable X with expoetial distributio is deoted by X E(λ), where λ > 0 is the rate, ad it is the distributio with pdf { λ exp( λu), if u 0, f X (u) = 0 otherwise The cdf is give by F X (u) = { exp( λu), if u 0, 0 otherwise ad so P(X > u) = exp( λu) for u 0 The mea is µ = λ, ad the variace is σ2 = λ 2 Oe importat property of a expoetial distributio is that σ 2 = µ 2 However, the estimators for mea ad variace that we derived above are ot guarateed to satisfy this relatioship I fact for the data i the above example we have that ˆµ 2 = 49 while ˆσ 2 = 452 This is ot a good situatio, because it meas that we are ot usig the full statistical kowledge available to us whe makig the estimates of mea ad variace Give this mismatch, we tur our attetio towards the questio of how we ca develop mea ad variace estimators that make better use of our statistical kowledge 2 Abstract Model We ca abstract this sceario ito the followig mathematical settig: Suppose X, X 2,, X are idepedet ad idetically distributed (iid) radom variables from a kow distributio with cdf F X (u; θ,, θ p ), but with p ukow parameters θ,, θ p that affect the shape of the distributio Two simple examples are: X N (θ, θ 2 ), where θ is a ukow mea of the Gaussia ad θ 2 is the ukow variace of the Gaussia; X E(θ ), where θ is the rate parameter of the expoetial distributio The mathematical questio we are iterested i aswerig is how to determie estimates of θ,, θ p? 22 Method of Momets Estimator Note that E(X j ) is the j-th momet of X, ad this is where the ame of the method comes from To simplify the otatio, let µ j = E(X j ) The methods of momets estimator is derived usig the followig steps: 3
Symbolically compute the first p momets ad express them as fuctios of the θ parameters: µ = E(X) = g (θ,, θ p ) µ 2 = E(X 2 ) = g 2 (θ,, θ p ) µ p = E(X p ) = g p (θ,, θ p ) 2 Symbolically ivert the system of equatios g,, g p, which gives a ew set of fuctios θ = h (µ,, µ p ) θ 2 = h 2 (µ,, µ p ) θ p = h p (µ,, µ p ) 3 Motivated by the ll, compute the followig estimates of the momets ˆµ = X i ˆµ 2 = ˆµ p = Xi 2 X p i 4 Compute estimates of the parameters by substitutig ˆµ j for µ j i the set of fuctios h,, h p, which gives the followig estimators of the parameters: ˆθ = h (ˆµ,, ˆµ p ) ˆθ 2 = h 2 (ˆµ,, ˆµ p ) ˆθ p = h p (ˆµ,, ˆµ p ) 23 Example: Expoetial Distributio We do ot cosider the example of a Gaussia distributio, because it turs out that the derivatio for the method of momets estimator is equivalet to the derivatio we did i the first sectio Istead, we compute the method of momets estimator for a expoetial distributio X E(θ ) Sice a expoetial distributio has a sigle parameter, we have p = Proceedig with the four steps, gives 4
Note µ = θ, which we kow by the properties of a expoetial distributio; 2 Ivertig this fuctio gives θ = µ ; 3 A estimator of the first momet is ˆµ = X i; 4 Substitutig this estimator gives ˆθ = X i Next, recall the data from the call ceter experimets: X = 3, X 2 =, X 3 = 20, X 4 = 6, X 5 = 5 Substitutig these values ito the equatio we have derived, we get ˆθ = X = i (3 + + 20 + 6 + 5) = 7 5 To estimate the mea ad variace, we substitute the estimated parameter ito the equatios for the mea ad variace Thus, we get ˆµ = ˆθ = 7 ˆσ 2 = ˆθ2 = 49 Comparig this estimated mea ad variace to the previous estimates, we see that the estimated mea remais the same but the estimated variace is larger However, this set of estimates is arguably better because these estimates satisfy the relatioships we would expect if the data is draw from a expoetial distributio 24 Example: Gamma Distributio Suppose X,, X are iid ad draw from a Gamma distributio, which is a distributio with pdf f X (u) = Γ(k)θ k uk exp( u/θ), where Γ( ) is the Gamma fuctio The mea of a Gamma distributio is give by µ = kθ, ad its variace is σ 2 = kθ 2 What are the equatios for the method of momets estimator for k, θ? There are two useful hits for this problem The first is that, i this example, the parameters are k = θ ad θ = θ 2 The secod is that the relatioship E(X 2 ) = µ 2 = σ 2 + µ 2 is true for every distributio where these quatities are fiite Proceedig with the four steps, gives From the properties of a Gamma distributio, we have µ = kθ µ 2 = kθ 2 + k 2 θ 2 ; 5
2 To ivert these fuctios, first ote that substitutig the equatio for µ ito the equatio for µ 2 gives µ 2 = µ θ + µ 2 θ = µ 2 µ 2 µ Substitutig this ito the equatio for µ gives µ = k(µ 2 µ 2 ) k = µ2 µ µ 2 µ 2 3 A estimator of the first ad secod momets is ˆµ = X i ˆµ 2 = Xi 2 ; 4 Substitutig this estimator ito the equatios for k, θ gives ( ˆµ ˆk 2 = = X 2 i) ˆµ 2 ˆµ 2 X2 i ( X i ˆθ = ˆµ 2 ˆµ 2 = X2 i ( X i ˆµ X i 25 Example: Liear Model of Buildig Eergy The amout of eergy cosumed i a buildig o the i-th day E i heavily depeds o the outside temperature o the i-th day T i Geerally, there are two situatios Whe the outside temperature is low, decreasig temperature leads to icreased eergy cosumptio because the buildig eeds to provider greater amouts of heatig Whe the outside temperature is high, icreasig temperature leads to icreased eergy cosumptio because the buildig eeds to provide greater amouts of coolig This is a geeral pheomeo, ad is empirically observed with the real data show i Figure, which was collected from Sutardja Dai Hall Based o the figure, we might guess that whe the outside temperature is below 59 F, the relatioship betwee eergy cosumptio ad outside temperature is give by E = a T + b + ɛ, where a, b are ukow costats, ad ɛ is a zero-mea/fiite-variace radom variable that is idepedet of T What is the method of momets estimate of a, b, σ 2 (ɛ) if we have measuremets of (T i, E i )? There are two useful hits for this problem The first is to use the radom variable ɛ = E a T b 6 ) 2 ) 2
Eergy (MWh) 22 2 8 6 4 45 50 55 60 65 70 75 Outside Air Temperature ( F) Figure : Real data (right) from Sutardja Dai Hall (left) is show The data marked with dots 22 ad crosses deote measuremets made ruig two differet 2 versios of software o the heatig, vetilatio, ad air-coditioig (HVAC) system i Sutardja 8 Dai Hall 6 The secod is to use the momets µ (ɛ) ad µ (ɛ T ) i 45 step 50, istead 55 60of the 65 momets 70 75 µ (ɛ) Outside Air Temperature ( F) ad µ 2 (ɛ) Eergy (MWh) 4 Proceedig with the four steps, gives Recall that µ (ɛ) = 0 sice ɛ is zero-mea, ad µ (ɛ T ) = 0 sice ɛ is zero-mea ad idepedet of T The momets are: 0 = µ (ɛ) = µ (E) aµ (T ) b 0 = µ (ɛ T ) = µ (ET ) aµ 2 (T ) bµ (T ); 2 This is a liear system of equatios, ad so i matrix otatio this is [ ] [ ] [ ] µ (T ) a µ (E) = µ 2 (T ) µ (T ) b µ (ET ) Solvig for a, b gives [ ] a = b Simplifyig this yields [ ] [ ] µ (T ) µ (E) (µ (T )) 2 µ 2 (T ) µ 2 (T ) µ (T ) µ (ET ) a = µ (E)µ (T ) µ (ET ) (µ (T )) 2 µ 2 (T ) b = µ (ET )µ (T ) µ (E)µ 2 (T ) (µ (T )) 2 µ 2 (T ) 7
3 A estimator of the remaiig momets is ˆµ (E) = E i ˆµ (T ) = T i ˆµ (ET ) = E i T i ˆµ 2 (T ) = T 2 i ; 4 Substitutig this estimator gives â = ( E i)( T i) E it i ( T i) 2 T 2 ( ˆb = E it i )( T i) ( E i)( ( T i) 2 T i 2 i T 2 However, this is a clumsy derivatio for a estimator of a, b I comig lectures, we will lear simpler approaches for costructig such liear models i ) 8