MIN-MAX CONFIDENCE INTERVALS

Transcription

1 MIN-MAX CONFIDENCE INTERVALS Johann Christoph Strelen Rheinische Friedrich Wilhelms Universität Bonn Römerstr. 164, Bonn, Germany July 2004

2 STOCHASTIC SIMULATION Random input = random output two different runs of the same model different output. Due to the stochastic nature of the simulation results, careful statistic analysis must be done for the correct interpretation of calculated values. If this is omitted, there is a significant probability of making erroneous inferences about the system under study. 2

3 CONFIDENCE INTERVALS Unknown parameter θ is to be estimated from an output process X 1, X 2,..., X n Confidence interval I(X 1,..., X n ) = [L(X 1,..., X n ), U(X 1,..., X n )] such that P {θ I(X 1,..., X n )} = 1 α Confidence level 1 α where the probability α, small, is given. Width U(X 1,..., X n ) L(X 1,..., X n ) should be small. 3

4 ESTIMATORS Functions T (X 1,..., X n ) for the estimation of the unknown parameter θ such that given an output x 1,..., x n one may expect T (x 1,..., x n ) θ E[T (X 1,..., X n )] = θ: T unbiased lim n E[T (X 1,..., X n )] = θ: T asymptotically unbiased 4

5 Statistical Theory For the construction of the confidence interval, probability distributions of the interval bounds, U(X 1,..., X n ) and L(X 1,..., X n ), are determined. Usual assumptions: the X 1,..., X n are independent random variables they are identically distributed often: they are normally distributed 5

6 CLASSICAL CONFIDENCE INTERVALS Ȳ ± t n 1,1 α/2 S2/n Sample (Y 1,..., Y n ) of independent, normally distributed random variables Confidence level 1 α, 0 < α < 1 Sample mean Ȳ = (Y Y n )/n Sample variance S2 = (Y Y 2 n )/(n 1) n n 1Ȳ 2 (1 α/2)-quantile of the Student distribution with n 1 degrees of freedom t n 1,1 α/2 But: In simulation, mostly the assumptions of the statistical theory are not fulfilled 6

7 Resort Central limit theorem: Y Y n is nearly normally distributed if the sample Y 1,..., Y n is IID and if n is large Independent replications of the simulation with different random number streams - the estimators in these runs are independent Grouping consecutive results of a long simulation run into batches - considered to be (nearly) independent Evaluating only the steady state phase of each simulation run - ignoring the transient phase 7

8 ACCURACY Inaccurate confidence intervals not unusual in simulation, e.g. assumed confidence level 90%, coverage only 80%. This means: In many different simulations, only approximately 80% of the confidence intervals contain the real value. Comparative numerical studies: More elaborated techniques (regenerative method, autoregressive processes, spectral estimation method, standardized time series method) may be less accurate than batch-means method and replication/deletion method. Median confidence intervals may be even more accurate. In long simulation runs, the accuracy of the confidence intervals is better. 8

9 Min-Max Confidence Intervals (MMCI), Median Confidence Intervals (MCI) A new confidence interval (CI) technique for simulation results Easy to apply Accurate Generally applicable 9

10 Main Features Easy to obtain: w independent replications (simulation runs) or a single simulation run with w subsequent phases for batches of data typically w = 5 or 6. The variance of the estimator is not used, correlated output is implicitly considered. Hence, a serious problem is omitted which usually arises when confidence intervals for simulation results are derived. Even if the variance does not exist, an MCI can be constructed whereas a classical CI cannot. Sequential procedure: If a median confidence interval (MCI) is too wide, given a confidence level, it can be narrowed: Each of the replications are augmented, beginning with the last state. Similar for batches. 10

11 If a measure is estimated with a function of some estimators, an MCI can be given. Example: Λ(n) estimates the throughput of a queue and W (n) the mean waitig time. Then the product Λ(n)W (n) estimates the mean number of customers in the queue (Littles Formula). For some samples of independent random variables, e.g. normally distributed, we found MCIs which are sometimes slightly wider than usual CIs. But such simple statistic occurs seldom in simulation. Here the output is usually dependent, and the distribution is unknown. Under these circumstances, classical CIs are usually too narrow, the confidence level is not realistic, the CIs too often do not cover the real unknown value, they are only approximate. MCIs are more accurate. 11

12 The MCI technique is exact when the median and the mean of an estimator coincide. This holds for symmetrical distributions the most important one in simulation is the normal distribution. Due to the central limit theorem and long simulation runs with fast computers many estimators are nearly normally distributed. But in principle, the MCI technique is not restricted to the case median = mean. In this general case, one must know a single value of the estimator distribution function F θ (x), namely the probability F = F θ (θ) where θ is the unknown parameter. Not each confidence level is possible, only the values 1 F w (1 F ) w, w = 2, 3,... Here, w is the number of independent replications or of batches of data. In the special case median = mean, F = 0.5 holds, and the possible confidence levels are 50%, 75%, 87.5%, 93.75%, %, 98.4%, 99.2%, 99.6%, 99.8%, 99.9%,... 12

13 The Basic Principle Sample X 1,1,..., X 1,m of random variables, one run of a steady-state simulation or of n terminating runs. θ unknown parameter to estimate. T (X 1,1,..., X 1,m ) estimator, distribution function F θ (x). Novel kind of confidence interval [ T min, T max ) (1) where and T min = min 1 i w T i, T max = max 1 i w T i, T i = T (X i,1,..., X i,m ), i = 1,..., w estimators for w independent replications X i,1,..., X i,m of the sample X 1,1,..., X 1,m. 13

14 Theorem 1 The interval (1) is a confidence interval for the parameter θ with the confidence level 1 F w (1 F ) w, i.e. P {T min θ < T max } = 1 F w (1 F ) w holds where F = F θ (θ), the value of the estimator distribution function at θ. The Most Important Special Case: Mean = Median Here, the unknown parameter is the median of the estimator, F θ (θ) = 1/2 This holds for unbiased estimators and symmetrical distributions, e.g. the estimator is normally distributed. Then for the confidence interval P {T min θ < T max } = w 1 holds, and the possible confidence levels are w 1, w = 2, 3,... 14

15 Batch Median Confidence Intervals for steady state statistics. We applied the idea of the batch means method: Grouping output data into batches and assuming these batches to being independent. A single simulation run: First the transient phase, then w phases for w batches of output data. From each batch one obtaines an estimate ˆT i, i = 1,..., w. The batch mean confidence interval (BMCI) is [ min 1 i w ˆT i, max 1 i w ˆT i ). 15

16 Interesting application where F can be calculated: Order statistics as estimates for quantiles. Consider samples X 1,..., X n and the according ordered sequence X (1),..., X (n), X (i) X (j) if i < j, where the X i are IID with the strictly increasing distribution function F (x). The q-quantile θ = x q, q (0, 1), F (x q ) = q, is estimated by X (r), r {1, 2,..., n}. Let F θ (x) denote the distribution function of the estimator, namely X (r). 16

17 Here, F = F θ (x) is known: Theorem 2 If the q-quantile x q is estimated by X (r), the min-max confidence interval (1) has precisely the confidence level of theorem 1 with F = n i=r n i q i (1 q) n i. (2) Remarks 1. Here the value F = F θ (x q ) is independent of the actual distribution function of the sample elements X i. 2. Theorem 2 is not useful for the simulation of the extremes, q = 0 or q = 1. Here one gets the confidence level Usually, k qn is chosen. 17

18 Corollary If the sample size n is odd, r = n/2 and q = 0.5, i.e. the median is estimated, F = 0.5 holds. 18

19 Confidence Intervals in Simulation are Usually Approximate Assumptions are not satisfied, in general What means approximate confidence The distribution of the estimator (normal, interval? If for a parameter of a simulation Student) Independency of the r.v. in the sample For some methods other assumptions For median confidence intervals the assumptions are weaker: Only symmetry of the distribution of the estimator model, many confidence intervals are calculated in many simulations, the real value lies in some of them, in the others it does not. The coverage C is the fraction of runs where it is within. If the limit of this coverage equals the confidence level CL = 1 α, the confidence interval technique is exact, otherwise approximate: The confidence level is not reached, Independency of the replications, not of CL C. the r.v. within them 19

20 Numerical Experience Many simulation studies. Comparison of classical confidence interval methods with median confidence intervals or with batch median confidence intervals. Each Study: Many independent simulation experiments for the estimation of the coverage of each considered confidence interval technique. Each simulation experiment: w= 5 independent replications for median confidence intervals (MCI) and for the replication/deletion method or w= 5 batches for batch median confidence intervals (BMCI) and for the batch means method. w= 5 implies a confidence level CL = 93.75% for the MCIs and BMCIs. Measure for the accuracy: The error CL C = confidence level observed coverage. 20

21 1. M/M/1 Queueing System: Waiting Times (Delays) Law and Kelton comparative study for different well known methods for confidence Batch Means intervals. Standardized Time Series Utilization 0.8; known to be statistically Spectrum Analysis difficult. Autoregressive Method independent simulation experiments for Regenerative Method Classical each run length n and each CI method, n Regenerative Method Jackknife = 2560 delays e.g. coverage C. We conducted an according simulation Batch Median Confidence Intervals Errors CL C study with the same model and the same run lengths including batch median confidence intervals (BMCI). Error means confidence level CL = 90%, observed coverage C = 75.5%, e.g. 21

22 2. M/M/1 Queueing System Comparison of the replication/deletion method (RD) and median confidence intervals (MCI). Low and high utilization (ρ = 0.25 and 0.8). Short and long simulation runs. ρ Run replication/deletion median confidence intervals Short Long Short Long Errors CL C Long runs: Both methods good Short runs: MCIs slightly better 22

23 3. M/M/1 Queueing System, Ratios of Estimators The same M/M/1-model as before. Comparison of jackknife intervals and median confidence intervals for the mean delay, Ŵ (r), as ratio of ˆQ/ˆλ of the mean number of jobs in the waiting room and the mean throughput. ρ Run RD, Jackknife Median Confidence Intervals Short Long Short Long Errors CL C Median confidence intervals are more accurate. 23

24 4. Pareto distribution We are interested in parameters of heavy-tailed Pareto distributions, F (x) = 1 x a, 0 < a <= 2, x 1, with expectation a/(a 1) for a > 1, median 2 1/a, the variance does not exist simulation experiments, each with sample size n = The classical confidence interval for the expectation does not exist. Median confidence intervals for the expectation: a CL C good bad 1.1 not acceptable Median confidence intervals for the order statistic for the median: a Confidence Interval Median CI Errors CL C

25 5. Reliability Model The model consists of three components and will function as long as component 1 works and either component 2 or 3 works. G i is the time to failure of component i, i = 1, 2, 3, and G = min{g 1, max{g 2, G 3 }} the time to failure of the whole system. The random variables G i are independent, and each G i has a Weibull distribution F (x) = 1 exp( x), x > 0. The estimator of the expectation of G has a very skewed and nonnormal distribution, all confidence intervals are quite inaccurate, for small sample sizes simulation experiments, each with sample size n = 5 or 40. n Classical CI Median CI Errors CL C 25

26 Potential Further Development of the Technique The assumption of symmetry of the estimator distribution can be omitted, even the estimator may be biased, only F = F θ (θ), the value of the estimator distribution function at θ, the unknown parameter, must be known. Then we speak of min-max confidence intervals (MMCI). They are exact if the w replications are independent, their confidence level is CL = 1 F w (1 F ) w. Crucial problem: This value F θ (θ). We do not know an adequate method for estimating it efficiently. 26

27 But this MMCI idea works, we tried a brute-force procedure: Very long and expensive simulations for an empirical distribution of the r.v. G of example 5, then the distribution function ˆF θ (x) of the estimator with convolution, and with an estimation of the unknown parameter, ˆθ, we obtained ˆF = ˆF θ (ˆθ) and an estimate ĈL. n Coverage ĈL Coverages and Estimated Confidence Levels Accurate, isn t it? But so not practicable 27