Lecture Notes: Variance, Law of Large Numbers, Central Limit Theorem

Transcription

1 Lecture Notes: Variance, Law of Large Numbers, Central Limit Theorem CS244-Randomness and Computation March 24, Variance Definition, Basic Examples The variance of a random variable is a measure of how much the value of the random variable differs from its expected value. Let X be a random variable, and let µ = E(X) be its expected value. Then the variance is defined by V ar(x) = E((X µ) 2 ). A related quantity, which you can think of as the average deviation of X from its mean, is the standard deviation of X, denoted σ X, defined by σ X = V ar(x). You might wonder why we don t use the more obvious E( X µ ) as a measure of the average deviation from the mean. The answer, in part, is given in the last section of these notes on the Central Limit Theorem. Example. Bernoulli Random Variable Let X be a Bernoulli random variable that has the value 1 with probability p and 0 with probability q = 1 p. As we ve seen, E(X) = p. So (X µ) 2 has the value (1 p) 2 = q 2 with probability p, and the value (0 p) 2 = p 2 with probability q. Thus V ar(x) = pq 2 + qp 2 = pq(p + q) = pq, and σ X = pq. 1

2 Figure 1: PMFs for a fair die and two differently loaded dice For instance if p = 1 then V ar(x) = 1 and σ 2 4 X = 1. This represents a kind of 2 extreme case, at least for Bernoulli random variables, of deviation from the mean. At the other extreme, if p = 1 then X never varies (it always has the value 1), and V ar(x) = σ X = 0. Example. Dice, loaded and unloaded. Figure 1 shows the PMFs for three different distributions of the outcome of a single die roll. The diagram at left shows the standard uniform random variable, where each of the six outcomes has probability 1/6. The diagram in the center shows a loaded die that always results in 1 or 6, each with probability 1/2, and the diagram at right is the case where the die always results in 3 or 4, again each with probability 1/2. In all three instances, the expected value of the random variable is 3.5. For the fair die, the variance is 6 j=1 1 6 (j 3.5)2 = , so the standard deviation is the square root of this, about For the die loaded to come out 1 or 6, the variance is just 1 2 ((1 3.5)2 + (6 3.5) 2 ) = 6.25, so the standard deviation is 2.5. Similarly for the die loaded to come out 3 or 4, the variance is 0.25 and the standard deviation 0.5. In other words, the center diagram 2

3 is the most spread out, because its value is always quite far from the mean, and the right diagram the least spread out. The uniform distribution has values both far from the mean and close to the mean, giving a variance those of the two loaded dice. A (usually) easier way to compute the variance. We can write (X µ) 2 = X 2 2µX + µ 2, so by linearity of expectation, and the fact that µ is constant, V ar(x) = E(X 2 2µX + µ 2 ) = E(X 2 ) E(2µX) + µ 2 = E(X 2 ) 2µE(X) + µ 2 = E(X 2 ) µ 2. Let s repeat the computation of the variance of the Bernoulli variable, using this simpler formula. Since X has values 0 and 1, X 2 = X, so E(X 2 ) = E(X) = p. Thus V ar(x) = E(X 2 ) µ 2 = p p 2 = p(1 p) = pq, just as we found above. For the fair die, E(X 2 ) = 1 6 ( ) = , so the variance is = , which agrees with the previous example. As another example, if we want to find E(X 2 ) for a continuous random variable X, we compute x 2 p(x)dx where p(x) is the probability density function. If we take X to be the outcome of a spinner with values between 0 and 1, then p is the uniform density that is 1 between 0 and 1 and 0 elsewhere. Thus Since E(X) = 1/2, we have E(X 2 ) = 1 0 x 2 dx = 1/3. V ar(x) = E(X 2 ) E(X) 2 = 1/3 (1/2) 2 = 1/12. As was the case with the expected value, the variance of an infinite discrete or continuous random variable might not be defined, because the underlying infinite series or improper interval does not converge. 3

4 2 Additivity for Independent Random Variables; Binomial Distribution For a constant c and random variable X we have, by linearity of expectations V ar(cx) = E((cX) 2 ) E(cX) 2 = E(c 2 X 2 ) (ce(x)) 2 = c 2 E(X 2 ) c 2 E(X) 2 = c 2 (E(X 2 ) E(X) 2 ) = c 2 V ar(x). What is the variance of the sum of two random variables? Again, by repeatedly applying the linearity of expectations, V ar(x + Y ) = E((X + Y ) 2 ) E(X + Y ) 2 = E(X 2 + 2XY + Y 2 ) (E(X) + E(Y )) 2 = E(X 2 ) + 2E(XY ) + E(Y 2 ) (E(X) 2 + 2E(X)E(Y ) + E(Y ) 2 ) = (E(X 2 ) E(X) 2 ) + (E(Y 2 ) E(Y ) 2 ) + 2(E(XY ) E(X)E(Y )) = V ar(x) + V ar(y ) + 2(E(XY ) E(X)E(Y )). The expression E(XY ) E(X)E(Y ) in the right-hand summand is called the covariance of X and Y, and we will see it again later. If X and Y are independent then this expression is 0, so for independent random variables, V ar(x + Y ) = V ar(x) + V ar(y ). This does not hold in general if the random variables are not independent. In a homework problem you were asked to show that if X and Y represent, respectively, the smaller and larger values of the dice in a roll of two dice, then E(X)E(Y ) E(XY ), so in this case the sum of the variances is not equal to the variance of the sum. Example. Binomial Random Variable The binomial random variable S n gives the probability of k heads on n coin tosses; as we ve seen before, ( ) n P [X n = k] = p k (1 p) n k k 4

5 if p is the probability of heads. X n is itself the sum of n pairwise independent copies of a Bernoulli random variable X each with probability p. As a result, using the summation result above, and V ar(s n ) = n V ar(x) = np(1 p), σ Sn = np(1 p). Further, let Y = S n /n be the average number of heads on n tosses of a coin. Then V ar(s n ) = V ar(x)/n 2 = p(1 p)/n. 3 Chebyshev Inequality The notion of variance allows us to obtain a rough estimate of the probability that a random value differs by a certain amount from its mean. Before we state and prove this, we ll establish a simpler property called the Markov inequality: If X is a positive-valued random variable with expected value µ, and c > 0, then P [X c] E(X). c Let s see why this is: We ll assume that X is a discrete random variable with finitely many outcomes 0 < c 1 < c 2 < c n. with probabilities p 1, p 2,..., p n, respectively. Let c i be the smallest of these values that is greater than or equal to c. Then P [X > c] = p i + p i p n = 1 c (cp i + cp i+1 + cp n ) 1 c (c ip i + c i+1 p i c n p n ) 1 c (c 1p 1 + c 2 p c n p n ) = E(X). c Essentially the same argument works for infinite discrete and continuous random variables. The Markov inequality by itself does not provide a great deal of information, but we will use it show something important. Let X be any random 5

6 variable with expected value µ, and let ɛ be a positive number. (Think of ɛ as small.) When we apply the Markov inequality to the random variable (X µ) 2 then we get P [(X µ) 2 ɛ 2 ] E((X µ)2 ) ɛ 2 = V ar(x) ɛ 2. Since the left-hand side is the same thing as P [ X µ ɛ], we can write this as P [ X µ ɛ] V ar(x) ɛ 2. This is called Chebyshev s inequality. It provides a probability bound on how likely it is for a random variable to deviate a certain amount from its expected value. Example. Let S n be the number of heads on n tosses of a fair coin. Let s estimate the probability that the number of heads for n = 100 is between 40 and 60. Since E(S 100 ) = 50, we are asking for the complement of the probability that the number of heads is at least 61 or at most 39; in other words, that it differs by at least 11 from it s mean. Since the variance of S 100 is 100/4 = 25, Chebyshev s inequality gives P [ S ] < 25/ , so the probability of the number of head being between 40 and 50 is at least As we shall see later, the probability is actually much closer to 1 than this. Chebyshev s inequality only gives a rough upper bound, not a close approximation. The advantage is that it applies to absolutely any random variable. If we take ɛ = tσ X, then we can write the inequality as P [ X µ tσ X ] 1 t 2. For instance, the probability of being at least two standard deviations from the mean is at most Law of Large Numbers The numerical outcome of an experiment is a random variable X. Let us suppose X has expected value µ and variance σ 2. If we perform n independent trials of 6

7 the experiment, then the random variable A n = (X X n )/n, giving the average of the outcomes of these trials, still has expected value µ, but now the variance is σ 2 /n. Thus by Chebyshev s inequality, for any ɛ > 0 P r[ A n µ > ɛ] < σ2 nɛ 2. What does this mean? Imagine that ɛ is rather small, say Then the right-hand side is 10 4 σ 2 /n. If we choose n to be large enough, then we can make this righthand side as small as we like, which means that we can guarantee with probability as large as we like, that the average A n is within 0.01 of its expected value. Put more simply, we can get as close to the mean as we like (with probability as high as we like) by repeating the experiment often enough. (Of course, we cannot make the probability exactly 1 no matter how many times we repeat the experiment, nor can we guarantee that the average will be exactly equal to the mean.) This is the precise statement of what is often described colloquially as the law of averages. In probability theory, it is called the Law of Large Numbers. 5 Normal Approximation to Binomial Distribution Figure 2 shows the PMFs of the random variables S n for n = 20, 60, 100, where S n denotes the number of heads on n tosses of a coin with heads probability p = 0.4. These PMFs are given by the binomial probability distribution ( ) n P [S n = k] = p k (1 p) n k. k The three random variables of course all have different expected values (10, 30 and 50, respectively) so the PMFs are nonzero on different parts of the numer line. By our previous calculations, the standard deviation of S n grows proportionally to the square root of n. As a result, the PMFs get more spread out as n increases. In Figure 3, S n is replaced by S n np, so that all three random variables have expected value 0. In Figure 4, we further change this to the random variables S n np np(1 p), so that all three have variance 1. The apparent result is that all three graphs seem to have the same basic shape, but just differ in the vertical scale. In Figure 5, the vertical scale is adjusted so 7

8 that all three have maximum value 1. All the points lie on the same smooth curve. What is this shape? The smooth curve was drawn by plotting the graph of y = e x2 /2, and the crucial result illustrated by these pictures is that this shape closely approximates the binomial distribution. In other words, this famous bell curve represents a continuous probability density that is a kind of limiting case of the binomial distributions as n grows large. Let s be a little more precise about this: The function e x2 /2 is not itself a probability density function, because the area under the curve is not 1, but it becomes a density function when we divide by the area under the curve. (This area is 2π, a fact that is far from obvious.) is called the standard normal density. Standard here means that it has mean 0 and standard deviation 1. The corresponding cumulative distribution function is Φ(x) = 1 2π x e t2 /2 dt. Since we cannot evaluate Φ(x) analytically, it has to be approximated numerically. You can compute Φ(x) in Python to high accuracy using a built-in related function erf, as *math.erf(x/math.sqrt(2)) Our observations above illustrate an important fact: the binomial distribution, adjusted to have mean 0 and standard deviation 1, is closely approximated by the normal distribution, especially as n gets larger. Here are a few examples. Example. Let us redo the problem of estimating the probability that on one hundred tosses of a fair coin, the number of heads is between 40 and 60. Let X be the random variable representing the number of heads, so we are asking for P [45 X 60]. We make the same modification as above, subtracting the expected value 50 and dividing by the standard deviation = 5. Thus we are looking for 4 P [ 1 X ]. 8

9 Figure 2: PMFs of binomial distribution with n = 20, 60, 100 and p = 0.4 9

10 Figure 3: The same distributions shifted to all have mean

11 Figure 4:...and scaled horizontally to have standard deviation 1 11

12 Figure 5: The previous figure stretched vertically so that all three PMFs appear with the same height, superimposed on the graph of e x2 /2 12

13 Figure 6: The standard normal density φ(x): the shaded area is Φ(1), the probability that the standard normal random variable has value less than 1. 13

14 Figure 7: The cumulative normal density Φ(x). 14

15 Approximating this by the standard normal distribution suggests that this probability is about Φ(1) Φ( 1) = The exact value, of course, is 55 j=45 ( 100 j ) = , so our approximation is not very impressive, only accurate to one decimal digit of precision. Part of the reason can be seen in the fact that the probability we are looking for is also equal to P [44 < X < 56]. One of the pitfalls of approximating a discrete distribution with a continuous one is that we don t necessarily know exactly where we should draw the lines between values of the random variable. It turns out that works well in this situation is to use values for the continuous distribution that are halfway between the relevant values for the discrete distribution: In this case, that means we should view the problem as one of calculating This gives an estimate of P [44.5 < X < 55.5]. Φ(1.1) Φ( 1.1) = , accurate to four decimal digits of precision. 6 Central Limit Theorem The last section illustrated the fact that the sum of independent identically distributed Bernoulli random variables is approximately normally distributed. This is an instance of a much more general phenomenon every random variable has this property! To be more precise, let X be a random variable with µ = E(X) and σ 2 = V ar(x) defined. Let X 1,..., X n be pairwise independent random variables, each with the same distribution as X. Think of this as making n independent repetitions of an experiment whose outcome is modeled by the random variable X. Our claim 15

16 is that the sum of the X i is approximately normally distributed. Again we adjust the mean and standard deviation to be 0 and 1; then the precise statement is lim P [a < X 1 + X n nµ n σ n < b] = Φ(b) Φ(a). This is called the Central Limit Theorem. Before we saw, with the Law of Large Numbers, that the deviation of the average of n independent identical random variables from its mean approaches 0 as n grows larger. The Central Limit Theorem says more: it tells us how that deviation is distributed. Example. Let s look at an experiment that was the subject of a question on the midterm: Spin two spinners, each giving a value uniformly distributed between 0 and 1, and let X be the larger of the two values. We saw that the cdf of X was given by y = x 2 between 0 and 1, and thus the pdf of X is y = 2x between 0 and 1, and 0 elsewhere. (See the posted exam solutions for the details.) We can then compute so µ = E(X) = E(X 2 ) = x 2xdx = 2 3, x 2 2xdx = 1 2, σ 2 = V ar(x) = 1 2 (2 3 )2 = 1 18, and σ = Suppose we perform this experiment 100 times. How likely is it that the sum is greater than 65? The expected value of the sum is Since the distribution of 3 the sum is approximately normal, and thus symmetric about the mean, we should expect a probability greater than one half. How likely is it that the sum is greater than 70? Here we should expect a probability less than one-half. We apply the Central Limit Theorem to obtain an estimate. We first try to compute P r[0 X X 100 < 65], so making our usual transformation with µ and σ, this is P r[ /(10/ 18) < X X / 18 < /(10/ 18)]. 16

17 The Central Limit Theorem says that this is approximately Φ( /(10/ 18)) Φ( /(10/ 18)). The right-hand expression is a very tiny number which we can treat as 0. (Alternatively, we could just as well have use as 0 in our computation.) So this gives the approximation Φ( /(10/ 18)) = , and thus the probability that the sum is greater than 65 is = , which is greater than one-half, as we expected. If we replace 65 by 70, then an identical calculation gives 1 Φ( /(10/ 18)) = = I simulated the experiment of 100 spinners spun twice and summed the maxima of the results. In 10,000 repetitions, I found that the number of times the sum was greater than 65 to be 7594 (as compared to the predicted 7602), and the number of times the sum was greater than 70 to be 780 (as compared to the predicted 787). 17