P (A) = lim P (A) = N(A)/N,

Transcription

1 1.1 Probability, Relative Frequency and Classical Definition. Probability is the study of random or non-deterministic experiments. Suppose an experiment can be repeated any number of times, so that we can produce a series of independent trials under identical conditions. This accumulated experience reveals a remarkable regularity of behaviour in the following sense. In each observation, depending on chance, a particular event A either occurs or does not occur. Let n be the total number of observations in this series and let n(a) denote the number of times A occurs. Then the ratio n(a)/n is called the relative frequency of the event A (in this given series of independent and identical trials). It has been empirically observed that the relative frequency becomes stable in the long run. This stability is the basis of probability theory. That is, there exists some constant P (A), called the probability of the event A, such that for large n, P (A) n(a)/n. In fact, the ratio approaches this limit such that P (A) = lim n n(a)/n. Although the preceding definition is certainly intuitively pleasing and should always be kept in mind, it possesses a serious drawback: How do we know that n(a)/n will converge to some constant limiting value that will be the same for each possible sequence of repetitions of the experiment? Proponents of this relative frequency definition of probability usually answer this objection by stating that the convergence of n(a)/n to a constant limiting value is an assumption, or an axiom, of the system. However, to assume that n(a)/n will necessarily converge to some constant value seems to be a very complex assumption, even though this convergence is indeed backed up by many of our empirical experiences. Historically, probability theory began with the study of games of chance. The starting point is an experiment with a finite number of mutually exclusive outcomes which are equiprobable or equally likely because of the nature and setup of the experiment. Let A denote some event associated with certain possible outcomes of the experiment. Then the probability P (A) of the event A is defined as the fraction of the outcomes in which A occurs. That is, P (A) = N(A)/N, where N is the total number of outcomes of the experiment and N(A) is the number of outcomes leading to the occurrence of event A. For example, in tossing a fair or unbiased die, let A be the event of getting an even number of spots. Then N = 6 and N(A) = 3. Hence P (A) = 1/2. This classical definition of probability is essentially circular since the idea of equally likely is the same as that of with equal probability, which has not been defined. The modern treatment of probability theory is purely axiomatic. This means that the probabilities of our events can be prefectly arbitrary, except that they must satisfy a set of simple axioms listed below. The classical theory will correspond to the special case of so-called equiprobable spaces. 1.2 Sample Space and Events. The set Ω of all mutually exclusive outcomes of some given random experiment is called the sample space. An elementary outcome ω, which is an element of Ω, is called a sample point. An event A is assoicated with the sample point ω if, given ωεω, we can always decide whether ω leads 1

2 2 to the occurrence of A. That is, event A occurs if and only if ωεa. Hence event A consists of a set of outcomes or, in other words, simply a subset of the underlying sample space Ω. The event {ω} consisting of a single sample point ωεω is called an elementary event. The empty set φ and Ω itself are events; φ is sometimes called the impossible event, and Ω the certain or sure event. Two events A and B are idential if A occurs iff B occurs. That is, they lead to the same subset of Ω. Two events A and B are mutually exclusive or incompatible if A and B cannot occur simultaneously. That is, A B = φ. If the sample space Ω is finite or countably infinite, then every subset of Ω is an event. On the other hand, if Ω is uncountable, we shall be concerned with the σ-algebra E of subsets of Ω, which is closed under the operations of set union, set intersection, set difference and set complement. 1.3 Axioms of Probability. Let Ω be a sample space, let E be the class of events, and let P be a real-valued function defined on E. Then P is called a probability function, and P (A) is called the probability of the event A if the following axioms hold: Axiom 1. For every event A, 0 P (A) 1. Axiom 2. P (Ω) = 1. Axiom 3. If A 1, A 2,... is a sequence of mutually exclusive events, then P (A 1 A 2... ) = P (A 1 ) + P (A 2 ) +. The assumption of the existence of a set function P, defined on the events of a sample space and satisfying Axioms 1, 2 and 3 constitutes the modern mathematical approach to probability theory. The axioms are natual and in accordance with our intuitive concept of probability as related to chance and randomness. Furthermore, using these axioms, we shall be able to prove that the relative frequency of a specific event A will equal P (A) with probability 1 under a series of independent and identical trials of an experiment. This is the well known result of the strong law of large numbers in probability theory. A few easy and useful results concerning the probabilities of events follow. Proposition 1. P (φ) = 0. Proposition 2. P (A) = 1 P (A), where A is the complement of A. Proposition 3. If A B, then P (A) P (B). Proposition 4. P (A B) = P (A) + P (B) P (A B). 1.4 Additive Laws. For a sequence of n mutually exclusive events A 1, A 2,..., A n, since A i A j = φ for i j, we have the additive law for mutually exclusive events n P ( A k ) = P (A k ). For a set of arbitrary events, there is the more general result stating that the probability of the union of n events equals the sum of the probabilities of these events taken one at a time, minus the sum of the probabilities of the events taken two at a time, plus the sum of the probabilities of these events taken three at a time, and so on.

3 3 Theorem. Given n events A 1, A 2,..., A n in Ω, let P 1 = P 2 = P 3 = n P (A i ) i=1 1 i<j n 1 i<j<k n.. P (A i A j ) P n = P (A 1 A 2 A n ) P (A i A j A k ) Then P ( A k ) = P 1 P 2 + P 3 + ( 1) n+1 P n. The proof is by induction on the number of events. For n = 2, we readily have P (A 1 A 2 ) = P (A 1 ) + P (A 2 ) P (A 1 A 2 ). Suppose the statement holds for n 1 events A 2, A 3,..., A n, i.e. Hence P ( P ( A k ) = k=2 A 1 A k ) = k=2 2 i n 2 i n P (A i ) P (A 1 A i ) 2 i<j n 2 i<j n Noting that n A k = A 1 ( n k=2 A k), we have P (A i A j ) + P (A 1 A i A j ) + P ( A k ) = P (A 1 ) + P ( A k ) P ( A 1 A k ) k=2 = P (A 1 ) + 2 i n 2 j n = P 1 P 2 + P 3 P (A i ) P (A 1 A j ) + k=2 2 i<j n 2 j<k n P (A i A j ) + 2 i<j<k n 2 i<j<k n 2 i<j<k n P (A 1 A j A k ) P (A i A j A k ) P (A 1 A i A j A k ) P (A i A j A k ) Example. Suppose n students have n identical raincoats all put together. Each student selects a raincoat at random. 1. What is the probability that at least one raincoat ends up with its original owner? 2. What is the probability that exactly k students select their own raincoats? Let A be the event that at least one raincoat ends up with its original owner. Let A k be the event that the k th student selects his own (the k th ) raincoat. Then A = n A k.

4 4 Every outcome of this experiment consisting of randomly selecting the raincoats is described by a permutation (i 1, i 2,..., i n ), where i k is the index of the raincoat selected by the k th student. For m n, the event A k1 A k2 A km occurs whenever i k1 = k 1, i k2 = k 2,..., i km = k m and the other n m indices of i k take the remaining n m values in any order. (In words, students k 1, k 2,..., k m select their own raincoats.) Therefore P (A k1 A k2 A km ) = N(A k 1 A k2 A km ) N = (n m)! n! for any set of m indices k 1, k 2,..., k m. There are exactly C n m distinct events of the type A k1 A k2 A km. It follows that P m = P (A k1 A k2 1 k 1 <k 2 < <k m n A km ) = Cm n (n m)! = 1 n! m!. Hence P (A) = P ( A k ) = P 1 P ( 1) n+1 P n = 1 1 2! + 1 3! + 1 ( 1)n+1 n!. (This is seen to be the partial sum of the power series expansion of 1 e 1.) Thus, for large n, P (A) 1 e The probability that none of the students selects his own raincoat equals P (A) = 1 P (A) = 1 2! 1 3! + + ( 1)n 1 n!, which is the well-known derangement number of order n. For large n, this is approximately equal to e (An incorrect intuition might leads us to think that P (A) would go to 1 as n goes to infinity.) To obtain the probability of event B such that exactly k students select their own raincoats, we first note that the number of ways in which a particular set of k students selecting their own is equal to the number of ways in which none of the other n k students selects his own raincoat. This is equal to (n k)! [derangement number of order n k] =(n k)! [ 1 2! 1 3! + + ( 1)n k 1 (n k)! ] As there are C n k possible selections of a group of k students, it follows that there are C n k (n k)! [ 1 2! 1 3! + + ( 1)n k 1 (n k)! ] ways in which exactly k of the students select their own raincoats. Thus P (B) = Ck n (n k)! [ 1 2! 1 3! ( 1)n k (n k)! ]/n! n k i=0 = ( 1)i /i!, k! which for n large is approximately e 1 /k!. The values e 1 /k!, k = 0, 1,... are of importance as they represent the values associated with the Possion distribution with parameter equal to unity. This will be elaborated upon when the Poisson approximation is discussed.

5 5 2.1 Continuity Property of Probability Function. Theorem. If A 1, A 2,... is an increasing sequence of events such that A 1 A 2, then denoting lim n A n = k A k, we have P ( k A k ) = P ( lim n A n) = lim n P (A n). Let us define events B 1, B 2,... as follows. B 1 A 1, B 2 A 2 A 1 = A 2 B 1, B 3 A 3 A 2 = A 3 (B 1 B 2 ),..., B n = A n n 1 B k,.... Thus (a) B 1, B 2,... are mutually exclusive, (b) k B k = k A k, and (c) A n = n B k A 3 A 2 A 1 B 1 B 2 B 3 Hence P ( k A k ) = P ( k B k ) = k P (B k ) = lim n = lim P ( n B k ) = lim P (A n). n n n P (B k ) Theorem. If A 1, A 2,... is a decreasing sequence of events such that A 1 A 2, then denoting lim n A n = k A k, we have P ( k A k ) = P ( lim n A n) = lim n P (A n). We have A 1 A 2. Hence P ( k A k ) = P (Ω k A k ) = 1 P ( k A k ) = 1 lim n P (A n ) = lim n [1 P (A n )] = lim n P (A n ). For mutually exclusive events A 1, A 2,..., we know that P ( k A k ) = k P (A k). For arbitrary events, there is the following. Theorem. P ( k A k ) k P (A k) for arbitrary events A 1, A 2,.... Again we define B n = A n n 1 B k as before. Then we have k A k = k B k and B k A k for all k. Hence P ( k A k ) = P ( k B k ) = k P (B k) k P (A k).

6 6 Remark. It is not only for increasing or decreasing sequence of events that we can define a limit. In general, for any sequence of events {A n, n 1}, define (a) lim sup n A n = n=1 i=n A i. [Interpretation: This consists of all points that are contained in an infinite number of the events A n, n 1.] (b) lim inf n A n = n=1 i=n A i. [Interpretation: This consists of all points that are contained in all but a finite number of the events A n, n 1.] Definition. We say that lim n A n exists if lim sup A n = lim inf A n ( lim A n). n n n Theorem. If lim n A n exists, then P (lim n A n ) = lim n P (A n ). (See S.M. Ross, A First Course in Probability, pp ) 2.2 Recurrence of Events. Theorem. (First Borel-Cantelli Lemma) Given a sequence of events A 1, A 2,... with probabilities P (A k ) = p k, k = 1, 2,.... Suppose k p k < converges. Then, with probability 1 only finitely many of the events A 1, A 2,... occur. Let B be the event that infinitely many of the events A 1, A 1,... occur. Let B n = k n A k, i.e. at least one of the events A n, A n+1,... occurs. Then B occurs iff B n occurs for every n = 1, 2,... (That is ωεb iff ωεb 1 B 2 ). Hence B = n B n = n ( k n A k ). Now as B 1 B 2 is a decreasing sequence, we have P (B) = P ( n B n ) = lim n P (B n). But P (B n ) = P ( k n A k ) k n P (A k) = k n p k approaches zero as n approaches. Hence P (B) = lim n P (B n ) = 0, or P (B) = 1 P (B) = 1 as asserted. Theorem. (Second Borel-Cantelli Lemma) Given a sequence of independent events A 1, A 2,... with probabilities P (A k ) = p k, k = 1, 2,.... Suppose k p k = diverges. Then, with probability 1 infinitely many of the events A 1, A 2,... occur. Let B n = k n A k and B = n B n = n ( k n A k ), so that B occurs iff each B n occur (i.e. infintiely many of A 1, A 2,..., occur). Taking complements yields B n = k n A k, B = n B n = n ( k n A k ).

7 7 Now B n = k n A k n+m k=n A k, m = 0, 1, 2,.... Therefore n+m P (B n ) P ( A k ) = P (A n )P (A n+1 ) P (A n+m ) k=n = (1 p n )(1 p n+1 ) (1 p n+m ) n+m e pn e pn+1 e pn+m = exp( p k ), where we used the fact that 1 x e x for x 0. But n+m k=n p k approaches as m approaches, hence P (B n ) = lim m P (B n) lim n+m exp( m k=n k=n Finally P (B) n P (B n) = 0 implies that P (B) = 1 P (B) = 1. p k ) = 0, n = 1, 2,. 3.1 Conditional Probability as a Probability Function. In terms of the dependency of two events A and B, we have: (a) A and B are mutually exclusive, i.e.ab = φ and P (AB) = 0. Hence A never occurs if B does. (b) A has complete dependence on B, i.e. A B and P (A) P (B). Hence A always occurs if B does. (c) A and B are independent, i.e. P (AB) = P (A)P (B). Here the outcomes of A and B are not influenced by that of the other. More generally, we define the conditional probability of A given B (with P (B) > 0) by P (A B) = P (AB)/P (B). Thus in terms of conditional probabilities: (a) P (A B) = 0 if A and B are mutually exclusive. (b) P (A B) = 1 if A is implied by B. (c) P (A B) = P (A) if A and B are independent. Conditional probabilities satisfy all the properties of ordinary probabilities. As a set function, P (A B) satisfies the three axioms while restricting the set of possible outcomes (sample space) to those that are in B. Theorem. Suppose B Ω is not an impossible event, i.e. P (B) > 0. (a) 0 P (A B) 1. (b) P (Ω B) = 1. (c) If A 1, A 2,... are mutually exclusive events, then P ( k A k B) = k P (A k B). (a) We have 0 P (AB)/P (B) P (B)/P (B) = 1. (b) P (ΩB)/P (B) = P (B)/P (B) = 1. (c) We have A 1 B, A 2 B,... being mutually exclusive, hence P ( k A k B)/P (B) = k P (A kb)/p (B) = k P (A k B).

8 8 The upshot is that all properties about ordinary probability function are true for conditional probability function. For example, it easily follows that P (A 1 A 2 B) = P (A 1 B) + P (A 2 B) P (A 1 A 2 B); and P (A 1 B) P (A 2 B) if A 1 A Total Probability Formula and Conditioning Argument. We say that B 1, B 2,... form a full set of events if they are mutually exclusive and collectively exhaustive, i.e. one and only one of B 1, B 2,... always occur and k B k = Ω. A powerful computation tool for the probability of an event is by way of the conditioning argument: P (A) = P (A B k )P (B k ). k The argument is simple. As A = AΩ = k AB k, the total probability formula yields P (A) = P ( k AB k ) = k P (AB k ), since B 1, B 2,... form a full set. In fact, many well-knwon formulas in probability theory are direct consequences of this. Proposition. (Baye s Formula) Suppose B 1, B 2,... form a full set of events, then P (B k A) = P (B ka) P (A) = P (B k)p (A B k ) k P (B k)p (A B k ). Example. (The Gambler s Ruin) Consider the game of calling head or tail on a toss of a fair coin. A correct (incorrect) call wins (loses) a dollar. Suppose the gambler s initial capital is k dollars and the game continues until he either wins an amount of m dollars, stipulated in advance, or else loses all his capital and is ruined. We would like to calculate the probability p(k) that the player will be ruined, starting with 0 < k < m dollars. Let B 1 be the event that the player wins the first call, and B 2 be that he loses the first call. Hence B 1 and B 2 form a full set for one play of the game. Now P (B 1 ) = P (B 2 ) = 1/2. Also P (Ruin B 1 ) = p(k + 1) and P (Ruin B 2 ) = p(k 1). Hence by conditioning on the first call, we have P (Ruin) = P (Ruin B 1 )P (B 1 ) + P (Ruin B 2 )P (B 2 ). That is, p(k) = p(k + 1)/2 + p(k 1)/2, 1 k m 1. Solution to this set of difference equations has the form of p(k) = c 1 + c 2 k. Using the boundary conditions that p(0) = 1 and p(m) = 0, we have c 1 = 1 and c 2 = 1/m. Hence p(k) = 1 k/m, 0 k m, is the required probability. [For general winning probability 0 < p < 1, p(k) = 1 (p/q)m k 1 (p/q) m in Probability, pp ]. See S.M. Ross, A First Course