Outline. Why Study Probability Theory? CPSC 531: Basic Probability Concepts

Transcription

1 CPSC 531: Basic Probability Concepts Instructor: Anirban Mahanti Office: ICT References: 1. Introduction to Probability Models by Sheldon Ross, Academic Press, Eight Edition, Probability and Statistics by M. DeGroot and M. Schervish, Third edition, Addison Wesley, Probability and Random Processes for Electrical and Computer Engineers by John Gubner, Cambridge Press, First Edition, Basic Probability Concepts 1 Outline Experiments, sample space, and events Review of set theory Probability: definition, property, interpretation Conditional probability Independent events Law of total probability and Bayes Counting Methods Basic Probability Concepts 2 Why Study Probability Theory? Probability theory is useful in understanding, studying, and analyzing complex real world systems Probability theory can be used to model and develop complex real world systems A good understanding of probability theory is needed to develop simulations E.g., How would you model network traffic? Should we use Markovian models? What is the impact of self similarity on design and analysis of communication networks Basic Probability Concepts 3 1

2 Experiment Experiment in probability theory refers to a process whose outcome is not known in advance with certainty E.g., Suppose a coin is tossed 10 times. How many times will we get heads? Experience tells us that, if the coin is fair, we will see on average 5 heads ; we can arrive at this result by performing the experiment many times and noting down the observations Instead of performing the experiments, we can use probability theory to develop a model of any system that yields uncertain measurements Basic Probability Concepts 4 Sample space, Outcome, Event Sample space (S) is the collection of all possible outcomes of a system or experiment. E.g., Rolling a six-sided dice can be modeled with the sample space S = {1, 2, 3, 4, 5, 6} E.g., Model of the number of packets queued at a router that can buffer up to 50 packets would use the sample space S = {0, 1, 2,, 50} Outcome is element in the sample space E.g., 2, 4, and 5 are outcomes in the rolling dice example Event is a subset of the sample space E.g., A is an event that an even number is rolled: A = {2, 4, 6} Basic Probability Concepts 5 Review of Set Theory Basic Probability Concepts 6 2

3 Countable vs. Uncountable Sets Some sets contain only finite number of elements, while others contain infinitely many elements. An infinite set can be classified as: Countable - if there is a one-to-one correspondence between elements of the set and the set of natural numbers E.g., the set of numbers from the roll of a dice E.g., the set of all odd numbers E.g., the set of all prime numbers Uncountable converse of countable; e.g., set of real numbers, numbers in the interval [0, 1] E.g., set of possible temperatures on Jan 1 in a city Basic Probability Concepts 7 Set Operations (1) Consider a set of points: S Let A and B be two collection of points in S Every point in A is also in B, then A B If A B and B A, then A = B Empty set: Ø is the null set Complement: A C is the complement of set A and contains all outcomes in the sample space S that do not belong to A Basic Probability Concepts 8 Set Operations (2) Union: A B = a set that consists of points that belong either to A or to B, or to both A B := {w Є S: w Є A or w Є B} Intersection: A B = a set that consists of points that belongs to both A and B A B := {w Є S: w Є A and w Є B} Two sets are said to be disjoint or mutually exclusive if A B = Ø Two sets are said to be mutually exclusive and exhaustive if A B = S and A B = Ø Basic Probability Concepts 9 3

4 Set Identities (1) Commutative laws A B = B A A B = B A Distributive laws A (B C) = (A B) (A C) A (B C) = (A B) (A C) Associative laws (A B) C = A (B C) = (A B) C (A B) C = A (B C) = (A B) C Basic Probability Concepts 10 Set Identities (2) Identity laws A Ø= A A U= A A U= U A Ø= Ø De Morgan s laws (A B) C = A C B C (A B) C = A C B C Basic Probability Concepts 11 Definitions of Probability Basic Probability Concepts 12 4

5 Classical Definition In the classical approach, the probability of an event A) is assigned a priori without performing any experiments. The probability of an event A is the ratio of the number of outcomes N A favorable to event A to the total number N of possible outcomes of the experiment. P (A) = N A N Basic Probability Concepts 13 Relative Frequency Definition Suppose n trials of an experiment are undertaken. In these trials, an event A occurs n A times. The probability of event A, A), can be defined as follows: P (A) = lim n n A n Note that the probability is assigned after (a posterior) the experiments. Basic Probability Concepts 14 Axiomatic Definition A probability measure, or simply a probability, on a sample space S is a specification of numbers.) that satisfies the following axioms: 1) 0 A) 1 for every event A; probability of an event A is a number between 0 and 1. 2) S) = 1; the probability of an outcome being a sample point in the sample space is 1. 3) For every infinite sequence of mutually exclusive events A 1, A 2, P = U Ai Ai ) i= 1 i= 1 Basic Probability Concepts 15 5

6 Probability Models (1) Probability theory helps develop mathematical models of complicated problems Example: Develop a mathematical model of rolling a fair die to determine probability of an even face. Let S = {1, 2, 3, 4, 5, 6} be the sample space. Let E i = {i}, i = 1, 2,, 6 be the event that die rolled the face i. Let E = {2, 4, 6} model the die rolling an even face. Here the classical definition of probability is used. E) = E / S = 1/2 Basic Probability Concepts 16 Probability Models (2) Some observations about the probability model: Ø) = Ø / S = 0. A) 0 for every event A. If A and B are disjoint events, then A B) = A) + B) When the die is rolled, something happens; this is represented by the fact that S) = 1. Our example considered the countable finite case. What about countable infinite outcomes? Let S = {1, 2, }. For w Є S, let p(w) be non-negative and p( w) = 1. w= 1 For any A S, A) = p(w). Basic Probability Concepts 17 Probability Models (3) In many cases, the number of possible outcomes are uncountable. E.g., the duration of Web browsing sessions, the duration of a phone call, the heights of adults in a city etc. We can define P as follows: Z P (A) = f(w)dw, A S A for some non-negative function f such that: f ( w) dw = 1. S Basic Probability Concepts 18 6

7 Properties of Probability (Complement) For any event A, A C ) = 1 A) (Impossible Set) Ø) = 0 (Monotonicity rule) If A B, A) B) (Inclusion-Exclusion rule) Given two events A and B, A U B) = A) + B) A B). Other rules can be derived. The inclusionexclusion rule can be extended as follows: Suppose A, B, and C are events in S. Then A B C) =P (A)+P (B)+P (C) P (A B) P (A C) P (B C)+P (A B C) Basic Probability Concepts 19 Example (1) Select one ball from a box containing white (W), red (R), blue (B), and green (G) balls. Suppose that R) = 0.1 and B) = 0.5. What is the probability of selecting a white or a green ball? Solution: W U R U B U G = S From axiom 3, we have S) = W) + R) + B) + G) Also, WUG) = W)+G) = 1 R) B) = 0.4 Basic Probability Concepts 20 Example (2) Rolling a fair dice: 1) = 2) = 3) = 4) = 5) = 6) = 1/6, where i) is the probability of rolling a face with i dots. Question: {1, 3}) =? {2, 4, 6}) =? Basic Probability Concepts 21 7

8 Conditional Probability Basic Probability Concepts 22 Conditional Probability (1) The conditional probability of event A given that event B has occurred is A B) A B) = B) The conditional probability A B) is undefined if B)=0 The frequency interpretation: If an experiment is repeated a large number of times, then event B occurs approximately B) fraction of the time and the event A and B both occur approximately A B) portion of time. Therefore, among the repetitions in which B occurs, the proportion of events in which A also occurs is A B) /B). A B) also denoted as AB). Basic Probability Concepts 23 Conditional Probability (2) Example: Two balls are selected at random, without replacement, from a bag containing r red balls and b blue balls. Determine probability p that the first selected ball is red and the second selected ball is blue. Solution A = event that first ball is red; B = event that second ball is blue A) = prob. first ball is red = r/(r+b) B A) = prob. blue in second draw = b/(r+b-1) B A) = AB)/A) AB) = (r.b)/((r+b)(r+b-1)) Basic Probability Concepts 24 8

9 Conditional Probability (3) AB) = A) A B); extended, as shown next Suppose that A 1, A 2, A 3,, A n are events such that B) > 0 and A 1 A 2 A 3 A n-1 ) > 0. Then, A 1 A 2 A 3 A n ) = A 1 )A 2 A 1 ) A 3 A 1 A 2 ) A n A 1 A 2 A 3 A n-1 ) Example: Draw 4 balls, one by one, without replacement, from a box containing r red balls and b blue balls. Probability of sequence red, red, blue, blue? Solution: Let R j and B j denote event that a red and blue ball is drawn, respectively. R 1 R 2 B 3 B 4 ) = R 1 )R 2 R 1 )B 3 R 1 R 2 ) B 4 R 1 R 2 B 3 ) complete the calculations on your own Basic Probability Concepts 25 Independent Events (1) Independent events are those that don t have any effect on each other. That is, knowing one of them occurs does not provide any information about the occurrence of the other event. Mathematically, A and B are independent if A B) = A) and B A) = B) From conditional probability definition, we have AB) = A B)B). Therefore, if A and B are independent events, AB) = A)B) Basic Probability Concepts 26 Independent Events (2) Example: Determine the probability p n that exactly n tosses are required for a head to appear for the first time. Solution We need to determine the probability of a sequence of (n-1) tails followed by a head. The probability of obtaining a heads or tails (for a fair coin) is ½. Therefore, p n = (½) n. Basic Probability Concepts 27 9

10 Law of Total Probability and Bayes Rule Basic Probability Concepts 28 Law of Total Probability Let B 1, B 2, B 3,, B k be mutually disjoint and collectively exhaustive events from the sample space S. Then, for any event A in S, we have k Explanation A = (B 1 A)U (B 2 A)U (B k A). The (B j A) s are disjoint events. Therefore, using laws of conditional probability we get: k k A) = B j A) = B j ) A B j ), j= 1 j= 1 if B j ) > 0 for j = 1,..., k A) = B j ) A B j ) j= 1 B 1 A B 2 B 4 B 3 Basic Probability Concepts 29 Bayes Theorem Partition: The events B 1, B 2, B 3,, B k form a partition of a set S if they are mutually disjoint and k B S Ui= i = 1 Bayes Theorem: Suppose that B 1, B 2, B 3,, B k form a partition of sample space S such that B j )> 0 for j = 1,, k. Let A be an event in S such that A)>0. Then, for i = 1,, k, Bi ) A Bi ) Bi A) = k j = 1 B j ) A B j ) Basic Probability Concepts 30 10

11 Counting Methods Basic Probability Concepts 31 Types of Counting Problems Two types of counting problems: Selection problems select k balls from an urn containing n distinguishable balls Allocation problems allocate k indistinguishable balls to a set of n distinguishable urns Consider the selection problem How do we replace selected balls? With replacement put ball back into urn Without replacement How to count events? Permutation order of selection important Combination order of selection unimportant Basic Probability Concepts 32 Permutations (1) The number of ordered ways k items can be selected without replacement from a pool of n items is: P n,k = n x (n-1) x X (n-k+1) = n!/(n-k)! The number of ordered ways k items can be selected with replacement from a pool of n items is: P r n,k = n x n x (k times) = n k Basic Probability Concepts 33 11

12 Permutations (2) When not all items are ordered: E.g., a club has 25 members and 2 officers (a president and a secretary). These officers are to be chosen from the members of the club. How many ways can the officers be selected? 25 ways to select the first officer and 24 ways to select the second officer. Answer = 25x24 = 600 Basic Probability Concepts 34 Permutations (3) What is the probability that at least two people share the same birthday? You may assume the following: 365 possible birthdays (ignore leap years) Our group consists of 2 k 365 people who are unrelated (no twins) Each birthday is equally likely (birth rate independent of the time of year) Basic Probability Concepts 35 Combinations (1) The number of combinations possible when selecting k items from a set of n items without replacement is denoted by C n,k. Note that one way to find P n,k is to compute C n,k (i.e., unordered selection of k items) and multiply by the number of ways of ordering k items (i.e., k!). Therefore, P n,k = C n,k x k! Pn, k n! Cn, k = = k! k!( n k)! Basic Probability Concepts 36 12

13 Combinations (2) The number of combinations possible when selecting k items from a set of n items with replacement is denoted by C r n,k. C r n,k = C n+k-1,k. Basic Probability Concepts 37 Combinations (3) Example: A fair coin is tossed 10 times. Determine the probability, p, of obtaining exactly 3 heads. Solution Each experiment has two possible outcome, heads or tails. Since the coins are fair, we can assume that each outcome is equally likely. Therefore, the total number of outcomes in this experiment is = 2 10 The number of different arrangements possible with 3 heads and 7 tails is = C 10,3 Therefore, 10, p = C = Basic Probability Concepts 38 Summary A review of basic probability and statistics was presented. You probably enjoyed it! Next, we will talk about discrete and continuous random variables. Basic Probability Concepts 39 13