STAT/MTHE 353: Probability II STAT/MTHE 353: Multiple Random Variables Administrative details Instructor: TamasLinder Email: linder@mast.queensu.ca T. Linder ueen s University Winter 2012 O ce: Je ery 401 Phone: 613-533-2417 O ce hours: Wednesday 2:30 3:30 pm Class web site: http://www.mast.queensu.ca/ stat353 All homework and solutions will be posted here. Check frequently for new announcements STAT/MTHE 353: Multiple Random Variables 1/ 34 STAT/MTHE 353: Multiple Random Variables 2/ 34 Review Tet: Fundamentals of Probability with Stochastic Processes, 3rd ed., bys.ghahramani,prenticehall. Lecture slides will be posted on the class web site. The slides are not self-contained; they only cover parts of the material. Homework: 9 HW assignments. Homework due Friday before noon in my mailbo (Je ery 401). No late homework will be accepted! Evaluation: the better of Homework 20%, midterm 20%, final eam 60% Homework 20%, final eam 80% Midterm Eam: Thursday, February 16 in class (9:30-10:30 am) S is the sample space; P is a probability measure on S: P is a function from a collection of subsets of S (called the events) to [0, 1]. P satisfies the aioms of probability; A random variable is a function : S! R. Thedistribution of is the probability measure associated with : P ( 2 A) P ({s : (s) 2 A}), for any reasonable A R. STAT/MTHE 353: Multiple Random Variables 3/ 34 STAT/MTHE 353: Multiple Random Variables 4/ 34
Here are the usual ways to describe the distribution of : Distribution function: It is always well defined. F : R! [0, 1] defined by F () P ( apple ). Probability mass function, orpmf:if is a discrete random variable, thenitspmfp : R! [0, 1] is p () P ( ), for all 2 R. Note: :since is discrete, there is a countable set R such that p () 0if /2. Probability density function, orpdf:if is a continuous random variable, thenitspdff : R! [0, 1) is a function such that P ( 2 A) f() d for all reasonable A R. A Joint Distributions If 1,..., n are random variables (defined on the same probability space), we can think of ( 1,..., n ) T as a random vector. (In this course ( 1,..., n ) is a row vector and its transpose, ( 1,..., n ) T, is a column vector.) Thus is a function : S! R n. Distribution of : For reasonable A R n,wedefine P ( 2 A) P ({s : (s) 2 A}). is called a random vector or vector random variable. STAT/MTHE 353: Multiple Random Variables 5/ 34 STAT/MTHE 353: Multiple Random Variables 6/ 34 We usually describe the distribution of by a function on R n : Joint cumulative distribution function (jcdf) is a the function defined for ( 1,..., n ) 2 R n by F () F 1,..., n ( 1,..., n ) P ( 1 apple 1, 2 apple 2,..., n apple n ) P ({ 1 apple 1 } \ { 2 apple 2 } \ \ { n apple n }) P 2 n ( 1, i ] i1 If 1,..., n are all discrete random variables, then their joint probability mass function (jpmf) is p () P ( ) P ( 1 1, 2 2,..., n n ), 2 R n The finite or countable set of values such that p () > 0 is called the support of the distribution of. STAT/MTHE 353: Multiple Random Variables 7/ 34 Properties of joint pmf: (1) 0 apple p () apple 1 for all 2 R n. (2) 2 p () 1,where is the support of. If 1,..., n are continuous random variables and there eists f : R n! [0, 1) such that for any reasonable A R n, P ( 2 A) f ( 1,..., n ) d 1 d n then The 1,..., n are called jointly continuous; A f () f 1,..., n ( 1,..., n ) is called the joint probability density function (jpdf) of. STAT/MTHE 353: Multiple Random Variables 8/ 34
Properties of joint pdf: Comments: (a) The joint pdf can be redefined on any set in R n that has zero volume. This will not change the distribution of. (b) The joint pdf may not eists even when each 1,..., n are all (individually) continuous random variables. Eample:... (1) f () 0 for all 2 R n (2) f () d R n R n f ( 1,..., n ) d 1 d n 1 The distributions for the various subsets of { 1,..., n } can be recovered from the joint distribution. These distributions are called the joint marginal distributions (here marginal is relative to the full set { 1,..., n }. STAT/MTHE 353: Multiple Random Variables 9/ 34 STAT/MTHE 353: Multiple Random Variables 10 / 34 Marginal joint probability mass functions Assume 1,..., n are discrete. Let 0 <k<nand {i 1,...,i k } {1,...,n} Then the marginal joint pmf of ( i1,..., ik ) can be obtained from p () p 1,..., n ( 1,..., n ) as p i1,..., ik ( i1,..., ik ) P ( i1 i1,..., ik ik ) P ( i1 i1,..., ik ik, j1 2 R,..., jn k 2 R) where {j 1,...,j n k } {1,...,n}\{i 1,...,i k } p 1,..., n ( 1,..., n ) j1 jn k Eample: In an urn there are n i objects of type i for i 1,...,r.The total number of objects is n 1 + + n r N. We randomly draw n objects (n apple N) without replacement. Let i #of objects of type i drawn. Find the joint pmf of ( 1,..., r ). Also find the marginal distribution of each i, i 1,...,r. Thus the joint pmf of i1,..., ik is obtained by summing p 1,..., n over all possible values of the complementary variables j1,..., jn k. STAT/MTHE 353: Multiple Random Variables 11 / 34 STAT/MTHE 353: Multiple Random Variables 12 / 34
Marginal joint probability density functions Let 1,..., n be jointly continuous with pdf f f. As before, let {i 1,...,i k } {1,...,n}, {j 1,...,j n k } {1,...,n}\{i 1,...,i k } Let B R k.then P ( i1,..., ik ) 2 B P ( i1,..., ik ) 2 B, j1 2 R,..., jn k 2 R 0 1 @ f( 1,..., n ) d j1 d jn A d k i1 d ik B R n k In conclusion, for {i 1,...,i k } {1,...,n}, f i1,..., ik ( i1,..., ik ) where {j 1,...,j n k } {1,...,n}\{i 1,...,i k }, R n k f( 1,..., n ) d j1 d jn k Note: In both the discrete and continuous cases it is important to always know where the joint pmf p and joint pdf f are zero and where they are positive. The latter set is called the support of p or f. That is, we integrate out the variables complementary to i1,..., ik. STAT/MTHE 353: Multiple Random Variables 13 / 34 STAT/MTHE 353: Multiple Random Variables 14 / 34 Eample: Suppose 1, 2, 3 are jointly continuous with jpdf 8 < 1 if 0 apple i apple 1, i 1, 2, 3 f( 1, 2, 3 ) : 0 otherwise Find the marginal pdfs of i, i 1, 2, 3, and the marginal jpdfs of ( i, j ), i 6 j. Eample: With 1, 2, 3 as in the previous problem, consider the quadratic equation 1 y 2 + 2 y + 3 0 in the variable y. Find the probability that both roots are real. Marginal joint cumulative distribution functions In all cases (discrete, continuous, or mied), F i1,..., ik ( i1,..., ik ) P ( i1 apple i1,..., ik apple ik ) P ( i1 apple i1,..., ik apple ik, j1 < 1,..., jn k < 1) lim lim F 1,..., n ( 1,..., n ) j1!1 jn k!1 That is, we let the variables complementary to i1,..., ik converge to 1 STAT/MTHE 353: Multiple Random Variables 15 / 34 STAT/MTHE 353: Multiple Random Variables 16 / 34
Independence Definition The random variables 1,..., n are independent if for all reasonable A 1,...,A n R, P ( 1 2 A 1,..., n 2 A n )P ( 1 2 A 1 ) P ( n 2 A n ) Remarks: (i) Independence among 1,..., n usually arises in a probability model by assumption. Such an assumption is reasonable if the outcome of i has no e ect on the outcomes of the other j s. (ii) The also definition applies to any n random quantities 1,..., n. E.g., each i can itself be a vector r.v. In this case the A i s have to be appropriately modified. (iii) Suppose g i : R! R, i 1,...,n are reasonable functions. Then if 1,..., n are independent, then so are g 1 ( 1 ),...,g n ( n ). Proof For A 1,...,A n R, P g 1 ( 1 ) 2 A 1,...,g n ( n ) 2 A n P 1 2 g 1 1 (A 1),..., n 2 g 1 (A n ) where g 1 i (A i ){ i : g i ( i ) 2 A i } P 1 2 g 1 1 (A 1) P n 2 g 1 (A n ) P g 1 ( 1 ) 2 A 1 P g n ( n ) 2 A n ) Since the sets A i were arbitrary, we obtain that g 1 ( 1 ),...,g n ( n ) are independent. Note: If we only know that i and j are independent for all i 6 j, it does not follow that 1,..., n are independent. STAT/MTHE 353: Multiple Random Variables 17 / 34 STAT/MTHE 353: Multiple Random Variables 18 / 34 Independence and cdf, pmf, pdf Theorem 1 Let F be the joint cdf of the random variables 1,..., n.then 1,..., n are independent if and only if F is the product of the marginal cdfs of the i,i.e.,forall( 1,..., n ) 2 R n, F ( 1,..., n )F 1 ( 1 )F 2 ( 2 ) F n ( n ) Theorem 2 Let 1,..., n be discrete r.v. s with joint pmf p. Then 1,..., n are independent if and only if p is the product of the marginal pmfs of the i,i.e.,forall( 1,..., n ) 2 R n, p( 1,..., n )p 1 ( 1 )p 2 ( 2 ) p n ( n ) Proof: If 1,..., n are independent, then Proof: If 1,..., n are independent, then F ( 1,..., n ) P ( 1 apple 1, 2 apple 2,..., n apple n ) P ( 1 apple 1 )P ( 2 apple 2 ) P ( n apple n ) F 1 ( 1 )F 2 ( 2 ) F n ( n ) p( 1,..., n ) P ( 1 1, 2 2,..., n n ) P ( 1 1 )P ( 2 2 ) P ( n n ) p 1 ( 1 )p 2 ( 2 ) p n ( n ) The converse that F ( 1,..., n ) n F i ( i ) for all ( 1,..., n ) 2 R n i1 implies independence is out the the scope of this class. STAT/MTHE 353: Multiple Random Variables 19 / 34 STAT/MTHE 353: Multiple Random Variables 20 / 34
Proof cont d: Conversely, suppose that p( 1,..., n ) n p i ( i ) for any 1,..., n. Then, for any A 1,A 2,...,A n R, P ( 1 2 A 1,..., n 2 A n ) 12A 1 12A 1 p 1 ( 1 ) 12A 1! i1 p( 1,..., n ) n2a n p 1 ( 1 ) p n ( n ) n2a n!! p 2 ( 2 ) p n ( n ) 22A 2 n2a n P ( 1 2 A 1 )P ( 2 2 A 2 ) P ( n 2 A n ) Thus 1,..., n are independent. Theorem 3 Let 1,..., n be jointly continuous r.v. s with joint pdf f. Then 1,..., n are independent if and only if f is the product of the marginal pdfs of the i,i.e.,forall( 1,..., n ) 2 R n, f( 1,..., n )f 1 ( 1 )f 2 ( 2 ) f n ( n ). Proof: Assumef( 1,..., n ) n f i ( i ) for any 1,..., n.then for any A 1,A 2,...,A n R, P ( 1 2 A 1,..., n 2 A n ) i1 f( 1,..., n ) d 1 d n A 1 A n f 1 ( 1 ) f n ( n ) d 1 d n A 1 A n!! f 1 ( 1 ) d 1 f n ( n ) d n A 1 A n P ( 1 2 A 1 )P ( 2 2 A 2 ) P ( n 2 A n ) STAT/MTHE 353: Multiple Random Variables 21 / 34 so 1,..., n are independent. STAT/MTHE 353: Multiple Random Variables 22 / 34 Proof cont d: For the converse, note that F ( 1,..., n ) P ( 1 apple 1, 2 apple 2,..., n apple n ) 1 n 1 1 By the fundamental theorem of calculus f(t 1,...,t n ) dt 1 dt n @ n @ 1 @ n F ( 1,..., n )f( 1,..., n ) Eample:... (assuming f is nice enough ). If 1,..., n are independent, then F ( 1,..., n ) n F i ( i ). Thus @ n f( 1,..., n ) F ( 1,..., n ) @ 1 @ n @ n F 1 ( 1 ) F n ( n ) @ 1 @ n f 1 ( 1 ) f n ( n ) i1 STAT/MTHE 353: Multiple Random Variables 23 / 34 STAT/MTHE 353: Multiple Random Variables 24 / 34
Epectations Involving Multiple Random Variables Recall that the epectation of a random variable is 8 >< p() E() 1 >: f() d 1 if is discrete if is continuous if the sum or the integral eist in the sense that P p() < 1 or R 1 f() d < 1. 1 Eample:... STAT/MTHE 353: Multiple Random Variables 25 / 34 If ( 1,..., n ) T is a random vector, we sometimes use the notation E() E( 1 ),...,E( n ) T For 1,..., n discrete, we still have E() P p() with the understanding that p() ( 1,..., n) ( 1,..., n ) T p( 1,..., n ) 1 p 1 ( 1 ), 2 p 2 ( 2 ),..., n p n ( n ) 1 2 n E( 1 ),...,E( n ) T Similarly, for jointly continuous 1,..., n, E() f() d R n ( 1,..., n ) T f( 1,..., n ) d 1 d n R n STAT/MTHE 353: Multiple Random Variables 26 / 34 T Theorem 4 ( Law of the unconscious statistician ) Suppose Y g() for some function g : R n! R k.then 8 >< g()p() E(Y ) >: g()f() d R n if 1,..., n are discrete if 1,..., n are jointly continuous Proof: We only prove the discrete case. Since ( 1,..., n ) can only take a countable number of values with positive probability, the same is true for (Y 1,...,Y k ) T Y g() so Y 1,...,Y k are discrete random variables. Proof cont d: Thus E(Y ) yp (Y y) yp (g() y) y y y P ( ) y :g()y g()p ( ) y :g()y g()p ( ) g()p() Eample: Linearity of epectation... E(a 0 + a 1 1 + a n n )a 0 + a 1 E( 1 )++ a n E( n ) STAT/MTHE 353: Multiple Random Variables 27 / 34 STAT/MTHE 353: Multiple Random Variables 28 / 34
Transformation of Multiple Random Variables Eample: Epected value of a binomial random variable... Eample: Suppose we have n bar magnets, each having negative polarity at one end and positive polarity at the other end. Line up the magnets end-to-end in such a way that the orientation of each magnet is random (the two choices are equally likely independently of the others). On the average, how many segments of magnets that stick together do we obtain? Suppose 1,..., n are jointly continuous with joint pdf f( 1,..., n ). Let h : R n! R n be a continuously di erentiable and one-to-one (invertible) function whose inverse g is also continuously di erentiable. Thus h is given by ( 1,..., n ) 7! (y 1,...,y n ),where y 1 h 1 ( 1,..., n ), y 2 h 2 ( 1,..., n ),..., y n h n ( 1,..., n ) We want to find the joint pdf of the vector Y (Y 1,...,Y n ) T, where Y 1 h 1 ( 1,..., n ) Y 2 h 2 ( 1,..., n ). Y n h n ( 1,..., n ) STAT/MTHE 353: Multiple Random Variables 29 / 34 STAT/MTHE 353: Multiple Random Variables 30 / 34 Let g : R n! R n denote the inverse of h. LetB R n be a nice set. We have P (Y 1,...,Y n ) 2 B P h() 2 B P ( 1,..., n ) 2 A where A { 2 R n : h() 2 B} h 1 (B) g(b). The multivariate change of variables formula for g(y) implies that P ( 1,..., n ) 2 A f( 1,..., n ) d 1 d n B B g(b) f g 1 (y 1,...,y n ),...,g n (y 1,...,y n ) J g (y 1,...,y n ) dy 1 dy n f(g(y)) J g (y) dy We have shown that for nice B R n P (Y 1,...,Y n ) 2 B This implies the following: B f(g(y)) J g (y) dy Theorem 5 (Transformation of Multiple Random Variables) Suppose 1,..., n are jointly continuous with joint pdf f( 1,..., n ). Let h : R n! R n be a continuously di erentiable and one-to-one function with continuously di erentiable inverse g. Then the joint pdf of Y (Y 1,...,Y n ) T h() is f Y (y 1,...,y n )f g 1 (y 1,...,y n ),...,g n (y 1,...,y n ) J g (y 1,...,y n ). where J g is the Jacobian of the transformation g. STAT/MTHE 353: Multiple Random Variables 31 / 34 STAT/MTHE 353: Multiple Random Variables 32 / 34
Eample: Suppose ( 1,..., n ) T has joint pdf f and let Y A, wherea is an invertible n n (real) matri. Find f Y. Often we are interested in the pdf of just a single function of 1,..., n,sayy 1 h 1 ( 1,..., n ). (1) Define Y i h i ( 1,..., n ), i 2,...,n in such a way that the mapping h (h 1,...,h n ) satisfies the conditions of the theorem (h has an inverse g which is continuously di erentiable). Then the theorem gives the joint pdf f Y (y 1,...,y n ) and we obtain f Y1 (y 1 ) by integrating out y 2,...,y n : f Y1 (y 1 ) f Y (y 1,...,y n ) dy 2...dy n R n 1 (2) Often it is easier to directly compute the cdf of Y 1 : F Y1 (y) P (Y 1 apple y) P h 1 ( 1,..., n ) apple y P ( 1,..., n ) 2 A y where A y {( 1,..., n ):h( 1,..., n ) apple y} f( 1,..., n ) d 1 d n A y Di erentiating F Y1 we obtain the pdf of Y 1. Eample: Let 1,..., n be independent with common distribution Uniform(0, 1). Determine the pdf of Y min( 1,..., n ). A common choice is Y i i, i 2,...,n. STAT/MTHE 353: Multiple Random Variables 33 / 34 STAT/MTHE 353: Multiple Random Variables 34 / 34