Random Variables, Expectation, Distributions

Transcription

1 Random Variables, Expectation, Distributions CS 5960/6960: Nonparametric Methods Tom Fletcher January 21, 2009

2 Review

3 Random Variables Definition A random variable is a function defined on a probability space. In other words, if (Ω, F, P) is a probability space, then a random variable is a function X : Ω V for some set V. Note: A random variable is neither random nor a variable. We will deal with integer-valued (V = Z) or real-valued (V = R) random variables. Technically, random variables are measurable functions.

4 Dice Example Let (Ω, F, P) be the probability space for rolling a pair of dice, and let X : Ω Z be the random variable that gives the sum of the numbers on the two dice. So, X[(1, 2)] = 3, X[(4, 4)] = 8, X[(6, 5)] = 11

5 Even Simpler Example Most of the time the random variable X will just be the identity function. For example, if the sample space is the real line, Ω = R, the identity function is a random variable. X : R R, X(s) = s

6 Defining Events via Random Variables Setting a real-valued random variable to a value or range of values defines an event. [X = x] = {s Ω : X(s) = x} [X < x] = {s Ω : X(s) < x} [a < X < b] = {s Ω : a < X(s) < b}

7 Cumulative Distribution Functions Definition Let X be a real-valued random variable on the probability space (Ω, F, P). Then the cumulative distribution function (cdf) of X is defined as F X (x) = P(X < x)

8 Properties of CDFs Let X be a real-valued random variable. Then F X has the following properties: 1. F X is monotonic increasing. 2. F X is right-continuous, that is, lim F X(x + ɛ) = F X (x), for all x R. ɛ lim x F X(x) = 0 and lim x F X (x) = 1.

9 Probability Mass Functions (Discrete) Definition The probability mass function (pmf) for a discrete real-valued random variable X, denoted f X, is defined as f X (x) = P(X = x). The cdf can be defined in terms of the pmf as F X (x) = P(X x) = k x f X (k).

10 Probability Density Functions (Continuous) Definition The probability density function (pdf) for a continuous real-valued random variable X, denoted f X, is defined as when this derivative exists. f X (x) = d dx F X(x), The cdf can be defined in terms of the pdf as F X (x) = P(X x) = x f X (t)dt.

11 Example: Uniform Distribution X Unif(0, 1) X is uniformly distributed between 0 and 1. f X (x) = { 1 0 x 1 0 otherwise 0 x < 0 F X (x) = x 0 x 1 1 x > 1

12 Joint Distributions Recall that given two events A, B, we can talk about the intersection of the two events A B and the probability P(A B) of both events happening. Given two random variables, X, Y, we can also talk about the intersection of the events these variables define. The distribution defined this way is called the joint distribution: F X,Y (x, y) = P(X x; Y y) = y x f X,Y (s, t)dsdt.

13 Marginal Distributions Definition Given a joint probability density f X,Y, the marginal densities of X and Y are given by f X (x) = f Y (y) = f X,Y (x, y)dy, f X,Y (x, y)dx. and

14 Conditional Densities Definition If X, Y are random variables with joint density f X,Y, then the conditional density of X given Y = y is f X Y=y (x) = f X,Y(x, y). f Y (y)

15 Independent Random Variables Definition Two random variables X, Y are called independent if f X,Y (x, y) = f X (x)f Y (y). If we integrate (or sum) both sides, we see this is equivalent to F X,Y (x, y) = F X (x)f Y (y).

16 Expectation Definition The expectation of a random variable X is E[X] = x f X (x)dx. This is the mean value of X, also denoted µ X = E[X].

17 Linearity of Expectation If X and Y are random variables, and a, b R, then E[aX + by] = ae[x] + be[y]. This extends the several random variables X i and constants a i : [ N ] N E a i X i = a i E[X i ]. i=1 i=1

18 Variance Definition The variance of a random variable X is defined as Var(X) = E[(X µ X ) 2 ]. This formula is equivalent to Var(X) = E[X 2 ] µ 2 X. The variance is a measure of the spread of the distribution. The standard deviation is the sqrt of variance: σ X = Var(X).

19 Example: Normal Distribution X N(µ, σ) X is normally distributed with mean µ and standard deviation σ. f X (x) = 1 2πσ exp F X (x) = x ( ) (x µ)2 2σ 2 f X (t)dt

20 Expectation of the Product of Two RVs We can take the expected value of the product of two random variables, X and Y: E[XY] = xy f X,Y (x, y)dxdy.

21 Covariance Definition The covariance of two random variables X and Y is Cov(X, Y) = E[(X µ X )(Y µ Y )] = E[XY] µ X µ Y. This is a measure of how much the variables X and Y change together. We ll also write σ XY = Cov(X, Y).

22 Correlation Definition The correlation of two random variables X and Y is ρ(x, Y) = σ XY σ X σ Y, or [( ) ( X µx Y µy ρ(x, Y) = E σ X σ Y )]. Correlation normalizes the covariance between [ 1, 1].

23 Independent RVs are Uncorrelated If X and Y are two independent RVs, then E[XY] = = = x f X (x)dx xy f X,Y (x, y)dxdy xy f X (x)f Y (y)dxdy = E[X]E[Y] = µ X µ Y So, σ XY = E[XY] µ X µ Y = 0. y f Y (y)dy

24 More on Indepdence and Correlation Warning: Independence implies uncorrelation, but uncorrelated variables are not necessarily independent!