# Probability & Statistics Primer Gregory J. Hakim University of Washington 2 January 2009 v2.0

Save this PDF as:

Size: px
Start display at page:

Download "Probability & Statistics Primer Gregory J. Hakim University of Washington 2 January 2009 v2.0"

## Transcription

1 Probability & Statistics Primer Gregory J. Hakim University of Washington 2 January 2009 v2.0 This primer provides an overview of basic concepts and definitions in probability and statistics. We shall denote a sample space by S, and define a random variable by the result of a rule (function) that associates a real number with each outcome in S. For simplicity, consider two discrete random variables that are associated with events in the sample space, A and B. The probability of A, denoted by P (A), can be viewed as the frequency of occurrence of the event (the frequentist perspective ). Equivalently, P (A) represents the likelihood of the event occurring; this is called the Bayesian perspective, which will prove useful for interpreting conditional probability. There are three basic probability axioms. Probability axiom #1 states that P (A) is a real number between 0 and 1. The complement of A, A c, is defined as all events in S that are not in A; P (A) = 1 P (A c ). A familiar example derives from rolling a one on an unbiased six-side die, which occurs with probability 1/6. We can check this by rolling the die many times, counting the number of ones, and dividing by the number of rolls. Probability axiom #2 says that P (S) = 1. Probability axiom #3 will be defined below, but basically it says that if we have n outcomes that don t overlap, the probability of all of them occurring is just the sum of the probabilities for each outcome. For discrete random variables, we shall use probability mass functions (PMFs). PMFs return a probability for the discrete random variable, X. For the unbiased six-side die P (X = 6) = 1/6, etc. For continuous variables, we shall use probability density functions (PDFs), denoted by p(x). A PDF is related to a probability through an integral relationship, which is unity over the whole sample space. For example, in one dimension, P ( < x < ) = The probability that x is between a and a + da is P (a x a + da) = a+da a p(x) dx = 1. (1) p(x) dx P (a) da (2) Note that P (a x a+da) may be a very small number even at the peak of a distribution if da is small; however, it will be larger than all others for similarly chosen da and different 1

2 x. Furthermore, it doesn t really make sense to say that, the probability of x = a is P(a) because as (2) shows, this limiting case gives P (x = a) = a a p(x) dx = 0. (3) Union and Intersection Sample space (S): the set of all possible outcomes. six-sided die consists of the integer values 1,2,3,4,5, and 6. P (S) = 1. Example: the sample space for a Figure 1: Venn diagrams for (left) mutually exclusive events (center) intersection or probability of A and B [P (A B)] and (right) union or probability of A or B [P (A B)]. Mutually exclusive events: A and B share no common outcomes (Fig. 1, left panel). Intersection P (A B), A and B : outcome is both A and B (Fig. 1, center panel). If A and B are mutually exclusive, then P (A B) = 0. The intersection is commutative, P (A B) = P (B A), and associative, P (A B C) = P ([A B] C) = P (A [B C]). Union P (A B), A or B : outcome is either A or B. From the Venn diagram in Fig. 1 (right panel) we can see that the probability of A or B is the sum of the individual probabilities, minus the intersection, which gets added twice: P (A B) = P (A) + P (B) P (A B). (4) If A and B are mutually exclusive, then P (A B) = P (A)+P (B); i.e. sum the probabilities. The union is commutative, P (A B) = P (B A), and associative, P (A B C) = P ([A B] C) = P (A [B C]). NOTE: Union and intersection are not associative: P ([A B] C) P (A [B C]), but they do satisfy the distributive property: P (A [B C]) = P ([A B] [A C]) and P (A [B C]) = P ([A B] [A C]). 2

3 We may now state Probability axiom #3: given n mutually exclusive events A 1, A 2,..., A n, then P (A 1 A 2... A n ) = n i=1 P (A i). Conditional probability Conditional probability is denoted by P (A B) and read as the probability of A given that B has occurred. Given that B has occurred, the sample space shrinks from S to B. We expect that P (A B) P (A B), since the sample space is now just B. Since P (B B) must be unity and P (B B) = P (B), the constant of proportionality must be 1/P (B): P (A B) = P (A B). (5) P (B) Similarly P (B A) = P (A B) P (A) (6) Using (6) to replace P (A B) in (5) gives Bayes Theorem, P (A B) = P (B A) P (A). (7) P (B) Note that if two events are mutually exclusive, then P (A B) = P (A B) = 0. Extending to three events (see Fig. 2 below), we expect that the probability of A given that B and C have occurred, P (A B C) or simply P (A B, C), is proportional to the probability of the intersection of A, B, and C, P (A B C) or simply P (A, B, C): P (A BC) = P (A, B, C) P (B, C), (8) where the constant of proportionality is deduced by similar logic to that for (5). A similar expression applies for P (B A, C): P (B A, C) = P (A, B, C) P (A, C). (9) Substituting P (A, B, C) from (9) into (8) gives P (A B, C) = P (B A, C)P (A, C). (10) P (B, C) Using the fact [i.e., (5)] that P (A, C) = P (A C)P (C) and P (B, C) = P (B C)P (C) yields P (A B, C) = P (B A, C)P (A C). (11) P (B C) 3

4 Figure 2: Probability of A and B and C [ P (A B C) = P (A, B, C)]. More generally, the chain rule of conditional probability allows one to factor joint probabilities. Previously we discussed the product rule for two variables, P (A, B) = P (A B)P (B) = P (B A)P (A), (12) and a similar result for three variables, P (A, B, C) = P (A B, C)P (B, C) = P (A B, C)P (B C)P (C). (13) Extending to n variables, P (A 1, A 2,..., A n ) = P (A 1 A 2,..., A n )P (A 2 A 3,..., A n ) P (A n 1 A n )P (A n ). (14) A Markov Chain, denoted A B C, is defined by a joint probability that factors as P (A, B, C) = P (C B)P (B A)P (A), which implies that C is independent of A. If A, B, C occur sequentially in time (as implied by the arrows in the definition), this means that in order to determine the probability at the new time, C, we only need to know the probability of the current time, B. Marginal probability Marginal probability represents the unconditional probability of one or more independent random variables without considering information from other random variables; e.g. consider P (A) instead of P (A, B). For discrete random variables, we obtain P (A) from P (A, B) by summation over the unwanted variable in the joint probability mass function: P (A) = P (A, B) = P (A B)P (B). Similarly, for continuous random variables, marginal B B probability density functions are defined by integrating over the unwanted variables in the joint probability density function; e.g., p(y) = p(x, y)dy = p(x y)p(y)dy. Likelihood 4 y y

5 With conditional probability, we are given specific information about an event that affects the probability of another event; e.g., knowing that B = b allows us to update the probability of A by P (A B = b) through (7). Conversely, if we view the conditional probability as a function of the second argument, B, we get a likelihood function, L(b A) = αp (A B = b), (15) where α is a constant parameter. For example, consider a sequence of observations of a two-sided coin with outcomes H and T. If we flip the coin once and observe H we can ask, what is the likelihood that the true probability, P (H) is 0.5? Fig. 3 (left panel) shows that L = 0.5, and in fact the most likely value is P(H) = 1; note that L(0) = 0 since H has been observed and therefore must have a nonzero probability. Suppose flipping the coin again also gives H. Maximum likelihood is still located at P (H) = 1, and the likelihood of smaller P (H) is diminished. Remember, that we can turn this around and ask, if we know that P (H) = 0.5 (i.e., the coin is fair), what is the probability of observing HH? This is conditional probability, which in this case is 0.25 (the same as L(0.5 HH ). Observing HHH gives a cubic function for L, and so one. If, on the other hand, one of the observations is also T, then both L(0) and L(1) must be zero (Fig. 3; right panel). Figure 3: Likelihood as a function of the probability of H on a two-sided coin. On the left, the first three trials give only H, whereas on the right T is observed as well. Note that, unlike conditional probability, likelihood is not a probability, since it may take values larger than unity, and does not integrate to one. Independence. Events A and B are independent if the outcome of one does not depend on the outcome of the other; i.e. P (A B) = P (A). (16) 5

6 Appealing to (5), (16) implies that P (A B) = P (A)P (B), (17) i.e., the outcome A and B is determined by a product of probabilities. This is a great simplification that is often made as a leading-order approximation. Of course it is perfect in some cases, as for example in rolling two dice, where the outcome on one die has no affect on the other; the probability of rolling a six on each is (1/6) (1/6). Note that independence can not be gleaned from examining a Venn Diagram. A general multiplication rule for independent events following directly from (16): P (A 1 A 2 A n ) = P (A 1 )P (A 2 ) P (A n ) (18) Given n mutually exclusive and exhaustive [ n i=1 P (A i) = 1] events A 1,..., A n, the Law of Total Probability. says that for some other event B or, equivalently P (B) = P (B A 1 )P (A 1 ) + + P (B A n )P (A n ) (19) P (B) = n P (B A i )P (A i ). (20) i=1 Since the A i are mutually exclusive, we just need to add up the intersections of each with B, and since they are exhaustive there is no probability unaccounted for by the A i. Note that we may use this result in the denominators of (5) and (7). Properties of Probability Density Functions (PDFs) Cumulative distribution function (CDF): F (x) = x p(y) dy. (21) The expected value of a PDF is equivalent to the center of mass in classical physics. It is determined by the expectation operator, E[ ], defined by E[x] = The expectation operator is linear, e.g. E[ax + by] = ae[x] + be[y]. x p(x) dx. (22) 6

7 The k th moment about the expected value is defined by µ k = (x µ) k p(x) dx, (23) where µ = E[x]. Note that the first moment is zero, and that the second moment is called the variance, and is usually denoted by σ 2 ; the standard deviation is defined as the square-root of the variance, σ. The covariance between two random variables is defined by cov(x, y) = E[(x µ)(y ν)] = E[xy] µν, (24) where µ and ν are E[x] and E[y], respectively. The covariance determines the linear relationship between two quantities, and the strength of this relationship is measured by the correlation coefficient, corr(x, y) = cov(x, y) σ x σ y, (25) which ranges between 1 and +1. If x and y are independent, then cov(x, y) = corr(x, y) = 0. Note that these definitions for PDFs extend naturally to multidimensional settings, and also to discrete random variables; in the discrete case, integrals are replace by sums. Gaussian (Normal) Probability Density Function Although there are many analytically defined PDFs, the Gaussian is particularly important in many areas of science. The assumption of Gaussian statistics is often invoked because it greatly simplifies the analysis, since all moments (23) of the PDF may be determined from the mean and covariance. A theoretical underpinning for this widespread use of the Gaussian PDF is given by the Central Limit Theorem, which states that the sum of many independent, identically distributed random variables is normally distributed (i.e., the mean value approaches nµ and the variance approaches nσ 2 for sufficiently large n); note that identically distributed means have the same distribution. In one dimension, the Gaussian (Normal) distribution is defined by p(x) = (2π) 1/2 (σ) 1 e 1 (x µ) 2 2 σ 2, (26) where µ is the expected value and σ 2 is the variance. A shorthand notation to denote a variable having a Gaussian distribution is x N(µ, σ 2 ). 7

8 Properties of the Gaussian include: If x is normally distributed, then so is a linear function of x. y = ax + b N(aµ + b, a 2 σ 2 ). If x N(µ, σ 2 ) then The normal sum (difference) distribution: adding (subtracting) different Gaussian distributions yields a Gaussian. Multivariate PDFs and Conditional Probability Set union discussed on page 2 for discrete variables generalizes to joint probability density functions for continuous random variables, p(x). For example, the n-dimensional generalization of the Gaussian distribution is p(x) = (2π) n/2 ( B ) 1/2 e 1 2 (x µ)t B 1 (x µ), (27) where x are µ are vectors and B is the covariance matrix. The vector x has one entry for each of the n degrees of freedom, as does µ, which represents the vector of expected values. The covariance matrix, real and symmetric, is defined by B = E[(x µ)(x µ) T ], and is the determinant. Diagonal elements of B represent the variance in each degree of freedom, and the off-diagonal elements represent cov(x i, x j ); in the case where all degrees of freedom are independent, B is diagonal. The superscripted T denotes the transpose. In the limiting case of n = 1, (27) recovers (26). Each property of the one-dimensional Gaussian has a multi-dimensional extension. One useful example is the linear transformation of the n 1 Gaussian random vector, x N(µ, B), by the m n matrix operator, A, Y = Ax, (28) where Y N(Aµ, ABA T ). The generalization of Bayes Theorem to multivariate continuous random variables is given by p(x y) = f(x, y) f(y) = p(y x) p(x). (29) p(y) Equivalently, using the definition for joint probability, (29) may be expressed as p(x y) = p(x, y) p(y). (30) To help illustrate conditional probability, consider a two-dimensional example for Gaussian variables x and y having zero mean, correlation ρ, and standard deviation σ x and σ y, 8

9 respectively. The conditional probability of y given x can be shown from (30) and (27) to be p(y x) = 1 e 1 (y µ) 2 2 σ 2 2πσ where µ = ρxσ y /σ x and σ = σ y (1 ρ 2 ) 1/2 are the mean and standard deviation, respectively, of the conditional probability density. Note that the from the definition of correlation, the mean can be re-written as µ = cov(x, y)x/σ x 2, which is the linear regression of y on x. Note also that the standard deviation in the conditional probability density is smaller than in the original, unconditional, density by the factor (1 ρ 2 ) 1/2, and therefore higher correlation results in lower variance in the conditional PDF. These results are illustrated in the figure below. Red lines in the main panel show contours of constant probability density, which reflect greater variance in the x variable compared to the y variable. Black lines in the subplots below and to the right show the marginal probabilities; that is, the integral over the other dimension. Suppose that new information becomes available showing that x = 1, as indicated by the thin blue line in the main panel. Applying Bayes theorem gives the blue curve in the right panel. The green line shows the linear regression of y on x, which intersects the blue line right at the peak of the conditional PDF shown in the right panel. Furthermore, note from the results derived above that the variance of the conditional PDF is independent of x, so that for other values of x, the conditional PDF shown in the right panel simply shifts along the y axis, following the green line, without change of shape. In contrast to the conditional PDF, the thin red line shows the value of the joint PDF at x = 1. First, notice that the area under the blue and red lines is different: the blue curve is a true probability density function, with unity integral, whereas the red line is not. The thin red line denotes the likelihood function, which is proportional to, but distinct from, conditional probability. Recall from the earlier discussion that, in the case of conditional probability, we are given information, in this case x = 1, that affects our knowledge of y as expressed by p(y x = 1). For likelihood, the first argument is held fixed and we have a function of the second argument, L(y = a x) = p(x y = a); i.e., if x = 1, what is the likelihood that y = 0? Notice for the example above that maximum likelihood occurs at the point of maximum conditional probability; they differ by a constant of proportionality. 9

10 10

### Joint Probability Distributions and Random Samples (Devore Chapter Five)

Joint Probability Distributions and Random Samples (Devore Chapter Five) 1016-345-01 Probability and Statistics for Engineers Winter 2010-2011 Contents 1 Joint Probability Distributions 1 1.1 Two Discrete

### L10: Probability, statistics, and estimation theory

L10: Probability, statistics, and estimation theory Review of probability theory Bayes theorem Statistics and the Normal distribution Least Squares Error estimation Maximum Likelihood estimation Bayesian

### Joint Probability Distributions and Random Samples. Week 5, 2011 Stat 4570/5570 Material from Devore s book (Ed 8), and Cengage

5 Joint Probability Distributions and Random Samples Week 5, 2011 Stat 4570/5570 Material from Devore s book (Ed 8), and Cengage Two Discrete Random Variables The probability mass function (pmf) of a single

### Jointly Distributed Random Variables

Jointly Distributed Random Variables COMP 245 STATISTICS Dr N A Heard Contents 1 Jointly Distributed Random Variables 1 1.1 Definition......................................... 1 1.2 Joint cdfs..........................................

### Bivariate Distributions

Chapter 4 Bivariate Distributions 4.1 Distributions of Two Random Variables In many practical cases it is desirable to take more than one measurement of a random observation: (brief examples) 1. What is

### Examination 110 Probability and Statistics Examination

Examination 0 Probability and Statistics Examination Sample Examination Questions The Probability and Statistics Examination consists of 5 multiple-choice test questions. The test is a three-hour examination

### For a partition B 1,..., B n, where B i B j = for i. A = (A B 1 ) (A B 2 ),..., (A B n ) and thus. P (A) = P (A B i ) = P (A B i )P (B i )

Probability Review 15.075 Cynthia Rudin A probability space, defined by Kolmogorov (1903-1987) consists of: A set of outcomes S, e.g., for the roll of a die, S = {1, 2, 3, 4, 5, 6}, 1 1 2 1 6 for the roll

### Probability and Statistics Prof. Dr. Somesh Kumar Department of Mathematics Indian Institute of Technology, Kharagpur

Probability and Statistics Prof. Dr. Somesh Kumar Department of Mathematics Indian Institute of Technology, Kharagpur Module No. #01 Lecture No. #15 Special Distributions-VI Today, I am going to introduce

### Summary of Formulas and Concepts. Descriptive Statistics (Ch. 1-4)

Summary of Formulas and Concepts Descriptive Statistics (Ch. 1-4) Definitions Population: The complete set of numerical information on a particular quantity in which an investigator is interested. We assume

### E3: PROBABILITY AND STATISTICS lecture notes

E3: PROBABILITY AND STATISTICS lecture notes 2 Contents 1 PROBABILITY THEORY 7 1.1 Experiments and random events............................ 7 1.2 Certain event. Impossible event............................

### 15.062 Data Mining: Algorithms and Applications Matrix Math Review

.6 Data Mining: Algorithms and Applications Matrix Math Review The purpose of this document is to give a brief review of selected linear algebra concepts that will be useful for the course and to develop

### Covariance and Correlation. Consider the joint probability distribution f XY (x, y).

Chapter 5: JOINT PROBABILITY DISTRIBUTIONS Part 2: Section 5-2 Covariance and Correlation Consider the joint probability distribution f XY (x, y). Is there a relationship between X and Y? If so, what kind?

### Topic 4: Multivariate random variables. Multiple random variables

Topic 4: Multivariate random variables Joint, marginal, and conditional pmf Joint, marginal, and conditional pdf and cdf Independence Expectation, covariance, correlation Conditional expectation Two jointly

### MT426 Notebook 3 Fall 2012 prepared by Professor Jenny Baglivo. 3 MT426 Notebook 3 3. 3.1 Definitions... 3. 3.2 Joint Discrete Distributions...

MT426 Notebook 3 Fall 2012 prepared by Professor Jenny Baglivo c Copyright 2004-2012 by Jenny A. Baglivo. All Rights Reserved. Contents 3 MT426 Notebook 3 3 3.1 Definitions............................................

### ST 371 (VIII): Theory of Joint Distributions

ST 371 (VIII): Theory of Joint Distributions So far we have focused on probability distributions for single random variables. However, we are often interested in probability statements concerning two or

### Chapter 1. Probability, Random Variables and Expectations. 1.1 Axiomatic Probability

Chapter 1 Probability, Random Variables and Expectations Note: The primary reference for these notes is Mittelhammer (1999. Other treatments of probability theory include Gallant (1997, Casella & Berger

### MULTIVARIATE PROBABILITY DISTRIBUTIONS

MULTIVARIATE PROBABILITY DISTRIBUTIONS. PRELIMINARIES.. Example. Consider an experiment that consists of tossing a die and a coin at the same time. We can consider a number of random variables defined

### Master s Theory Exam Spring 2006

Spring 2006 This exam contains 7 questions. You should attempt them all. Each question is divided into parts to help lead you through the material. You should attempt to complete as much of each problem

### Machine Learning Math Essentials

Machine Learning Math Essentials Jeff Howbert Introduction to Machine Learning Winter 2012 1 Areas of math essential to machine learning Machine learning is part of both statistics and computer science

### Chapter 4 Lecture Notes

Chapter 4 Lecture Notes Random Variables October 27, 2015 1 Section 4.1 Random Variables A random variable is typically a real-valued function defined on the sample space of some experiment. For instance,

### ECE302 Spring 2006 HW7 Solutions March 11, 2006 1

ECE32 Spring 26 HW7 Solutions March, 26 Solutions to HW7 Note: Most of these solutions were generated by R. D. Yates and D. J. Goodman, the authors of our textbook. I have added comments in italics where

### Joint Exam 1/P Sample Exam 1

Joint Exam 1/P Sample Exam 1 Take this practice exam under strict exam conditions: Set a timer for 3 hours; Do not stop the timer for restroom breaks; Do not look at your notes. If you believe a question

### Estimation with Minimum Mean Square Error

C H A P T E R 8 Estimation with Minimum Mean Square Error INTRODUCTION A recurring theme in this text and in much of communication, control and signal processing is that of making systematic estimates,

### Data Modeling & Analysis Techniques. Probability & Statistics. Manfred Huber 2011 1

Data Modeling & Analysis Techniques Probability & Statistics Manfred Huber 2011 1 Probability and Statistics Probability and statistics are often used interchangeably but are different, related fields

### Section 6.1 Joint Distribution Functions

Section 6.1 Joint Distribution Functions We often care about more than one random variable at a time. DEFINITION: For any two random variables X and Y the joint cumulative probability distribution function

### A Tutorial on Probability Theory

Paola Sebastiani Department of Mathematics and Statistics University of Massachusetts at Amherst Corresponding Author: Paola Sebastiani. Department of Mathematics and Statistics, University of Massachusetts,

### Change of Continuous Random Variable

Change of Continuous Random Variable All you are responsible for from this lecture is how to implement the Engineer s Way (see page 4) to compute how the probability density function changes when we make

### Chapters 5. Multivariate Probability Distributions

Chapters 5. Multivariate Probability Distributions Random vectors are collection of random variables defined on the same sample space. Whenever a collection of random variables are mentioned, they are

### Overview of Violations of the Basic Assumptions in the Classical Normal Linear Regression Model

Overview of Violations of the Basic Assumptions in the Classical Normal Linear Regression Model 1 September 004 A. Introduction and assumptions The classical normal linear regression model can be written

### 3 Multiple Discrete Random Variables

3 Multiple Discrete Random Variables 3.1 Joint densities Suppose we have a probability space (Ω, F,P) and now we have two discrete random variables X and Y on it. They have probability mass functions f

### Exercises with solutions (1)

Exercises with solutions (). Investigate the relationship between independence and correlation. (a) Two random variables X and Y are said to be correlated if and only if their covariance C XY is not equal

### Discrete Mathematics and Probability Theory Fall 2009 Satish Rao,David Tse Note 11

CS 70 Discrete Mathematics and Probability Theory Fall 2009 Satish Rao,David Tse Note Conditional Probability A pharmaceutical company is marketing a new test for a certain medical condition. According

### Normal distribution Approximating binomial distribution by normal 2.10 Central Limit Theorem

1.1.2 Normal distribution 1.1.3 Approimating binomial distribution by normal 2.1 Central Limit Theorem Prof. Tesler Math 283 October 22, 214 Prof. Tesler 1.1.2-3, 2.1 Normal distribution Math 283 / October

### Random variables, probability distributions, binomial random variable

Week 4 lecture notes. WEEK 4 page 1 Random variables, probability distributions, binomial random variable Eample 1 : Consider the eperiment of flipping a fair coin three times. The number of tails that

### Some probability and statistics

Appendix A Some probability and statistics A Probabilities, random variables and their distribution We summarize a few of the basic concepts of random variables, usually denoted by capital letters, X,Y,

### Probability and Statistics

CHAPTER 2: RANDOM VARIABLES AND ASSOCIATED FUNCTIONS 2b - 0 Probability and Statistics Kristel Van Steen, PhD 2 Montefiore Institute - Systems and Modeling GIGA - Bioinformatics ULg kristel.vansteen@ulg.ac.be

### MATHEMATICS (CLASSES XI XII)

MATHEMATICS (CLASSES XI XII) General Guidelines (i) All concepts/identities must be illustrated by situational examples. (ii) The language of word problems must be clear, simple and unambiguous. (iii)

### Mathematical Background

Appendix A Mathematical Background A.1 Joint, Marginal and Conditional Probability Let the n (discrete or continuous) random variables y 1,..., y n have a joint joint probability probability p(y 1,...,

### Statistiek (WISB361)

Statistiek (WISB361) Final exam June 29, 2015 Schrijf uw naam op elk in te leveren vel. Schrijf ook uw studentnummer op blad 1. The maximum number of points is 100. Points distribution: 23 20 20 20 17

### ELEC-E8104 Stochastics models and estimation, Lecture 3b: Linear Estimation in Static Systems

Stochastics models and estimation, Lecture 3b: Linear Estimation in Static Systems Minimum Mean Square Error (MMSE) MMSE estimation of Gaussian random vectors Linear MMSE estimator for arbitrarily distributed

### Sections 2.11 and 5.8

Sections 211 and 58 Timothy Hanson Department of Statistics, University of South Carolina Stat 704: Data Analysis I 1/25 Gesell data Let X be the age in in months a child speaks his/her first word and

### Lecture Note 1 Set and Probability Theory. MIT 14.30 Spring 2006 Herman Bennett

Lecture Note 1 Set and Probability Theory MIT 14.30 Spring 2006 Herman Bennett 1 Set Theory 1.1 Definitions and Theorems 1. Experiment: any action or process whose outcome is subject to uncertainty. 2.

### Probability & Probability Distributions

Probability & Probability Distributions Carolyn J. Anderson EdPsych 580 Fall 2005 Probability & Probability Distributions p. 1/61 Probability & Probability Distributions Elementary Probability Theory Definitions

### Notes for STA 437/1005 Methods for Multivariate Data

Notes for STA 437/1005 Methods for Multivariate Data Radford M. Neal, 26 November 2010 Random Vectors Notation: Let X be a random vector with p elements, so that X = [X 1,..., X p ], where denotes transpose.

### Lecture Notes 1. Brief Review of Basic Probability

Probability Review Lecture Notes Brief Review of Basic Probability I assume you know basic probability. Chapters -3 are a review. I will assume you have read and understood Chapters -3. Here is a very

### The Big 50 Revision Guidelines for S1

The Big 50 Revision Guidelines for S1 If you can understand all of these you ll do very well 1. Know what is meant by a statistical model and the Modelling cycle of continuous refinement 2. Understand

### 4. Joint Distributions of Two Random Variables

4. Joint Distributions of Two Random Variables 4.1 Joint Distributions of Two Discrete Random Variables Suppose the discrete random variables X and Y have supports S X and S Y, respectively. The joint

### Basic concepts in probability. Sue Gordon

Mathematics Learning Centre Basic concepts in probability Sue Gordon c 2005 University of Sydney Mathematics Learning Centre, University of Sydney 1 1 Set Notation You may omit this section if you are

### P (x) 0. Discrete random variables Expected value. The expected value, mean or average of a random variable x is: xp (x) = v i P (v i )

Discrete random variables Probability mass function Given a discrete random variable X taking values in X = {v 1,..., v m }, its probability mass function P : X [0, 1] is defined as: P (v i ) = Pr[X =

### Math/Stats 425 Introduction to Probability. 1. Uncertainty and the axioms of probability

Math/Stats 425 Introduction to Probability 1. Uncertainty and the axioms of probability Processes in the real world are random if outcomes cannot be predicted with certainty. Example: coin tossing, stock

### Diagonal, Symmetric and Triangular Matrices

Contents 1 Diagonal, Symmetric Triangular Matrices 2 Diagonal Matrices 2.1 Products, Powers Inverses of Diagonal Matrices 2.1.1 Theorem (Powers of Matrices) 2.2 Multiplying Matrices on the Left Right by

### ST 371 (IV): Discrete Random Variables

ST 371 (IV): Discrete Random Variables 1 Random Variables A random variable (rv) is a function that is defined on the sample space of the experiment and that assigns a numerical variable to each possible

### Chapter 3: The basic concepts of probability

Chapter 3: The basic concepts of probability Experiment: a measurement process that produces quantifiable results (e.g. throwing two dice, dealing cards, at poker, measuring heights of people, recording

### We have discussed the notion of probabilistic dependence above and indicated that dependence is

1 CHAPTER 7 Online Supplement Covariance and Correlation for Measuring Dependence We have discussed the notion of probabilistic dependence above and indicated that dependence is defined in terms of conditional

### Joint Distributions. Tieming Ji. Fall 2012

Joint Distributions Tieming Ji Fall 2012 1 / 33 X : univariate random variable. (X, Y ): bivariate random variable. In this chapter, we are going to study the distributions of bivariate random variables

### 4: Probability. What is probability? Random variables (RVs)

4: Probability b binomial µ expected value [parameter] n number of trials [parameter] N normal p probability of success [parameter] pdf probability density function pmf probability mass function RV random

### Covariance and Correlation

Covariance and Correlation ( c Robert J. Serfling Not for reproduction or distribution) We have seen how to summarize a data-based relative frequency distribution by measures of location and spread, such

### Probability and Statistics Vocabulary List (Definitions for Middle School Teachers)

Probability and Statistics Vocabulary List (Definitions for Middle School Teachers) B Bar graph a diagram representing the frequency distribution for nominal or discrete data. It consists of a sequence

### Stat260: Bayesian Modeling and Inference Lecture Date: February 1, Lecture 3

Stat26: Bayesian Modeling and Inference Lecture Date: February 1, 21 Lecture 3 Lecturer: Michael I. Jordan Scribe: Joshua G. Schraiber 1 Decision theory Recall that decision theory provides a quantification

### A crash course in probability and Naïve Bayes classification

Probability theory A crash course in probability and Naïve Bayes classification Chapter 9 Random variable: a variable whose possible values are numerical outcomes of a random phenomenon. s: A person s

### PROBABILITIES AND PROBABILITY DISTRIBUTIONS

Published in "Random Walks in Biology", 1983, Princeton University Press PROBABILITIES AND PROBABILITY DISTRIBUTIONS Howard C. Berg Table of Contents PROBABILITIES PROBABILITY DISTRIBUTIONS THE BINOMIAL

### TRANSFORMATIONS OF RANDOM VARIABLES

TRANSFORMATIONS OF RANDOM VARIABLES 1. INTRODUCTION 1.1. Definition. We are often interested in the probability distributions or densities of functions of one or more random variables. Suppose we have

### Probability OPRE 6301

Probability OPRE 6301 Random Experiment... Recall that our eventual goal in this course is to go from the random sample to the population. The theory that allows for this transition is the theory of probability.

### Outline. Random Variables. Examples. Random Variable

Outline Random Variables M. Sami Fadali Professor of Electrical Engineering University of Nevada, Reno Random variables. CDF and pdf. Joint random variables. Correlated, independent, orthogonal. Correlation,

### MATRIX ALGEBRA AND SYSTEMS OF EQUATIONS

MATRIX ALGEBRA AND SYSTEMS OF EQUATIONS Systems of Equations and Matrices Representation of a linear system The general system of m equations in n unknowns can be written a x + a 2 x 2 + + a n x n b a

### + Section 6.2 and 6.3

Section 6.2 and 6.3 Learning Objectives After this section, you should be able to DEFINE and APPLY basic rules of probability CONSTRUCT Venn diagrams and DETERMINE probabilities DETERMINE probabilities

### Multivariate Normal Distribution

Multivariate Normal Distribution Lecture 4 July 21, 2011 Advanced Multivariate Statistical Methods ICPSR Summer Session #2 Lecture #4-7/21/2011 Slide 1 of 41 Last Time Matrices and vectors Eigenvalues

### Statistics 100 Binomial and Normal Random Variables

Statistics 100 Binomial and Normal Random Variables Three different random variables with common characteristics: 1. Flip a fair coin 10 times. Let X = number of heads out of 10 flips. 2. Poll a random

### 1. The sample space S is the set of all possible outcomes. 2. An event is a set of one or more outcomes for an experiment. It is a sub set of S.

1 Probability Theory 1.1 Experiment, Outcomes, Sample Space Example 1 n psychologist examined the response of people standing in line at a copying machines. Student volunteers approached the person first

### m (t) = e nt m Y ( t) = e nt (pe t + q) n = (pe t e t + qe t ) n = (qe t + p) n

1. For a discrete random variable Y, prove that E[aY + b] = ae[y] + b and V(aY + b) = a 2 V(Y). Solution: E[aY + b] = E[aY] + E[b] = ae[y] + b where each step follows from a theorem on expected value from

### DISCRETE RANDOM VARIABLES

DISCRETE RANDOM VARIABLES DISCRETE RANDOM VARIABLES Documents prepared for use in course B01.1305, New York University, Stern School of Business Definitions page 3 Discrete random variables are introduced

### Lesson 5 Chapter 4: Jointly Distributed Random Variables

Lesson 5 Chapter 4: Jointly Distributed Random Variables Department of Statistics The Pennsylvania State University 1 Marginal and Conditional Probability Mass Functions The Regression Function Independence

### Statistics - Written Examination MEC Students - BOVISA

Statistics - Written Examination MEC Students - BOVISA Prof.ssa A. Guglielmi 26.0.2 All rights reserved. Legal action will be taken against infringement. Reproduction is prohibited without prior consent.

### Common probability distributionsi Math 217/218 Probability and Statistics Prof. D. Joyce, 2016

Introduction. ommon probability distributionsi Math 7/8 Probability and Statistics Prof. D. Joyce, 06 I summarize here some of the more common distributions used in probability and statistics. Some are

### ( ) = P Z > = P( Z > 1) = 1 Φ(1) = 1 0.8413 = 0.1587 P X > 17

4.6 I company that manufactures and bottles of apple juice uses a machine that automatically fills 6 ounce bottles. There is some variation, however, in the amounts of liquid dispensed into the bottles

### Stat 704 Data Analysis I Probability Review

1 / 30 Stat 704 Data Analysis I Probability Review Timothy Hanson Department of Statistics, University of South Carolina Course information 2 / 30 Logistics: Tuesday/Thursday 11:40am to 12:55pm in LeConte

### Chapter 4. Multivariate Distributions

1 Chapter 4. Multivariate Distributions Joint p.m.f. (p.d.f.) Independent Random Variables Covariance and Correlation Coefficient Expectation and Covariance Matrix Multivariate (Normal) Distributions Matlab

### Correlation in Random Variables

Correlation in Random Variables Lecture 11 Spring 2002 Correlation in Random Variables Suppose that an experiment produces two random variables, X and Y. What can we say about the relationship between

### Lecture 9: Introduction to Pattern Analysis

Lecture 9: Introduction to Pattern Analysis g Features, patterns and classifiers g Components of a PR system g An example g Probability definitions g Bayes Theorem g Gaussian densities Features, patterns

### Probability for Estimation (review)

Probability for Estimation (review) In general, we want to develop an estimator for systems of the form: x = f x, u + η(t); y = h x + ω(t); ggggg y, ffff x We will primarily focus on discrete time linear

### Slides for Risk Management VaR and Expected Shortfall

Slides for Risk Management VaR and Expected Shortfall Groll Seminar für Finanzökonometrie Prof. Mittnik, PhD Groll (Seminar für Finanzökonometrie) Slides for Risk Management Prof. Mittnik, PhD 1 / 133

### Welcome to Stochastic Processes 1. Welcome to Aalborg University No. 1 of 31

Welcome to Stochastic Processes 1 Welcome to Aalborg University No. 1 of 31 Welcome to Aalborg University No. 2 of 31 Course Plan Part 1: Probability concepts, random variables and random processes Lecturer:

### Basics of Probability

Basics of Probability 1 Sample spaces, events and probabilities Begin with a set Ω the sample space e.g., 6 possible rolls of a die. ω Ω is a sample point/possible world/atomic event A probability space

### A Primer on Mathematical Statistics and Univariate Distributions; The Normal Distribution; The GLM with the Normal Distribution

A Primer on Mathematical Statistics and Univariate Distributions; The Normal Distribution; The GLM with the Normal Distribution PSYC 943 (930): Fundamentals of Multivariate Modeling Lecture 4: September

### Lecture 6: Discrete & Continuous Probability and Random Variables

Lecture 6: Discrete & Continuous Probability and Random Variables D. Alex Hughes Math Camp September 17, 2015 D. Alex Hughes (Math Camp) Lecture 6: Discrete & Continuous Probability and Random September

### Slides for Risk Management

Slides for Risk Management VaR and Expected Shortfall Groll Seminar für Finanzökonometrie Prof. Mittnik, PhD 1 Introduction Value-at-Risk Expected Shortfall Model risk Multi-period / multi-asset case 2

### Lecture 8: Signal Detection and Noise Assumption

ECE 83 Fall Statistical Signal Processing instructor: R. Nowak, scribe: Feng Ju Lecture 8: Signal Detection and Noise Assumption Signal Detection : X = W H : X = S + W where W N(, σ I n n and S = [s, s,...,

### THE MULTINOMIAL DISTRIBUTION. Throwing Dice and the Multinomial Distribution

THE MULTINOMIAL DISTRIBUTION Discrete distribution -- The Outcomes Are Discrete. A generalization of the binomial distribution from only 2 outcomes to k outcomes. Typical Multinomial Outcomes: red A area1

### The Bivariate Normal Distribution

The Bivariate Normal Distribution This is Section 4.7 of the st edition (2002) of the book Introduction to Probability, by D. P. Bertsekas and J. N. Tsitsiklis. The material in this section was not included

### Important Probability Distributions OPRE 6301

Important Probability Distributions OPRE 6301 Important Distributions... Certain probability distributions occur with such regularity in real-life applications that they have been given their own names.

### The sample space for a pair of die rolls is the set. The sample space for a random number between 0 and 1 is the interval [0, 1].

Probability Theory Probability Spaces and Events Consider a random experiment with several possible outcomes. For example, we might roll a pair of dice, flip a coin three times, or choose a random real

### MATRIX ALGEBRA AND SYSTEMS OF EQUATIONS. + + x 2. x n. a 11 a 12 a 1n b 1 a 21 a 22 a 2n b 2 a 31 a 32 a 3n b 3. a m1 a m2 a mn b m

MATRIX ALGEBRA AND SYSTEMS OF EQUATIONS 1. SYSTEMS OF EQUATIONS AND MATRICES 1.1. Representation of a linear system. The general system of m equations in n unknowns can be written a 11 x 1 + a 12 x 2 +

### DECISION MAKING UNDER UNCERTAINTY:

DECISION MAKING UNDER UNCERTAINTY: Models and Choices Charles A. Holloway Stanford University TECHNISCHE HOCHSCHULE DARMSTADT Fachbereich 1 Gesamtbibliothek Betrtebswirtscrtaftslehre tnventar-nr. :...2>2&,...S'.?S7.

### Math 431 An Introduction to Probability. Final Exam Solutions

Math 43 An Introduction to Probability Final Eam Solutions. A continuous random variable X has cdf a for 0, F () = for 0 <

### Probability. A random sample is selected in such a way that every different sample of size n has an equal chance of selection.

1 3.1 Sample Spaces and Tree Diagrams Probability This section introduces terminology and some techniques which will eventually lead us to the basic concept of the probability of an event. The Rare Event

### Lesson 1. Basics of Probability. Principles of Mathematics 12: Explained! www.math12.com 314

Lesson 1 Basics of Probability www.math12.com 314 Sample Spaces: Probability Lesson 1 Part I: Basic Elements of Probability Consider the following situation: A six sided die is rolled The sample space

### Section 5.1 Continuous Random Variables: Introduction

Section 5. Continuous Random Variables: Introduction Not all random variables are discrete. For example:. Waiting times for anything (train, arrival of customer, production of mrna molecule from gene,

### MASSACHUSETTS INSTITUTE OF TECHNOLOGY 6.436J/15.085J Fall 2008 Lecture 5 9/17/2008 RANDOM VARIABLES

MASSACHUSETTS INSTITUTE OF TECHNOLOGY 6.436J/15.085J Fall 2008 Lecture 5 9/17/2008 RANDOM VARIABLES Contents 1. Random variables and measurable functions 2. Cumulative distribution functions 3. Discrete