Ex. 2.1 (Davide Basilio Bartolini)

Size: px
Start display at page:

Download "Ex. 2.1 (Davide Basilio Bartolini)"

Transcription

1 ECE 54: Elements of Information Theory, Fall 00 Homework Solutions Ex.. (Davide Basilio Bartolini) Text Coin Flips. A fair coin is flipped until the first head occurs. Let X denote the number of flips required. (a) Find the Entropy H(X) in bits (b) A random variable X is drawn according to this distribution. Find an efficient sequence of yes-no questions of the form, Is X contained in the set S?. Compare H(X) to the expected number of questions required to determine X. Solution (a) The random variable X is on the domain X = {,,,...} and it denotes the number of flips needed to get the first head, i.e. + the number of consecutive tails appeared before the first head. Since the coin is said to be fair, we have p( head ) = p( tail ) = and hence (exploiting the independence of the coin flips): p(x = ) = p( head ) = p(x = ) = p( tail ) p( head ) = ( =. ntimes { }} { p(x = n) = p( tail )... p( tail ) p( head ) =... ( = ) ) n from this, it is clear that the probability mass distribution of X is: p X (x) = ( ) x

2 Once the distribution is known, H(X) can be computed from the definition: H(X) = p X (x) log p X (x) x X ( ) x ( ) x = log x= ( ) x ( ) x = log (since the summed expr. equals 0 for x = 0) ( ) x ( ) = x log (property of logarithms) ( ) ( ) x = log x ( ) ( ) x = x = ( ) = [bit] exploiting (k) x k x = ( k) (b) Since the most likely value for X is (p(x = ) = ), the most efficient first question is: Is X =? ; the next question will be Is X =? and so on, until a positive answer is found. If this strategy is used, the random variable Y representing the number of questions will have the same distribution as X and it will be: E [Y ] = y y=0 ( ) y = ( ) = which is exactly equal to the entropy of X. An interpretation of this fact could be that bits (which is the entropy value for X) are the amount of memory required to store the outcomes of the two binary questions which are enough (on average) to get a positive answer on the value of X. Exercise.4 (Matteo Carminati) Entropy of functions of a random variable. Let X be a discrete random variable. Show that the entropy of a function of X is less than or equal to the entropy of X by justifying the following steps: H(X, g(x)) (a) = H(X) + H(g(X) X) (b) = H(X) () H(X, g(x)) (c) = H(g(X)) + H(X g(x)) (d) H(g(X)) ()

3 Thus, H(g(X)) H(X). Solution (a) It comes from entropy s chain rule applied to random variables X and g(x), i.e. H(X, Y ) = H(X) + H(Y X), so H(X, g(x)) = H(X) + H(g(X) X). (b) Intuitively, if g(x) depends only on X and if the value of X is known, g(x) is completely specified and it has a deterministic value. The entropy of a deterministic value is 0, so H(g(X) X) = 0 and H(X) + H(g(X) X) = H(X). (c) Again, this formula comes from the entropy s chain rule, in the form: H(X, Y ) = H(Y ) + H(X Y ). (d) Proving that H(g(X)) + H(X g(x)) H(g(X)) means proving that H(X g(x)) 0: the non-negativity is one of the property of entropy and can be proved from its definition by noting that the logarithm of a probability (a quantity always less than or equal to ) is non-positive. In particular H(X g(x)) = 0 if the knowledge of the value of g(x) allows to totally specify the value of X; otherwise H(X g(x)) > 0 (for example if g(x) is an injective function). Ex..7(a) (Davide Basilio Bartolini) Text Coin weighing. Suppose that one has n coins, among which there may or may not be one counterfeit coin. If there is a counterfeit coin, it may be either heavier or lighter than the other coins. The coins are to be weighed by a balance. Find an upper bound on the number of coins n so that k weighings will find the counterfeit coin (if any) and correctly declare it to be heavier or lighter. Solution Let X be a string of n characters on the alphabet X = {, 0, } n, each of which represents one coin. Each of the characters of X may have three different values (say if the coin is heavier than a normal one, 0 if it is regular, if it is lighter). Since only one of the coins may be counterfeit, X may be a string of all 0 (if all the coins are regular) or may present either a or a at only one position. Thus, the possible configurations for X are n +. Under the hypothesis of a uniform distribution of the probability of which coin is counterfeit, the entropy of X will be: H(X) = log (n + )

4 Now let Z = [Z, Z,..., Z k ] be a random variable representing the weighings; each of the Z i will have three possible values to indicate whether the result of the weighing is balanced, left arm heavier or right arm heavier. The entropy of each Z i will be upper-bounded by the three possible values it can assume: H(Z i ) log, i [, k] and for Z (under the hypothesis of independence of the weighings): H(Z) = H(Z, Z,..., Z k ) (ChainR.) = (Indep.) = k H(Z i ) log i= k H(Z i Z i,..., Z ) i= Since we want to know how many weghings will yield the same amount of information which is given by the configuration of X (i.e. we want to know how many weighings will be needed to find out which coin - if any - is counterfeit), we can write: H(X) = H(Z) log log (n + ) log n + k n k, which is the wanted upper bound. Ex.. (Kenneth Palacio) X Y 0 0 / / 0 / Table : p(x,y) for problem.. Find: (a) H(X), H(Y ). (b) H(X Y ), H(Y X). (c) H(X, Y ). (d) H(Y ) H(Y X). (e) I(X; Y ). (f) Draw a Venn diagram for the quantities in parts (a) through (e). Solution: 4

5 Compute of marginal distributions: p(x) = [, ] p(y) = [, ] (a) H(X), H(Y ). H(X) = log ( ) log ( ) H(X) = 0.98bits (4) () H(Y ) = log ( ) log ( ) H(Y ) = 0.98bits (6) (5) Figure : H(X), H(Y) (b) H(X Y ), H(Y X). H(X Y ) = p(y = i)h(x Y = y) (7) i=0 H(X Y ) = H(X Y = 0) + H(X Y = ) (8) H(X Y ) = H(, 0) + H(/, /) (9) H(X Y ) = / (0) 5

6 H(Y X) = X p(x = i)h(y X = x) () i=0 H(Y X) = H(Y X = 0) + H(Y X = ) H(Y X) = H(/, /) + H(0, ) H(Y X) = / () () (4) Figure : H(X Y ), H(Y X) (c) H(X, Y ). H(X, Y ) =, X p(x, y) log p(x, y) (5),y=0 H(X, Y ) = log H(X, Y ) = bits Figure : H(X,Y) 6 (6) (7)

7 (d) H(Y ) H(Y X). H(Y ) H(Y X) = 0.98 / (8) H(Y ) H(Y X) = 0.54 (9) Figure 4: H(Y ) H(Y X) (e) I(X; Y ). p(x, y) I(X; Y ) = p(x, y) log p(x)p(y) x,y / / / I(X; Y ) = log + log + log (/)(/) (/)(/) (/)(/) I(X; Y ) = X (0) () () Figure 5: I(X;Y) (f) Venn diagram is already shown for each item. Ex..0 (Kenneth Palacio) Run-length coding. Let X,X,..., Xn be (possibly dependent) binary random variables. Suppose that one calculates the run lengths R = (R, R,...) of this sequence (in order as they occur). For example, the sequence X = yields run lengths R = (,,,, ). Compare 7

8 H(X, X,..., X n ), H(R), and H(Xn, R). Show all equalities and inequalities, and bound all the differences. Solution: Lets assume that one random variable Xj (0 < j n) is known, then if R is also known, H(Xj,R) will provide the same information about uncertainty than H(X,X,.. Xj,..,Xn), since the whole sequence of X can be completely recovered from the knowledge of Xj and R. For example, with X5 = and the run lengths R = (,,,, ) it s possible to recover the original sequence as follows: X5 =, R = (,,,, ) leads to recover the sequence: X = It can be concluded that: H(Xj, R) = H(X, X,...Xn) Using the chain rule, H(Xj,R) can be written as: H(Xj, R) = H(R) + H(Xj R) H(Xj R) H(Xj), since conditioning reduces entropy. Then it s possible to write: H(Xj) H(X, X,...Xn) H(R) H(Xj) + H(R) H(X, X,...Xn) Computing H(Xj) = n x p(xj) log Xj, where the distribution of Xj is unknown, it can be assumed to be: a probability of p for Xj=0 and of (-p) for Xj=. It can be observed that the maximum entropy is given when p=/ leading max H(Xj)=. Then: + H(R) H(X, X,...Xn) Considering the results obtained in problem.4, we can write also that: H(R) H(X). Because R is a function of X. 8

Chapter 2: Entropy and Mutual Information

Chapter 2: Entropy and Mutual Information Chapter 2: Entropy and Mutual Information Chapter 2 outline Definitions Entropy Joint entropy, conditional entropy Relative entropy, mutual information Chain rules Jensen s inequality Log-sum inequality

More information

Exercises with solutions (1)

Exercises with solutions (1) Exercises with solutions (). Investigate the relationship between independence and correlation. (a) Two random variables X and Y are said to be correlated if and only if their covariance C XY is not equal

More information

An Introduction to Information Theory

An Introduction to Information Theory An Introduction to Information Theory Carlton Downey November 12, 2013 INTRODUCTION Today s recitation will be an introduction to Information Theory Information theory studies the quantification of Information

More information

Information Theory and Coding Prof. S. N. Merchant Department of Electrical Engineering Indian Institute of Technology, Bombay

Information Theory and Coding Prof. S. N. Merchant Department of Electrical Engineering Indian Institute of Technology, Bombay Information Theory and Coding Prof. S. N. Merchant Department of Electrical Engineering Indian Institute of Technology, Bombay Lecture - 17 Shannon-Fano-Elias Coding and Introduction to Arithmetic Coding

More information

LECTURE 4. Last time: Lecture outline

LECTURE 4. Last time: Lecture outline LECTURE 4 Last time: Types of convergence Weak Law of Large Numbers Strong Law of Large Numbers Asymptotic Equipartition Property Lecture outline Stochastic processes Markov chains Entropy rate Random

More information

Question: What is the probability that a five-card poker hand contains a flush, that is, five cards of the same suit?

Question: What is the probability that a five-card poker hand contains a flush, that is, five cards of the same suit? ECS20 Discrete Mathematics Quarter: Spring 2007 Instructor: John Steinberger Assistant: Sophie Engle (prepared by Sophie Engle) Homework 8 Hints Due Wednesday June 6 th 2007 Section 6.1 #16 What is the

More information

Lecture 2: October 24

Lecture 2: October 24 Machine Learning: Foundations Fall Semester, 2 Lecture 2: October 24 Lecturer: Yishay Mansour Scribe: Shahar Yifrah, Roi Meron 2. Bayesian Inference - Overview This lecture is going to describe the basic

More information

Random variables P(X = 3) = P(X = 3) = 1 8, P(X = 1) = P(X = 1) = 3 8.

Random variables P(X = 3) = P(X = 3) = 1 8, P(X = 1) = P(X = 1) = 3 8. Random variables Remark on Notations 1. When X is a number chosen uniformly from a data set, What I call P(X = k) is called Freq[k, X] in the courseware. 2. When X is a random variable, what I call F ()

More information

Recitation 4. 24xy for 0 < x < 1, 0 < y < 1, x + y < 1 0 elsewhere

Recitation 4. 24xy for 0 < x < 1, 0 < y < 1, x + y < 1 0 elsewhere Recitation. Exercise 3.5: If the joint probability density of X and Y is given by xy for < x

More information

Chapter 4 Lecture Notes

Chapter 4 Lecture Notes Chapter 4 Lecture Notes Random Variables October 27, 2015 1 Section 4.1 Random Variables A random variable is typically a real-valued function defined on the sample space of some experiment. For instance,

More information

National Sun Yat-Sen University CSE Course: Information Theory. Gambling And Entropy

National Sun Yat-Sen University CSE Course: Information Theory. Gambling And Entropy Gambling And Entropy 1 Outline There is a strong relationship between the growth rate of investment in a horse race and the entropy of the horse race. The value of side information is related to the mutual

More information

Lecture 1: Entropy and mutual information

Lecture 1: Entropy and mutual information Lecture 1: Entropy and mutual information 1 Introduction Imagine two people Alice and Bob living in Toronto and Boston respectively. Alice (Toronto) goes jogging whenever it is not snowing heavily. Bob

More information

T Cryptology Spring 2009

T Cryptology Spring 2009 T-79.5501 Cryptology Spring 2009 Homework 2 Tutor : Joo Y. Cho joo.cho@tkk.fi 5th February 2009 Q1. Let us consider a cryptosystem where P = {a, b, c} and C = {1, 2, 3, 4}, K = {K 1, K 2, K 3 }, and the

More information

3 Multiple Discrete Random Variables

3 Multiple Discrete Random Variables 3 Multiple Discrete Random Variables 3.1 Joint densities Suppose we have a probability space (Ω, F,P) and now we have two discrete random variables X and Y on it. They have probability mass functions f

More information

1. (First passage/hitting times/gambler s ruin problem:) Suppose that X has a discrete state space and let i be a fixed state. Let

1. (First passage/hitting times/gambler s ruin problem:) Suppose that X has a discrete state space and let i be a fixed state. Let Copyright c 2009 by Karl Sigman 1 Stopping Times 1.1 Stopping Times: Definition Given a stochastic process X = {X n : n 0}, a random time τ is a discrete random variable on the same probability space as

More information

Convex Functions. Convex Functions. Optimization Carlos Guestrin Carnegie Mellon University. March 17 th, Function f:r n R is convex if

Convex Functions. Convex Functions. Optimization Carlos Guestrin Carnegie Mellon University. March 17 th, Function f:r n R is convex if Convex Functions Optimization - 10725 Carlos Guestrin Carnegie Mellon University March 17 th, 2010 1 Convex Functions Function f:r n R is convex if Domain is convex Generalization: Jensen s inequality:

More information

Basics of information theory and information complexity

Basics of information theory and information complexity Basics of information theory and information complexity a tutorial Mark Braverman Princeton University June 1, 2013 1 Part I: Information theory Information theory, in its modern format was introduced

More information

Lecture 4: AC 0 lower bounds and pseudorandomness

Lecture 4: AC 0 lower bounds and pseudorandomness Lecture 4: AC 0 lower bounds and pseudorandomness Topics in Complexity Theory and Pseudorandomness (Spring 2013) Rutgers University Swastik Kopparty Scribes: Jason Perry and Brian Garnett In this lecture,

More information

Gambling and Data Compression

Gambling and Data Compression Gambling and Data Compression Gambling. Horse Race Definition The wealth relative S(X) = b(x)o(x) is the factor by which the gambler s wealth grows if horse X wins the race, where b(x) is the fraction

More information

The Basics of Graphical Models

The Basics of Graphical Models The Basics of Graphical Models David M. Blei Columbia University October 3, 2015 Introduction These notes follow Chapter 2 of An Introduction to Probabilistic Graphical Models by Michael Jordan. Many figures

More information

The sample space for a pair of die rolls is the set. The sample space for a random number between 0 and 1 is the interval [0, 1].

The sample space for a pair of die rolls is the set. The sample space for a random number between 0 and 1 is the interval [0, 1]. Probability Theory Probability Spaces and Events Consider a random experiment with several possible outcomes. For example, we might roll a pair of dice, flip a coin three times, or choose a random real

More information

Discrete Mathematics: Solutions to Homework (12%) For each of the following sets, determine whether {2} is an element of that set.

Discrete Mathematics: Solutions to Homework (12%) For each of the following sets, determine whether {2} is an element of that set. Discrete Mathematics: Solutions to Homework 2 1. (12%) For each of the following sets, determine whether {2} is an element of that set. (a) {x R x is an integer greater than 1} (b) {x R x is the square

More information

CHAPTER 2 Estimating Probabilities

CHAPTER 2 Estimating Probabilities CHAPTER 2 Estimating Probabilities Machine Learning Copyright c 2016. Tom M. Mitchell. All rights reserved. *DRAFT OF January 24, 2016* *PLEASE DO NOT DISTRIBUTE WITHOUT AUTHOR S PERMISSION* This is a

More information

Linear Threshold Units

Linear Threshold Units Linear Threshold Units w x hx (... w n x n w We assume that each feature x j and each weight w j is a real number (we will relax this later) We will study three different algorithms for learning linear

More information

Discrete Math in Computer Science Homework 7 Solutions (Max Points: 80)

Discrete Math in Computer Science Homework 7 Solutions (Max Points: 80) Discrete Math in Computer Science Homework 7 Solutions (Max Points: 80) CS 30, Winter 2016 by Prasad Jayanti 1. (10 points) Here is the famous Monty Hall Puzzle. Suppose you are on a game show, and you

More information

Statistics 100A Homework 3 Solutions

Statistics 100A Homework 3 Solutions Chapter Statistics 00A Homework Solutions Ryan Rosario. Two balls are chosen randomly from an urn containing 8 white, black, and orange balls. Suppose that we win $ for each black ball selected and we

More information

Lecture 18 October 30

Lecture 18 October 30 EECS 290S: Network Information Flow Fall 2008 Lecture 18 October 30 Lecturer: Anant Sahai and David Tse Scribe: Changho Suh In this lecture, we studied two types of one-to-many channels: (1) compound channels

More information

4. Continuous Random Variables, the Pareto and Normal Distributions

4. Continuous Random Variables, the Pareto and Normal Distributions 4. Continuous Random Variables, the Pareto and Normal Distributions A continuous random variable X can take any value in a given range (e.g. height, weight, age). The distribution of a continuous random

More information

arxiv:1112.0829v1 [math.pr] 5 Dec 2011

arxiv:1112.0829v1 [math.pr] 5 Dec 2011 How Not to Win a Million Dollars: A Counterexample to a Conjecture of L. Breiman Thomas P. Hayes arxiv:1112.0829v1 [math.pr] 5 Dec 2011 Abstract Consider a gambling game in which we are allowed to repeatedly

More information

Notes from Week 1: Algorithms for sequential prediction

Notes from Week 1: Algorithms for sequential prediction CS 683 Learning, Games, and Electronic Markets Spring 2007 Notes from Week 1: Algorithms for sequential prediction Instructor: Robert Kleinberg 22-26 Jan 2007 1 Introduction In this course we will be looking

More information

MTH135/STA104: Probability

MTH135/STA104: Probability MTH135/STA14: Probability Homework # 8 Due: Tuesday, Nov 8, 5 Prof Robert Wolpert 1 Define a function f(x, y) on the plane R by { 1/x < y < x < 1 f(x, y) = other x, y a) Show that f(x, y) is a joint probability

More information

An-Najah National University Faculty of Engineering Industrial Engineering Department. Course : Quantitative Methods (65211)

An-Najah National University Faculty of Engineering Industrial Engineering Department. Course : Quantitative Methods (65211) An-Najah National University Faculty of Engineering Industrial Engineering Department Course : Quantitative Methods (65211) Instructor: Eng. Tamer Haddad 2 nd Semester 2009/2010 Chapter 5 Example: Joint

More information

Section 6.1 Joint Distribution Functions

Section 6.1 Joint Distribution Functions Section 6.1 Joint Distribution Functions We often care about more than one random variable at a time. DEFINITION: For any two random variables X and Y the joint cumulative probability distribution function

More information

Chapter Prove or disprove: A (B C) = (A B) (A C). Ans: True, since

Chapter Prove or disprove: A (B C) = (A B) (A C). Ans: True, since Chapter 2 1. Prove or disprove: A (B C) = (A B) (A C)., since A ( B C) = A B C = A ( B C) = ( A B) ( A C) = ( A B) ( A C). 2. Prove that A B= A B by giving a containment proof (that is, prove that the

More information

Almost every lossy compression system contains a lossless compression system

Almost every lossy compression system contains a lossless compression system Lossless compression in lossy compression systems Almost every lossy compression system contains a lossless compression system Lossy compression system Transform Quantizer Lossless Encoder Lossless Decoder

More information

Review for Calculus Rational Functions, Logarithms & Exponentials

Review for Calculus Rational Functions, Logarithms & Exponentials Definition and Domain of Rational Functions A rational function is defined as the quotient of two polynomial functions. F(x) = P(x) / Q(x) The domain of F is the set of all real numbers except those for

More information

Introduction to Learning & Decision Trees

Introduction to Learning & Decision Trees Artificial Intelligence: Representation and Problem Solving 5-38 April 0, 2007 Introduction to Learning & Decision Trees Learning and Decision Trees to learning What is learning? - more than just memorizing

More information

Influences in low-degree polynomials

Influences in low-degree polynomials Influences in low-degree polynomials Artūrs Bačkurs December 12, 2012 1 Introduction In 3] it is conjectured that every bounded real polynomial has a highly influential variable The conjecture is known

More information

Problem Set I Lossless Coding

Problem Set I Lossless Coding Problem Set I Lossless Coding Christopher Tsai Problem #2 Markov- Source Model The two-dimensional entropy rate function, is, Problem 2[ii] - Entropy Rate Function of a Markov- Source.9 Entropy H(α, β).9.7.3..7.3

More information

Sophomore College Mathematics of the Information Age Entropy

Sophomore College Mathematics of the Information Age Entropy Sophomore College Mathematics of the Information Age Entropy Let s recall the fundamental notions of information and entropy. To repeat from the first class, Shannon s emphasis is on selecting a given

More information

Lecture 8. Confidence intervals and the central limit theorem

Lecture 8. Confidence intervals and the central limit theorem Lecture 8. Confidence intervals and the central limit theorem Mathematical Statistics and Discrete Mathematics November 25th, 2015 1 / 15 Central limit theorem Let X 1, X 2,... X n be a random sample of

More information

Convex Functions (cont. 2)

Convex Functions (cont. 2) Convex Functions (cont. 2) Optimization - 10725 Carlos Guestrin Carnegie Mellon University March 5 th, 2008 1 Convex Functions Function f:r n R is convex if Domain is convex Generalization: Jensen s inequality:

More information

Convex Functions (cont. 2)

Convex Functions (cont. 2) Convex Functions (cont. 2) Optimization - 10725 Carlos Guestrin Carnegie Mellon University March 5 th, 2008 1 Convex Functions Function f:r n R is convex if Domain is convex Generalization: Jensen s inequality:

More information

MATH 425, PRACTICE FINAL EXAM SOLUTIONS.

MATH 425, PRACTICE FINAL EXAM SOLUTIONS. MATH 45, PRACTICE FINAL EXAM SOLUTIONS. Exercise. a Is the operator L defined on smooth functions of x, y by L u := u xx + cosu linear? b Does the answer change if we replace the operator L by the operator

More information

Information Theoretic Analysis of Proactive Routing Overhead in Mobile Ad Hoc Networks

Information Theoretic Analysis of Proactive Routing Overhead in Mobile Ad Hoc Networks Information Theoretic Analysis of Proactive Routing Overhead in obile Ad Hoc Networks Nianjun Zhou and Alhussein A. Abouzeid 1 Abstract This paper considers basic bounds on the overhead of link-state protocols

More information

2.8 Expected values and variance

2.8 Expected values and variance y 1 b a 0 y 0.0 0.2 0.4 0.6 0.8 1.0 1.2 0 a b (a) Probability density function x 0 a b (b) Cumulative distribution function x Figure 2.3: The probability density function and cumulative distribution function

More information

P (x) 0. Discrete random variables Expected value. The expected value, mean or average of a random variable x is: xp (x) = v i P (v i )

P (x) 0. Discrete random variables Expected value. The expected value, mean or average of a random variable x is: xp (x) = v i P (v i ) Discrete random variables Probability mass function Given a discrete random variable X taking values in X = {v 1,..., v m }, its probability mass function P : X [0, 1] is defined as: P (v i ) = Pr[X =

More information

Master s Theory Exam Spring 2006

Master s Theory Exam Spring 2006 Spring 2006 This exam contains 7 questions. You should attempt them all. Each question is divided into parts to help lead you through the material. You should attempt to complete as much of each problem

More information

Probability Generating Functions

Probability Generating Functions page 39 Chapter 3 Probability Generating Functions 3 Preamble: Generating Functions Generating functions are widely used in mathematics, and play an important role in probability theory Consider a sequence

More information

Definition: Suppose that two random variables, either continuous or discrete, X and Y have joint density

Definition: Suppose that two random variables, either continuous or discrete, X and Y have joint density HW MATH 461/561 Lecture Notes 15 1 Definition: Suppose that two random variables, either continuous or discrete, X and Y have joint density and marginal densities f(x, y), (x, y) Λ X,Y f X (x), x Λ X,

More information

Measuring Intrusion Detection Capability: An Information-Theoretic Approach

Measuring Intrusion Detection Capability: An Information-Theoretic Approach Measuring Intrusion Detection Capability: An Information-Theoretic Approach Guofei Gu, Prahlad Fogla, David Dagon, Boris Škorić Wenke Lee Philips Research Laboratories, Netherlands Georgia Institute of

More information

Probability and Statistics

Probability and Statistics CHAPTER 2: RANDOM VARIABLES AND ASSOCIATED FUNCTIONS 2b - 0 Probability and Statistics Kristel Van Steen, PhD 2 Montefiore Institute - Systems and Modeling GIGA - Bioinformatics ULg kristel.vansteen@ulg.ac.be

More information

MAS108 Probability I

MAS108 Probability I 1 QUEEN MARY UNIVERSITY OF LONDON 2:30 pm, Thursday 3 May, 2007 Duration: 2 hours MAS108 Probability I Do not start reading the question paper until you are instructed to by the invigilators. The paper

More information

Chapter 3: Discrete Random Variable and Probability Distribution. January 28, 2014

Chapter 3: Discrete Random Variable and Probability Distribution. January 28, 2014 STAT511 Spring 2014 Lecture Notes 1 Chapter 3: Discrete Random Variable and Probability Distribution January 28, 2014 3 Discrete Random Variables Chapter Overview Random Variable (r.v. Definition Discrete

More information

Math 55: Discrete Mathematics

Math 55: Discrete Mathematics Math 55: Discrete Mathematics UC Berkeley, Fall 2011 Homework # 5, due Wednesday, February 22 5.1.4 Let P (n) be the statement that 1 3 + 2 3 + + n 3 = (n(n + 1)/2) 2 for the positive integer n. a) What

More information

Lecture 4: BK inequality 27th August and 6th September, 2007

Lecture 4: BK inequality 27th August and 6th September, 2007 CSL866: Percolation and Random Graphs IIT Delhi Amitabha Bagchi Scribe: Arindam Pal Lecture 4: BK inequality 27th August and 6th September, 2007 4. Preliminaries The FKG inequality allows us to lower bound

More information

MASSACHUSETTS INSTITUTE OF TECHNOLOGY 6.436J/15.085J Fall 2008 Lecture 5 9/17/2008 RANDOM VARIABLES

MASSACHUSETTS INSTITUTE OF TECHNOLOGY 6.436J/15.085J Fall 2008 Lecture 5 9/17/2008 RANDOM VARIABLES MASSACHUSETTS INSTITUTE OF TECHNOLOGY 6.436J/15.085J Fall 2008 Lecture 5 9/17/2008 RANDOM VARIABLES Contents 1. Random variables and measurable functions 2. Cumulative distribution functions 3. Discrete

More information

Important Probability Distributions OPRE 6301

Important Probability Distributions OPRE 6301 Important Probability Distributions OPRE 6301 Important Distributions... Certain probability distributions occur with such regularity in real-life applications that they have been given their own names.

More information

What is Statistics? Lecture 1. Introduction and probability review. Idea of parametric inference

What is Statistics? Lecture 1. Introduction and probability review. Idea of parametric inference 0. 1. Introduction and probability review 1.1. What is Statistics? What is Statistics? Lecture 1. Introduction and probability review There are many definitions: I will use A set of principle and procedures

More information

Topic 4: Multivariate random variables. Multiple random variables

Topic 4: Multivariate random variables. Multiple random variables Topic 4: Multivariate random variables Joint, marginal, and conditional pmf Joint, marginal, and conditional pdf and cdf Independence Expectation, covariance, correlation Conditional expectation Two jointly

More information

Homework Zero. Max H. Farrell Chicago Booth BUS41100 Applied Regression Analysis. Complete before the first class, do not turn in

Homework Zero. Max H. Farrell Chicago Booth BUS41100 Applied Regression Analysis. Complete before the first class, do not turn in Homework Zero Max H. Farrell Chicago Booth BUS41100 Applied Regression Analysis Complete before the first class, do not turn in This homework is intended as a self test of your knowledge of the basic statistical

More information

MULTIVARIATE PROBABILITY DISTRIBUTIONS

MULTIVARIATE PROBABILITY DISTRIBUTIONS MULTIVARIATE PROBABILITY DISTRIBUTIONS. PRELIMINARIES.. Example. Consider an experiment that consists of tossing a die and a coin at the same time. We can consider a number of random variables defined

More information

Pennies and Blood. Mike Bomar

Pennies and Blood. Mike Bomar Pennies and Blood Mike Bomar In partial fulfillment of the requirements for the Master of Arts in Teaching with a Specialization in the Teaching of Middle Level Mathematics in the Department of Mathematics.

More information

Introduction to the Practice of Statistics Fifth Edition Moore, McCabe Section 4.4 Homework

Introduction to the Practice of Statistics Fifth Edition Moore, McCabe Section 4.4 Homework Introduction to the Practice of Statistics Fifth Edition Moore, McCabe Section 4.4 Homework 4.65 You buy a hot stock for $1000. The stock either gains 30% or loses 25% each day, each with probability.

More information

Chapter Objectives. Chapter 9. Sequential Search. Search Algorithms. Search Algorithms. Binary Search

Chapter Objectives. Chapter 9. Sequential Search. Search Algorithms. Search Algorithms. Binary Search Chapter Objectives Chapter 9 Search Algorithms Data Structures Using C++ 1 Learn the various search algorithms Explore how to implement the sequential and binary search algorithms Discover how the sequential

More information

Random variables, probability distributions, binomial random variable

Random variables, probability distributions, binomial random variable Week 4 lecture notes. WEEK 4 page 1 Random variables, probability distributions, binomial random variable Eample 1 : Consider the eperiment of flipping a fair coin three times. The number of tails that

More information

Worked examples Random Processes

Worked examples Random Processes Worked examples Random Processes Example 1 Consider patients coming to a doctor s office at random points in time. Let X n denote the time (in hrs) that the n th patient has to wait before being admitted

More information

MATH 201. Final ANSWERS August 12, 2016

MATH 201. Final ANSWERS August 12, 2016 MATH 01 Final ANSWERS August 1, 016 Part A 1. 17 points) A bag contains three different types of dice: four 6-sided dice, five 8-sided dice, and six 0-sided dice. A die is drawn from the bag and then rolled.

More information

The random variable X - the no. of defective items when three electronic components are tested would be

The random variable X - the no. of defective items when three electronic components are tested would be RANDOM VARIABLES and PROBABILITY DISTRIBUTIONS Example: Give the sample space giving a detailed description of each possible outcome when three electronic components are tested, where N - denotes nondefective

More information

Adaptive Online Gradient Descent

Adaptive Online Gradient Descent Adaptive Online Gradient Descent Peter L Bartlett Division of Computer Science Department of Statistics UC Berkeley Berkeley, CA 94709 bartlett@csberkeleyedu Elad Hazan IBM Almaden Research Center 650

More information

The mathematics behind wireless communication

The mathematics behind wireless communication June 2008 Questions and setting In wireless communication, information is sent through what is called a channel. The channel is subject to noise, so that there will be some loss of information. How should

More information

4. Joint Distributions of Two Random Variables

4. Joint Distributions of Two Random Variables 4. Joint Distributions of Two Random Variables 4.1 Joint Distributions of Two Discrete Random Variables Suppose the discrete random variables X and Y have supports S X and S Y, respectively. The joint

More information

Data Analysis and Uncertainty Part 3: Hypothesis Testing/Sampling

Data Analysis and Uncertainty Part 3: Hypothesis Testing/Sampling Data Analysis and Uncertainty Part 3: Hypothesis Testing/Sampling Instructor: Sargur N. University at Buffalo The State University of New York srihari@cedar.buffalo.edu Topics 1. Hypothesis Testing 1.

More information

1 Approximating Set Cover

1 Approximating Set Cover CS 05: Algorithms (Grad) Feb 2-24, 2005 Approximating Set Cover. Definition An Instance (X, F ) of the set-covering problem consists of a finite set X and a family F of subset of X, such that every elemennt

More information

ST 371 (IV): Discrete Random Variables

ST 371 (IV): Discrete Random Variables ST 371 (IV): Discrete Random Variables 1 Random Variables A random variable (rv) is a function that is defined on the sample space of the experiment and that assigns a numerical variable to each possible

More information

5.4. HUFFMAN CODES 71

5.4. HUFFMAN CODES 71 5.4. HUFFMAN CODES 7 Corollary 28 Consider a coding from a length n vector of source symbols, x = (x... x n ), to a binary codeword of length l(x). Then the average codeword length per source symbol for

More information

Summary of Formulas and Concepts. Descriptive Statistics (Ch. 1-4)

Summary of Formulas and Concepts. Descriptive Statistics (Ch. 1-4) Summary of Formulas and Concepts Descriptive Statistics (Ch. 1-4) Definitions Population: The complete set of numerical information on a particular quantity in which an investigator is interested. We assume

More information

Math Review. for the Quantitative Reasoning Measure of the GRE revised General Test

Math Review. for the Quantitative Reasoning Measure of the GRE revised General Test Math Review for the Quantitative Reasoning Measure of the GRE revised General Test www.ets.org Overview This Math Review will familiarize you with the mathematical skills and concepts that are important

More information

STAT 430/510 Probability Lecture 14: Joint Probability Distribution, Continuous Case

STAT 430/510 Probability Lecture 14: Joint Probability Distribution, Continuous Case STAT 430/510 Probability Lecture 14: Joint Probability Distribution, Continuous Case Pengyuan (Penelope) Wang June 20, 2011 Joint density function of continuous Random Variable When X and Y are two continuous

More information

Probabilities. Probability of a event. From Random Variables to Events. From Random Variables to Events. Probability Theory I

Probabilities. Probability of a event. From Random Variables to Events. From Random Variables to Events. Probability Theory I Victor Adamchi Danny Sleator Great Theoretical Ideas In Computer Science Probability Theory I CS 5-25 Spring 200 Lecture Feb. 6, 200 Carnegie Mellon University We will consider chance experiments with

More information

Compression techniques

Compression techniques Compression techniques David Bařina February 22, 2013 David Bařina Compression techniques February 22, 2013 1 / 37 Contents 1 Terminology 2 Simple techniques 3 Entropy coding 4 Dictionary methods 5 Conclusion

More information

1 Definition of Conditional Expectation

1 Definition of Conditional Expectation 1 DEFINITION OF CONDITIONAL EXPECTATION 4 1 Definition of Conditional Expectation 1.1 General definition Recall the definition of conditional probability associated with Bayes Rule P(A B) P(A B) P(B) For

More information

Foundation of Video Coding Part I: Overview and Binary Encoding

Foundation of Video Coding Part I: Overview and Binary Encoding Foundation of Video Coding Part I: Overview and Binary Encoding Yao Wang Polytechnic University, Brooklyn, NY11201 http://eeweb.poly.edu Based on: Y. Wang, J. Ostermann, and Y.-Q. Zhang, Video Processing

More information

STA 247 Answers for practice problem set #1

STA 247 Answers for practice problem set #1 STA 247 Answers for practice problem set #1 Question 1: The random variable X has a range of {0, 1, 2} and the random variable Y has a range of {1, 2}. The joint distribution of X and Y is given by the

More information

Lecture 1-4. Architecture: Design Principles

Lecture 1-4. Architecture: Design Principles 6.976/ESD.937 Quantitative Foundations of Engineering Systems Feb 13-15-21, 2006 Lecturer: D. Shah and S. Mitter Lecture 1-4 Scribe: J. Laracy and D. Shah Architecture: Design Principles Architecture is

More information

2WB05 Simulation Lecture 8: Generating random variables

2WB05 Simulation Lecture 8: Generating random variables 2WB05 Simulation Lecture 8: Generating random variables Marko Boon http://www.win.tue.nl/courses/2wb05 January 7, 2013 Outline 2/36 1. How do we generate random variables? 2. Fitting distributions Generating

More information

Tail inequalities for order statistics of log-concave vectors and applications

Tail inequalities for order statistics of log-concave vectors and applications Tail inequalities for order statistics of log-concave vectors and applications Rafał Latała Based in part on a joint work with R.Adamczak, A.E.Litvak, A.Pajor and N.Tomczak-Jaegermann Banff, May 2011 Basic

More information

Stat 110. Unit 3: Random Variables Chapter 3 in the Text

Stat 110. Unit 3: Random Variables Chapter 3 in the Text Stat 110 Unit 3: Random Variables Chapter 3 in the Text 1 Unit 3 Outline Random Variables (RVs) and Distributions Discrete RVs and Probability Mass Functions (PMFs) Bernoulli and Binomial Distributions

More information

On the Sum Capacity of the Gaussian Multiple Access Channel with Feedback

On the Sum Capacity of the Gaussian Multiple Access Channel with Feedback Forty-Seventh Annual Allerton Conference Allerton House, UIUC, Illinois, USA September 30 - October, 009 On the Sum Capacity of the Gaussian Multiple Access Channel with Feedback Ehsan Ardetsanizadeh,

More information

2. Discrete random variables

2. Discrete random variables 2. Discrete random variables Statistics and probability: 2-1 If the chance outcome of the experiment is a number, it is called a random variable. Discrete random variable: the possible outcomes can be

More information

Binomial distribution From Wikipedia, the free encyclopedia See also: Negative binomial distribution

Binomial distribution From Wikipedia, the free encyclopedia See also: Negative binomial distribution Binomial distribution From Wikipedia, the free encyclopedia See also: Negative binomial distribution In probability theory and statistics, the binomial distribution is the discrete probability distribution

More information

H/wk 13, Solutions to selected problems

H/wk 13, Solutions to selected problems H/wk 13, Solutions to selected problems Ch. 4.1, Problem 5 (a) Find the number of roots of x x in Z 4, Z Z, any integral domain, Z 6. (b) Find a commutative ring in which x x has infinitely many roots.

More information

For a partition B 1,..., B n, where B i B j = for i. A = (A B 1 ) (A B 2 ),..., (A B n ) and thus. P (A) = P (A B i ) = P (A B i )P (B i )

For a partition B 1,..., B n, where B i B j = for i. A = (A B 1 ) (A B 2 ),..., (A B n ) and thus. P (A) = P (A B i ) = P (A B i )P (B i ) Probability Review 15.075 Cynthia Rudin A probability space, defined by Kolmogorov (1903-1987) consists of: A set of outcomes S, e.g., for the roll of a die, S = {1, 2, 3, 4, 5, 6}, 1 1 2 1 6 for the roll

More information

Lectures 5-6: Taylor Series

Lectures 5-6: Taylor Series Math 1d Instructor: Padraic Bartlett Lectures 5-: Taylor Series Weeks 5- Caltech 213 1 Taylor Polynomials and Series As we saw in week 4, power series are remarkably nice objects to work with. In particular,

More information

ECEN 5682 Theory and Practice of Error Control Codes

ECEN 5682 Theory and Practice of Error Control Codes ECEN 5682 Theory and Practice of Error Control Codes Convolutional Codes University of Colorado Spring 2007 Linear (n, k) block codes take k data symbols at a time and encode them into n code symbols.

More information

Math 431 An Introduction to Probability. Final Exam Solutions

Math 431 An Introduction to Probability. Final Exam Solutions Math 43 An Introduction to Probability Final Eam Solutions. A continuous random variable X has cdf a for 0, F () = for 0 <

More information

Tossing a Biased Coin

Tossing a Biased Coin Tossing a Biased Coin Michael Mitzenmacher When we talk about a coin toss, we think of it as unbiased: with probability one-half it comes up heads, and with probability one-half it comes up tails. An ideal

More information

MT426 Notebook 3 Fall 2012 prepared by Professor Jenny Baglivo. 3 MT426 Notebook 3 3. 3.1 Definitions... 3. 3.2 Joint Discrete Distributions...

MT426 Notebook 3 Fall 2012 prepared by Professor Jenny Baglivo. 3 MT426 Notebook 3 3. 3.1 Definitions... 3. 3.2 Joint Discrete Distributions... MT426 Notebook 3 Fall 2012 prepared by Professor Jenny Baglivo c Copyright 2004-2012 by Jenny A. Baglivo. All Rights Reserved. Contents 3 MT426 Notebook 3 3 3.1 Definitions............................................

More information

Chapter There are non-isomorphic rooted trees with four vertices. Ans: 4.

Chapter There are non-isomorphic rooted trees with four vertices. Ans: 4. Use the following to answer questions 1-26: In the questions below fill in the blanks. Chapter 10 1. If T is a tree with 999 vertices, then T has edges. 998. 2. There are non-isomorphic trees with four

More information

THE NUMBER OF GRAPHS AND A RANDOM GRAPH WITH A GIVEN DEGREE SEQUENCE. Alexander Barvinok

THE NUMBER OF GRAPHS AND A RANDOM GRAPH WITH A GIVEN DEGREE SEQUENCE. Alexander Barvinok THE NUMBER OF GRAPHS AND A RANDOM GRAPH WITH A GIVEN DEGREE SEQUENCE Alexer Barvinok Papers are available at http://www.math.lsa.umich.edu/ barvinok/papers.html This is a joint work with J.A. Hartigan

More information