# Ex. 2.1 (Davide Basilio Bartolini)

Save this PDF as:

Size: px
Start display at page:

## Transcription

1 ECE 54: Elements of Information Theory, Fall 00 Homework Solutions Ex.. (Davide Basilio Bartolini) Text Coin Flips. A fair coin is flipped until the first head occurs. Let X denote the number of flips required. (a) Find the Entropy H(X) in bits (b) A random variable X is drawn according to this distribution. Find an efficient sequence of yes-no questions of the form, Is X contained in the set S?. Compare H(X) to the expected number of questions required to determine X. Solution (a) The random variable X is on the domain X = {,,,...} and it denotes the number of flips needed to get the first head, i.e. + the number of consecutive tails appeared before the first head. Since the coin is said to be fair, we have p( head ) = p( tail ) = and hence (exploiting the independence of the coin flips): p(x = ) = p( head ) = p(x = ) = p( tail ) p( head ) = ( =. ntimes { }} { p(x = n) = p( tail )... p( tail ) p( head ) =... ( = ) ) n from this, it is clear that the probability mass distribution of X is: p X (x) = ( ) x

2 Once the distribution is known, H(X) can be computed from the definition: H(X) = p X (x) log p X (x) x X ( ) x ( ) x = log x= ( ) x ( ) x = log (since the summed expr. equals 0 for x = 0) ( ) x ( ) = x log (property of logarithms) ( ) ( ) x = log x ( ) ( ) x = x = ( ) = [bit] exploiting (k) x k x = ( k) (b) Since the most likely value for X is (p(x = ) = ), the most efficient first question is: Is X =? ; the next question will be Is X =? and so on, until a positive answer is found. If this strategy is used, the random variable Y representing the number of questions will have the same distribution as X and it will be: E [Y ] = y y=0 ( ) y = ( ) = which is exactly equal to the entropy of X. An interpretation of this fact could be that bits (which is the entropy value for X) are the amount of memory required to store the outcomes of the two binary questions which are enough (on average) to get a positive answer on the value of X. Exercise.4 (Matteo Carminati) Entropy of functions of a random variable. Let X be a discrete random variable. Show that the entropy of a function of X is less than or equal to the entropy of X by justifying the following steps: H(X, g(x)) (a) = H(X) + H(g(X) X) (b) = H(X) () H(X, g(x)) (c) = H(g(X)) + H(X g(x)) (d) H(g(X)) ()

3 Thus, H(g(X)) H(X). Solution (a) It comes from entropy s chain rule applied to random variables X and g(x), i.e. H(X, Y ) = H(X) + H(Y X), so H(X, g(x)) = H(X) + H(g(X) X). (b) Intuitively, if g(x) depends only on X and if the value of X is known, g(x) is completely specified and it has a deterministic value. The entropy of a deterministic value is 0, so H(g(X) X) = 0 and H(X) + H(g(X) X) = H(X). (c) Again, this formula comes from the entropy s chain rule, in the form: H(X, Y ) = H(Y ) + H(X Y ). (d) Proving that H(g(X)) + H(X g(x)) H(g(X)) means proving that H(X g(x)) 0: the non-negativity is one of the property of entropy and can be proved from its definition by noting that the logarithm of a probability (a quantity always less than or equal to ) is non-positive. In particular H(X g(x)) = 0 if the knowledge of the value of g(x) allows to totally specify the value of X; otherwise H(X g(x)) > 0 (for example if g(x) is an injective function). Ex..7(a) (Davide Basilio Bartolini) Text Coin weighing. Suppose that one has n coins, among which there may or may not be one counterfeit coin. If there is a counterfeit coin, it may be either heavier or lighter than the other coins. The coins are to be weighed by a balance. Find an upper bound on the number of coins n so that k weighings will find the counterfeit coin (if any) and correctly declare it to be heavier or lighter. Solution Let X be a string of n characters on the alphabet X = {, 0, } n, each of which represents one coin. Each of the characters of X may have three different values (say if the coin is heavier than a normal one, 0 if it is regular, if it is lighter). Since only one of the coins may be counterfeit, X may be a string of all 0 (if all the coins are regular) or may present either a or a at only one position. Thus, the possible configurations for X are n +. Under the hypothesis of a uniform distribution of the probability of which coin is counterfeit, the entropy of X will be: H(X) = log (n + )

4 Now let Z = [Z, Z,..., Z k ] be a random variable representing the weighings; each of the Z i will have three possible values to indicate whether the result of the weighing is balanced, left arm heavier or right arm heavier. The entropy of each Z i will be upper-bounded by the three possible values it can assume: H(Z i ) log, i [, k] and for Z (under the hypothesis of independence of the weighings): H(Z) = H(Z, Z,..., Z k ) (ChainR.) = (Indep.) = k H(Z i ) log i= k H(Z i Z i,..., Z ) i= Since we want to know how many weghings will yield the same amount of information which is given by the configuration of X (i.e. we want to know how many weighings will be needed to find out which coin - if any - is counterfeit), we can write: H(X) = H(Z) log log (n + ) log n + k n k, which is the wanted upper bound. Ex.. (Kenneth Palacio) X Y 0 0 / / 0 / Table : p(x,y) for problem.. Find: (a) H(X), H(Y ). (b) H(X Y ), H(Y X). (c) H(X, Y ). (d) H(Y ) H(Y X). (e) I(X; Y ). (f) Draw a Venn diagram for the quantities in parts (a) through (e). Solution: 4

5 Compute of marginal distributions: p(x) = [, ] p(y) = [, ] (a) H(X), H(Y ). H(X) = log ( ) log ( ) H(X) = 0.98bits (4) () H(Y ) = log ( ) log ( ) H(Y ) = 0.98bits (6) (5) Figure : H(X), H(Y) (b) H(X Y ), H(Y X). H(X Y ) = p(y = i)h(x Y = y) (7) i=0 H(X Y ) = H(X Y = 0) + H(X Y = ) (8) H(X Y ) = H(, 0) + H(/, /) (9) H(X Y ) = / (0) 5

6 H(Y X) = X p(x = i)h(y X = x) () i=0 H(Y X) = H(Y X = 0) + H(Y X = ) H(Y X) = H(/, /) + H(0, ) H(Y X) = / () () (4) Figure : H(X Y ), H(Y X) (c) H(X, Y ). H(X, Y ) =, X p(x, y) log p(x, y) (5),y=0 H(X, Y ) = log H(X, Y ) = bits Figure : H(X,Y) 6 (6) (7)

7 (d) H(Y ) H(Y X). H(Y ) H(Y X) = 0.98 / (8) H(Y ) H(Y X) = 0.54 (9) Figure 4: H(Y ) H(Y X) (e) I(X; Y ). p(x, y) I(X; Y ) = p(x, y) log p(x)p(y) x,y / / / I(X; Y ) = log + log + log (/)(/) (/)(/) (/)(/) I(X; Y ) = X (0) () () Figure 5: I(X;Y) (f) Venn diagram is already shown for each item. Ex..0 (Kenneth Palacio) Run-length coding. Let X,X,..., Xn be (possibly dependent) binary random variables. Suppose that one calculates the run lengths R = (R, R,...) of this sequence (in order as they occur). For example, the sequence X = yields run lengths R = (,,,, ). Compare 7

8 H(X, X,..., X n ), H(R), and H(Xn, R). Show all equalities and inequalities, and bound all the differences. Solution: Lets assume that one random variable Xj (0 < j n) is known, then if R is also known, H(Xj,R) will provide the same information about uncertainty than H(X,X,.. Xj,..,Xn), since the whole sequence of X can be completely recovered from the knowledge of Xj and R. For example, with X5 = and the run lengths R = (,,,, ) it s possible to recover the original sequence as follows: X5 =, R = (,,,, ) leads to recover the sequence: X = It can be concluded that: H(Xj, R) = H(X, X,...Xn) Using the chain rule, H(Xj,R) can be written as: H(Xj, R) = H(R) + H(Xj R) H(Xj R) H(Xj), since conditioning reduces entropy. Then it s possible to write: H(Xj) H(X, X,...Xn) H(R) H(Xj) + H(R) H(X, X,...Xn) Computing H(Xj) = n x p(xj) log Xj, where the distribution of Xj is unknown, it can be assumed to be: a probability of p for Xj=0 and of (-p) for Xj=. It can be observed that the maximum entropy is given when p=/ leading max H(Xj)=. Then: + H(R) H(X, X,...Xn) Considering the results obtained in problem.4, we can write also that: H(R) H(X). Because R is a function of X. 8

### Chapter 2: Entropy and Mutual Information

Chapter 2: Entropy and Mutual Information Chapter 2 outline Definitions Entropy Joint entropy, conditional entropy Relative entropy, mutual information Chain rules Jensen s inequality Log-sum inequality

### Exercises with solutions (1)

Exercises with solutions (). Investigate the relationship between independence and correlation. (a) Two random variables X and Y are said to be correlated if and only if their covariance C XY is not equal

### An Introduction to Information Theory

An Introduction to Information Theory Carlton Downey November 12, 2013 INTRODUCTION Today s recitation will be an introduction to Information Theory Information theory studies the quantification of Information

### Information Theory and Coding Prof. S. N. Merchant Department of Electrical Engineering Indian Institute of Technology, Bombay

Information Theory and Coding Prof. S. N. Merchant Department of Electrical Engineering Indian Institute of Technology, Bombay Lecture - 17 Shannon-Fano-Elias Coding and Introduction to Arithmetic Coding

### LECTURE 4. Last time: Lecture outline

LECTURE 4 Last time: Types of convergence Weak Law of Large Numbers Strong Law of Large Numbers Asymptotic Equipartition Property Lecture outline Stochastic processes Markov chains Entropy rate Random

### Question: What is the probability that a five-card poker hand contains a flush, that is, five cards of the same suit?

ECS20 Discrete Mathematics Quarter: Spring 2007 Instructor: John Steinberger Assistant: Sophie Engle (prepared by Sophie Engle) Homework 8 Hints Due Wednesday June 6 th 2007 Section 6.1 #16 What is the

### Lecture 2: October 24

Machine Learning: Foundations Fall Semester, 2 Lecture 2: October 24 Lecturer: Yishay Mansour Scribe: Shahar Yifrah, Roi Meron 2. Bayesian Inference - Overview This lecture is going to describe the basic

### Random variables P(X = 3) = P(X = 3) = 1 8, P(X = 1) = P(X = 1) = 3 8.

Random variables Remark on Notations 1. When X is a number chosen uniformly from a data set, What I call P(X = k) is called Freq[k, X] in the courseware. 2. When X is a random variable, what I call F ()

### Recitation 4. 24xy for 0 < x < 1, 0 < y < 1, x + y < 1 0 elsewhere

Recitation. Exercise 3.5: If the joint probability density of X and Y is given by xy for < x

### Chapter 4 Lecture Notes

Chapter 4 Lecture Notes Random Variables October 27, 2015 1 Section 4.1 Random Variables A random variable is typically a real-valued function defined on the sample space of some experiment. For instance,

### National Sun Yat-Sen University CSE Course: Information Theory. Gambling And Entropy

Gambling And Entropy 1 Outline There is a strong relationship between the growth rate of investment in a horse race and the entropy of the horse race. The value of side information is related to the mutual

### Lecture 1: Entropy and mutual information

Lecture 1: Entropy and mutual information 1 Introduction Imagine two people Alice and Bob living in Toronto and Boston respectively. Alice (Toronto) goes jogging whenever it is not snowing heavily. Bob

### T Cryptology Spring 2009

T-79.5501 Cryptology Spring 2009 Homework 2 Tutor : Joo Y. Cho joo.cho@tkk.fi 5th February 2009 Q1. Let us consider a cryptosystem where P = {a, b, c} and C = {1, 2, 3, 4}, K = {K 1, K 2, K 3 }, and the

### 3 Multiple Discrete Random Variables

3 Multiple Discrete Random Variables 3.1 Joint densities Suppose we have a probability space (Ω, F,P) and now we have two discrete random variables X and Y on it. They have probability mass functions f

### 1. (First passage/hitting times/gambler s ruin problem:) Suppose that X has a discrete state space and let i be a fixed state. Let

Copyright c 2009 by Karl Sigman 1 Stopping Times 1.1 Stopping Times: Definition Given a stochastic process X = {X n : n 0}, a random time τ is a discrete random variable on the same probability space as

### Convex Functions. Convex Functions. Optimization Carlos Guestrin Carnegie Mellon University. March 17 th, Function f:r n R is convex if

Convex Functions Optimization - 10725 Carlos Guestrin Carnegie Mellon University March 17 th, 2010 1 Convex Functions Function f:r n R is convex if Domain is convex Generalization: Jensen s inequality:

### Basics of information theory and information complexity

Basics of information theory and information complexity a tutorial Mark Braverman Princeton University June 1, 2013 1 Part I: Information theory Information theory, in its modern format was introduced

### Lecture 4: AC 0 lower bounds and pseudorandomness

Lecture 4: AC 0 lower bounds and pseudorandomness Topics in Complexity Theory and Pseudorandomness (Spring 2013) Rutgers University Swastik Kopparty Scribes: Jason Perry and Brian Garnett In this lecture,

### Gambling and Data Compression

Gambling and Data Compression Gambling. Horse Race Definition The wealth relative S(X) = b(x)o(x) is the factor by which the gambler s wealth grows if horse X wins the race, where b(x) is the fraction

### The Basics of Graphical Models

The Basics of Graphical Models David M. Blei Columbia University October 3, 2015 Introduction These notes follow Chapter 2 of An Introduction to Probabilistic Graphical Models by Michael Jordan. Many figures

### The sample space for a pair of die rolls is the set. The sample space for a random number between 0 and 1 is the interval [0, 1].

Probability Theory Probability Spaces and Events Consider a random experiment with several possible outcomes. For example, we might roll a pair of dice, flip a coin three times, or choose a random real

### Discrete Mathematics: Solutions to Homework (12%) For each of the following sets, determine whether {2} is an element of that set.

Discrete Mathematics: Solutions to Homework 2 1. (12%) For each of the following sets, determine whether {2} is an element of that set. (a) {x R x is an integer greater than 1} (b) {x R x is the square

### CHAPTER 2 Estimating Probabilities

CHAPTER 2 Estimating Probabilities Machine Learning Copyright c 2016. Tom M. Mitchell. All rights reserved. *DRAFT OF January 24, 2016* *PLEASE DO NOT DISTRIBUTE WITHOUT AUTHOR S PERMISSION* This is a

### Linear Threshold Units

Linear Threshold Units w x hx (... w n x n w We assume that each feature x j and each weight w j is a real number (we will relax this later) We will study three different algorithms for learning linear

### Discrete Math in Computer Science Homework 7 Solutions (Max Points: 80)

Discrete Math in Computer Science Homework 7 Solutions (Max Points: 80) CS 30, Winter 2016 by Prasad Jayanti 1. (10 points) Here is the famous Monty Hall Puzzle. Suppose you are on a game show, and you

### Statistics 100A Homework 3 Solutions

Chapter Statistics 00A Homework Solutions Ryan Rosario. Two balls are chosen randomly from an urn containing 8 white, black, and orange balls. Suppose that we win \$ for each black ball selected and we

### Lecture 18 October 30

EECS 290S: Network Information Flow Fall 2008 Lecture 18 October 30 Lecturer: Anant Sahai and David Tse Scribe: Changho Suh In this lecture, we studied two types of one-to-many channels: (1) compound channels

### 4. Continuous Random Variables, the Pareto and Normal Distributions

4. Continuous Random Variables, the Pareto and Normal Distributions A continuous random variable X can take any value in a given range (e.g. height, weight, age). The distribution of a continuous random

### arxiv:1112.0829v1 [math.pr] 5 Dec 2011

How Not to Win a Million Dollars: A Counterexample to a Conjecture of L. Breiman Thomas P. Hayes arxiv:1112.0829v1 [math.pr] 5 Dec 2011 Abstract Consider a gambling game in which we are allowed to repeatedly

### Notes from Week 1: Algorithms for sequential prediction

CS 683 Learning, Games, and Electronic Markets Spring 2007 Notes from Week 1: Algorithms for sequential prediction Instructor: Robert Kleinberg 22-26 Jan 2007 1 Introduction In this course we will be looking

### MTH135/STA104: Probability

MTH135/STA14: Probability Homework # 8 Due: Tuesday, Nov 8, 5 Prof Robert Wolpert 1 Define a function f(x, y) on the plane R by { 1/x < y < x < 1 f(x, y) = other x, y a) Show that f(x, y) is a joint probability

### An-Najah National University Faculty of Engineering Industrial Engineering Department. Course : Quantitative Methods (65211)

An-Najah National University Faculty of Engineering Industrial Engineering Department Course : Quantitative Methods (65211) Instructor: Eng. Tamer Haddad 2 nd Semester 2009/2010 Chapter 5 Example: Joint

### Section 6.1 Joint Distribution Functions

Section 6.1 Joint Distribution Functions We often care about more than one random variable at a time. DEFINITION: For any two random variables X and Y the joint cumulative probability distribution function

### Chapter Prove or disprove: A (B C) = (A B) (A C). Ans: True, since

Chapter 2 1. Prove or disprove: A (B C) = (A B) (A C)., since A ( B C) = A B C = A ( B C) = ( A B) ( A C) = ( A B) ( A C). 2. Prove that A B= A B by giving a containment proof (that is, prove that the

### Almost every lossy compression system contains a lossless compression system

Lossless compression in lossy compression systems Almost every lossy compression system contains a lossless compression system Lossy compression system Transform Quantizer Lossless Encoder Lossless Decoder

### Review for Calculus Rational Functions, Logarithms & Exponentials

Definition and Domain of Rational Functions A rational function is defined as the quotient of two polynomial functions. F(x) = P(x) / Q(x) The domain of F is the set of all real numbers except those for

### Introduction to Learning & Decision Trees

Artificial Intelligence: Representation and Problem Solving 5-38 April 0, 2007 Introduction to Learning & Decision Trees Learning and Decision Trees to learning What is learning? - more than just memorizing

### Influences in low-degree polynomials

Influences in low-degree polynomials Artūrs Bačkurs December 12, 2012 1 Introduction In 3] it is conjectured that every bounded real polynomial has a highly influential variable The conjecture is known

### Problem Set I Lossless Coding

Problem Set I Lossless Coding Christopher Tsai Problem #2 Markov- Source Model The two-dimensional entropy rate function, is, Problem 2[ii] - Entropy Rate Function of a Markov- Source.9 Entropy H(α, β).9.7.3..7.3

### Sophomore College Mathematics of the Information Age Entropy

Sophomore College Mathematics of the Information Age Entropy Let s recall the fundamental notions of information and entropy. To repeat from the first class, Shannon s emphasis is on selecting a given

### Lecture 8. Confidence intervals and the central limit theorem

Lecture 8. Confidence intervals and the central limit theorem Mathematical Statistics and Discrete Mathematics November 25th, 2015 1 / 15 Central limit theorem Let X 1, X 2,... X n be a random sample of

### Convex Functions (cont. 2)

Convex Functions (cont. 2) Optimization - 10725 Carlos Guestrin Carnegie Mellon University March 5 th, 2008 1 Convex Functions Function f:r n R is convex if Domain is convex Generalization: Jensen s inequality:

### Convex Functions (cont. 2)

Convex Functions (cont. 2) Optimization - 10725 Carlos Guestrin Carnegie Mellon University March 5 th, 2008 1 Convex Functions Function f:r n R is convex if Domain is convex Generalization: Jensen s inequality:

### MATH 425, PRACTICE FINAL EXAM SOLUTIONS.

MATH 45, PRACTICE FINAL EXAM SOLUTIONS. Exercise. a Is the operator L defined on smooth functions of x, y by L u := u xx + cosu linear? b Does the answer change if we replace the operator L by the operator

### Information Theoretic Analysis of Proactive Routing Overhead in Mobile Ad Hoc Networks

Information Theoretic Analysis of Proactive Routing Overhead in obile Ad Hoc Networks Nianjun Zhou and Alhussein A. Abouzeid 1 Abstract This paper considers basic bounds on the overhead of link-state protocols

### 2.8 Expected values and variance

y 1 b a 0 y 0.0 0.2 0.4 0.6 0.8 1.0 1.2 0 a b (a) Probability density function x 0 a b (b) Cumulative distribution function x Figure 2.3: The probability density function and cumulative distribution function

### P (x) 0. Discrete random variables Expected value. The expected value, mean or average of a random variable x is: xp (x) = v i P (v i )

Discrete random variables Probability mass function Given a discrete random variable X taking values in X = {v 1,..., v m }, its probability mass function P : X [0, 1] is defined as: P (v i ) = Pr[X =

### Master s Theory Exam Spring 2006

Spring 2006 This exam contains 7 questions. You should attempt them all. Each question is divided into parts to help lead you through the material. You should attempt to complete as much of each problem

### Probability Generating Functions

page 39 Chapter 3 Probability Generating Functions 3 Preamble: Generating Functions Generating functions are widely used in mathematics, and play an important role in probability theory Consider a sequence

### Definition: Suppose that two random variables, either continuous or discrete, X and Y have joint density

HW MATH 461/561 Lecture Notes 15 1 Definition: Suppose that two random variables, either continuous or discrete, X and Y have joint density and marginal densities f(x, y), (x, y) Λ X,Y f X (x), x Λ X,

### Measuring Intrusion Detection Capability: An Information-Theoretic Approach

Measuring Intrusion Detection Capability: An Information-Theoretic Approach Guofei Gu, Prahlad Fogla, David Dagon, Boris Škorić Wenke Lee Philips Research Laboratories, Netherlands Georgia Institute of

### Probability and Statistics

CHAPTER 2: RANDOM VARIABLES AND ASSOCIATED FUNCTIONS 2b - 0 Probability and Statistics Kristel Van Steen, PhD 2 Montefiore Institute - Systems and Modeling GIGA - Bioinformatics ULg kristel.vansteen@ulg.ac.be

### MAS108 Probability I

1 QUEEN MARY UNIVERSITY OF LONDON 2:30 pm, Thursday 3 May, 2007 Duration: 2 hours MAS108 Probability I Do not start reading the question paper until you are instructed to by the invigilators. The paper

### Chapter 3: Discrete Random Variable and Probability Distribution. January 28, 2014

STAT511 Spring 2014 Lecture Notes 1 Chapter 3: Discrete Random Variable and Probability Distribution January 28, 2014 3 Discrete Random Variables Chapter Overview Random Variable (r.v. Definition Discrete

### Math 55: Discrete Mathematics

Math 55: Discrete Mathematics UC Berkeley, Fall 2011 Homework # 5, due Wednesday, February 22 5.1.4 Let P (n) be the statement that 1 3 + 2 3 + + n 3 = (n(n + 1)/2) 2 for the positive integer n. a) What

### Lecture 4: BK inequality 27th August and 6th September, 2007

CSL866: Percolation and Random Graphs IIT Delhi Amitabha Bagchi Scribe: Arindam Pal Lecture 4: BK inequality 27th August and 6th September, 2007 4. Preliminaries The FKG inequality allows us to lower bound

### MASSACHUSETTS INSTITUTE OF TECHNOLOGY 6.436J/15.085J Fall 2008 Lecture 5 9/17/2008 RANDOM VARIABLES

MASSACHUSETTS INSTITUTE OF TECHNOLOGY 6.436J/15.085J Fall 2008 Lecture 5 9/17/2008 RANDOM VARIABLES Contents 1. Random variables and measurable functions 2. Cumulative distribution functions 3. Discrete

### Important Probability Distributions OPRE 6301

Important Probability Distributions OPRE 6301 Important Distributions... Certain probability distributions occur with such regularity in real-life applications that they have been given their own names.

### What is Statistics? Lecture 1. Introduction and probability review. Idea of parametric inference

0. 1. Introduction and probability review 1.1. What is Statistics? What is Statistics? Lecture 1. Introduction and probability review There are many definitions: I will use A set of principle and procedures

### Topic 4: Multivariate random variables. Multiple random variables

Topic 4: Multivariate random variables Joint, marginal, and conditional pmf Joint, marginal, and conditional pdf and cdf Independence Expectation, covariance, correlation Conditional expectation Two jointly

### Homework Zero. Max H. Farrell Chicago Booth BUS41100 Applied Regression Analysis. Complete before the first class, do not turn in

Homework Zero Max H. Farrell Chicago Booth BUS41100 Applied Regression Analysis Complete before the first class, do not turn in This homework is intended as a self test of your knowledge of the basic statistical

### MULTIVARIATE PROBABILITY DISTRIBUTIONS

MULTIVARIATE PROBABILITY DISTRIBUTIONS. PRELIMINARIES.. Example. Consider an experiment that consists of tossing a die and a coin at the same time. We can consider a number of random variables defined

### Pennies and Blood. Mike Bomar

Pennies and Blood Mike Bomar In partial fulfillment of the requirements for the Master of Arts in Teaching with a Specialization in the Teaching of Middle Level Mathematics in the Department of Mathematics.

### Introduction to the Practice of Statistics Fifth Edition Moore, McCabe Section 4.4 Homework

Introduction to the Practice of Statistics Fifth Edition Moore, McCabe Section 4.4 Homework 4.65 You buy a hot stock for \$1000. The stock either gains 30% or loses 25% each day, each with probability.

### Chapter Objectives. Chapter 9. Sequential Search. Search Algorithms. Search Algorithms. Binary Search

Chapter Objectives Chapter 9 Search Algorithms Data Structures Using C++ 1 Learn the various search algorithms Explore how to implement the sequential and binary search algorithms Discover how the sequential

### Random variables, probability distributions, binomial random variable

Week 4 lecture notes. WEEK 4 page 1 Random variables, probability distributions, binomial random variable Eample 1 : Consider the eperiment of flipping a fair coin three times. The number of tails that

### Worked examples Random Processes

Worked examples Random Processes Example 1 Consider patients coming to a doctor s office at random points in time. Let X n denote the time (in hrs) that the n th patient has to wait before being admitted

### MATH 201. Final ANSWERS August 12, 2016

MATH 01 Final ANSWERS August 1, 016 Part A 1. 17 points) A bag contains three different types of dice: four 6-sided dice, five 8-sided dice, and six 0-sided dice. A die is drawn from the bag and then rolled.

### The random variable X - the no. of defective items when three electronic components are tested would be

RANDOM VARIABLES and PROBABILITY DISTRIBUTIONS Example: Give the sample space giving a detailed description of each possible outcome when three electronic components are tested, where N - denotes nondefective

Adaptive Online Gradient Descent Peter L Bartlett Division of Computer Science Department of Statistics UC Berkeley Berkeley, CA 94709 bartlett@csberkeleyedu Elad Hazan IBM Almaden Research Center 650

### The mathematics behind wireless communication

June 2008 Questions and setting In wireless communication, information is sent through what is called a channel. The channel is subject to noise, so that there will be some loss of information. How should

### 4. Joint Distributions of Two Random Variables

4. Joint Distributions of Two Random Variables 4.1 Joint Distributions of Two Discrete Random Variables Suppose the discrete random variables X and Y have supports S X and S Y, respectively. The joint

### Data Analysis and Uncertainty Part 3: Hypothesis Testing/Sampling

Data Analysis and Uncertainty Part 3: Hypothesis Testing/Sampling Instructor: Sargur N. University at Buffalo The State University of New York srihari@cedar.buffalo.edu Topics 1. Hypothesis Testing 1.

### 1 Approximating Set Cover

CS 05: Algorithms (Grad) Feb 2-24, 2005 Approximating Set Cover. Definition An Instance (X, F ) of the set-covering problem consists of a finite set X and a family F of subset of X, such that every elemennt

### ST 371 (IV): Discrete Random Variables

ST 371 (IV): Discrete Random Variables 1 Random Variables A random variable (rv) is a function that is defined on the sample space of the experiment and that assigns a numerical variable to each possible

### 5.4. HUFFMAN CODES 71

5.4. HUFFMAN CODES 7 Corollary 28 Consider a coding from a length n vector of source symbols, x = (x... x n ), to a binary codeword of length l(x). Then the average codeword length per source symbol for

### Summary of Formulas and Concepts. Descriptive Statistics (Ch. 1-4)

Summary of Formulas and Concepts Descriptive Statistics (Ch. 1-4) Definitions Population: The complete set of numerical information on a particular quantity in which an investigator is interested. We assume

### Math Review. for the Quantitative Reasoning Measure of the GRE revised General Test

Math Review for the Quantitative Reasoning Measure of the GRE revised General Test www.ets.org Overview This Math Review will familiarize you with the mathematical skills and concepts that are important

### STAT 430/510 Probability Lecture 14: Joint Probability Distribution, Continuous Case

STAT 430/510 Probability Lecture 14: Joint Probability Distribution, Continuous Case Pengyuan (Penelope) Wang June 20, 2011 Joint density function of continuous Random Variable When X and Y are two continuous

### Probabilities. Probability of a event. From Random Variables to Events. From Random Variables to Events. Probability Theory I

Victor Adamchi Danny Sleator Great Theoretical Ideas In Computer Science Probability Theory I CS 5-25 Spring 200 Lecture Feb. 6, 200 Carnegie Mellon University We will consider chance experiments with

### Compression techniques

Compression techniques David Bařina February 22, 2013 David Bařina Compression techniques February 22, 2013 1 / 37 Contents 1 Terminology 2 Simple techniques 3 Entropy coding 4 Dictionary methods 5 Conclusion

### 1 Definition of Conditional Expectation

1 DEFINITION OF CONDITIONAL EXPECTATION 4 1 Definition of Conditional Expectation 1.1 General definition Recall the definition of conditional probability associated with Bayes Rule P(A B) P(A B) P(B) For

### Foundation of Video Coding Part I: Overview and Binary Encoding

Foundation of Video Coding Part I: Overview and Binary Encoding Yao Wang Polytechnic University, Brooklyn, NY11201 http://eeweb.poly.edu Based on: Y. Wang, J. Ostermann, and Y.-Q. Zhang, Video Processing

### STA 247 Answers for practice problem set #1

STA 247 Answers for practice problem set #1 Question 1: The random variable X has a range of {0, 1, 2} and the random variable Y has a range of {1, 2}. The joint distribution of X and Y is given by the

### Lecture 1-4. Architecture: Design Principles

6.976/ESD.937 Quantitative Foundations of Engineering Systems Feb 13-15-21, 2006 Lecturer: D. Shah and S. Mitter Lecture 1-4 Scribe: J. Laracy and D. Shah Architecture: Design Principles Architecture is

### 2WB05 Simulation Lecture 8: Generating random variables

2WB05 Simulation Lecture 8: Generating random variables Marko Boon http://www.win.tue.nl/courses/2wb05 January 7, 2013 Outline 2/36 1. How do we generate random variables? 2. Fitting distributions Generating

### Tail inequalities for order statistics of log-concave vectors and applications

Tail inequalities for order statistics of log-concave vectors and applications Rafał Latała Based in part on a joint work with R.Adamczak, A.E.Litvak, A.Pajor and N.Tomczak-Jaegermann Banff, May 2011 Basic

### Stat 110. Unit 3: Random Variables Chapter 3 in the Text

Stat 110 Unit 3: Random Variables Chapter 3 in the Text 1 Unit 3 Outline Random Variables (RVs) and Distributions Discrete RVs and Probability Mass Functions (PMFs) Bernoulli and Binomial Distributions

### On the Sum Capacity of the Gaussian Multiple Access Channel with Feedback

Forty-Seventh Annual Allerton Conference Allerton House, UIUC, Illinois, USA September 30 - October, 009 On the Sum Capacity of the Gaussian Multiple Access Channel with Feedback Ehsan Ardetsanizadeh,

### 2. Discrete random variables

2. Discrete random variables Statistics and probability: 2-1 If the chance outcome of the experiment is a number, it is called a random variable. Discrete random variable: the possible outcomes can be

### H/wk 13, Solutions to selected problems

H/wk 13, Solutions to selected problems Ch. 4.1, Problem 5 (a) Find the number of roots of x x in Z 4, Z Z, any integral domain, Z 6. (b) Find a commutative ring in which x x has infinitely many roots.

### For a partition B 1,..., B n, where B i B j = for i. A = (A B 1 ) (A B 2 ),..., (A B n ) and thus. P (A) = P (A B i ) = P (A B i )P (B i )

Probability Review 15.075 Cynthia Rudin A probability space, defined by Kolmogorov (1903-1987) consists of: A set of outcomes S, e.g., for the roll of a die, S = {1, 2, 3, 4, 5, 6}, 1 1 2 1 6 for the roll

### Lectures 5-6: Taylor Series

Math 1d Instructor: Padraic Bartlett Lectures 5-: Taylor Series Weeks 5- Caltech 213 1 Taylor Polynomials and Series As we saw in week 4, power series are remarkably nice objects to work with. In particular,

### ECEN 5682 Theory and Practice of Error Control Codes

ECEN 5682 Theory and Practice of Error Control Codes Convolutional Codes University of Colorado Spring 2007 Linear (n, k) block codes take k data symbols at a time and encode them into n code symbols.

### Math 431 An Introduction to Probability. Final Exam Solutions

Math 43 An Introduction to Probability Final Eam Solutions. A continuous random variable X has cdf a for 0, F () = for 0 <

### Tossing a Biased Coin

Tossing a Biased Coin Michael Mitzenmacher When we talk about a coin toss, we think of it as unbiased: with probability one-half it comes up heads, and with probability one-half it comes up tails. An ideal

### MT426 Notebook 3 Fall 2012 prepared by Professor Jenny Baglivo. 3 MT426 Notebook 3 3. 3.1 Definitions... 3. 3.2 Joint Discrete Distributions...

MT426 Notebook 3 Fall 2012 prepared by Professor Jenny Baglivo c Copyright 2004-2012 by Jenny A. Baglivo. All Rights Reserved. Contents 3 MT426 Notebook 3 3 3.1 Definitions............................................