Ex. 2.1 (Davide Basilio Bartolini)


 Paul Green
 2 years ago
 Views:
Transcription
1 ECE 54: Elements of Information Theory, Fall 00 Homework Solutions Ex.. (Davide Basilio Bartolini) Text Coin Flips. A fair coin is flipped until the first head occurs. Let X denote the number of flips required. (a) Find the Entropy H(X) in bits (b) A random variable X is drawn according to this distribution. Find an efficient sequence of yesno questions of the form, Is X contained in the set S?. Compare H(X) to the expected number of questions required to determine X. Solution (a) The random variable X is on the domain X = {,,,...} and it denotes the number of flips needed to get the first head, i.e. + the number of consecutive tails appeared before the first head. Since the coin is said to be fair, we have p( head ) = p( tail ) = and hence (exploiting the independence of the coin flips): p(x = ) = p( head ) = p(x = ) = p( tail ) p( head ) = ( =. ntimes { }} { p(x = n) = p( tail )... p( tail ) p( head ) =... ( = ) ) n from this, it is clear that the probability mass distribution of X is: p X (x) = ( ) x
2 Once the distribution is known, H(X) can be computed from the definition: H(X) = p X (x) log p X (x) x X ( ) x ( ) x = log x= ( ) x ( ) x = log (since the summed expr. equals 0 for x = 0) ( ) x ( ) = x log (property of logarithms) ( ) ( ) x = log x ( ) ( ) x = x = ( ) = [bit] exploiting (k) x k x = ( k) (b) Since the most likely value for X is (p(x = ) = ), the most efficient first question is: Is X =? ; the next question will be Is X =? and so on, until a positive answer is found. If this strategy is used, the random variable Y representing the number of questions will have the same distribution as X and it will be: E [Y ] = y y=0 ( ) y = ( ) = which is exactly equal to the entropy of X. An interpretation of this fact could be that bits (which is the entropy value for X) are the amount of memory required to store the outcomes of the two binary questions which are enough (on average) to get a positive answer on the value of X. Exercise.4 (Matteo Carminati) Entropy of functions of a random variable. Let X be a discrete random variable. Show that the entropy of a function of X is less than or equal to the entropy of X by justifying the following steps: H(X, g(x)) (a) = H(X) + H(g(X) X) (b) = H(X) () H(X, g(x)) (c) = H(g(X)) + H(X g(x)) (d) H(g(X)) ()
3 Thus, H(g(X)) H(X). Solution (a) It comes from entropy s chain rule applied to random variables X and g(x), i.e. H(X, Y ) = H(X) + H(Y X), so H(X, g(x)) = H(X) + H(g(X) X). (b) Intuitively, if g(x) depends only on X and if the value of X is known, g(x) is completely specified and it has a deterministic value. The entropy of a deterministic value is 0, so H(g(X) X) = 0 and H(X) + H(g(X) X) = H(X). (c) Again, this formula comes from the entropy s chain rule, in the form: H(X, Y ) = H(Y ) + H(X Y ). (d) Proving that H(g(X)) + H(X g(x)) H(g(X)) means proving that H(X g(x)) 0: the nonnegativity is one of the property of entropy and can be proved from its definition by noting that the logarithm of a probability (a quantity always less than or equal to ) is nonpositive. In particular H(X g(x)) = 0 if the knowledge of the value of g(x) allows to totally specify the value of X; otherwise H(X g(x)) > 0 (for example if g(x) is an injective function). Ex..7(a) (Davide Basilio Bartolini) Text Coin weighing. Suppose that one has n coins, among which there may or may not be one counterfeit coin. If there is a counterfeit coin, it may be either heavier or lighter than the other coins. The coins are to be weighed by a balance. Find an upper bound on the number of coins n so that k weighings will find the counterfeit coin (if any) and correctly declare it to be heavier or lighter. Solution Let X be a string of n characters on the alphabet X = {, 0, } n, each of which represents one coin. Each of the characters of X may have three different values (say if the coin is heavier than a normal one, 0 if it is regular, if it is lighter). Since only one of the coins may be counterfeit, X may be a string of all 0 (if all the coins are regular) or may present either a or a at only one position. Thus, the possible configurations for X are n +. Under the hypothesis of a uniform distribution of the probability of which coin is counterfeit, the entropy of X will be: H(X) = log (n + )
4 Now let Z = [Z, Z,..., Z k ] be a random variable representing the weighings; each of the Z i will have three possible values to indicate whether the result of the weighing is balanced, left arm heavier or right arm heavier. The entropy of each Z i will be upperbounded by the three possible values it can assume: H(Z i ) log, i [, k] and for Z (under the hypothesis of independence of the weighings): H(Z) = H(Z, Z,..., Z k ) (ChainR.) = (Indep.) = k H(Z i ) log i= k H(Z i Z i,..., Z ) i= Since we want to know how many weghings will yield the same amount of information which is given by the configuration of X (i.e. we want to know how many weighings will be needed to find out which coin  if any  is counterfeit), we can write: H(X) = H(Z) log log (n + ) log n + k n k, which is the wanted upper bound. Ex.. (Kenneth Palacio) X Y 0 0 / / 0 / Table : p(x,y) for problem.. Find: (a) H(X), H(Y ). (b) H(X Y ), H(Y X). (c) H(X, Y ). (d) H(Y ) H(Y X). (e) I(X; Y ). (f) Draw a Venn diagram for the quantities in parts (a) through (e). Solution: 4
5 Compute of marginal distributions: p(x) = [, ] p(y) = [, ] (a) H(X), H(Y ). H(X) = log ( ) log ( ) H(X) = 0.98bits (4) () H(Y ) = log ( ) log ( ) H(Y ) = 0.98bits (6) (5) Figure : H(X), H(Y) (b) H(X Y ), H(Y X). H(X Y ) = p(y = i)h(x Y = y) (7) i=0 H(X Y ) = H(X Y = 0) + H(X Y = ) (8) H(X Y ) = H(, 0) + H(/, /) (9) H(X Y ) = / (0) 5
6 H(Y X) = X p(x = i)h(y X = x) () i=0 H(Y X) = H(Y X = 0) + H(Y X = ) H(Y X) = H(/, /) + H(0, ) H(Y X) = / () () (4) Figure : H(X Y ), H(Y X) (c) H(X, Y ). H(X, Y ) =, X p(x, y) log p(x, y) (5),y=0 H(X, Y ) = log H(X, Y ) = bits Figure : H(X,Y) 6 (6) (7)
7 (d) H(Y ) H(Y X). H(Y ) H(Y X) = 0.98 / (8) H(Y ) H(Y X) = 0.54 (9) Figure 4: H(Y ) H(Y X) (e) I(X; Y ). p(x, y) I(X; Y ) = p(x, y) log p(x)p(y) x,y / / / I(X; Y ) = log + log + log (/)(/) (/)(/) (/)(/) I(X; Y ) = X (0) () () Figure 5: I(X;Y) (f) Venn diagram is already shown for each item. Ex..0 (Kenneth Palacio) Runlength coding. Let X,X,..., Xn be (possibly dependent) binary random variables. Suppose that one calculates the run lengths R = (R, R,...) of this sequence (in order as they occur). For example, the sequence X = yields run lengths R = (,,,, ). Compare 7
8 H(X, X,..., X n ), H(R), and H(Xn, R). Show all equalities and inequalities, and bound all the differences. Solution: Lets assume that one random variable Xj (0 < j n) is known, then if R is also known, H(Xj,R) will provide the same information about uncertainty than H(X,X,.. Xj,..,Xn), since the whole sequence of X can be completely recovered from the knowledge of Xj and R. For example, with X5 = and the run lengths R = (,,,, ) it s possible to recover the original sequence as follows: X5 =, R = (,,,, ) leads to recover the sequence: X = It can be concluded that: H(Xj, R) = H(X, X,...Xn) Using the chain rule, H(Xj,R) can be written as: H(Xj, R) = H(R) + H(Xj R) H(Xj R) H(Xj), since conditioning reduces entropy. Then it s possible to write: H(Xj) H(X, X,...Xn) H(R) H(Xj) + H(R) H(X, X,...Xn) Computing H(Xj) = n x p(xj) log Xj, where the distribution of Xj is unknown, it can be assumed to be: a probability of p for Xj=0 and of (p) for Xj=. It can be observed that the maximum entropy is given when p=/ leading max H(Xj)=. Then: + H(R) H(X, X,...Xn) Considering the results obtained in problem.4, we can write also that: H(R) H(X). Because R is a function of X. 8
Chapter 2: Entropy and Mutual Information
Chapter 2: Entropy and Mutual Information Chapter 2 outline Definitions Entropy Joint entropy, conditional entropy Relative entropy, mutual information Chain rules Jensen s inequality Logsum inequality
More informationExercises with solutions (1)
Exercises with solutions (). Investigate the relationship between independence and correlation. (a) Two random variables X and Y are said to be correlated if and only if their covariance C XY is not equal
More informationAn Introduction to Information Theory
An Introduction to Information Theory Carlton Downey November 12, 2013 INTRODUCTION Today s recitation will be an introduction to Information Theory Information theory studies the quantification of Information
More informationInformation Theory and Coding Prof. S. N. Merchant Department of Electrical Engineering Indian Institute of Technology, Bombay
Information Theory and Coding Prof. S. N. Merchant Department of Electrical Engineering Indian Institute of Technology, Bombay Lecture  17 ShannonFanoElias Coding and Introduction to Arithmetic Coding
More informationLECTURE 4. Last time: Lecture outline
LECTURE 4 Last time: Types of convergence Weak Law of Large Numbers Strong Law of Large Numbers Asymptotic Equipartition Property Lecture outline Stochastic processes Markov chains Entropy rate Random
More informationQuestion: What is the probability that a fivecard poker hand contains a flush, that is, five cards of the same suit?
ECS20 Discrete Mathematics Quarter: Spring 2007 Instructor: John Steinberger Assistant: Sophie Engle (prepared by Sophie Engle) Homework 8 Hints Due Wednesday June 6 th 2007 Section 6.1 #16 What is the
More informationLecture 2: October 24
Machine Learning: Foundations Fall Semester, 2 Lecture 2: October 24 Lecturer: Yishay Mansour Scribe: Shahar Yifrah, Roi Meron 2. Bayesian Inference  Overview This lecture is going to describe the basic
More informationRandom variables P(X = 3) = P(X = 3) = 1 8, P(X = 1) = P(X = 1) = 3 8.
Random variables Remark on Notations 1. When X is a number chosen uniformly from a data set, What I call P(X = k) is called Freq[k, X] in the courseware. 2. When X is a random variable, what I call F ()
More informationRecitation 4. 24xy for 0 < x < 1, 0 < y < 1, x + y < 1 0 elsewhere
Recitation. Exercise 3.5: If the joint probability density of X and Y is given by xy for < x
More informationChapter 4 Lecture Notes
Chapter 4 Lecture Notes Random Variables October 27, 2015 1 Section 4.1 Random Variables A random variable is typically a realvalued function defined on the sample space of some experiment. For instance,
More informationNational Sun YatSen University CSE Course: Information Theory. Gambling And Entropy
Gambling And Entropy 1 Outline There is a strong relationship between the growth rate of investment in a horse race and the entropy of the horse race. The value of side information is related to the mutual
More informationLecture 1: Entropy and mutual information
Lecture 1: Entropy and mutual information 1 Introduction Imagine two people Alice and Bob living in Toronto and Boston respectively. Alice (Toronto) goes jogging whenever it is not snowing heavily. Bob
More informationT Cryptology Spring 2009
T79.5501 Cryptology Spring 2009 Homework 2 Tutor : Joo Y. Cho joo.cho@tkk.fi 5th February 2009 Q1. Let us consider a cryptosystem where P = {a, b, c} and C = {1, 2, 3, 4}, K = {K 1, K 2, K 3 }, and the
More information3 Multiple Discrete Random Variables
3 Multiple Discrete Random Variables 3.1 Joint densities Suppose we have a probability space (Ω, F,P) and now we have two discrete random variables X and Y on it. They have probability mass functions f
More information1. (First passage/hitting times/gambler s ruin problem:) Suppose that X has a discrete state space and let i be a fixed state. Let
Copyright c 2009 by Karl Sigman 1 Stopping Times 1.1 Stopping Times: Definition Given a stochastic process X = {X n : n 0}, a random time τ is a discrete random variable on the same probability space as
More informationConvex Functions. Convex Functions. Optimization Carlos Guestrin Carnegie Mellon University. March 17 th, Function f:r n R is convex if
Convex Functions Optimization  10725 Carlos Guestrin Carnegie Mellon University March 17 th, 2010 1 Convex Functions Function f:r n R is convex if Domain is convex Generalization: Jensen s inequality:
More informationBasics of information theory and information complexity
Basics of information theory and information complexity a tutorial Mark Braverman Princeton University June 1, 2013 1 Part I: Information theory Information theory, in its modern format was introduced
More informationLecture 4: AC 0 lower bounds and pseudorandomness
Lecture 4: AC 0 lower bounds and pseudorandomness Topics in Complexity Theory and Pseudorandomness (Spring 2013) Rutgers University Swastik Kopparty Scribes: Jason Perry and Brian Garnett In this lecture,
More informationGambling and Data Compression
Gambling and Data Compression Gambling. Horse Race Definition The wealth relative S(X) = b(x)o(x) is the factor by which the gambler s wealth grows if horse X wins the race, where b(x) is the fraction
More informationThe Basics of Graphical Models
The Basics of Graphical Models David M. Blei Columbia University October 3, 2015 Introduction These notes follow Chapter 2 of An Introduction to Probabilistic Graphical Models by Michael Jordan. Many figures
More informationThe sample space for a pair of die rolls is the set. The sample space for a random number between 0 and 1 is the interval [0, 1].
Probability Theory Probability Spaces and Events Consider a random experiment with several possible outcomes. For example, we might roll a pair of dice, flip a coin three times, or choose a random real
More informationDiscrete Mathematics: Solutions to Homework (12%) For each of the following sets, determine whether {2} is an element of that set.
Discrete Mathematics: Solutions to Homework 2 1. (12%) For each of the following sets, determine whether {2} is an element of that set. (a) {x R x is an integer greater than 1} (b) {x R x is the square
More informationCHAPTER 2 Estimating Probabilities
CHAPTER 2 Estimating Probabilities Machine Learning Copyright c 2016. Tom M. Mitchell. All rights reserved. *DRAFT OF January 24, 2016* *PLEASE DO NOT DISTRIBUTE WITHOUT AUTHOR S PERMISSION* This is a
More informationLinear Threshold Units
Linear Threshold Units w x hx (... w n x n w We assume that each feature x j and each weight w j is a real number (we will relax this later) We will study three different algorithms for learning linear
More informationDiscrete Math in Computer Science Homework 7 Solutions (Max Points: 80)
Discrete Math in Computer Science Homework 7 Solutions (Max Points: 80) CS 30, Winter 2016 by Prasad Jayanti 1. (10 points) Here is the famous Monty Hall Puzzle. Suppose you are on a game show, and you
More informationStatistics 100A Homework 3 Solutions
Chapter Statistics 00A Homework Solutions Ryan Rosario. Two balls are chosen randomly from an urn containing 8 white, black, and orange balls. Suppose that we win $ for each black ball selected and we
More informationLecture 18 October 30
EECS 290S: Network Information Flow Fall 2008 Lecture 18 October 30 Lecturer: Anant Sahai and David Tse Scribe: Changho Suh In this lecture, we studied two types of onetomany channels: (1) compound channels
More information4. Continuous Random Variables, the Pareto and Normal Distributions
4. Continuous Random Variables, the Pareto and Normal Distributions A continuous random variable X can take any value in a given range (e.g. height, weight, age). The distribution of a continuous random
More informationarxiv:1112.0829v1 [math.pr] 5 Dec 2011
How Not to Win a Million Dollars: A Counterexample to a Conjecture of L. Breiman Thomas P. Hayes arxiv:1112.0829v1 [math.pr] 5 Dec 2011 Abstract Consider a gambling game in which we are allowed to repeatedly
More informationNotes from Week 1: Algorithms for sequential prediction
CS 683 Learning, Games, and Electronic Markets Spring 2007 Notes from Week 1: Algorithms for sequential prediction Instructor: Robert Kleinberg 2226 Jan 2007 1 Introduction In this course we will be looking
More informationMTH135/STA104: Probability
MTH135/STA14: Probability Homework # 8 Due: Tuesday, Nov 8, 5 Prof Robert Wolpert 1 Define a function f(x, y) on the plane R by { 1/x < y < x < 1 f(x, y) = other x, y a) Show that f(x, y) is a joint probability
More informationAnNajah National University Faculty of Engineering Industrial Engineering Department. Course : Quantitative Methods (65211)
AnNajah National University Faculty of Engineering Industrial Engineering Department Course : Quantitative Methods (65211) Instructor: Eng. Tamer Haddad 2 nd Semester 2009/2010 Chapter 5 Example: Joint
More informationSection 6.1 Joint Distribution Functions
Section 6.1 Joint Distribution Functions We often care about more than one random variable at a time. DEFINITION: For any two random variables X and Y the joint cumulative probability distribution function
More informationChapter Prove or disprove: A (B C) = (A B) (A C). Ans: True, since
Chapter 2 1. Prove or disprove: A (B C) = (A B) (A C)., since A ( B C) = A B C = A ( B C) = ( A B) ( A C) = ( A B) ( A C). 2. Prove that A B= A B by giving a containment proof (that is, prove that the
More informationAlmost every lossy compression system contains a lossless compression system
Lossless compression in lossy compression systems Almost every lossy compression system contains a lossless compression system Lossy compression system Transform Quantizer Lossless Encoder Lossless Decoder
More informationReview for Calculus Rational Functions, Logarithms & Exponentials
Definition and Domain of Rational Functions A rational function is defined as the quotient of two polynomial functions. F(x) = P(x) / Q(x) The domain of F is the set of all real numbers except those for
More informationIntroduction to Learning & Decision Trees
Artificial Intelligence: Representation and Problem Solving 538 April 0, 2007 Introduction to Learning & Decision Trees Learning and Decision Trees to learning What is learning?  more than just memorizing
More informationInfluences in lowdegree polynomials
Influences in lowdegree polynomials Artūrs Bačkurs December 12, 2012 1 Introduction In 3] it is conjectured that every bounded real polynomial has a highly influential variable The conjecture is known
More informationProblem Set I Lossless Coding
Problem Set I Lossless Coding Christopher Tsai Problem #2 Markov Source Model The twodimensional entropy rate function, is, Problem 2[ii]  Entropy Rate Function of a Markov Source.9 Entropy H(α, β).9.7.3..7.3
More informationSophomore College Mathematics of the Information Age Entropy
Sophomore College Mathematics of the Information Age Entropy Let s recall the fundamental notions of information and entropy. To repeat from the first class, Shannon s emphasis is on selecting a given
More informationLecture 8. Confidence intervals and the central limit theorem
Lecture 8. Confidence intervals and the central limit theorem Mathematical Statistics and Discrete Mathematics November 25th, 2015 1 / 15 Central limit theorem Let X 1, X 2,... X n be a random sample of
More informationConvex Functions (cont. 2)
Convex Functions (cont. 2) Optimization  10725 Carlos Guestrin Carnegie Mellon University March 5 th, 2008 1 Convex Functions Function f:r n R is convex if Domain is convex Generalization: Jensen s inequality:
More informationConvex Functions (cont. 2)
Convex Functions (cont. 2) Optimization  10725 Carlos Guestrin Carnegie Mellon University March 5 th, 2008 1 Convex Functions Function f:r n R is convex if Domain is convex Generalization: Jensen s inequality:
More informationMATH 425, PRACTICE FINAL EXAM SOLUTIONS.
MATH 45, PRACTICE FINAL EXAM SOLUTIONS. Exercise. a Is the operator L defined on smooth functions of x, y by L u := u xx + cosu linear? b Does the answer change if we replace the operator L by the operator
More informationInformation Theoretic Analysis of Proactive Routing Overhead in Mobile Ad Hoc Networks
Information Theoretic Analysis of Proactive Routing Overhead in obile Ad Hoc Networks Nianjun Zhou and Alhussein A. Abouzeid 1 Abstract This paper considers basic bounds on the overhead of linkstate protocols
More information2.8 Expected values and variance
y 1 b a 0 y 0.0 0.2 0.4 0.6 0.8 1.0 1.2 0 a b (a) Probability density function x 0 a b (b) Cumulative distribution function x Figure 2.3: The probability density function and cumulative distribution function
More informationP (x) 0. Discrete random variables Expected value. The expected value, mean or average of a random variable x is: xp (x) = v i P (v i )
Discrete random variables Probability mass function Given a discrete random variable X taking values in X = {v 1,..., v m }, its probability mass function P : X [0, 1] is defined as: P (v i ) = Pr[X =
More informationMaster s Theory Exam Spring 2006
Spring 2006 This exam contains 7 questions. You should attempt them all. Each question is divided into parts to help lead you through the material. You should attempt to complete as much of each problem
More informationProbability Generating Functions
page 39 Chapter 3 Probability Generating Functions 3 Preamble: Generating Functions Generating functions are widely used in mathematics, and play an important role in probability theory Consider a sequence
More informationDefinition: Suppose that two random variables, either continuous or discrete, X and Y have joint density
HW MATH 461/561 Lecture Notes 15 1 Definition: Suppose that two random variables, either continuous or discrete, X and Y have joint density and marginal densities f(x, y), (x, y) Λ X,Y f X (x), x Λ X,
More informationMeasuring Intrusion Detection Capability: An InformationTheoretic Approach
Measuring Intrusion Detection Capability: An InformationTheoretic Approach Guofei Gu, Prahlad Fogla, David Dagon, Boris Škorić Wenke Lee Philips Research Laboratories, Netherlands Georgia Institute of
More informationProbability and Statistics
CHAPTER 2: RANDOM VARIABLES AND ASSOCIATED FUNCTIONS 2b  0 Probability and Statistics Kristel Van Steen, PhD 2 Montefiore Institute  Systems and Modeling GIGA  Bioinformatics ULg kristel.vansteen@ulg.ac.be
More informationMAS108 Probability I
1 QUEEN MARY UNIVERSITY OF LONDON 2:30 pm, Thursday 3 May, 2007 Duration: 2 hours MAS108 Probability I Do not start reading the question paper until you are instructed to by the invigilators. The paper
More informationChapter 3: Discrete Random Variable and Probability Distribution. January 28, 2014
STAT511 Spring 2014 Lecture Notes 1 Chapter 3: Discrete Random Variable and Probability Distribution January 28, 2014 3 Discrete Random Variables Chapter Overview Random Variable (r.v. Definition Discrete
More informationMath 55: Discrete Mathematics
Math 55: Discrete Mathematics UC Berkeley, Fall 2011 Homework # 5, due Wednesday, February 22 5.1.4 Let P (n) be the statement that 1 3 + 2 3 + + n 3 = (n(n + 1)/2) 2 for the positive integer n. a) What
More informationLecture 4: BK inequality 27th August and 6th September, 2007
CSL866: Percolation and Random Graphs IIT Delhi Amitabha Bagchi Scribe: Arindam Pal Lecture 4: BK inequality 27th August and 6th September, 2007 4. Preliminaries The FKG inequality allows us to lower bound
More informationMASSACHUSETTS INSTITUTE OF TECHNOLOGY 6.436J/15.085J Fall 2008 Lecture 5 9/17/2008 RANDOM VARIABLES
MASSACHUSETTS INSTITUTE OF TECHNOLOGY 6.436J/15.085J Fall 2008 Lecture 5 9/17/2008 RANDOM VARIABLES Contents 1. Random variables and measurable functions 2. Cumulative distribution functions 3. Discrete
More informationImportant Probability Distributions OPRE 6301
Important Probability Distributions OPRE 6301 Important Distributions... Certain probability distributions occur with such regularity in reallife applications that they have been given their own names.
More informationWhat is Statistics? Lecture 1. Introduction and probability review. Idea of parametric inference
0. 1. Introduction and probability review 1.1. What is Statistics? What is Statistics? Lecture 1. Introduction and probability review There are many definitions: I will use A set of principle and procedures
More informationTopic 4: Multivariate random variables. Multiple random variables
Topic 4: Multivariate random variables Joint, marginal, and conditional pmf Joint, marginal, and conditional pdf and cdf Independence Expectation, covariance, correlation Conditional expectation Two jointly
More informationHomework Zero. Max H. Farrell Chicago Booth BUS41100 Applied Regression Analysis. Complete before the first class, do not turn in
Homework Zero Max H. Farrell Chicago Booth BUS41100 Applied Regression Analysis Complete before the first class, do not turn in This homework is intended as a self test of your knowledge of the basic statistical
More informationMULTIVARIATE PROBABILITY DISTRIBUTIONS
MULTIVARIATE PROBABILITY DISTRIBUTIONS. PRELIMINARIES.. Example. Consider an experiment that consists of tossing a die and a coin at the same time. We can consider a number of random variables defined
More informationPennies and Blood. Mike Bomar
Pennies and Blood Mike Bomar In partial fulfillment of the requirements for the Master of Arts in Teaching with a Specialization in the Teaching of Middle Level Mathematics in the Department of Mathematics.
More informationIntroduction to the Practice of Statistics Fifth Edition Moore, McCabe Section 4.4 Homework
Introduction to the Practice of Statistics Fifth Edition Moore, McCabe Section 4.4 Homework 4.65 You buy a hot stock for $1000. The stock either gains 30% or loses 25% each day, each with probability.
More informationChapter Objectives. Chapter 9. Sequential Search. Search Algorithms. Search Algorithms. Binary Search
Chapter Objectives Chapter 9 Search Algorithms Data Structures Using C++ 1 Learn the various search algorithms Explore how to implement the sequential and binary search algorithms Discover how the sequential
More informationRandom variables, probability distributions, binomial random variable
Week 4 lecture notes. WEEK 4 page 1 Random variables, probability distributions, binomial random variable Eample 1 : Consider the eperiment of flipping a fair coin three times. The number of tails that
More informationWorked examples Random Processes
Worked examples Random Processes Example 1 Consider patients coming to a doctor s office at random points in time. Let X n denote the time (in hrs) that the n th patient has to wait before being admitted
More informationMATH 201. Final ANSWERS August 12, 2016
MATH 01 Final ANSWERS August 1, 016 Part A 1. 17 points) A bag contains three different types of dice: four 6sided dice, five 8sided dice, and six 0sided dice. A die is drawn from the bag and then rolled.
More informationThe random variable X  the no. of defective items when three electronic components are tested would be
RANDOM VARIABLES and PROBABILITY DISTRIBUTIONS Example: Give the sample space giving a detailed description of each possible outcome when three electronic components are tested, where N  denotes nondefective
More informationAdaptive Online Gradient Descent
Adaptive Online Gradient Descent Peter L Bartlett Division of Computer Science Department of Statistics UC Berkeley Berkeley, CA 94709 bartlett@csberkeleyedu Elad Hazan IBM Almaden Research Center 650
More informationThe mathematics behind wireless communication
June 2008 Questions and setting In wireless communication, information is sent through what is called a channel. The channel is subject to noise, so that there will be some loss of information. How should
More information4. Joint Distributions of Two Random Variables
4. Joint Distributions of Two Random Variables 4.1 Joint Distributions of Two Discrete Random Variables Suppose the discrete random variables X and Y have supports S X and S Y, respectively. The joint
More informationData Analysis and Uncertainty Part 3: Hypothesis Testing/Sampling
Data Analysis and Uncertainty Part 3: Hypothesis Testing/Sampling Instructor: Sargur N. University at Buffalo The State University of New York srihari@cedar.buffalo.edu Topics 1. Hypothesis Testing 1.
More information1 Approximating Set Cover
CS 05: Algorithms (Grad) Feb 224, 2005 Approximating Set Cover. Definition An Instance (X, F ) of the setcovering problem consists of a finite set X and a family F of subset of X, such that every elemennt
More informationST 371 (IV): Discrete Random Variables
ST 371 (IV): Discrete Random Variables 1 Random Variables A random variable (rv) is a function that is defined on the sample space of the experiment and that assigns a numerical variable to each possible
More information5.4. HUFFMAN CODES 71
5.4. HUFFMAN CODES 7 Corollary 28 Consider a coding from a length n vector of source symbols, x = (x... x n ), to a binary codeword of length l(x). Then the average codeword length per source symbol for
More informationSummary of Formulas and Concepts. Descriptive Statistics (Ch. 14)
Summary of Formulas and Concepts Descriptive Statistics (Ch. 14) Definitions Population: The complete set of numerical information on a particular quantity in which an investigator is interested. We assume
More informationMath Review. for the Quantitative Reasoning Measure of the GRE revised General Test
Math Review for the Quantitative Reasoning Measure of the GRE revised General Test www.ets.org Overview This Math Review will familiarize you with the mathematical skills and concepts that are important
More informationSTAT 430/510 Probability Lecture 14: Joint Probability Distribution, Continuous Case
STAT 430/510 Probability Lecture 14: Joint Probability Distribution, Continuous Case Pengyuan (Penelope) Wang June 20, 2011 Joint density function of continuous Random Variable When X and Y are two continuous
More informationProbabilities. Probability of a event. From Random Variables to Events. From Random Variables to Events. Probability Theory I
Victor Adamchi Danny Sleator Great Theoretical Ideas In Computer Science Probability Theory I CS 525 Spring 200 Lecture Feb. 6, 200 Carnegie Mellon University We will consider chance experiments with
More informationCompression techniques
Compression techniques David Bařina February 22, 2013 David Bařina Compression techniques February 22, 2013 1 / 37 Contents 1 Terminology 2 Simple techniques 3 Entropy coding 4 Dictionary methods 5 Conclusion
More information1 Definition of Conditional Expectation
1 DEFINITION OF CONDITIONAL EXPECTATION 4 1 Definition of Conditional Expectation 1.1 General definition Recall the definition of conditional probability associated with Bayes Rule P(A B) P(A B) P(B) For
More informationFoundation of Video Coding Part I: Overview and Binary Encoding
Foundation of Video Coding Part I: Overview and Binary Encoding Yao Wang Polytechnic University, Brooklyn, NY11201 http://eeweb.poly.edu Based on: Y. Wang, J. Ostermann, and Y.Q. Zhang, Video Processing
More informationSTA 247 Answers for practice problem set #1
STA 247 Answers for practice problem set #1 Question 1: The random variable X has a range of {0, 1, 2} and the random variable Y has a range of {1, 2}. The joint distribution of X and Y is given by the
More informationLecture 14. Architecture: Design Principles
6.976/ESD.937 Quantitative Foundations of Engineering Systems Feb 131521, 2006 Lecturer: D. Shah and S. Mitter Lecture 14 Scribe: J. Laracy and D. Shah Architecture: Design Principles Architecture is
More information2WB05 Simulation Lecture 8: Generating random variables
2WB05 Simulation Lecture 8: Generating random variables Marko Boon http://www.win.tue.nl/courses/2wb05 January 7, 2013 Outline 2/36 1. How do we generate random variables? 2. Fitting distributions Generating
More informationTail inequalities for order statistics of logconcave vectors and applications
Tail inequalities for order statistics of logconcave vectors and applications Rafał Latała Based in part on a joint work with R.Adamczak, A.E.Litvak, A.Pajor and N.TomczakJaegermann Banff, May 2011 Basic
More informationStat 110. Unit 3: Random Variables Chapter 3 in the Text
Stat 110 Unit 3: Random Variables Chapter 3 in the Text 1 Unit 3 Outline Random Variables (RVs) and Distributions Discrete RVs and Probability Mass Functions (PMFs) Bernoulli and Binomial Distributions
More informationOn the Sum Capacity of the Gaussian Multiple Access Channel with Feedback
FortySeventh Annual Allerton Conference Allerton House, UIUC, Illinois, USA September 30  October, 009 On the Sum Capacity of the Gaussian Multiple Access Channel with Feedback Ehsan Ardetsanizadeh,
More information2. Discrete random variables
2. Discrete random variables Statistics and probability: 21 If the chance outcome of the experiment is a number, it is called a random variable. Discrete random variable: the possible outcomes can be
More informationBinomial distribution From Wikipedia, the free encyclopedia See also: Negative binomial distribution
Binomial distribution From Wikipedia, the free encyclopedia See also: Negative binomial distribution In probability theory and statistics, the binomial distribution is the discrete probability distribution
More informationH/wk 13, Solutions to selected problems
H/wk 13, Solutions to selected problems Ch. 4.1, Problem 5 (a) Find the number of roots of x x in Z 4, Z Z, any integral domain, Z 6. (b) Find a commutative ring in which x x has infinitely many roots.
More informationFor a partition B 1,..., B n, where B i B j = for i. A = (A B 1 ) (A B 2 ),..., (A B n ) and thus. P (A) = P (A B i ) = P (A B i )P (B i )
Probability Review 15.075 Cynthia Rudin A probability space, defined by Kolmogorov (19031987) consists of: A set of outcomes S, e.g., for the roll of a die, S = {1, 2, 3, 4, 5, 6}, 1 1 2 1 6 for the roll
More informationLectures 56: Taylor Series
Math 1d Instructor: Padraic Bartlett Lectures 5: Taylor Series Weeks 5 Caltech 213 1 Taylor Polynomials and Series As we saw in week 4, power series are remarkably nice objects to work with. In particular,
More informationECEN 5682 Theory and Practice of Error Control Codes
ECEN 5682 Theory and Practice of Error Control Codes Convolutional Codes University of Colorado Spring 2007 Linear (n, k) block codes take k data symbols at a time and encode them into n code symbols.
More informationMath 431 An Introduction to Probability. Final Exam Solutions
Math 43 An Introduction to Probability Final Eam Solutions. A continuous random variable X has cdf a for 0, F () = for 0 <
More informationTossing a Biased Coin
Tossing a Biased Coin Michael Mitzenmacher When we talk about a coin toss, we think of it as unbiased: with probability onehalf it comes up heads, and with probability onehalf it comes up tails. An ideal
More informationMT426 Notebook 3 Fall 2012 prepared by Professor Jenny Baglivo. 3 MT426 Notebook 3 3. 3.1 Definitions... 3. 3.2 Joint Discrete Distributions...
MT426 Notebook 3 Fall 2012 prepared by Professor Jenny Baglivo c Copyright 20042012 by Jenny A. Baglivo. All Rights Reserved. Contents 3 MT426 Notebook 3 3 3.1 Definitions............................................
More informationChapter There are nonisomorphic rooted trees with four vertices. Ans: 4.
Use the following to answer questions 126: In the questions below fill in the blanks. Chapter 10 1. If T is a tree with 999 vertices, then T has edges. 998. 2. There are nonisomorphic trees with four
More informationTHE NUMBER OF GRAPHS AND A RANDOM GRAPH WITH A GIVEN DEGREE SEQUENCE. Alexander Barvinok
THE NUMBER OF GRAPHS AND A RANDOM GRAPH WITH A GIVEN DEGREE SEQUENCE Alexer Barvinok Papers are available at http://www.math.lsa.umich.edu/ barvinok/papers.html This is a joint work with J.A. Hartigan
More information