Entropy Rates of a Stochastic Process

Save this PDF as:
 WORD  PNG  TXT  JPG

Size: px
Start display at page:

Download "Entropy Rates of a Stochastic Process"

Transcription

1 Etropy Rates of a Stochastic Process Best Achievable Data Compressio Radu Trîmbiţaş October Etropy Rates of a Stochastic Process Etropy rates The AEP states that H(X) bits suffice o the average for i.i.d. RVs What for depedet RVs? For statioary processes H(X 1, X 2,..., X ) grows (asymptotically) liearly with at a rate H(X ) the etropy rate of the process A stochastic process {X i } i I is a idexed sequece of radom variables, X i : S X is a RV i I If I N, {X 1, X 2,... } is a discrete stochastic process, called also a discrete iformatio source. A discrete stochastic process is characterized by the joit probability mass fuctio P((X 1, X 2,..., X ) = (x 1, x 2,..., x )) = p(x 1, x 2,..., x ) where (x 1, x 2,..., x ) X. 1.1 Markov chais Markov chais Defiitio 1. A stochastic process is said to be statioary if the joit distributio of ay subset of the sequece of radom variables is ivariat with respect to shifts i the time idex P(X 1 = x 1,..., X = x ) = P(X 1+l = x 1,..., X +l = x ) (1), l ad x 1, x 2,..., x X. 1

2 Defiitio 2. A discrete stochastic process {X 1, X 2,... } is said to be a Markov chai or Markov process if for = 1, 2,... P (X +1 = x +1 X = x, X 1 = x 1,..., X 1 = x 1 ) = P(X +1 = x +1 X = x ), x 1, x 2,..., x, x +1 X. (2) The joit pmf ca be writte as p(x 1, x 2,..., x ) = p(x 1 )p(x 2 x 1 )... p(x x 1 ). (3) Defiitio 3. A Markov chai is said to be time ivariat (time homogeeous) if the coditioal probability p(x +1 x ) does ot deped o ; that is for = 1, 2,... P (X +1 = b X = a) = P(X 2 = b X 1 = a), a, b X. (4) This property is assumed uless otherwise stated. {X i } Markov chai, X is called the state at time A time-ivariat Markov chai is characterized by its iitial state ad a probability trasitio matrix P = [P ij ], i, j = 1,..., m, where P ij = P(X +1 = j X = i). The Markov chai {X } is irreducible if it is possible to go from ay state to aother with a probability > 0 The Markov chai {X } is aperiodic if state a, the possible times to go from a to a have highest commo factor = 1. Markov chais are ofte described by a directed graph where the edges are labeled by the probability of goig from oe state to aother. p(x ) - pmf of the radom variable at time p(x +1 ) = x p(x )P x x +1 (5) A distributio o the states such that the distributio at time + 1 is the same as the distributio at time is called a statioary distributio - so called because if the iitial state of a Markov chai is draw accordig to a statioary distributio, the Markov chai form a statioary process. If the fiite-state Markov chai is irreducible ad periodic, the statioary distributio is uique, ad from ay startig distributio, the distributio of X teds to a statioary distributio as. Example 4. Cosider a two-state Markov chai with a probability trasitio matrix [ ] 1 α α P = β 1 β (Figure 1) 2

3 Figure 1: Two-state Markov chai The statioary probability is the solutio of µp = µ or (I P T )µ T = 0. We add the coditio µ 1 + µ 2 = 0. The solutio is µ 1 = β α + β, µ 2 = α α + β. Click here for a Maple solutio Markovex1.html. The etropy of X is ( ) β H(X ) = H α + β, α. α + β 1.2 Etropy rate Etropy rate Defiitio 5. The etropy rate of a stochastic process {X i } is defied by whe the limit exists. Examples 1 H(X ) = lim H (X 1,..., X ) (6) 1. Typewriter - m equally likely output letters; he(she) ca produce m sequeces of legth, all of them equally likely. H (X 1,..., X ) = log m, ad the etropy rate is H(X ) = log m bits per symbol. 2. X 1, X 2,... i.i.d. RVs H (X H(X ) = lim 1,..., X ) H(X = lim 1 ) = H (X 1 ). 3. X 1, X 2,... idepedet, but ot idetically distributed RVs H (X 1,..., X ) = It is possible that 1 H(x i) does ot exists H(X i ) 3

4 Defiitio 6. H (X ) = lim H (X X 1, X 2,... X 1 ). (7) H(X ) is etropy per symbol of the RVs; H (X ) is the coditioal etropy of the last RV give the past. For statioary processes both limits exist ad are equal. Lemma 7. For a statioary stochastic process, H(X X 1,..., X 1 ) is oicreasig i ad has a limit H (X ). Proof. H(X +1 X 1, X 2,..., X ) H(X +1 X,..., X 2 ) coditioig = H(X X 1,..., X 1 ). statioarity (H(X X 1,..., X 1 )) is decreasig ad oegative, so it has a limit H (X ). Lemma 8 (Cesáro). If a a ad b = 1 a i the b a. Theorem 9. For a statioary stochastic process H(X ) (give by (6)) ad H (X ) (give by (7)) exist ad H(X ) = H (X ). (8) Proof. By the chai rule, But, H (X 1,..., X ) = 1 H (X H(X ) = lim 1,..., X ) 1 = lim H(X i X i 1,..., X 1 ) H(X i X i 1,..., X 1 ). = lim H(X X 1,..., X 1 ) (Lemma 8) = H (X ) (Lemma 7) 1.3 Etropy rate for Markov chai Etropy rate for Markov chai For a statioary Markov chai, the etropy rate is give by H (X ) = H (X ) = lim H (X X 1,..., X 1 ) = lim H (X X 1 ) = H(X 2 X 1 ), (9) where the coditioal etropy is calculated usig the give statioary distributio. 4

5 The statioary distributio µ is the solutio of the equatios Expressio of coditioal etropy: µ j = µ i P ij, j. i Theorem 10. {X i } statioary Markov chai with statioary distributio µ ad trasitio matrix P. Let X 1 µ. the the etropy rate is H(X ) = i Proof. H(X ) = H(X 2 X 1 ) = i µ i ( j P ij log P ij ). µ i P ij log P ij. (10) j Example 11 (Two-state Markov chai). The etropy rate of the two-state Markov chai i Figure 1 is H(X ) = H(X 2 X 1 ) = β α + β H(α) + α α + β H(β). Remark. If the Markov chai is irreducible ad aperiodic, it has a uique statioary distributio o the states, ad ay iitial distributio teds to the statioary distributio as. I this case, eve though the iitial distributio is ot the statioary distributio, the etropy rate, which is defied i terms of log-term behavior, is H(X ), as defied i (9) ad (10). 1.4 Fuctios of Markov chais Fuctios of Markov chais X 1, X 2,..., X,... statioary Markov chai, Y i = φ(x i ), H(Y) =? i may cases Y 1, Y 2,..., Y,... is ot a Markov chai, but it is statioary lower boud Lemma 12. Proof. For k = 1, 2,... H(Y Y 1,..., Y 2, X 1 ) H(Y). (11) H(Y Y 1,..., Y 2, X 1 ) (a) = H(Y Y 1,..., Y 2, Y 1, X 1 ) (b) = H(Y Y 1,..., Y 2, Y 1, X 1, X 0, X 1,..., X k ) (c) = H (Y Y 1,..., Y 2, Y 1, X 1, X 0, X 1,..., X k, Y 0,..., Y k ) (d) H (Y Y 1,..., Y 2, Y 1, Y 0,..., Y k ) (e) = H (Y +k+1 Y +k,..., Y 1 ), 5

6 (a) follows from the fact that Y 1 = φ(x 1 ), (b) from the Markovity, (c) from Y i = φ(x i ), (d) coditioig reduces etropy, (e) statioarity. Proof - cotiuatio. Sice iequality is true for all k, i the limit H(Y Y 1,..., Y 2, X 1 ) lim H (Y +k+1 Y +k,..., Y 1 ) k = H(Y). Lemma 13. H(Y Y 1,..., Y 2, X 1 ) H(Y Y 1,..., Y 2, Y 1, X 1 ) 0. (12) Proof. Expressio of iterval legth: H(Y Y 1,..., Y 2, X 1 ) H(Y Y 1,..., Y 2, Y 1, X 1 ) = I(X 1 ; Y Y 1,..., Y 1 ). By properties of mutual iformatio, I(X 1 ; Y 1,..., Y ) H(X 1 ), ad I(X 1 ; Y 1,..., Y ) icreases with. Thus, lim I(X 1 ; Y 1,..., Y ) exists ad lim I(X 1; Y 1,..., Y ) H(X 1 ). Proof - cotiuatio. By the chai rule H(X 1 ) lim I(X 1 ; Y 1,..., Y ) = lim = I(X 1 ; Y i Y i 1,..., Y 1 ) I(X 1 ; Y i Y i 1,..., Y 1 ) The geeral term of the series must ted to 0 lim I (X 1 ; Y Y 1,..., Y 1 ) = 0. The last two lemmas imply Theorem 14. X 1, X 2,..., X,... statioary Markov chai, Y i = φ(x i ) H(Y Y 1,..., Y 1, X 1 ) H(Y) H(Y Y 1,..., Y 1 ) (13) ad lim H(Y Y 1,..., Y 1, X 1 ) = H(Y) = lim H(Y Y 1,..., Y 1 ) (14) 6

7 Hide Markov models We could cosider Y i to be a stochastic fuctio of X i X 1, X 2,..., X,... statioary Markov chai, Y 1, Y 2,..., Y,... a ew process where Y i is draw accordig to p(y i x i ), coditioally idepedet of all the other X j, j = i p(x, y 1 ) = p(x 1 ) p(x i+1 x i ) p(y i x i ). Y 1, Y 2,..., Y,... is called a hidde Markov model (HMM) Applied to speech recogitio, hadwritig recogitio, ad so o. The same argumet as for fuctios of Markov chai works for HMMs. Refereces Refereces [1] Thomas M. Cover, Joy A. Thomas, Elemets of Iformatio Theory, 2d editio, Wiley, [2] David J.C. MacKay, Iformatio Theory, Iferece, ad Learig Algorithms, Cambridge Uiversity Press, [3] Robert M. Gray, Etropy ad Iformatio Theory, Spriger,

Properties of MLE: consistency, asymptotic normality. Fisher information.

Properties of MLE: consistency, asymptotic normality. Fisher information. Lecture 3 Properties of MLE: cosistecy, asymptotic ormality. Fisher iformatio. I this sectio we will try to uderstad why MLEs are good. Let us recall two facts from probability that we be used ofte throughout

More information

Chapter 7 Methods of Finding Estimators

Chapter 7 Methods of Finding Estimators Chapter 7 for BST 695: Special Topics i Statistical Theory. Kui Zhag, 011 Chapter 7 Methods of Fidig Estimators Sectio 7.1 Itroductio Defiitio 7.1.1 A poit estimator is ay fuctio W( X) W( X1, X,, X ) of

More information

An example of non-quenched convergence in the conditional central limit theorem for partial sums of a linear process

An example of non-quenched convergence in the conditional central limit theorem for partial sums of a linear process A example of o-queched covergece i the coditioal cetral limit theorem for partial sums of a liear process Dalibor Volý ad Michael Woodroofe Abstract A causal liear processes X,X 0,X is costructed for which

More information

Department of Computer Science, University of Otago

Department of Computer Science, University of Otago Departmet of Computer Sciece, Uiversity of Otago Techical Report OUCS-2006-09 Permutatios Cotaiig May Patters Authors: M.H. Albert Departmet of Computer Sciece, Uiversity of Otago Micah Colema, Rya Fly

More information

A probabilistic proof of a binomial identity

A probabilistic proof of a binomial identity A probabilistic proof of a biomial idetity Joatho Peterso Abstract We give a elemetary probabilistic proof of a biomial idetity. The proof is obtaied by computig the probability of a certai evet i two

More information

UC Berkeley Department of Electrical Engineering and Computer Science. EE 126: Probablity and Random Processes. Solutions 9 Spring 2006

UC Berkeley Department of Electrical Engineering and Computer Science. EE 126: Probablity and Random Processes. Solutions 9 Spring 2006 Exam format UC Bereley Departmet of Electrical Egieerig ad Computer Sciece EE 6: Probablity ad Radom Processes Solutios 9 Sprig 006 The secod midterm will be held o Wedesday May 7; CHECK the fial exam

More information

SAMPLE QUESTIONS FOR FINAL EXAM. (1) (2) (3) (4) Find the following using the definition of the Riemann integral: (2x + 1)dx

SAMPLE QUESTIONS FOR FINAL EXAM. (1) (2) (3) (4) Find the following using the definition of the Riemann integral: (2x + 1)dx SAMPLE QUESTIONS FOR FINAL EXAM REAL ANALYSIS I FALL 006 3 4 Fid the followig usig the defiitio of the Riema itegral: a 0 x + dx 3 Cosider the partitio P x 0 3, x 3 +, x 3 +,......, x 3 3 + 3 of the iterval

More information

In nite Sequences. Dr. Philippe B. Laval Kennesaw State University. October 9, 2008

In nite Sequences. Dr. Philippe B. Laval Kennesaw State University. October 9, 2008 I ite Sequeces Dr. Philippe B. Laval Keesaw State Uiversity October 9, 2008 Abstract This had out is a itroductio to i ite sequeces. mai de itios ad presets some elemetary results. It gives the I ite Sequeces

More information

Chapter 5 Discrete Probability Distributions

Chapter 5 Discrete Probability Distributions Slides Prepared by JOHN S. LOUCKS St. Edward s Uiversity Slide Chapter 5 Discrete Probability Distributios Radom Variables Discrete Probability Distributios Epected Value ad Variace Poisso Distributio

More information

Module 4: Mathematical Induction

Module 4: Mathematical Induction Module 4: Mathematical Iductio Theme 1: Priciple of Mathematical Iductio Mathematical iductio is used to prove statemets about atural umbers. As studets may remember, we ca write such a statemet as a predicate

More information

The second difference is the sequence of differences of the first difference sequence, 2

The second difference is the sequence of differences of the first difference sequence, 2 Differece Equatios I differetial equatios, you look for a fuctio that satisfies ad equatio ivolvig derivatives. I differece equatios, istead of a fuctio of a cotiuous variable (such as time), we look for

More information

Lecture 4: Cheeger s Inequality

Lecture 4: Cheeger s Inequality Spectral Graph Theory ad Applicatios WS 0/0 Lecture 4: Cheeger s Iequality Lecturer: Thomas Sauerwald & He Su Statemet of Cheeger s Iequality I this lecture we assume for simplicity that G is a d-regular

More information

Approximating the Sum of a Convergent Series

Approximating the Sum of a Convergent Series Approximatig the Sum of a Coverget Series Larry Riddle Ages Scott College Decatur, GA 30030 lriddle@agesscott.edu The BC Calculus Course Descriptio metios how techology ca be used to explore covergece

More information

Overview of some probability distributions.

Overview of some probability distributions. Lecture Overview of some probability distributios. I this lecture we will review several commo distributios that will be used ofte throughtout the class. Each distributio is usually described by its probability

More information

The Common Information for N Dependent Random Variables

The Common Information for N Dependent Random Variables Syracuse Uiversity SURFACE Electrical Egieerig ad Computer Sciece Techical Reports College of Egieerig ad Computer Sciece 4--20 The Commo Iformatio for N Depedet Radom Variables Wei Liu Syracuse Uiversity

More information

Sequences II. Chapter 3. 3.1 Convergent Sequences

Sequences II. Chapter 3. 3.1 Convergent Sequences Chapter 3 Sequeces II 3. Coverget Sequeces Plot a graph of the sequece a ) = 2, 3 2, 4 3, 5 + 4,...,,... To what limit do you thik this sequece teds? What ca you say about the sequece a )? For ǫ = 0.,

More information

Asymptotic Growth of Functions

Asymptotic Growth of Functions CMPS Itroductio to Aalysis of Algorithms Fall 3 Asymptotic Growth of Fuctios We itroduce several types of asymptotic otatio which are used to compare the performace ad efficiecy of algorithms As we ll

More information

1 The Gaussian channel

1 The Gaussian channel ECE 77 Lecture 0 The Gaussia chael Objective: I this lecture we will lear about commuicatio over a chael of practical iterest, i which the trasmitted sigal is subjected to additive white Gaussia oise.

More information

Universal coding for classes of sources

Universal coding for classes of sources Coexios module: m46228 Uiversal codig for classes of sources Dever Greee This work is produced by The Coexios Project ad licesed uder the Creative Commos Attributio Licese We have discussed several parametric

More information

Joint Probability Distributions and Random Samples

Joint Probability Distributions and Random Samples STAT5 Sprig 204 Lecture Notes Chapter 5 February, 204 Joit Probability Distributios ad Radom Samples 5. Joitly Distributed Radom Variables Chapter Overview Joitly distributed rv Joit mass fuctio, margial

More information

Sequences and Series

Sequences and Series CHAPTER 9 Sequeces ad Series 9.. Covergece: Defiitio ad Examples Sequeces The purpose of this chapter is to itroduce a particular way of geeratig algorithms for fidig the values of fuctios defied by their

More information

1 Introduction to reducing variance in Monte Carlo simulations

1 Introduction to reducing variance in Monte Carlo simulations Copyright c 007 by Karl Sigma 1 Itroductio to reducig variace i Mote Carlo simulatios 11 Review of cofidece itervals for estimatig a mea I statistics, we estimate a uow mea µ = E(X) of a distributio by

More information

Discrete Random Variables and Probability Distributions. Random Variables. Chapter 3 3.1

Discrete Random Variables and Probability Distributions. Random Variables. Chapter 3 3.1 UCLA STAT A Applied Probability & Statistics for Egieers Istructor: Ivo Diov, Asst. Prof. I Statistics ad Neurology Teachig Assistat: Neda Farziia, UCLA Statistics Uiversity of Califoria, Los Ageles, Sprig

More information

Infinite Sequences and Series

Infinite Sequences and Series CHAPTER 4 Ifiite Sequeces ad Series 4.1. Sequeces A sequece is a ifiite ordered list of umbers, for example the sequece of odd positive itegers: 1, 3, 5, 7, 9, 11, 13, 15, 17, 19, 21, 23, 25, 27, 29...

More information

A RANDOM PERMUTATION MODEL ARISING IN CHEMISTRY

A RANDOM PERMUTATION MODEL ARISING IN CHEMISTRY J. Appl. Prob. 45, 060 070 2008 Prited i Eglad Applied Probability Trust 2008 A RANDOM PERMUTATION MODEL ARISING IN CHEMISTRY MARK BROWN, The City College of New York EROL A. PEKÖZ, Bosto Uiversity SHELDON

More information

. P. 4.3 Basic feasible solutions and vertices of polyhedra. x 1. x 2

. P. 4.3 Basic feasible solutions and vertices of polyhedra. x 1. x 2 4. Basic feasible solutios ad vertices of polyhedra Due to the fudametal theorem of Liear Programmig, to solve ay LP it suffices to cosider the vertices (fiitely may) of the polyhedro P of the feasible

More information

Lecture 5: Span, linear independence, bases, and dimension

Lecture 5: Span, linear independence, bases, and dimension Lecture 5: Spa, liear idepedece, bases, ad dimesio Travis Schedler Thurs, Sep 23, 2010 (versio: 9/21 9:55 PM) 1 Motivatio Motivatio To uderstad what it meas that R has dimesio oe, R 2 dimesio 2, etc.;

More information

Chapter 6: Variance, the law of large numbers and the Monte-Carlo method

Chapter 6: Variance, the law of large numbers and the Monte-Carlo method Chapter 6: Variace, the law of large umbers ad the Mote-Carlo method Expected value, variace, ad Chebyshev iequality. If X is a radom variable recall that the expected value of X, E[X] is the average value

More information

5 Boolean Decision Trees (February 11)

5 Boolean Decision Trees (February 11) 5 Boolea Decisio Trees (February 11) 5.1 Graph Coectivity Suppose we are give a udirected graph G, represeted as a boolea adjacecy matrix = (a ij ), where a ij = 1 if ad oly if vertices i ad j are coected

More information

0.7 0.6 0.2 0 0 96 96.5 97 97.5 98 98.5 99 99.5 100 100.5 96.5 97 97.5 98 98.5 99 99.5 100 100.5

0.7 0.6 0.2 0 0 96 96.5 97 97.5 98 98.5 99 99.5 100 100.5 96.5 97 97.5 98 98.5 99 99.5 100 100.5 Sectio 13 Kolmogorov-Smirov test. Suppose that we have a i.i.d. sample X 1,..., X with some ukow distributio P ad we would like to test the hypothesis that P is equal to a particular distributio P 0, i.e.

More information

The Euler Totient, the Möbius and the Divisor Functions

The Euler Totient, the Möbius and the Divisor Functions The Euler Totiet, the Möbius ad the Divisor Fuctios Rosica Dieva July 29, 2005 Mout Holyoke College South Hadley, MA 01075 1 Ackowledgemets This work was supported by the Mout Holyoke College fellowship

More information

Ekkehart Schlicht: Economic Surplus and Derived Demand

Ekkehart Schlicht: Economic Surplus and Derived Demand Ekkehart Schlicht: Ecoomic Surplus ad Derived Demad Muich Discussio Paper No. 2006-17 Departmet of Ecoomics Uiversity of Muich Volkswirtschaftliche Fakultät Ludwig-Maximilias-Uiversität Müche Olie at http://epub.ub.ui-mueche.de/940/

More information

Week 3 Conditional probabilities, Bayes formula, WEEK 3 page 1 Expected value of a random variable

Week 3 Conditional probabilities, Bayes formula, WEEK 3 page 1 Expected value of a random variable Week 3 Coditioal probabilities, Bayes formula, WEEK 3 page 1 Expected value of a radom variable We recall our discussio of 5 card poker hads. Example 13 : a) What is the probability of evet A that a 5

More information

A gentle introduction to Expectation Maximization

A gentle introduction to Expectation Maximization A getle itroductio to Expectatio Maximizatio Mark Johso Brow Uiversity November 2009 1 / 15 Outlie What is Expectatio Maximizatio? Mixture models ad clusterig EM for setece topic modelig 2 / 15 Why Expectatio

More information

3. Covariance and Correlation

3. Covariance and Correlation Virtual Laboratories > 3. Expected Value > 1 2 3 4 5 6 3. Covariace ad Correlatio Recall that by takig the expected value of various trasformatios of a radom variable, we ca measure may iterestig characteristics

More information

Parameter estimation for nonlinear models: Numerical approaches to solving the inverse problem. Lecture 11 04/01/2008. Sven Zenker

Parameter estimation for nonlinear models: Numerical approaches to solving the inverse problem. Lecture 11 04/01/2008. Sven Zenker Parameter estimatio for oliear models: Numerical approaches to solvig the iverse problem Lecture 11 04/01/2008 Sve Zeker Review: Trasformatio of radom variables Cosider probability distributio of a radom

More information

Lecture 4: Cauchy sequences, Bolzano-Weierstrass, and the Squeeze theorem

Lecture 4: Cauchy sequences, Bolzano-Weierstrass, and the Squeeze theorem Lecture 4: Cauchy sequeces, Bolzao-Weierstrass, ad the Squeeze theorem The purpose of this lecture is more modest tha the previous oes. It is to state certai coditios uder which we are guarateed that limits

More information

THE ABRACADABRA PROBLEM

THE ABRACADABRA PROBLEM THE ABRACADABRA PROBLEM FRANCESCO CARAVENNA Abstract. We preset a detailed solutio of Exercise E0.6 i [Wil9]: i a radom sequece of letters, draw idepedetly ad uiformly from the Eglish alphabet, the expected

More information

A Constant-Factor Approximation Algorithm for the Link Building Problem

A Constant-Factor Approximation Algorithm for the Link Building Problem A Costat-Factor Approximatio Algorithm for the Lik Buildig Problem Marti Olse 1, Aastasios Viglas 2, ad Ilia Zvedeiouk 2 1 Ceter for Iovatio ad Busiess Developmet, Istitute of Busiess ad Techology, Aarhus

More information

Our aim is to show that under reasonable assumptions a given 2π-periodic function f can be represented as convergent series

Our aim is to show that under reasonable assumptions a given 2π-periodic function f can be represented as convergent series 8 Fourier Series Our aim is to show that uder reasoable assumptios a give -periodic fuctio f ca be represeted as coverget series f(x) = a + (a cos x + b si x). (8.) By defiitio, the covergece of the series

More information

Soving Recurrence Relations

Soving Recurrence Relations Sovig Recurrece Relatios Part 1. Homogeeous liear 2d degree relatios with costat coefficiets. Cosider the recurrece relatio ( ) T () + at ( 1) + bt ( 2) = 0 This is called a homogeeous liear 2d degree

More information

Incremental calculation of weighted mean and variance

Incremental calculation of weighted mean and variance Icremetal calculatio of weighted mea ad variace Toy Fich faf@cam.ac.uk dot@dotat.at Uiversity of Cambridge Computig Service February 009 Abstract I these otes I eplai how to derive formulae for umerically

More information

x(x 1)(x 2)... (x k + 1) = [x] k n+m 1

x(x 1)(x 2)... (x k + 1) = [x] k n+m 1 1 Coutig mappigs For every real x ad positive iteger k, let [x] k deote the fallig factorial ad x(x 1)(x 2)... (x k + 1) ( ) x = [x] k k k!, ( ) k = 1. 0 I the sequel, X = {x 1,..., x m }, Y = {y 1,...,

More information

THE REGRESSION MODEL IN MATRIX FORM. For simple linear regression, meaning one predictor, the model is. for i = 1, 2, 3,, n

THE REGRESSION MODEL IN MATRIX FORM. For simple linear regression, meaning one predictor, the model is. for i = 1, 2, 3,, n We will cosider the liear regressio model i matrix form. For simple liear regressio, meaig oe predictor, the model is i = + x i + ε i for i =,,,, This model icludes the assumptio that the ε i s are a sample

More information

Chapter 7 - Sampling Distributions. 1 Introduction. What is statistics? It consist of three major areas:

Chapter 7 - Sampling Distributions. 1 Introduction. What is statistics? It consist of three major areas: Chapter 7 - Samplig Distributios 1 Itroductio What is statistics? It cosist of three major areas: Data Collectio: samplig plas ad experimetal desigs Descriptive Statistics: umerical ad graphical summaries

More information

Convexity, Inequalities, and Norms

Convexity, Inequalities, and Norms Covexity, Iequalities, ad Norms Covex Fuctios You are probably familiar with the otio of cocavity of fuctios. Give a twicedifferetiable fuctio ϕ: R R, We say that ϕ is covex (or cocave up) if ϕ (x) 0 for

More information

Optimal Lossless Data Compression: Non-Asymptotics and Asymptotics Ioannis Kontoyiannis, Fellow, IEEE, and Sergio Verdú, Fellow, IEEE

Optimal Lossless Data Compression: Non-Asymptotics and Asymptotics Ioannis Kontoyiannis, Fellow, IEEE, and Sergio Verdú, Fellow, IEEE IEEE TRANSACTIONS ON INFORMATION THEORY, VOL. 60, NO. 2, FEBRUARY 204 777 Optimal Lossless Data Compressio: No-Asymptotics ad Asymptotics Ioais Kotoyiais, Fellow, IEEE, ad Sergio Verdú, Fellow, IEEE Abstract

More information

Distributions of Order Statistics

Distributions of Order Statistics Chapter 2 Distributios of Order Statistics We give some importat formulae for distributios of order statistics. For example, where F k: (x)=p{x k, x} = I F(x) (k, k + 1), I x (a,b)= 1 x t a 1 (1 t) b 1

More information

Permutations, the Parity Theorem, and Determinants

Permutations, the Parity Theorem, and Determinants 1 Permutatios, the Parity Theorem, ad Determiats Joh A. Guber Departmet of Electrical ad Computer Egieerig Uiversity of Wiscosi Madiso Cotets 1 What is a Permutatio 1 2 Cycles 2 2.1 Traspositios 4 3 Orbits

More information

Linear Algebra II. 4 Determinants. Notes 4 1st November Definition of determinant

Linear Algebra II. 4 Determinants. Notes 4 1st November Definition of determinant MTH6140 Liear Algebra II Notes 4 1st November 2010 4 Determiats The determiat is a fuctio defied o square matrices; its value is a scalar. It has some very importat properties: perhaps most importat is

More information

1. MATHEMATICAL INDUCTION

1. MATHEMATICAL INDUCTION 1. MATHEMATICAL INDUCTION EXAMPLE 1: Prove that for ay iteger 1. Proof: 1 + 2 + 3 +... + ( + 1 2 (1.1 STEP 1: For 1 (1.1 is true, sice 1 1(1 + 1. 2 STEP 2: Suppose (1.1 is true for some k 1, that is 1

More information

Installment Joint Life Insurance Actuarial Models with the Stochastic Interest Rate

Installment Joint Life Insurance Actuarial Models with the Stochastic Interest Rate Iteratioal Coferece o Maagemet Sciece ad Maagemet Iovatio (MSMI 4) Istallmet Joit Life Isurace ctuarial Models with the Stochastic Iterest Rate Nia-Nia JI a,*, Yue LI, Dog-Hui WNG College of Sciece, Harbi

More information

Sequences and Series Using the TI-89 Calculator

Sequences and Series Using the TI-89 Calculator RIT Calculator Site Sequeces ad Series Usig the TI-89 Calculator Norecursively Defied Sequeces A orecursively defied sequece is oe i which the formula for the terms of the sequece is give explicitly. For

More information

Chair for Network Architectures and Services Institute of Informatics TU München Prof. Carle. Network Security. Chapter 2 Basics

Chair for Network Architectures and Services Institute of Informatics TU München Prof. Carle. Network Security. Chapter 2 Basics Chair for Network Architectures ad Services Istitute of Iformatics TU Müche Prof. Carle Network Security Chapter 2 Basics 2.4 Radom Number Geeratio for Cryptographic Protocols Motivatio It is crucial to

More information

MARTINGALES AND A BASIC APPLICATION

MARTINGALES AND A BASIC APPLICATION MARTINGALES AND A BASIC APPLICATION TURNER SMITH Abstract. This paper will develop the measure-theoretic approach to probability i order to preset the defiitio of martigales. From there we will apply this

More information

SECTION 1.5 : SUMMATION NOTATION + WORK WITH SEQUENCES

SECTION 1.5 : SUMMATION NOTATION + WORK WITH SEQUENCES SECTION 1.5 : SUMMATION NOTATION + WORK WITH SEQUENCES Read Sectio 1.5 (pages 5 9) Overview I Sectio 1.5 we lear to work with summatio otatio ad formulas. We will also itroduce a brief overview of sequeces,

More information

Discrete Mathematics and Probability Theory Spring 2014 Anant Sahai Note 13

Discrete Mathematics and Probability Theory Spring 2014 Anant Sahai Note 13 EECS 70 Discrete Mathematics ad Probability Theory Sprig 2014 Aat Sahai Note 13 Itroductio At this poit, we have see eough examples that it is worth just takig stock of our model of probability ad may

More information

THE LEAST COMMON MULTIPLE OF A QUADRATIC SEQUENCE

THE LEAST COMMON MULTIPLE OF A QUADRATIC SEQUENCE THE LEAST COMMON MULTIPLE OF A QUADRATIC SEQUENCE JAVIER CILLERUELO Abstract. We obtai, for ay irreducible quadratic olyomial f(x = ax 2 + bx + c, the asymtotic estimate log l.c.m. {f(1,..., f(} log. Whe

More information

NATIONAL SENIOR CERTIFICATE GRADE 12

NATIONAL SENIOR CERTIFICATE GRADE 12 NATIONAL SENIOR CERTIFICATE GRADE MATHEMATICS P EXEMPLAR 04 MARKS: 50 TIME: 3 hours This questio paper cosists of 8 pages ad iformatio sheet. Please tur over Mathematics/P DBE/04 NSC Grade Eemplar INSTRUCTIONS

More information

Confidence Interval Estimation of the Shape. Parameter of Pareto Distribution. Using Extreme Order Statistics

Confidence Interval Estimation of the Shape. Parameter of Pareto Distribution. Using Extreme Order Statistics Applied Mathematical Scieces, Vol 6, 0, o 93, 467-4640 Cofidece Iterval Estimatio of the Shape Parameter of Pareto Distributio Usig Extreme Order Statistics Aissa Omar, Kamarulzama Ibrahim ad Ahmad Mahir

More information

The following example will help us understand The Sampling Distribution of the Mean. C1 C2 C3 C4 C5 50 miles 84 miles 38 miles 120 miles 48 miles

The following example will help us understand The Sampling Distribution of the Mean. C1 C2 C3 C4 C5 50 miles 84 miles 38 miles 120 miles 48 miles The followig eample will help us uderstad The Samplig Distributio of the Mea Review: The populatio is the etire collectio of all idividuals or objects of iterest The sample is the portio of the populatio

More information

Maximum Likelihood Estimators.

Maximum Likelihood Estimators. Lecture 2 Maximum Likelihood Estimators. Matlab example. As a motivatio, let us look at oe Matlab example. Let us geerate a radom sample of size 00 from beta distributio Beta(5, 2). We will lear the defiitio

More information

Plug-in martingales for testing exchangeability on-line

Plug-in martingales for testing exchangeability on-line Plug-i martigales for testig exchageability o-lie Valetia Fedorova, Alex Gammerma, Ilia Nouretdiov, ad Vladimir Vovk Computer Learig Research Cetre Royal Holloway, Uiversity of Lodo, UK {valetia,ilia,alex,vovk}@cs.rhul.ac.uk

More information

Lecture 13. Lecturer: Jonathan Kelner Scribe: Jonathan Pines (2009)

Lecture 13. Lecturer: Jonathan Kelner Scribe: Jonathan Pines (2009) 18.409 A Algorithmist s Toolkit October 27, 2009 Lecture 13 Lecturer: Joatha Keler Scribe: Joatha Pies (2009) 1 Outlie Last time, we proved the Bru-Mikowski iequality for boxes. Today we ll go over the

More information

LIMIT MODEL FOR THE RELIABILITY OF A REGULAR AND HOMOGENEOUS SERIES- PARALLEL SYSTEM

LIMIT MODEL FOR THE RELIABILITY OF A REGULAR AND HOMOGENEOUS SERIES- PARALLEL SYSTEM REVSTAT Statistical Joural Volume 7, Number 3, November 2009, 227 243 LIMIT MODEL FOR THE RELIABILITY OF A REGULAR AND HOMOGENEOUS SERIES- PARALLEL SYSTEM Authors: Paula Reis CEAUL ad EST, Politecic Istitut

More information

Notes on Hypothesis Testing

Notes on Hypothesis Testing Probability & Statistics Grishpa Notes o Hypothesis Testig A radom sample X = X 1,..., X is observed, with joit pmf/pdf f θ x 1,..., x. The values x = x 1,..., x of X lie i some sample space X. The parameter

More information

Approximating Area under a curve with rectangles. To find the area under a curve we approximate the area using rectangles and then use limits to find

Approximating Area under a curve with rectangles. To find the area under a curve we approximate the area using rectangles and then use limits to find 1.8 Approximatig Area uder a curve with rectagles 1.6 To fid the area uder a curve we approximate the area usig rectagles ad the use limits to fid 1.4 the area. Example 1 Suppose we wat to estimate 1.

More information

{{1}, {2, 4}, {3}} {{1, 3, 4}, {2}} {{1}, {2}, {3, 4}} 5.4 Stirling Numbers

{{1}, {2, 4}, {3}} {{1, 3, 4}, {2}} {{1}, {2}, {3, 4}} 5.4 Stirling Numbers . Stirlig Numbers Whe coutig various types of fuctios from., we quicly discovered that eumeratig the umber of oto fuctios was a difficult problem. For a domai of five elemets ad a rage of four elemets,

More information

Irreducible polynomials with consecutive zero coefficients

Irreducible polynomials with consecutive zero coefficients Irreducible polyomials with cosecutive zero coefficiets Theodoulos Garefalakis Departmet of Mathematics, Uiversity of Crete, 71409 Heraklio, Greece Abstract Let q be a prime power. We cosider the problem

More information

4.1 Sigma Notation and Riemann Sums

4.1 Sigma Notation and Riemann Sums 0 the itegral. Sigma Notatio ad Riema Sums Oe strategy for calculatig the area of a regio is to cut the regio ito simple shapes, calculate the area of each simple shape, ad the add these smaller areas

More information

Taking DCOP to the Real World: Efficient Complete Solutions for Distributed Multi-Event Scheduling

Taking DCOP to the Real World: Efficient Complete Solutions for Distributed Multi-Event Scheduling Taig DCOP to the Real World: Efficiet Complete Solutios for Distributed Multi-Evet Schedulig Rajiv T. Maheswara, Milid Tambe, Emma Bowrig, Joatha P. Pearce, ad Pradeep araatham Uiversity of Souther Califoria

More information

Math 113 HW #11 Solutions

Math 113 HW #11 Solutions Math 3 HW # Solutios 5. 4. (a) Estimate the area uder the graph of f(x) = x from x = to x = 4 usig four approximatig rectagles ad right edpoits. Sketch the graph ad the rectagles. Is your estimate a uderestimate

More information

University of California, Los Angeles Department of Statistics. Distributions related to the normal distribution

University of California, Los Angeles Department of Statistics. Distributions related to the normal distribution Uiversity of Califoria, Los Ageles Departmet of Statistics Statistics 100B Istructor: Nicolas Christou Three importat distributios: Distributios related to the ormal distributio Chi-square (χ ) distributio.

More information

A Note on Sums of Greatest (Least) Prime Factors

A Note on Sums of Greatest (Least) Prime Factors It. J. Cotemp. Math. Scieces, Vol. 8, 203, o. 9, 423-432 HIKARI Ltd, www.m-hikari.com A Note o Sums of Greatest (Least Prime Factors Rafael Jakimczuk Divisio Matemática, Uiversidad Nacioal de Luá Bueos

More information

Solving Divide-and-Conquer Recurrences

Solving Divide-and-Conquer Recurrences Solvig Divide-ad-Coquer Recurreces Victor Adamchik A divide-ad-coquer algorithm cosists of three steps: dividig a problem ito smaller subproblems solvig (recursively) each subproblem the combiig solutios

More information

Algebra Vocabulary List (Definitions for Middle School Teachers)

Algebra Vocabulary List (Definitions for Middle School Teachers) Algebra Vocabulary List (Defiitios for Middle School Teachers) A Absolute Value Fuctio The absolute value of a real umber x, x is xifx 0 x = xifx < 0 http://www.math.tamu.edu/~stecher/171/f02/absolutevaluefuctio.pdf

More information

Groups of diverse problem solvers can outperform groups of high-ability problem solvers

Groups of diverse problem solvers can outperform groups of high-ability problem solvers Groups of diverse problem solvers ca outperform groups of high-ability problem solvers Lu Hog ad Scott E. Page Michiga Busiess School ad Complex Systems, Uiversity of Michiga, A Arbor, MI 48109-1234; ad

More information

Exponential function: For a > 0, the exponential function with base a is defined by. f(x) = a x

Exponential function: For a > 0, the exponential function with base a is defined by. f(x) = a x MATH 11011 EXPONENTIAL FUNCTIONS KSU AND THEIR APPLICATIONS Defiitios: Expoetial fuctio: For a > 0, the expoetial fuctio with base a is defied by fx) = a x Horizotal asymptote: The lie y = c is a horizotal

More information

Chapter 14 Nonparametric Statistics

Chapter 14 Nonparametric Statistics Chapter 14 Noparametric Statistics A.K.A. distributio-free statistics! Does ot deped o the populatio fittig ay particular type of distributio (e.g, ormal). Sice these methods make fewer assumptios, they

More information

arxiv:1012.1336v2 [cs.cc] 8 Dec 2010

arxiv:1012.1336v2 [cs.cc] 8 Dec 2010 Uary Subset-Sum is i Logspace arxiv:1012.1336v2 [cs.cc] 8 Dec 2010 1 Itroductio Daiel M. Kae December 9, 2010 I this paper we cosider the Uary Subset-Sum problem which is defied as follows: Give itegers

More information

Engineering 323 Beautiful Homework Set 3 1 of 7 Kuszmar Problem 2.51

Engineering 323 Beautiful Homework Set 3 1 of 7 Kuszmar Problem 2.51 Egieerig 33 eautiful Homewor et 3 of 7 Kuszmar roblem.5.5 large departmet store sells sport shirts i three sizes small, medium, ad large, three patters plaid, prit, ad stripe, ad two sleeve legths log

More information

.04. This means $1000 is multiplied by 1.02 five times, once for each of the remaining sixmonth

.04. This means $1000 is multiplied by 1.02 five times, once for each of the remaining sixmonth Questio 1: What is a ordiary auity? Let s look at a ordiary auity that is certai ad simple. By this, we mea a auity over a fixed term whose paymet period matches the iterest coversio period. Additioally,

More information

Cooley-Tukey. Tukey FFT Algorithms. FFT Algorithms. Cooley

Cooley-Tukey. Tukey FFT Algorithms. FFT Algorithms. Cooley Cooley Cooley-Tuey Tuey FFT Algorithms FFT Algorithms Cosider a legth- sequece x[ with a -poit DFT X[ where Represet the idices ad as +, +, Cooley Cooley-Tuey Tuey FFT Algorithms FFT Algorithms Usig these

More information

Gregory Carey, 1998 Linear Transformations & Composites - 1. Linear Transformations and Linear Composites

Gregory Carey, 1998 Linear Transformations & Composites - 1. Linear Transformations and Linear Composites Gregory Carey, 1998 Liear Trasformatios & Composites - 1 Liear Trasformatios ad Liear Composites I Liear Trasformatios of Variables Meas ad Stadard Deviatios of Liear Trasformatios A liear trasformatio

More information

LECTURE 4. Last time: Lecture outline

LECTURE 4. Last time: Lecture outline LECTURE 4 Last time: Types of convergence Weak Law of Large Numbers Strong Law of Large Numbers Asymptotic Equipartition Property Lecture outline Stochastic processes Markov chains Entropy rate Random

More information

8.1 Arithmetic Sequences

8.1 Arithmetic Sequences MCR3U Uit 8: Sequeces & Series Page 1 of 1 8.1 Arithmetic Sequeces Defiitio: A sequece is a comma separated list of ordered terms that follow a patter. Examples: 1, 2, 3, 4, 5 : a sequece of the first

More information

Non-life insurance mathematics. Nils F. Haavardsson, University of Oslo and DNB Skadeforsikring

Non-life insurance mathematics. Nils F. Haavardsson, University of Oslo and DNB Skadeforsikring No-life isurace mathematics Nils F. Haavardsso, Uiversity of Oslo ad DNB Skadeforsikrig Mai issues so far Why does isurace work? How is risk premium defied ad why is it importat? How ca claim frequecy

More information

Case Study. Normal and t Distributions. Density Plot. Normal Distributions

Case Study. Normal and t Distributions. Density Plot. Normal Distributions Case Study Normal ad t Distributios Bret Halo ad Bret Larget Departmet of Statistics Uiversity of Wiscosi Madiso October 11 13, 2011 Case Study Body temperature varies withi idividuals over time (it ca

More information

CME 302: NUMERICAL LINEAR ALGEBRA FALL 2005/06 LECTURE 8

CME 302: NUMERICAL LINEAR ALGEBRA FALL 2005/06 LECTURE 8 CME 30: NUMERICAL LINEAR ALGEBRA FALL 005/06 LECTURE 8 GENE H GOLUB 1 Positive Defiite Matrices A matrix A is positive defiite if x Ax > 0 for all ozero x A positive defiite matrix has real ad positive

More information

Solutions to Selected Problems In: Pattern Classification by Duda, Hart, Stork

Solutions to Selected Problems In: Pattern Classification by Duda, Hart, Stork Solutios to Selected Problems I: Patter Classificatio by Duda, Hart, Stork Joh L. Weatherwax February 4, 008 Problem Solutios Chapter Bayesia Decisio Theory Problem radomized rules Part a: Let Rx be the

More information

Hypothesis testing. Null and alternative hypotheses

Hypothesis testing. Null and alternative hypotheses Hypothesis testig Aother importat use of samplig distributios is to test hypotheses about populatio parameters, e.g. mea, proportio, regressio coefficiets, etc. For example, it is possible to stipulate

More information

INFINITE SERIES KEITH CONRAD

INFINITE SERIES KEITH CONRAD INFINITE SERIES KEITH CONRAD. Itroductio The two basic cocepts of calculus, differetiatio ad itegratio, are defied i terms of limits (Newto quotiets ad Riema sums). I additio to these is a third fudametal

More information

Theorems About Power Series

Theorems About Power Series Physics 6A Witer 20 Theorems About Power Series Cosider a power series, f(x) = a x, () where the a are real coefficiets ad x is a real variable. There exists a real o-egative umber R, called the radius

More information

Chapter 5: Inner Product Spaces

Chapter 5: Inner Product Spaces Chapter 5: Ier Product Spaces Chapter 5: Ier Product Spaces SECION A Itroductio to Ier Product Spaces By the ed of this sectio you will be able to uderstad what is meat by a ier product space give examples

More information

Section 1.6: Proof by Mathematical Induction

Section 1.6: Proof by Mathematical Induction Sectio.6 Proof by Iductio Sectio.6: Proof by Mathematical Iductio Purpose of Sectio: To itroduce the Priciple of Mathematical Iductio, both weak ad the strog versios, ad show how certai types of theorems

More information

10-705/36-705 Intermediate Statistics

10-705/36-705 Intermediate Statistics 0-705/36-705 Itermediate Statistics Larry Wasserma http://www.stat.cmu.edu/~larry/=stat705/ Fall 0 Week Class I Class II Day III Class IV Syllabus August 9 Review Review, Iequalities Iequalities September

More information

CS103A Handout 23 Winter 2002 February 22, 2002 Solving Recurrence Relations

CS103A Handout 23 Winter 2002 February 22, 2002 Solving Recurrence Relations CS3A Hadout 3 Witer 00 February, 00 Solvig Recurrece Relatios Itroductio A wide variety of recurrece problems occur i models. Some of these recurrece relatios ca be solved usig iteratio or some other ad

More information

Estimating the Mean and Variance of a Normal Distribution

Estimating the Mean and Variance of a Normal Distribution Estimatig the Mea ad Variace of a Normal Distributio Learig Objectives After completig this module, the studet will be able to eplai the value of repeatig eperimets eplai the role of the law of large umbers

More information

Stock Market Trading via Stochastic Network Optimization

Stock Market Trading via Stochastic Network Optimization PROC. IEEE CONFERENCE ON DECISION AND CONTROL (CDC), ATLANTA, GA, DEC. 2010 1 Stock Market Tradig via Stochastic Network Optimizatio Michael J. Neely Uiversity of Souther Califoria http://www-rcf.usc.edu/

More information

Confidence Intervals for One Mean

Confidence Intervals for One Mean Chapter 420 Cofidece Itervals for Oe Mea Itroductio This routie calculates the sample size ecessary to achieve a specified distace from the mea to the cofidece limit(s) at a stated cofidece level for a

More information