Entropy Rates of a Stochastic Process


 Allyson Wheeler
 1 years ago
 Views:
Transcription
1 Etropy Rates of a Stochastic Process Best Achievable Data Compressio Radu Trîmbiţaş October Etropy Rates of a Stochastic Process Etropy rates The AEP states that H(X) bits suffice o the average for i.i.d. RVs What for depedet RVs? For statioary processes H(X 1, X 2,..., X ) grows (asymptotically) liearly with at a rate H(X ) the etropy rate of the process A stochastic process {X i } i I is a idexed sequece of radom variables, X i : S X is a RV i I If I N, {X 1, X 2,... } is a discrete stochastic process, called also a discrete iformatio source. A discrete stochastic process is characterized by the joit probability mass fuctio P((X 1, X 2,..., X ) = (x 1, x 2,..., x )) = p(x 1, x 2,..., x ) where (x 1, x 2,..., x ) X. 1.1 Markov chais Markov chais Defiitio 1. A stochastic process is said to be statioary if the joit distributio of ay subset of the sequece of radom variables is ivariat with respect to shifts i the time idex P(X 1 = x 1,..., X = x ) = P(X 1+l = x 1,..., X +l = x ) (1), l ad x 1, x 2,..., x X. 1
2 Defiitio 2. A discrete stochastic process {X 1, X 2,... } is said to be a Markov chai or Markov process if for = 1, 2,... P (X +1 = x +1 X = x, X 1 = x 1,..., X 1 = x 1 ) = P(X +1 = x +1 X = x ), x 1, x 2,..., x, x +1 X. (2) The joit pmf ca be writte as p(x 1, x 2,..., x ) = p(x 1 )p(x 2 x 1 )... p(x x 1 ). (3) Defiitio 3. A Markov chai is said to be time ivariat (time homogeeous) if the coditioal probability p(x +1 x ) does ot deped o ; that is for = 1, 2,... P (X +1 = b X = a) = P(X 2 = b X 1 = a), a, b X. (4) This property is assumed uless otherwise stated. {X i } Markov chai, X is called the state at time A timeivariat Markov chai is characterized by its iitial state ad a probability trasitio matrix P = [P ij ], i, j = 1,..., m, where P ij = P(X +1 = j X = i). The Markov chai {X } is irreducible if it is possible to go from ay state to aother with a probability > 0 The Markov chai {X } is aperiodic if state a, the possible times to go from a to a have highest commo factor = 1. Markov chais are ofte described by a directed graph where the edges are labeled by the probability of goig from oe state to aother. p(x )  pmf of the radom variable at time p(x +1 ) = x p(x )P x x +1 (5) A distributio o the states such that the distributio at time + 1 is the same as the distributio at time is called a statioary distributio  so called because if the iitial state of a Markov chai is draw accordig to a statioary distributio, the Markov chai form a statioary process. If the fiitestate Markov chai is irreducible ad periodic, the statioary distributio is uique, ad from ay startig distributio, the distributio of X teds to a statioary distributio as. Example 4. Cosider a twostate Markov chai with a probability trasitio matrix [ ] 1 α α P = β 1 β (Figure 1) 2
3 Figure 1: Twostate Markov chai The statioary probability is the solutio of µp = µ or (I P T )µ T = 0. We add the coditio µ 1 + µ 2 = 0. The solutio is µ 1 = β α + β, µ 2 = α α + β. Click here for a Maple solutio Markovex1.html. The etropy of X is ( ) β H(X ) = H α + β, α. α + β 1.2 Etropy rate Etropy rate Defiitio 5. The etropy rate of a stochastic process {X i } is defied by whe the limit exists. Examples 1 H(X ) = lim H (X 1,..., X ) (6) 1. Typewriter  m equally likely output letters; he(she) ca produce m sequeces of legth, all of them equally likely. H (X 1,..., X ) = log m, ad the etropy rate is H(X ) = log m bits per symbol. 2. X 1, X 2,... i.i.d. RVs H (X H(X ) = lim 1,..., X ) H(X = lim 1 ) = H (X 1 ). 3. X 1, X 2,... idepedet, but ot idetically distributed RVs H (X 1,..., X ) = It is possible that 1 H(x i) does ot exists H(X i ) 3
4 Defiitio 6. H (X ) = lim H (X X 1, X 2,... X 1 ). (7) H(X ) is etropy per symbol of the RVs; H (X ) is the coditioal etropy of the last RV give the past. For statioary processes both limits exist ad are equal. Lemma 7. For a statioary stochastic process, H(X X 1,..., X 1 ) is oicreasig i ad has a limit H (X ). Proof. H(X +1 X 1, X 2,..., X ) H(X +1 X,..., X 2 ) coditioig = H(X X 1,..., X 1 ). statioarity (H(X X 1,..., X 1 )) is decreasig ad oegative, so it has a limit H (X ). Lemma 8 (Cesáro). If a a ad b = 1 a i the b a. Theorem 9. For a statioary stochastic process H(X ) (give by (6)) ad H (X ) (give by (7)) exist ad H(X ) = H (X ). (8) Proof. By the chai rule, But, H (X 1,..., X ) = 1 H (X H(X ) = lim 1,..., X ) 1 = lim H(X i X i 1,..., X 1 ) H(X i X i 1,..., X 1 ). = lim H(X X 1,..., X 1 ) (Lemma 8) = H (X ) (Lemma 7) 1.3 Etropy rate for Markov chai Etropy rate for Markov chai For a statioary Markov chai, the etropy rate is give by H (X ) = H (X ) = lim H (X X 1,..., X 1 ) = lim H (X X 1 ) = H(X 2 X 1 ), (9) where the coditioal etropy is calculated usig the give statioary distributio. 4
5 The statioary distributio µ is the solutio of the equatios Expressio of coditioal etropy: µ j = µ i P ij, j. i Theorem 10. {X i } statioary Markov chai with statioary distributio µ ad trasitio matrix P. Let X 1 µ. the the etropy rate is H(X ) = i Proof. H(X ) = H(X 2 X 1 ) = i µ i ( j P ij log P ij ). µ i P ij log P ij. (10) j Example 11 (Twostate Markov chai). The etropy rate of the twostate Markov chai i Figure 1 is H(X ) = H(X 2 X 1 ) = β α + β H(α) + α α + β H(β). Remark. If the Markov chai is irreducible ad aperiodic, it has a uique statioary distributio o the states, ad ay iitial distributio teds to the statioary distributio as. I this case, eve though the iitial distributio is ot the statioary distributio, the etropy rate, which is defied i terms of logterm behavior, is H(X ), as defied i (9) ad (10). 1.4 Fuctios of Markov chais Fuctios of Markov chais X 1, X 2,..., X,... statioary Markov chai, Y i = φ(x i ), H(Y) =? i may cases Y 1, Y 2,..., Y,... is ot a Markov chai, but it is statioary lower boud Lemma 12. Proof. For k = 1, 2,... H(Y Y 1,..., Y 2, X 1 ) H(Y). (11) H(Y Y 1,..., Y 2, X 1 ) (a) = H(Y Y 1,..., Y 2, Y 1, X 1 ) (b) = H(Y Y 1,..., Y 2, Y 1, X 1, X 0, X 1,..., X k ) (c) = H (Y Y 1,..., Y 2, Y 1, X 1, X 0, X 1,..., X k, Y 0,..., Y k ) (d) H (Y Y 1,..., Y 2, Y 1, Y 0,..., Y k ) (e) = H (Y +k+1 Y +k,..., Y 1 ), 5
6 (a) follows from the fact that Y 1 = φ(x 1 ), (b) from the Markovity, (c) from Y i = φ(x i ), (d) coditioig reduces etropy, (e) statioarity. Proof  cotiuatio. Sice iequality is true for all k, i the limit H(Y Y 1,..., Y 2, X 1 ) lim H (Y +k+1 Y +k,..., Y 1 ) k = H(Y). Lemma 13. H(Y Y 1,..., Y 2, X 1 ) H(Y Y 1,..., Y 2, Y 1, X 1 ) 0. (12) Proof. Expressio of iterval legth: H(Y Y 1,..., Y 2, X 1 ) H(Y Y 1,..., Y 2, Y 1, X 1 ) = I(X 1 ; Y Y 1,..., Y 1 ). By properties of mutual iformatio, I(X 1 ; Y 1,..., Y ) H(X 1 ), ad I(X 1 ; Y 1,..., Y ) icreases with. Thus, lim I(X 1 ; Y 1,..., Y ) exists ad lim I(X 1; Y 1,..., Y ) H(X 1 ). Proof  cotiuatio. By the chai rule H(X 1 ) lim I(X 1 ; Y 1,..., Y ) = lim = I(X 1 ; Y i Y i 1,..., Y 1 ) I(X 1 ; Y i Y i 1,..., Y 1 ) The geeral term of the series must ted to 0 lim I (X 1 ; Y Y 1,..., Y 1 ) = 0. The last two lemmas imply Theorem 14. X 1, X 2,..., X,... statioary Markov chai, Y i = φ(x i ) H(Y Y 1,..., Y 1, X 1 ) H(Y) H(Y Y 1,..., Y 1 ) (13) ad lim H(Y Y 1,..., Y 1, X 1 ) = H(Y) = lim H(Y Y 1,..., Y 1 ) (14) 6
7 Hide Markov models We could cosider Y i to be a stochastic fuctio of X i X 1, X 2,..., X,... statioary Markov chai, Y 1, Y 2,..., Y,... a ew process where Y i is draw accordig to p(y i x i ), coditioally idepedet of all the other X j, j = i p(x, y 1 ) = p(x 1 ) p(x i+1 x i ) p(y i x i ). Y 1, Y 2,..., Y,... is called a hidde Markov model (HMM) Applied to speech recogitio, hadwritig recogitio, ad so o. The same argumet as for fuctios of Markov chai works for HMMs. Refereces Refereces [1] Thomas M. Cover, Joy A. Thomas, Elemets of Iformatio Theory, 2d editio, Wiley, [2] David J.C. MacKay, Iformatio Theory, Iferece, ad Learig Algorithms, Cambridge Uiversity Press, [3] Robert M. Gray, Etropy ad Iformatio Theory, Spriger,
Properties of MLE: consistency, asymptotic normality. Fisher information.
Lecture 3 Properties of MLE: cosistecy, asymptotic ormality. Fisher iformatio. I this sectio we will try to uderstad why MLEs are good. Let us recall two facts from probability that we be used ofte throughout
More informationChapter 7 Methods of Finding Estimators
Chapter 7 for BST 695: Special Topics i Statistical Theory. Kui Zhag, 011 Chapter 7 Methods of Fidig Estimators Sectio 7.1 Itroductio Defiitio 7.1.1 A poit estimator is ay fuctio W( X) W( X1, X,, X ) of
More informationAn example of nonquenched convergence in the conditional central limit theorem for partial sums of a linear process
A example of oqueched covergece i the coditioal cetral limit theorem for partial sums of a liear process Dalibor Volý ad Michael Woodroofe Abstract A causal liear processes X,X 0,X is costructed for which
More informationDepartment of Computer Science, University of Otago
Departmet of Computer Sciece, Uiversity of Otago Techical Report OUCS200609 Permutatios Cotaiig May Patters Authors: M.H. Albert Departmet of Computer Sciece, Uiversity of Otago Micah Colema, Rya Fly
More informationA probabilistic proof of a binomial identity
A probabilistic proof of a biomial idetity Joatho Peterso Abstract We give a elemetary probabilistic proof of a biomial idetity. The proof is obtaied by computig the probability of a certai evet i two
More informationUC Berkeley Department of Electrical Engineering and Computer Science. EE 126: Probablity and Random Processes. Solutions 9 Spring 2006
Exam format UC Bereley Departmet of Electrical Egieerig ad Computer Sciece EE 6: Probablity ad Radom Processes Solutios 9 Sprig 006 The secod midterm will be held o Wedesday May 7; CHECK the fial exam
More informationSAMPLE QUESTIONS FOR FINAL EXAM. (1) (2) (3) (4) Find the following using the definition of the Riemann integral: (2x + 1)dx
SAMPLE QUESTIONS FOR FINAL EXAM REAL ANALYSIS I FALL 006 3 4 Fid the followig usig the defiitio of the Riema itegral: a 0 x + dx 3 Cosider the partitio P x 0 3, x 3 +, x 3 +,......, x 3 3 + 3 of the iterval
More informationIn nite Sequences. Dr. Philippe B. Laval Kennesaw State University. October 9, 2008
I ite Sequeces Dr. Philippe B. Laval Keesaw State Uiversity October 9, 2008 Abstract This had out is a itroductio to i ite sequeces. mai de itios ad presets some elemetary results. It gives the I ite Sequeces
More informationChapter 5 Discrete Probability Distributions
Slides Prepared by JOHN S. LOUCKS St. Edward s Uiversity Slide Chapter 5 Discrete Probability Distributios Radom Variables Discrete Probability Distributios Epected Value ad Variace Poisso Distributio
More informationModule 4: Mathematical Induction
Module 4: Mathematical Iductio Theme 1: Priciple of Mathematical Iductio Mathematical iductio is used to prove statemets about atural umbers. As studets may remember, we ca write such a statemet as a predicate
More informationThe second difference is the sequence of differences of the first difference sequence, 2
Differece Equatios I differetial equatios, you look for a fuctio that satisfies ad equatio ivolvig derivatives. I differece equatios, istead of a fuctio of a cotiuous variable (such as time), we look for
More informationLecture 4: Cheeger s Inequality
Spectral Graph Theory ad Applicatios WS 0/0 Lecture 4: Cheeger s Iequality Lecturer: Thomas Sauerwald & He Su Statemet of Cheeger s Iequality I this lecture we assume for simplicity that G is a dregular
More informationApproximating the Sum of a Convergent Series
Approximatig the Sum of a Coverget Series Larry Riddle Ages Scott College Decatur, GA 30030 lriddle@agesscott.edu The BC Calculus Course Descriptio metios how techology ca be used to explore covergece
More informationOverview of some probability distributions.
Lecture Overview of some probability distributios. I this lecture we will review several commo distributios that will be used ofte throughtout the class. Each distributio is usually described by its probability
More informationThe Common Information for N Dependent Random Variables
Syracuse Uiversity SURFACE Electrical Egieerig ad Computer Sciece Techical Reports College of Egieerig ad Computer Sciece 420 The Commo Iformatio for N Depedet Radom Variables Wei Liu Syracuse Uiversity
More informationSequences II. Chapter 3. 3.1 Convergent Sequences
Chapter 3 Sequeces II 3. Coverget Sequeces Plot a graph of the sequece a ) = 2, 3 2, 4 3, 5 + 4,...,,... To what limit do you thik this sequece teds? What ca you say about the sequece a )? For ǫ = 0.,
More informationAsymptotic Growth of Functions
CMPS Itroductio to Aalysis of Algorithms Fall 3 Asymptotic Growth of Fuctios We itroduce several types of asymptotic otatio which are used to compare the performace ad efficiecy of algorithms As we ll
More information1 The Gaussian channel
ECE 77 Lecture 0 The Gaussia chael Objective: I this lecture we will lear about commuicatio over a chael of practical iterest, i which the trasmitted sigal is subjected to additive white Gaussia oise.
More informationUniversal coding for classes of sources
Coexios module: m46228 Uiversal codig for classes of sources Dever Greee This work is produced by The Coexios Project ad licesed uder the Creative Commos Attributio Licese We have discussed several parametric
More informationJoint Probability Distributions and Random Samples
STAT5 Sprig 204 Lecture Notes Chapter 5 February, 204 Joit Probability Distributios ad Radom Samples 5. Joitly Distributed Radom Variables Chapter Overview Joitly distributed rv Joit mass fuctio, margial
More informationSequences and Series
CHAPTER 9 Sequeces ad Series 9.. Covergece: Defiitio ad Examples Sequeces The purpose of this chapter is to itroduce a particular way of geeratig algorithms for fidig the values of fuctios defied by their
More information1 Introduction to reducing variance in Monte Carlo simulations
Copyright c 007 by Karl Sigma 1 Itroductio to reducig variace i Mote Carlo simulatios 11 Review of cofidece itervals for estimatig a mea I statistics, we estimate a uow mea µ = E(X) of a distributio by
More informationDiscrete Random Variables and Probability Distributions. Random Variables. Chapter 3 3.1
UCLA STAT A Applied Probability & Statistics for Egieers Istructor: Ivo Diov, Asst. Prof. I Statistics ad Neurology Teachig Assistat: Neda Farziia, UCLA Statistics Uiversity of Califoria, Los Ageles, Sprig
More informationInfinite Sequences and Series
CHAPTER 4 Ifiite Sequeces ad Series 4.1. Sequeces A sequece is a ifiite ordered list of umbers, for example the sequece of odd positive itegers: 1, 3, 5, 7, 9, 11, 13, 15, 17, 19, 21, 23, 25, 27, 29...
More informationA RANDOM PERMUTATION MODEL ARISING IN CHEMISTRY
J. Appl. Prob. 45, 060 070 2008 Prited i Eglad Applied Probability Trust 2008 A RANDOM PERMUTATION MODEL ARISING IN CHEMISTRY MARK BROWN, The City College of New York EROL A. PEKÖZ, Bosto Uiversity SHELDON
More information. P. 4.3 Basic feasible solutions and vertices of polyhedra. x 1. x 2
4. Basic feasible solutios ad vertices of polyhedra Due to the fudametal theorem of Liear Programmig, to solve ay LP it suffices to cosider the vertices (fiitely may) of the polyhedro P of the feasible
More informationLecture 5: Span, linear independence, bases, and dimension
Lecture 5: Spa, liear idepedece, bases, ad dimesio Travis Schedler Thurs, Sep 23, 2010 (versio: 9/21 9:55 PM) 1 Motivatio Motivatio To uderstad what it meas that R has dimesio oe, R 2 dimesio 2, etc.;
More informationChapter 6: Variance, the law of large numbers and the MonteCarlo method
Chapter 6: Variace, the law of large umbers ad the MoteCarlo method Expected value, variace, ad Chebyshev iequality. If X is a radom variable recall that the expected value of X, E[X] is the average value
More information5 Boolean Decision Trees (February 11)
5 Boolea Decisio Trees (February 11) 5.1 Graph Coectivity Suppose we are give a udirected graph G, represeted as a boolea adjacecy matrix = (a ij ), where a ij = 1 if ad oly if vertices i ad j are coected
More information0.7 0.6 0.2 0 0 96 96.5 97 97.5 98 98.5 99 99.5 100 100.5 96.5 97 97.5 98 98.5 99 99.5 100 100.5
Sectio 13 KolmogorovSmirov test. Suppose that we have a i.i.d. sample X 1,..., X with some ukow distributio P ad we would like to test the hypothesis that P is equal to a particular distributio P 0, i.e.
More informationThe Euler Totient, the Möbius and the Divisor Functions
The Euler Totiet, the Möbius ad the Divisor Fuctios Rosica Dieva July 29, 2005 Mout Holyoke College South Hadley, MA 01075 1 Ackowledgemets This work was supported by the Mout Holyoke College fellowship
More informationEkkehart Schlicht: Economic Surplus and Derived Demand
Ekkehart Schlicht: Ecoomic Surplus ad Derived Demad Muich Discussio Paper No. 200617 Departmet of Ecoomics Uiversity of Muich Volkswirtschaftliche Fakultät LudwigMaximiliasUiversität Müche Olie at http://epub.ub.uimueche.de/940/
More informationWeek 3 Conditional probabilities, Bayes formula, WEEK 3 page 1 Expected value of a random variable
Week 3 Coditioal probabilities, Bayes formula, WEEK 3 page 1 Expected value of a radom variable We recall our discussio of 5 card poker hads. Example 13 : a) What is the probability of evet A that a 5
More informationA gentle introduction to Expectation Maximization
A getle itroductio to Expectatio Maximizatio Mark Johso Brow Uiversity November 2009 1 / 15 Outlie What is Expectatio Maximizatio? Mixture models ad clusterig EM for setece topic modelig 2 / 15 Why Expectatio
More information3. Covariance and Correlation
Virtual Laboratories > 3. Expected Value > 1 2 3 4 5 6 3. Covariace ad Correlatio Recall that by takig the expected value of various trasformatios of a radom variable, we ca measure may iterestig characteristics
More informationParameter estimation for nonlinear models: Numerical approaches to solving the inverse problem. Lecture 11 04/01/2008. Sven Zenker
Parameter estimatio for oliear models: Numerical approaches to solvig the iverse problem Lecture 11 04/01/2008 Sve Zeker Review: Trasformatio of radom variables Cosider probability distributio of a radom
More informationLecture 4: Cauchy sequences, BolzanoWeierstrass, and the Squeeze theorem
Lecture 4: Cauchy sequeces, BolzaoWeierstrass, ad the Squeeze theorem The purpose of this lecture is more modest tha the previous oes. It is to state certai coditios uder which we are guarateed that limits
More informationTHE ABRACADABRA PROBLEM
THE ABRACADABRA PROBLEM FRANCESCO CARAVENNA Abstract. We preset a detailed solutio of Exercise E0.6 i [Wil9]: i a radom sequece of letters, draw idepedetly ad uiformly from the Eglish alphabet, the expected
More informationA ConstantFactor Approximation Algorithm for the Link Building Problem
A CostatFactor Approximatio Algorithm for the Lik Buildig Problem Marti Olse 1, Aastasios Viglas 2, ad Ilia Zvedeiouk 2 1 Ceter for Iovatio ad Busiess Developmet, Istitute of Busiess ad Techology, Aarhus
More informationOur aim is to show that under reasonable assumptions a given 2πperiodic function f can be represented as convergent series
8 Fourier Series Our aim is to show that uder reasoable assumptios a give periodic fuctio f ca be represeted as coverget series f(x) = a + (a cos x + b si x). (8.) By defiitio, the covergece of the series
More informationSoving Recurrence Relations
Sovig Recurrece Relatios Part 1. Homogeeous liear 2d degree relatios with costat coefficiets. Cosider the recurrece relatio ( ) T () + at ( 1) + bt ( 2) = 0 This is called a homogeeous liear 2d degree
More informationIncremental calculation of weighted mean and variance
Icremetal calculatio of weighted mea ad variace Toy Fich faf@cam.ac.uk dot@dotat.at Uiversity of Cambridge Computig Service February 009 Abstract I these otes I eplai how to derive formulae for umerically
More informationx(x 1)(x 2)... (x k + 1) = [x] k n+m 1
1 Coutig mappigs For every real x ad positive iteger k, let [x] k deote the fallig factorial ad x(x 1)(x 2)... (x k + 1) ( ) x = [x] k k k!, ( ) k = 1. 0 I the sequel, X = {x 1,..., x m }, Y = {y 1,...,
More informationTHE REGRESSION MODEL IN MATRIX FORM. For simple linear regression, meaning one predictor, the model is. for i = 1, 2, 3,, n
We will cosider the liear regressio model i matrix form. For simple liear regressio, meaig oe predictor, the model is i = + x i + ε i for i =,,,, This model icludes the assumptio that the ε i s are a sample
More informationChapter 7  Sampling Distributions. 1 Introduction. What is statistics? It consist of three major areas:
Chapter 7  Samplig Distributios 1 Itroductio What is statistics? It cosist of three major areas: Data Collectio: samplig plas ad experimetal desigs Descriptive Statistics: umerical ad graphical summaries
More informationConvexity, Inequalities, and Norms
Covexity, Iequalities, ad Norms Covex Fuctios You are probably familiar with the otio of cocavity of fuctios. Give a twicedifferetiable fuctio ϕ: R R, We say that ϕ is covex (or cocave up) if ϕ (x) 0 for
More informationOptimal Lossless Data Compression: NonAsymptotics and Asymptotics Ioannis Kontoyiannis, Fellow, IEEE, and Sergio Verdú, Fellow, IEEE
IEEE TRANSACTIONS ON INFORMATION THEORY, VOL. 60, NO. 2, FEBRUARY 204 777 Optimal Lossless Data Compressio: NoAsymptotics ad Asymptotics Ioais Kotoyiais, Fellow, IEEE, ad Sergio Verdú, Fellow, IEEE Abstract
More informationDistributions of Order Statistics
Chapter 2 Distributios of Order Statistics We give some importat formulae for distributios of order statistics. For example, where F k: (x)=p{x k, x} = I F(x) (k, k + 1), I x (a,b)= 1 x t a 1 (1 t) b 1
More informationPermutations, the Parity Theorem, and Determinants
1 Permutatios, the Parity Theorem, ad Determiats Joh A. Guber Departmet of Electrical ad Computer Egieerig Uiversity of Wiscosi Madiso Cotets 1 What is a Permutatio 1 2 Cycles 2 2.1 Traspositios 4 3 Orbits
More informationLinear Algebra II. 4 Determinants. Notes 4 1st November Definition of determinant
MTH6140 Liear Algebra II Notes 4 1st November 2010 4 Determiats The determiat is a fuctio defied o square matrices; its value is a scalar. It has some very importat properties: perhaps most importat is
More information1. MATHEMATICAL INDUCTION
1. MATHEMATICAL INDUCTION EXAMPLE 1: Prove that for ay iteger 1. Proof: 1 + 2 + 3 +... + ( + 1 2 (1.1 STEP 1: For 1 (1.1 is true, sice 1 1(1 + 1. 2 STEP 2: Suppose (1.1 is true for some k 1, that is 1
More informationInstallment Joint Life Insurance Actuarial Models with the Stochastic Interest Rate
Iteratioal Coferece o Maagemet Sciece ad Maagemet Iovatio (MSMI 4) Istallmet Joit Life Isurace ctuarial Models with the Stochastic Iterest Rate NiaNia JI a,*, Yue LI, DogHui WNG College of Sciece, Harbi
More informationSequences and Series Using the TI89 Calculator
RIT Calculator Site Sequeces ad Series Usig the TI89 Calculator Norecursively Defied Sequeces A orecursively defied sequece is oe i which the formula for the terms of the sequece is give explicitly. For
More informationChair for Network Architectures and Services Institute of Informatics TU München Prof. Carle. Network Security. Chapter 2 Basics
Chair for Network Architectures ad Services Istitute of Iformatics TU Müche Prof. Carle Network Security Chapter 2 Basics 2.4 Radom Number Geeratio for Cryptographic Protocols Motivatio It is crucial to
More informationMARTINGALES AND A BASIC APPLICATION
MARTINGALES AND A BASIC APPLICATION TURNER SMITH Abstract. This paper will develop the measuretheoretic approach to probability i order to preset the defiitio of martigales. From there we will apply this
More informationSECTION 1.5 : SUMMATION NOTATION + WORK WITH SEQUENCES
SECTION 1.5 : SUMMATION NOTATION + WORK WITH SEQUENCES Read Sectio 1.5 (pages 5 9) Overview I Sectio 1.5 we lear to work with summatio otatio ad formulas. We will also itroduce a brief overview of sequeces,
More informationDiscrete Mathematics and Probability Theory Spring 2014 Anant Sahai Note 13
EECS 70 Discrete Mathematics ad Probability Theory Sprig 2014 Aat Sahai Note 13 Itroductio At this poit, we have see eough examples that it is worth just takig stock of our model of probability ad may
More informationTHE LEAST COMMON MULTIPLE OF A QUADRATIC SEQUENCE
THE LEAST COMMON MULTIPLE OF A QUADRATIC SEQUENCE JAVIER CILLERUELO Abstract. We obtai, for ay irreducible quadratic olyomial f(x = ax 2 + bx + c, the asymtotic estimate log l.c.m. {f(1,..., f(} log. Whe
More informationNATIONAL SENIOR CERTIFICATE GRADE 12
NATIONAL SENIOR CERTIFICATE GRADE MATHEMATICS P EXEMPLAR 04 MARKS: 50 TIME: 3 hours This questio paper cosists of 8 pages ad iformatio sheet. Please tur over Mathematics/P DBE/04 NSC Grade Eemplar INSTRUCTIONS
More informationConfidence Interval Estimation of the Shape. Parameter of Pareto Distribution. Using Extreme Order Statistics
Applied Mathematical Scieces, Vol 6, 0, o 93, 4674640 Cofidece Iterval Estimatio of the Shape Parameter of Pareto Distributio Usig Extreme Order Statistics Aissa Omar, Kamarulzama Ibrahim ad Ahmad Mahir
More informationThe following example will help us understand The Sampling Distribution of the Mean. C1 C2 C3 C4 C5 50 miles 84 miles 38 miles 120 miles 48 miles
The followig eample will help us uderstad The Samplig Distributio of the Mea Review: The populatio is the etire collectio of all idividuals or objects of iterest The sample is the portio of the populatio
More informationMaximum Likelihood Estimators.
Lecture 2 Maximum Likelihood Estimators. Matlab example. As a motivatio, let us look at oe Matlab example. Let us geerate a radom sample of size 00 from beta distributio Beta(5, 2). We will lear the defiitio
More informationPlugin martingales for testing exchangeability online
Plugi martigales for testig exchageability olie Valetia Fedorova, Alex Gammerma, Ilia Nouretdiov, ad Vladimir Vovk Computer Learig Research Cetre Royal Holloway, Uiversity of Lodo, UK {valetia,ilia,alex,vovk}@cs.rhul.ac.uk
More informationLecture 13. Lecturer: Jonathan Kelner Scribe: Jonathan Pines (2009)
18.409 A Algorithmist s Toolkit October 27, 2009 Lecture 13 Lecturer: Joatha Keler Scribe: Joatha Pies (2009) 1 Outlie Last time, we proved the BruMikowski iequality for boxes. Today we ll go over the
More informationLIMIT MODEL FOR THE RELIABILITY OF A REGULAR AND HOMOGENEOUS SERIES PARALLEL SYSTEM
REVSTAT Statistical Joural Volume 7, Number 3, November 2009, 227 243 LIMIT MODEL FOR THE RELIABILITY OF A REGULAR AND HOMOGENEOUS SERIES PARALLEL SYSTEM Authors: Paula Reis CEAUL ad EST, Politecic Istitut
More informationNotes on Hypothesis Testing
Probability & Statistics Grishpa Notes o Hypothesis Testig A radom sample X = X 1,..., X is observed, with joit pmf/pdf f θ x 1,..., x. The values x = x 1,..., x of X lie i some sample space X. The parameter
More informationApproximating Area under a curve with rectangles. To find the area under a curve we approximate the area using rectangles and then use limits to find
1.8 Approximatig Area uder a curve with rectagles 1.6 To fid the area uder a curve we approximate the area usig rectagles ad the use limits to fid 1.4 the area. Example 1 Suppose we wat to estimate 1.
More information{{1}, {2, 4}, {3}} {{1, 3, 4}, {2}} {{1}, {2}, {3, 4}} 5.4 Stirling Numbers
. Stirlig Numbers Whe coutig various types of fuctios from., we quicly discovered that eumeratig the umber of oto fuctios was a difficult problem. For a domai of five elemets ad a rage of four elemets,
More informationIrreducible polynomials with consecutive zero coefficients
Irreducible polyomials with cosecutive zero coefficiets Theodoulos Garefalakis Departmet of Mathematics, Uiversity of Crete, 71409 Heraklio, Greece Abstract Let q be a prime power. We cosider the problem
More information4.1 Sigma Notation and Riemann Sums
0 the itegral. Sigma Notatio ad Riema Sums Oe strategy for calculatig the area of a regio is to cut the regio ito simple shapes, calculate the area of each simple shape, ad the add these smaller areas
More informationTaking DCOP to the Real World: Efficient Complete Solutions for Distributed MultiEvent Scheduling
Taig DCOP to the Real World: Efficiet Complete Solutios for Distributed MultiEvet Schedulig Rajiv T. Maheswara, Milid Tambe, Emma Bowrig, Joatha P. Pearce, ad Pradeep araatham Uiversity of Souther Califoria
More informationMath 113 HW #11 Solutions
Math 3 HW # Solutios 5. 4. (a) Estimate the area uder the graph of f(x) = x from x = to x = 4 usig four approximatig rectagles ad right edpoits. Sketch the graph ad the rectagles. Is your estimate a uderestimate
More informationUniversity of California, Los Angeles Department of Statistics. Distributions related to the normal distribution
Uiversity of Califoria, Los Ageles Departmet of Statistics Statistics 100B Istructor: Nicolas Christou Three importat distributios: Distributios related to the ormal distributio Chisquare (χ ) distributio.
More informationA Note on Sums of Greatest (Least) Prime Factors
It. J. Cotemp. Math. Scieces, Vol. 8, 203, o. 9, 423432 HIKARI Ltd, www.mhikari.com A Note o Sums of Greatest (Least Prime Factors Rafael Jakimczuk Divisio Matemática, Uiversidad Nacioal de Luá Bueos
More informationSolving DivideandConquer Recurrences
Solvig DivideadCoquer Recurreces Victor Adamchik A divideadcoquer algorithm cosists of three steps: dividig a problem ito smaller subproblems solvig (recursively) each subproblem the combiig solutios
More informationAlgebra Vocabulary List (Definitions for Middle School Teachers)
Algebra Vocabulary List (Defiitios for Middle School Teachers) A Absolute Value Fuctio The absolute value of a real umber x, x is xifx 0 x = xifx < 0 http://www.math.tamu.edu/~stecher/171/f02/absolutevaluefuctio.pdf
More informationGroups of diverse problem solvers can outperform groups of highability problem solvers
Groups of diverse problem solvers ca outperform groups of highability problem solvers Lu Hog ad Scott E. Page Michiga Busiess School ad Complex Systems, Uiversity of Michiga, A Arbor, MI 481091234; ad
More informationExponential function: For a > 0, the exponential function with base a is defined by. f(x) = a x
MATH 11011 EXPONENTIAL FUNCTIONS KSU AND THEIR APPLICATIONS Defiitios: Expoetial fuctio: For a > 0, the expoetial fuctio with base a is defied by fx) = a x Horizotal asymptote: The lie y = c is a horizotal
More informationChapter 14 Nonparametric Statistics
Chapter 14 Noparametric Statistics A.K.A. distributiofree statistics! Does ot deped o the populatio fittig ay particular type of distributio (e.g, ormal). Sice these methods make fewer assumptios, they
More informationarxiv:1012.1336v2 [cs.cc] 8 Dec 2010
Uary SubsetSum is i Logspace arxiv:1012.1336v2 [cs.cc] 8 Dec 2010 1 Itroductio Daiel M. Kae December 9, 2010 I this paper we cosider the Uary SubsetSum problem which is defied as follows: Give itegers
More informationEngineering 323 Beautiful Homework Set 3 1 of 7 Kuszmar Problem 2.51
Egieerig 33 eautiful Homewor et 3 of 7 Kuszmar roblem.5.5 large departmet store sells sport shirts i three sizes small, medium, ad large, three patters plaid, prit, ad stripe, ad two sleeve legths log
More information.04. This means $1000 is multiplied by 1.02 five times, once for each of the remaining sixmonth
Questio 1: What is a ordiary auity? Let s look at a ordiary auity that is certai ad simple. By this, we mea a auity over a fixed term whose paymet period matches the iterest coversio period. Additioally,
More informationCooleyTukey. Tukey FFT Algorithms. FFT Algorithms. Cooley
Cooley CooleyTuey Tuey FFT Algorithms FFT Algorithms Cosider a legth sequece x[ with a poit DFT X[ where Represet the idices ad as +, +, Cooley CooleyTuey Tuey FFT Algorithms FFT Algorithms Usig these
More informationGregory Carey, 1998 Linear Transformations & Composites  1. Linear Transformations and Linear Composites
Gregory Carey, 1998 Liear Trasformatios & Composites  1 Liear Trasformatios ad Liear Composites I Liear Trasformatios of Variables Meas ad Stadard Deviatios of Liear Trasformatios A liear trasformatio
More informationLECTURE 4. Last time: Lecture outline
LECTURE 4 Last time: Types of convergence Weak Law of Large Numbers Strong Law of Large Numbers Asymptotic Equipartition Property Lecture outline Stochastic processes Markov chains Entropy rate Random
More information8.1 Arithmetic Sequences
MCR3U Uit 8: Sequeces & Series Page 1 of 1 8.1 Arithmetic Sequeces Defiitio: A sequece is a comma separated list of ordered terms that follow a patter. Examples: 1, 2, 3, 4, 5 : a sequece of the first
More informationNonlife insurance mathematics. Nils F. Haavardsson, University of Oslo and DNB Skadeforsikring
Nolife isurace mathematics Nils F. Haavardsso, Uiversity of Oslo ad DNB Skadeforsikrig Mai issues so far Why does isurace work? How is risk premium defied ad why is it importat? How ca claim frequecy
More informationCase Study. Normal and t Distributions. Density Plot. Normal Distributions
Case Study Normal ad t Distributios Bret Halo ad Bret Larget Departmet of Statistics Uiversity of Wiscosi Madiso October 11 13, 2011 Case Study Body temperature varies withi idividuals over time (it ca
More informationCME 302: NUMERICAL LINEAR ALGEBRA FALL 2005/06 LECTURE 8
CME 30: NUMERICAL LINEAR ALGEBRA FALL 005/06 LECTURE 8 GENE H GOLUB 1 Positive Defiite Matrices A matrix A is positive defiite if x Ax > 0 for all ozero x A positive defiite matrix has real ad positive
More informationSolutions to Selected Problems In: Pattern Classification by Duda, Hart, Stork
Solutios to Selected Problems I: Patter Classificatio by Duda, Hart, Stork Joh L. Weatherwax February 4, 008 Problem Solutios Chapter Bayesia Decisio Theory Problem radomized rules Part a: Let Rx be the
More informationHypothesis testing. Null and alternative hypotheses
Hypothesis testig Aother importat use of samplig distributios is to test hypotheses about populatio parameters, e.g. mea, proportio, regressio coefficiets, etc. For example, it is possible to stipulate
More informationINFINITE SERIES KEITH CONRAD
INFINITE SERIES KEITH CONRAD. Itroductio The two basic cocepts of calculus, differetiatio ad itegratio, are defied i terms of limits (Newto quotiets ad Riema sums). I additio to these is a third fudametal
More informationTheorems About Power Series
Physics 6A Witer 20 Theorems About Power Series Cosider a power series, f(x) = a x, () where the a are real coefficiets ad x is a real variable. There exists a real oegative umber R, called the radius
More informationChapter 5: Inner Product Spaces
Chapter 5: Ier Product Spaces Chapter 5: Ier Product Spaces SECION A Itroductio to Ier Product Spaces By the ed of this sectio you will be able to uderstad what is meat by a ier product space give examples
More informationSection 1.6: Proof by Mathematical Induction
Sectio.6 Proof by Iductio Sectio.6: Proof by Mathematical Iductio Purpose of Sectio: To itroduce the Priciple of Mathematical Iductio, both weak ad the strog versios, ad show how certai types of theorems
More information10705/36705 Intermediate Statistics
0705/36705 Itermediate Statistics Larry Wasserma http://www.stat.cmu.edu/~larry/=stat705/ Fall 0 Week Class I Class II Day III Class IV Syllabus August 9 Review Review, Iequalities Iequalities September
More informationCS103A Handout 23 Winter 2002 February 22, 2002 Solving Recurrence Relations
CS3A Hadout 3 Witer 00 February, 00 Solvig Recurrece Relatios Itroductio A wide variety of recurrece problems occur i models. Some of these recurrece relatios ca be solved usig iteratio or some other ad
More informationEstimating the Mean and Variance of a Normal Distribution
Estimatig the Mea ad Variace of a Normal Distributio Learig Objectives After completig this module, the studet will be able to eplai the value of repeatig eperimets eplai the role of the law of large umbers
More informationStock Market Trading via Stochastic Network Optimization
PROC. IEEE CONFERENCE ON DECISION AND CONTROL (CDC), ATLANTA, GA, DEC. 2010 1 Stock Market Tradig via Stochastic Network Optimizatio Michael J. Neely Uiversity of Souther Califoria http://wwwrcf.usc.edu/
More informationConfidence Intervals for One Mean
Chapter 420 Cofidece Itervals for Oe Mea Itroductio This routie calculates the sample size ecessary to achieve a specified distace from the mea to the cofidece limit(s) at a stated cofidece level for a
More information