How To Calculate A Radom Umber From A Probability Fuctio

Transcription

1 Iteral Report SUF PFY/96 Stockholm, December 996 st revisio, 3 October 998 last modificatio September 7 Had-book o STATISTICAL DISTRIBUTIONS for experimetalists by Christia Walck Particle Physics Group Fysikum Uiversity of Stockholm ( walck@physto.se)

2

3 Cotets Itroductio. Radom Number Geeratio Probability Desity Fuctios 3. Itroductio Momets Errors of Momets Characteristic Fuctio Probability Geeratig Fuctio Cumulats Radom Number Geeratio Cumulative Techique Accept-Reject techique Compositio Techiques Multivariate Distributios Multivariate Momets Errors of Bivariate Momets Joit Characteristic Fuctio Radom Number Geeratio Beroulli Distributio 3. Itroductio Relatio to Other Distributios Beta distributio 3 4. Itroductio Derivatio of the Beta Distributio Characteristic Fuctio Momets Probability Cotet Radom Number Geeratio Biomial Distributio 6 5. Itroductio Momets Probability Geeratig Fuctio Cumulative Fuctio Radom Number Geeratio Estimatio of Parameters Probability Cotet Biormal Distributio 6. Itroductio Coditioal Probability Desity Characteristic Fuctio Momets Box-Muller Trasformatio Probability Cotet i

4 6.7 Radom Number Geeratio Cauchy Distributio 6 7. Itroductio Momets Normalizatio Characteristic Fuctio Locatio ad Scale Parameters Breit-Wiger Distributio Compariso to Other Distributios Trucatio Sum ad Average of Cauchy Variables Estimatio of the Media Estimatio of the HWHM Radom Number Geeratio Physical Picture Ratio Betwee Two Stadard Normal Variables Chi-square Distributio Itroductio Momets Characteristic Fuctio Cumulative Fuctio Origi of the Chi-square Distributio Approximatios Radom Number Geeratio Cofidece Itervals for the Variace Hypothesis Testig Probability Cotet Eve Number of Degrees of Freedom Odd Number of Degrees of Freedom Fial Algorithm Chi Distributio Compoud Poisso Distributio Itroductio Brachig Process Momets Probability Geeratig Fuctio Radom Number Geeratio Double-Expoetial Distributio 47. Itroductio Momets Characteristic Fuctio Cumulative Fuctio Radom Number Geeratio ii

5 Doubly No-Cetral F -Distributio 49. Itroductio Momets Cumulative Distributio Radom Number Geeratio Doubly No-Cetral t-distributio 5. Itroductio Momets Cumulative Distributio Radom Number Geeratio Error Fuctio Itroductio Probability Desity Fuctio Expoetial Distributio Itroductio Cumulative Fuctio Momets Characteristic Fuctio Radom Number Geeratio Method by vo Neuma Method by Marsaglia Method by Ahres Extreme Value Distributio Itroductio Cumulative Distributio Characteristic Fuctio Momets Radom Number Geeratio F-distributio 6 6. Itroductio Relatios to Other Distributios /F Characteristic Fuctio Momets F-ratio Variace Ratio Aalysis of Variace Calculatio of Probability Cotet The Icomplete Beta fuctio Fial Formulæ Radom Number Geeratio iii

6 7 Gamma Distributio Itroductio Derivatio of the Gamma Distributio Momets Characteristic Fuctio Probability Cotet Radom Number Geeratio Erlagia distributio Geeral case Asymptotic Approximatio Geeralized Gamma Distributio Itroductio Cumulative Fuctio Momets Relatio to Other Distributios Geometric Distributio Itroductio Momets Probability Geeratig Fuctio Radom Number Geeratio Hyperexpoetial Distributio 77. Itroductio Momets Characteristic Fuctio Radom Number Geeratio Hypergeometric Distributio 79. Itroductio Probability Geeratig Fuctio Momets Radom Number Geeratio Logarithmic Distributio 8. Itroductio Momets Probability Geeratig Fuctio Radom Number Geeratio Logistic Distributio Itroductio Cumulative Distributio Characteristic Fuctio Momets Radom umbers iv

7 4 Log-ormal Distributio Itroductio Momets Cumulative Distributio Radom Number Geeratio Maxwell Distributio Itroductio Momets Cumulative Distributio Kietic Theory Radom Number Geeratio Moyal Distributio 9 6. Itroductio Normalizatio Characteristic Fuctio Momets Cumulative Distributio Radom Number Geeratio Multiomial Distributio Itroductio Histogram Momets Probability Geeratig Fuctio Radom Number Geeratio Sigificace Levels Equal Group Probabilities Multiormal Distributio Itroductio Coditioal Probability Desity Probability Cotet Radom Number Geeratio Negative Biomial Distributio 9. Itroductio Momets Probability Geeratig Fuctio Relatios to Other Distributios Poisso Distributio Gamma Distributio Logarithmic Distributio Brachig Process Poisso ad Gamma Distributios Radom Number Geeratio v

8 3 No-cetral Beta-distributio 8 3. Itroductio Derivatio of distributio Momets Cumulative distributio Radom Number Geeratio No-cetral Chi-square Distributio 3. Itroductio Characteristic Fuctio Momets Cumulative Distributio Approximatios Radom Number Geeratio No-cetral F -Distributio 3 3. Itroductio Momets Cumulative Distributio Approximatios Radom Number Geeratio No-cetral t-distributio Itroductio Derivatio of distributio Momets Cumulative Distributio Approximatio Radom Number Geeratio Normal Distributio Itroductio Momets Cumulative Fuctio Characteristic Fuctio Additio Theorem Idepedece of x ad s Probability Cotet Radom Number Geeratio Cetral Limit Theory Approach Exact Trasformatio Polar Method Trapezoidal Method Ceter-tail method Compositio-rejectio Methods Method by Marsaglia Histogram Techique Ratio of Uiform Deviates Compariso of radom umber geerators vi

9 34.9 Tests o Parameters of a Normal Distributio Pareto Distributio Itroductio Cumulative Distributio Momets Radom Numbers Poisso Distributio Itroductio Momets Probability Geeratig Fuctio Cumulative Distributio Additio Theorem Derivatio of the Poisso Distributio Histogram Radom Number Geeratio Rayleigh Distributio Itroductio Momets Cumulative Distributio Two-dimesioal Kietic Theory Radom Number Geeratio Studet s t-distributio Itroductio History Momets Cumulative Fuctio Relatios to Other Distributios t-ratio Oe Normal Sample Two Normal Samples Paired Data Cofidece Levels Testig Hypotheses Calculatio of Probability Cotet Eve umber of degrees of freedom Odd umber of degrees of freedom Fial algorithm Radom Number Geeratio Triagular Distributio Itroductio Momets Radom Number Geeratio vii

10 4 Uiform Distributio 5 4. Itroductio Momets Radom Number Geeratio Weibull Distributio 5 4. Itroductio Cumulative Distributio Momets Radom Number Geeratio Appedix A: The Gamma ad Beta Fuctios Itroductio The Gamma Fuctio Numerical Calculatio Formulæ Digamma Fuctio Polygamma Fuctio The Icomplete Gamma Fuctio Numerical Calculatio Formulæ Special Cases The Beta Fuctio The Icomplete Beta Fuctio Numerical Calculatio Approximatio Relatios to Probability Desity Fuctios The Beta Distributio The Biomial Distributio The Chi-squared Distributio The F -distributio The Gamma Distributio The Negative Biomial Distributio The Normal Distributio The Poisso Distributio Studet s t-distributio Summary Appedix B: Hypergeometric Fuctios Itroductio Hypergeometric Fuctio Cofluet Hypergeometric Fuctio Mathematical Costats Errata et Addeda viii

11 Refereces Idex List of Tables Percetage poits of the chi-square distributio Extreme cofidece levels for the chi-square distributio Extreme cofidece levels for the chi-square distributio (as χ /d.f. values) Exact ad approximate values for the Beroulli umbers Percetage poits of the F -distributio Probability cotet from z to z of Gauss distributio i % Stadard ormal distributio z-values for a specific probability cotet Percetage poits of the t-distributio Expressios for the Beta fuctio B(m, ) for iteger ad half-iteger argumets. 79 ix

12 x

13 Itroductio I experimetal work e.g. i physics oe ofte ecouters problems where a stadard statistical probability desity fuctio is applicable. It is ofte of great help to be able to hadle these i differet ways such as calculatig probability cotets or geeratig radom umbers. For these purposes there are excellet text-books i statistics e.g. the classical work of Maurice G. Kedall ad Ala Stuart [,] or more moder text-books as [3] ad others. Some books are particularly aimed at experimetal physics or eve specifically at particle physics [4,5,6,7,8]. Cocerig umerical methods a valuable refereces worth metioig is [9] which has bee surpassed by a ew editio []. Also had-books, especially [], has bee of great help throughout. However, whe it comes to actual applicatios it ofte turs out to be hard to fid detailed explaatios i the literature ready for implemetatio. This work has bee collected over may years i parallel with actual experimetal work. I this way some material may be historical ad sometimes be aïve ad have somewhat clumsy solutios ot always made i the mathematically most striget may. We apologize for this but still hope that it will be of iterest ad help for people who is strugglig to fid methods to solve their statistical problems i makig real applicatios ad ot oly learig statistics as a course. Eve if oe has the skill ad may be able to fid solutios it seems worthwhile to have easy ad fast access to formulæ ready for applicatio. Similar books ad reports exist e.g. [,3] but we hope the preset work may compete i describig more distributios, beig more complete, ad icludig more explaatios o relatios give. The material could most probably have bee divided i a more logical way but we have chose to preset the distributios i alphabetic order. I this way it is more of a had-book tha a proper text-book. After the first release the report has bee modestly chaged. Mior chages to correct misprits is made wheever foud. I a few cases subsectios ad tables have bee added. These alteratios are described o page 8. I October 998 the first somewhat bigger revisio was made where i particular a lot of material o the o-cetral samplig distributios were added.. Radom Number Geeratio I moder computig Mote Carlo simulatios are of vital importace ad we give methods to achieve radom umbers from the distributios. A earlier report dealt etirely with these matters [4]. Not all text-books o statistics iclude iformatio o this subject which we fid extremely useful. Large simulatios are commo i particle physics as well as i other areas but ofte it is also useful to make small toy Mote Carlo programs to ivestigate ad study aalysis tools developed o ideal, but statistically soud, radom samples. A related ad importat field which we will oly metio briefly here, is how to get good basic geerators for achievig radom umbers uiformly distributed betwee zero ad oe. Those are the basis for all the methods described i order to get radom umbers

14 from specific distributios i this documet. For a review see e.g. [5]. From older methods ofte usig so called multiplicative cogruetial method or shiftgeerators G. Marsaglia et al [6] itroduced i 989 a ew uiversal geerator which became the ew stadard i may fields. We implemeted this i our experimets at CERN ad also made a package of routies for geeral use [7]. This method is still a very good choice but later alteratives, claimed to be eve better, have tured up. These are based o o the same type of lagged Fiboacci sequeces as is used i the uiversal geerator ad was origially proposed by the same authors [8]. A implemetatios of this method was proposed by F. James [5] ad this versio was further developed by M. Lüscher [9]. A similar package of routie as was prepared for the uiversal geerator has bee implemeted for this method [].

15 Probability Desity Fuctios. Itroductio Probability desity fuctios i oe, discrete or cotiuous, variable are deoted p(r) ad f(x), respectively. They are assumed to be properly ormalized such that p(r) = r ad f(x)dx = where the sum or the itegral are take over all relevat values for which the probability desity fuctio is defied. Statisticias ofte use the distributio fuctio or as physicists more ofte call it the cumulative fuctio which is defied as r P (r) = p(i) ad F (x) = x i= f(t)dt. Momets Algebraic momets of order r are defied as the expectatio value µ r = E(x r ) = k k r p(k) or x r f(x)dx Obviously µ = from the ormalizatio coditio ad µ is equal to the mea, sometimes called the expectatio value, of the distributio. Cetral momets of order r are defied as µ r = E((k E(k)) r ) or E((x E(x)) r ) of which the most commoly used is µ which is the variace of the distributio. Istead of usig the third ad fourth cetral momets oe ofte defies the coefficiets of skewess γ ad kurtosis γ by γ = µ 3 ad γ µ 3 = µ 4 µ 3 where the shift by 3 uits i γ assures that both measures are zero for a ormal distributio. Distributios with positive kurtosis are called leptokurtic, those with kurtosis aroud zero mesokurtic ad those with egative kurtosis platykurtic. Leptokurtic distributios are ormally more peaked tha the ormal distributio while platykurtic distributios are more flat topped. From greek kyrtosis = curvature from kyrt(ós) = curved, arched, roud, swellig, bulgig. Sometimes, especially i older literature, γ is called the coefficiet of excess. 3

16 .. Errors of Momets For a thorough presetatio of how to estimate errors o momets we refer to the classical books by M. G. Kedall ad A. Stuart [] (pp 8 45). Below oly a brief descriptio is give. For a sample with observatios x, x,..., x we defie the momet-statistics for the algebraic ad cetral momets m r ad m r as m r = x r ad m r = r= (x m ) r r= The otatio m r ad m r are thus used for the statistics (sample values) while we deote the true, populatio, values by µ r ad µ r. The mea value of the r:th ad the samplig covariace betwee the q:th ad r:th momet-statistic are give by. E(m r) = µ r Cov(m q, m r) = ( µ q+r µ qµ r) These formula are exact. Formulæ for momets about the mea are ot as simple sice the mea itself is subject to samplig fluctuatios. E(m r ) = µ r Cov(m q, m r ) = (µ q+r µ q µ r + rqµ µ r µ q rµ r µ q+ qµ r+ µ q ) to order / ad /, respectively. The covariace betwee a algebraic ad a cetral momet is give by to order /. Note especially that Cov(m r, m q) = (µ q+r µ q µ r rµ q+ µ r ) V (m r) = ( ) µ r µ r V (m r ) = ( ) µr µ r + r µ µ r rµ r µ r+ Cov(m, m r ) = (µ r+ rµ µ r ).3 Characteristic Fuctio For a distributio i a cotiuous variable x the Fourier trasform of the probability desity fuctio φ(t) = E(e ıxt ) = 4 e ıxt f(x)dx

17 is called the characteristic fuctio. It has the properties that φ() = ad φ(t) for all t. If the cumulative, distributio, fuctio F (x) is cotiuous everywhere ad df (x) = f(x)dx the we reverse the trasform such that f(x) = π φ(t)e ıxt dt The characteristic fuctio is related to the momets of the distributio by φ x (t) = E(e ıtx ) = = (ıt) E(x )! = = (ıt) µ! e.g. algebraic momets may be foud by µ r = ( r d φ(t) ı dt) r To fid cetral momets (about the mea µ) use t= φ x µ (t) = E ( e ıt(x µ)) = e ıtµ φ x (t) ad thus µ r = ( r d e ı dt) ıtµ φ(t) r t= A very useful property of the characteristic fuctio is that for idepedet variables x ad y φ x+y (t) = φ x (t) φ y (t) As a example regard the sum a i z i where the z i s are distributed accordig to ormal distributios with meas µ i ad variaces σ i. The the liear combiatio will also be distributed accordig to the ormal distributio with mea a i µ i ad variace a i σ i. To show that the characteristic fuctio i two variables factorizes is the best way to show idepedece betwee two variables. Remember that a vaishig correlatio coefficiet does ot imply idepedece while the reversed is true..4 Probability Geeratig Fuctio I the case of a distributio i a discrete variable r the characteristic fuctio is give by φ(t) = E(e ıtr ) = p(r)e ıtr I this case it is ofte coveiet to write z = e ıt ad defie the probability geeratig fuctio as G(z) = E(z r ) = p(r)z r 5

18 Derivatives of G(z) evaluated at z = are related to factorial momets of the distributio G () = d dz G(z) z= = E(r) G() = (ormalizatio) G () = d dz G(z) = E(r(r )) z= G 3 () = d3 dz G(z) 3 = E(r(r )(r )) z= G k () = dk dz G(z) k = E(r(r )(r ) (r k + )) z= Lower order algebraic momets are the give by µ = G () µ = G () + G () µ 3 = G 3 () + 3G () + G () µ 4 = G 4 () + 6G 3 () + 7G () + G () while expressio for cetral momets become more complicated. A useful property of the probability geeratig fuctio is for a brachig process i steps where G(z) = G (G (... G (G (z))...)) with G k (z) the probability geeratig fuctio for the distributio i the k:th step. As a example see sectio o page 5..5 Cumulats Although ot much used i physics the cumulats, κ r, are of statistical iterest. Oe reaso for this is that they have some useful properties such as beig ivariat for a shift i scale (except the first cumulat which is equal to the mea ad is shifted alog with the scale). Multiplyig the x-scale by a costat a has the same effect as for algebraic momets amely to multiply κ r by a r. As the algebraic momet µ is the coefficiet of (ıt) /! i the expasio of φ(t) the cumulat κ is the coefficiet of (ıt) /! i the expasio of the logarithm of φ(t) (sometimes called the cumulat geeratig fuctio) i.e. (ıt) l φ(t) = κ =! ad thus κ r = ( r d l φ(t) ı dt) r t= Relatios betwee cumulats ad cetral momets for some lower orders are as follows 6

19 κ = µ κ = µ µ = κ κ 3 = µ 3 µ 3 = κ 3 κ 4 = µ 4 3µ µ 4 = κ 4 + 3κ κ 5 = µ 5 µ 3 µ µ 5 = κ 5 + κ 3 κ κ 6 = µ 6 5µ 4 µ µ 3 + 3µ 3 µ 6 = κ 6 + 5κ 4 κ + κ 3 + 5κ 3 κ 7 = µ 7 µ 5 µ 35µ 4 µ 3 + µ 3 µ µ 7 = κ 7 + κ 5 κ + 35κ 4 κ 3 + 5κ 3 κ κ 8 = µ 8 8µ 6 µ 56µ 5 µ 3 35µ 4+ µ 8 = κ 8 + 8κ 6 κ + 56κ 5 κ κ 4+ +4µ 4 µ + 56µ 3µ 63µ 4 +κ 4 κ + 8κ 3κ + 5κ 4.6 Radom Number Geeratio Whe geeratig radom umbers from differet distributio it is assumed that a good geerator for uiform pseudoradom umbers betwee zero ad oe exist (ormally the ed-poits are excluded)..6. Cumulative Techique The most direct techique to obtai radom umbers from a cotiuous probability desity fuctio f(x) with a limited rage from x mi to x max is to solve for x i the equatio ξ = F (x) F (x mi) F (x max ) F (x mi ) where ξ is uiformly distributed betwee zero ad oe ad F (x) is the cumulative distributio (or as statisticias say the distributio fuctio). For a properly ormalized probability desity fuctio thus x = F (ξ) The techique is sometimes also of use i the discrete case if the cumulative sum may be expressed i aalytical form as e.g. for the geometric distributio. Also for geeral cases, discrete or cotiuous, e.g. from a arbitrary histogram the cumulative method is coveiet ad ofte faster tha more elaborate methods. I this case the task is to costruct a cumulative vector ad assig a radom umber accordig to the value of a uiform radom umber (iterpolatig withi bis i the cotiuous case)..6. Accept-Reject techique A useful techique is the acceptace-rejectio, or hit-miss, method where we choose f max to be greater tha or equal to f(x) i the etire iterval betwee x mi ad x max ad proceed as follows i Geerate a pair of uiform pseudoradom umbers ξ ad ξ. ii Determie x = x mi + ξ (x max x mi ). iii Determie y = f max ξ. iv If y f(x) > reject ad go to i else accept x as a pseudoradom umber from f(x). 7

20 The efficiecy of this method depeds o the average value of f(x)/f max over the iterval. If this value is close to oe the method is efficiet. O the other had, if this average is close to zero, the method is extremely iefficiet. If α is the fractio of the area f max (x max x mi ) covered by the fuctio the average umber of rejects i step iv is α ad uiform pseudoradom umbers are required o average. α The efficiecy of this method ca be icreased if we are able to choose a fuctio h(x), from which radom umbers are more easily obtaied, such that f(x) αh(x) = g(x) over the etire iterval uder cosideratio (where α is a costat). A radom sample from f(x) is obtaied by i Geerate i x a radom umber from h(x). ii Geerate a uiform radom umber ξ. iii If ξ f(x)/g(x) go back to i else accept x as a pseudoradom umber from f(x). Yet aother situatio is whe a fuctio g(x), from which fast geeratio may be obtaied, ca be iscribed i such a way that a big proportio (f) of the area uder the fuctio is covered (as a example see the trapezoidal method for the ormal distributio). The proceed as follows: i Geerate a uiform radom umber ξ. ii If ξ < f the geerate a radom umber from g(x). iii Else use the acceptace/rejectio techique for h(x) = f(x) g(x) (i subitervals if more efficiet)..6.3 Compositio Techiques If f(x) may be writte i the form f(x) = g z (x)dh(z) where we kow how to sample radom umbers from the p.d.f. g(x) ad the distributio fuctio H(z). A radom umber from f(x) is the obtaied by i Geerate two uiform radom umbers ξ ad ξ. ii Determie z = H (ξ ). iii Determie x = G z (ξ ) where G z is the distributio fuctio correspodig to the p.d.f. g z (x). For more detailed iformatio o the Compositio techique see [] or []. 8

21 A combiatio of the compositio ad the rejectio method has bee proposed by J. C. Butcher [3]. If f(x) ca be writte f(x) = α i f i (x)g i (x) i= where α i are positive costats, f i (x) p.d.f. s for which we kow how to sample a radom umber ad g i (x) are fuctios takig values betwee zero ad oe. The method is the as follows: i Geerate uiform radom umbers ξ ad ξ. ii Determie a iteger k from the discrete distributio p i = α i /(α + α α ) usig ξ. iii Geerate a radom umber x from f k (x). iv Determie g k (x) ad if ξ > g k (x) the go to i. v Accept x as a radom umber from f(x)..7 Multivariate Distributios Joit probability desity fuctios i several variables are deoted by f(x, x,..., x ) ad p(r, r,..., r ) for cotiuous ad discrete variables, respectively. It is assumed that they are properly ormalized i.e. itegrated (or summed) over all variables the result is uity..7. Multivariate Momets The geeralizatio of algebraic ad cetral momets to multivariate distributios is straightforward. As a example we take a bivariate distributio f(x, y) i two cotiuous variables x ad y ad defie algebraic ad cetral bivariate momets of order k, l as µ kl E(x k y l ) = x k y l f(x, y)dxdy µ kl E((x µ x ) k (y µ y ) l ) = (x µ x ) k (y µ y ) l f(x, y)dxdy where µ x ad µ y are the mea values of x ad y. The covariace is a cetral bivariate momet of order, i.e. Cov(x, y) = µ. Similarly oe easily defies multivariate momets for distributio i discrete variables..7. Errors of Bivariate Momets Algebraic (m rs) ad cetral (m rs ) bivariate momets are defied by: m rs = x r i yi s ad m rs = i= (x i m ) r (y i m ) s i= Whe there is a risk of ambiguity we write m r,s istead of m rs. 9

22 The otatios m rs ad m rs are used for the statistics (sample values) while we write µ rs ad µ rs for the populatio values. The errors of bivariate momets are give by Cov(m rs, m uv) = (µ r+u,s+v µ rsµ uv) especially Cov(m rs, m uv ) = (µ r+u,s+v µ rs µ uv + ruµ µ r,s µ u,v + svµ µ r,s µ u,v +rvµ µ r,s µ u,v + suµ µ r,s µ u,v uµ r+,s µ u,v vµ r,s+ µ u,v rµ r,s µ u+,v sµ r,s µ u,v+ ) V (m rs) = (µ r,s µ rs) V (m rs ) = (µ r,s µ rs + r µ µ r,s + s µ µ r,s +rsµ µ r,s µ r,s rµ r+,s µ r,s sµ r,s+ µ r,s ) For the covariace (m ) we get by error propagatio V (m ) = (µ µ ) Cov(m, m ) = µ Cov(m, m ) = (µ 3 µ µ ) For the correlatio coefficiet (deoted by ρ = µ / µ µ for the populatio value ad by r for the sample value) we get V (r) = ρ { µ + [ µ4 + µ 4 + µ ] [ µ3 + µ ]} 3 µ 4 µ µ µ µ µ µ µ Beware, however, that the samplig distributio of r teds to ormality very slowly..7.3 Joit Characteristic Fuctio The joit characteristic fuctio is defied by φ(t, t,..., t ) = E(e ıt x +ıt x +...t x ) = =... e ıt x +ıt x +...+ıt x f(x, x,..., x )dx dx... dx From this fuctio multivariate momets may be obtaied e.g. for a bivariate distributio algebraic bivariate momets are give by µ rs = E(x r x s ) = r+s φ(t, t ) (ıt ) r (ıt ) s t =t =

23 .7.4 Radom Number Geeratio Radom samplig from a may dimesioal distributio with a joit probability desity fuctio f(x, x,..., x ) ca be made by the followig method: Defie the margial distributios g m (x, x,..., x m ) = f(x,..., x )dx m+ dx m+...dx = g m+ (x,..., x m+ )dx m+ Cosider the coditioal desity fuctio h m give by h m (x m x, x,...x m ) g m (x, x,..., x m )/g m (x, x,..., x m ) We see that g = f ad that h m (x m x, x,..., x m )dx m = from the defiitios. Thus h m is the coditioal distributio i x m give fixed values for x, x,..., x m. We ca ow factorize f as f(x, x,..., x ) = h (x )h (x x )... h (x x, x,..., x ) We sample values for x, x,..., x from the joit probability desity fuctio f by: Geerate a value for x from h (x ). Use x ad sample x from h (x x ). Proceed step by step ad use previously sampled values for x, x,..., x m to obtai a value for x m+ from h m+ (x m+ x, x,..., x m ). Cotiue util all x i :s have bee sampled. If all x i :s are idepedet the coditioal desities will equal the margial desities ad the variables ca be sampled i ay order.

24 3 Beroulli Distributio 3. Itroductio The Beroulli distributio, amed after the swiss mathematicia Jacques Beroulli (654 75), describes a probabilistic experimet where a trial has two possible outcomes, a success or a failure. The parameter p is the probability for a success i a sigle trial, the probability for a failure thus beig p (ofte deoted by q). Both p ad q is limited to the iterval from zero to oe. The distributio has the simple form p(r; p) = { p = q if r = (failure) p if r = (success) ad zero elsewhere. The work of J. Beroulli, which costitutes a foudatio of probability theory, was published posthumously i Ars Cojectadi (73) [4]. The probability geeratig fuctio is G(z) = q +pz ad the distributio fuctio give by P () = q ad P () =. A radom umbers are easily obtaied by usig a uiform radom umber variate ξ ad puttig r = (success) if ξ p ad r = else (failure). 3. Relatio to Other Distributios From the Beroulli distributio we may deduce several probability desity fuctios described i this documet all of which are based o series of idepedet Beroulli trials: Biomial distributio: expresses the probability for r successes i a experimet with trials ( r ). Geometric distributio: expresses the probability of havig to wait exactly r trials before the first successful evet (r ). Negative Biomial distributio: expresses the probability of havig to wait exactly r trials util k successes have occurred (r k). This form is sometimes referred to as the Pascal distributio. Sometimes this distributio is expressed as the umber of failures occurrig while waitig for k successes ( ).

25 4 Beta distributio 4. Itroductio The Beta distributio is give by f(x; p, q) = B(p, q) xp ( x) q where the parameters p ad q are positive real quatities ad the variable x satisfies x. The quatity B(p, q) is the Beta fuctio defied i terms of the more commo Gamma fuctio as B(p, q) = Γ(p)Γ(q) Γ(p + q) For p = q = the Beta distributio simply becomes a uiform distributio betwee zero ad oe. For p = ad q = or vise versa we get triagular shaped distributios, f(x) = x ad f(x) = x. For p = q = we obtai a distributio of parabolic shape, f(x) = 6x( x). More geerally, if p ad q both are greater tha oe the distributio has a uique mode at x = (p )/(p + q ) ad is zero at the ed-poits. If p ad/or q is less tha oe f() ad/or f() ad the distributio is said to be J-shaped. I figure below we show the Beta distributio for two cases: p = q = ad p = 6, q = 3. Figure : Examples of Beta distributios 4. Derivatio of the Beta Distributio If y m ad y are two idepedet variables distributed accordig to the chi-squared distributio with m ad degrees of freedom, respectively, the the ratio y m /(y m + y ) follows a Beta distributio with parameters p = m ad q =. 3

26 To show this we make a chage of variables to x = y m /(y m + y ) ad y = y m + y which implies that y m = xy ad y = y( x). We obtai f(x, y) = = = y m x y x y y y m y y y ( x ym x Γ ( ) m+ Γ ( m ) Γ ( f(y m, y ) = ) m e ym Γ ( ) m ( y )x m ( x) ) e y ) Γ ( ( ) m y + e y Γ ( ) m+ = which we recogize as a product of a Beta distributio i the variable x ad a chi-squared distributio with m + degrees of freedom i the variable y (as expected for the sum of two idepedet chi-square variables). 4.3 Characteristic Fuctio The characteristic fuctio of the Beta distributio may be expressed i terms of the cofluet hypergeometric fuctio (see sectio 43.3) as 4.4 Momets φ(t) = M(p, p + q; ıt) The expectatio value, variace, third ad fourth cetral momet are give by E(x) = V (x) = µ 3 = µ 4 = p p + q pq (p + q) (p + q + ) pq(q p) (p + q) 3 (p + q + )(p + q + ) 3pq((p + q) + pq(p + q 6)) (p + q) 4 (p + q + )(p + q + )(p + q + 3) More geerally algebraic momets are give i terms of the Beta fuctio by µ k = B(p + k, q) B(p, q) 4.5 Probability Cotet I order to fid the probability cotet for a Beta distributio we form the cumulative distributio x F (x) = t p ( t) q dt = B x(p, q) B(p, q) B(p, q) = I x(p, q) 4

27 where both B x ad I x seems to be called the icomplete Beta fuctio i the literature. The icomplete Beta fuctio I x is coected to the biomial distributio for iteger values of a by ( ) a I x (a, b) = I x (b, a) = ( x) a+b ( a + b x i x or expressed i the opposite directio s=a i= ( ) p s ( p) s = I p (a, a + ) s Also to the egative biomial distributio there is a coectio by the relatio ( ) + s p q s = I q (a, ) s=a s The icomplete Beta fuctio is also coected to the probability cotet of Studet s t-distributio ad the F -distributio. See further sectio 4.7 for more iformatio o I x. 4.6 Radom Number Geeratio I order to obtai radom umbers from a Beta distributio we first sigle out a few special cases. For p = ad/or q = we may easily solve the equatio F (x) = ξ where F (x) is the cumulative fuctio ad ξ a uiform radom umber betwee zero ad oe. I these cases p = x = ξ /q q = x = ξ /p For p ad q half-itegers we may use the relatio to the chi-square distributio by formig the ratio y m y m + y with y m ad y two idepedet radom umbers from chi-square distributios with m = p ad = q degrees of freedom, respectively. Yet aother way of obtaiig radom umbers from a Beta distributio valid whe p ad q are both itegers is to take the l:th out of k ( l k) idepedet uiform radom umbers betwee zero ad oe (sorted i ascedig order). Doig this we obtai a Beta distributio with parameters p = l ad q = k + l. Coversely, if we wat to geerate radom umbers from a Beta distributio with iteger parameters p ad q we could use this techique with l = p ad k = p+q. This last techique implies that for low iteger values of p ad q simple code may be used, e.g. for p = ad q = we may simply take max(ξ, ξ ) i.e. the maximum of two uiform radom umbers. ) i 5

28 5 Biomial Distributio 5. Itroductio The Biomial distributio is give by p(r; N, p) = ( ) N p r ( p) N r r where the variable r with r N ad the parameter N (N > ) are itegers ad the parameter p ( p ) is a real quatity. The distributio describes the probability of exactly r successes i N trials if the probability of a success i a sigle trial is p (we sometimes also use q = p, the probability for a failure, for coveiece). It was first preseted by Jacques Beroulli i a work which was posthumously published [4]. 5. Momets The expectatio value, variace, third ad fourth momet are give by E(r) = Np V (r) = Np( p) = Npq µ 3 = Np( p)( p) = Npq(q p) µ 4 = Np( p) [ + 3p( p)(n )] = Npq [ + 3pq(N )] Cetral momets of higher orders may be obtaied by the recursive formula µ r+ = pq { Nrµ r + µ r p startig with µ = ad µ =. The coefficiets of skewess ad kurtosis are give by γ = q p Npq ad γ = 6pq Npq 5.3 Probability Geeratig Fuctio The probability geeratig fuctio is give by ) N G(z) = E(z r ) = z r( N p r ( p) N r = (pz + q) N r= r ad the characteristic fuctio thus by φ(t) = G(e ıt ) = ( q + pe ıt) N } 6

29 5.4 Cumulative Fuctio For fixed N ad p oe may easily costruct the cumulative fuctio P (r) by a recursive formula, see sectio o radom umbers below. However, a iterestig ad useful relatio exist betwee P (r) ad the icomplete Beta fuctio I x amely k P (k) = p(r; N, p) = I p (N k, k + ) r= For further iformatio o I x see sectio Radom Number Geeratio I order to achieve radom umbers from a biomial distributio we may either Geerate N uiform radom umbers ad accumulate the umber of such that are less or equal to p, or Use the cumulative techique, i.e. costruct the cumulative, distributio, fuctio ad by use of this ad oe uiform radom umber obtai the required radom umber, or for larger values of N, say N >, use a approximatio to the ormal distributio with mea Np ad variace Npq. Except for very small values of N ad very high values of p the cumulative techique is the fastest for umerical calculatios. This is especially true if we proceed by costructig the cumulative vector oce for all (as opposed to makig this at each call) usig the recursive formula p(i) = p(i ) p N + i q i for i =,,..., N startig with p() = q N. However, usig the relatio give i the previous sectio with a well optimized code for the icomplete Beta fuctio (see [] or sectio 4.7) turs out to be a umerically more stable way of creatig the cumulative distributio tha a simple loop addig up the idividual probabilities. 5.6 Estimatio of Parameters Experimetally the quatity r, the relative umber of successes i N trials, ofte is of more N iterest tha r itself. This variable has expectatio E( r ) = p ad variace V ( r ) = pq. N N N The estimated value for p i a experimet givig r successes i N trials is ˆp = r. N If p is ukow a ubiased estimate of the variace of a biomial distributio is give by V (r) = N ( ) ( r N N r ) = N N ˆp( ˆp) N N N This is possible oly if we require radom umbers from oe ad the same biomial distributio with fixed values of N ad p. 7

30 To fid lower ad upper cofidece levels for p we proceed as follows. For lower limits fid a p low such that N r=k ( ) N p r r low( p low ) N r = α or expressed i terms of the icomplete Beta fuctio I p (N k +, k) = α for upper limits fid a p up such that k r= ( ) N p r r up( p up ) N r = α which is equivalet to I p (N k, k + ) = α i.e. I p (k +, N k) = α. As a example we take a experimet with N = where a certai umber of successes k N have bee observed. The cofidece levels correspodig to 9%, 95%, 99% as well as the levels correspodig to oe, two ad three stadard deviatios for a ormal distributio (84.3%, 97.7% ad 99.87% probability cotet) are give below. Lower cofidece levels Upper cofidece levels k 3σ 99% σ 95% 9% σ ˆp σ 9% 95% σ 99% 3σ Probability Cotet It is sometimes of iterest to judge the sigificace level of a certai outcome give the hypothesis that p =. If N trials are made ad we fid k successes (let s say k < N/ else use N k istead of k) we wat to estimate the probability to have k or fewer successes plus the probability for N k or more successes. Sice the assumptio is that p = we wat the two-tailed probability cotet. To calculate this either sum the idividual probabilities or use the relatio to the icomplete beta fuctio. The former may seem more straightforward but the latter may be computatioally easier give a routie for the icomplete beta fuctio. If k = N/ we watch up ot to add the cetral term twice (i this case the requested probability is % ayway). I the table below we show such cofidece levels i % for values of N ragig from to. E.g. the probability to observe 3 successes (or failures) or less ad failures (or successes) or more for = 5 is 3.5%. 8

31 k N

32 6 Biormal Distributio 6. Itroductio As a geeralizatio of the ormal or Gauss distributio to two dimesios we defie the biormal distributio as ( ( ) ( ) ) x µ x f(x, x ) = πσ σ ρ e ( ρ + µ ρ x µ x µ ) σ σ σ σ where µ ad µ are the expectatio values of x ad x, σ ad σ their stadard deviatios ad ρ the correlatio coefficiet betwee them. Puttig ρ = we see that the distributio becomes the product of two oe-dimesioal Gauss distributios. 4 3 x x Figure : Biormal distributio I figure we show cotours for a stadardized Biormal distributio i.e puttig µ = µ = ad σ = σ = (these parameters are ayway shift- ad scale-parameters oly). I the example show ρ =.5. Usig stadardized variables the cotours rage from a perfect circle for ρ = to gradually thier ellipses i the ±45 directio as ρ ±. The cotours show correspod to the oe, two, ad three stadard deviatio levels. See sectio o probability cotet below for details.

33 6. Coditioal Probability Desity The coditioal desity of the biormal distributio is give by f(x y) = f(x, y)/f(y) = = exp πσx ρ σx( ρ ) ( = N µ x + ρ σ ) x (y µ y ), σ σ x( ρ ) y [ x ( µ x + ρσ )] x (y µ y ) σ y = which is see to be a ormal distributio which for ρ = is, as expected, give by N(µ x, σ x) but geerally has a mea shifted from µ x ad a variace which is smaller tha σ x. 6.3 Characteristic Fuctio The characteristic fuctio of the biormal distributio is give by φ(t, t ) = E(e ıt x +ıt x ) = = exp { ıt µ + ıt µ + e ıt x +ıt x f(x, x )dx dx = [ (ıt ) σ + (ıt ) σ + (ıt )(ıt )ρσ σ ]} which shows that if the correlatio coefficiet ρ is zero the the characteristic fuctio factorizes i.e. the variables are idepedet. This is a uique property of the ormal distributio sice i geeral ρ = does ot imply idepedece. 6.4 Momets To fid bivariate momets of the biormal distributio the simplest, but still quite tedious, way is to use the characteristic fuctio give above (see sectio.7.3). Algebraic bivariate momets for the biormal distributio becomes somewhat complicated but ormally they are of less iterest tha the cetral oes. Algebraic momets of the type µ k ad µ k are, of course, equal to momets of the margial oe-dimesioal ormal distributio e.g. µ = µ, µ = µ + σ, ad µ 3 = µ (σ + µ ) (for µ k simply exchage the subscripts o µ ad σ). Some other lower order algebraic bivariate momets are give by µ = µ µ + ρσ σ µ = ρσ σ µ + σ µ + µ µ µ = σ σ + σ µ + σ µ + µ µ + ρ σ σ + 4ρσ σ µ µ Beware of the somewhat cofusig otatio where µ with two subscripts deotes bivariate momets while µ with oe subscript deotes expectatio values. Lower order cetral bivariate momets µ kl, arraged i matrix form, are give by