On the Multivariate t Distribution

Transcription

1 Technical report from Automatic Control at Linköpings universitet On the Multivariate t Distribution Michael Roth Division of Automatic Control roth@isy.liu.se 7th April 03 Report no.: LiTH-ISY-R-3059 Aress: Department of Electrical Engineering Linköpings universitet SE Linköping, Sween WWW: AUTOMATIC CONTROL REGLERTEKNIK LINKÖPINGS UNIVERSITET Technical reports from the Automatic Control group in Linköping are available from

2 Abstract This technical report summarizes a number of results for the multivariate t istribution which can exhibit heavier tails than the Gaussian istribution. It is shown how t ranom variables can be generate, the probability ensity function (pf) is erive, an marginal an conitional ensities of partitione t ranom vectors are presente. Moreover, a brief comparison with the multivariate Gaussian istribution is provie. The erivations of several results are given in an extensive appenix. Keywors: Stuent's t istribution, heavy tails, Gaussian istribution.

3 On the Multivariate t Distribution Michael Roth April 7, 03 contents Introuction Representation an moments 3 Probability ensity function 3 4 Affine transformations an marginal ensities 3 5 Conitional ensity 4 6 Comparison with the Gaussian istribution 4 6. Gaussian istribution as limit case 5 6. Tail behavior Depenence an correlation Probability regions an istributions of quaratic forms Parameter estimation from ata 8 a Probability theory backgroun an etaile erivations 8 a. Remarks on notation 8 a. The transformation theorem 9 a.3 Utilize probability istributions 9 a.4 Derivation of the t pf a.5 Useful matrix computations 4 a.6 Derivation of the conitional t pf 5 a.7 Mathematica coe 7 introuction This technical report summarizes a number of results for the multivariate t istribution [, 3, 7] which can exhibit heavier tails than the Gaussian istribution. It is shown how t ranom variables can be generate, the probability ensity function (pf) is erive, an marginal an conitional ensities of partitione t ranom vectors are presente. Moreover, a brief comparison with the multivariate Gaussian istribution is provie. The erivations of several results are given in an extensive appenix. The reaer is assume to have a working knowlege of probability theory an the Gaussian istribution in particular. Nevertheless, important techniques an utilize probability istributions are briefly covere in the appenix.

4 Any t ranom variable is escribe by its egrees of freeom parameter ν (scalar an positive), a mean µ (imension ), an a symmetric matrix parameter Σ (imension ). In general, Σ is not the covariance of X. In case Σ is positive efinite, we can write the pf of a t ranom vector X (imension ) as St(x; µ, Σ, ν) = Γ( ν+ ) Γ( ν ) (νπ) ( + +ν et(σ) ν (x µ)t Σ (x µ). (.) Equation (.) is presente in, for instance, [, 3, 7]. representation an moments Multivariate t ranom variables can be generate in a number of ways [7]. We here combine Gamma an Gaussian ranom variables an show some properties of the resulting quantity without specifying its istribution. The fact that the erive ranom variable amits inee a t istribution with pf (.) will be postpone to Section 3. The use of the more general Gamma istribution instea of a chi-square istribution [3] shall turn out to be beneficial in eriving more involve results in Section 5. Let V > 0 be a ranom variable that amits the Gamma istribution Gam(α, β) with shape parameter α > 0, an rate (or inverse scale) parameter β > 0. The relate pf is given in (A.3). Furthermore, consier the -imensional, zero mean, ranom variable Y with Gaussian istribution N (0, Σ). Σ is the positive efinite covariance matrix of Y. The relate pf is shown in (A.). Given that V an Y are inepenent of one another, the following ranom variable is generate X = µ + V Y. (.) with a vector µ. Realizations of X are scattere aroun µ. The sprea is intuitively influence by the pf of V, an it can be easily seen that for V = v we have X v N (µ, v Σ), a Gaussian with scale covariance matrix. In the limit α = β, the Gamma pf (A.3) approaches a Dirac impulse centere at one, which results in V = with probability. It follows (without further specification of the istribution) that X in (.) then reuces to a Gaussian ranom variable. With mere knowlege of (.) we can erive moments of X. Trivially taking the expectation reveals E(X) = µ. For higher moments, it is convenient work with W = V which amits an inverse Gamma istribution IGam(α, β). The ranom variable W has the pf (A.5) an the expecte value E(W) = β α for α >. The covariance computations cov(x) = E(X E(X))(X E(X)) T = E(WYY T ) = β Σ, α > (.) α

5 reveal that Σ is in general not a covariance matrix, an finite only if α >. Higher moments can in principle be compute in a similar manner, again with conitions for existence. 3 probability ensity function The probability ensity function of X in (.) can be erive using a transformation theorem that can be foun in many textbooks on multivariate statistics, for instance [4] or [9]. The theorem is presente, in a basic version, in Appenix A.. We immeiately present the main result but show etaile calculations in Appenix A.4. Let V Gam(α, β) an Y N (0, Σ). Then X = µ + V Y amits a t istribution with pf p(x) = Γ(α + ) Γ(α) (βπ) ( + α et(σ) β (3.) where = (x µ) T Σ (x µ). = St(x; µ, β Σ, α) (3.) α Obviously, (3.) is a t pf in isguise. For α = β = ν, (3.) turns into the familiar (.). Intermeiate steps to obtain (3.) an (3.) inclue introuction of an auxiliary variable an subsequent marginalization. Intermeiate results reveal that β an Σ only appear as prouct βσ. Consequently, β can be set to any positive number without altering (3.), if the change is compensate for by altering Σ. This relates to the fact that t ranom variables can be constructe using a wie range of rules [7]. The parametrization using α an β is use to erive the conitional pf of a partitione t ranom variable in Appenix A.6. 4 affine transformations an marginal ensities The following property is common to both t an Gaussian istribution. Consier a z x matrix A an a z -imensional vector b. Let X be a t ranom variable with pf (.). Then, the ranom variable Z = AX + b has the pf p(z) = St(z; Aµ + b, AΣA T, ν). (4.) The result follows from (.) an the formulas for affine transformations of Gaussian ranom variables []. Here, A must be such that AΣA T is invertible for the pf to exist. The egrees of freeom parameter ν is not altere by affine transformations. There is an immeiate consequence for partitione ranom variables X T = [ X T X T ] which are compose of X an X, with respective imensions an. From the joint pf p(x) = p(x, x ) = St( [ x x ] ; [ µ µ ], [ Σ Σ Σ T Σ ], ν), (4.) 3

6 the marginal pf of either X or X is easily obtaine by applying a linear transformation. Choosing A = [ 0 I ], for instance, yiels the marginal ensity of X p(x ) = St (x ; µ, Σ, ν). (4.3) 5 conitional ensity The main result of this section is that X given X = x amits another t istribution if X an X are jointly t istribute with pf (4.). The erivations are straightforwar but somewhat involve, an therefore provie in Appenix A.6. The conitional ensity is given by p(x x ) = p(x, x ) p(x ) = St(x ; µ, Σ, ν ), (5.) where ν = ν +, (5.) µ = µ + Σ Σ (x µ ), (5.3) Σ = ν + (x µ ) T Σ (x µ ) (Σ ν + Σ Σ ). (5.4) These expressions are in accorance with the formulas given in [6] (without erivation) an [3] (erivation is given as exercise). The book [7] misleaingly states that the conitional istribution is in general not t. However, a generalize t istribution is presente in Chapter 5 of [7], which provies the property that the conitional istribution is again t. The erivations provie here show that there is no nee to consier a generalization of the t for such a closeness property. Obviously, conitioning increases the egrees of freeom. The expression for µ is the same as the conitional mean in the Gaussian case []. The matrix parameter Σ, in contrast to the Gaussian case, is also influence by the realization x. By letting ν go to infinity, we can recover the Gaussian conitional covariance: lim ν Σ = Σ Σ Σ ΣT. 6 comparison with the gaussian istribution The Gaussian istribution appears to be the most popular istribution among continuous multivariate unimoal istributions. Motivation for this is given by the many convenient properties: simple marginalization, conitioning, parameter estimation from ata. We here compare the t istribution with the Gaussian to assess whether some of the known Gaussian results generalize. After all, the t istribution can be interprete as generalization of the Gaussian (or conversely, the Gaussian as special case of the t istribution). 4

7 6. Gaussian istribution as limit case It is a well known result [] that, as the egrees of freeom ν ten to infinity, the t istribution converges to a Gaussian istribution. This fact can be establishe, for instance, using (.) an the properties of the Gamma istribution. Another encounter with this result is escribe in Appenix A Tail behavior One reason to employ the t istribution is its ability to generate outliers, which facilitates more accurate moeling of many real worl scenarios. In contrast to the Gaussian, the t istribution has heavy tails. To illustrate this, we consier the -imensional example with parameters µ = [ ] 0, Σ = 0 [ ] 0.4, ν = 5. (6.) 0.4 Figure contains 5 samples each, rawn from the Gaussian N (µ, Σ) an the t istribution St(µ, Σ, ν). Close to µ, there is no obvious ifference between Gaussian an t samples. More t samples are scattere further from the mean, with one outlier being far off. The illustrate phenomenon can be explaine by the t istribution having heavier tails: the pf oes not ecay as quickly when moving away from the mean Figure : 5 ranom samples each from the -imensional istributions N (µ, Σ) (crosses) an St(µ, Σ, ν) (circles) with the parameters given in (6.). The ellipses illustrate regions that contain 95 percent of the probability mass, soli for the Gaussian an ashe for the t istribution. For the presente two-imensional case, we can compute both Gaussian an t pf on a gri an visualize the results. Figure contains heat maps that illustrate the pf logarithms ln N (x; µ, Σ) an ln St(x; µ, Σ, ν), where the logarithmic scale helps highlight the ifference in tail behavior. The coloring is the same in Fig. a an Fig. b, but Fig. a contains a large proportion of white ink which correspons to values below 5. While the ensities appear similar close to µ, the Gaussian rops quickly with increasing istance whereas the t pf shows a less rapi ecay. The connection between 5

8 (a) ln N ( x; µ, Σ). (b) ln St( x; µ, Σ, ν). Figure : Heat maps of ln N ( x; µ, Σ) an ln St( x; µ, Σ, ν) for a two-imensional x. The logarithmic presentation helps highlight the ifference in tail behavior. White areas in (a) correspon to values below 5. The parameters are given by (6.). the occurrence of samples that lie on ellipses that are further away from µ an the corresponing values of the pf is apparent D D -40 (a) Linear scale. (b) Logarithmic scale. Figure 3: Gaussian pf (soli) an t pf (ashe) with = 3 an ν = 3, as functions of the Mahalanobis istance. The general -imensional case is a bit trickier to illustrate. An investigation of the ensity expression (.) an (A.), however, reveals that both can be written as functions of a scalar quantity = ( x µ)t Σ ( x µ). A constant correspons to an ellipsoial region (an ellipse for = ) with constant pf, as seen in Figure. can be interprete as a square weighte istance an is wiely known as square Mahalanobis istance []. Having unerstoo the istance epenence, it is reasonable to plot each pf as function of. Furthermore, Σ can be set to unity without loss of generality when comparing N ( x; µ, Σ) an St( x; µ, Σ, ν), because of the common factor et(σ) /. Figure 3 contains the Gaussian an t pf as functions of with Σ = I. The shape of the curves is etermine by ν an only. For =, such illustrations coincie with the univariate ensities (over the positive ray). Here, we isplay the functions for = 3 an ν = 3 instea but provie the 6

9 mathematica coe snippet that can be use to prouce interactively manipulable plots for arbitrary an ν in Section A.7. For the chosen parameters, the t pf appears more peake towars the mean (small in Fig. 3a). The logarithmic isplay (Fig. 3b) again shows the less rapi ecay of the t pf with increasing, commonly known as heavy tails. 6.3 Depenence an correlation Gaussian ranom variables that are not correlate are by efault inepenent: the joint pf is the prouct of marginal ensities. This is not the case for arbitrary other istributions, as we shall now establish for the t istribution. Consier the a partitione t ranom vector X with pf (4.). For Σ = 0 we see from the covariance result (.) that X an X are not correlate. The joint pf, however, p(x, x ) ( + ν (x µ ) T Σ (x µ ) + ν (x µ ) T Σ (x ν µ ) oes not factor trivially. Consequently, uncorrelate t ranom variables are in general not inepenent. Furthermore, the prouct of two t ensities is in general not a t pf (unless we have p(x x )p(x )). 6.4 Probability regions an istributions of quaratic forms The following paragraphs make heavy use of the results provie in Appenix A.3. For a Gaussian ranom variable X N (µ, Σ), it is well known [3] that the quaratic form (or square Mahalanobis istance) = (X µ) T Σ (X µ) amits a chi-square istribution χ (). This can be use to efine symmetric regions aroun µ in which a fraction, say P percent, of the probability mass is concentrate. Such a P percent probability region contains all x for which δ (x µ) T Σ (x µ) δp, with P χ (r; )r = P/00. (6.) The require δ P can be obtaine from the inverse cumulative istribution function of χ (), which is implemente in many software packages. For X St(µ, Σ, ν) similar statements can be mae. The involve computations, however, are slightly more involve. Recall from (.) that X can be generate using a Gaussian ranom vector Y N (0, Σ) an the Gamma ranom variable V Gam( ν, ν ). Furthermore, observe that V can be written as V = W/ν with a chi-square ranom variable W χ (ν), as explaine in Appenix A.3. Using 0 X = µ + V Y = µ + W/ν Y, (6.3) 7

10 we can re-arrange the square form = (X µ) T Σ (X µ) = YT Σ Y/ W/ν (6.4) an obtain a ratio of two scale chi-square variables, since Y T Σ Y χ (). Knowing that such a ratio amits an F istribution, we have (X µ)t Σ (X µ) F(, ν) (6.5) with pf given by (A.6). Following the steps that were taken for the Gaussian case, we can erive the symmetric P percent probability region for X St(µ, Σ, ν) as the set of all x for which (x µ) T Σ (x µ) r P, with rp 0 F(r;, ν)r = P/00. (6.6) Again, the require r P can be obtaine from an inverse cumulative istribution function, that is available in many software packages. Figure contains two ellipses, a soli for the Gaussian an a ashe for the t istribution. Both illustrate regions which contain 95 percent of the probability mass, an have been obtaine with the techniques explaine in this section. It can be seen that the 95 percent ellipse for the t istribution stretches out further than the Gaussian ellipse, which of course relates to the heavy tails of the t istribution. 6.5 Parameter estimation from ata Methos for fining the parameters of a Gaussian ranom variables from a number of inepenent an ientically istribute realizations are well known. A comprehensive treatment of maximum likelihoo an Bayesian methos is given in []. The solutions are available in close form an comparably simple. The maximum likelihoo estimate for the mean of a Gaussian, for instance, coincies with the average over all realizations. For the t istribution, the are no such simple close form solutions. However, iterative maximum likelihoo schemes base on the expectation maximization algorithm are available. The reaer is referre to [8] for further etails. a probability theory backgroun an etaile erivations a. Remarks on notation We make a istinction between the istribution a ranom variable amits an its pf. For example, X N (µ, Σ) enotes that X amits a Gaussian istribution with parameters µ an Σ. In contrast, p(x) = N (x; µ, Σ) (with the extra argument) enotes the Gaussian probability ensity function. Furthermore, ranom variables are capitalize whereas the (ummy) argument of a pf is lower 8

11 case. Where require, the pf carries an extra inex that clarifies the corresponing ranom variable, for instance p X (x) in Section A.. a. The transformation theorem We here introuce a powerful tool which we plainly refer to as the transformation theorem. Uner some conitions, it can be use to fin the pf of a ranom variable that is transforme by a function. We only present the theorem in its most basic form, but refer the reaer to [4, 9] for further etails. For clarity purposes all probability ensity functions carry an extra inex. Transformation theorem: Let X be a scalar ranom variable with pf p X (x), that amits values from some set D X. Let f in y = f (x) be a one-to-one mapping for which an inverse g with x = g(y) exists, with an existing erivative g (y) that is continuous. Then, the ranom variable Y = f (X) has the pf p Y (y) = { px ( g(y) ) g (y), y D Y, 0, otherwise, (A.) with D Y = { f (x) : x D X }. The theorem is presente in [9], Ch.3, or [4], Ch., incluing proofs an extensions to cases when f (x) is not a one-to-one mapping (bijection). The occurrence of g (y) is the result of a change of integration variables, an in the multivariate case g (y) is merely replace by the eterminant of the Jacobian matrix of g(y). In case Y is of lower imension than X, auxiliary variables nee to be introuce an subsequently marginalize. An example of the entire proceure is given for the erivation of the multivariate t pf in Section A.4. a.3 Utilize probability istributions We here present some well known probability istributions that are use in this ocument. All of these are covere in a plethora of text books. We aopt the presentation of [] an [3]. Gaussian istribution A -imensional ranom variable X with Gaussian istribution N (µ, Σ) is characterize by its mean µ an covariance matrix Σ. For positive efinite Σ, the pf is given by N (x; µ, Σ) = (π) ( exp ) et(σ) (x µ)t Σ (x µ). (A.) Information about the many convenient properties of the Gaussian istribution can be foun in almost any of the references at the en of this ocument. A particularly extensive account is given in []. We shall therefore only briefly mention some aspects. Gaussian ranom variables that are uncorrelate are also inepenent. The marginal an conitional istribution of a partitione Gaussian ranom vector are again Gaussian, which relates to the Gaussian being its own conjugate 9

12 prior in Bayesian statistics []. The formulas for marginal an conitional pf can be use to erive the Kalman filter from a Bayesian perspective []. Gamma istribution A scalar ranom variable X amits the Gamma istribution Gam(α, β) with shape parameter α > 0 an rate (or inverse scale) parameter β > 0 if it has the pf [, 3] Gam(x; α, β) = { β α Γ(α) xα exp( βx), x > 0, 0, otherwise. (A.3) Γ(z) = 0 e t t z t is the Gamma function. Two parametrizations of the Gamma istribution exist in literature: the rate version that we aopt, an a version that uses the inverse rate (calle scale parameter) instea. Hence, care must be taken to have the correct syntax in e.g. software packages. Using the transformation theorem of Section A. an X Gam(α, β), we can erive that cx Gam(α, β c ). Mean an variance of X Gam(α, β) are E(X) = α β an var(x) = α β, respectively. For α = β the support of the pf collapses an Gam(x; α, β) approaches a Dirac impulse centere at one. chi-square istribution A special case of the Gamma istribution for α = ν, β = is the chi-square istribution χ (ν) with egrees of freeom ν > 0 an pf ν χ (x; ν) = Gam(x; ν, ) = Γ( ν ) x ν exp( x), x > 0, 0, otherwise. (A.4) From the scaling property of the Gamma istribution we have that X/ν Gam( ν, ν ) for X χ (ν), which relates to a common generation rule for t ranom variables [3, 7]. Inverse Gamma istribution In orer to compute the moments of t ranom vectors, it is require to work with the inverse of a Gamma ranom variable. We shall here present the relate pf, an even sketch the erivation as it is not containe in [] or [3]. Let Y Gam(α, β). Then its inverse X = /Y has an inverse Gamma istribution with parameters α > 0 an β > 0, an pf IGam(x; α, β) = { β α Γ(α) x α exp( β/x), x > 0, 0, otherwise. (A.5) 0

13 The result follows from the transformation theorem of Section A.. Using g(x) = /x, we obtain p X (x) = p Y (g(x)) g (x) = Gam(x ; α, β)x which gives (A.5). The pf for one set of parameters is illustrate in Figure 4. The mean of Figure 4: The pf of the inverse of a Gamma ranom variable, IGam(x;.5,.5). X IGam(α, β) is E(X) = moments to exist. β α an is finite for α >. Similar conitions are require for higher F istribution We conclue this presentation of univariate ensities (A.3)-(A.5) with the F istribution [3]. Consier the (scale) ratio of two chi-square ranom variables Z = X/a Y/b with X χ (a) an Y χ (b). Recall that this is also a raius of two Gamma ranom variables. Z then has an F istribution [3] with a > 0 an b > 0 egrees of freeom, with pf Γ( a+b ) Γ( F(z; a, b) = a )Γ( b ) a a b b z a, z > 0, (b+az) a+b 0, otherwise. An illustration of the pf for one set of parameters is given in Figure 5. The mean value of Z F(a, b) is E(Z) = b (a )b b, an exists for b >. The moe, i.e. the maximum of F(z; a, b), is locate at a(b+), again for b >. The meian can be compute but the expression is not as conveniently written own as mean an moe. (A.6) a.4 Derivation of the t pf In this appenix we make use of the transformation theorem of Appenix A. to erive the pf of the ranom vector in (.). This technique involves the introuction of an auxiliary variable, a change of integration variables, an subsequent marginalization of the auxiliary variable. In the following calculations, we merely require (.) an the joint pf of V an Y.

14 x Figure 5: The pf of a scale ratio of chi-square ranom variables, F(x; 3, 3). V an Y are inepenent, an hence their joint ensity is the prouct of iniviual pfs p V,Y (v, y) = p V (v)p Y (y). We now introuce the pair U an X, which are relate to V an Y through X = µ + V Y; U = V. Here, it is X that we are actually intereste in. U is an auxiliary variable that we have to introuce in orer to get a one to one relation between (U, X) an (V, Y): Y = U(X µ); V = U. Accoring to the transformation theorem of Appenix A., the ensity of (U, X) is given by p U,X (u, x) = p V (u)p Y ( u(x µ) ) et(j) (A.7) with J as the Jacobian of the variable change relation [ ] [ ] y/ x T y/ u ui J = v/ x T = (x µ) u. v/ u 0 With J being upper triangular, the eterminant simplifies to et(j) = et( ui ) = u. We next write own each factor of the joint ensity (A.7) p Y ( u(x µ)) = (π) ( exp u(x µ) et(σ) T Σ ) u(x µ) = u N (x; µ, Σ/u), p V (u) = Gam(u; α, β),

15 an combine them accoring to (A.7) to obtain the joint ensity p U,X (u, x) = Gam(u; α, β)n (x; µ, Σ/u). From the above expression we can now integrate out u to obtain the sought ensity p X (x) = 0 Gam(u; α, β)n (x; µ, Σ/u)u. (A.8) The alert reaer might at this stage recognize the integral representation of the t pf [] for the case α = β = ν. As we procee further an compute the integral, the connections become more evient. Also, we see that if the Gamma part reuces to a Dirac impulse δ(u ) for α = β, the result of the integration is a Gaussian ensity. To simplify the upcoming calculations we introuce the square Mahalanobis istance [] = (x µ) T Σ (x µ) an procee: p X (x) = = 0 0 Γ(α) uα exp( βu) (π) ) u +α exp ( u(β + ) u β α (β + α Γ(α + ). ( exp u et(σ/u) ) u Arranging terms an re-introucing the constant factor eventually yiels the result of 3.: p X (x) = Γ(α + ) Γ(α) = Γ(α + ) Γ(α) β α ( β + α (π) et(σ) ( + α. (A.9) (βπ) et(σ) β Although not visible at first sight, the parameters β an Σ are relate. To show this we investigate the parts of (A.9) where β an Σ occur, β = (x µ)t (βσ (x µ), et(σ)β = et(βσ), (A.0) an see that the prouct of both etermines the ensity. This means that any ranom variable generate using (.) with V Gam(α, β c ) an Y N (y; 0, cσ) amits the same istribution for any c > 0. Consequently we can choose c = β α an thereby have V Gam(α, α) without loss of generality. More specifically, we can replace α = β = ν in (A.9) an recover the t pf [, 3, 7] of (.). 3

16 a.5 Useful matrix computations We here present results that are use in the erivation of the conitional pf of a partitione t ranom vector, as presente in Section A.6. However, similar expressions show up when working with the Gaussian istribution []. Consier the a partitione ranom vector X that amits the pf (.) with X = [ X X ], µ = [ µ µ ], Σ = [ Σ Σ Σ T Σ ], (A.) where X an X have imension an, respectively. Mean an matrix parameter are compose of blocks of appropriate imensions. Sometimes it is more convenient to work with the inverse matrix parameter Σ = Λ, another block matrix: [ ] [ ] Σ Σ Λ Λ Σ T = Σ Λ T. (A.) Λ Using formulas for block matrix inversion [5] we can fin the expressions Λ = (Σ Σ Σ, (A.3) Λ = (Σ Σ Σ Σ Σ = Λ Σ Σ (A.4) Σ = (Λ Λ T Λ Λ. (A.5) Furthermore, we have for the eterminant of Σ [ ] Σ Σ et( Σ T ) = et(σ ) et(σ Σ Σ Σ ΣT ) = et(σ ) et(λ ). (A.6) We next investigate the quaratic form in which x enters the t pf (.). Following [] we term square Mahalanobis istance. Inserting the parameter blocks (A.) an grouping powers of x yiels = (x µ)λ(x µ) = x T Λ x x T (Λ µ Λ (x µ )) + µ T Λ µ µ T Λ (x µ ) + (x µ ) T Λ (x µ ). With the intention of establishing compact quaratic forms we introuce the quantities µ = (Λ (Λ µ Λ (x µ )) = µ Λ Λ (x µ ) (A.7) = (x µ ) T Λ (x µ ) (A.8) 4

17 an procee. Completing the square in x an subsequent simplification yiels = µt Λ µ + µ T Λ µ µ T Λ (x µ ) + (x µ ) T Λ (x µ ) = + (x µ ) T( ) Λ Λ T Λ Λ (x µ ) = +, (A.9) where we use (A.5) an introuce = (x µ ) T Σ (x µ ). a.6 Derivation of the conitional t pf We here erive the conitional pf of X given that X = x, as presente in Section 5. Although basic probability theory provies the result as ratio of joint an marginal pf, the expressions take a very complicate form. Step by step, we isclose that the resulting pf is again Stuent s t. In the course of the erivations, we make heavy use of the results from appenix A.5. We furthermore work on the α, β parametrization of the t pf given in (3.). The conitional pf is given as ratio of joint (X an X ) an marginal (X only) pf p(x x ) = p(x, x ) p(x ) = Γ(α + ) Γ(α) (βπ) Γ(α + ) Γ(α) ( + α et(σ) β ( + (βπ) et(σ ) β α (A.0) where we combine (4.), (4.3), an (3.). For the moment, we ignore the normalization constants an focus on the bracket term of p(x, x ) (which is colore in re in (A.0)). Using the result (A.9) we can rewrite it ( + α ( = + β = ( + β β + β α α ( + β + The bracket expressions (both re an blue) in (A.0) now simplify to ( + α ( + β = ( + β β + α. α ( ) + +α β ( + β + α. (A.) 5

18 The remaining bits of (A.0), i.e. the ratio of normalization constants can be written as Γ(α + ) (βπ) Γ(α + ) (βπ) et(σ ) = Γ(α + + et(σ) Γ(α + ) ) (βπ) et(λ ), (A.) where we use the eterminant result (A.6). Combining the above results (A.) an (A.) gives the (still complicate) expression p(x x ) = Γ(α + + ( ) ( ) et(λ ) Γ(α α ) (βπ) β β +, which we shall now slowly shape into a recognizable t pf. Therefore, we introuce α = α +, an recall from (A.8) that = (x µ ) T Λ (x µ ): p(x x ) = Γ(α + ( ) ) ( et(λ ) + + α Γ(α ) (βπ) β β + ( et(λ ) = Γ(α + ) Γ(α ) ( π ( β + )) + (x µ ) T Λ (x µ ) β + α Comparison with (3.) reveals that this is also a t istribution in α, β parametrization with β = (β + ) = β + (x µ ) T Σ (x µ ), Σ = Λ = Σ Σ Σ ΣT. Obviously, α = β. Because, however, only the prouct β Σ enters a t pf, the parameters can be ajuste as iscusse in Section 3 an Appenix A.4. Converting back to ν parametrization, as in (.), we obtain p(x x ) = St(x ; µ, Σ, ν ) with ν = α = α + = ν +, µ = µ Λ Λ (x µ ) = µ + Σ Σ (x µ ), Σ = β α Σ which are the parameters provie in Section 5. = β + (x µ ) T Σ (x µ ) α + (Σ Σ Σ ΣT ) = ν + (x µ ) T Σ (x µ ) (Σ ν + Σ Σ ΣT ), 6

19 a.7 Mathematica coe The following coe snippet can be use to re-prouce Figure 3b. Moreover, by using mathematica s recently evelope Manipulate function, the means to interactively change an ν via two slier buttons are provie. More information can be foun on Manipulate[Plot[{Log[/(E^(Delta^/)*(*Pi)^(/))], Log[Gamma[( + nu)/]/(gamma[nu/]*((pi*nu)^(/)* (Delta^/nu + )^(( + nu)/)))]}, {Delta, 0, 0}, PlotStyle -> {{Thick, Black}, {Thick, Black, Dashe}}, PlotRange -> {-40, 0}, AxesLabel -> {"Delta", ""}], {,, 0,, Appearance -> "Labele"}, {nu, 3, 00,, Appearance -> "Labele"}] references [] B. D. Anerson an J. B. Moore. Optimal Filtering. Prentice Hall, June 979. [] C. M. Bishop. Pattern Recognition an Machine Learning. Springer, Aug [3] M. H. DeGroot. Optimal Statistical Decisions. Wiley-Interscience, WCL eition, Apr [4] A. Gut. An Intermeiate Course in Probability. Springer, secon eition, June 009. [5] T. Kailath, A. H. Saye, an B. Hassibi. Linear Estimation. Prentice Hall, Apr [6] G. Koop. Bayesian Econometrics. Wiley-Interscience, July 003. [7] S. Kotz an S. Naarajah. Multivariate t Distributions an Their Applications. Cambrige University Press, Feb [8] G. J. McLachlan an T. Krishnan. The EM Algorithm an Extensions. Wiley-Interscience, secon eition, Mar [9] C. R. Rao. Linear Statistical Inference an its Applications. Wiley-Interscience, secon eition, Dec

20

21 Avelning, Institution Division, Department Datum Date Division of Automatic Control Department of Electrical Engineering Språk Language Rapporttyp Report category ISBN Svenska/Sweish Licentiatavhanling ISRN Engelska/English Examensarbete C-uppsats D-uppsats Övrig rapport Serietitel och serienummer Title of series, numbering ISSN URL för elektronisk version LiTH-ISY-R-3059 Titel Title On the Multivariate t Distribution Författare Author Michael Roth Sammanfattning Abstract This technical report summarizes a number of results for the multivariate t istribution which can exhibit heavier tails than the Gaussian istribution. It is shown how t ranom variables can be generate, the probability ensity function (pf) is erive, an marginal an conitional ensities of partitione t ranom vectors are presente. Moreover, a brief comparison with the multivariate Gaussian istribution is provie. The erivations of several results are given in an extensive appenix. Nyckelor Keywors Stuent's t istribution, heavy tails, Gaussian istribution.