Relaxation Methods for Iterative Solution to Linear Systems of Equations

Size: px

Start display at page:

Download "Relaxation Methods for Iterative Solution to Linear Systems of Equations"

Warren Cunningham
7 years ago
Views:

1 Relaxato Methods for Iteratve Soluto to Lear Systems of Equatos Gerald Recktewald Portlad State Uversty Mechacal Egeerg Departmet Prmary Topcs Basc Cocepts Statoary Methods a.k.a. Relaxato methods Gauss Sedel Jacob SOR Relaxato Soluto to Lear Systems page 1

2 Drect ad Iteratve Methods Solve Ax = b where A s matrx, x ad b are elemet colum vectors. Gaussa elmato wth backward substtuto s a Drect Method A Drect Method obtas the soluto a fte umber of steps Soluto s equvalet to x = A 1 b Iteratve Methods obta a sequece of approxmatos to the soluto x k k =0, 1, 2,... such that (b Ax k ) 0 as k. Relaxato Soluto to Lear Systems page 2 Resdual The resdual r = b Ax s zero whe x s the soluto to Ax = b. For a teratve method the resdual at terato k s r k = b Ax k k =0, 1, 2,... If a teratve methods coverges, ther k 0 as k. Relaxato Soluto to Lear Systems page 3

Methods obta a sequece of approxmatos to the soluto x k k =0, 1, 2,... such that (b Ax k ) 0 as k.

3 Relaxato Methods (1) Overvew Smple to program Coverges slowly for large systems of equatos (large ) Not a useful stad aloe soluto method Key gredet to multgrd methods Examples Jacob Gauss-Sedel SOR Relaxato Soluto to Lear Systems page 4 Relaxato Methods (2) Basc Idea Update elemets of the soluto vector oe elemet at a tme Solve a odal equato by assumg other odal values are kow N Apply the fte volume method to get a S φ S a W φ W + a P φ P a E φ E a N φ N = b W P S E If k s the terato couter, we solve for φ k+1 P φ k+1 P b + a S φ k S a + a W φ k W + a Eφ k E + a Nφ k N P Values of φ W, φ E, φ N ad φ S are updated by applyg the teratve formula to the cells cetered at those eghbor odes. Relaxato Soluto to Lear Systems page 5

values are kow N Apply the fte volume method to get a S φ S a W φ W + a P φ P a E φ E a N φ N = b W P S E If k s the terato couter, we solve for φ k+1 P φ k+1 P b + a S φ k S a + a W

4 Matrx Notato (1) O a structured mesh the teror odes ca be umbered sequetally wth a mappg lke J= y +2 = x y = I 1+(J 2) x 3 = x +1 where x s the umber of cells the x drecto ad x s the value of the th depedet feld varable whch s located at the cells wth dces I ad J. The compass pot otato y 2 =1 =2 = x J=1 I=1 2 3 I= x +2 x a S φ S a W φ W + a P φ P a E φ E a N φ N = b becomes a S, x x a W, x 1 + a P, x a E, x +1 a W, x + x = b Relaxato Soluto to Lear Systems page 6 Matrx Notato (2) The atural orderg of odes o a two-dmesoal structured mesh yelds a coeffcet matrx wth fve dagoals. Coeffcets o the dagoals correspod to a P coeffcet of φ P x ad the four eghbor coeffcets a S, a W, a E,ada N. A = a S a W a P a E a N Relaxato Soluto to Lear Systems page 7

The compass pot otato y 2 =1 =2 = x J=1 I=1 2 3 I= x +2 x a S φ S a W φ W + a P φ P a E φ E a N φ N = b becomes a S, x x a W, x 1 + a P, x a E, x +1 a W, x + x = b Relaxato Soluto to Lear

5 Matrx Notato (3) I a ustructured mesh, the coeffcet matrx s also sparse, but the o-zero elemets of the coeffcet matrx do ot le alog dagoals. Ustructured Mesh No-zeros coeffcet matrx To smplfy the presetato, we ll use a structured mesh z = 361 Relaxato Soluto to Lear Systems page 8 Relaxato Methods Use relaxato methods to solve Ax = b Relaxato methods do ot work for all A, buttheydoworkforstadardfte-dfferece ad fte-volume dscretzatos of the Posso equato. Jacob Gauss-Sedel SOR Successve Over-relaxato Relaxato Soluto to Lear Systems page 9

90 0 10 20 30 40 50 60 70 80 90 z = 361 Relaxato Soluto to Lear Systems page 8 Relaxato Methods Use relaxato methods to solve Ax = b Relaxato methods

6 Jacob Iterato (1) A dscrete model of the Posso equato yelds a system of equatos that ca be wrtte Ax = b. The equato for the th odal value s a x = b =1 where s the total umber of odes the doma. Extract the dagoal term from the sum a x + a x = b =1 = The precedg equato wll ot be satsfed utl, the lmt as the umber of teratos creased to. Relaxato Soluto to Lear Systems page 10 Jacob Iterato (2) Solve for the ext guess x. x b a a x =1 = () I terms of the compass-pot otato for the two-dmesoal fte-volume method, the Equato () s φ (k+1) P b + a S φ (k) S a + a W φ (k) W + a Eφ (k) E P + a Nφ (k) N Equato () s ot really the soluto for x utl all the other x are kow. I other words, the value of x from Equato () wll be correct oly whe the system of equatos Ax = b s solved. If matrx A has favorable propertes, the value of x from Equato () wll be closer to the soluto tha the prevous guess at x. Relaxato Soluto to Lear Systems page 11

Relaxato Soluto to Lear Systems page 10 Jacob Iterato (2) Solve for the ext guess x.

7 Jacob Iterato (3) Let k be the terato couter. The Jacob terato formula s b a =1 = a x (k) =1,..., Ths formula mples that two copes of the x vector are mataed memory: oe for terato k ad oe for terato k +1. for k=1:maxt % -- For odes away from the boudares for =(x+1):-x xew() = (b() + as()*x(-x) + aw()*x(-1)... + a()*x(+x) + ae()*x(+1) ) / ap(); ed x = xew; ed Relaxato Soluto to Lear Systems page 12 Jacob Iterato (4) The Jacob teratve formula ca be wrtte as a matrx equato. Let D = dag(a 11,...,a ) ad B = D A the b a =1 = a x (k) s equvalet to = D 1 b + Bx (k) (1) where ad x (k) are the soluto vectors at terato k +1ad k, respectvely. Relaxato Soluto to Lear Systems page 13

for k=1:maxt % -- For odes away from the boudares for =(x+1):-x xew() = (b() + as()*x(-x) + aw()*x(-1).

8 Jacob Iterato (5) The matrx form of the terato ca be rewrtte = Hx (k) + d (2) where H = D 1 B d = D 1 b The matrx formulato s prmarly useful the aalyss of teratve methods ad toy mplemetatos Matlab. The egevalues of H determe the covergece rate of the teratos. Relaxato Soluto to Lear Systems page 14 Stoppg Crtera (1) Recall that the goal s to solve Ax = b where A s matrx, x ad b are elemet colum vectors. Iteratve Methods obta a sequece of approxmatos to the soluto x k k =0, 1, 2,... such that b Ax k 0 as k. The resdual r = b Ax s zero whe x s the soluto to Ax = b. Relaxato Soluto to Lear Systems page 15

Relaxato Soluto to Lear Systems page 14 Stoppg Crtera (1) Recall that the goal s to solve Ax = b where A s matrx, x ad b are elemet colum vectors.

9 Stoppg Crtera (2) For a teratve method the resdual at terato k s r k = b Ax k k =0, 1, 2,... If a teratve methods coverges, ther k 0 as k. Sce r s a vector, what does t mea that r k 0? Use a coveet vector orm to test whether r s small eough. Relaxato Soluto to Lear Systems page 16 Jacob Iterato (3) Soluto to a toy problem Soluto 2 T x + 2 T 2 y = o 0 x L, 0 y W 4 T =0o three boudares 3.5 uform q o fourth boudary Soluto s obtaed by Matlab code demojacob Relaxato Soluto to Lear Systems page 17

Use a coveet vector orm to test whether r s small eough.

10 Jacob Iterato (4) Covergece of Jacob teratos slow as mesh s refed x 8 16 x x r 1 / r Iterato Relaxato Soluto to Lear Systems page 18 Gauss-Sedel Iterato (1) I the Jacob method s obtaed from the froze values of x (k). The Gauss-Sedel method uses a smlar formula for updatg, but the ew value of s used as soo as t s avalable values from terato k y values from terato k+1 x Relaxato Soluto to Lear Systems page 19

Lear Systems page 18 Gauss-Sedel Iterato (1) I the Jacob method s obtaed from the froze values of x (k).

11 Gauss-Sedel Iterato (2) Replace odal values as sweep progresses through the doma for k=1:maxt ed % -- For odes away from the boudares for =(x+1):-x x() = (b() + as()*x(-x) + aw()*x(-1)... + a()*x(+x) + ae()*x(+1) ) / ap(); ed Relaxato Soluto to Lear Systems page 20 Gauss-Sedel Iterato (3) Cosder the sweep through the odal values order of creasg : 1 b 1 a 11 =2 a 1 x (k) 2 b 2 a 21 1 a 22 =3 a 1 x (k) 3 b 3 a 31 1 a 32 2 a b a a =1 =+1 =4 a 1 x (k) a 1 x (k) (3) Relaxato Soluto to Lear Systems page 21

.. + a()*x(+x) + ae()*x(+1) ) / ap(); ed Relaxato Soluto to Lear Systems page 20 Gauss-Sedel Iterato (3) Cosder the sweep

12 Gauss-Sedel Iterato (4) Coeffcets that multply ew values A x = b ew values old values = Coeffcets that multply old values Aga, let ad defe D = dag(a 11,...,a ) L = lower tragular part of A U = upper tragular part of A The Gauss-Sedel teratos correspod to the splttg of A A = D L U Note that A = LU,.e.,,L ad U are ot the usual factors of A. Relaxato Soluto to Lear Systems page 22 Equato (3) ca be rewrtte as Gauss-Sedel Iterato (5) or or a a 1 = b 1 =1 =1 a a = b =+1 =+1 a 1 x (k) a 1 x (k) D L = b + Ux (k) = (D L) = Ux (k) + b or =(D L) 1 (Ux (k) + b) (4) Relaxato Soluto to Lear Systems page 23

e.,,L ad U are ot the usual factors of A.

13 Gauss-Sedel Iterato (6) Note: The order whch the odes are processed the Gauss-Sedel teratos wll affect the path toward the soluto. I other words, for two dfferet ordergs of the odes, the termedate terates of x wl be dfferet. The Jacob terato does ot deped o the order whch the odes are umbered. Much of the classcal theory of relaxato methods assumes that the odes ad umbered atural order. Relaxato Soluto to Lear Systems page 24 SOR: Successve Over-Relaxato (1) Let ˆ be the updated value of x from the Gauss-Sedel formula, vz. ˆ b a a x (k) a < > (5) Istead of takg ˆ as the value of x at the k +1step, use = ωˆ + (1 ω)x (k) (6) where ω s a scalar weghtg factor. The basc step Equato (5) s called relaxato. ω =1for Gauss-Sedel ω < 1 causes the teratos to more slowly move toward the soluto. Ths s called uder-relaxato. ω > 1 causes the teratos to accelerate,.e. to be more aggressve. Ths s called over-relaxato. Relaxato Soluto to Lear Systems page 25

Much of the classcal theory of relaxato methods assumes that the odes ad umbered atural order.

14 Relaxato as a update We ca rewrte Equato (6) as = x (k) + ω ˆ x (k) (7) or where = x (k) + ω (8) =ˆ x (k) s the update to x terato k. ω s the drecto whch the soluto s chagg. s a magfed (ω > 1) orreduced(ω < 1) chage to the soluto. Relaxato Soluto to Lear Systems page 26 SOR: Successve Over-Relaxato (2) Substtute Equato (6) to Equato (5) = (1 ω)x k + ω b a a x (k) a < > (9) Multply through by a ad rearrage to get a + ω < a = (1 ω)a x k ω > a x (k) + ωb Use the splttg A = D L U ad the precedg equato becomes D ωl = (1 ω)dx (k) + ωux (k) + ωb (D ωl) = (1 ω)dx (k) + ωu x (k) + ωb =(D ωl) 1 (1 ω)dx (k) + ωu x (k) + ω(d ωl) 1 b Relaxato Soluto to Lear Systems page 27

Relaxato Soluto to Lear Systems page 26 SOR: Successve Over-Relaxato (2) Substtute Equato (6) to Equato (5) = (1 ω)x k + ω b a a x (k) a < > (9) Multply through by a

15 SOR: Successve Over-Relaxato (3) The update equato s =(D ωl) 1 (1 ω)dx (k) + ωu x (k) + ω(d ωl) 1 b Thus where = Hx (k) + d H =(D ωl) 1 (1 ω)dx (k) + ωu d = ω(d ωl) 1 b The speed of covergece depeds o the egevalues of H. Relaxato Soluto to Lear Systems page 28 O a coarse 8 8 mesh Compare Relaxato Covergece 10 4 Jacob Gauss Sedel SOR 10 3 r Iterato Relaxato Soluto to Lear Systems page 29

16 O a coarse mesh Compare Relaxato Covergece 10 5 Jacob Gauss Sedel SOR 10 4 r Iterato Relaxato Soluto to Lear Systems page 30

1 10 3 10 2 10 1 0 50 100 150 200 250 300

STATISTICAL PROPERTIES OF LEAST SQUARES ESTIMATORS. x, where. = y - ˆ " 1

STATISTICAL PROPERTIES OF LEAST SQUARES ESTIMATORS Recall Assumpto E(Y x) η 0 + η x (lear codtoal mea fucto) Data (x, y ), (x 2, y 2 ),, (x, y ) Least squares estmator ˆ E (Y x) ˆ " 0 + ˆ " x, where ˆ