Solving Linear Systems:Direct Methods

Transcription

1 Solving Linear Systems:Direct Methods A number of engineering problems or models can be formulated in terms of systems of equations Examples: Electrical Circuit Analysis, Radiative Heat Transfer (or equivalently, global diffuse illumination compuation), etc. Need a variety of computational techniques, depending on nature of system (sparse systems, for instance, benefit from iterative solutions). Systems can be very large, thousands of equations is not uncommon. Large systems require computational/numerical solutions. ITCS 4133/5133: Intro. to Numerical Methods 1 Solving Linear Systems

2 Example: Current in an Electrical Circuit ITCS 4133/5133: Intro. to Numerical Methods 2 Solving Linear Systems

3 Solving Linear Systems: Direct Methods Basic Gaussian Elimination (GE) Gaussian Elimination with Row Pivoting Tridiagonal Systems (Gauss-Thomas Method) Gauss-Jordan Elimination ITCS 4133/5133: Intro. to Numerical Methods 3 Solving Linear Systems

4 Linear Systems of Equations a 11 X 1 +a 12 X 2 + +a 1n Xn = C 1 a 21 X 1 +a 22 X 2 + +a 2n Xn = C a n1 X 1 +a n2 X 2 + +a nn Xn = C n where a ij s are the coefficients, X j s are the unknowns, and C i s are the known constants. Assumptions Number of unknowns equal the number of equations Equations are linearly independent ITCS 4133/5133: Intro. to Numerical Methods 4 Solving Linear Systems

5 Solution of Small Systems of Equations For small systems such as can be solved by substitution a 11 X 1 + a 12 X 2 = C 1 a 21 X 1 + a 22 X 2 = C 2 Complex engineering problems could contain thousands of equations. Substitution is not a feasible technique for large systems ITCS 4133/5133: Intro. to Numerical Methods 5 Solving Linear Systems

6 Linear System Classification With solutions Without solutions With infinite solutions ITCS 4133/5133: Intro. to Numerical Methods 6 Solving Linear Systems

7 Gaussian Elimination Forward Pass: convert the system of equations into an upper triangular matrix, Back Substitution: to solve the system. Permissible Operations Changing the order of the equations does not change the solution An equation may be scaled (multiplied/divided) by a non-zero constant without changing the solution May add 2 equations together and replace either one with the new equation, termed (linear combination) ITCS 4133/5133: Intro. to Numerical Methods 7 Solving Linear Systems

8 Gaussian Elimination:Matrix Representation a 11 a 12 a 1n X 1 a 21 a 22 a 2n X = a n1 a n2 a nn X n V 1 V 2. V n or, a 11 a 12 a 1n C 1 a 21 a 22 a 2n C a n1 a n2 a nn C n ITCS 4133/5133: Intro. to Numerical Methods 8 Solving Linear Systems

9 Gaussian Elimination:Forward Pass Results in the upper triangular matrix: 1 d 12 d 13 d 1n e1 0 1 d 23 d 2n e d 3n e e n ITCS 4133/5133: Intro. to Numerical Methods 9 Solving Linear Systems

10 Gaussian Elimination:Back Substitution X 1 + d 12 X 2 + d 13 X d 1,n 1 X n 1 + d 1n X n = e 1 X 2 + d 23 X d 2,n 1 X n 1 + d 2n X n = e 2 X d 3,n 1 X n 1 + d 3n X n = e 3 X n 2 + d n 2,n 1 X n 1 + d n 2,n X n X n 1 + d n 1,n X n X n = e n 2 = e n 1 = e n Implementation Diagonalize the matrix, resulting in the solution vector in the last column. ITCS 4133/5133: Intro. to Numerical Methods 10 Solving Linear Systems

11 Gaussian Elimination:Algorithm Forward Pass: Loop over each row i, making it the pivot row Normalize the pivot row, i (pivot element becomes 1) a ij = a ij /a ii, j = i + 1, i + 2,..., (n + 1) C i = C i /a ii a ii = 1 Loop over rows (i + 1) to n and reduce elements in each row. a kj = a kj a ki a ij, j = i,..., n C k = C k a ki C i, k = (i + 1),..., n ITCS 4133/5133: Intro. to Numerical Methods 11 Solving Linear Systems

12 Gaussian Elimination:Algorithm Forward Pass (Alternate) Uses a multiplier, without normalizing pivot element Input: a ij, b i, n Column: for k = 1 to n 1 for i = k + 1 to n Multiplier: m ik = a ik /a kk Row: for j = k + 1 to n a ij = a ij m ik a kj end b i = b i m ik b k end end Output: Upper Triangular Matrix ITCS 4133/5133: Intro. to Numerical Methods 12 Solving Linear Systems

13 Gaussian Elimination:Algorithm Back Substitution: For the last row, X n = Cn For rows (n 1) through 1, n X i = C i a ij X j, j = (i + 1),..., n and i = (n 1),..., 1 j=i+1 ITCS 4133/5133: Intro. to Numerical Methods 13 Solving Linear Systems

14 Gaussian Elimination: Algorithm ITCS 4133/5133: Intro. to Numerical Methods 14 Solving Linear Systems

15 Example 1:Electrical Circuit ITCS 4133/5133: Intro. to Numerical Methods 15 Solving Linear Systems

16 Roundoff Errors and Pivoting Truncation and Roundoff errors can occur when coefficients differ by orders of magnitude. To minimize errors, diagonal elements should contain the largest coefficient This can be accomplished by swapping rows during Gaussian elimination, a process called partial pivoting Zero Pivot Element: rearrange equations to prevent zero coefficients in the pivot element Equations must be linearly independent, else we have a singular coefficient matrix ITCS 4133/5133: Intro. to Numerical Methods 16 Solving Linear Systems

17 GE with Pivoting: Motivation Consider the following example: ITCS 4133/5133: Intro. to Numerical Methods 17 Solving Linear Systems

18 GE with Pivoting: Algorithm ITCS 4133/5133: Intro. to Numerical Methods 18 Solving Linear Systems

19 GE with Pivoting: Example ITCS 4133/5133: Intro. to Numerical Methods 19 Solving Linear Systems

20 Banded Systems Systems of equations with banded structure, with non-zero elements on the diagonal and a small number of off-diagonals. Such systems can be solved more efficiently. Example: Tridiagonal system d 1 x 1 a 1 x 2 = r 1 b 2 x 1 d 2 x 2 a 2 x 3 = r b n 1 x n 2 d n 1 x n 1 a n 1 x n = r n 1 b n x n 1 d n x n = r n Both memory and computational savings! ITCS 4133/5133: Intro. to Numerical Methods 20 Solving Linear Systems

21 Gauss-Thomas Method Representation:Vectors d (diagonal), a (above diagonal elements), b (below diagonal elements) Example: ITCS 4133/5133: Intro. to Numerical Methods 21 Solving Linear Systems

22 Gauss-Thomas Method (contd) ITCS 4133/5133: Intro. to Numerical Methods 22 Solving Linear Systems

23 Gauss-Thomas Method:Algorithm ITCS 4133/5133: Intro. to Numerical Methods 23 Solving Linear Systems

24 Gauss-Jordan Elimination In Gaussian elimination, we create upper triangular matrix, followed by back-substitution In Gauss-Jordan elimination, the input matrix is diagonalized in the forward pass, completing the solution More efficient when the same system is used to solve multiple rightside vectors. ITCS 4133/5133: Intro. to Numerical Methods 24 Solving Linear Systems

25 Matrix Condition Number Defined for non-singular matrices Defined in terms of the norm of the matrix, A Can be derived from vector norms: Matrix norms are defined as x 1 = x 1 + x x n x 2 = ( x x x n 2 ) 1/2 x = Max x i A 1 = max(over j) [sum of column j] A = max (over i) [sum of row i] A 2 = ( Max Eigen value of A H A) 1/2 ITCS 4133/5133: Intro. to Numerical Methods 25 Solving Linear Systems

26 Matrix Condition Number (contd) Ill-conditioned: Sensitive to small changes in parameters defining the problem. In general, condition number is defined as Spectral condition number: κ(a) = A A 1 κ(a) = A 2 A 1 2 Relative change in solution : depends on product of condition number and relative change in RHS vector (b). ITCS 4133/5133: Intro. to Numerical Methods 26 Solving Linear Systems

27 Analysis: Gaussian Elimination Why does Gaussian Elimination work? Equation forming a linear combination of two equations also passes through a point common (intersection) to both equations. Consider S 1 : a 0 + a 1 x a n x n = 0 T 1 : a 0 + b 1 x b n x n = 0 If r = (r 1, r 2,..., r n ) is a solution vector it also satisfies m 1 S 1 + m 2 T 1 Elimination produces a system solvable by back substitution. ITCS 4133/5133: Intro. to Numerical Methods 27 Solving Linear Systems

28 Analysis: Gaussian Elimination - Computational Complexity Measure Flops. Row calculations: multiplier calculation (1) + n multiplications and n additions + RHS update Repeat for (n 1) rows, resulting in (n+1)(n-1) multiplications and divisions, n(n-1) additions k=n 1 k=1 (n k + 1)(n k) = = k=n 1 k=1 k=n 1 k=1 = n3 3 n 3 n 2 2nk + k 2 + n k (n 2 + n) k=n 1 k=1 (2n + 1)k + k=n 1 ITCS 4133/5133: Intro. to Numerical Methods 28 Solving Linear Systems k=1 k 2

29 Analysis: GE with Pivoting Some form of row scaling can also be used. However, ill-conditioned matrices are not helped by scaling. Scaling is difficult to incorporate in general purpose packages. Computational complexity is O(n 3 ), so preferable to use more efficient methods. ITCS 4133/5133: Intro. to Numerical Methods 29 Solving Linear Systems

30 Analysis: GT Method (Tridiagonal Systems) First equation: 2 divisions (scaling the equation) Next (n-2) equations: 2 mults, 2 divisions. Last equation: 2 mults, 1 division Total (Forward Step): 5 + 4(n 2) Back Substitution: (n-1) multiplies. ITCS 4133/5133: Intro. to Numerical Methods 30 Solving Linear Systems