# Yousef Saad University of Minnesota Computer Science and Engineering. CRM Montreal - April 30, 2008

Save this PDF as:

Size: px
Start display at page:

## Transcription

1 A tutorial on: Iterative methods for Sparse Matrix Problems Yousef Saad University of Minnesota Computer Science and Engineering CRM Montreal - April 30, 2008

2 Outline Part 1 Sparse matrices and sparsity Basic iterative techniques Projection methods Krylov subspace methods Part 2 Preconditioned iterations Preconditioning techniques Part 3 Parallel implementations Multigrid methods Part 4 Eigenvalue problems Applications CRM April 30,

3 INTRODUCTION TO SPARSE MATRICES

4 Typical Problem: Physical Model Nonlinear PDEs Discretization Linearization (Newton) Sequence of Sparse Linear Systems Ax = b CRM April 30,

5 Introduction: Linear System Solvers Problem considered: Linear systems Ax = b Can view the problem from somewhat different angles: Discretized problem coming from a PDE An algebraic system of equations [ignore origin] Sometimes a system of equations where A is not explicitly available CRM April 30,

6 Direct sparse Solvers A x = b u = f + bc Iterative Methods Preconditioned Krylov General Purpose Specialized Fast Poisson Solvers Multigrid Methods CRM April 30,

7 Introduction: Linear System Solvers Much of recent work on solvers has focussed on: (1) Parallel implementation scalable performance (2) Improving Robustness, developing more general preconditioners CRM April 30,

8 A few observations Problems are getting harder for Sparse Direct methods (more 3-D models, much bigger problems,..) Problems are also getting difficult for iterative methods Cause: more complex models - away from Poisson Researchers in iterative methods are borrowing techniques from direct methods: preconditioners The inverse is also happening: Direct methods are being adapted for use as preconditioners CRM April 30,

9 What are sparse matrices? Common definition:..matrices that allow special techniques to take advantage of the large number of zero elements and the structure. A few applications of sparse matrices: Structural Engineering, Reservoir simulation, Electrical Networks, optimization problems,... Goals: Much less storage and work than dense computations. Observation: A 1 is usually dense, but L and U in the LU factorization may be reasonably sparse (if a good technique is used). CRM April 30,

10 Nonzero patterns of a few sparse matrices ARC130: Unsymmetric matrix from laser problem. a.r.curtis, oct 1974 SHERMAN5: fully implicit black oil simulator 16 by 23 by 3 grid, 3 unk

11 PORES3: Unsymmetric MATRIX FROM PORES BP_1000: UNSYMMETRIC BASIS FROM LP PROBLEM BP CRM April 30,

12 Two types of matrices: structured (e.g. Sherman5) and unstructured (e.g. BP 1000) Main goal of Sparse Matrix Techniques: To perform standard matrix computations economically i.e., without storing the zeros of the matrix. Example: To add two square dense matrices of size n requires O(n 2 ) operations. To add two sparse matrices A and B requires O(nnz(A) + nnz(b)) where nnz(x) = number of nonzero elements of a matrix X. For typical Finite Element /Finite difference matrices, number of nonzero elements is O(n). CRM April 30,

13 Graph Representations of Sparse Matrices Graph theory is a fundamental tool in sparse matrix techniques. Graph G = (V, E) of an n n matrix A defined by Vertices V = {1, 2,..., N}. Edges E = {(i, j) a ij 0}. Graph is undirected if matrix has symmetric structure: a ij 0 iff a ji 0. CRM April 30,

14 CRM April 30,

15 Example: Adjacency graph of: A =. Example: For any matrix A, what is the graph of A 2? [interpret in terms of paths in the graph of A] CRM April 30,

16 Direct versus iterative methods Background. Two types of methods: Direct methods : based on sparse Gaussian eimination, sparse Cholesky,.. Iterative methods: compute a sequence of iterates which converge to the solution - preconditioned Krylov methods.. Remark: These two classes of methods have always been in competition. 40 years ago solving a system with n = 10, 000 was a challenge Now you can solve this in < 1 sec. on a laptop. CRM April 30,

17 Sparse direct methods made huge gains in efficiency. As a result they are very competitive for 2-D problems. 3-D problems lead to more challenging systems [inherent to the underlying graph] Problems with many unknowns per grid point similar to 3-D problems Remarks: No robust black-box iterative solvers. Robustness often conflicts with efficiency However, situation improved in last decade Line between direct and iterative solvers blurring CRM April 30,

18 Direct Sparse Matrix Techniques Principle of sparse matrix techniques: Store only the nonzero elements of A. Try to minimize computations and (perhaps more importantly) storage. Difficulty in Gaussian elimination: Fill-in Trivial Example: A = CRM April 30,

19 Reorder equations and unknowns in order N, N 1,..., 1 A stays sparse during Gaussian eliminatin i.e., no fill-in. A = Finding the best ordering to minimize fill-in is NP-complete. A number of heuristics developed. Among the best known: Minimum degree ordering (Tinney Scheme 2) Nested Dissection Ordering. Approximate Minimal Degree... CRM April 30,

20 Reorderings and graphs Let π = {i 1,, i n } a permutation A π, = { a π(i),j }i,j=1,...,n = matrix A with its i-th row replaced by row number π(i). A,π = matrix A with its j-th column replaced by column π(j). Define P π = I π, = Permutation matrix Then: (1) Each row (column) of P π consists of zeros and exactly one 1 (2) A π, = P π A (3) P π Pπ T = I (4) A,π = APπ T CRM April 30,

21 Consider now: A = A π,π = P π AP T π Entry (i, j) in matrix A is exactly entry in position (π(i), π(j)) in A, i.e., (a ij = a π(i),π(j)) (i, j) E A (π(i), π(j)) E A General picture : i j π i π j CRM April 30,

22 Example A 9 9 arrow matrix and its adjacency graph CRM April 30,

23 Graph and matrix after permuting the nodes in reverse order CRM April 30,

24 Cuthill-McKee & reverse Cuthill-McKee A class of reordering techniques proceeds by levels in the graph. Related to Breadth First Search (BFS) traversal in graph theory. Idea of BFS is to visit the nodes by levels. Level 0 = level of starting node. Start with a node, visit its neighbors, then the (unmarked) neighbors of its neighbors, etc... CRM April 30,

25 Example: A B H I * *-----* * / / / / BFS from node A: / Level 0: A C* * D / Level 1: B, C; \ / Level 2: E, D, H; \ * K Level 3: I, K, E, F, G, H. F \ E * *----* G \ / \ / \ / * H CRM April 30,

26 Implementation using levels Algorithm BF S(G, v) by level sets Initialize S = {v}, seen = 1; Mark v; While seen < n Do S new = ; For each node v in S do For each unmarked w in adj(v) do Add w to S new ; Mark w; seen + +; S := S new CRM April 30,

27 A few properties of Breadth-First-Search If G is a connected undirected graph then each vertex will be visited once each edge will be inspected at least once Therefore, for a connected undirected graph, The cost of BFS is O( V + E ) Distance = level number; For each node v we have: min dist(s, v) = level number(v) = depth T (v) Several reordering algorithms are based on variants of Breadth- First-Search CRM April 30,

28 Cuthill McKee ordering Algorithm proceeds by levels. Same as BFS except: in each level, nodes are ordered by increasing degree Example A B E D G C F Level Nodes Deg. Order 0 A 2 A 1 B, C 4, 3 C, B 2 D, E, F 3, 4, 2 F, D, E 3 G 2 G CRM April 30,

29 ALGORITHM : 1 Cuthill Mc Kee ordering 0. Find an intial node for the traversal 1. Initialize S = {v}, seen = 1, π(seen) = v; Mark v; 2. While seen < n Do 3. S new = ; 4. For each node v, going from lowest to highest degree, Do: 5. π(+ + seen) = v; 6. For each unmarked w in adj(v) do 7. Add w to S new ; 8. Mark w; 9. EndDo 10. S := S new 11. EndDo 12. EndWhile CRM April 30,

30 Reverse Cuthill McKee ordering The Cuthill - Mc Kee ordering has a tendency to create small arrow matrices (going the wrong way): Origimal matrix CM ordering nz = nz = 377 CRM April 30,

31 Idea: Take the reverse ordering 0 RCM ordering nz = 377 Reverse Cuthill M Kee ordering (RCM). CRM April 30,

32 Nested Dissection ordering The idea of divide and conquer recursively divide graph in two using a separator CRM April 30,

33 Nested dissection for a small mesh Original Grid First dissection

34 Second Dissection Third Dissection CRM April 30,

35 Nested dissection: cost for a regular mesh In 2-D consider an n n problem, N = n 2 In 3-D consider an n n n problem, N = n 3 2-D 3-D space (fill) O(N log N) O(N 4/3 ) time (flops) O(N 3/2 ) O(N 2 ) Significant difference in complexity between 2-D and 3-D CRM April 30,

36 Ordering techniques for direct methods in practice In practice: Nested dissection (+ variants) is preferred for parallel processing Good implementations of Min. Degree algorithm work well in practice. Currently AMD and AMF are best known implementations/variants/ Best practical reordering algorithms usually combine Nested dissection and min. degree algorithms. CRM April 30,

37 BASIC RELAXATION METHODS

38 Basic Relaxation Schemes Relaxation schemes: based on the decomposition A = D E F - E - F D D = diag(a), E = strict lower part of A and F its strict upper part. Gauss-Seidel iteration for solving Ax = b: (D E)x (k+1) = F x (k) + b idea: correct the j-th component of the current approximate solution, j = 1, 2,..n, to zero the j th component of residual. CRM April 30,

39 Can also define a backward Gauss-Seidel Iteration: (D F )x (k+1) = Ex (k) + b and a Symmetric Gauss-Seidel Iteration: forward sweep followed by backward sweep. Over-relaxation is based on the decomposition: ωa = (D ωe) (ωf + (1 ω)d) successive overrelaxation, (SOR): (D ωe)x (k+1) = [ωf + (1 ω)d]x (k) + ωb CRM April 30,

40 Iteration matrices Jacobi, Gauss-Seidel, SOR, & SSOR iterations are of the form x (k+1) = Mx (k) + f M Jac = D 1 (E + F ) = I D 1 A M GS (A) = (D E) 1 F == I (D E) 1 A M SOR (A) = (D ωe) 1 (ωf +(1 ω)d) = I (ω 1 D E) 1 A M SSOR (A) = I (2ω 1 1)(ω 1 D F ) 1 D(ω 1 D E) 1 A = I ω(2ω 1)(D ωf ) 1 D(D ωe) 1 A CRM April 30,

41 General convergence result Consider the iteration: x (k+1) = Gx (k) + f (1) Assume that ρ(a) < 1. Then I G is non-singular and G has a fixed point. Iteration converges to a fixed point for any f and x (0). (2) If iteration converges for any f and x (0) then ρ(g) < 1. Example: Richardson s iteration x (k+1) = x (k) + α(b A (k) ) Assume Λ(A) R. When does the iteration converge? Jacobi and Gauss-Seidel converge for diagonal dominant A SOR converges for 0 < ω < 2 for SPD matrices CRM April 30,

42 An observation. Introduction to Preconditioning The iteration x (k+1) = Mx (k) + f is attempting to solve (I M)x = f. Since M is of the form M = I P 1 A this system can be rewritten as P 1 Ax = P 1 b where for SSOR, we have P SSOR = (D ωe)d 1 (D ωf ) referred to as the SSOR preconditioning matrix. In other words: Relaxation Scheme Preconditioned Fixed Point Iteration CRM April 30,

### 6. Cholesky factorization

6. Cholesky factorization EE103 (Fall 2011-12) triangular matrices forward and backward substitution the Cholesky factorization solving Ax = b with A positive definite inverse of a positive definite matrix

### Numerical Solution of Linear Systems

Numerical Solution of Linear Systems Chen Greif Department of Computer Science The University of British Columbia Vancouver B.C. Tel Aviv University December 17, 2008 Outline 1 Direct Solution Methods

### Numerical Methods I Solving Linear Systems: Sparse Matrices, Iterative Methods and Non-Square Systems

Numerical Methods I Solving Linear Systems: Sparse Matrices, Iterative Methods and Non-Square Systems Aleksandar Donev Courant Institute, NYU 1 donev@courant.nyu.edu 1 Course G63.2010.001 / G22.2420-001,

### DATA ANALYSIS II. Matrix Algorithms

DATA ANALYSIS II Matrix Algorithms Similarity Matrix Given a dataset D = {x i }, i=1,..,n consisting of n points in R d, let A denote the n n symmetric similarity matrix between the points, given as where

### 7. LU factorization. factor-solve method. LU factorization. solving Ax = b with A nonsingular. the inverse of a nonsingular matrix

7. LU factorization EE103 (Fall 2011-12) factor-solve method LU factorization solving Ax = b with A nonsingular the inverse of a nonsingular matrix LU factorization algorithm effect of rounding error sparse

### Iterative Solvers for Linear Systems

9th SimLab Course on Parallel Numerical Simulation, 4.10 8.10.2010 Iterative Solvers for Linear Systems Bernhard Gatzhammer Chair of Scientific Computing in Computer Science Technische Universität München

### Link-based Analysis on Large Graphs. Presented by Weiren Yu Mar 01, 2011

Link-based Analysis on Large Graphs Presented by Weiren Yu Mar 01, 2011 Overview 1 Introduction 2 Problem Definition 3 Optimization Techniques 4 Experimental Results 2 1. Introduction Many applications

### (57) (58) (59) (60) (61)

Module 4 : Solving Linear Algebraic Equations Section 5 : Iterative Solution Techniques 5 Iterative Solution Techniques By this approach, we start with some initial guess solution, say for solution and

### MATH 304 Linear Algebra Lecture 8: Inverse matrix (continued). Elementary matrices. Transpose of a matrix.

MATH 304 Linear Algebra Lecture 8: Inverse matrix (continued). Elementary matrices. Transpose of a matrix. Inverse matrix Definition. Let A be an n n matrix. The inverse of A is an n n matrix, denoted

### A note on fast approximate minimum degree orderings for symmetric matrices with some dense rows

NUMERICAL LINEAR ALGEBRA WITH APPLICATIONS Numer. Linear Algebra Appl. (2009) Published online in Wiley InterScience (www.interscience.wiley.com)..647 A note on fast approximate minimum degree orderings

### Parallel implementation of the deflated GMRES in the PETSc package

Work done while visiting the Jointlab @NCSA (10/26-11/24) 1/17 Parallel implementation of the deflated in the package in WAKAM 1, Joint work with Jocelyne ERHEL 1, William D. GROPP 2 1 INRIA Rennes, 2

### 9. Numerical linear algebra background

Convex Optimization Boyd & Vandenberghe 9. Numerical linear algebra background matrix structure and algorithm complexity solving linear equations with factored matrices LU, Cholesky, LDL T factorization

### AN INTRODUCTION TO NUMERICAL METHODS AND ANALYSIS

AN INTRODUCTION TO NUMERICAL METHODS AND ANALYSIS Revised Edition James Epperson Mathematical Reviews BICENTENNIAL 0, 1 8 0 7 z ewiley wu 2007 r71 BICENTENNIAL WILEY-INTERSCIENCE A John Wiley & Sons, Inc.,

### 10.2 ITERATIVE METHODS FOR SOLVING LINEAR SYSTEMS. The Jacobi Method

578 CHAPTER 1 NUMERICAL METHODS 1. ITERATIVE METHODS FOR SOLVING LINEAR SYSTEMS As a numerical technique, Gaussian elimination is rather unusual because it is direct. That is, a solution is obtained after

### Multigrid preconditioning for nonlinear (degenerate) parabolic equations with application to monument degradation

Multigrid preconditioning for nonlinear (degenerate) parabolic equations with application to monument degradation M. Donatelli 1 M. Semplice S. Serra-Capizzano 1 1 Department of Science and High Technology

### Elementary Matrices and The LU Factorization

lementary Matrices and The LU Factorization Definition: ny matrix obtained by performing a single elementary row operation (RO) on the identity (unit) matrix is called an elementary matrix. There are three

### FRIEDRICH-ALEXANDER-UNIVERSITÄT ERLANGEN-NÜRNBERG

FRIEDRICH-ALEXANDER-UNIVERSITÄT ERLANGEN-NÜRNBERG INSTITUT FÜR INFORMATIK (MATHEMATISCHE MASCHINEN UND DATENVERARBEITUNG) Lehrstuhl für Informatik 10 (Systemsimulation) Massively Parallel Multilevel Finite

### SOLVING LINEAR SYSTEMS

SOLVING LINEAR SYSTEMS Linear systems Ax = b occur widely in applied mathematics They occur as direct formulations of real world problems; but more often, they occur as a part of the numerical analysis

### Direct Methods for Solving Linear Systems. Matrix Factorization

Direct Methods for Solving Linear Systems Matrix Factorization Numerical Analysis (9th Edition) R L Burden & J D Faires Beamer Presentation Slides prepared by John Carroll Dublin City University c 2011

### Matrix Inverse and Determinants

DM554 Linear and Integer Programming Lecture 5 and Marco Chiarandini Department of Mathematics & Computer Science University of Southern Denmark Outline 1 2 3 4 and Cramer s rule 2 Outline 1 2 3 4 and

### Scalable Distributed Schur Complement Solvers for Internal and External Flow Computations on Many-Core Architectures

Scalable Distributed Schur Complement Solvers for Internal and External Flow Computations on Many-Core Architectures Dr.-Ing. Achim Basermann, Dr. Hans-Peter Kersken, Melven Zöllner** German Aerospace

### Solution of Linear Systems

Chapter 3 Solution of Linear Systems In this chapter we study algorithms for possibly the most commonly occurring problem in scientific computing, the solution of linear systems of equations. We start

### SECTIONS 1.5-1.6 NOTES ON GRAPH THEORY NOTATION AND ITS USE IN THE STUDY OF SPARSE SYMMETRIC MATRICES

SECIONS.5-.6 NOES ON GRPH HEORY NOION ND IS USE IN HE SUDY OF SPRSE SYMMERIC MRICES graph G ( X, E) consists of a finite set of nodes or vertices X and edges E. EXMPLE : road map of part of British Columbia

### (a) The transpose of a lower triangular matrix is upper triangular, and the transpose of an upper triangular matrix is lower triangular.

Theorem.7.: (Properties of Triangular Matrices) (a) The transpose of a lower triangular matrix is upper triangular, and the transpose of an upper triangular matrix is lower triangular. (b) The product

### MATH 240 Fall, Chapter 1: Linear Equations and Matrices

MATH 240 Fall, 2007 Chapter Summaries for Kolman / Hill, Elementary Linear Algebra, 9th Ed. written by Prof. J. Beachy Sections 1.1 1.5, 2.1 2.3, 4.2 4.9, 3.1 3.5, 5.3 5.5, 6.1 6.3, 6.5, 7.1 7.3 DEFINITIONS

### Fast Multipole Method for particle interactions: an open source parallel library component

Fast Multipole Method for particle interactions: an open source parallel library component F. A. Cruz 1,M.G.Knepley 2,andL.A.Barba 1 1 Department of Mathematics, University of Bristol, University Walk,

### Iterative Methods for Linear Systems

Iterative Methods for Linear Systems We present the basic concepts for three of the most fundamental and well known iterative techniques for solving a system of linear equations of the form Ax = b. Iterative

### HSL and its out-of-core solver

HSL and its out-of-core solver Jennifer A. Scott j.a.scott@rl.ac.uk Prague November 2006 p. 1/37 Sparse systems Problem: we wish to solve where A is Ax = b LARGE Informal definition: A is sparse if many

### Matrix Multiplication

Matrix Multiplication CPS343 Parallel and High Performance Computing Spring 2016 CPS343 (Parallel and HPC) Matrix Multiplication Spring 2016 1 / 32 Outline 1 Matrix operations Importance Dense and sparse

### TESLA Report 2003-03

TESLA Report 23-3 A multigrid based 3D space-charge routine in the tracking code GPT Gisela Pöplau, Ursula van Rienen, Marieke de Loos and Bas van der Geer Institute of General Electrical Engineering,

### Distributed block independent set algorithms and parallel multilevel ILU preconditioners

J Parallel Distrib Comput 65 (2005) 331 346 wwwelseviercom/locate/jpdc Distributed block independent set algorithms and parallel multilevel ILU preconditioners Chi Shen, Jun Zhang, Kai Wang Department

### 1 Introduction to Matrices

1 Introduction to Matrices In this section, important definitions and results from matrix algebra that are useful in regression analysis are introduced. While all statements below regarding the columns

### Lecture 3. Linear Programming. 3B1B Optimization Michaelmas 2015 A. Zisserman. Extreme solutions. Simplex method. Interior point method

Lecture 3 3B1B Optimization Michaelmas 2015 A. Zisserman Linear Programming Extreme solutions Simplex method Interior point method Integer programming and relaxation The Optimization Tree Linear Programming

### Social Media Mining. Graph Essentials

Graph Essentials Graph Basics Measures Graph and Essentials Metrics 2 2 Nodes and Edges A network is a graph nodes, actors, or vertices (plural of vertex) Connections, edges or ties Edge Node Measures

### Mesh Generation and Load Balancing

Mesh Generation and Load Balancing Stan Tomov Innovative Computing Laboratory Computer Science Department The University of Tennessee April 04, 2012 CS 594 04/04/2012 Slide 1 / 19 Outline Motivation Reliable

### 7 Gaussian Elimination and LU Factorization

7 Gaussian Elimination and LU Factorization In this final section on matrix factorization methods for solving Ax = b we want to take a closer look at Gaussian elimination (probably the best known method

### Graph Algorithms. Ananth Grama, Anshul Gupta, George Karypis, and Vipin Kumar

Graph Algorithms Ananth Grama, Anshul Gupta, George Karypis, and Vipin Kumar To accompany the text Introduction to Parallel Computing, Addison Wesley, 3. Topic Overview Definitions and Representation Minimum

### 5. Orthogonal matrices

L Vandenberghe EE133A (Spring 2016) 5 Orthogonal matrices matrices with orthonormal columns orthogonal matrices tall matrices with orthonormal columns complex matrices with orthonormal columns 5-1 Orthonormal

### General Framework for an Iterative Solution of Ax b. Jacobi s Method

2.6 Iterative Solutions of Linear Systems 143 2.6 Iterative Solutions of Linear Systems Consistent linear systems in real life are solved in one of two ways: by direct calculation (using a matrix factorization,

### Technical Report. Parallel iterative solution method for large sparse linear equation systems. Rashid Mehmood, Jon Crowcroft. Number 650.

Technical Report UCAM-CL-TR-65 ISSN 1476-2986 Number 65 Computer Laboratory Parallel iterative solution method for large sparse linear equation systems Rashid Mehmood, Jon Crowcroft October 25 15 JJ Thomson

### P013 INTRODUCING A NEW GENERATION OF RESERVOIR SIMULATION SOFTWARE

1 P013 INTRODUCING A NEW GENERATION OF RESERVOIR SIMULATION SOFTWARE JEAN-MARC GRATIEN, JEAN-FRANÇOIS MAGRAS, PHILIPPE QUANDALLE, OLIVIER RICOIS 1&4, av. Bois-Préau. 92852 Rueil Malmaison Cedex. France

### Linear Algebra Review Part 2: Ax=b

Linear Algebra Review Part 2: Ax=b Edwin Olson University of Michigan The Three-Day Plan Geometry of Linear Algebra Vectors, matrices, basic operations, lines, planes, homogeneous coordinates, transformations

### Factorization Theorems

Chapter 7 Factorization Theorems This chapter highlights a few of the many factorization theorems for matrices While some factorization results are relatively direct, others are iterative While some factorization

### Home Page. Data Structures. Title Page. Page 1 of 24. Go Back. Full Screen. Close. Quit

Data Structures Page 1 of 24 A.1. Arrays (Vectors) n-element vector start address + ielementsize 0 +1 +2 +3 +4... +n-1 start address continuous memory block static, if size is known at compile time dynamic,

### Scientific Computing: An Introductory Survey

Scientific Computing: An Introductory Survey Chapter 3 Linear Least Squares Prof. Michael T. Heath Department of Computer Science University of Illinois at Urbana-Champaign Copyright c 2002. Reproduction

### Eigenvalues of C SMS (16) 1 A. Eigenvalues of A (32) 1 A (16,32) 1 A

Negative Results for Multilevel Preconditioners in Image Deblurring C. R. Vogel Department of Mathematical Sciences Montana State University Bozeman, MT 59717-0240 USA Abstract. A one-dimensional deconvolution

### P164 Tomographic Velocity Model Building Using Iterative Eigendecomposition

P164 Tomographic Velocity Model Building Using Iterative Eigendecomposition K. Osypov* (WesternGeco), D. Nichols (WesternGeco), M. Woodward (WesternGeco) & C.E. Yarman (WesternGeco) SUMMARY Tomographic

### A matrix-free preconditioner for sparse symmetric positive definite systems and least square problems

A matrix-free preconditioner for sparse symmetric positive definite systems and least square problems Stefania Bellavia Dipartimento di Ingegneria Industriale Università degli Studi di Firenze Joint work

### CS3220 Lecture Notes: QR factorization and orthogonal transformations

CS3220 Lecture Notes: QR factorization and orthogonal transformations Steve Marschner Cornell University 11 March 2009 In this lecture I ll talk about orthogonal matrices and their properties, discuss

### Diagonal, Symmetric and Triangular Matrices

Contents 1 Diagonal, Symmetric Triangular Matrices 2 Diagonal Matrices 2.1 Products, Powers Inverses of Diagonal Matrices 2.1.1 Theorem (Powers of Matrices) 2.2 Multiplying Matrices on the Left Right by

### Computer Science Master's Project Report Rensselaer Polytechnic Institute Troy, NY 12180

Computer Science Master's Project Report Rensselaer Polytechnic Institute Troy, NY 12180 Development of A Two-Level Iterative Computational Method for Solution of the Franklin Approximation Algorithm for

### AN INTERFACE STRIP PRECONDITIONER FOR DOMAIN DECOMPOSITION METHODS

AN INTERFACE STRIP PRECONDITIONER FOR DOMAIN DECOMPOSITION METHODS by M. Storti, L. Dalcín, R. Paz Centro Internacional de Métodos Numéricos en Ingeniería - CIMEC INTEC, (CONICET-UNL), Santa Fe, Argentina

### Preconditioning Sparse Matrices for Computing Eigenvalues and Solving Linear Systems of Equations. Tzu-Yi Chen

Preconditioning Sparse Matrices for Computing Eigenvalues and Solving Linear Systems of Equations by Tzu-Yi Chen B.S. (Massachusetts Institute of Technology) 1995 B.S. (Massachusetts Institute of Technology)

### Performance of Dynamic Load Balancing Algorithms for Unstructured Mesh Calculations

Performance of Dynamic Load Balancing Algorithms for Unstructured Mesh Calculations Roy D. Williams, 1990 Presented by Chris Eldred Outline Summary Finite Element Solver Load Balancing Results Types Conclusions

### Abstract: We describe the beautiful LU factorization of a square matrix (or how to write Gaussian elimination in terms of matrix multiplication).

MAT 2 (Badger, Spring 202) LU Factorization Selected Notes September 2, 202 Abstract: We describe the beautiful LU factorization of a square matrix (or how to write Gaussian elimination in terms of matrix

### Solving Linear Systems of Equations. Gerald Recktenwald Portland State University Mechanical Engineering Department gerry@me.pdx.

Solving Linear Systems of Equations Gerald Recktenwald Portland State University Mechanical Engineering Department gerry@me.pdx.edu These slides are a supplement to the book Numerical Methods with Matlab:

### We seek a factorization of a square matrix A into the product of two matrices which yields an

LU Decompositions We seek a factorization of a square matrix A into the product of two matrices which yields an efficient method for solving the system where A is the coefficient matrix, x is our variable

### Approximation Algorithms

Approximation Algorithms or: How I Learned to Stop Worrying and Deal with NP-Completeness Ong Jit Sheng, Jonathan (A0073924B) March, 2012 Overview Key Results (I) General techniques: Greedy algorithms

### Mathematics Notes for Class 12 chapter 3. Matrices

1 P a g e Mathematics Notes for Class 12 chapter 3. Matrices A matrix is a rectangular arrangement of numbers (real or complex) which may be represented as matrix is enclosed by [ ] or ( ) or Compact form

### Modification of the Minimum-Degree Algorithm by Multiple Elimination

Modification of the Minimum-Degree Algorithm by Multiple Elimination JOSEPH W. H. LIU York University The most widely used ordering scheme to reduce fills and operations in sparse matrix computation is

### Network (Tree) Topology Inference Based on Prüfer Sequence

Network (Tree) Topology Inference Based on Prüfer Sequence C. Vanniarajan and Kamala Krithivasan Department of Computer Science and Engineering Indian Institute of Technology Madras Chennai 600036 vanniarajanc@hcl.in,

### = [a ij ] 2 3. Square matrix A square matrix is one that has equal number of rows and columns, that is n = m. Some examples of square matrices are

This document deals with the fundamentals of matrix algebra and is adapted from B.C. Kuo, Linear Networks and Systems, McGraw Hill, 1967. It is presented here for educational purposes. 1 Introduction In

### SHARP BOUNDS FOR THE SUM OF THE SQUARES OF THE DEGREES OF A GRAPH

31 Kragujevac J. Math. 25 (2003) 31 49. SHARP BOUNDS FOR THE SUM OF THE SQUARES OF THE DEGREES OF A GRAPH Kinkar Ch. Das Department of Mathematics, Indian Institute of Technology, Kharagpur 721302, W.B.,

### APPLICATIONS OF MATRICES. Adj A is nothing but the transpose of the co-factor matrix [A ij ] of A.

APPLICATIONS OF MATRICES ADJOINT: Let A = [a ij ] be a square matrix of order n. Let Aij be the co-factor of a ij. Then the n th order matrix [A ij ] T is called the adjoint of A. It is denoted by adj

### (67902) Topics in Theory and Complexity Nov 2, 2006. Lecture 7

(67902) Topics in Theory and Complexity Nov 2, 2006 Lecturer: Irit Dinur Lecture 7 Scribe: Rani Lekach 1 Lecture overview This Lecture consists of two parts In the first part we will refresh the definition

### System Interconnect Architectures. Goals and Analysis. Network Properties and Routing. Terminology - 2. Terminology - 1

System Interconnect Architectures CSCI 8150 Advanced Computer Architecture Hwang, Chapter 2 Program and Network Properties 2.4 System Interconnect Architectures Direct networks for static connections Indirect

### DETERMINANTS. b 2. x 2

DETERMINANTS 1 Systems of two equations in two unknowns A system of two equations in two unknowns has the form a 11 x 1 + a 12 x 2 = b 1 a 21 x 1 + a 22 x 2 = b 2 This can be written more concisely in

### Approximating the Minimum Chain Completion problem

Approximating the Minimum Chain Completion problem Tomás Feder Heikki Mannila Evimaria Terzi Abstract A bipartite graph G = (U, V, E) is a chain graph [9] if there is a bijection π : {1,..., U } U such

### Computational Optical Imaging - Optique Numerique. -- Deconvolution --

Computational Optical Imaging - Optique Numerique -- Deconvolution -- Winter 2014 Ivo Ihrke Deconvolution Ivo Ihrke Outline Deconvolution Theory example 1D deconvolution Fourier method Algebraic method

### 5 INTEGER LINEAR PROGRAMMING (ILP) E. Amaldi Fondamenti di R.O. Politecnico di Milano 1

5 INTEGER LINEAR PROGRAMMING (ILP) E. Amaldi Fondamenti di R.O. Politecnico di Milano 1 General Integer Linear Program: (ILP) min c T x Ax b x 0 integer Assumption: A, b integer The integrality condition

### UNIT 2 MATRICES - I 2.0 INTRODUCTION. Structure

UNIT 2 MATRICES - I Matrices - I Structure 2.0 Introduction 2.1 Objectives 2.2 Matrices 2.3 Operation on Matrices 2.4 Invertible Matrices 2.5 Systems of Linear Equations 2.6 Answers to Check Your Progress

### SECTION 8.3: THE INVERSE OF A SQUARE MATRIX

(Section 8.3: The Inverse of a Square Matrix) 8.47 SECTION 8.3: THE INVERSE OF A SQUARE MATRIX PART A: (REVIEW) THE INVERSE OF A REAL NUMBER If a is a nonzero real number, then aa 1 = a 1 a = 1. a 1, or

### MATH 551 - APPLIED MATRIX THEORY

MATH 55 - APPLIED MATRIX THEORY FINAL TEST: SAMPLE with SOLUTIONS (25 points NAME: PROBLEM (3 points A web of 5 pages is described by a directed graph whose matrix is given by A Do the following ( points

### LINEAR SYSTEMS. Consider the following example of a linear system:

LINEAR SYSTEMS Consider the following example of a linear system: Its unique solution is x +2x 2 +3x 3 = 5 x + x 3 = 3 3x + x 2 +3x 3 = 3 x =, x 2 =0, x 3 = 2 In general we want to solve n equations in

### 10. Graph Matrices Incidence Matrix

10 Graph Matrices Since a graph is completely determined by specifying either its adjacency structure or its incidence structure, these specifications provide far more efficient ways of representing a

### ALGEBRAIC EIGENVALUE PROBLEM

ALGEBRAIC EIGENVALUE PROBLEM BY J. H. WILKINSON, M.A. (Cantab.), Sc.D. Technische Universes! Dsrmstedt FACHBEREICH (NFORMATiK BIBL1OTHEK Sachgebieto:. Standort: CLARENDON PRESS OXFORD 1965 Contents 1.

### 15.062 Data Mining: Algorithms and Applications Matrix Math Review

.6 Data Mining: Algorithms and Applications Matrix Math Review The purpose of this document is to give a brief review of selected linear algebra concepts that will be useful for the course and to develop

### Scientific Computing: An Introductory Survey

Scientific Computing: An Introductory Survey Chapter 10 Boundary Value Problems for Ordinary Differential Equations Prof. Michael T. Heath Department of Computer Science University of Illinois at Urbana-Champaign

### 1 Determinants and the Solvability of Linear Systems

1 Determinants and the Solvability of Linear Systems In the last section we learned how to use Gaussian elimination to solve linear systems of n equations in n unknowns The section completely side-stepped

### On fast factorization pivoting methods for sparse symmetric indefinite systems

On fast factorization pivoting methods for sparse symmetric indefinite systems by Olaf Schenk 1, and Klaus Gärtner 2 Technical Report CS-2004-004 Department of Computer Science, University of Basel Submitted

### Algebraic and Combinatorial Circuits

C H A P T E R Algebraic and Combinatorial Circuits Algebraic circuits combine operations drawn from an algebraic system. In this chapter we develop algebraic and combinatorial circuits for a variety of

### ESSENTIAL COMPUTATIONAL FLUID DYNAMICS

ESSENTIAL COMPUTATIONAL FLUID DYNAMICS Oleg Zikanov WILEY JOHN WILEY & SONS, INC. CONTENTS PREFACE xv 1 What Is CFD? 1 1.1. Introduction / 1 1.2. Brief History of CFD / 4 1.3. Outline of the Book / 6 References

### RESEARCH OF MULTICORE-BASED PARALLEL GABP ALGORITHM WITH DYNAMIC LOAD-BALANCE

INTERNATIONAL JOURNAL OF NUMERICAL ANALYSIS AND MODELING, SERIES B Volume 5, Number 1-2, Pages 123 135 c 2014 Institute for Scientific Computing and Information RESEARCH OF MULTICORE-BASED PARALLEL GABP

### Lecture 12: Partitioning and Load Balancing

Lecture 12: Partitioning and Load Balancing G63.2011.002/G22.2945.001 November 16, 2010 thanks to Schloegel,Karypis and Kumar survey paper and Zoltan website for many of today s slides and pictures Partitioning

### POISSON AND LAPLACE EQUATIONS. Charles R. O Neill. School of Mechanical and Aerospace Engineering. Oklahoma State University. Stillwater, OK 74078

21 ELLIPTICAL PARTIAL DIFFERENTIAL EQUATIONS: POISSON AND LAPLACE EQUATIONS Charles R. O Neill School of Mechanical and Aerospace Engineering Oklahoma State University Stillwater, OK 74078 2nd Computer

### PARDISO. User Guide Version 5.0.0

P a r a l l e l S p a r s e D i r e c t A n d M u l t i - R e c u r s i v e I t e r a t i v e L i n e a r S o l v e r s PARDISO User Guide Version 5.0.0 (Updated February 07, 2014) O l a f S c h e n k

### Asking Hard Graph Questions. Paul Burkhardt. February 3, 2014

Beyond Watson: Predictive Analytics and Big Data U.S. National Security Agency Research Directorate - R6 Technical Report February 3, 2014 300 years before Watson there was Euler! The first (Jeopardy!)

### Practical Guide to the Simplex Method of Linear Programming

Practical Guide to the Simplex Method of Linear Programming Marcel Oliver Revised: April, 0 The basic steps of the simplex algorithm Step : Write the linear programming problem in standard form Linear

Facts About Eigenvalues By Dr David Butler Definitions Suppose A is an n n matrix An eigenvalue of A is a number λ such that Av = λv for some nonzero vector v An eigenvector of A is a nonzero vector v

### High Performance Computing for Operation Research

High Performance Computing for Operation Research IEF - Paris Sud University claude.tadonki@u-psud.fr INRIA-Alchemy seminar, Thursday March 17 Research topics Fundamental Aspects of Algorithms and Complexity

### Logistic Regression. Jia Li. Department of Statistics The Pennsylvania State University. Logistic Regression

Logistic Regression Department of Statistics The Pennsylvania State University Email: jiali@stat.psu.edu Logistic Regression Preserve linear classification boundaries. By the Bayes rule: Ĝ(x) = arg max

### AN ALGORITHM FOR DETERMINING WHETHER A GIVEN BINARY MATROID IS GRAPHIC

AN ALGORITHM FOR DETERMINING WHETHER A GIVEN BINARY MATROID IS GRAPHIC W. T. TUTTE. Introduction. In a recent series of papers [l-4] on graphs and matroids I used definitions equivalent to the following.

### Using row reduction to calculate the inverse and the determinant of a square matrix

Using row reduction to calculate the inverse and the determinant of a square matrix Notes for MATH 0290 Honors by Prof. Anna Vainchtein 1 Inverse of a square matrix An n n square matrix A is called invertible

### Eigenvalues and eigenvectors of a matrix

Eigenvalues and eigenvectors of a matrix Definition: If A is an n n matrix and there exists a real number λ and a non-zero column vector V such that AV = λv then λ is called an eigenvalue of A and V is

### 1 Determinants. Definition 1

Determinants The determinant of a square matrix is a value in R assigned to the matrix, it characterizes matrices which are invertible (det 0) and is related to the volume of a parallelpiped described

### Cofactor Expansion: Cramer s Rule

Cofactor Expansion: Cramer s Rule MATH 322, Linear Algebra I J. Robert Buchanan Department of Mathematics Spring 2015 Introduction Today we will focus on developing: an efficient method for calculating

### Similarity and Diagonalization. Similar Matrices

MATH022 Linear Algebra Brief lecture notes 48 Similarity and Diagonalization Similar Matrices Let A and B be n n matrices. We say that A is similar to B if there is an invertible n n matrix P such that

### 4. MATRICES Matrices

4. MATRICES 170 4. Matrices 4.1. Definitions. Definition 4.1.1. A matrix is a rectangular array of numbers. A matrix with m rows and n columns is said to have dimension m n and may be represented as follows: