Direct methods for sparse matrices

Transcription

1 Direct methods for sparse matrices Iain S. Duff STFC Rutherford Appleton Laboratory Oxfordshire, UK. and CERFACS, Toulouse, France CEA-EDF-INDRIA Schools. Sophia Antipolis. March 30 - April p.1/50

2 Introduction Lecture 1 Sparse matrices and Gaussian elimination CEA-EDF-INDRIA Schools. Sophia Antipolis. March 30 - April p.2/50

3 Introduction Wish to solve Ax = b where A is LARGE and S P A R S E CEA-EDF-INDRIA Schools. Sophia Antipolis. March 30 - April p.3/50

4 Introduction By LARGE we mean matrices of large order n. This is a function of time. t n(t) , , , , , ,000, ,000,000 The meaning of sparse is not so simple SPARSE... NUMBER ENTRIES kn k 2 - logn CEA-EDF-INDRIA Schools. Sophia Antipolis. March 30 - April p.4/50

5 Introduction NUMERICAL APPLICATIONS Stiff ODEs... BDF... Sparse Jacobian Linear Programming... simplex... interior point Optimization/Nonlinear Equations Elliptic Partial Differential equations Eigensystem Solution Two Point Boundary Value Problems Least Squares Calculations CEA-EDF-INDRIA Schools. Sophia Antipolis. March 30 - April p.5/50

6 Introduction APPLICATION AREAS Physics Chemistry Civil engineering Electrical engineering Geography Demography Economics Behavioural sciences Politics Psychology Business administration Operations research CFD Lattice gauge Atomic spectra Quantum chemistry Chemical engineering Structural analysis Power systems Circuit simulation Device simulation Geodesy Migration Economic modelling Industrial relations Trading Social dominance Bureaucracy Linear Programming CEA-EDF-INDRIA Schools. Sophia Antipolis. March 30 - April p.6/50

7 Sparse matrices THERE FOLLOWS PICTURES OF SPARSE MATRICES FROM VARIOUS APPLICATIONS This is done to illustrate different structures for sparse matrices CEA-EDF-INDRIA Schools. Sophia Antipolis. March 30 - April p.7/50

8 Sparse matrices Thermal Simulation; SHERMAN2 CEA-EDF-INDRIA Schools. Sophia Antipolis. March 30 - April p.8/50

9 Sparse matrices Weather Matrix; FS CEA-EDF-INDRIA Schools. Sophia Antipolis. March 30 - April p.9/50

10 Sparse matrices Dynamic Calculation in Structures; BCSSTM13 CEA-EDF-INDRIA Schools. Sophia Antipolis. March 30 - April p.10/50

11 Sparse matrices Power Systems; BCSPWR07 CEA-EDF-INDRIA Schools. Sophia Antipolis. March 30 - April p.11/50

12 Sparse matrices Simulation of Computing Systems; GRE 1107 CEA-EDF-INDRIA Schools. Sophia Antipolis. March 30 - April p.12/50

13 Sparse matrices Chemical Engineering; WEST0381 CEA-EDF-INDRIA Schools. Sophia Antipolis. March 30 - April p.13/50

14 Sparse matrices Economic Modelling; ORANI678 CEA-EDF-INDRIA Schools. Sophia Antipolis. March 30 - April p.14/50

15 Sparse matrices STANDARD SETS OF SPARSE MATRICES Original set Harwell-Boeing Sparse Matrix Collection. Available through the web page Extended set of test matrices available from: Tim Davis and Matrix market CEA-EDF-INDRIA Schools. Sophia Antipolis. March 30 - April p.15/50

16 Sparse matrices Large and increasing collection maintained by the GRID-TLSE Project Also see the report: The Rutherford-Boeing Sparse Matrix Collection. Iain S. Duff, Roger G. Grimes and John G. Lewis. Report RAL-TR , Rutherford Appleton Laboratory. CEA-EDF-INDRIA Schools. Sophia Antipolis. March 30 - April p.16/50

17 Solving sparse systems Ax = b has solution x = A 1 b BUT This is notational only. Do not use or even think of inverse of A. For sparse A, A 1 is usually dense. Examples are: Tridiagonal and arrowhead CEA-EDF-INDRIA Schools. Sophia Antipolis. March 30 - April p.17/50

18 Solving sparse systems DIRECT METHOD Gaussian Elimination PAQ LU Permutations P and Q chosen to preserve sparsity and maintain stability L : Lower triangular (sparse) U : Upper triangular (sparse) SOLVE: Ax = b by Ly = Pb then UQ T x = y CEA-EDF-INDRIA Schools. Sophia Antipolis. March 30 - April p.18/50

19 Direct methods DIRECT METHODS Easy to package High accuracy Method of choice in many applications Not dramatically affected by conditioning Reasonably independent of structure However High time and storage requirement Typically limited to n So Use on subproblem Use as preconditioner CEA-EDF-INDRIA Schools. Sophia Antipolis. March 30 - April p.19/50

20 Hybrid methods COMBINING DIRECT AND ITERATIVE METHODS (can be thought of as sophisticated preconditioning) Multigrid Using direct method as coarse grid solver. Domain Decomposition Using direct method on local subdomains and direct preconditioner on interface. Block Iterative Methods Direct solver on sub-blocks. Partial factorization as preconditioner. Factorization of nearby problem as a preconditioner. CEA-EDF-INDRIA Schools. Sophia Antipolis. March 30 - April p.20/50

21 Matrix storage schemes For efficient solution of sparse equations we must Only store nonzeros (or exceptionally a few zeros also) Only perform arithmetic with nonzeros Preserve sparsity during computation CEA-EDF-INDRIA Schools. Sophia Antipolis. March 30 - April p.21/50

22 Direct methods Direct Sparse Matrix Codes A Paradigm for Large Scale Scientific Computing 1. Much non-numeric processing 2. A significant amount of data handling 3. Sometimes out-of-core 4. Sometimes much time outwith innermost loop 5. Innermost loop sometimes quite complicated CEA-EDF-INDRIA Schools. Sophia Antipolis. March 30 - April p.22/50

23 Direct codes TYPICAL INNERMOST LOOP DO 590 JJ = J1, J2 J = ICN (JJ) IF (IQ(J). GT.0) GO TO 590 IOP = IOP + 1 PIVROW = IJPOS - IQ (J) A(JJ) = A(JJ) + AU x A (PIVROW) IF (LBIG) BIG = DMAXI (DABS(A(JJ)), BIG) IF (DABS(A(JJ)). LT. TOL) IDROP = IDROP + 1 ICN (PIVROW) = ICN (PIVROW) 590 CONTINUE CEA-EDF-INDRIA Schools. Sophia Antipolis. March 30 - April p.23/50

24 Addition of sparse vectors a + b a j 1 j 2 j 3 j 4 a 1 a 2 a 3 a 4 + b j 5 b 1 j 2 b 2 j 6 b 3 j 4 b 4 a is stored as a 1 a 2 a 3 a 4 j 1 j 2 j 3 j 4 b is stored as b 1 b 2 b 3 b 4 j 5 j 2 j 6 j 4 CEA-EDF-INDRIA Schools. Sophia Antipolis. March 30 - April p.24/50

25 Addition of sparse vectors a + b Expand a into a full vector W then: Scan index list of vector b Add value of b entry to each component of W Merge index lists from a and b Collect result vector from components of W corresponding to this merged list CEA-EDF-INDRIA Schools. Sophia Antipolis. March 30 - April p.25/50

26 Addition of sparse vectors Benefits of scheme No scans of length n No need to hold indices in order No need to scan two sparse vectors in parallel However, It requires a vector of length n In identifying fill-in (that is, computing pattern of vector a + b), there can be a confusion between explicitly held zeros and entries not in the sparsity pattern CEA-EDF-INDRIA Schools. Sophia Antipolis. March 30 - April p.26/50

27 Addition of sparse vectors EXERCISE Design an algorithm to add two sparse vectors without requiring a real vector of length n and which does not confuse explicitly held zeros with entries not in the sparsity pattern. The algorithm should also be efficiently adapted to the case where different multiples of a single vector are added to a sequence of different vectors (as is the case in Gaussian elimination). You are allowed temporarily to have two copies of the pivot row and you can assume that there exists an integer vector of length n all of whose entries are positive. However, any work in restoring this vector to its original form must be included. You should show that this algorithm also satisfies the requirements that the vectors need not be ordered, and that scans of length n are avoided. You should show the complexity of this algorithm, probably by coding it and running on vectors of different length and density. The algorithm should be independent of n and linear in the number of nonzero entries. CEA-EDF-INDRIA Schools. Sophia Antipolis. March 30 - April p.27/50

28 The ordering problem In using Gaussian elimination to solve Ax = b, we perform the decomposition PAQ = LU where P and Q (P if A is symmetric) are permutation matrices chosen to: 1. preserve sparsity and reduce work in decomposition and solution 2. enable use of efficient data structures 3. ensure stability 4. take advantage of structure CEA-EDF-INDRIA Schools. Sophia Antipolis. March 30 - April p.28/50

29 Ordering Benefits of Sparsity Ordering on Matrix of Order 2021 with 7353 entries Total storage Flops Time (secs) Procedure (Kwords) (10 6 ) CRAY J90 Treating system as dense Storing and operating only on nonzero entries Using sparsity pivoting CEA-EDF-INDRIA Schools. Sophia Antipolis. March 30 - April p.29/50

30 Ordering Two categories... LOCAL Ordering and GLOBAL Ordering CEA-EDF-INDRIA Schools. Sophia Antipolis. March 30 - April p.30/50

31 Ordering for symmetric matrices L/U L/U CEA-EDF-INDRIA Schools. Sophia Antipolis. March 30 - April p.31/50

32 Minimum degree (Tinney S ) At each stage choose diagonal entry with least number of entries in row of reduced matrix. Using this to minimize the fill-in is NP-complete Usually it is a good heuristic and gives a good ordering. However, being a local algorithm, it does not take full account of the global underlying structure of the problem. Its performance can be unpredictable and it is sensitive to how ties are broken. The code for efficient implementation can be remarkably complicated. Often the most time consuming part of the algorithm is the degree update Approximate Minimum Degree (AMD). CEA-EDF-INDRIA Schools. Sophia Antipolis. March 30 - April p.32/50

33 Local ordering for unsymmetric matrices v v v v v v r i = 3, c j = 4 Minimize (r i 1) (c j 1) MARKOWITZ ORDERING (Markowitz 1957) Choose nonzero satisfying a numerical criterion with best Markowitz count. CEA-EDF-INDRIA Schools. Sophia Antipolis. March 30 - April p.33/50

34 Threshold pivoting for stability a ij is admissible as a pivot if a ij umax k a kj 0 < u 1 So choose as pivot at each stage the entry with lowest Markowitz cost satisfying the above inequality. Called threshold pivoting CEA-EDF-INDRIA Schools. Sophia Antipolis. March 30 - April p.34/50

35 Growth at one step is bounded by max i and overall growth by max i Threshold pivoting for stability a (k+1) ij (1 + 1/u)max a (k) i ij a (k) ij (1 + 1/u)p j max a ij i where p j is the number of off-diagonal entries in the jth column of U. If then H = ˆLÛ A h ij 5.01ɛnmax k a (k) ij This means that through u, the threshold parameter, we can directly influence backward error in factorization. CEA-EDF-INDRIA Schools. Sophia Antipolis. March 30 - April p.35/50

36 Threshold pivoting for stability Threshold Entries in factors Error in solution Effect of changing threshold parameter. Matrix is order 541 with 4285 entries. CEA-EDF-INDRIA Schools. Sophia Antipolis. March 30 - April p.36/50

37 Efficient implementation of Markowitz U L A (k) If you search all of A (k) each time, the complexity is O(nτ) O(n 2 ). Want O(1) search every time... Search rows/columns in order of increasing sparsity Can limit search to small number of rows/columns CEA-EDF-INDRIA Schools. Sophia Antipolis. March 30 - April p.37/50

38 Local ordering for unsymmetric matrices (PAQ) T (PAQ) = Q T A T P T PAQ = Q T A T AQ So choose the column ordering Q using the minimum degree ordering algorithm on A T A COLAMD (Matlab) does this without forming A T A Row ordering can be chosen later for stability. CEA-EDF-INDRIA Schools. Sophia Antipolis. March 30 - April p.38/50

39 Global ordering for symmetric matrices PARTITIONING Divide and conquer paradigm (Nested) dissection Domain decomposition Substructuring CEA-EDF-INDRIA Schools. Sophia Antipolis. March 30 - April p.39/50

40 Global ordering for symmetric matrices Computational Graph CEA-EDF-INDRIA Schools. Sophia Antipolis. March 30 - April p.40/50

41 Global ordering for symmetric matrices Chooses last pivots first Dissect problem into two parts Resulting matrix has form: Dissection Hence to nested dissection CEA-EDF-INDRIA Schools. Sophia Antipolis. March 30 - April p.41/50

42 Nested dissection Nested dissection was originally proposed by Alan George in his thesis in Although good on regular grids and for complexity was not used for general symmetric matrices until 1990s Key to achieving good ordering on general problems is Bisection Algorithm CEA-EDF-INDRIA Schools. Sophia Antipolis. March 30 - April p.42/50

43 Spectral Bisection (Pothen, Simon, Liou 1990) Laplacian matrix, L, of a matrix A l ii = degree of i l ij = 0, a ij = 0, i j l ij If x is such that x i = +1 for i in partition C and x i = 1 for i in partition D, then x T Lx = a i,j 0 = 1, a ij 0, i j (x i x j ) 2 = 4 #cut edges CEA-EDF-INDRIA Schools. Sophia Antipolis. March 30 - April p.43/50

44 Spectral Bisection (Pothen, Simon, Liou 1990) So graph bisection problem is: min x T Lx x x i = ±1 xi = 0 Continuous problem is to min {x T Lx/x T x} over all vectors that are orthogonal to the vector of all ones. So we search for the Fiedler vector of L. This is the eigenvector corresponding to the second smallest eigenvalue. CEA-EDF-INDRIA Schools. Sophia Antipolis. March 30 - April p.44/50

45 Comparison of AMD and Nested Dissection A nested dissection ordering can be generated by the graph partitioning software called METIS (Karypis and Kumar) For large problems this often beats a local ordering. For example, AMD (Amestoy, Davis, and Duff) Matrices are from PARASOL test set and runs are from a very early version of MUMPS (more about this code later). Entries in factors Operations in factorization (10 6 ) (10 9 ) MIXTANK AMD ND INVEXTR1 AMD ND BBMAT AMD ND CEA-EDF-INDRIA Schools. Sophia Antipolis. March 30 - April p.45/50

46 Sparse Complexity COMPLEXITY of LU factorization on dense matrices of order n 2 3 n3 + O (n 2 ) flops n 2 storage For band matrix (order n, bandwidth k) 2k 2 n work, nk storage Five-diagonal matrix (on k k grid) O (k 3 ) work and O (k 2 logk) storage Tridiagonal + Arrowhead matrix O (n) work and storage Target O (n) + O (τ) for sparse matrix of order n with τ entries. CEA-EDF-INDRIA Schools. Sophia Antipolis. March 30 - April p.46/50

47 MA48 An example of a code that uses Markowitz and Threshold Pivoting is the HSL Code MA48 CEA-EDF-INDRIA Schools. Sophia Antipolis. March 30 - April p.47/50

48 Features of MA48 Uses Markowitz with threshold pivoting Factorizes rectangular and singular systems Block Triangularization Done at higher level Internal routines work only on single block Switch to full code at all phases of solution Three factorize entries Only pivot order is input Fast factorize.. uses structures generated in first factorize Only factorizes later columns of matrix Iterative refinement and error estimator in Solve phase Can employ drop tolerance Low storage in analyse phase CEA-EDF-INDRIA Schools. Sophia Antipolis. March 30 - April p.48/50

49 Sparse HSL code MA48 Matrix Order Entries MA48 DGESV KALLMAN FS GRE WEST ORANI LNS LHR07C Comparison between a sparse code (MA48 from HSL) and a dense code (DGESV from LAPACK). Times for analyse/factorize/solve in seconds on a Compaq/HP Alpha DS 20 workstation. CEA-EDF-INDRIA Schools. Sophia Antipolis. March 30 - April p.49/50

50 MA48.. why look at other codes Why look at other codes?? MA48 does not naturally vectorize or parallelize short vector lengths There is heavy use of indirect addressing in MA48 (viz. references of form W(IND(I)) Even with h/ware indirect addressing is 2-4 times slower more memory traffic non-localization of data MA48 can perform poorly when there is significant fill-in We would like to take more advantage of the Level 3 BLAS CEA-EDF-INDRIA Schools. Sophia Antipolis. March 30 - April p.50/50