Espaces grossiers adaptatifs pour les méthodes de décomposition de domaines à deux niveaux

Transcription

1 Espaces grossiers adaptatifs pour les méthodes de décomposition de domaines à deux niveaux Frédéric Nataf Laboratory J.L. Lions (LJLL), CNRS, Alpines et Univ. Paris VI joint work with Victorita Dolean (Univ. Nice Sophia-Antipolis) Patrice Hauret (Michelin, Clermont-Ferrand) Frédéric Hecht (LJLL) Pierre Jolivet (LJLL) Clemens Pechstein (Johannes Kepler Univ., Linz) Robert Scheichl (Univ. Bath) Nicole Spillane (LJLL) CEA Saclay 2013

2 Outline 1 Introduction 2 An abstract 2-level Schwarz: the GenEO algorithm 3 Conclusion F. Nataf et al GENEO 2 / 31

3 Outline 1 Introduction Classical coarse space method 2 An abstract 2-level Schwarz: the GenEO algorithm 3 Conclusion F. Nataf et al GENEO 3 / 31

4 Motivation Large discretized system of PDEs strongly heterogeneous coefficients (high contrast, nonlinear, multiscale) E.g. Darcy pressure equation, P 1 -finite elements: Applications: flow in heterogeneous / stochastic / layered media structural mechanics electromagnetics etc. AU = F cond(a) α max α min h 2 Goal: iterative solvers robust in size and heterogeneities F. Nataf et al GENEO 4 / 31

5 Linear Algebra from the End User point of view Direct DDM Iterative Cons: Memory Pro: Flexible Pros: Memory Difficult to Naturally Easy to Pros: Robustness Cons: Robustness solve(mat,rhs,sol) Few black box routines solve(mat,rhs,sol) Few implementations of efficient DDM Multigrid methods: very efficient but may lack robustness, not always applicable (Helmholtz type problems, complex systems) and difficult to parallelize. F. Nataf et al GENEO 5 / 31

6 The First Domain Decomposition Method The original Schwarz Method (H.A. Schwarz, 1870) (u) = f in Ω u = 0 on Ω. Ω 1 Ω 2 Schwarz Method : (u1 n, un 2 ) (un+1 1, u n+1 2 ) with (u n+1 1 ) = f in Ω 1 u n+1 1 = 0 on Ω 1 Ω u n+1 1 = u2 n on Ω 1 Ω 2. (u n+1 2 ) = f in Ω 2 u n+1 2 = 0 on Ω 2 Ω u n+1 2 = u n+1 1 on Ω 2 Ω 1. Parallel algorithm, converges but very slowly, overlapping subdomains only. The parallel version is called Jacobi Schwarz method (JSM). F. Nataf et al GENEO 6 / 31

7 Strong and Weak scalability How to evaluate the efficiency of a domain decomposition? Strong scalability (Amdahl) How the solution time varies with the number of processors for a fixed total problem size Weak scalability (Gustafson) How the solution time varies with the number of processors for a fixed problem size per processor. Not achieved with the one level method Number of subdomains ASM The iteration number increases linearly with the number of subdomains in one direction. F. Nataf et al GENEO 7 / 31

8 How to achieve scalability Stagnation corresponds to a few very low eigenvalues in the spectrum of the preconditioned problem. They are due to the lack of a global exchange of information in the preconditioner. u = f in Ω u = 0 on Ω The mean value of the solution in domain i depends on the value of f on all subdomains. A classical remedy consists in the introduction of a coarse problem that couples all subdomains. This is closely related to deflation technique classical in linear algebra (see Nabben and Vuik s papers in SIAM J. Sci. Comp, 200X). F. Nataf et al GENEO 8 / 31

9 Adding a coarse space We add a coarse space correction (aka second level) Let V H be the coarse space and Z be a basis, V H = span Z, writing R 0 = Z T we define the two level preconditioner as: M 1 ASM,2 := RT 0 (R 0AR T 0 ) 1 R 0 + N i=1 R T i A 1 i R i. The Nicolaides approach is to use the kernel of the operator as a coarse space, this is the constant vectors, in local form this writes: Z := (Ri T D i R i 1) 1 i N where D i are chosen so that we have a partition of unity: N Ri T D i R i = Id. i=1 F. Nataf et al GENEO 9 / 31

10 Theoretical convergence result Theorem (Widlund, Dryija) Let M 1 ASM,2 be the two-level additive Schwarz method: κ(m 1 ASM,2 (1 A) C + H ) δ where δ is the size of the overlap between the subdomains and H the subdomain size. This does indeed work very well Number of subdomains ASM ASM + Nicolaides F. Nataf et al GENEO 10 / 31

11 Failure for Darcy equation with heterogeneities (α(x, y) u) = 0 in Ω R 2, u = 0 on Ω D, u n = 0 on Ω N. IsoValue Our approach Decomposition α(x, y) Jump ASM ASM + Nicolaides Fix the problem by an optimal and proven choice of a coarse space Z. F. Nataf et al GENEO 11 / 31

12 Outline 1 Introduction 2 An abstract 2-level Schwarz: the GenEO algorithm Choice of the coarse space First Numerical results Parallel implementation 3 Conclusion F. Nataf et al GENEO 12 / 31

13 Objectives Strategy Define an appropriate coarse space V H 2 = span(z 2 ) and use the framework previously introduced, writing R 0 = Z2 T the two level preconditioner is: P 1 ASM 2 := RT 0 (R 0AR T 0 ) 1 R 0 + N i=1 R T i A 1 i R i. The coarse space must be Local (calculated on each subdomain) parallel Adaptive (calculated automatically) Easy and cheap to compute Robust (must lead to an algorithm whose convergence is proven not to depend on the partition nor the jumps in coefficients) F. Nataf et al GENEO 13 / 31

14 Abstract eigenvalue problem Gen.EVP per subdomain: Find p j,k V h Ωj and λ j,k 0: a Ωj (p j,k, v) = λ j,k a Ω j (Ξ j p j,k, Ξ j v) v V h Ωj A j p j,k = λ j,k X j A j X j p j,k (X j... diagonal) a D... restriction of a to D In the two-level ASM: Choose first m j eigenvectors per subdomain: V 0 = span { Ξ j p j,k } j=1,...,n k=1,...,m j This automatically includes Zero Energy Modes. F. Nataf et al GENEO 14 / 31

15 Abstract eigenvalue problem Gen.EVP per subdomain: Find p j,k V h Ωj and λ j,k 0: a Ωj (p j,k, v) = λ j,k a Ω j (Ξ j p j,k, Ξ j v) v V h Ωj A j p j,k = λ j,k X j A j X j p j,k (X j... diagonal) a D... restriction of a to D In the two-level ASM: Choose first m j eigenvectors per subdomain: V 0 = span { Ξ j p j,k } j=1,...,n k=1,...,m j This automatically includes Zero Energy Modes. F. Nataf et al GENEO 14 / 31

16 Comparison with existing works Galvis & Efendiev (SIAM 2010): κ p j,k v dx = λ j,k κ p j,k v dx Ω j Ω j v V h Ωj Efendiev, Galvis, Lazarov & Willems (submitted): a Ωj (p j,k, v) = λ j,k a Ωj (ξ j ξ i p j,k, ξ j ξ i v) v V Ωj Our gen.evp: i neighb(j) ξ j... partition of unity, calculated adaptively (MS) a Ωj (p j,k, v) = λ j,k a Ω j (Ξ j p j,k, Ξ j v) v V h Ωj both matrices typically singular = λ j,k [0, ] F. Nataf et al GENEO 15 / 31

17 Theory Two technical assumptions. Theorem (Spillane, Dolean, Hauret, N., Pechstein, Scheichl) If for all j: 0 < λ j,mj+1 < : [ κ(m 1 ASM,2 A) (1 + k 0) 2 + k 0 (2k 0 + 1) N max j=1 ( )] λ j,mj +1 Possible criterion for picking m j : (used in our Numerics) λ j,mj +1 < δ j H j H j... subdomain diameter, δ j... overlap F. Nataf et al GENEO 16 / 31

18 Numerical results via a Domain Specific Language FreeFem++ ( with: Metis Karypis and Kumar 1998 SCOTCH Chevalier and Pellegrini 2008 UMFPACK Davis 2004 ARPACK Lehoucq et al MPI Snir et al Intel MKL PARDISO Schenk et al MUMPS Amestoy et al PaStiX Hénon et al Why use a DS(E)L instead of C/C++/Fortran/..? performances close to low-level language implementation, hard to beat something as simple as: varf a(u, v) = int3d(mesh)([dx(u), dy(u), dz(u)]' * [dx(v), dy(v), dz(v)]) + int3d(mesh)(f * v) + on(boundary mesh)(u = 0) F. Nataf et al GENEO 17 / 31

19 Numerics 2D Elasticity E IsoValue e e e e e e e e e e e e e e e e e e e e+11 E 1 = ν 1 = 0.3 E 2 = ν 2 = 0.45 METIS partitions with 2 layers added subd. dofs AS-1 AS-ZEM (V H ) GENEO (V H ) (12) 42 (42) (48) 45 (159) (75) 47 (238) (192) 45 (519) F. Nataf et al GENEO 18 / 31

20 Numerics 3D Elasticity Iterations (CG) vs. number of subdomains E 1 = ν 1 = 0.3 Relative error vs. iterations 16 regular subdomains E 2 = ν 2 = 0.45 subd. dofs AS-1 AS-ZEM (V H ) GENEO (V H ) (24) 16 (46) (48) 16 (102) (96) 16 (214) AS-ZEM (Rigid body motions): m j = 6 F. Nataf et al GENEO 19 / 31

21 Numerical results Optimality Layers of hard and soft elastic materials m i is given automatically by the strategy. #Z per subd. one level ZEM GenEO max(m i 1, 3) 2600 (93) m i 5.1 e5 (184) 1.4 e4 (208) 53 (35) m i (25) condition number (iteration count) for one and two level ASMs Taking one fewer eigenvalue has a huge influence on the iteration count Taking one more has only a small influence F. Nataf et al GENEO 20 / 31

22 Eigenvalues and eigenvectors Eigenvector number 1 is e-15; exageration coefficient is: Eigenvector number 2 is e-16; exageration coefficient is: Eigenvector number 3 is e-15; exageration coefficient is: Eigenvector number 4 is e-05; exageration coefficient is: Eigenvector number 5 is e-05; exageration coefficient is: Eigenvector number 6 is e-05; exageration coefficient is: Eigenvector number 7 is ; exageration coefficient is: Eigenvector number 8 is ; exageration coefficient is: Eigenvector number 9 is ; exageration coefficient is: Eigenvector number 10 is ; exageration coefficient is: Eigenvector number 11 is ; exageration coefficient is: Eigenvector number 12 is ; exageration coefficient is: Eigenvector number 13 is ; exageration coefficient is: Eigenvector number 14 is ; exageration coefficient is: Eigenvector number 15 is ; exageration coefficient is: eigenvalue (log scale) eigenvalue number Logarithmic scale F. Nataf et al GENEO 21 / 31

23 Darcy pressure equation κ(x, y) x y Figure : Two dimensional diffusivity κ F. Nataf et al GENEO 22 / 31

24 Channels and inclusion IsoValue e e e e e e+06 IsoValue Channels and inclusions: 1 α , the solution and partitionings (Metis or not) F. Nataf et al GENEO 23 / 31

25 Parallel implementation PhD of Pierre Jolivet. Since version 1.16, bundled with the Message Passing Interface. FreeFem++ is working on the following parallel architectures (among others): N of cores Memory Peak perf hpc1@ljll 160@2.00 Ghz 640 Go 10 TFLOP/s titane@cea 12192@2.93 Ghz 37 To 140 TFLOP/s babel@idris 40960@850 Mhz 20 To 139 TFLOP/s curie@cea 92000@2.93 Ghz 315 To 1.6 PFLOP/s Bruyères-le-Châtel, France. Orsay, France. F. Nataf et al GENEO 24 / 31

26 Strong scalability in two dimensions heterogeneous elasticity Elasticity problem with heterogeneous coefficients Timing relative to processes Linear speedup 10 #iterations #processes Speed-up for a 1.2 billion unknowns 2D problem. Direct solvers in the subdomains. Peak performance wall-clock time: 26s. F. Nataf et al GENEO 25 / 31

27 Strong scalability in three dimensions heterogeneous elasticity Elasticity problem with heterogeneous coefficients Timing relative to processes Linear speedup 10 #iterations #processes Speed-up for a 160 million unknowns 3D problem. Direct solvers in subdomains. Peak performance wall-clock time: 36s. F. Nataf et al GENEO 26 / 31

28 Weak scalability in two dimensions Darcy problems with heterogeneous coefficients Weak efficiency relative to processes 100 % 80 % 60 % 40 % 20 % 0 % #d.o.f. (in millions) #processes Efficiency for a 2D problem. Direct solvers in the subdomains. Final size: 22 billion unknowns. Wall-clock time: 200s. F. Nataf et al GENEO 27 / 31

29 Weak scalability in three dimensions Darcy problems with heterogeneous coefficients Weak e ciency relative to processes 100 % 80 % 60 % 40 % 20 % 0% #d.o.f. (in millions) #processes Efficiency for a 3D problem. Direct solvers in the subdomains. Final size: 2 billion unknowns. Wall-clock time: 200s. F. Nataf et al GENEO 28 / 31

30 Outline 1 Introduction 2 An abstract 2-level Schwarz: the GenEO algorithm 3 Conclusion F. Nataf et al GENEO 29 / 31

31 Conclusion GenEO for ASM Implementation requires only element stiffness matrices + connectivity FreeFem++ MPI implementation Not shown here Outlook GenEO for BNN/FETI methods Nonlinear time dependent problem (Reuse of the coarse space) Coarse problem satisfies assembling property multilevel method link to σamge? Other discretizations: finite volume, finite difference,... F. Nataf et al GENEO 30 / 31

32 Conclusion GenEO for ASM Implementation requires only element stiffness matrices + connectivity FreeFem++ MPI implementation Not shown here Outlook GenEO for BNN/FETI methods Nonlinear time dependent problem (Reuse of the coarse space) Coarse problem satisfies assembling property multilevel method link to σamge? Other discretizations: finite volume, finite difference,... F. Nataf et al GENEO 30 / 31

33 Conclusion GenEO for ASM Implementation requires only element stiffness matrices + connectivity FreeFem++ MPI implementation Not shown here Outlook GenEO for BNN/FETI methods Nonlinear time dependent problem (Reuse of the coarse space) Coarse problem satisfies assembling property multilevel method link to σamge? Other discretizations: finite volume, finite difference,... F. Nataf et al GENEO 30 / 31

34 Bibliography Schwarz: DtN and GenEO Preprints available on HAL: N. Spillane, V. Dolean, P. Hauret, F. Nataf, C. Pechstein, R. Scheichl, An Algebraic Local Generalized Eigenvalue in the Overlapping Zone Based Coarse Space : A first introduction, C. R. Mathématique, Vol. 349, No , pp , F. Nataf, H. Xiang, V. Dolean, N. Spillane, A coarse space construction based on local Dirichlet to Neumann maps, SIAM J. Sci Comput., Vol. 33, No.4, pp , F. Nataf, H. Xiang, V. Dolean, A two level domain decomposition preconditioner based on local Dirichlet-to-Neumann maps, C. R. Mathématique, Vol. 348, No , pp , V. Dolean, F. Nataf, R. Scheichl, N. Spillane, Analysis of a two-level Schwarz method with coarse spaces based on local Dirichlet to Neumann maps, CMAM, 2012 vol. 12, P. Jolivet, V. Dolean, F. Hecht, F. Nataf, C. Prud homme, N. Spillane, High Performance domain decomposition methods on massively parallel architectures with FreeFem++, J. of Numerical Mathematics, 2012 vol. 20. N. Spillane, V. Dolean, P. Hauret, F. Nataf, C. Pechstein, R. Scheichl, Abstract Robust Coarse Spaces for Systems of PDEs via Generalized Eigenproblems in the Overlaps, submitted to Numerische Mathematik, V. Dolean, F. Nataf, P. Jolivet: Domain decomposition methods. Theory and parallel implementation with FreeFEM++, Lecture notes (coming soon). THANK YOU FOR YOUR ATTENTION! F. Nataf et al GENEO 31 / 31

35 Bibliography Schwarz: DtN and GenEO Preprints available on HAL: N. Spillane, V. Dolean, P. Hauret, F. Nataf, C. Pechstein, R. Scheichl, An Algebraic Local Generalized Eigenvalue in the Overlapping Zone Based Coarse Space : A first introduction, C. R. Mathématique, Vol. 349, No , pp , F. Nataf, H. Xiang, V. Dolean, N. Spillane, A coarse space construction based on local Dirichlet to Neumann maps, SIAM J. Sci Comput., Vol. 33, No.4, pp , F. Nataf, H. Xiang, V. Dolean, A two level domain decomposition preconditioner based on local Dirichlet-to-Neumann maps, C. R. Mathématique, Vol. 348, No , pp , V. Dolean, F. Nataf, R. Scheichl, N. Spillane, Analysis of a two-level Schwarz method with coarse spaces based on local Dirichlet to Neumann maps, CMAM, 2012 vol. 12, P. Jolivet, V. Dolean, F. Hecht, F. Nataf, C. Prud homme, N. Spillane, High Performance domain decomposition methods on massively parallel architectures with FreeFem++, J. of Numerical Mathematics, 2012 vol. 20. N. Spillane, V. Dolean, P. Hauret, F. Nataf, C. Pechstein, R. Scheichl, Abstract Robust Coarse Spaces for Systems of PDEs via Generalized Eigenproblems in the Overlaps, submitted to Numerische Mathematik, V. Dolean, F. Nataf, P. Jolivet: Domain decomposition methods. Theory and parallel implementation with FreeFEM++, Lecture notes (coming soon). THANK YOU FOR YOUR ATTENTION! F. Nataf et al GENEO 31 / 31