Large-Scale Reservoir Simulation and Big Data Visualization

Large-Scale Reservoir Simulation and Big Data Visualization Dr. Zhangxing John Chen NSERC/Alberta Innovates Energy Environment Solutions/Foundation CMG Chair Alberta Innovates Technology Future (icore) Chair Director, Foundation CMG/Frank-Sarah Meyer Collaboration Center, University of Calgary Slide 1

Outlines Motivations - Importance Simulation and Visualization Case Studies Summary Remarks Slide 2

Outlines Motivations - Importance Parallelization Means Case Studies Summary Remarks Slide 3

Motivations Accuracy Stability Robustness Speed capital savings Slide 4

Speed Issue 3D thermal models: with 500,000 cells geomodels and multiprocessor computers, runs take hours to weeks. For typical optimization run, 400-1,000 runs required. If done serially, hundreds to thousands of days for each optimization run required. With parallelization, optimization runtimes will drop. More geological descriptions can be added. Slide 5

Motivations Grid (scale) requirements Physics requirements Process requirements Slide 6

Grid Requirements Reservoir (rock and fluid) heterogeneity Thin moving thermal front Thin mobile solvent-rich layers ~ 1cm Discontinuity: presence of faults, fractures, and mud and shale layers Small dispersion/diffusion Local reaction (chemistry) zones Slide 7

Grid Requirements Rock heterogeneity Slide 8

Fluid heterogeneity Grid Requirements Slide 9

Grid Requirements Thin moving thermal and solvent fronts Slide 10

Grid Requirements Discontinuity: presence of faults, fractures, and mud and shale layers Slide 11

Grid Requirements Small dispersion/diffusion Slide 12

Grid Requirements Local reaction (chemistry) zones Slide 13

Physics Requirements Disparate data with different scales Thermal and solvent effects Mass and heat transfer Phase behavior Geomechanics Wellbore flow Slide 14

Process Requirements Thermal Recovery Processes - SAGD (steam assisted gravity drainage) - CSS (cyclic steam stimulation) - ISC (in situ combustion) Solvent Recovery Processes - VAPEX (vapor extraction) - SAP (solvent aided process) Slide 15

Business Competiveness Field management Equipment management Cost management SIMULATION TECHNOLOGY Strategy Safety Slide 16

Outlines Motivations - Importance Simulation and Visualization Case Studies Summary Remarks Slide 17

Petroleum Reservoir Simulators Black oil simulator (water flooding, fractured reservoirs) Compositional simulator (CO 2, N 2, CBM, tight and shale oil and gas) Thermal simulator (CSS, SAGD, ISC, THAI, Steam flooding) Chemical simulator (SAP+Foam) Wellbore module Geomechanical module Slide 18

Acceleration Means Software Acceleration Hardware Acceleration Slide 19

Software Acceleration Grid Management Discretization (Numerical) Methods System Solvers Parallelization (Matrices and Vectors) Software Design Slide 20

Grid Management The most critical modules - Type, choice of numerical methods - Partition, workload, communication - Data structure, info and data distribution - Numbering, bandwidth, and input Slide 21

Discretization Methods Finite difference methods Finite volume (control volume) methods Finite element methods Slide 22

System Solvers Preconditioners Linear Solvers A x = b Nonlinear Solvers Slide 23

Polynomial Approximate Inverse ILUT(p, tol) ILU(k) Preconditioners Domain Decomposition (Restricted Additive Schwarz) Algebraic Multigrid: classical and smoothed aggregation Slide 24

Linear Solvers GMRES, CG, BICGSTAB and GCR CGS, Orthomin and Orthodir Classical AMG Smoothed Aggregation AMG Slide 25

Nonlinear Solvers Newton Iterations Newton-Raphson Iterations Slide 26

Software Design OpenMP - Multi-core, easy to use, limited scalability MPI - communication - MPI-IO Programming languages - C, C++, Fortran, Slide 27

Hardware Acceleration CPUs (central processing units) GPUs (graph processing units) Slide 28

CPU VS GPU GPU (C2050) CPU (X5570) Cores 448 6 Memory 144 GB/s 10 GB/s Float Performance 1030G (s) / 515G (d) ~20G GPU is around 10 times faster than CPU! Slide 29

Outlines Motivations Simulation and Visualization Case Studies Summary Remarks Slide 30

AMG: Example 1 Two-dimensional elliptic problem Grid size: 1,000x1,000, non-zeros: 4,996,000 Tol: 1e-6, level: 8 36 iterations GPU: 1.30s CPU: 12.26s Speedup: 9.33 Slide 31

AMG: Example 2 Two-dimensional elliptic problem Grid size: 1,500x1,500, non-zeros: 11,244,000 Tol: 1e-6, level: 8 31 iterations GPU: 2.58s CPU: 24.50s Speedup: 9.42 Slide 32

AMG: Example 3 Three-dimensional elliptic problem Grid size: 100x100x100, non-zeros: 6,940,000 Tol: 1e-6, level: 8 21 iterations GPU: 1.13s CPU: 10.80s Speedup: 9.58 Slide 33

AMG: Example 4 Three-dimensional elliptic problem Grid size: 130x130x130, non-zeros: 15,277,000 Tol: 1e-6, level: 8 25 iterations GPU: 3.02s CPU: 37.86s Speedup: 12.47 Slide 34

Numerical Studies on CPUs Parallel (University of Calgary), Westgrid 528 standard nodes - 26-core Intel Xeon E5649 processors - 24 G memory InfiniBand 4X QDR, 40 Gbit/s Slide 35

15M Case: Example 5 GMRES + DDM Grid: 250x250x250 Unknowns: 15 million # processors 2 4 8 16 32 64 Grid time (s) 44.93 21.74 10.63 5.36 2.81 1.59 Overall time (s) 122.68 59.82 27.48 13.54 6.90 3.49 Slide 36

125M Case: Example 6 GMRES + DDM Grid: 500x500x500 Unknowns: 125 million # processors 16 32 64 128 Grid time (s) 49.19 24.18 11.84 5.45 Overall time (s) 1258.55 662.10 338.68 166.54 Slide 37

200M Case: Example 7 GMRES + DDM Grid: 585x585x585 Unknowns: 200 million # processors 16 32 64 128 Grid time (s) 80.54 38.93 19.82 9.52 Overall time (s) 2471.98 1286.08 670.83 346.07 Slide 38

Billion Case: Example 8 GMRES + DDM Grid size: billion (B) # processors 64 128 160 200 Grid size 1 B 1B 2 B 3 B Grid time (s) 115.76 57.56 94.74 123.62 Overall time (s) 4141.05 2029.93 3229.67 4060.92 Slide 39

Stereo 3D and Immersive Visualization Immersive visualization has been shown to enable interactions that are more intuitive and that let users focus on analysis. Slide 40

Stereo 3D and Immersive Visualization Slide 41

Stereo 3D and Immersive Visualization Slide 42

Case Study T.Q.C. Dang, L.X. Nghiem, Z. Chen, and T.B.N. Nguyen, CO 2 low-salinity water alternating gas: A promising new approach for EOR, Journal of Petroleum Technology, January 2015, 84-86 Slide 43

Case Study Geological Modeling Facies Modeling Clay Mapping Advanced Field Scale LSW Modeling Sensitivity Analysis History Matching Optimization Uncertainty Assessment Reservoir Simulation Flow, Ion Exchange, Geochemistry, Wettability Alteration CEC & Porosity Modeling Permeability Modeling Slide 44

Case Study Slide 45

Case Study Optimal Pattern Slide 46

Case Study Slide 47

Outlines Motivations Simulation and Visualization Case Studies Summary Remarks Slide 48

Summary Remarks Ultimate Goals Increasing reserves Reducing operating costs Enhancing petroleum recovery Capital savings Slide 49

Sponsors Slide 50