ECLIPSE Performance Benchmarks and Profiling. January 2009

Size: px

Start display at page:

Download "ECLIPSE Performance Benchmarks and Profiling. January 2009"

Gwen Gilmore
8 years ago
Views:

1 ECLIPSE Performance Benchmarks and Profiling January 2009

2 Note The following research was performed under the HPC Advisory Council activities AMD, Dell, Mellanox, Schlumberger HPC Advisory Council Cluster Center For more info please refer to

HPC Advisory Council Cluster Center For more info please refer

3 Schlumberger ECLIPSE Oil and gas reservoir simulation software Developed by Schlumberger Offers multiple choices of numerical simulation techniques for accurate and fast simulation for Black-oil Compositional Thermal Streamline Others ECLIPSE support MPI to achieve high performance and scalability 3

for accurate and fast simulation for Black-oil Compositional Thermal

4 Objectives The presented research was done to provide best practices ECLIPSE performance benchmarking Interconnect performance comparisons Ways to increase ECLIPSE productivity Understanding ECLIPSE communication patterns Power-efficient simulations 4

performance comparisons Ways to increase ECLIPSE productivity

5 Test Cluster Configuration System Upgrade Dell PowerEdge SC node cluster Quad-Core AMD Opteron Model 2382 processors ( Shanghai ) Mellanox InfiniBand ConnectX DDR HCAs Mellanox InfiniBand DDR Switch Memory: 16GB memory, DDR2 800MHz per node OS: RHEL5U2, OFED 1.3 InfiniBand SW stack MPI: Platform MPI Application: Schlumberger ECLIPSE Simulators Benchmark Workload 4 million cell model ( ) Blackoil 3 phase model with ~ 800 wells 5

DDR2 800MHz per node OS: RHEL5U2, OFED 1.3 InfiniBand SW stack MPI: Platform MPI 5.6.

6 Mellanox InfiniBand Solutions Industry Standard Hardware, software, cabling, management Design for clustering and storage interconnect Price and Performance 40Gb/s node-to-node 120Gb/s switch-to-switch 1us application latency Most aggressive roadmap in the industry Reliable with congestion management Efficient RDMA and Transport Offload Kernel bypass CPU focuses on application processing Scalable for Petascale computing & beyond End-to-end quality of service Virtualization acceleration I/O consolidation Including storage The InfiniBand Performance Gap is Increasing 60Gb/s 20Gb/s 120Gb/s 40Gb/s 240Gb/s (12X) 80Gb/s (4X) Ethernet Fibre Channel InfiniBand Delivers the Lowest Latency 6

Offload Kernel bypass CPU focuses on application processing Scalable for Petascale computing & beyond End-to-end quality of service Virtualization acceleration I/O

7 Quad-Core AMD Opteron Processor Performance Quad-Core Enhanced CPU IPC Dual Channel Reg DDR2 4x 512K L2 cache 6MB L3 Cache 8 GB/S 8 GB/S Direct Connect Architecture HyperTransport technology Up to 24 GB/s peak per processor Floating Point 8 GB/S 8 GB/S PCI-E PCI-E 128-bit FPU per core Bridge Bridge Bridge Bridge 4 FLOPS/clk peak per core 8 GB/S Integrated Memory Controller Up to 12.8 GB/s USB USB I/O I/O Hub Hub DDR2-800 MHz or DDR2-667 MHz PCI PCI Scalability 48-bit Physical Addressing Compatibility Same power/thermal envelopes as 2 nd / 3 rd generation AMD Opteron processor 7 November5,

Bridge Bridge Bridge 4 FLOPS/clk peak per core 8 GB/S Integrated Memory Controller Up to 12.

8 Dell PowerEdge Servers helping Simplify IT System Structure and Sizing Guidelines 24-node cluster build with Dell PowerEdge SC 1435 Servers Servers optimized for High Performance Computing environments Building Block Foundations for best price/performance and performance/watt Dell HPC Solutions Scalable Architectures for High Performance and Productivity Dell's comprehensive HPC services help manage the lifecycle requirements. Integrated, Tested and Validated Architectures Workload Modeling Optimized System Size, Configuration and Workloads Test-bed Benchmarks ISV Applications Characterization Best Practices & Usage Analysis 8

Architectures for High Performance and Productivity Dell's comprehensive HPC services help manage the lifecycle requirements.

9 ECLIPSE Performance Results - Interconnect InfiniBand enables highest scalability Performance accelerates with cluster size Performance over GigE and 10GigE is not scaling Slowdown occurs beyond 8 nodes 6000 Schlumberger ECLIPSE (FOURMILL) Elapsed Time (Seconds) Number of Nodes Lower is better GigE 10GigE InfiniBand Single job per cluster size 9

beyond 8 nodes 6000 Schlumberger ECLIPSE (FOURMILL) Elapsed Time (Seconds) 5000 4000 3000 2000

10 ECLIPSE Performance Results - Interconnect 600% Schlumberger ECLIPSE (InfiniBand vs GigE & 10GigE) Performance Advantage 500% 400% 300% 200% 100% 0% Number of Nodes GigE 10GigE InfiniBand outperforms GigE by up to 500% and 10GigE by up to 457% As node number increases, bigger advantage is gained 10

4 8 12 16 20 22 24 Number of Nodes GigE 10GigE InfiniBand outperforms GigE by

11 ECLIPSE Performance Results - Productivity InfiniBand increases productivity by allowing multiple jobs to run simultaneously Providing required productivity for reservoir simulations Three cases are presented Single job over the entire systems Four jobs, each on two cores per CPU per server Eight jobs, each on one CPU core per server Eight jobs per node increases productivity by up to 142% 300 Schlumberger ECLIPSE (FOURMILL) Number of Jobs Number of Nodes Higher is better 1 Job per Node 4 Jobs per Node 8 Jobs per Node InfiniBand 11

server Eight jobs, each on one CPU core per server Eight jobs per node increases productivity by up to 142% 300 Schlumberger ECLIPSE (FOURMILL)

12 ECLIPSE Performance Results - Productivity InfiniBand offers unparalleled productivity compared to Ethernet GigE shows performance decrease beyond 8 nodes 10GigE demonstrates no scaling beyond 16 nodes 250 Schlumberger ECLIPSE (FOURMILL) Number of Jobs Number of Nodes Higher is better GigE 10GigE InfiniBand 4 Jobs on each node 12

scaling beyond 16 nodes 250 Schlumberger ECLIPSE (FOURMILL) Number of Jobs 200 150 100 50

13 ECLIPSE Profiling Data Transferred ECLIPSE MPI Profiliing MPI_Isend 10 Number of Messages (Millions) [0..128B] [128B..1KB] [1..8KB] [8..256KB] [256KB..1M] [1M..Infinity] Message Size 4nodes 8nodes 12nodes 16nodes 20nodes 24nodes 13

14 ECLIPSE Profiling Data Transferred ECLIPSE MPI Profiliing MPI_Recv Number of Messages (Millions) [0..128B] [128B..1KB] [1..8KB] [8..256KB] [256KB..1M] [1M..Infinity] Message Size 4nodes 8nodes 12nodes 16nodes 20nodes 24nodes 14

15 ECLIPSE Profiling Message Distribution Eclipse MPI Profiliing Percentage of Messages 60% 50% 40% 30% Number of Nodes MPI_Isend < 128 Bytes MPI_Recv < 128 Bytes MPI_Isend < 256 KB MPI_Recv < 256 KB Majority of MPI messages are large size Demonstrating the need for highest throughput 15

Bytes MPI_Recv < 128 Bytes MPI_Isend < 256 KB MPI_Recv < 256 KB Majority of

16 Interconnect Usage by ECLIPSE Total server throughput increases rapidly with cluster size Data Transferred (MB/s) Data Transferred (MB/s) Data Sent 4 Nodes Data Sent 8 Nodes Data Transferred (MB/s) Timing (s) Timing (s) Data Sent 16 Nodes Data Sent 24 Nodes Timing (s) Data Transferred (MB/s) This data is per node based Timing (s) 16

1000 800 600 400 200 0 1 48 95 142 189 236 283 330 377 424 471 518 565 612 659 706 753 800 847 Timing (s) Data Transferred (MB/s) This data is per node based 1400 1200 1000 800 600

17 ECLIPSE Profiling Summary - Interconnect ECLIPSE was profiled to determine networking dependency Majority of data transferred between compute nodes Done with 8KB-256KB message size Data transferred increases with cluster size Most used message sizes <128B messages mainly synchronizations 8KB-256KB data transferring Message size distribution Percentage of smaller messages (<128B) slightly decreases with cluster size Percentage of mid-size messages (8KB-256KB) increases with cluster size ECLIPSE interconnects sensitivity points Interconnect latency and throughput for <256KB message range As node number increases, interconnect throughput becomes more critical 17

size distribution Percentage of smaller messages (<128B) slightly decreases with cluster size Percentage of mid-size messages (8KB-256KB) increases with cluster size

18 Power Consumption/Productivity Comparison InfiniBand enables power efficient simulations Reducing system power/job consumption up to 66% vs GigE and 33% vs 10GigE For productivity case 4 jobs per node When using single job approach, InfiniBand reduces power/job consumption by more than 82% compared to 10GigE Power Consumption Power per Job (Wh) % 33% GigE 10GigE InfiniBand 4 Jobs on each node 18

single job approach, InfiniBand reduces power/job consumption by more than 82% compared to 10GigE Power

19 Conclusions Eclipse is widely used to perform reservoir simulation Developed by Schulmberger ECLIPSE performance and productivity relies on Scalable HPC systems and interconnect solutions Low latency and high throughout interconnect technology NUMA aware application for fast access to memory Reasonable job distribution can dramatically improve productivity Increasing number of jobs per day while maintaining fast run time Interconnect comparison shows InfiniBand delivers superior performance and productivity in every cluster size Scalability requires low latency and zero scalable latency InfiniBand enables lowest power consumption per job Optimizing power/job ratio 19

dramatically improve productivity Increasing number of jobs per day while maintaining fast run time Interconnect comparison shows InfiniBand delivers superior

20 Thank You HPC Advisory Council All trademarks are property of their respective owners. All information is provided As-Is without any kind of warranty. The HPC Advisory Council makes no representation to the accuracy and completeness of the information contained herein. HPC Advisory Council Mellanox undertakes no duty and assumes no obligation to update or correct any information presented herein 20 20

The HPC Advisory Council makes no representation to the accuracy and completeness of the information

LS DYNA Performance Benchmarks and Profiling. January 2009

LS DYNA Performance Benchmarks and Profiling January 2009 Note The following research was performed under the HPC Advisory Council activities AMD, Dell, Mellanox HPC Advisory Council Cluster Center The