AIX NFS Client Performance Improvements for Databases on NAS

AIX NFS Client Performance Improvements for Databases on NAS October 20, 2005 Sanjay Gulabani Sr. Performance Engineer Network Appliance, Inc. gulabani@netapp.com Diane Flemming Advisory Software Engineer IBM Corporation dianefl@us.ibm.com Slide 1

Agenda Why Databases on NAS? AIX NFS improvements - CIO Simple IO Performance Tests OLTP Performance Comparison Future work Conclusion Q&A Slide 2

Databases on SAN Databases do block based I/O and prefer raw blocks But, most admins still put a volume manager and a file system: JFS2, VxFS, UFS over HW RAID storage Reason: Simplicity -- Easier backups and provisioning Slide 3

NAS Myths Myth: NFS consumes more CPU for Databases Reality: Most NFS client performance problems with DB apps are due to kernel locking and are fixed in good NFS clients Slide 4

NAS Myths Myth: Ethernet is slower than SAN Reality: Ethernet is 1Gb, FC is 2Gb. 4Gb FC is almost here but so is 10GbE. Ethernet is likely to takeover bandwidth of FC soon; cost effectively Its easy to setup multiple wires to match FC bandwidth with just 1GbE, today Storage latencies at database layer are measured in msecs, differences between SAN and NAS latencies are in usecs 78usec more for 8K blocks < 0.1msec difference! Slide 5

Why NAS? Networking is simpler Networking promotes sharing Sharing = Higher utilization (~grid storage) NAS solutions are cheaper than SAN No one has won against Ethernet! Seriously, even blocks based storage is moving to Ethernet (iscsi) Slide 6

Why DB on NAS? Complex storage management is offloaded from the database servers to storage appliances Performance vs. manageability Database requirements different than traditional NFS workloads: home directories, distributed source code single writer lock can be a problem Slide 7

Industry Support for NAS Oracle On-Demand hosts ERP+CRM 1000+ Linux servers on NetApp NFS Yahoo runs Database on NetApp Sun 2003 presentation McDougall + Colaco promoted NFS NetApp #1 Leader of NAS $1.59b(FY05) IBM AIX 5.3 ML-03 improvements now No one got fired for buying from IBM! (IBM resells NetApp products) Slide 8

Oracle s Austin Datacenter More than 15,000 x86 servers 3.1 Petabytes of NetApp storage 100s of mission-critical hosted Oracle apps Source: Oracle Magazine Mar/Apr 2005 Slide 9

I/O, I/O It s off to disk I go Raw logical volumes No memory management overhead Buffered I/O Variety of kernel services Direct I/O No memory management overhead defaults to synchronous accesses over NFS Concurrent I/O DIO plus no inode/rnode contention Slide 10

Metrics: NFS File Transfer CPU utilization and throughput in KB/sec Options: Default (buffered I/O) and CIO AIX Client and Server p630 1.45GHz 4-way GbE 1500-byte MTU AIX 5.3 ML-03 Demonstrates algorithm behavior Atomic, Single-Threaded Slide 11

Throughput (KB/sec) 120000 100000 80000 60000 40000 20000 0 NFS File Transfer Sequential Reads and Writes Seq Write Buff I/O Throughput Buff I/O CPU Util Seq Read CIO Throughput CIO CPU Util 40 30 20 10 0 CPU Util (%) 256MB File, NFS V3, TCP, auth=sys, 32KB RPCs Slide 12

Metrics Simple I/O Load Generator CPU utilization, IOPS, throughput, etc. Options NFS client mount options to specify I/O mode: Default (buffered I/O), DIO, CIO NetApp Filer FAS880 ONTAP 6.5.1 AIX Client Model p630 1.45 GHz 4-way AIX 5.3 ML-03 GbE 1500-byte MTU Slide 13

SIO Throughput 50% Writers, 256MB file KB/sec 80000 70000 60000 50000 40000 30000 20000 10000 0 1 2 4 8 16 32 64 128 256 Buff I/O CIO DIO Number of Threads 256MB File, NFS V3, TCP, auth=sys, AIX v5.3 ML-3 client Slide 14

SIO Throughput 100% Writers, 256MB File KB/sec 60000 50000 40000 30000 20000 10000 0 1 2 4 8 16 32 64 128 256 Number of Threads Buff I/O CIO DIO 256MB File, NFS V3, TCP, auth=sys, AIX v5.3 ML-3 client Slide 15

SIO Scaled Throughput using CIO, 256MB file 200000 KBps/CPU 150000 100000 50000 0 1 2 4 8 16 32 64 128 256 Number of Threads 100% Writers 50% Writers 256MB File, NFS V3, TCP, auth=sys, AIX v5.3 ML-3 client Slide 16

NFS FT and SIO Summary CIO not appropriate for single threaded sequential read/write workloads. CIO aimed at workloads with higher concurrency such as databases. Significant gain in raw throughput performance using CIO vs. Buffered I/O for workloads with higher concurrency. CPU utilization illustrates issue with TCP socket lock contention with increased concurrency. Slide 17

OLTP Performance AIX 5.2 ML-5 and AIX 5.3 ML-03 (beta) PowerPC pseries 650 (2 * 1.2 GHz, 4 GB RAM) Oracle 10.1 1GbE for NFS or iscsi or 2Gb FCP card FAS 940c 48 Disk Spindles (4 x DS14) 144GB 10K RPM ONTAP 6.5.1 Slide 18

Transactions on AIX 5.2, 5.3, iscsi, FCP Chart 1: Transactions for protocols 16000 14000 12000 11817 12171 14139 transactions 10000 8000 6000 4710 OETs 4000 2000 0 kernel NFS 5.2 @60u NFS 5.3* 60u iscsi raw 5.3* 60u FCP raw 5.3* 60u OETs = Order Entry Transactions (an Oracle OLTP benchmark, approx 2:1 read to write ratio) Slide 19

Host CPU Utilization Chart 2: Transactions and cpu utilization for protocols 16000 14000 98 90 14139 100 12000 11817 12171 88 80 transactions 10000 8000 6000 4710 49 60 40 cpu OETs cpu 4000 20 2000 0 0 kernel NFS 5.2 @60u NFS 5.3 60u iscsi raw 5.3 60u FCP raw 5.3 60u Slide 20

Protocol Efficiencies Computed using (%CPU used/ OETs) * K: where K is a constant such that NFS CPU cost per K transactions is 1.0 and relative cost for iscsi and FCP is computed using the same constant K transactions. Chart 3: Relative CPU cost per protocol for fixed transactions 1.2 normalized cpu cost (lower is better) 1 0.8 0.6 0.4 0.2 1 0.89 0.75 relative cpu cost 0 NFS 5.3 60u iscsi raw 5.3 60u FCP raw 5.3 60u protocol Slide 21

Oracle Block Read Response Time Not typical response time. Random workload used. Filer with more cache improves block response time or more sequential reads improve avg. block response time. Chart 4: Oracle Block Read Response Time Block read Resp Time (lower is better) 35 30 25 20 15 10 5 0 29 kernel NFS 5.2 @60u 13 14 NFS 5.3 60u iscsi raw 5.3 60u 12 FCP raw 5.3 60u block read resp time protocol Slide 22

Future Work Testing and analysis on systems with higher numbers of CPUs. Further investigation on socket lock scaling issue on AIX Slide 23

Conclusions AIX 5.3 ML-03 with new Concurrent I/O CIO mount option delivers an OLTP performance on NFS comparable to that of block-based protocols iscsi and FCP Don t be afraid to deploy databases on AIX NFS, we will support you. Proof of AIX NFS performance up to 4 CPU completed. Slide 24

References AIX Performance with NFS, iscsi and FCP Using an Oracle Database on NetApp White Paper @ http://www.netapp.com/library/tr/3408.pdf NetApp Best Practices Paper located at: http://www.ibm.com/servers/storage/support/nas/ Download SIO tool http://now.netapp.com/now/download/tools/sio_ntap/index.shtml Improving Database Performance with AIX Concurrent I/O: http://www-03.ibm.com/servers/aix/whitepapers/db_perf_aix.pdf Slide 25

Questions/Answers Slide 26