Linux on System z Performance Tips David Simpson - IT Specialist IBM Corporation [email protected]

Transcription

1 Linux on System z Performance Tips David Simpson - IT Specialist IBM Corporation [email protected]

2 Copyright and Trademark Information For IBM can be found at For Oracle can be found at

3 Agenda CPU Memory I/O Networking Monitoring 3

4 z/vm and Linux Dynamic Changes z/vm LPAR Linux Guest Add CPU Yes Yes Increase weight / share Yes Yes Add memory Yes Yes Add I/O adapter Yes Yes Remove CPU Yes Yes Decrease weight / share Yes Yes Remove memory No Yes Remove I/O adapter Yes Yes RHEL 6.2 and SLES 11 SP2 or later, z/vm 5.4 or later

5 Sizing Linux on System z Workload For existing/running workloads take system utilization data for as long a period as possible or at least peak periods combined with make/model of server. If new workload we use a sizing questionnaire with our TechLine sizing team and used industry standards to size the workload. 5

6 Sizing Consolidated CPU consumption equivalent MIPS 2937 MIPS

7 Stay current with your Linux updates Check updates for performance enhancements RHEL 5.9 VDSO HyperPAV SLES 11 SP2+ GRO / TSO Security updates need to be considered as well 7

8 Linux configuration Disable all not needed services splash, postfix, nfs, Disable cgroup memory Kernel parameter cgroup_disable=memory Saves 1% of memory per guest. Red Hat - disable selinux or follow Red Hat procedure (you will need rpms) Kernel parameter selinux=0 8

9 Oracle Parallelism Default Value: PARALLEL_MAX_SERVERS = (CPU_COUNT x PARALLEL_THREADS_PER_CPU x 10) If too many query server processes, memory contention (paging), I/O contention, or excessive context switching can occur Contention can reduce system throughput to a level lower than if parallel execution were not used. Can utilize Oracle Consumer Group to limit processes for certain types of users/jobs 9

10 Monitoring CPU Run Levels / Oracle Parallel Query # vmstat 3 (on virtual 2 way machine) procs memory swap io---- -system cpu r b swpd free buff cache si so bi bo in cs us sy id wa st st Value is avg from last startup i.e. Typically ignore r run queue how many processes currently waiting for CPU b how many processes waiting in uninterruptible sleep Steal time (st) is the percentage of time a virtual CPU waits for a real CPU while the hypervisor is servicing another virtual processor. When sustained load average/run queue is > 4x number of virtuals with no cpu idle then this may be an indication of lack of cpu. In this example Linux swap (lack of Linux guest memory) seems to be the main culprit. 10

11 CPUPLUGD CPUPLUGD Daemon can be configured to add or reduce the number of Virtual processors based on the load Oracle dynamically changes the Oracle internal parameter cpu_count based on the number of Virtual processors available. This should be the default! Explicitly setting cpu_count will disable the automatic adaption of Oracle DB to cpuplugd changes CPUPLUGD configuration recommendations Need fast sampling interval (1s) Create sensitive configuration for CPU add 11

12 VDSO Linux cpu Improvements Virtual Dynamically-linked Shared Object (VDSO) is a shared library provided by the kernel. This allows normal programs to do certain system calls without the usual overhead of system calls like switching address spaces. On a z196 system for example by using the VDSO implementation six times reduction in the function calls are possible. Newer Linux distributions (RHEL 5.9 & 6.x, SLES 11) have this feature and it's enabled by default. Oracle calls Linux gettimeofday() hundreds of times a second for 12 reporting statistics.

13 Agenda CPU Memory I/O Networking Monitoring 13

14 Memory Sizing Oracle with Linux on System z Linux Customer attempted install 11gR2 with 512mb could not re-link on install. Oracle recommends 4GB for all Linux Platforms, smallest we would suggest is 2GB of Virtual Memory for a Single Oracle 11g/12c instance. One customer experienced 200 MB more RAM consumption 10gR2 to 11gR2 Right Size the Virtual Memory based on What is needed: All SGA s (including ASM) consider Large Pages Oracle PGA s (not eligible for Large Pages small pages) User Connections to the database (4.5mb per connection small pages) Linux Page Tables and Linux Kernel Memory (small pages) Try NOT to oversize the Linux Guest under z/vm, use VDISKs Leave room (5-10%) such that kswapd and OOM (out of mem mgr) don t kick in, 14

15 Linux paging / swappiness With the default swappiness setting of 60 Linux does proactive paging Oracle data / code on a Linux (or VM) paging disk has a performance hit when it s needed Observed long (>10s) waits at swap in Guest was sized correctly Guest was using database on a file system without direct I/O Recommendation: set swappiness to zero In /etc/syctl.conf add vm.swappiness=0 15 Largepages are ineligible for swapping.

16 Swap Sizing Oracle on System z Linux and 11gR2 Example of VDISK for 1 st and or 2 nd Level Swap with higher priority and then DASD as a lower priority swap in case of an unexpected memory pattern May want to recycle the swap from time to time to free swap slots (check swapcache in /proc/meminfo) Ensure there is enough memory (e.g. at night) drop caches swapoff / swapon 16

17 Linux Huge Pages Consider Using Linux Huge Pages for Oracle Database Memory In general 10-15% can be gained by the reduction in CPU usage as well as having a lot more memory for applications that would be consumed in Linux Page Tables 17

18 /proc/meminfo customer example (before) MemTotal: kb MemFree: kb Buffers: 4956 kb Cached: kb SwapCached: kb Active: kb Inactive: kb HighTotal: 0 kb HighFree: 0 kb LowTotal: kb LowFree: kb SwapTotal: kb SwapFree: kb Dirty: 468 kb 18 Writeback: 0 kb AnonPages: kb Mapped: kb Slab: kb PageTables: kb NFS_Unstable: 0 kb Bounce: 0 kb CommitLimit: kb Committed_AS: kb VmallocTotal: kb VmallocUsed: kb VmallocChunk: kb HugePages_Total: 0 HugePages_Free: 0 HugePages_Rsvd: 0 Hugepagesize: 2048 kb

19 /proc/meminfo customer example (after) MemTotal: kb MemFree: kb Buffers: kb Cached: kb SwapCached: 0 kb Active: kb Inactive: kb HighTotal: 0 kb HighFree: 0 kb LowTotal: kb LowFree: kb SwapTotal: kb SwapFree: kb Dirty: 504 kb 19 Writeback: 108 kb AnonPages: kb Mapped: kb Slab: kb PageTables: kb NFS_Unstable: 0 kb Bounce: 0 kb CommitLimit: kb Committed_AS: kb VmallocTotal: kb VmallocUsed: kb VmallocChunk: kb HugePages_Total: HugePages_Free: 1208 HugePages_Rsvd: 1205 Hugepagesize: 2048 kb

20 HugePage Considerations: Can not use MEMORY_TARGET with Huge Pages. Set manually to SGA_TARGET not including the PGA_AGGREGATE_TARGET. Not swappable: Huge Pages are not swappable General guideline consider when combined Oracle SGA s are greater than 8 GB (particularly if a lots of connections) Decreased page table overhead; more memory can be freed up for other uses. i.e. more Oracle SGA memory, and less physical I/O s (See also Oracle Note: ) 20

21 Recommendation: Use Huge Pages - even under z/vm Under z/vm (which has 4K pages) it s still recommended to use Huge Pages for SGA s > 10GB particularly with many connections Saves Memory that would otherwise be used for pagetables Stability for user process spikes (avoiding swap) Less work to manage smaller number of pagetables ~10% improvement (ROT) for memory intensive databases 21

22 Agenda Linux CPU Memory I/O Networking Monitoring 22

23 Verify I/O Performance with Oracle Orion Oracle ORION Simulates Oracle reads and writes, without having to create a database No Longer Download from Oracle it is now included with Oracle Code in $ORACLE_HOME/bin/orion./orion_zlinux -run oltp -testname test -num_disks 2 -duration 30 -simulate raid0 ORION VERSION Commandline: -run oltp -testname mytest -num_disks 2 -duration 30 -simulate raid0 This maps to this test: Test: mytest Small IO size: 8 KB Large IO size: 1024 KB IO Types: Small Random IOs, Large Random IOs Simulated Array Type: RAID 0 Stripe Depth: 1024 KB Write: 0% Cache Size: Not Entered Duration for each Data Point: 30 seconds Small Columns:, 2, 4, 6, 8, 10, 12, 14, 16, 18, 20, 22, 24, 26, 28, 30, 32, 34, 36, 38, 40 Large Columns:, 0 Total Data Points: 22 Name: /dev/dasdq1 Size: Name: /dev/dasdr1 Size: FILEs found. Maximum Small Small=40 and Large=0 Minimum Small Small=2 and Large=0 23

24 ASM or LVM LVM Logical Volume Manager in Linux ASM Automated Storage Management provided by Oracle Oracle RAC One and Oracle RAC will require ASM LVM pro Direct control on setting and layout Can choose file system ASM Automated, out of the box environment Very good integration with Oracle con Complex setup RMAN required for backup Overall recommendation: ASM Don t combine both! 24

25 Database files on Filesystem: Disable read ahead & Use Async / direct I/O Oracle parameter file systems: filesystemio_options=setall Provides asynchronous & direct I/O (avoids linux file system cache) Reduce Linux Read-Ahead for LVM file systems. lvchange -r none <lv device name> 25

26 Kernel I/O Scheduler The Linux 2.6 kernel offers a choice of four different I/O schedulers: Noop Scheduler (noop) Deadline Scheduler (deadline) Anticipatory Scheduler (as) Complete Fair Queuing Scheduler (cfq) Some Linux kernels default is the cfq scheduler: Designed to optimize access to physical disks Check in /sys/block/<device>/queue/scheduler noop anticipatory [deadline] cfq Default configurable by setting the elevator=[ ] boot parameter in /etc/zipl.conf Recommendation: deadline or noop 26

27 HyperPAV (1) HyperPAV allows multiple IO operations on the same sub channel Very important for random access workload with relative small data transfers HyperPAV aliases per LCU showed best performance gains in our internal tests. Recommendation: Enable HyperPAV whenever using ECKD devices lsdasd u bc dasda IBM NA bc eac ca dasdk IBM NA ca eac d0 alias IBM NA xx d1 alias IBM NA xx

28 HyperPAV (2) ECKD Devices: Scaling HyperPAV aliases Normalized Transactional throughput and total Disk I/O (read + write) 160% 120 normalized transactional throughput 140% 120% 100% 80% 60% 40% 20% 0% MiB/sec # alias devices normalized transactional throughput Total MB/sec 28

29 FICON / FCP comparison (1) 160% 140% 120% 100% 80% 60% 40% 20% normalized transactional throughput 0% 4 CPUs 6 CPUs FICON (20 aliases) FCP (rr_min_io=100) 29

30 FICON / FCP comparison (2) 120% Normalized CPU cost per transaction CPU cost per transactional throughput 100% 80% 60% 40% 20% 0% 4 CPUs 6 CPUs FICON (20 aliases) FCP (rr_min_io=100) 30

31 FICON / FCP comparison (3) FCP offers better throughput and performance FICON uses less CPU per transaction You have to tune both environments Recommendation: it depends 31

32 Linux multipathing rr_min_io / rr_min_io_rq For FCP attached devices multipathing is needed for availability Guidance for SLES11 + RHEL6 is to use multibus option in multipath.conf (opposed to failover). rr_min_io defines the number of I/O operations that are send to path before switching to the next (round robin) Defined in multipath.conf In RHEL6.2+ / SLES11 SP2+ now called rr_min_io_rq Device mapper does accept old rr_min_io but does nothing with it Watch for upgrades! The rr_min_io value is storage dependent 32 For DS8K rr_min_io=100 provided good results XIV recommends rr_min_io=15

33 Sample multipath.conf defaults { dev_loss_tmo 90 #zseries specific, no. of secs wait before marking path bad failback "immediate" # failback "manual" #use manual with RAC clusters to prevent ping-pong effect fast_io_fail_tmo 5 #zseries specific, length time to wait before failing I/O getuid_callout "/lib/udev/scsi_id --whitelisted --device=/dev/%n" # getuid_callout "/sbin/scsi_id -g -u -s /block/%n" #use with Red Hat 5.x,SLES 10 # uid_attribute "ID_SERIAL" #use uid_attribute instead of getuid for SLES 11SP3+ & RH7 max_fds "max" #Red Hat 6.3+, SLES 11SP3+ # max_fds 8192 #Use for Older SLES and Red Hat distros no_path_retry "queue" #XiV recommendation if no paths available, use fail with RAC # no_path_retry "fail" # use fail if using Oracle RAC path_selector "round-robin 0" #round-robin for SLES 11 SP2+ and RedHat 6.x # path_selector "service-time 0" #default for SLES 11 SP3+, RedHat 7+ NOTE: test first path_grouping_policy "multibus" # SLES 11 SP1+ and Red Hat 6.x # path_grouping_policy "failover" # SLES 10, Red Hat 5.x path_checker "tur" # Determines state of path, XiV, DS8000, verify with vendor polling_interval 30 # Time in seconds to check paths prio "const" queue_without_daemon "yes" # rr_min_io 100 #rr_min_io for older Linux distro's (Red Hat 5.x,SLES 11sp1 & older) # rr_min_io_rq 1 #RamSan RS-710/810, use 4 RamSan RS-720/820 # rr_min_io_rq 15 #IBM XiV rr_min_io_rq 100 #IBM DS8000, rr_min_io_rq for newer Linux (SLES 11sp2+ RH 6+) rr_weight "uniform" user_friendly_names "yes" } 33 blacklist { devnode "^(dasd)[0-9]*" } multipaths { multipath { #use /lib/udev/scsi_id --whitelisted --device=/dev/sda to get unique wwid wwid ffcbbf ef00 alias ASMFCP1 } Cust: By the way, with your additional settings I got 552MB/s (from 505mb/s)

34 File System Types & Noatime mount option Recommended file system types: SUSE ext3 file system recommended for database files. new SUSE xfs testing, see Oracle Note for latest: Red Hat 6.x - ext4 recommended for database files. Reiser (the default) does not perform that well with Oracle databases. Linux ext3/4 file systems keep track of different times: access, modify, change time Problem is nearly always the access time Updated at every read Only very few applications make use of that Turned on by default Oracle db file systems on ext3/4 mount with the noatime, (nodiratime) options Reduces CPU usage and I/Os - see Oracle Note

35 Linux I/O - Queue Depth - nr_requests / queue_depth Default of 32 generally pretty good (check with Storage vendor) Set in /sys/bus/scsi/devices/<scsi device>/queue_depth Reasons to decrease value: Latency problems (pretty rare) Storage subsystem overload Reasons to increase value: System with heavy I/O load Storage vendor suggestion / recommendation Use with care, due to the overload problem 35

36 Changing I/O scheduler with UDEV Rule I/O settings you may want to consider adjusting (based on storage vendor recommendations): scheduler I/O scheduler (noop or deadline) nr_requests number of I/O requests default 128 (may want to increase to 1024 for Flash Storage devices) rotational 1 default for spinning disk, 0 for Flash nomerges default 0, 1 for Flash max_sectors_kb default 1024, 4096 for Flash rq_infinity default 1, 2 for Flash add_random default 1, 0 for Flash Sample UDEV rules (/etc/udev/rules.d) 99-udev-oracle.rules KERNEL=="sda",OWNER="oracle",GROUP="dba",MODE="660",RUN+="/bin/sh -c echo noop > /sys/$devpath/queue/scheduler && echo 1024 > /sys/$devpath/queue/nr_requests'" KERNEL=="dasd*1",ID==" ",OWNER="oracle",GROUP= dba,mode="0660",symlink+="asm0300 # cat /sys/block/sda/queue/nr_requests dm-permissions.rules #ENV{DM_UUID}=="mpath ffcbbf ef01", OWNER:="oracle", GROUP:="dba", MODE:="660" ENV{DM_NAME}=="ASM*",OWNER:="oracle",GROUP:="dba",MODE:="660",RUN+="/bin/sh -c 'echo noop > /sys/$devpath/queue/scheduler && echo 1024 > /sys/$devpath/queue/nr_requests'" # cat /sys/devices/virtual/block/dm-0/queue/nr_requests 1024 Restarting UDEV rules (Sles 11 sp3/red Hat 6): # udevadm control reload-rules # udevadm trigger 36

37 Separate Redo log files from database (1) Conflicting kind of I/O Logs are large sequential writes (good to optimize) Normal database workloads are many small random read / writes Storage subsystem can t optimize if everything put together Watch Oracle events log file sync and log file parallel write Recommendation: put logs and data in different ASM disk groups 37

38 Separate Redo log files from database (2) Data and Logs - Disk Setup 30% 25% 20% % Total Call Time 15% 10% 5% 0% Log file sync log file parallel write 4 disks, log + data mixed 4 disk data + 2 disks log 38

39 Oracle Cache, Memory and I/O Access 39

40 AWR Analysis for an I/O Bound System - Recommendations Summary I/O bound AWR s can benefit from Flash Storage in the following way: 21x reduction in response times 957%improvement in IO wait time Other Recommendations Rerun AWR reports after implementing to validate outcomes Use application partitioning on the app-tier to ensure data files and log files are located on the faster storage.

41 AWR Analysis for an I/O Bound DB -Wait Time Impact Top 5 Foreground User I/O Wait Events The Top 5 User IO foreground Wait Events indicate substantive improvements in IO response time. The IO wait time would go down from 53.6% of overall wait time to 5%, an improvement of 957%. 41

42 AWR Analysis - Latency Impact The microsecond response times would significantly reduce latency, more than 21x, driving higher utilization at the server and application level. Latency down from 4.2ms to.28ms. 42

43 Monitoring I/O performance (iostat x) 43

44 Dasdstat 44

45 Dasdstat - cont 45

46 Dasdstat (contd) 46

47 Ziomon 47 Red Hat Error -> ziomon: Error: Debugfs not mounted on /sys/kernel/debug Fix: mount -t debugfs nodev /sys/kernel/debug

48 Ziomon (contd) 48

49 Ziomon (contd) ziorep_config => 49

50 Agenda CPU Memory I/O Networking Monitoring 50

51 System z Networking z/os z/os System z z/vm LPAR Linux Linux VSWITCH Dedicated OSA OSA HiperSockets To Ethernet networks Shared OSA OSA To Ethernet networks OSA To Ethernet networks

52 Networking Choose correct MTU size (should match with App Servers) Network queue length For high transactional applications try to reduce number of hops (latency) between app and db server 52 hsi Linux z/vm hsi Linux VSWITCH Hipersocket hsi Linux eth eth eth eth OSA OSA LAN VSWITCH z/vm hsi Linux

53 Choose the Correct Network MTU size netstat s of Interconnect MTU Size of 1492 (default) MTU Size of 8992 (with 8K DB block size, especially RAC) Before reassemblies 43,530,572 1,563,179 After reassemblies 54,281,987 1,565,071 Delta assemblies 10,751,415 1,892 53

54 Network Queue Length The device queue length should be increased from the default size of 1000 to at least 2000 using sysctl: sysctl -w net.core.netdev_max_backlog =

55 Networking: Hipersockets Checksumming Disable HiperSockets does not require network checksum since it is a memory to-memory operation. To save CPU cycles, switch checksumming off: SUSE SLES10: in /etc/sysconfig/hardware/hwcfg-qeth-bus-ccw- 0.0.F200 add QETH_OPTIONS="checksumming=no_checksumming" SUSE SLES11: in /etc/udev/rules.d/51-qeth-0.0.f200.rules add ACTION=="add", SUBSYSTEM=="ccwgroup", KERNEL=="0.0.f200", ATTR{checksumming}="no_checksumming Red Hat: in /etc/sysconfig/network-scripts/ifcfg-eth0 add 55 OPTIONS="checksumming=no_checksumming

56 Oracle Network Configuration Testing VSwitch (Active / Passive), Linux Bonding, VSwitch Link Aggregation and Oracle s HAIP Tests included shared OSA cards across multiple System z machines Separation of Interconnect traffic (application server as well) including VLANs improves performance and stability. Multiple Write/Write intensive databases performed best with Link Aggregation or HAIP 56

57 Agenda CPU Memory I/O Networking Monitoring 57

58 Collect Linux performance data Standalone performance collection in Linux is sysstat For standard monitoring use same interval as for your z/vm monitoring Always monitor your system Include monitoring for disks (default off) 58

59 Linux Monitoring Tools Collect Linux monitoring data with historical analysis Standalone performance collection - sysstat rpm (sadc, sar, vmstat) Include monitoring for disks (default off) OSWatcher, Linux nmon, and some z/vm tools can be integrated to providing graphing capabilities. OSW Document Doc ID , OSWg Document Doc ID , LTOM Document Doc ID Linux nmon reports: LTOM Graphs: 59

60 Performance Management Strategies Linux on System z Database wide problems with Oracle with Linux on System z it s important to have the right performance management reports at all levels. (z/vm, Linux & Database). Collect and retain z/vm performance data Other tooling from ISVs (Velocity -> new with Oracle Integration, CSL wave, Tivoli OmegaMon XE) Velocity zmon (ESAMON) CSL Wave IBM Performance Toolkit 60