1 Performance monitoring in the GNU/Linux environment Linux is like a wigwam - no Windows, no Gates, Apache inside! 2 1
Post-conditions To be familiar with some performance-tuning options To be able to select appropriate performance monitoring tools under Linux To be able to apply reasonable interpretation to the output of performance monitoring tools (under Linux). 3 References Fink, J.R., & M.D. Sherer, (2002). Linux Performance Tuning and Capacity Planning. SAMS: Indianapolis. Loukides, M. (1991). System Performance Tuning. O Reilly: Sebastapol, CA. 4 2
Reactive versus Proactive Tuning Reactive: observe a bottleneck and respond Proactive: predict a bottleneck (based on capacity planning) and anticipate needs 5 Typical targets of tuning CPU Memory Swap Disk X11 Application settings Networking but not a focus of this unit Other sound, etc. 6 3
Compute-bound vs. I/O-bound Compute-bound applications bound by CPU speed I/O-bound applications bound by I/O speed 7 Typical tuning activities Monitoring performance monitoring tools bench-marking tests Tuning hardware upgrades driver upgrades application tuning and management kernel configuration and compilation 8 4
Some Linux Monitoring Tools 9 free Statistics: total: used: free: Total available memory Total amount of memory used Total amount of free memory shared: Total shared memory being used buffers: Size of disk buffer cache cached: Amount of memory that has been cached to disk 10 5
free Note that Linux uses an aggressive caching policy which may cause surprise when observing memory usage. 11 free YIKES! $ free total used free shared buffers cached Mem: 130592 126464 4128 0 6012 50344 -/+ buffers/cache: 70108 60484 Swap: 249440 0 249440 Used memory (no buffers/cache) Free memory (with buffers/cache) 12 6
uptime Statistics: The length of time that the system has been up. The average number of jobs in the run queue for the last 1 minute, 5 minutes, 10 minutes values over 1 indicate resource starvation 13 uptime Mean queue (10 mins) $ uptime 4:50am up 1:21, 1 user, load average: 0.07, 0.04, 0.01 Up for 1 hour, 21 minutes Mean queue (5 mins) Problems sleeping! Mean queue (last minute) What There can are you no warning say about signs; the the system system based is lightly on this loaded. output? 14 7
uptime $ uptime 5:52am up 2:23, 1 user, load average: 4.18, 3.01, 2.86 What The CPU can is you a bottleneck say about the under system current based loads, on and this there output? is evidence of backlog. 15 vmstat procs r: number of processes waiting to run b: number of processes interrupted from sleep mode w: number of processes swapped out and yet to run memory swpd: total amount of virtual memory in use free: total amount of free memory buff: total amount of buffered memory cache: amount of cache in use swap si: memory transferred from swap space into memory so: memory swapped to disk 16 8
vmstat io bi: disk block sent to disk devices (blocks / sec) bo: outbound disk activity system in: interrupts / sec (including CPU clocks) cs: context switches per second cpu us: % cycles on user processes sy: % cycles on system processes id: % cycles idle 17 vmstat $ vmstat procs memory swap io system cpu r b w swpd free buff cache si so bi bo in cs us sy id 0 0 0 0 2900 6012 52012 0 0 22 1 128 114 2 1 97 18 9
vmstat $ vmstat 2 6 procs memory swap io system cpu r b w swpd free buff cache si so bi bo in cs us sy id 0 0 0 0 2820 6012 52092 0 0 19 1 124 103 2 1 97 0 0 0 0 2820 6012 52092 0 0 0 0 102 50 0 0 100 0 0 0 0 2820 6012 52092 0 0 0 0 102 35 0 0 100 0 0 0 0 2820 6012 52092 0 0 0 2 102 37 0 0 100 0 0 0 0 2820 6012 52092 0 0 0 0 103 32 0 0 100 0 0 0 0 2820 6012 52092 0 0 0 0 102 33 0 0 100 19 The /proc pseudo file-system /proc is a pseudo file-system, providing a hook into the kernel. Typically it is used for reading various kernel parameters. It may also be used for changing such parameters (without the need to recompile the kernel!) 20 10
/proc/cpuinfo $ cat /proc/cpuinfo processor : 0 vendor_id : GenuineIntel cpu family : 6 model : 8 model name : Pentium III (Coppermine) stepping : 6 cpu MHz : 651.495 cache size : 256 KB fdiv_bug : no hlt_bug : no sep_bug : no f00f_bug : no coma_bug : no fpu : yes fpu_exception : yes cpuid level : 2 wp : yes flags : fpu vme de pse tsc msr pae mce cx8 sep mtrr pge mca cmov pat pse36 mmx fxsr xmm bogomips : 1300.89 21 /proc/meminfo $ cat /proc/meminfo total: used: free: shared: buffers: cached: Mem: 133726208 93433856 40292352 0 2916352 19476480 Swap: 255426560 221184 255205376 MemTotal: 130592 kb MemFree: 39348 kb MemShared: 0 kb Buffers: 2848 kb Cached: 19020 kb BigTotal: 0 kb BigFree: 0 kb SwapTotal: 249440 kb SwapFree: 249224 kb 22 11
Some other examples /proc/devices /proc/interrupts /proc/loadavg /proc/modules /proc/mounts /proc/partitions 23 bonnie Bonnie performs a disk I/O benchmark test: Writing to disk: Sequentially, one character at a time Sequentially, in larger blocks Reading from disk: Sequentially, one character at a time Sequentially, in larger blocks Random seeks 24 12
bonnie $ bonnie File './Bonnie.9318', size: 104857600 Writing with putc()...done Rewriting...done Writing intelligently...done Reading with getc()...done Reading intelligently...done Seeker 1...Seeker 2...Seeker 3...start 'em...done...done...done... -------Sequential Output-------- ---Sequential Input-- --Random-- -Per Char- --Block--- -Rewrite-- -Per Char- --Block--- --Seeks--- Machine MB K/sec %CPU K/sec %CPU K/sec %CPU K/sec %CPU K/sec %CPU /sec %CPU 100 2555 32.1 3242 7.8 1238 22.7 2577 65.6 4143 62.0 99.3 4.7 25 top Status: Up time number of processes CPU states memory swap 26 13
top Per-process details: PID process ID number USER user name PRI priority NI nice value SIZE total, including code, data, and stack space (KB) RSS physical memory used SHARE shared memory used STATE sleeping (S), uninterrupted (D), runnable (R), stopped (T), zombie (Z) TIME CPU time since starting %CPU CPU time since last update %MEM share of physical memory COMMAND command name 27 top $ top 5:03am up 1:34, 1 user, load average: 0.00, 0.00, 0.00 112 processes: 111 sleeping, 1 running, 0 zombie, 0 stopped CPU states: 1.5% user, 1.0% system, 0.0% nice, 97.3% idle Mem: 130592K av, 127940K used, 2652K free, 0K shrd, 6012K buff Swap: 249440K av, 0K used, 249440K free 52084K cached PID USER PRI NI SIZE RSS SHARE STAT %CPU %MEM TIME COMMAND 9268 tony 16 0 928 928 680 R 2.8 0.7 0:00 top 1 root 0 0 468 468 404 S 0.0 0.3 0:05 init 2 root 0 0 0 0 0 SW 0.0 0.0 0:00 kflushd 3 root 0 0 0 0 0 SW 0.0 0.0 0:00 kupdate 4 root 0 0 0 0 0 SW 0.0 0.0 0:00 kswapd 5 root -20-20 0 0 0 SW< 0.0 0.0 0:00 mdrecoveryd 32 root 0 0 348 348 296 S 0.0 0.2 0:00 open 40 root 0 0 1860 1860 444 S 0.0 1.4 0:11 wsmonitorbin 381 root 0 0 0 0 0 SW 0.0 0.0 0:00 khubd 411 bin 0 0 392 392 312 S 0.0 0.3 0:00 portmap 476 root 0 0 580 580 484 S 0.0 0.4 0:00 automount 492 root 0 0 616 616 508 S 0.0 0.4 0:00 automount 540 root 0 0 464 464 404 S 0.0 0.3 0:00 apmd 556 root 0 0 588 588 444 S 0.0 0.4 0:00 syslogd Processes swapped out 28 14
xload 29 xosview Process load average CPU usage: user, nice, sys, free Memory: user-allocated, buffer, cache, free Swap: in-use, free Paging: paging in, paging out, idle Disk: read, write, idle Interrupts: 0 to 23 30 15
Responses 31 CPU tuning Hardware solutions: Faster processor Multiprocessor environment especially if the offending applications fork into multiple processes or, if the application was specifically designed to optimise in SMP (shared memory multiprocessor ) environments Linux scales well to SMP (at least to 8 CPUs) Different architecture 32 16
CPU tuning Software solutions: Application configuration turn off capabilities that aren t being used System configuration turn off competing processes that aren t useful 33 CPU tuning Kernel solutions: Not a lot of options can define the specific CPU type and features in the build which enables specific optimisations to be used base kernel usually assumes low-end capabilities some kernel patches may address the issue in other ways e.g., low latency patch for music 34 17
Memory tuning Hardware solutions: More memory Faster memory newer memory is often much faster Cache size hardware caches generally improve access, but over-large caches may involve longer look-up times 35 Memory tuning Software solutions: Application choice bad coding can result in performance that appears to be due to a lack of memory user threads vs. POSIX threads Addressable memory Intel chips often impose a 4GB-addressable memory limit Processor Address Extension extends this, but requires tuning. Otherwise, consider a 64b architecture 36 18
Memory tuning Kernel solutions: Contiguous memory needs may respond to a bigmem kernel patch Monolithic versus modular kernels Modular kernels only load in the relevant functionality as it is required. If you don t use a particular piece of hardware often, for example, then having that driver modular will conserve kernel memory space 37 Memory tuning Swap-space planning: Consider programs that will run concurrently Determine their memory requirements and add a few extra MB Subtract physical memory from this amount the remainder is the minimum swap space if this is 3 times physical RAM, you need more RAM. 38 19
X-Window issues (overview) x11perf benchmark Increase priority of the X server using nice or renice, if it is already running a nice value of -10 is recommended nice - -10 /usr/x11r6/bin/x 39 X-Window issues (overview) Check that the configuration file represents the system accurately /etc/x11/xf86config-4 Decrease colour depth Decrease resolution Turn off unused modules e.g., if you don t use 3D comment out the GLCore and glx modules in the configuration file 40 20
X-Window issues (overview) Load less fonts Upgrade the X server Upgrade the hardware Tune the desktop environment X-client tuning is a whole performance domain in itself! Without X, 16-32MB memory should be fine With small window managers, 32-48MB needed Larger WMs may need 64-128MB RAM. 41 That s not all, folks! 42 21
Post-conditions To be familiar with some performance-tuning options To be able to select appropriate performance monitoring tools under Linux To be able to apply reasonable interpretation to the output of performance monitoring tools (under Linux). 43 Sources Images (in order) http://www.funnyfreepics.com/images/hello.jpg http://funnyfunnypictures.tripod.com/sitebuildercontent/sitebuilderpictures/image24.jpg http://www.superlaugh.com/pets/funny.jpg http://www.linux-for-less.com/cards/images/extreme_lg.jpg Other resources (first resource in particular) Fink, J.R., & M.D. Sherer, (2002). Linux Performance Tuning and Capacity Planning. SAMS: Indianapolis. Loukides, M. (1991). System Performance Tuning. O Reilly: Sebastapol, CA. 44 22