Performance monitoring. in the GNU/Linux environment. Linux is like a wigwam - no Windows, no Gates, Apache inside!

Transcription

1 1 Performance monitoring in the GNU/Linux environment Linux is like a wigwam - no Windows, no Gates, Apache inside! 2 1

2 Post-conditions To be familiar with some performance-tuning options To be able to select appropriate performance monitoring tools under Linux To be able to apply reasonable interpretation to the output of performance monitoring tools (under Linux). 3 References Fink, J.R., & M.D. Sherer, (2002). Linux Performance Tuning and Capacity Planning. SAMS: Indianapolis. Loukides, M. (1991). System Performance Tuning. O Reilly: Sebastapol, CA. 4 2

3 Reactive versus Proactive Tuning Reactive: observe a bottleneck and respond Proactive: predict a bottleneck (based on capacity planning) and anticipate needs 5 Typical targets of tuning CPU Memory Swap Disk X11 Application settings Networking but not a focus of this unit Other sound, etc. 6 3

4 Compute-bound vs. I/O-bound Compute-bound applications bound by CPU speed I/O-bound applications bound by I/O speed 7 Typical tuning activities Monitoring performance monitoring tools bench-marking tests Tuning hardware upgrades driver upgrades application tuning and management kernel configuration and compilation 8 4

5 Some Linux Monitoring Tools 9 free Statistics: total: used: free: Total available memory Total amount of memory used Total amount of free memory shared: Total shared memory being used buffers: Size of disk buffer cache cached: Amount of memory that has been cached to disk 10 5

6 free Note that Linux uses an aggressive caching policy which may cause surprise when observing memory usage. 11 free YIKES! $ free total used free shared buffers cached Mem: /+ buffers/cache: Swap: Used memory (no buffers/cache) Free memory (with buffers/cache) 12 6

7 uptime Statistics: The length of time that the system has been up. The average number of jobs in the run queue for the last 1 minute, 5 minutes, 10 minutes values over 1 indicate resource starvation 13 uptime Mean queue (10 mins) $ uptime 4:50am up 1:21, 1 user, load average: 0.07, 0.04, 0.01 Up for 1 hour, 21 minutes Mean queue (5 mins) Problems sleeping! Mean queue (last minute) What There can are you no warning say about signs; the the system system based is lightly on this loaded. output? 14 7

8 uptime $ uptime 5:52am up 2:23, 1 user, load average: 4.18, 3.01, 2.86 What The CPU can is you a bottleneck say about the under system current based loads, on and this there output? is evidence of backlog. 15 vmstat procs r: number of processes waiting to run b: number of processes interrupted from sleep mode w: number of processes swapped out and yet to run memory swpd: total amount of virtual memory in use free: total amount of free memory buff: total amount of buffered memory cache: amount of cache in use swap si: memory transferred from swap space into memory so: memory swapped to disk 16 8

9 vmstat io bi: disk block sent to disk devices (blocks / sec) bo: outbound disk activity system in: interrupts / sec (including CPU clocks) cs: context switches per second cpu us: % cycles on user processes sy: % cycles on system processes id: % cycles idle 17 vmstat $ vmstat procs memory swap io system cpu r b w swpd free buff cache si so bi bo in cs us sy id

10 vmstat $ vmstat 2 6 procs memory swap io system cpu r b w swpd free buff cache si so bi bo in cs us sy id The /proc pseudo file-system /proc is a pseudo file-system, providing a hook into the kernel. Typically it is used for reading various kernel parameters. It may also be used for changing such parameters (without the need to recompile the kernel!) 20 10

11 /proc/cpuinfo $ cat /proc/cpuinfo processor : 0 vendor_id : GenuineIntel cpu family : 6 model : 8 model name : Pentium III (Coppermine) stepping : 6 cpu MHz : cache size : 256 KB fdiv_bug : no hlt_bug : no sep_bug : no f00f_bug : no coma_bug : no fpu : yes fpu_exception : yes cpuid level : 2 wp : yes flags : fpu vme de pse tsc msr pae mce cx8 sep mtrr pge mca cmov pat pse36 mmx fxsr xmm bogomips : /proc/meminfo $ cat /proc/meminfo total: used: free: shared: buffers: cached: Mem: Swap: MemTotal: kb MemFree: kb MemShared: 0 kb Buffers: 2848 kb Cached: kb BigTotal: 0 kb BigFree: 0 kb SwapTotal: kb SwapFree: kb 22 11

12 Some other examples /proc/devices /proc/interrupts /proc/loadavg /proc/modules /proc/mounts /proc/partitions 23 bonnie Bonnie performs a disk I/O benchmark test: Writing to disk: Sequentially, one character at a time Sequentially, in larger blocks Reading from disk: Sequentially, one character at a time Sequentially, in larger blocks Random seeks 24 12

13 bonnie $ bonnie File './Bonnie.9318', size: Writing with putc()...done Rewriting...done Writing intelligently...done Reading with getc()...done Reading intelligently...done Seeker 1...Seeker 2...Seeker 3...start 'em...done...done...done Sequential Output Sequential Input-- --Random-- -Per Char- --Block--- -Rewrite-- -Per Char- --Block--- --Seeks--- Machine MB K/sec %CPU K/sec %CPU K/sec %CPU K/sec %CPU K/sec %CPU /sec %CPU top Status: Up time number of processes CPU states memory swap 26 13

14 top Per-process details: PID process ID number USER user name PRI priority NI nice value SIZE total, including code, data, and stack space (KB) RSS physical memory used SHARE shared memory used STATE sleeping (S), uninterrupted (D), runnable (R), stopped (T), zombie (Z) TIME CPU time since starting %CPU CPU time since last update %MEM share of physical memory COMMAND command name 27 top $ top 5:03am up 1:34, 1 user, load average: 0.00, 0.00, processes: 111 sleeping, 1 running, 0 zombie, 0 stopped CPU states: 1.5% user, 1.0% system, 0.0% nice, 97.3% idle Mem: K av, K used, 2652K free, 0K shrd, 6012K buff Swap: K av, 0K used, K free 52084K cached PID USER PRI NI SIZE RSS SHARE STAT %CPU %MEM TIME COMMAND 9268 tony R :00 top 1 root S :05 init 2 root SW :00 kflushd 3 root SW :00 kupdate 4 root SW :00 kswapd 5 root SW< :00 mdrecoveryd 32 root S :00 open 40 root S :11 wsmonitorbin 381 root SW :00 khubd 411 bin S :00 portmap 476 root S :00 automount 492 root S :00 automount 540 root S :00 apmd 556 root S :00 syslogd Processes swapped out 28 14

15 xload 29 xosview Process load average CPU usage: user, nice, sys, free Memory: user-allocated, buffer, cache, free Swap: in-use, free Paging: paging in, paging out, idle Disk: read, write, idle Interrupts: 0 to

16 Responses 31 CPU tuning Hardware solutions: Faster processor Multiprocessor environment especially if the offending applications fork into multiple processes or, if the application was specifically designed to optimise in SMP (shared memory multiprocessor ) environments Linux scales well to SMP (at least to 8 CPUs) Different architecture 32 16

17 CPU tuning Software solutions: Application configuration turn off capabilities that aren t being used System configuration turn off competing processes that aren t useful 33 CPU tuning Kernel solutions: Not a lot of options can define the specific CPU type and features in the build which enables specific optimisations to be used base kernel usually assumes low-end capabilities some kernel patches may address the issue in other ways e.g., low latency patch for music 34 17

18 Memory tuning Hardware solutions: More memory Faster memory newer memory is often much faster Cache size hardware caches generally improve access, but over-large caches may involve longer look-up times 35 Memory tuning Software solutions: Application choice bad coding can result in performance that appears to be due to a lack of memory user threads vs. POSIX threads Addressable memory Intel chips often impose a 4GB-addressable memory limit Processor Address Extension extends this, but requires tuning. Otherwise, consider a 64b architecture 36 18

19 Memory tuning Kernel solutions: Contiguous memory needs may respond to a bigmem kernel patch Monolithic versus modular kernels Modular kernels only load in the relevant functionality as it is required. If you don t use a particular piece of hardware often, for example, then having that driver modular will conserve kernel memory space 37 Memory tuning Swap-space planning: Consider programs that will run concurrently Determine their memory requirements and add a few extra MB Subtract physical memory from this amount the remainder is the minimum swap space if this is 3 times physical RAM, you need more RAM

20 X-Window issues (overview) x11perf benchmark Increase priority of the X server using nice or renice, if it is already running a nice value of -10 is recommended nice /usr/x11r6/bin/x 39 X-Window issues (overview) Check that the configuration file represents the system accurately /etc/x11/xf86config-4 Decrease colour depth Decrease resolution Turn off unused modules e.g., if you don t use 3D comment out the GLCore and glx modules in the configuration file 40 20

21 X-Window issues (overview) Load less fonts Upgrade the X server Upgrade the hardware Tune the desktop environment X-client tuning is a whole performance domain in itself! Without X, 16-32MB memory should be fine With small window managers, 32-48MB needed Larger WMs may need MB RAM. 41 That s not all, folks! 42 21

22 Post-conditions To be familiar with some performance-tuning options To be able to select appropriate performance monitoring tools under Linux To be able to apply reasonable interpretation to the output of performance monitoring tools (under Linux). 43 Sources Images (in order) Other resources (first resource in particular) Fink, J.R., & M.D. Sherer, (2002). Linux Performance Tuning and Capacity Planning. SAMS: Indianapolis. Loukides, M. (1991). System Performance Tuning. O Reilly: Sebastapol, CA