Linux Performance Profiling and Monitoring. Georg Schönberger

Transcription

1 Linux Performance Profiling and Monitoring Georg Schönberger

2 Thomas-Krenn.AG A server manufacturer in Bavaria Well visited knowledge base, Thomas-Krenn Wiki 2

3 Agenda Collect Statistics Sysstat Package dstat nicstat /proc raw counters sar and sadc Watch online 3 top htop iotop iftop Tracing perfevents ftrace perf-tools Flame graphs LXC MySQL Slow Query Log innotop

4 find / -type f -name statistics

5 From Brendan Gregg,

6 From Brendan Gregg,

7 25

8 mpstat Without Interval/Count since system startup CPU usage per Core Including Hyperthreading lscpu grep -E 'core socket' Thread(s) per core: 2 Core(s) per socket: 2 Check how well usage is balanced mpstat -P ALL Linux generic (X22) x8664 (4 CPU) 14:28:21 CPU %usr %nice %sys %iowait %irq %soft 14:28:21 all 11,59,9 3,62,3,4 14:28:21 6,45,5 1,87,4,7 14:28: ,44,11 5,56,1 14:28: ,15,14 5,55,3,5 14:28: ,27,11 4,89,1,2 8 %steal %guest %gnice %idle 84,64 91,53 77,89 77,8 78,7

9 mpstat mpstat -P ALL 1 2 Linux generic (X22) x :24:44 15:24:45 15:24:45 15:24:45 15:24:45 15:24:45 CPU all %usr 5,21 1,43 11,88 4,12 3,3 %nice %sys %iowait 7,12 17,81 1,43 23,76 64,36 1,3 1,1 %irq 15:24:45 15:24:46 15:24:46 15:24:46 15:24:46 15:24:46 CPU all %usr 5,74 2,99 11,88 6, 1,1 %nice %sys %iowait 7,1 17,76 1,49 23,76 64,36 1, 1,1 %irq (4 CPU) %soft %steal %guest,27 2,86 Core not idle 1 is also and deals with %iowait %soft,55 2,99 %steal %guest %gnice %idle 69,59 94,29 94,85 95,96 %gnice %idle 68,85 92,54 93, 97,98

10 vmstat High Level Statistics about Virtual memory Swap/Paging I/O statistics System interrupts and context switches CPU statistics vmstat 1 procs memory swap io---- -system cpu----r b swpd free buff cache si so bi bo in cs us sy id wa st

11 vmstat Memory statistics buff Raw disk blocks like filesystem metadata cache Memory used for data information, pages with actual contents $ vmstat 1 procs memory swap io---- -system cpu----r b swpd free buff cache si so bi bo in cs us sy id wa st $ free total used free shared buffers cached Mem: /+ buffers/cache: Swap:

12 vmstat Process related fields r The number of runnable processes (running or waiting for run time) If high indicator for saturation b The number of processes in uninterruptible sleep Mostly waiting for I/O vmstat 1 procs memory swap io---- -system cpu----r b swpd free buff cache si so bi bo in cs us sy id wa st Processes doing I/O [...] can bei in waiting state $ ps -eo ppid,pid,user,stat,pcpu,comm,wchan:32 grep ext4 [...] Kernel function root Ds 3.2 fio ext4filewrite is sleeping on root Ds 3.2 fio ext4filewrite root Ds 3.2 fio ext4filewrite 12 process

13 vmstat plots Drawing interrupts and context switches 13

14 But we are not satisfied with summaries and overviews...

15 What is PID 959 doing?

16 pidstat Report statistics for tasks being managed by kernel CPU bound identify peak activity $ top -b -n 1 -d 2 -o %CPU head [...] PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND 959 gschoenb R 96,9,3 :2.14 python 1 root S,, :2.35 init $ pidstat -p 959 -u 1 -l Linux generic (X22) x8664 (4 CPU) 1:11:4 UID 1:11:5 1 -i matrix.in 1:11:6 1 -i matrix.in 1:11:7 1 -i matrix.in 16 PID 959 %usr %system 1 %guest %CPU 1 CPU Command python ijk-matrix.py python ijk-matrix.py 959 Even check 1 command 1 line arguments! python ijk-matrix.py

17 pidstat I/O bound device report mpstat -P ALL 1 1:25:31 CPU %usr %nice %sys %iowait %irq %soft %steal %guest %gnice 1:25:32 all 14,88 9,4 13,84 1,4 1:25:32 22,45 1,2 1:25: ,73 34,31 51,96 1:25: ,86 3,57 Which process 1:25:32 3 6,12 is causing %iowait? pidstat -d 1 Linux generic (X22) x8664 (4 CPU) 1:26:35 1:26:36 1:26:36 1:26:36 1:26:36 UID PID kbrd/s kbwr/s kbccwr/s 233, ,15 223,8 1284,62 Device report reveals command and I/O 17 Command fio fio fio fio %idle 6,84 76,53 78,57 93,88

18 pidstat How much memory is PID 8461 using? Major faults require I/O operations, good indicator you need more RAM! pidstat -r -p Linux generic (X22) :9:6 1:9:7 1:9:8 1:9:9 Average: UID PID minflt/s 8, 11, 23, 14, x8664 (4 CPU) majflt/s Minor and major page faults 18 VSZ RSS %MEM 9,76 9,76 9,77 9,77 Command firefox firefox firefox firefox Current used share of physical memory

19 iostat I/O subsystem statistics CPU or device utilization report Without argument summary since boot Skip that with -y option iostat Linux generic (X22) avg-cpu: Device: sda 19 %user 16,16 %nice %system %iowait,9 4,79,46 tps 83,8 kbread/s 41,64 x8664 %steal kbwrtn/s 531,43 (4 CPU) %idle 78,5 kbread kbwrtn

20 iostat CPU util report %iowait Not really reliable %iowait is some kind of %idle time taskset 1 fio rw=randwrite [...] & iostat -y -c 1 3 [ ] avg-cpu: %user %nice %system %iowait %steal 17,32 6,56 13,65 taskset 1 sh -c "while true; do true; done" & iostat -y -c 1 3 avg-cpu: %user %nice %system %iowait %steal 35,59 7,2 2 %idle 62,47 %idle 57,39

21 iostat Extended device util report %util man iostat for devices serving requests in parallel, such as RAID arrays and modern SSDs, this number does not reflect their performance limits. In theory 21 94,4% util 2332 IOPS 99,6% util 243 IOPS

22 iostat iostat -y -d -x 1 3 Linux generic (X22) x8664 (4 CPU) Device: rrqm/s wrqm/s r/s w/s rkb/s wkb/s avgrq-sz avgqu-sz rawait wawait svctm %util sda 2, 2332, 92136, 8, 2,9,13,4 94,4 iostat -y -d -x 1 3 Linux generic (X22) x8664 (4 CPU) await Device: rrqm/s wrqm/s rawait wawait svctm %util sda 2917, 3,15,2 99,6 await 22 r/s w/s 43175, rkb/s wkb/s avgrq-sz avgqu-sz ,55,13 135,75 Only 5% util increase, but IOPS nearly doubled! 3,15

23 23

24 iostat avgqu-sz Avg. queue length of requests issued (delta[timeinqueue] / interval) / 1. timeinqueue Requets waiting for device, effected by inflight await Avg. time requests being served delta[readticks + writeticks] / delta[readios + writeios] ticks also effected by inflight Therefore serving more requests while await is not increasing, is a good performance indicator 24 - Monitoring IO Performance using iostat and pt-diskstats - Block layer statistics

25 25 4

26 dstat Combines several classic tools 26 Prints metrics and uses colors Has a plugin concept

27 nicstat Print network device statistics 27 %Util depends on speed and duplex mode Sat also takes errors into account nicstat -l Check if your network is Int Loopback Mbit/s Duplex State saturated, Drops can be vboxnet No unkn up an indicator! eth No 1 full up lo Yes unkn up wlan No unkn up nicstat -i eth 1 5 Time Int rkb/s wkb/s rpk/s wpk/s ravs wavs %Util Sat 14:52:21 eth :52:22 eth :52:23 eth nicstat -i eth -t :57:36 InKB OutKB InSeg OutSeg Reset AttF %ReTX InConn OutCon Drops TCP :57:37 InKB OutKB InSeg OutSeg Reset AttF %ReTX InConn OutCon Drops TCP

28 Do you have a basic chronicle of your system's data? Yes No

29 sar It's easy with system activity reporter 29 sar, sadc, sa1 and sa2, sadf

30 ksar Mitigates character encoding and number format problems LCALL=POSIX sar -A -f sa1 > ksar.out.txt 3

31 The /proc filesystem /proc knows everything $ sudo procinfo -n 2 Memory: Total RAM: Swap: Bootup: Thu Apr user : nice : system: IOwait: hw irq: sw irq: idle : uptime: [...] 31 Used :16: :45:48.75 :2:35.9 1:48:13.52 ::48.3 ::.19 :1: d 17:48: d 6:59: %.1% 3.6%.%.%.% 84.5% Free Buffers Load average: / page in : page out: page act: page dea: page flt: swap in : swap out: context :

32 Overview Collect Statistics Sysstat Package dstat nicstat /proc raw counters sar and sadc Watch online 32 Tracing perfevents ftrace perf-tools Flame graphs strace top LXC htop iotop MySQL iftop Slow Query Log innotop

33 top System summary at beginning Per process metrics afterwards 1, 5 and 15 min load average Default sorted by CPU usage $ top -b -n 1 head -15 top - 15:33:5 up 3 days, 19:2, 3 users, load average:.13,.51,.59 Tasks: 668 total, 1 running, 667 sleeping, stopped, zombie Cpu(s): 1.5%us,.3%sy,.1%ni, 98.1%id,.%wa,.%hi,.%si,.%st Mem: k total, k used, k free, 1612k buffers Swap: k total, k used, k free, k cached 33 PID USER root 1193 gschoenb root 1997 www-data 775 root 3712 root 1287 root root PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND S :55.72 ossec-syscheckd R 4. :.2 top S 4. 7:38.88 openvpn 2 197m 61m 52m S 4. :6.18 apache2 2 S 2. 8:3.13 md3raid S 2. 22:45.85 kipmi -3 S 2. 6:2.3 drbd2asender 2 S 2. 12:4.19 drbd1receiver

34 top Memory usage VIRT The total size of virtual memory for the process Also including e.g. not already mapped heap or swap RES How many blocks are really allocated and mapped to address space resident Also includes file-backed memory (like shared libraries, mmap) Can be used concurrently by processes SHR is shared or file-backed memory RES SHR = anon mem (malloc) $ cat /proc/17692/statm

35 top Can consume resources on it's own Toggle f and select fields, e.g. SWAP -u let's you see processes from a user Toggle k to kill a PID Toggle r to renice a PID But top can miss short living processes high %CPU so what? 35 Keep an eye on the tracing part

36 htop Super advanced top 36 Uses colors, views can be customized

37 iotop Simple top like I/O monitor Which process is causing I/O Filtering specific PID is possible Show writes, reads and command in realtime iotop -o -b Total DISK READ :. B/s Total DISK WRITE : M/s Actual DISK READ:. B/s Actual DISK WRITE: 63.9 M/s TID PRIO USER DISK READ DISK WRITE SWAPIN IO COMMAND be/4 root. B/s M/s. % % fio --rw=randwrite --name=test --filename=test.fio --size=3m --direct=1 --bs=4k be/4 gschoenb. B/s K/s. %. % firefox [mozstorage 1] iotop -o -b Total DISK READ : 69.2 M/s Total DISK WRITE : K/s Actual DISK READ: 69.2 M/s Actual DISK WRITE: K/s TID PRIO USER DISK READ DISK WRITE SWAPIN IO COMMAND be/4 root 69.2 M/s. B/s. % % fio --rw=read --name=test --filename=test.fio --size=3m --direct=1 --bs=8k 37

38 Bandwidth live usage iftop Per interface usage nethogs Per proces NetHogs version.8. PID ? USER gschoenb root gschoenb root TOTAL 38 PROGRAM /usr/lib/firefox/firefox /usr/bin/ssh evolution unknown TCP DEV eth eth eth SENT RECEIVED.194 KB/sec. KB/sec. KB/sec. KB/sec KB/sec

39 25 13

40 Overview Collect Statistics Sysstat Package dstat nicstat /proc raw counters sar and sadc Watch online 4 top htop iotop iftop Tracing perfevents ftrace perf-tools Flame graphs LXC MySQL Slow Query Log innotop

41 whereis tracing

42 Profiling Create profile about usage characteristics Count specific samples/events Count objects Next slides focus on system profiling ftrace perfevents and perf Collecting statistics about tracepoints 42 Lines of kernel code with defined event

43 ftrace Part of the Linux kernel since (28) What is going on inside the kernel Common task is to trace events With ftrace configured, only debugfs is required cat /proc/sys/kernel/ftraceenabled 1 mount grep debug none on /sys/kernel/debug type debugfs (rw) /sys/kernel/debug/tracing cat availabletracers blk mmiotrace functiongraph wakeuprt wakeup function nop 43

44 ftrace Interact with files in sys Easier with trace-cmd!/bin/bash DEBUGFS=`grep debugfs /proc/mounts awk '{ print $2; }'` echo echo echo exec echo 44 $$ > $DEBUGFS/tracing/setftracepid function > $DEBUGFS/tracing/currenttracer 1 > $DEBUGFS/tracing/tracingon $* > $DEBUGFS/tracing/tracingon

45 perfevents and perf Used to be called performance counters for Linux A lot of updates for kernel CPU performance counters, tracepoints, kprobes and uprobes Per package with linux-tools-common which perf /usr/bin/perf dpkg -S /usr/bin/perf linux-tools-common: /usr/bin/perf 45

46 perf list perf list 46 Shows supported events perf list wc -l 1779 perf list grep Hardware cpu-cycles OR cycles instructions cache-references cache-misses branch-instructions OR branches branch-misses bus-cycles stalled-cycles-frontend OR idle-cycles-frontend stalled-cycles-backend OR idle-cycles-backend ref-cycles L1-dcache-loads L1-dcache-load-misses L1-dcache-stores L1-dcache-store-misses This also includes static tracepoints [Hardware [Hardware [Hardware [Hardware [Hardware [Hardware [Hardware [Hardware [Hardware [Hardware [Hardware [Hardware [Hardware [Hardware event] event] event] event] event] event] event] event] event] event] cache event] cache event] cache event] cache event]

47 Raw CPU counters Each CPU has it's own raw counters They should be documented by the hardware manufacturer libpfm4 is a nice way to find raw masks perf list grep rnnn rnnn [Raw hardware event descriptor] git clone git://perfmon2.git.sourceforge.net/gitroot/perfmon2/libpfm4 cd libpfm4 make cd examples/./showevtinfo grep LLC grep MISSES Name : LLCMISSES Now we collect last [...] level cache misses./checkevents LLCMISSES grep Codes with the raw mask Codes : x53412e perf stat -e r53412e sleep 5 47

48 Tracepoints perf also has trace functionalities perf list grep -i trace wc -l 1716 Filesystem Block layer Syscalls perf stat -e 'syscalls:sysentermmap'./helloworld.out Hello world! Performance counter stats for './helloworld.out': 8 syscalls:sysentermmap seconds time elapsed 48

49 perf stat Get a counter summary Easy to compare performance of different algorithms perf stat python numpy-matrix.py -i matrix.in Performance counter stats for 'python numpy-matrix.py -i matrix.in': 576, task-clock (msec) context-switches cpu-migrations page-faults cycles stalled-cycles-frontend stalled-cycles-backend instructions branches branch-misses, seconds time elapsed 49,93,554 7,17 3,27 47,68% 26,31% 1,6,3 995,26,63% CPUs utilized K/sec K/sec M/sec GHz frontend cycles idle backend cycles idle insns per cycle stalled cycles per insn M/sec of all branches [82,63%] [83,75%] [67,26%] [84,28%] [83,78%] [82,7%]

50 perf record Record samples to a file Can be post-processed with perf report -a records on all CPUs -g records call graphs Install debug symbols perf record -a -g sleep 5 [ perf record: Woken up 4 times to write data ] [ perf record: Captured and wrote MB perf.data (~94254 samples) ] Nice way to record what's currently running on all CPUs 5

51 perf report Displays profile of a record 51 Can be sorted and or filtered Shows all samples

52 52 perf report -i perf.data.dd --stdio --showcpuutilization --sort comm,dso [...] Overhead sys usr Command Shared Object % 95.%.% dd [kernel.kallsyms] %-- aesnienc1 ablkencrypt ablkencrypt Command and shared cryptscatterlist cryptextent ecryptfsencryptpage Traced method ecryptfswriteend genericfilebufferedwrite genericfileaiowrite genericfileaiowrite dosyncwrite vfswrite syswrite systemcallfastpath GIlibcwrite x415f65643d %-- condresched dd writes data %-- ext4dirtyinode markinodedirty genericwriteend ext4dawriteend genericfilebufferedwrite object

53 perf-tools By Brendan Gregg Mostly quick hacks, read Warnings! Nice, this are simple bash scripts! Using perfevents and ftrace Good examples what can be done with perf and ftrace 53 iosnoop Shows I/O access for commands, including latency cachestat Linux page cache hit/miss statistics functrace Count kernel functions matching wildcards

54 19 25

55 view flamegraph

56 Flamegraph Visualization how resources are distributed among code Powered 56

57 Flamegraph perf record -g dd if=/dev/zero of=test.data count=1 bs=1m mv perf.data perf.data.dd perf script -i perf.data.dd./flamegraph/stackcollapse-perf.pl > out.dd.folded./flamegraph/flamegraph.pl out.dd.folded > out.perf.dd.svg Flame Graph nativewritemsrsafe intelpmuenableall x86pmuenable perfpmuenable per.. perfeventcontextschedin perfeventcomm settaskcomm setupnewexec loadelfbinary searchbinaryhandler doexecvecommon.isra.22 sysexecve stubexecve [unknown] : aesnienc1 ablkencrypt a.. ablkencrypt all.. cryptscatterlist p.. cryptextent gra.. ecryptfsencryptpage ecr.. ecryptfswriteend genericfilebufferedwrite genericfileaiowrite genericfileaiowrite dosyncwrite vfswrite syswrite systemcallfastpath GIlibcwrite [unknown] dd co.. ext.. ext.. e.. co.. ext.. ext.. mem.. ext4dirt.. b.. ext.. jbd.. markin.. blo.. b.. e.. genericwrite.. cop.. ext4daw.. ext4dawrite.. genericfilebufferedwrite sca.. genericfileaiowrite blk.. genericfileaiowrite cbc.. ext4filewrite dosyncwrite vfswrite kernelwrite ecryptfswritelower com.. mut.. app.. ext.. sec.. do.. rw.. vfswrite kernelwr.. ecryptfs.. ecryptfs.. fin.. fil.. d.. han.. d.. do.. pag.. dl.. khu.. exp.. set.. loa.. sea.. do.. sys.. stu.. r.. cle.. get.. allocp.. allocpag.. handlemm.. dopage.. dopagef.. pagefault readzero.. readzero vfsread sysread systemca.. GIli.. nl.. pag.. unm.. unm.. unm.. exi.. mmput do.. do.. sys.. sys..

58 tail -n 2 /special/lxc

59 Linux Container Lightweight virtual machines using features provided by a modern Linux kernel cgroups Aggregate or partition tasks and their children to hierarchical groups to isolate resources namespaces Wrap a resource in an abstraction so that it appears to processes they have their own isolated resource Each container shares the kernel running on the host 59 Some may refer to it as native performance

60 Linux Container cgroups are divided into subsystems, e.g. cpusets blkio memory Image from Boden Russel, 6

61 Linux Container cgroup created per container per subsystem lxc-ls --fancy NAME STATE IPV4 IPV6 GROUPS AUTOSTART ubuntu1 RUNNING NO lxc-info -n ubuntu1 Name: ubuntu1 State: RUNNING PID: 7548 IP: CPU use: 1.8 seconds BlkIO use: MiB Memory use: 3.85 MiB KMem use: bytes Link: vethc8tjut TX bytes: 3.33 KiB RX bytes: 3.49 KiB Total bytes: 6.82 KiB 61

62 Linux Container lxc-info takes cgroups into account Value Origin CPU use lxc-cgroup -n ubuntu1 cpuacct.usage BlkIO use lxc-cgroup -n ubuntu1 blkio.throttle.ioservicebytes Memory use lxc-cgroup -n ubuntu1 memory.usageinbytes KMem use lxc-cgroup -n ubuntu1 memory.kmem.usageinbytes Link cat /sys/class/net/vethep3qm/statistics/*bytes cgroups providing further info 62 memory.stat memory.failcnt cpuset.cpus

63 Linux Container lxc-top monitors container overall usage lxc-top Container Name ubuntu1 ubuntu2 TOTAL 2 of 2 CPU Used CPU Sys CPU User BlkIO Total MB 1.61 MB MB Mem Used MB MB MB Traditional tools without lxcfs do not work! root@host lxc-cgroup -n ubuntu1 memory.limitinbytes root@container free -h total used free shared buffers Mem: 489M 22M 287M 488K 26M 63 cached 115M

64 sh -c "while true; do mysql; done"

65 MySQL Percona provides a lot of good tools First step, generate a summery pt-mysql-summary Percona Toolkit MySQL Summary Report Instances Port Data Directory Nice OOM Socket ===== ========================== ==== === ====== /var/lib/mysql /var/lib/mysql/mysql.sock MySQL Executable Path to executable /usr/sbin/mysqld Has symbols Yes Report On Port 336 User root@localhost Time :49:9 (CEST) Hostname mysql1 Databases 15 Datadir /var/lib/mysql/ [...] 65

66 MySQL Extended status prints also counters Can be monitored with pmp-check-mysql-status Plugin mysqladmin ext wc -l 345 mysqladmin ext grep Threadsrunning Threadsrunning 3 mysqladmin ext grep Innodbbufferpoolpagesfree Innodbbufferpoolpagesfree Slow Query log 66 Queries exceeding a specific runtime OFF by default, runtime and log file must be defined Query Cache is ignored

67 MySQL Easy way to log all queries longquerytime pt-query-digest 67 Process slow log and generate report pt-query-digest mysql-slow.log [...] Attribute total min max ============ ======= ======= ======= Exec time s 9s 419s Lock time 15s 3s Rows sent 5.5M 2.65M Rows examine 3.23G M Query size M M Profile Rank Query ID Response time ==== ================== ================ 1 x7a8eb8c13a4a % 2 xa45c5fb6d66119b % 3 x67a347a df % 4 xd7a9797e % avg 95% stddev median ======= ======= ======= ======= 51s 151s 45s 42s 4ms 71ms k 1.69M k k 2.49M 5.53M 44.37k 35.91k 68.96k 136.3k 1.29k Calls R/Call ===== ======== V/M ===== Item ========== SELECT SELECT SELECT SELECT

68 MySQL innotop Live analysis of SQL queries 68 Sort by execution time Not only for innodb

69 Thanks for your

70 Backup slides

71 iostat CPU util report %iowait With iostat, avg. over CPUs Take mpstat into account iostat -y -c 1 3 Linux generic (X22) x8664 (4 CPU) %user %nice %system %iowait %steal %idle 11,41 9,28 12,2 67,11 mpstat -P ALL 1 Linux generic (X22) x8664 (4 CPU) avg-cpu: 12:4:47 12:4:48 12:4:48 12:4:48 12:4:48 12:4:48 71 CPU all %usr 1,8 13,68 18,81 1,43 4,4 %nice %sys %iowait 7,36 16,8 25,26 61,5,99 1,43 1,1 %irq %soft,82 4,29 %steal %guest %gnice %idle 65,67 8,2 92,86 94,95

72 The /proc filesystem /proc knows everything Obviously ps uses /proc $ cat /proc/self/status head Name:cat State: R (running) Pid: Uid: Gid: Or values from free $ cat /proc/meminfo grep Swap SwapCached: kb SwapTotal: kb SwapFree: 1484 kb $ cat /proc/meminfo grep -E 'Buffer Cache' Buffers: kb Cached: kb SwapCached: kb 72

73 varnishstat and top Good example for application based statistics varnishstat -1 wc -l 311 varnishstat -1 grep -E 'cache nuked' MAIN.cachehit Cache hits MAIN.cachehitpass Cache hits for pass MAIN.cachemiss Cache misses MAIN.nlrunuked. Number of LRU nuked objects varnishtop -i ReqUR ReqURL /de/wikide/api.php 384. ReqURL /favicon.ico 321. ReqURL /en/wikien/api.php 282. ReqURL /res/font/fsmeweb/bold/fsmeweb-bold.woff 282. ReqURL /res/font/fsmeweb/regular/fsmeweb-regular.woff 28. ReqURL /de/wikide/skins/tkskin/images/thomas-krenn-logo-grey.png 278. ReqURL /de/wikide/skins/tkskin/images/tklogo2x9.png 73

74 blktrace blktrace Captures I/O traces Trace is stored in a binary format blkparse Reads traces recorded by blktrace btt 74 blktrace timeline Post-processing tool

75 8, , , , Actions 5 61 A8, IO was remapped to a different device 6 request Q8, IO handled by queue code 61 G8, Getrequest P8, Plug request I 8, IO inserted onto queue 9 request U8, Unplug request D IO issued to driver [...] 8, , , , , , , Pr oc es s PI D Ac tio RW n BS St ar tb lo ck p st am Ti m e nc e ue Se q De vi ce CP U + of bl oc k s /var/log head -n 1 syslog.1 blktrace -d /dev/sda (run parallel) blkparse -i sda.blktrace.[...] A Q G I D A Q G I D WS WS WS WS WS WS WS WS WS WS <- (8,3) [kworker/:3] + RWBS 8 [kworker/:3] read + R8 [kworker/:3] write +W 8 [kworker/:3] + D8 <- block (8,3)discard B barrier operation + 8 [kworker/:3] synchronous operation + S8 [kworker/:3] + 8 [kworker/:3] + 8 [kworker/:3] A Q G P I U D R R R N R N R [head] [head] <- (8,3) [head] + 32 [head] + 32 [head] + 32 [head]

76 CPU (sda): Reads Queued: Read Dispatches: Reads Requeued: per CPU Reads Completed: Read Merges: details Read depth: IO unplugs: CPU1 (sda): Reads Queued: Read Dispatches: Reads Requeued: Reads Completed: Read Merges: Read depth: IO unplugs: CPU3 (sda): Reads Queued: Read Dispatches: Reads Requeued: Reads Completed: Read Merges: Read depth: IO unplugs: Total (sda): Reads Queued: Read Dispatches: Reads Requeued: Reads Completed: Read Merges: IO unplugs:,,,, 1,, 1,, 1 KiB KiB Writes Queued: Write Dispatches: Writes Requeued: KiB Writes Completed: KiB Write Merges: writes submitted Write depth: on this Timer CPU unplugs: KiB KiB 16KiB KiB 1, 16KiB 1, 16KiB, KiB per, devicekib 1details 1 1, 1, 1,, 1 16KiB 16KiB 16KiB KiB Throughput (R/W): 37KiB/s / 74KiB/s Events (sda): 62 entries Skips: forward (.%) Writes Queued: Write Dispatches: Writes Requeued: Writes Completed: Write Merges: Write depth: Timer unplugs: 7, 7, 5,, 7 28KiB 28KiB 2, 1, 5,, 7 4KiB 4KiB writes completed, this, CPU Writes Queued: Write Dispatches:on Writes Requeued: Writes Completed: Write Merges: Write depth: Timer unplugs: Writes Queued: Write Dispatches: Writes Requeued: Writes Completed: Write Merges: Timer unplugs:,, 7 9, 8, 1,, avg. throughput 2KiB KiB 12KiB KiB KiB KiB KiB KiB 32KiB 32KiB 32KiB KiB

77 btt -i sda.blktrace. ==================== All Devices ==================== ALL MIN AVG MAX N Q2Q Q2G G2I I2D D2C Q2C [...] D2C Driver and device time the average time from when the actual IO was issued to the driver until is completed (completion trace) back to the block IO layer. Q2C Measures the times for the complete life cycle of IOs during the run.

78 MySQL pt-query-digest 78 Query 1:. QPS,.1x concurrency, ID x67a347a df at byte This item is included in the report because it matches --limit. Scores: V/M = Time range: ::26 to :1:33 Attribute pct total min max avg 95% stddev median ============ === ======= ======= ======= ======= ======= ======= ======= Count Exec time s 3s 2629s 41s 13s 84s 17s Lock time 531ms 599us 47us 93us 28us 38us Rows sent G M 7.53M 46.53M 13.84M k Rows examine G M 7.53M 46.53M 13.84M k Query size k String: Databases XXXXX (1159/9%)... 1 more Hosts localhost (9524/84%), XXXXX (1743/15%) Users XXXXX (9738/86%), XXXXX (1529/13%) Querytime distribution 1us 1ms 1ms 1ms 1s 1s+

79 MySQL Performance Schema Per default on in 5.6, older profiling commands are deprecated with A structured way in SQL to get timing information Runtime and query execution statistics Sys scchema (pshelper) provides a more comfortable way > select * from schematablestatistics where tableschema='sbtest' limit 1 \G *************************** 1. row *************************** tableschema: sbtest tablename: sbtest rowsfetched: [...] 79