GSE z/os Systems Working Group s390-tools

Bertrand Jouniaux 2011/06/15 GSE z/os Systems Working Group s390-tools

Based on s390-tools: The Swiss Army Knife for Linux on System z System Administration Hans-Joachim Picht Running Linux on System z as a z/vm Guest: Useful Things to Know Hans-Joachim Picht 2

S390-TOOLS: THE SWISS ARMY KNIFE FOR LINUX ON SYSTEM Z SYSTEM ADMINISTRATION 3

The Linux on System z Life Cycle IPL INIT Maintain Debug You need the s390-tool at every stage of the life cycle of a Linux guest (LPAR & z/vm) 4

What is the s390-tools package? s390-tools is a package with a set of user space utilities to be used with the Linux on System z distributions It is the essential tool box for Linux on System z It contains everything from the boot loader to dump related tools for system crash analysis The latest version is 1.13.0 and was released in 2011/05/19 This software package is contained in all major (and IBM supported) distributions which support s390 RedHat Enterprise Linux 4 (s390-tools-1.3.2) RedHat Enterprise Linux 5 (s390-tools-1.8.1 since RHEL 5.4) SuSE Linux Enterprise Server 10 (s390-tools-1.6.3 since SLES 10 SP2 ) SuSE Linux Enterprise Server 11 (s390-tools-1.8.0) Website: http://www.ibm.com/developerworks/linux/linux390/s390-tools.html 5

What is the s390-tools package? More than 50 tools exclusively dedicated to Linux on z 6

The 10 tools boxes 7 Change chccwdev chchp chreipl chshut chzcrypt Display lscss lschp lsdasd lsluns lsqeth lsreipl lsshut lstape lszcrypt lszfcp DASD dasdfmt dasdinfo dasdview fdasd tunedasd MONITOR mon_fsstatd mon_procd ziomon Network ip_watcher osasnmpd qetharp qethconf Tape tape390_crypt tape390_display Debug dbginfo dumpconf zfcpdump zfcpdbf zgetdump scsi_logging_level zvm vmconvert vmcp vmur Misc cpuplugd iucvconn Iucvtty ts-shell Boot zipl

Selection of 14 essential tools 8 Change chccwdev chchp chreipl chshut chzcrypt Display lscss lschp lsdasd lsluns lsqeth lsreipl lsshut lstape lszcrypt lszfcp DASD dasdfmt dasdinfo dasdview fdasd tunedasd MONITOR mon_fsstatd mon_procd ziomon Network ip_watcher osasnmpd qetharp qethconf Tape tape390_crypt tape390_display Debug dbginfo dumpconf zfcpdump zfcpdbf zgetdump scsi_logging_level zvm vmconvert vmcp vmur Misc cpuplugd iucvconn Iucvtty ts-shell Boot zipl

CHANGE chccwdev chchp chreipl chshut chzcrypt chccwdev The chccwdev command with the -e flag activates the unused DASD. And with the -d flag deactivate a DASD Example # chccwdev -e 0.0.0200 Setting device 0.0.0200 online 9

CHANGE chccwdev chchp chreipl chshut chzcrypt Chreipl (Shutdown action tools ) Configure a disk or change a an entry in the Boot menu for the next boot cycle. Example ~> chreipl node /dev/dasda ~> chreipl node /dev/sda 10

CHANGE chccwdev chchp chreipl chshut chzcrypt Chshut (Shutdown action tools ) Change the entries in /sys/firmware to configure the shutdown behaviour Example 11 chshut halt ipl chshut halt vmcmd LOGOFF chshut poff vmcmd "MSG MASTER Going down" vmcmd "LOGOFF"

CHANGE chccwdev chchp chreipl chshut chzcrypt Chzcrypt (modify zcrypt configuration) Use the chzcrypt command to configure cryptographic adapters managed by zcrypt and modify zcrypt s AP bus attributes. Example chzcrypt -e 0 1 4 5 12 chzcrypt -d -a 12

DASD dasdfmt dasdinfo dasdview fdasd tunedasd Dasdfmt (formatting of DASD (ECKD) disk drives) dasdfmt formats a DASD (ECKD) disk drive to prepare it for usage with Linux. The device is the node of the device (e.g. /dev/dasda ). Any device node created by udev for kernel 2.6 can be used (e.g. /dev/dasd/0.0.b100/disc ). Before formatting a DASD, you must bring it online with the chccwdev command. Example dasdfmt -b 4096 -y -f /dev/dasda 13

DASD dasdfmt dasdinfo dasdview fdasd tunedasd Dasdinfo dasdinfo displays specific information about a specified DASD device. It is normally called from a udev rule, to provide udev with a unique id string and additional information (type, serial) for an S390 DASD drive. Udev can use this information to create symlinks in /dev/disk/by-id and /dev/disk/ by-label to the real device node. Example dasdinfo -u -i 0.0.e910 dasdinfo -u -b dasdb dasdinfo -u -d /dev/dasdb All three examples should return the same unique ID for the same DASD device, e.g. IBM.75000000092461.e900.10. 14 In case this uid is not available, dasdinfo will return the volume label instead, e.g. 0XE910.

Debug dbginfo dumpconf zfcpdump zfcpdbf zgetdump scsi_logging_level Dbginfo.sh dbginfo.sh is a script to collect various system related files, for debugging purposes. It generates a tar-archive which can be attached to PMRs / Bugzilla entries. Please use the data from this tool is you open a Bugzilla (Novell/ RedHat) or a PMR. 15

Debug dbginfo dumpconf zfcpdump zfcpdbf zgetdump scsi_logging_level dumpconf Configure automatic dump for Linux on z. The command can be installed as service script under /etc/init.d/ dumpconf or can be called manually. Start service: service dumpconf start It reads the configuration file /etc/sysconfig/dumpconf. 16

Debug dbginfo dumpconf zfcpdump zfcpdbf zgetdump scsi_logging_level dumpconf Example Example configuration for FCP dump device (SCSI disk): ON_PANIC=dump DUMP_TYPE=fcp DEVICE=0.0.4714 Example configuration for re-ipl without taking a dump, if a kernel panic occurs: ON_PANIC=reipl Example of executing a CP command, and rebooting from device 4711 if a kernel panic occurs (MASTER is a VM Guest Name): ON_PANIC=vmcmd VMCMD_1="MSG MASTER Starting VMDUMP" VMCMD_2="VMDUMP" VMCMD_3="IPL 4711" 17

zvm vmconvert vmcp vmur vmcp Using the z/vm CP interface device driver (vmcp), you can send control program (CP) commands to the VM hypervisor and display VM s response. Before using this command, you must load the vmcp module by using the modprobe command Example ~> vmcp "q dasd grep T6345057 DASD 4DE0 ATTACHED TO T6345057 4DE0 R/W 0X4DE0 DASD 4DE1 ATTACHED TO T6345057 4DE1 R/W 0X4DE1 DASD 4DE2 ATTACHED TO T6345057 4DE2 R/W 0X4DE DASD 4DE3 ATTACHED TO T6345057 4DE3 R/W 0X4DE3 18

Misc cpuplugd iucvconn Iucvtty ts-shell Cpuplugd (CPU and Memory Hotplug Daemon) cpuplugd is a daemon that dynamically enables and disables cpus or increases the memory based on a set of rules. The rules can incorporate certain system load variables. Manage memory under z/vm. Configuration file: /etc/sysconfig/cpuplugd Init-Script: /etc/init.d/cpuplugd {start, stop, restart} 19

Misc cpuplugd iucvconn Iucvtty ts-shell Cpuplugd (CPU and Memory Hotplug Daemon) Example (/etc/sysconfig/cpuplugd) update="60 cpu_min="2 cpu_max="5 cmm_min="32 cmm_max="4096 cmm_inc="128 hotplug = "(loadavg + 0.75 > onumcpus) & (idle < 10.0) hotunplug = "(loadavg < onumcpus - 0.25) (idle > 50) memplug = "(swaprate +10 > freemem) & (freemem+10 < apcr) memunplug = "(swaprate > (freemem +10000))" 20

Display lscss lsdasd lsreipl Lsshut Lscss (list channel subsystem devices) The lscss command is used to list all or a subset of devices that are managed by to common I/O subsystem. Example lscss List all devices that are managed by the common I/O subsystem. lscss -t 3390 Same as above but shows only 3390 devices. 21

Display lscss lsdasd lsreipl Lsshut Lsdasd (list channel subsystem devices) The lsdasd command provides an overview of available DASD devices. Example # lsdasd 0.0.0100(ECKD) at (94: 0) is dasda : active at blocksize 4096, 600840 blocks, 2347 MB 0.0.0101(ECKD) at (94: 4) is dasdb : active at blocksize 4096, 543240 blocks, 2122 MB 0.0.0102(ECKD) at (94: 8) is dasdc : active at blocksize 4096, 600840 blocks, 2347 MB 0.0.0103(ECKD) at (94:12) is dasdd : active at blocksize 4096, 600840 blocks, 2347 MB 22

Display lscss lsdasd lsreipl Lsshut Lsreipl (List the re-ipl configuration for Linux on System z) lsreipl lists the re-ipl (and ipl) configuration for Linux on System z. Using this tool you can see from which device your system will boot after you issue the reboot command. Furthermore you can query the system for information about the current boot device. Example ~> lsreipl Re-IPL type: ccw Device: 0.0.4bb8 Loadparm: 23

Display lscss lsdasd lsreipl Lsshut lsshut(print the shutdown action configuration for Linux on System z) Command to see what the system should do in one of the following states. Example ~> lsshut Trigger Action ======================== Halt stop Panic stop Power off stop Reboot reipl 24

Boot zipl Zipl (boot loader for IBM S/390 and zseries architectures) zipl is a boot loader tool for IBM S/390 and zseries machines. It can be used to prepare devices for initial program load (IPL). The following functions are supported: - booting a Linux kernel with optional ramdisk and kernel command line - taking a snapshot of the current system status (system dump) - loading a data file to initialize named saved segments (NSS) Example (/etc/zipl.conf) [defaultboot] default=ipl 25 [ipl] target=/boot/zipl image=/boot/zilo-kernel/image #ramdisk=/boot/initrd parameters="dasd=0150 root=/dev/dasda2 noinitrd"

RUNNING LINUX ON SYSTEM Z AS A Z/VM GUEST: USEFUL THINGS TO KNOW 26

Logging on to z/vm (creating a virtual machine) 27

How can you read files on a CMS disk with Linux? Allows to mount a z/vm minidisk to a Linux mount point The cmsfs fuse file system transparently integrates the files on the minidisk into the Linux VFS, no special command required ~> cmsfs-fuse /dev/dasde /mnt/cms ~> ls -la /mnt/cms/profile.exec -r--r--- 1 root root 3360 Jun 26 2009 /mnt/fuse PROFILE.EXEC Mount with '-t' to get automatic EBCDIC to ASCII conversion ~> cmsfs-fuse -t /dev/dasde /mnt/cms Use fusermount to unmount the file system ~> fusermount -u /mnt/cms 28

Configuring standby memory To see how much central and expanded storage (memory) are installed and allocated to a system use the QUERY STORAGE and QUERY XSTOR commands. For example : ==> q stor STORAGE = 16G CONFIGURED = 16G INC = 256M STANDBY = 0 RESERVED = 0 Modify the directory entry by adding a COMMAND statement. This will give the virtual machine an additional 768 MB of standby memory : USER LINUX01 LNX4VM 256M 2G G INCLUDE LNXDFLT COMMAND DEFINE STORAGE 256M STANDBY 768M OPTION APPLMON MDISK 100 3390 3339 3338 UM63A9 MR LNX4VM LNX4VM LNX4VM MDISK 101 3390 6677 3338 UM63A9 MR LNX4VM LNX4VM LNX4VM 29

chmem Setting memory online or offline The chmem command sets a particular size or range of memory online or offline Setting memory online might fail if the hypervisor does not have enough memory left For example, because memory was overcommitted Setting memory offline might fail if Linux cannot free the memory If only part of the requested memory could be set online or offline, a message tells you how much memory was set online or offline instead of the requested amount 30

chmem Setting memory online or offline To request 1024 MB of memory to be set online, issue : # chmem --enable 1024 This command revoke 512 MB of memory : # chmem --disable 512 31

z/vm 5.4: Virtual CPU SHARE Redistribution Allows z/vm guests to expand or contract the number of virtual processors it uses without affecting the overall CPU capacity it is allowed to consume Guests can dynamically optimize their multiprogramming capacity based on workload demand Starting and stopping virtual CPUs does not affect the total amount of CPU capacity the guest is authorized to use Linux CPU hotplug daemon starts and stops virtual CPUs based on Linux load average value Helps enhance the overall efficiency of a Linux-on-z/VM environment Previously, stopped virtual processors were given a portion of the guest share. 32

z/vm 5.4: Virtual CPU SHARE Redistribution 33

Cooperative Memory Management (CMM) 34

CMM: Linux Implementation 35

CMM: Linux Implementation Linux uses a IUCV special message interface for z/vm interaction (CMMSHRINK/CMMRELEASE/CMMREUSE) ~# modprobe cmm ~# echo 100 > /proc/sys/vm/cmm_timed_pages ~# echo 10 1 > /proc/sys/vm/cmm_timeout ~# echo 100 > /proc/sys/vm/cmm_pages 36

appldata - Linux monitoring modules Kernel modules which gather information from the Linux kernel appldata_os CPU utilization, processes appldata_mem memory, paging, cache appldata_net_sum packets, bytes, errors ~# modprobe appldata_os ~# echo 1 > /proc/sys/appldata/os ~# modprobe appldata_mem ~# echo 1 > /proc/sys/appldata/mem ~# modprobe appldata_net_sum ~# echo 1 > /proc/sys/appldata/net_sum 37

appldata - Linux monitoring modules Linux monitoring data collected by appldata_os as processed and displayed by z/vm Performance Toolkit: 38

appldata - Linux monitoring modules Linux monitoring data collected by appldata_mem as processed and displayed by z/vm Performance Toolkit: 39

Monitoring with hypfs Virtual Linux file system Uses diagnose calls to gather guest data from hypervisor Works with LPAR hypervisor or z/vm Resources controlled by hypervisor, i.e. physical CPUs Resources provided to guest systems, i.e virtual CPUs Preconditions LPAR: enable Global performance data control checkbox in HMC activation profile of the guest where hypfs is mounted z/vm: privilege class B required for the guest where hypfs is mounted 40

Monitoring with hypfs mounting hypfs: ~# mount -t s390_hypfs /sys/hypervisor/s390/ hypfs is populated with initial data when being mounted and hypfs data is only updated on request: ~# echo 1 > /sys/hypervisor/s390/update 41

Monitoring with hypfs 42

Monitoring with hypfs /sys/hypervisor/s390/systems/h4245005/mem/share_kib: 319004 /sys/hypervisor/s390/systems/h4245005/mem/used_kib: 319004 /sys/hypervisor/s390/systems/h4245005/mem/max_kib: 1048576 /sys/hypervisor/s390/systems/h4245005/mem/min_kib: 0 /sys/hypervisor/s390/systems/h4245005/cpus/weight_cur: 100 /sys/hypervisor/s390/systems/h4245005/cpus/weight_max: 10000 /sys/hypervisor/s390/systems/h4245005/cpus/weight_min: 6 /sys/hypervisor/s390/systems/h4245005/cpus/count: 6 /sys/hypervisor/s390/systems/h4245005/cpus/dedicated: 0 /sys/hypervisor/s390/systems/h4245005/cpus/capped: 0 /sys/hypervisor/s390/systems/h4245005/cpus/cputime_us: 203792603 /sys/hypervisor/s390/systems/h4245005/onlinetime_us: 166806841739 43

hyptop - Display hypervisor performance data The hyptop command provides a dynamic real-time view of a hypervisor environment on System z. It works with both the z/vm and the LPAR PR/SM hypervisor. Depending on the available data it shows, for example, CPU and memory information about running LPARs or z/vm guest operating systems. 44

hyptop - Display hypervisor performance data The following things are required to run hyptop: The debugfs file system must be mounted. The hyptop user must have read permission for the required debugfs files: z/vm: <debugfs mount point>/s390_hypfs/diag_2fc LPAR: <debugfs mount point>/s390_hypfs/diag_204 To monitor all LPARs or z/vm guest operating systems of the hypervisor, your system must have additional permissions: For z/vm: The guest must be privilege class B. For LPAR: On the HMC or SE security menu of the LPAR activation profile, select the Global performance data control checkbox. 45

hyptop - Display hypervisor performance data 46

hyptop - Display hypervisor performance data 47

END 48