IBM Certification Study Guide - AIX 5L Problem Determination Tools and Techniques

Transcription

1 Front cover IBM Certification Study Guide - AIX 5L Problem Determination Tools and Techniques Developed specifically for the purpose of preparing for AIX certification Makes an excellent companion to classroom education For experienced AIX professionals Tim Dasgupta Stephen Sommer ibm.com/redbooks

2

3 International Technical Support Organization IBM ^ Certification Study Guide - AIX 5L Problem Determination Tools and Techniques January 2003 SG

4 Note: Before using this information and the product it supports, read the information in Notices on page xv. Second Edition (January 2003) This edition applies to AIX 5L Version 5.1 (5765-E61) and subsequent releases running on an IBM ^ pseries or RS/6000 server. Copyright International Business Machines Corporation 2000, All rights reserved. Note to U.S. Government Users Restricted Rights -- Use, duplication or disclosure restricted by GSA ADP Schedule Contract with IBM Corp.

5 Contents Figures xi Tables xiii Notices xv Trademarks xvi Preface xvii The team that wrote this redbook xviii Become a published author xix Comments welcome xx Chapter 1. Certification overview Certification requirements Required prerequisite Recommended prerequisite Information and registration for the certification exam Core requirements Certification education courses Chapter 2. Customer relations Defining the problem Collecting information from the user Collecting information about the system Quiz Answers Chapter 3. Booting problem determination A general overview of the boot process BIST - POST MCA systems PCI systems Boot phase Boot phase LED 551, 555, or LED 552, 554, or LED The alog command Boot phase Copyright IBM Corp All rights reserved. iii

6 3.5.1 The /etc/inittab file LED LED c LED pseries servers Boot-related information in the error log Boot summary Command summary The errpt command The w command Quiz Answers Exercises Chapter 4. Hardware problem determination Hardware basics Hardware inventory Running diagnostics Concurrent mode Stand-alone diagnostics from disk - service mode Stand-alone diagnostics from CD-ROM Task selection or service aids Serial Storage Architecture disks General SSA setup rules SSA devices SSA disk considerations Three-digit display values Common boot time LEDs in the three-digit display Command summary The chdev command The lsattr command Quiz Answers Exercises Chapter 5. System dumps Configuring the dump device Starting a system dump Using the command line Using the SMIT interface Using the Reset button Using special key sequences iv IBM ^ Certification Study Guide - AIX 5L Problem Determination Tools and Techniques

7 5.2.5 System Hang Detection System dump status check Status codes Increasing the size of the dump device Configuring remote dump devices Copying a system dump Reading dumps Core dumps Checking for core dump Locating a core dump Determining the program that caused the core dump Command summary The crash command Types of crashes The snap command The strings command The sysdumpdev command The sysdumpstart command Quiz Answers Exercises Chapter 6. Error reports The error daemon The errdemon command The errpt command Error classes The errclear command Accounting Setting up an accounting system Setting up disk-usage accounting The syslogd daemon Quiz Answers Exercises Chapter 7. LVM, file system, and disk problem determination LVM data Physical volumes Volume groups Logical volumes Object Data Manager (ODM) LVM problem determination Contents v

8 7.2.1 Data relocation Backup data ODM resynchronization Understanding importvg problems Extending the number of max physical partitions Disk replacement Replacing a disk Recovering an incorrectly removed disk The AIX JFS Creating a JFS Increasing the file system size File system verification and recovery Sparse file allocation Unmount problems Removing file systems Different output from du and df commands Enhanced journaled file system The /proc file system Disk quota Paging space Recommendations for creating or enlarging paging space Determining if more paging space is needed Reducing and removing paging space Command summary The lsvg command The chvg command The importvg command The rmlvcopy command The reducevg command The rmdev command The syncvg command Quiz Answers Exercises Chapter 8. Network problem determination Network interface problems Routing problems Dynamic or static routing Name resolution problems The tcpdump and iptrace commands NFS troubleshooting General steps for NFS problem solving vi IBM ^ Certification Study Guide - AIX 5L Problem Determination Tools and Techniques

9 8.4.2 NFS mount problems Increasing NFS Socket Buffer Size The biod and nfsd daemons Command summary The chdev command The exportfs command The ifconfig command The iptrace command The lsattr command The netstat command The route command The tcpdump command Quiz Answers Exercises Chapter 9. System access and printer problem determination User license problems Telnet troubleshooting Network problems The telnet subserver Slow telnet login Telnet error FTP troubleshooting File limits System settings Adjusting AIX kernel parameters The su command A full file system Tracing Trace hook IDs Starting a trace Trace reports Tracing example Managing mail logging TTY troubleshooting Respawning too rapidly errors Printing Local printing problem Remote printing problem The /var file system Default printing subsystem Command summary Contents vii

10 9.9.1 The lslicense command The lssrc command The startsrc command The trace command The trcrpt command Quiz Answers Exercises Chapter 10. Performance problem determination CPU-bound system The sar command The vmstat command The ps command The tprof command Memory-bound system The vmstat command The ps command The svmon command The schedtune command Disk I/O bound system The iostat command The filemon command The fileplace command The sar command Network I/O bound system The netstat command The nfsstat command The netpmon command Workload Manager (WLM) WLM concepts and architecture Automatic assignment Manual assignment Backward compatibility Resource sets rset registry System debuggers The dbx command The kdb command Summary Command summary The sar command The ps command viii IBM ^ Certification Study Guide - AIX 5L Problem Determination Tools and Techniques

11 The netstat command The nfsstat command Quiz Answers Chapter 11. Software updates Overview Terminology Software layout Software states Installing a software patch Software patch installation procedure Software inventory Command summary The lslpp command The installp command The instfix command The lppchk command Quiz Answers Exercises Chapter 12. Online documentation Installing the Web browser Installing the Web server Installing the Documentation Library Service Configuring the Documentation Library Service Installing online manuals Invoking the Documentation Library Service Exercises Abbreviations and acronyms Related publications IBM Redbooks Other resources Referenced Web sites How to get IBM Redbooks IBM Redbooks collections Index Contents ix

12 x IBM ^ Certification Study Guide - AIX 5L Problem Determination Tools and Techniques

13 Figures 3-1 General boot order Function selection menu in diag Task selection menu in diag Display/alter bootlist menu in diag SMS main menu Boot phase Boot phase 2, part one Boot phase 2, part two Boot phase Example of rc.boot 3 in /etc/inittab Main diagnostics menu Format of the 103 code message Case study SMIT dump screen SMIT Add a TTY screen - Remote reboot options Error log case study Case study The errpt command error log report process The errpt command error record template repository process Disk problem mail from Automatic Error Log Analysis (diagela) JFS organization Case study SMIT menu to change the number of licensed users The smitty crjfsbf fast path SMIT screen for changing AIX operating system characteristics Output display of the topas command Change/Show Current Print Subsystem option General performance tuning flowchart CPU penalty example Web-based System Manager Overview and Tasks dialog Hierarchy of classes Resources cascading through tiers SMIT with the class creation attributes screen SMIT panel shows the additional localshm attribute Resource set definition to a specific class SMIT main panel for resource set management SMIT panel for rset registry management SMIT panel to add a new resource set Copyright IBM Corp All rights reserved. xi

14 10-12 Performance tuning flowchart SMIT Software Maintenance and Utilities panel Install and Update Software panel Netscape filesets HTTP filesets Documentation Library Service filesets Documentation Library Service xii IBM ^ Certification Study Guide - AIX 5L Problem Determination Tools and Techniques

15 Tables 3-1 Common MCA LED codes MCA POST LEDs Boot phase 2 LED codes Boot phase 3 LED codes Commonly used flags of the errpt command Commonly used flags of the w command SSA adapter information Common MCA LED codes Location code mapping table Commonly used flags of the chdev command Commonly used flags of the lsattr command REMOTE reboot ENABLE settings SHD default values Differences in system dump The vmmerrlog structure Commonly used flags of the snap command Commonly used flags of the strings command Commonly used flags of the sysdumpdev command Commonly used flags of the sysdumpstart command Commonly used flags of the errdemon command Commonly used flags of the errpt command List of error classes Commonly used flags of the errclear command System Resource Controller commands Journaled file system specifications Functions of pseudo files in /proc/<pid> directory Commonly used flag of the lsvg command Commonly used flag of the chvg command Commonly used flags of the importvg command Commonly used flags of the reducevg command Commonly used flags of the rmdev command Commonly used flag of the syncvg command Commonly used flags of the chdev command Commonly used flags of the exportfs command Commonly used flags of the ifconfig command Commonly used flags of the iptrace command Commonly used flags of the lsattr command Commonly used flags of the netstat command Copyright IBM Corp All rights reserved. xiii

16 8-7 Commonly used flags of the route command Commonly used flags of the tcpdump command Managing TTY devices Comparison of print subsystem functions Commonly used flags of the lssrc command Commonly used flags of the startsrc command Commonly used flags of the trace command Commonly used flags of the trcrpt command CPU-related ps output Memory-related ps output Some commonly used flags for the schedtune command List of process types Examples of class assignment rules Sample dbx subcommands Sample kdb subcommands Commonly used flags of the sar command Commonly used flags of the ps command Commonly used flags of the netstat command Commonly used flags of the nfsstat command Commonly used flags of the lppchk command Commonly used flags of the lslpp command Commonly used flags of the installp command Commonly used flags of the instfix command Commonly used flags of the lppchk command xiv IBM ^ Certification Study Guide - AIX 5L Problem Determination Tools and Techniques

17 Notices This information was developed for products and services offered in the U.S.A. IBM may not offer the products, services, or features discussed in this document in other countries. Consult your local IBM representative for information on the products and services currently available in your area. Any reference to an IBM product, program, or service is not intended to state or imply that only that IBM product, program, or service may be used. Any functionally equivalent product, program, or service that does not infringe any IBM intellectual property right may be used instead. However, it is the user's responsibility to evaluate and verify the operation of any non-ibm product, program, or service. IBM may have patents or pending patent applications covering subject matter described in this document. The furnishing of this document does not give you any license to these patents. You can send license inquiries, in writing, to: IBM Director of Licensing, IBM Corporation, North Castle Drive Armonk, NY U.S.A. The following paragraph does not apply to the United Kingdom or any other country where such provisions are inconsistent with local law: INTERNATIONAL BUSINESS MACHINES CORPORATION PROVIDES THIS PUBLICATION "AS IS" WITHOUT WARRANTY OF ANY KIND, EITHER EXPRESS OR IMPLIED, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF NON-INFRINGEMENT, MERCHANTABILITY OR FITNESS FOR A PARTICULAR PURPOSE. Some states do not allow disclaimer of express or implied warranties in certain transactions, therefore, this statement may not apply to you. This information could include technical inaccuracies or typographical errors. Changes are periodically made to the information herein; these changes will be incorporated in new editions of the publication. IBM may make improvements and/or changes in the product(s) and/or the program(s) described in this publication at any time without notice. Any references in this information to non-ibm Web sites are provided for convenience only and do not in any manner serve as an endorsement of those Web sites. The materials at those Web sites are not part of the materials for this IBM product and use of those Web sites is at your own risk. IBM may use or distribute any of the information you supply in any way it believes appropriate without incurring any obligation to you. Information concerning non-ibm products was obtained from the suppliers of those products, their published announcements or other publicly available sources. IBM has not tested those products and cannot confirm the accuracy of performance, compatibility or any other claims related to non-ibm products. Questions on the capabilities of non-ibm products should be addressed to the suppliers of those products. This information contains examples of data and reports used in daily business operations. To illustrate them as completely as possible, the examples include the names of individuals, companies, brands, and products. All of these names are fictitious and any similarity to the names and addresses used by an actual business enterprise is entirely coincidental. COPYRIGHT LICENSE: This information contains sample application programs in source language, which illustrates programming techniques on various operating platforms. You may copy, modify, and distribute these sample programs in any form without payment to IBM, for the purposes of developing, using, marketing or distributing application programs conforming to the application programming interface for the operating platform for which the sample programs are written. These examples have not been thoroughly tested under all conditions. IBM, therefore, cannot guarantee or imply reliability, serviceability, or function of these programs. You may copy, modify, and distribute these sample programs in any form without payment to IBM for the purposes of developing, using, marketing, or distributing application programs conforming to IBM's application programming interfaces. Copyright IBM Corp All rights reserved. xv