Concepts, Planning, and Installation Guide

Transcription

1 General Parallel File System Concepts, Planning, and Installation Guide Version 3 Release 2 GA

2

3 General Parallel File System Concepts, Planning, and Installation Guide Version 3 Release 2 GA

4 Note: Before using this information and the product it supports, be sure to read the general information under Notices on page 99. Second Edition (October 2007) This edition applies to version 3 release 2 of IBM General Parallel File System Multiplatform (program number 5724-N94), IBM General Parallel File System for POWER (program number 5765-G66), and to all subsequent releases and modifications until otherwise indicated in new editions. Significant changes or additions to the text and illustrations are indicated by a vertical line () to the left of the change. IBM welcomes your comments. A form for your comments may be provided at the back of this publication, or you may address your comments to the following: International Business Machines Corporation Department H6MA, Mail Station P South Road Poughkeepsie, NY United States of America FAX (United States and Canada): FAX (Other Countries): Your International Access Code IBMLink (United States customers only): IBMUSM10(MHVRCFS) Internet mhvrcfs@us.ibm.com If you would like a reply, be sure to include your name, address, telephone number, or FAX number. Make sure to include the following in your comment or note: v Title and order number of this book v Page number or topic related to your comment When you send information to IBM, you grant IBM a nonexclusive right to use or distribute the information in any way it believes appropriate without incurring any obligation to you. Copyright International Business Machines Corporation 1998, All rights reserved. US Government Users Restricted Rights Use, duplication or disclosure restricted by GSA ADP Schedule Contract with IBM Corp.

5 Contents Figures vii Tables ix About this publication xi Who should read this publication xi Conventions used in this publication xi Prerequisite and related information xii ISO xii Using LookAt to look up message explanations xii How to send your comments xiii Summary of changes xv Chapter 1. Introducing General Parallel File System The strengths of GPFS Shared file system access among GPFS clusters Improved system performance File consistency High recoverability and increased data availability Enhanced system flexibility Simplified storage management Simplified administration The basic GPFS structure GPFS administration commands The GPFS kernel extension The GPFS daemon The GPFS open source portability layer GPFS cluster configurations Interoperable cluster requirements Chapter 2. Planning for GPFS Hardware requirements Software requirements Recoverability considerations Node failure Network Shared Disk server and disk failure Reduced recovery time using Persistent Reserve GPFS cluster creation considerations GPFS node adapter interface names Nodes in your GPFS cluster GPFS cluster configuration servers Remote shell command Remote file copy command Cluster name User ID domain for the cluster Starting GPFS automatically Cluster configuration file Managing distributed tokens Disk considerations NSD creation considerations NSD server considerations File system descriptor quorum Copyright IBM Corp. 1998, 2007 iii

6 File system creation considerations Device name of the file system List of disk descriptors NFS V4 deny-write open lock Disks for your file system Deciding how the file system is mounted Block size atime values mtime values Block allocation map File system authorization Strict replication Internal log file File system recoverability parameters Number of nodes mounting the file system Maximum number of files Mountpoint directory Assign mount command options Automatic quota activation Enable DMAPI A sample file system creation Chapter 3. Steps to establishing and starting your GPFS cluster Chapter 4. Installing GPFS on Linux nodes Files to ease the installation process Verifying the level of prerequisite software Installation procedures Accepting the electronic license agreement Creating the GPFS directory Installing the GPFS man pages Installing GPFS over a network Verifying the GPFS installation Building your GPFS portability layer Using the automatic configuration tool to build GPFS portability layer Chapter 5. Installing GPFS on AIX 5L nodes Files to ease the installation process Verifying the level of prerequisite software Installation procedures Accepting the electronic license agreement Creating the GPFS directory Creating the GPFS installation table of contents file Installing the GPFS man pages Installing GPFS over a network Reconciling existing GPFS files Verifying the GPFS installation Chapter 6. Migration, coexistence and compatibility Migrating to GPFS 3.2 from GPFS Migrating to GPFS 3.2 from GPFS Migrating to GPFS 3.2 from GPFS 2.2 or earlier releases of GPFS Completing the migration to a new level of GPFS Additional considerations when migrating GPFS 2.3 and earlier file systems Reverting to the previous level of GPFS Reverting to a previous level of GPFS when you have not issued mmchconfig release=latest 56 iv GPFS: Concepts, Planning, and Installation Guide

7 Reverting to a previous level of GPFS when you have issued mmchconfig release=latest Coexistence Compatibility Applying maintenance to your GPFS system Chapter 7. Configuring and tuning your system for GPFS General system configuration and tuning considerations Clock synchronization GPFS administration security Cache usage GPFS I/O Access patterns Aggregate network interfaces Swap space Linux configuration and tuning considerations updatedb considerations SUSE LINUX considerations GPFS helper threads Communications I/O Disk I/O AIX configuration and tuning considerations Communications I/O Disk I/O Switch pool eserver High Performance Switch IBM Virtual Shared Disk GPFS use with Oracle Chapter 8. Steps to permanently uninstall GPFS Chapter 9. GPFS architecture Special management functions The GPFS cluster manager The file system manager The metanode Use of disk storage and file structure within a GPFS file system Quota files GPFS recovery logs GPFS and memory Pinned and non-pinned memory GPFS and network communication GPFS daemon communication GPFS administration commands Application and user interaction with GPFS Operating system commands Operating system calls GPFS command processing NSD disk discovery Recovery Cluster configuration data files GPFS backup data Chapter 10. IBM Virtual Shared Disk considerations Virtual shared disk server considerations Disk distribution Disk connectivity Contents v

8 Virtual shared disk creation considerations Virtual shared disk server and disk failure Chapter 11. Considerations for GPFS applications Exceptions to Open Group technical standards Determining if a file system is controlled by GPFS GPFS exceptions and limitations to NFS V4 ACLs Accessibility features for GPFS Accessibility features Keyboard navigation IBM and accessibility Notices Trademarks Glossary Index vi GPFS: Concepts, Planning, and Installation Guide

9 Figures 1. A Linux-only cluster with disks that are SAN-attached to all nodes A Linux-only cluster with an NSD server An AIX and Linux cluster with an NSD server An AIX and Linux cluster providing remote access to disks through the High Performance Switch (HPS) for the AIX nodes and a LAN connection for the Linux nodes An AIX and Linux cluster that provides remote access to disks through multiple NSD servers 9 6. An AIX cluster with an NSD server GPFS clusters providing shared file system access GPFS configuration utilizing node quorum GPFS configuration utilizing node quorum with tiebreaker disks RAID/ESS Controller twin-tailed in a SAN configuration GPFS configuration specifying multiple NSD servers connected to a common disk controller utilizing RAID5 with four data disks and one parity disk GPFS utilizes failure groups to minimize the probability of a service disruption due to a single component failure GPFS files have a typical UNIX structure Basic failure groups with servers and disks Failure groups with twin-tailed disks Primary node serving RAID device Backup node serving RAID device RAID/ESS Controller multi-tailed to the primary and secondary virtual shared disk servers Concurrent node serving device Copyright IBM Corp. 1998, 2007 vii

10 viii GPFS: Concepts, Planning, and Installation Guide

11 Tables 1. Typographic conventions xi 2. GPFS cluster creation options Disk descriptor usage for the GPFS disk commands File system creation options Copyright IBM Corp. 1998, 2007 ix

12 x GPFS: Concepts, Planning, and Installation Guide

13 About this publication The General Parallel File System: Concepts, Planning, and Installation Guide describes: v The IBM General Parallel File System (GPFS ) Multiplatform licensed program, 5724-N94 v The IBM GPFS for POWER licensed program, 5765-G66 This publication includes information about these topics: v Introducing GPFS v Planning concepts for GPFS v SNMP support v Installing GPFS v Migration, coexistence and compatibility v Applying maintenance v Configuration and tuning v Steps to uninstall GPFS Who should read this publication This publication is intended for system administrators, analysts, installers, planners, and programmers of GPFS clusters. It assumes that you are very experienced with and fully understand the operating systems on which your cluster is based. Use this publication if you are: v Planning for GPFS v Installing GPFS on a supported cluster configuration, consisting of: Linux nodes AIX 5L nodes An interoperable cluster comprised of both operating systems Conventions used in this publication Table 1 describes the typographic conventions used in this publication. Table 1. Typographic conventions Typographic convention Usage Bold Bold words or characters represent system elements that you must use literally, such as commands, flags, path names, directories, file names, values, and selected menu options. Bold Underlined Bold Underlined keywords are defaults. These take effect if you fail to specify a different keyword. Italic v Italic words or characters represent variable values that you must supply. v Italics are also used for publication titles and for general emphasis in text. Copyright IBM Corp. 1998, 2007 xi

14 Table 1. Typographic conventions (continued) Typographic convention Usage Constant width All of the following are displayed in constant width typeface: v Displayed information v Message text v Example text v Specified text typed by the user v Field names as displayed on the screen v Prompts from the system v References to example text [ ] Brackets enclose optional items in format and syntax descriptions. { } Braces enclose a list from which you must choose an item in format and syntax descriptions. A vertical bar separates items in a list of choices. (In other words, it means or ) <> Angle brackets (less-than and greater-than) enclose the name of a key on the keyboard. For example, <Enter> refers to the key on your terminal or workstation that is labeled with the word Enter.... An ellipsis indicates that you can repeat the preceding item one or more times. <Ctrl-x> The notation <Ctrl-x> indicates a control character sequence. For example, <Ctrl-c> means that you hold down the control key while pressing <c>. \ The continuation character is used in programming examples in this publication for formatting purposes. Prerequisite and related information For updates to this publication, see publib.boulder.ibm.com/infocenter/clresctr/topic/ com.ibm.cluster.gpfs.doc/gpfsbooks.html. For the latest support information, see the GPFS Frequently Asked Questions at publib.boulder.ibm.com/ infocenter/clresctr/topic/com.ibm.cluster.gpfs.doc/gpfs_faqs/gpfsclustersfaq.html. ISO 9000 ISO 9000 registered quality systems were used in the development and manufacturing of this product. Using LookAt to look up message explanations LookAt is an online facility that lets you look up explanations for most of the IBM messages you encounter, as well as for some system abends and codes. You can use LookAt from the following locations to find IBM message explanations for Clusters for AIX and Linux: v The Internet. You can access IBM message explanations directly from the LookAt Web site: v Your wireless handheld device. You can use the LookAt Mobile Edition with a handheld device that has wireless access and an Internet browser (for example, Internet Explorer for Pocket PCs, Blazer, or Eudora for Palm OS, or Opera for Linux handheld devices). Link to the LookAt Mobile Edition from the LookAt Web site. xii GPFS: Concepts, Planning, and Installation Guide

15 How to send your comments Your feedback is important in helping us to produce accurate, high-quality information. If you have any comments about this publication or any other GPFS documentation: v Send your comments by to: mhvrcfs@us.ibm.com Include the publication title and order number, and, if applicable, the specific location of the information you have comments on (for example, a page number or a table number). v Fill out one of the forms at the back of this publication and return it by mail, by fax, or by giving it to an IBM representative. To contact the IBM cluster development organization, send your comments by to: cluster@us.ibm.com. About this publication xiii

16 xiv GPFS: Concepts, Planning, and Installation Guide

17 Summary of changes The following sections summarize changes to the GPFS licensed program and the GPFS library for version 3 release 2. Within each book in the library, a vertical line to the left of text and illustrations indicates technical changes or additions made to the previous edition of the book. Summary of changes for GPFS Version 3 Release 2 as updated, October 2007 Changes to GPFS and to the GPFS library for version 3 release 2 include: v New information: In the past, migrating to a new release of GPFS required shutting down GPFS and upgrading all nodes before GPFS could be restarted. GPFS V3.2 supports rolling upgrades and a limited form of backward compatibility: - Rolling upgrades enable you to install new GPFS code one node at a time without shutting down GPFS on other nodes. It is expected that all nodes will be upgraded within a short time. Some features become available on each node as soon as the node is upgraded, while other features become available as soon as all participating nodes are upgraded. - Backward compatibility allows running with a mixture of old and new nodes. Multi-cluster environments may be able to upgrade the local cluster while still allowing mounts from remote nodes in other clusters that have not yet been upgraded. You can designate up to eight NSD servers to simultaneously service I/O requests from different clients. Each of these NSD servers must have physical access to the same logical unit number (LUN). Different servers can serve I/O to different non-intersecting sets of clients for a variety of reasons, such as load balancing on the server, network partitioning by balancing the load on different networks, or workload partitioning. Multiple NSD server functions require all (peer) NSD servers to be part of the same GPFS cluster. The existing subnet functions in GPFS determine which NSD server should serve a particular client. The assumption is that nodes within a subnet are connected via high-speed networks. GPFS for Linux offers Clustered NFS (CNFS) to enable highly-available NFS exporting of file systems. Some or all nodes in an existing GPFS cluster also serve NFS and are part of the NFS cluster. All of the nodes in the NFS cluster export the same file systems to the NFS clients. This support includes the following: - Monitoring. Every node in the NFS cluster runs an NFS monitoring utility that monitors GPFS and the NFS and networking components on the node. After failure detection, and based on customer configuration, the monitoring utility may invoke a failover. - Failover. The automatic failover procedure transfers the NFS serving load from the failing node to another node in the NFS cluster. The failure is managed by the GPFS cluster, including lock and state recovery. - Load balancing. The IP address is the load unit that can be moved from one node to another because of failure or load balancing needs. This solution supports a failover of all the node s load (all NFS IP addresses) as one unit to another node. However, if no locks are outstanding, individual IP addresses can be moved to other nodes for load balancing purposes. The GPFS InfiniBand Remote Direct Memory Access (RDMA) code uses RDMA for NSC client file I/O requests. RDMA transfers data directly between the NSD client memory and the NSD server memory instead of sending and receiving the data over the TCP socket. Using RDMA may improve bandwidth and decrease CPU utilization. GPFS can use the SCSI-3 (Persistent Reserve) standard to provide fast failover with improved recovery times. To exploit this functionality the file system has to be created on SCSI-3 capable disks. Copyright IBM Corp. 1998, 2007 xv

18 To enable this function, you must set up PR when you create each NSD. The basic requirements for PR are that the NSD server must be an AIX node and that the disks must be regular AIX hdisks. To enable PR, use the new mmchconfig command option called usepersistentreserve. Monitoring is enabled with the SNMP-based management application, Net-SNMP. SNMP requires a Linux node installed in order to collect the data. Monitoring involves gaining a view of the GPFS system. Monitored information can be roughly grouped into the categories of configuration, status, and performance: - Configuration denotes the initially customized aspects of the system s current state. - Status information is dynamic information that expresses the current health of nodes, disks, and other hardware, including any reported error conditions disk utilization, and fragmentation. - Performance information includes quantitative measurement of the workings of a system. Performance enhancements include support for parallel defragmentation of a file system, larger pagepool support, and directory-locking: - Defragmentation of a file system can now be run in parallel across nodes in a cluster. - Maximum pagepool support is increased to 256 GB. - Directory-locking improvements for concurrent file creates and deletes from multiple nodes (this function will be available in APAR IZ01431). When a socket connection breaks due to a network failure, GPFS now tries to re-establish the connection rather than immediately initiating node expulsion procedures. Mount up to 256 file systems. GPFS V3.2 extends Information Lifecycle Management (ILM) functionality to integrate with HSM products. A single set of policies can be used to move data across different storage pools of a file system and to move data from GPFS storage pools to near-line storage and from near-line storage to GPFS storage pools. Additional enhancements include the ability to use policies for backup and restore. - Subroutine gpfs_fputattrswithpathname for backup and restore functions has been added. This subroutine sets all the extended file attributes for a file and invokes the policy engine for restoring files. - Subroutines gpfs_fgetattrs and gpfs_fputattrs have been enhanced with new flags. GPFS V3.2 enables the policy code to run in parallel across all nodes in the home cluster that have the file system mounted. The policy evaluation can then scale with the size of the cluster. Administration enhancements, which include: New commands: - mmchnode, which changes node attributes - mmnsddiscover, which rediscovers paths to the specified network shared disks on the specified nodes - mmtracectl, which sets up and enables GPFS tracing Updated commands: - mmapplypolicy to add the -M, -N, and -S options - mmchconfig to add several new configuration parameters - mmchdisk to add the -F option - mmchfs to add full or compat to the -V option - mmchmgr to add the -c option - mmchnsd to state that you can specify up to 8 NSD servers in the disk descriptors - mmcrfs: v To remove the mandatory MountPoint positional parameter v To add the -L and -T options v To change the default for the -D option from posix to nfs4 xvi GPFS: Concepts, Planning, and Installation Guide

19 v To change the default for the -k option from posix to all v To add the --version Version option v To change the default values for the -R and -M options to 2 - mmcrnsd to state that you can specify up to 8 NSD servers in the disk descriptors and to change PrimaryServer and BackupServer to serverlist - mmdefragfs to add the -P and -N options and to remove the -v option - mmdf to remove the -q option - mmedquota to add Device:Fileset to the -j option - mmlscluster to add the --cnfs option which displays the cluster NFS - mmlsfs to add the all_local and all_remote parameters - mmlsmgr to add the -c and -C parameters - mmmount to add the all_local and all_remote parameters Tracing is improved to increase the reliability of trace data gathering. It enhances the ability to acquire accurate and reliable problem determination information. With the new mmtracectl trace command, you can: - Turn trace on or off on the next session to automatically start trace when GPFS starts - Allow for predefined trace levels: io, all, def, and user-specified trace levels - Change the size of trace buffers - Allow a user-defined directory for keeping trace files - Control trace recycling during daemon termination: (off, local, global, or globalonshutdown) v Changed information: The terminology "cluster configuration manager" or "configuration manager" has been changed to "cluster manager". v Deleted information: The mmsanrepairfs command has been removed. All references to SANergy have been removed. Summary of changes xvii

20 xviii GPFS: Concepts, Planning, and Installation Guide

21 Chapter 1. Introducing General Parallel File System IBM s General Parallel File System (GPFS) provides file system services to parallel and serial applications. GPFS allows parallel applications simultaneous access to the same files, or different files, from any node which has the GPFS file system mounted while managing a high level of control over all file system operations. GPFS is particularly appropriate in an environment where the aggregate peak need for data bandwidth exceeds the capability of a distributed file system server. GPFS allows users shared file access within a single GPFS cluster and across multiple GPFS clusters. A GPFS cluster consists of: v AIX 5L nodes, Linux nodes, or a combination thereof (see GPFS cluster configurations on page 6). A node may be: An individual operating system image on a single computer within a cluster. A system partition containing an operating system. Some IBM System p5 and IBM System p, machines allow multiple system partitions, each of which is considered to be a node within the GPFS cluster. v Network shared disks (NSDs) created and maintained by the NSD component of GPFS All disks utilized by GPFS must first be given a globally accessible NSD name. The GPFS NSD component provides a method for cluster-wide disk naming and access. On Linux machines running GPFS, you may give an NSD name to: - Physical disks - Logical partitions of a disk - Representations of physical disks (such as LUNs) On AIX machines running GPFS, you may give an NSD name to: - Physical disks - Virtual shared disks - Representations of physical disks (such as LUNs) v A shared network for GPFS communications allowing a single network view of the configuration. A single network, a LAN or a switch, is used for GPFS communication, including the NSD communication. The strengths of GPFS GPFS is a powerful file system offering: v Shared file system access among GPFS clusters v Improved system performance on page 2 v File consistency on page 3 v High recoverability and increased data availability on page 3 v Enhanced system flexibility on page 3 v Simplified storage management on page 4 v Simplified administration on page 4 Shared file system access among GPFS clusters GPFS allows users shared access to files in either the cluster where the file system was created or other GPFS clusters. Each site in the network is managed as a separate cluster, while allowing shared file system access. When multiple clusters are configured to access the same GPFS file system, Open Secure Sockets Layer (OpenSSL) is used to authenticate and check authorization for all network connections. Copyright IBM Corp. 1998,