Procedure - Lustre MDT 1.8 to 2.0+ Migration Author: Colin Faber Version 1.0 October 2011 Notices The information in this document is subject to change without notice. While every effort has been made to ensure that all information in this document is accurate, the document is provided "as-is" and Xyratex accepts no liability for any errors that may arise. Reliance on this document and its content is at the sole risk of the viewer or reader. 2011 Xyratex (the trading name of Xyratex Technology Limited). Registered Office: Langstone Road, Havant, Hampshire, PO9 1SA, England. Registered number 03134912. No part of this document may be transmitted or copied in any form, or by any means, for any purpose, without the written permission of Xyratex. Xyratex is a trademark of Xyratex Technology Limited. Lustre is a registered trademark of Oracle. For more information please contact marketing@xyratex.com. 1
Revision History Version Date Author Comment 1.0 October 2011 Colin Faber, Kevin Canady Initial document release. 2
Introduction This is a step-by-step procedure to successfully migrate a Lustre MDT from 1.8 software to 2.x software. This document also describes how to perform successful file-level backup and restores in a 2.x environment. About the Migration Process The on-disk format of the Meta Data Server (MDS) file system has changed between 1.8 and 2.x versions of the Lustre file system software. Although Lustre 2.x software can manage Lustre 1.8 file systems, I/O performance is not optimized and the use of several 2.x features, such as backup/restore and changelogs is limited because of this architectural change. To improve file system performance and enable full 2.x functionality, we recommend upgrading the MDS/MDT component from Lustre 1.8 to 2.x level software. Primary Goals of the 1.8 to 2.x Migration Convert the on-disk format of the MDS from Lustre 1.8 to native Lustre 2.x Restore file system metadata to the newly-formatted 2.x MDS (after a standard backup and restore operation) Who Should Migrate from Lustre 1.8 to 2.x? The migration process is intended for Lustre users who: Have large data sets that are sustained and cannot be easily reconfigured or moved to Lustre 2.x Need to change on-disk metadata file system options, such as inode spacing and stripe/ stride alignment Need to perform file-level backup and restore (tar) Want to utilise changelog functionality Want to improve overall metadata performance Do not want to run in MDT compatibility mode 3
Requirements Make sure that these prerequisites have been met before continuing with the migration procedure: Lustre 2.1 (or later) packages (RPMs) that support the MDT mount upgrade option. These packages can be obtained from Xyratex. tar archiver version 1.26 (or later) and/or dd Free space (several TBs depending on the size of the MDT) Lustre 1.8 MDT file system Console (screen) tool Step 1 - Back Up the MDT There are various ways to back up the MDT. This procedure describes two backup methods (using dd or tar). 1. Unmount the MDT. Run: umount <MDT device path> 2. Back up the MDT using either dd or tar. Both methods are described below. Backing up with dd : a. Run: dd if=<mdt device path> of=<backup image file> bs=4m b. Verify that the dd image is identical to the MDT: i. Mount the dd image with a loopback device. Run: losetup /dev/ loop4 <backup image file> ; mount -t ldiskfs /dev/ loop4 <temporary mount point> ii. Select a few random files and verify that the permissions and EA information match between the dd image and the MDT. iii. If the random checks (5 or 10 files) all match, remove the temporary data. c. Wait for the backup process to complete. The process time depends on variables such as MDT size, MDS performance, and backup device performance. NOTE: On some dd versions, you can verify the transfer rate and total amount of data transferred using kill -USR1 <dd process ID>. Check your dd manual for information about this option. 4
Backing up with tar : a. Mount the MDT as ldiskfs. Run: mount -t ldiskfs <MDT device path> <MDT mount point> b. Start a new screen session. Run: screen c. Run: cd <mount point> ; tar -cvf <backup file> --xattrs - -acls --sparse --exclude=catalogs. d. Verify that the <backup file> contains identical file data to the MDT by partially extracting the tar archive to a temporary path (NOTE: the file system must have EA support enabled prior to extracting the data): i. Run: cd <Temp path> ; mkdir mdt ; cd mdt ; tar -xvf <Backup File> --sparse. Then ^C after it starts extracting the ROOT files. ii. Select a few random files and verify that the permissions and EA information match between the dd image and the MDT. iii. If the random checks (5 or 10 files) all match, remove the temporary data. e. Unmount the MDT. Run: cd / ; umount <MDT mount point> The backup process may run slowly. The dd process may run faster, but it requires free space equal to or greater than the size of the MDT device. The tar process uses much less space. In both cases, the images can be heavily compressed. Step 2 - Upgrade your Base OS Before migrating Lustre, we recommend upgrading your base operating system. Lustre 2.1 supports a limited number of OS distributions. Currently, RHEL6 and SL6 are fully supported. For complete instructions to upgrade to Lustre 2.0, see: http://wiki.lustre.org/manual/ LustreManual20_HTML/UpgradingLustre.html#50438205_51369 To obtain Lustre 2.1 (or later) with MDT upgrade support, see: http://www.clusterstor.com/ releases Step 3 - Upgrade Lustre MDT from 1.8 to 2.1 Verify that the MDT parameter set is valid. Change or remove all non-valid parameters. 1. Print the current MDT parameter set. Run: tunefs.lustre <MDT device> 2. Modify the parameters as needed and omit any deprecated options. Run: tunefs.lustre --erase-param < --param > --writeconf <MDT device> 3. Mount the MDT with the upgrade option. Run: mount -t lustre -o upgrade <MDT device> <MDT mount point> 5
The upgrade process starts on the file system and blocks the console until it completes. Once the MDT is upgraded, metadata services start normally. Lustre 2.x features such as changelogs, OST pools, and version-based recovery should be fully functional as the MDT is now running 2.x software. Performing a File-level Backup / Restore This is a two-step procedure. First, perform a file-level backup of the MDT and then restore and upgrade the MDT. Back Up the MDT 1. Unmount the MDT. Run: umount <MDT device> 2. Start the tar backup procedure outlined in Step 1 - Back Up the MDT (above). This ensures that a good copy of the MDT is saved. 3. One the backup is complete, remount the MDT normally to bring MDS operations back online. Restore the MDT 1. Restore the backup copy of the MDT. a. Unmount the MDT (if it is mounted). b. Mount the MDT as ldiskfs and change to the MDT root directory. c. Untar the file with this command: tar -xvf <backup file> --sparse -- acls --xattrs. 2. Remove any dangling objects in the <MDT mount>/objects/* and the <MDT mount>/ CATALOG files if they were captured during the MDT backup process. These files are outdated and will be automatically recreated when the MDT is remounted. 3. Unmount the MDT. 4. Remount the MDT normally and specify the upgrade option. Run: mount -t lustre -o upgrade <MDT device> <MDT mount point> Statistics and Real-time Information During the upgrade process, various statistics are provided via two procfs entries. These statistics provide a full count of all directories, files, and extents changed: /proc/fs/lustre/mdd/*/lu_stat Number of LinkEAs added - LinkEA added Number of LInkEAs removed - LinkEA removed" Number of files verified and possibly modified - "files handled" Number of directories verified and possibly modified - "dirs handled" 6
/proc/fs/lustre/osd-ldiskfs/*/lu_stat Number of LMAs created - "LMA created" Number of OI entries added - "OI entry added" Number of OI entries removed - "OI entry removed" Number of FIDs in direntrys added - "FID-in-direntry added" /proc/fs/lustre/mdd/*/per_cpu_rebuild This file shows the number of directories and files handled in a two-column display. Each column represents one of the possible CPU threads. 7