EXAScaler Version 2.0.1 Product Release Notes Revision A0 December 2013
Important Information Information in this document is subject to change without notice and does not represent a commitment on the part of DataDirect Networks, Inc. No part of this manual may be reproduced or transmitted in any form or by any means, electronic or mechanical, including photocopying and recording, for any purpose other than the purchaser s personal use without the written permission of DataDirect Networks, Inc. 2013 DataDirect Networks, Inc. All rights reserved. DataDirect Networks, the DataDirect Networks logo, DirectOS, DirectProtect, EXAScaler, GRIDScaler, Information in Motion, NAS Scaler, DirectMon, NoFS, ObjectAssure, SATAssure, Silicon Storage Appliance, S2A, Storage Fusion Architecture, SFA, Storage Fusion Fabric, xstreamscaler, Web Object Scaler, WOS are registered trademarks or trademarks of DataDirect Networks, Inc. All other brand and product names are trademarks of their respective holders. DataDirect Networks makes no warranties, express or implied, including without limitation the implied warranties of merchantability and fitness for a particular purpose of any products or software. DataDirect Networks does not warrant, guarantee or make any representations regarding the use or the results of the use of any products or software in terms of correctness, accuracy, reliability, or otherwise. The entire risk as to the results and performance of the product and software are assumed by you. The exclusion of implied warranties is not permitted by some jurisdictions; this exclusion may not apply to you. In no event will DataDirect Networks, their directors, officers, employees, or agents (collectively DataDirect Networks) be liable to you for any consequential, incidental, or indirect damages, including damages for loss of business profits, business interruption, loss of business information, and the like, arising out of the use or inability to use any DataDirect product or software even if DataDirect Networks has been advised of the possibility of such damages by you. Because some jurisdictions do not allow the exclusion or limitation of liability for consequential or incidental damages, these limitations may not apply to you. DataDirect Networks liability to you for actual damages from any cause whatsoever, and regardless of the form of the action (whether in contract, tort including negligence, product liability or otherwise), is limited to the sum you paid for the DataDirect product or software. DataDirect Networks EXAScaler 2.0.1 Product Release Notes, Rev. A0 2
Table of Contents 1.0 OVERVIEW... 4 2.0 FEATURES... 4 3.0 RELEASE COMPATIBILITY MATRIX... 5 4.0 INSTALLATION... 5 4.1 Block storage... 5 4.2 Embedded... 6 5.0 SOFTWARE VERSIONS... 6 6.0 UPGRADES... 6 6.1 Upgrading Systems Running EXAScaler Versions 1.6.0 through 2.0.0... 6 6.2 Upgrading Systems Running EXAScaler v1.5.x to 1.6.2... 7 6.2.1 Incompatibilities... 7 6.2.2 Process... 7 7.0 RESOLVED ISSUES... 8 8.0 KNOWN ISSUES... 8 9.0 SUPPORT... 8 10.0 APPENDIX A: SUPPORTED SOLUTION COMPONENTS... 9 DataDirect Networks EXAScaler 2.0.1 Product Release Notes, Rev. A0 3
1.0 Overview This document covers the features, installation, and errata for the 2.0.1-9 release of EXAScaler. This release is applicable to the standard EXAScaler solution for block storage and embedded storage platforms. 2.0 Features NOTE: On SFA12K, SFA12K-E, and SFA10K controllers, EXAScaler 2.0.1 requires SFA OS 2.0 or later. Therefore, SFA storage controllers SFA12K, SFA12K-E, and SFA10K must be running SFA OS 2.0 or later prior to installing or upgrading to EXAScaler 2.0.1. NOTE: On SFA10K-E and SFA 7700 controllers, EXAScaler 2.0.1 requires SFA OS 2.1.1 or later. Therefore, SFA storage controllers SFA10K-E and SFA 7700 must be running SFA OS 2.1.1 or later prior to installing or upgrading to EXAScaler 2.0.1. This release of EXAScaler (ES) updates the system software stack and includes the following new features relative to v1.6.2: EXAScaler I/O Software Stack Components: CentOS 6.4 Kernel - 2.6.32-358.18.1.el6.x86_64 Lustre 2.4.1 InfiniBand - MLNX OFED 1.5.3-4.0.41 DDN InfiniBand SRP 1.10 DDN Multipath 1.5.7 (RHEL6) Corosync 1.4.1-15.el6_4.1 Pacemaker 1.1.8-7 New Features in ES 2.0.1: Better correctness checking when loading exascaler.conf Removed the requirement to have nic_list set in host defaults. kill-heartbeat now kills corosync as well as heartbeat. Default routes are now set correctly on embedded platforms. es_mount now accepts --serial to start Lustre targets serially. Added support for shadow config files which allows es_showall to create debug dumps without including sensitive information. New Features in ES 2.0.0: Lustre 2.4.1 o Distributed Namespace (DNE) o Imperative Recovery o Large RPCs (4MB IOs) SFX Out-of-Band Hints OST Pools with SFX Please refer to Appendix A for list of supported server, networking, and storage platforms DataDirect Networks EXAScaler 2.0.1 Product Release Notes, Rev. A0 4
The Lustre version included in this release is a DDN build of Lustre 2.4.1 (tag 2.4.1) patched to build against MLNX_OFED 1.5.3-4.0.41. EXAScaler can be used with clients running Lustre versions 1.8.9, Lustre 2.1.6, Lustre 2.4.0 or newer clients. Older versions of Lustre clients should not be used with EXAScaler 2.0.0. Note that new features available in Lustre 2.4 like Distributed Namespace(DNE) and Imperative Recovery are only available if all clients are using Lustre 2.4.0 or above. Lustre 2.4.1 clients are strongly recommended. 3.0 Release Compatibility Matrix EXAScaler 2.0.1 support is shown in the following table. Firmware Versions Platforms SFA7700 SFA12K SFA12K-E SFA10K-X SFA10K-E EF3015 S2A6620 SFA 2.1.1 Supported Supported Supported Supported Supported N/A Not Supported SFA 2.0.0 Not Supported Supported Supported Supported Not Supported N/A Not Supported T230R30 N/A N/A N/A N/A N/A Supported N/A 4.0 Installation EXAScaler is released as a pair of ISO files that are applicable to different configurations; a CentOS ISO for block storage and SFA ISO for embedded systems. For all systems the default root password is DDNSolutions4U and should be changed by the user for security purposes. 4.1 Block storage OS Support: CentOS 6.4 (es-hpc-2.0.1-cent-r9-x86_64-dvd.iso) Server Support: Dell R620/R720 Server Dell R610/R710 Server Other x86_64 server with local storage should work with the caveat that customizations may be necessary to the EXAScaler install process To install EXAScaler on an external server, burn the ISO file to a CD/DVD, insert into a node, and boot from the optical drive. The install process will complete automatically with no additional user import required. If desired the user can control the install process by using the keyboard at the grub prompt and selecting a menu item. If no key is pressed, EXAScaler will be installed to the server after a 60-second delay. By default eth0 will be configured to use DHCP to obtain an IP address automatically if possible. Care should be taken to remove any storage connections to a server (IB or FC) during install or data loss may result. DataDirect Networks EXAScaler 2.0.1 Product Release Notes, Rev. A0 5
After the install has completed the ISO will be ejected and the node will reboot. Repeat the procedure for each server. 4.2 Embedded OS Support: CentOS 6.4 (es-sfa-2.0.1-r9-x86_64-dvd.iso) Embedded VM Support: SFA12K-20E SFA10K-E VM images reside on the system drives. EXAScaler software is typically pre-installed in the embedded appliance. NOTE: If you are using the external MDS setting with embedded OSS VMs and block-storage attached MDSes, then the non-embedded version of EXAScaler should be used on the external MDS servers. 5.0 Software Versions These are the versions of important software included in EXAScaler v2.0.0: Lustre 2.4.1 CentOS6.4 Linux Kernel 2.6.32-358.18.1.el6.x86_64 "inkernel" MLNX OFED 1.5.3-4.0.41 exascaler-tools-1.1-11 sfx-server-1.0-20 sfx-client-1.0.26 6.0 Upgrades EXAScaler 2.0.1 supports upgrading from systems already running EXAScaler version 2.0.0, 1.6.2, 1.6.1, 1.6.0 or 1.5.x. It does not support upgrading from systems running any other EXAScaler releases. Section 6.1 covers upgrading systems from EXAScaler versions 1.6.0 through 2.0.0. EXAScaler 1.5.x will need to be upgraded to 1.6.x before it can be upgraded to 2.0.1. For questions, please contact DDN support at support@ddn.com. 6.1 Upgrading Systems Running EXAScaler Versions 1.6.0 through 2.0.0 The update pack is distributed as a single tar file which is applicable to all EXAScaler installs, regardless of original flavor (CentOS/SFA-E). Before starting, download the tar file, ES_2.0.1-9.tar, and distribute it to every node. Detailed instructions for all platforms: 1. Stop all Lustre file systems on the node to be upgraded. 2. Copy the tar file onto the node and extract the contents. DataDirect Networks EXAScaler 2.0.1 Product Release Notes, Rev. A0 6
3. cd into the ES_2.0.1-9 directory. 4. Run the update_node script. If there are errors at this stage, the contents of the update_node script will have to be executed by hand. The script can be run as:./update_node. 5. Reboot the node. 6. Repeat steps 1 5 for all nodes. All servers must be upgraded at the same time, while some or all clients may be upgraded. NOTE: Lustre 2.4 introduces Distributed Namespace. Clients running a release prior to the Lustre 2.4 release can only see the namespace hosted by MDT0 and will return an I/O error if an attempt is made to access a directory on another MDT. NOTE: If you want to enable the quota feature after upgrade to Lustre 2.4, enter on both the MDS and OSTs: tunefs.lustre quota 6.2 Upgrading Systems Running EXAScaler v1.5.x to 1.6.2 EXAScaler 1.5 is based on CentOS 5 with Lustre 1.8. EXAScaler 1.6 is based on CentOS 6 with Lustre 2.1. EXAScaler 1.5 systems can be upgraded to run 1.6 however there is not a simple process for doing this. The process needs to be performed by DDN directly and in scale is comparable to doing an initial deployment when a system is delivered. The contents of the file system can be maintained during the upgrade. 6.2.1 Incompatibilities Lustre 2.x drops support for Lustre 1.6 clients. It is believed very few customers still run 1.6 clients however there are some so a check does need to be made. Both Lustre 1.8 and Lustre 2.1 support Lustre 1.8 clients however modern client versions are strongly recommended. At a minimum all clients should be running 1.8.6 and preferably Lustre 1.8.9. 6.2.2 Process Due to complexity of the process the upgrade should be performed by DDN employees only. It is possible to complete in one day; however, for more complex systems, two are recommended. Physical access to the system is a prerequisite due to the need to use ISO images to install nodes. Any clients running Lustre prior to 1.8.6 should be upgraded to Lustre 1.8.9. All Lustre clients should be shut down or disconnected from the file system. The exascaler.conf file from /etc on the Lustre servers should be copied to somewhere elsewhere on the network. The Lustre servers should be shut down. Any physical connections to the storage should be disconnected. The ISO cannot be installed when there are storage controllers connected to the servers. The servers should be re-installed from the 1.6 ISO. exascaler.conf should be copied back onto the nodes. The system should be configured as normal, skipping the es_mkfs step. The servers should be reconnected to the storage and the servers rebooted. es_tunefs writeconf needs to be run on every node to reset the Lustre configuration on the storage. Note that the contents of the file system remain intact. DataDirect Networks EXAScaler 2.0.1 Product Release Notes, Rev. A0 7
The file system should be re-started one target at a time, initially the MGS, then the MDT, then each OST in order. Any site-specific customization required should be re-applied to the new OS on the servers. The file system can now be recommissioned and any clients can now be allowed to reconnect. 7.0 Resolved Issues Re-instated support for network (Ethernet) bonding. Support for more than 16 MDTs in a single file system. Support for servers using Intel Matrix RAID for internal drives. Fixed some scalability problems in pacemaker to allow larger HA domains. 8.0 Known Issues 9.0 Support On block-storage solutions, multipath configuration is not automatically handled, copy /etc/multipath.conf.ddn to /etc/multipath.conf and add appropriate entries for the storage devices. NOTE: When requesting support, please run es_showall --upload-to-ddn and inform DDN of the file name if the system has Internet connectivity or run es_showall and send the generated debug dump to DDN if it is not. Please contact DataDirect Networks Support at any time for assistance. Support can be reached by the following methods: Web: http://www.ddn.com/support-overview Email: support@ddn.com North America: +1.888.634.2374 International: +1.818.718.8507 DataDirect Networks EXAScaler 2.0.1 Product Release Notes, Rev. A0 8
10.0 Appendix A: Supported Solution Components The following solution components are supported by this release of EXAScaler: Storage SFA12K (SFA OS 2.0 and above) SFA12K-E (SFA OS 2.0 and above) SFA10K (SFA OS 2.0 and above) SFA10K-E (SFA OS 2.1.1) SFA7700 (SFA OS 2.1.1) EF3015 (External Server - T230R30) SAN Connectivity Fibre-Channel (8G) InfiniBand (QDR, FDR) Servers x86_64 architecture only Client Networks 1G/10G/40G Ethernet InfiniBand (IPoIB, RDMA verbs on QDR/FDR) Bonded 1G/10G/40G Ethernet DataDirect Networks EXAScaler 2.0.1 Product Release Notes, Rev. A0 9
World Headquarters 9351 Deering Avenue Chatsworth, CA 91311 ddn.com Phone: +1.818.700.7600 Fax: +1.818.700.7601