EMC DiskXtender for NAS Release 3.0

Transcription

1 EMC DiskXtender for NAS Release 3.0 Theory of Operations P/N REV A02 EMC Corporation Corporate Headquarters: Hopkinton, MA

2 Copyright EMC Corporation. All rights reserved. Published July, 2007 EMC believes the information in this publication is accurate as of its publication date. The information is subject to change without notice. THE INFORMATION IN THIS PUBLICATION IS PROVIDED AS IS. EMC CORPORATION MAKES NO REPRESENTATIONS OR WARRANTIES OF ANY KIND WITH RESPECT TO THE INFORMATION IN THIS PUBLICATION, AND SPECIFICALLY DISCLAIMS IMPLIED WARRANTIES OF MERCHANTABILITY OR FITNESS FOR A PARTICULAR PURPOSE. Use, copying, and distribution of any EMC software described in this publication requires an applicable software license. For the most up-to-date listing of EMC product names, see EMC Corporation Trademarks on EMC.com. All other trademarks used herein are the property of their respective owners. 2 EMC DiskXtender for NAS Release 3.0 Theory of Operations

3 Contents Contents... 3 Preface Chapter 1 Chapter 2 Components of the DX-NAS Solution About the EMC Celerra About the FileMover Functionality in DART About the NetApp Filer About the DX-NAS Policy Engine Recalling Migrated Files About Secondary Storage About NAS server to NAS server Migration NAS Server to Separate Host Migration NAS Server Direct to EMC Centera NAS Server to Other Storage Device Migration DX-NAS server port usage Celerra requirements NetApp filer requirements EMC Centera Recommendations Basic DX-NAS Configurations About File Migration Creating Jobs in the GUI Running Jobs Monitoring Jobs Stopping Jobs EMC DiskXtender for NAS Release 3.0 Theory of Operations 3

4 Contents File Recall Recalling All Files from Celerra Recalling All Files from NetApp About Orphan Files About Source Scans Orphan File Management About File Versioning Storage Reporting About the Migration Report About the Orphan File Report About the Primary Storage Report About the Secondary Storage Report About Printing Reports Error Reporting About Alerts Printing Alerts About Logs Chapter 3 Chapter 4 Chapter 5 Differences between Celerra and NetApp with DX-NAS Differences between DX-NAS on Microsoft Windows and UNIX Configuration Planning and Performance Tuning DX-NAS Configurations Two Data Movers on the Same Celerra One Data Mover on the Same Celerra Two Celerras NetApp Filer to Celerra Celerra or NetApp to DX-NAS Server Local Disk Celerra or NetApp to Network Disk Destination Celerra or NetApp to CLARiiON ATA Celerra or NetApp to FSM-Managed File System Celerra or NetApp to DiskXtender for Windows Server Celerra or NetApp Direct to EMC Centera Primary Storage Considerations Effects of Source File Size on DX-NAS Effects of Volatile Source Files on DX-NAS The Effect of the Full Readback Method Primary Storage Sizing Formula EMC DiskXtender for NAS Release 3.0 Theory of Operations

5 Contents DX-NAS Server Local Disk Considerations Sizing Secondary Storage Effect of Source File Size on Secondary Storage Effect of Volatile Source on Secondary Storage Managing Secondary Storage with Orphan File Scans Secondary Storage Considerations Deploying Multiple DX-NAS Servers Chapter 6 Chapter 7 Chapter 8 Backup and Recovery for DX-NAS Backing Up Full Data Orphan File Management with Full Data Backups Forcing a Full Data Backup to Read Stubs in Passthrough Mode Backing up Stubs Preparing primary storage for Stub Backups on Celerra Performing Orphan File Management without Backup Synchronization Performing Orphan File Management with Backup Synchronization Backing up Secondary Storage Restoring Stubs from Backup Media About Orphan File Previews with Restored Stubs Determining Offline File Name from the GUI Determining File Attributes from the CLI Using Celerra CheckPoints with DX-NAS About Checkpoints Checkpointing Source Directories Copying Stubs from Checkpoint Directories About Checkpoints and Readback Method Override Using a Checkpoint Directory as a DX-NAS Source Checkpointing Secondary Storage Integrating DX-NAS with other Products Using Anti-Virus Software with DX-NAS Using VisualSRM Configuring VisualSRM to Work with DX-NAS Integrating with DiskXtender for Windows Integrating with DXUL File System Manager EMC DiskXtender for NAS Release 3.0 Theory of Operations 5

6 Contents Integrating with DXUL Storage Manager Configuring DXUL-SM with DX-NAS DXUL-SM and Celerra DX-NAS Using EMC Celerra HighRoad Chapter 9 Chapter 10 High Availability Solutions for DX-NAS High Availability on the Celerra Data Mover Failover Celerra Replication High Availability on NetApp High Availability for the DX-NAS server DX-NAS Server and Microsoft Server Clustering A DX-NAS Configuration on a Microsoft Cluster High Availability for Secondary Storage HA Secondary Storage on the Celerra HA Secondary Storage on the NetApp HA Secondary Storage on a Microsoft or Linux Cluster HA on an EMC Centera Using DX Windows HA on an EMC Centera Using FSM Disaster Tolerant Strategies for DX-NAS Disaster Tolerant Strategies About Synchronous Data Mirroring About Point-In-Time Data Mirroring About Asynchronous Data Mirroring Failing Over a Disaster Tolerant Configuration Index EMC DiskXtender for NAS Release 3.0 Theory of Operations

7 Figures 1 EMC Celerra provides file sharing for clients Celerra perspective of file access on local disk arrays Celerra perspective on file access for a migrated or offline file NetApp Filer provides file sharing for clients DX-NAS migrating files from one NAS server to another DX-NAS migrating files to a remote host DX-NAS migrating files directly to an EMC Centera DX-NAS migrating files to other storage devices using FSM Sources and Destinations in the DX-NAS GUI Example of creating a job using the CLI The Policy Wizard combines DX-NAS elements and search behaviors Readback methods available for the Celerra Migration Job Preview in the DX-NAS GUI Orphan Job Preview in the DX-NAS GUI Details of a migration job in progress Details of an orphan file job in progress Details of a source scan job in progress Connection string listing from Celerra Examples of how orphan files can be created Example of directory structure on secondary storage Example of tracking multiple versions of files on Celerra Example of a Migration Report An example of an Orphan File Report Alerts Tab in the DX-NAS GUI Two Data Movers on the Same Celerra System Configuration One Data Mover Configuration Celerra to Destination Celerra System Configuration NetApp Filer to Celerra Destination Celerra or NetApp to Local Disk Configuration Celerra or NetApp to Network Disk Configuration EMC DiskXtender for NAS Release 3.0 Theory of Operations 7

8 31 Celerra or NetApp to CLARiiON ATA Configuration Celerra or NetApp to FSM File System Configuration Celerra or NetApp to FSM to AVALONidm Configuration Celerra or NetApp to DiskXtender for Windows Configuration DX-NAS migrating files directly to an EMC Centera Example of full data backup of migrated files on Celerra Stub backup with corresponding secondary storage backup Example of stub backup on EMC Legato Networker Example of using the Windows file attributes GUI Example of using the CLI fileattrs command Example of the pfs12 file system with a WorkingData directory Example of the pfs12 file system with checkpoints DX-NAS policy set to find PDF files VisualSRM filter set to find PDF files DXUL-SM 2.9 running on different machine than DX-NAS DXUL-SM 2.9 running on Solaris with DX-NAS DX-NAS Components that need HA configuration A Celerra configured with a standby Data Mover for failover A Celerra configured with data mover failover and replication AutoStart Configuration DX-NAS installed in a two-node cluster configuration DX-NAS using a Celerra to provide HA for secondary storage FSM configured for HA using Autostart Example of Synchronous Mirroring with DX-NAS Example of Asynchronous Mirroring with DX-NAS EMC DiskXtender for NAS Release 3.0 Theory of Operations

9 Tables Title Page 1 Ports used by DX-NAS Support matrix for Data ONTAP Example of information file used to track orphan files Column definition for the Primary Storage Report Column definition for the Primary Storage Report Differences between Celerra and NetApp as source hosts Differences between DX-NAS on Microsoft Windows and UNIX Source File Size Considerations for DX-NAS migration Source File System Volatility Considerations for DX-NAS DX-NAS Server Disk Space Requirements on Microsoft Windows Sizing Secondary Storage DX-NAS and cluster specifications EMC DiskXtender for NAS Release 3.0 Theory of Operations 9

10 10 EMC DiskXtender for NAS Release 3.0 Theory of Operations

11 Preface As part of an effort to improve and enhance the performance and capabilities of its product lines, EMC periodically releases revisions of its hardware and software. Therefore, some functions described in this document may not be supported by all versions of the software or hardware currently in use. For the most up-to-date information on product features, refer to your product release notes. If a product does not function properly or does not function as described in this document, please contact your EMC representative. Audience This document is part of the EMC DiskXtender for NAS (DX-NAS), Release 3.0, documentation set, and is intended for use by trained DX-NAS system administrators. Readers of this document are expected to be familiar with the following topics: EMC Celerra administration EMC Centera administration Celerra FileMover API (formerly known as DHSM) NetApp Filer administration Solaris, Linux, or Microsoft Windows operating system configuration and management EMC DiskXtender for NAS Release 3.0 Theory of Operations 11

12 Preface Related documentation Related documents include: DiskXtender for NAS Release 3.0 Multiplatform Version Release Notes DiskXtender for NAS Release 3.0 Microsoft Windows Version Installation and Configuration Guide DiskXtender for NAS Release 3.0 UNIX/Linux Version Installation and Configuration Guide DiskXtender for NAS Release 3.0 Multiplatform Version Administrator s Guide Autostart Module For DiskXtender for NAS Release 3.0 Using Celerra FileMover Technical Module Conventions used in this document EMC uses the following conventions for special notices. Note: A note presents information that is important, but not hazard-related.! CAUTION A caution contains information essential to avoid data loss or damage to the system or equipment. The caution may apply to hardware or software.! IMPORTANT An important notice contains information essential to operation of the software. The important notice applies only to software. 12 EMC DiskXtender for NAS Release 3.0 Theory of Operations

13 Preface Typographical conventions EMC uses the following type style conventions in this document: Normal Used in running (nonprocedural) text for: Names of interface elements (such as names of windows, dialog boxes, buttons, fields, and menus) Names of resources, attributes, pools, Boolean expressions, buttons, DQL statements, keywords, clauses, environment variables, filenames, functions, utilities URLs, pathnames, filenames, directory names, computer names, links, groups, service keys, file systems, notifications Bold: Italic: Courier: Courier bold: Used in running (nonprocedural) text for: Names of commands, daemons, options, programs, processes, services, applications, utilities, kernels, notifications, system call, man pages Used in procedures for: Names of interface elements (such as names of windows, dialog boxes, buttons, fields, and menus) What user specifically selects, clicks, presses, or types Used in all text (including procedures) for: Full titles of publications referenced in text Emphasis (for example a new term) Variables Used for: System output, such as an error message or script URLs, complete paths, filenames, prompts, and syntax when shown outside of running text Used for: Specific user input (such as commands) Courier italic: Used in procedures for: Variables on command line User input variables < > Angle brackets enclose parameter or variable values supplied by the user [ ] Square brackets enclose optional values Vertical bar indicates alternate selections - the bar means or { } Braces indicate content that you must specify (that is, x or y or z)... Ellipses indicate nonessential information omitted from the example EMC DiskXtender for NAS Release 3.0 Theory of Operations 13

14 Preface Where to get help EMC support, product, and licensing information can be obtained as follows. Product information For documentation, release notes, software updates, or for information about EMC products, licensing, and service, go to the EMC Powerlink website (registration required) at: Technical support For technical support, go to EMC Customer Service on Powerlink. To open a service request through Powerlink, you must have a valid support agreement. Please contact your EMC sales representative for details about obtaining a valid support agreement or to answer any questions about your account. Your comments Comments and suggestions about our product documentation are always welcome. To provide feedback: 1. Go to: 2. Click the Feedback link. 14 EMC DiskXtender for NAS Release 3.0 Theory of Operations

15 Invisible Body Tag 1 Components of the DX-NAS Solution The DiskXtender File System Manager for NAS (DX-NAS) is an archiving solution that allows you to free storage space on your NAS server while maintaining NAS client access to archived files. The solution consists of integrating the following three components: An EMC Celerra or NetApp Filer The DX-NAS 3.0 policy engine Secondary storage Components of the DX-NAS Solution 15

16 PS0 PS1 PS2 PS3 PS4 SMB0 SMB1 Components of the DX-NAS Solution About the EMC Celerra The EMC Celerra provides Network Attached Storage (NAS) to UNIX (Solaris or Linux) and Microsoft Windows clients. Windows clients connect to the Celerra using the CIFS protocol, and UNIX clients connect using NFS. From the perspective of these clients, they are simply mapping drives to, or mounting volumes on, a specialized server in the Celerra cabinet called a Data Mover. Windows NAS Client Celerra SB0 SB1 SB2 SB3 SB4 SB5 SB6 SB7 SB8 SB9 SB10 SB11 SB12 SB13 SB14 SB15 UNIX NAS Client Figure 1 EMC Celerra provides file sharing for clients Generally, NAS clients are not concerned with the configuration of the file system on the back-end disk array nor the physical location of their files. However, from the Celerra perspective, the Data Mover manages the client connection and functions as an intermediate layer between the NAS clients and the physical storage (see Figure 2). 16 EMC DiskXtender for NAS Release 3.0 Theory of Operations

17 Components of the DX-NAS Solution Celerra NAS Client Windows NAS Client UNIX Data Mover G:\file.avi /home/file.mpeg File System Back-End Disk Array Figure 2 Celerra perspective of file access on local disk arrays As with all storage solutions, there are physical limits to the amount of storage that a Celerra can manage. Consequently, functionality was included in the Data Mover operating system (called DART) to allow files to be moved from a NAS file system to another location while preserving NAS client access to those files. The DX-NAS solution exploits this functionality by moving files through the DART FileMover API. (In earlier releases of DART, this functionality was called DHSM.) About the FileMover Functionality in DART When a file is moved to another location using the Celerra File Mover API, the only change in the way the NAS client sees the file is that the file may now have an offline attribute set. However, from the perspective of the Celerra, the file has now been moved to a separate location (most often referred to as secondary storage) leaving only a small stub on the Celerra. This stub is one block (usually 8 KB) in size and contains file attribute information as About the EMC Celerra 17

18 Components of the DX-NAS Solution well as the location of the data portion of the file. 1 When a NAS client attempts to read a file that is offline, the Data Mover retrieves the file from the secondary location (see Figure 3). Celerra NAS Client Windows NAS Client UNIX Data Mover :\file.avi /home/file.mpeg Secondary Storage on Remote Host Shared File System File System Back-End Disk Array Figure 3 Celerra perspective on file access for a migrated or offline file 1. In some cases if additional alternate data streams are associated with a file, the stub left behind on the Celerra will use one block for each data stream. In these cases the stub could be as large as 64 KB. 18 EMC DiskXtender for NAS Release 3.0 Theory of Operations

19 Components of the DX-NAS Solution About the NetApp Filer NetApp filers provide Network Attached Storage (NAS) to UNIX and Microsoft Windows clients as well, and DX-NAS can be configured to move files to secondary storage while leaving a stub on the NetApp Filer for NAS clients to access. The stub on the NetApp filer is also 8KB in size; however, DX-NAS will not migrate a file smaller than 8KB off the NetApp filer. All migrations off the NetApp filer use a readback method of Full. Note: NetApp file migration is not available for DX-NAS servers running on Solaris. Windows NAS Client NetApp UNIX NAS Client Figure 4 NetApp Filer provides file sharing for clients About the NetApp Filer 19

20 Components of the DX-NAS Solution About the DX-NAS Policy Engine The DX-NAS policy engine does not run on the Celerra or on the NetApp Filer. Rather, it runs on a separate host and connects to Celerra or NetApp like any other NAS client. It has four primary functions: Identifying files that need to be migrated Copying the files to the new location Creating a stub on Celerra or NetApp for continued access by NAS Clients Performing maintenance on orphaned files (i.e. stubs that no longer have a corresponding data portion on secondary storage, or files on secondary storage than no longer correspond to a stub) The DX-NAS administrator establishes criteria to identify which files should be archived or migrated to secondary storage. This criteria can be based on things like file size, file creation date, or days that have passed since the file was last accessed. To facilitate the articulation of this criteria, the DX-NAS GUI and CLI offer a preview function that allows administrators to see what files meet the defined criteria before they are actually moved. The DX-NAS policy engine is the component of the solution that copies the files. Consequently, DX-NAS must be able to access or mount locations on Celerra and NetApp as well as on any hosts where the secondary storage will be located. (DX-NAS needs to have full read and write privileges on the drives or mount points being used as secondary storage.) Recalling Migrated Files When NAS clients access migrated files on Celerra or NetApp the data portion of the files is recalled. On the Celerra, the recall is performed by the Celerra data mover. When the file was originally migrated, DX-NAS wrote the location of the data file into the stub, and the data mover uses this information to retrieve the file. Consequently, even if DX-NAS is not present, the data mover on the Celerra can retrieve the data portion of migrated files. The exception is configurations where DX-NAS is migrating files from NetApp or migrating files directly to an EMC Centera. 20 EMC DiskXtender for NAS Release 3.0 Theory of Operations

21 Components of the DX-NAS Solution In the case of a Celerra recalling files directly from an EMC Centera, the Celerra data mover contacts the DX-NAS server, and the DX-NAS server retrieves the requested data from the EMC Centera and passes it back to the Celerra. (DX-NAS must be present for a Celerra to recall files that have been migrated directly to an EMC Centera.) Similarly, in all NetApp configurations, the DX-NAS server retrieves the data portion of migrated files. (DX-NAS must be present for NetApp to recall any files that have been migrated.) About the DX-NAS Policy Engine 21

22 Components of the DX-NAS Solution About Secondary Storage In the DX-NAS solution, secondary storage can be any file system on any host that the DX-NAS policy engine can connect to. In Microsoft Windows, DX-NAS needs to be able to access secondary storage using UNC paths. For Solaris and Linux, DX-NAS needs to be able to mount the file system on secondary storage. While there are a number of ways that DX-NAS can be implemented, secondary storage configurations generally fall into the following categories: NAS server file systems File systems on remote hosts EMC Centera Other Storage Devices About NAS server to NAS server Migration In some cases system administrators merely want to move files from expensive storage in a Celerra or NetApp cabinet to less-expensive storage in the same cabinet or on a different Celerra or NetApp Filer (see Figure 5). Expensive storage might be, for example, a file system with a RAID 5 configuration. In such a case, a system administrator might want to archive older files to a file system which is just a bunch of disks (JBOD). Using DX-NAS terminology, then, the RAID 5 file system would be the source, and the JBOD file system would be destination. 22 EMC DiskXtender for NAS Release 3.0 Theory of Operations

23 PS0 PS1 PS2 PS3 PS4 SM B0SM B1 SB8 SB6 SB4 Components of the DX-NAS Solution DX-NAS Source Data Mover 1 Destination Data Mover 2 SB0 SB1 SB2 SB3 SB4 SB5 SB6 SB7 SB8 SB9 SB2 SB0 PS0 PS1 PS2 PS3 PS4 SMB0 SMB1 SB9 PS0 PS1 PS2 PS3 PS4 SM B0SM B1 SB10 SB12 SB14 SB11 SB13 SB15 SB0 SB1 SB2 SB3 SB4 SB5 SB6 SB7 SB8 SB7 SB9 SB5 SB3 SB1 SB10 SB12 SB14 SB11 SB13 SB15 Figure 5 DX-NAS migrating files from one NAS server to another NAS Server to Separate Host Migration Another common DX-NAS configuration is to migrate files to a file system on a separate host. These destination file systems can be a CIFS (Windows) or an NFS (UNIX) based file system, and the only requirement is that both the DX-NAS policy engine and the Celerra (if part of the configuration) need to be able to access secondary storage using a UNC paths (Windows) or by mounting the file system on the secondary storage as an NFS share (Solaris or Linux). SB14 SB15 DX-NAS Remote Host Data Mover 1 Source SB12 SB10 SB13 SB11 NFS/CIFS Destination Celerra or NetApp Figure 6 DX-NAS migrating files to a remote host About Secondary Storage 23

24 PS0 PS1 PS2 PS3 PS4 SM B0SM B1 Components of the DX-NAS Solution NAS Server Direct to EMC Centera It is also possible for DX-NAS to migrate data directly to an EMC Centera (see Figure 7). This requires the Celerra to be running Dart 5.5 and the EMC Centera to be running CentraStar version 2.4, 3.0, 3.1, or higher. Any version of NetApp Ontap can migrate to the Centera. In a Celerra configuration, DX-NAS provides a web server interface to the EMC Centera. When the Celerra needs to recall a file that has been migrated to the EMC Centera, it makes an HTTP call to the DX-NAS web server which retrieves the file from the EMC Centera and supplies it to the Celerra. This configuration requires you to identify the EMC Centera secondary storage as type http in the connection string on the Celerra. For example a connection string for direct to EMC Centera migration might look like this: fs_dhsm -connection myfilesystem -create -type http -secondary -httpport cgi y With NetApp, the DX-NAS server tracks all migrated files in its database, and DX-NAS knows if a file needs to be retrieved from the EMC Centera. Note: With EMC Centera, the DX-NAS server is part of the restore path for migrated files. Consequently, if you want to uninstall DX-NAS you will want to migrate all your files back first. For more information on how this is accomplished see Recalling All Files from Celerra on page 45. Data Mover 1 SB10 SB12 SB14 SB11 SB13 SB15 DX-NAS Source SB0 SB2 SB4 SB6 SB8 SB1 SB3 SB5 SB7 SB9 Destination (DX-NAS HTTP Server) Celerra or NetApp EMC Centera Figure 7 DX-NAS migrating files directly to an EMC Centera 24 EMC DiskXtender for NAS Release 3.0 Theory of Operations

25 Components of the DX-NAS Solution NAS Server to Other Storage Device Migration It is possible to migrate files to devices such as a tape library. However the migration must be done indirectly through an intermediary product like the DiskXtender File System Manager (DX-FSM) or DiskXtender for Microsoft Windows (DX-Windows). With DX-FSM or DX-Windows, a managed file system is created where all the files placed on that file system are moved to an EMC Centera or to a supported storage device. Using DX-FSM or DX-Windows, then, DX-NAS would migrate files to the managed file system, which in turn would migrate them to the storage device. (Figure 8) The restore path would be similar: when an offline file is accessed, Celerra or DX-NAS (in the case of NetApp) would attempt to read the file from the FSM managed file system, and it would have to wait for DX-FSM or DX-Windows to retrieve the file from the About Secondary Storage 25

26 PS0 PS1 PS2 PS3 PS4 SMB0 SMB1 Components of the DX-NAS Solution storage device before transferring the contents of the file back to the requesting client. Windows 2000/2003 (Destination) SB8 SB10 SB12 SB14 SB9 SB11 SB13 SB15 Secondary storage SB0 SB2 SB4 SB6 SB1 SB3 SB5 SB7 Celerra or NetApp DX-NAS DiskXtender FSM, or DiskXtender for Windows Tertiary storage NAS Device MediaStor EMC Centera Drive, library, and tower devices Figure 8 DX-NAS migrating files to other storage devices using FSM 26 EMC DiskXtender for NAS Release 3.0 Theory of Operations

27 Components of the DX-NAS Solution DX-NAS server port usage Table 1 on page 27 provides a list of the ports used by the DX-NAS server. Table 1 Ports used by DX-NAS Port number Description 139 NetApp callback (not configurable) Note: If DX-NAS is used to migrate files from a NetApp filer, you cannot run Samba on the same machine where the DX-NAS server is running. 445 CIFS 2049 NFS 1976 DX-NAS database 3682 EMC Centera CLI EMC Centera Viewer EMC Centera communication 8080 DX-NAS web server DX-NAS server (configurable) If you configured a fire wall on the network where the DX-NAS server and client are located, enable access to ports 1976, 11001, and DX-NAS server port usage 27

28 Components of the DX-NAS Solution Celerra requirements The DX-NAS software supports the following versions of data access in real time (DART): or later or later Note: Versions of DART that are before are not supported. The EMC NAS Support Matrix provides more information on supported software and hardware, Fibre Channel switches, and applications for Celerra network-attached storage products. The EMC NAS Support Matrix can be viewed at 28 EMC DiskXtender for NAS Release 3.0 Theory of Operations

29 Components of the DX-NAS Solution NetApp filer requirements DX-NAS release 3.0 is the first release to support NetApp filers as primary storage. The following requirements for using NetApp filers apply: DX-NAS server software installed on either Red Hat Linux or Microsoft Windows. Note: You cannot use a NetApp filer as primary storage if you are using DX-NAS server software on Solaris. Data ONTAP version 7.0.x or later installed on the NetApp filer. Table 2 on page 29 provides a support matrix for ONTAP versions that can be used with DX-NAS. Table 2 Support matrix for Data ONTAP ONTAP version File systems DX-NAS server 7.0.x CIFS only Note: ONTAP 7.0.x does not support NFS. Microsoft Windows CIFS Microsoft Windows NFS Red Hat Linux and CIFS Microsoft Windows NFS Red Hat Linux NetApp filer requirements 29

30 Components of the DX-NAS Solution EMC Centera Recommendations When migrating from primary storage to an EMC Centera, consider the following recommendations: Configure the EMC Centera to use its single instance storage abilities rather than use the DX-NAS check-summing feature. Single instance storage is the ability of the Centera to recognize that the same unchanged file is being migrated more than once and only store one instance of the file. By default, this feature comes turned on regardless of the Centera model (Basic, Governance, or Compliance Use retention classes rather than retention periods. Where possible, use EMC Centera virtual pools. 30 EMC DiskXtender for NAS Release 3.0 Theory of Operations

31 Invisible Body Tag 2 Basic DX-NAS Configurations The primary objective of DX-NAS is to move files from primary storage to secondary storage. To accomplish this effectively, DX-NAS needs to provide the following basic services: File Migration File Recall File Versioning Orphan File Management Storage Reporting Error Reporting Basic DX-NAS Configurations 31

32 Basic DX-NAS Configurations About File Migration In DX-NAS, file migration is accomplished by creating and running jobs. Jobs consist of the following elements: A Source the folder or directory on a Celerra or NetApp Filer where the policy engine looks for files that are candidates for migration. On Microsoft Windows a source is specified as a UNC path (for example, \\server2\wip). On Solaris or Linux a source is specified as a a directory or mount point (for example, /Celerra/server2/wip). A Destination the folder or directory where the file will be migrated to. The machine the DX-NAS server is running on needs to be able to access the destination in one of the following ways: As a local directory As a UNC path (for CIFS shares) As a mounted NFS share (for Solaris or Linux) A Policy a description of what files to migrate and how to migrate them. When a policy is created, match criteria is established to identify which files are candidates for migration. This match criteria can be based on a variety of file attributes like file name, file size, last modification time, etc. After jobs have been created, the DX-NAS server can be scheduled to run the jobs at times that will have the least impact on normal NAS client use. Creating Jobs in the GUI The structure of a job is implied in the layout of the DX-NAS GUI. The tabs in the GUI reflect the elements or steps necessary to create a job. 32 EMC DiskXtender for NAS Release 3.0 Theory of Operations

33 Basic DX-NAS Configurations In Figure 9, the first three tabs on the user interface correspond to the elements of a job identified above and represent the first three steps in creating a job. Figure 9 Sources and Destinations in the DX-NAS GUI The fourth step--the step of actually creating a job on the Jobs tab of the GUI--consists primarily in choosing which policy the job will implement. About File Migration 33

34 Basic DX-NAS Configurations Jobs can also be created in the DX-NAS command line interface (CLI). Figure 10 shows an example of the commands used to create a job in the CLI. 34 EMC DiskXtender for NAS Release 3.0 Theory of Operations

35 Basic DX-NAS Configurations DXNAS Ready>host -c server229.lab.local -type celerra -user dhsm_user -pass dhsm_user -i "Celerra host from CLI" Source Host Configurations: ========================== SourceHost :server229.lab.local Description=Celerra host from CLI FileMover Username=dhsm_user NAS Filer Type=EMC Celerra DART Hostname=server229.lab.local Number of sources=0 DXNAS Ready>source -c "Celerra CIFS Source" -h server229.lab.local -p \\server229.lab.local\wayne_test\win -i "Celerra CIFS source from CLI" Adding source : Celerra CIFS Source Source Configurations: ========================== Source :Celerra CIFS Source Description=Celerra CIFS source from CLI Sourcehost=server229.lab.local Source path=\\server229.lab.local\wayne_test\win DXNAS Ready>dest -c "Windows Dest for Celerra" -type fs -h TP lab.local -p \\TP lab.local\WinStorage -fmp \\TP lab.local\WinStorage -i "Windows Storage for Celerra referencing a connection string." Destination Configurations: ========================== Destination :Windows Dest for Celerra Type=Disk Filesystem Offline directory=\\tp lab.local\winstorage Description=Windows Storage for Celerra referencing a connection string. Destination path=\\tp lab.local\winstorage Host Name=TP lab.local DXNAS Ready>policy -c "Migrate all.mpg to File System" -type mig -s "Celerra CIFS Source" -dest Windows Dest for Celerra" -mode filemover -rb passthrough -rule "-name *.mpg" -md5 off -i "Migrate to Local Disk" Policy Configurations: ========================== Policy :Migrate all.mpg to File System Migration Source=Celerra CIFS Source Match Criteria= Use MD5 deduplication=false Policy Type=Migration Policy Readback Method=Passthrough Description=Migrate to Local Disk Destination=Windows Dest for Celerra Source Host=server229.lab.local Source Path=\\server229.lab.local\wayne_test\Win Find Match Criteria=-type f -name *.mpg Celerra Offline Directory=\\TP lab.local\WinStorage Destination Path=\\TP lab.local\WinStorage DXNAS Ready>job -c "Migrate MPG" -p "Migrate all.mpg to File System" -m 15 -t 0 -i "Job created from CLI" Job Configurations and Running Jobs: ========================== Job :Migrate MPG Migration Source=Celerra CIFS Source Readback Method=Passthrough Destination=Windows Dest for Celerra Time Limit (minutes)=0.0 Find Match Criteria=-type f -name *.mpg Celerra Offline Directory=\\TP lab.local\WinStorage Match Criteria= Policy Type=Migration Policy Use MD5 deduplication=false Policy=Migrate all.mpg to File System Description=Job created from CLI Source Host=server229.lab.local Source Path=\\server229.lab.local\wayne_test\Win Number of Threads=15 Destination Path=\\TP lab.local\WinStorage Figure 10 Example of creating a job using the CLI About File Migration 35

36 Basic DX-NAS Configurations About Policies There are three types of DX-NAS policies: Migration Policies--which identify files that should be migrated from primary storage to secondary storage Orphan File Policies--which identify migrated files on secondary storage that no longer have a valid corresponding stub on primary storage Source Scan Policies--which identify stubs on primary storage that no longer have valid corresponding files on secondary storage Each of these policies is created with the Policy Wizard that is launched from the Policies tab. Figure 11 The Policy Wizard combines DX-NAS elements and search behaviors 36 EMC DiskXtender for NAS Release 3.0 Theory of Operations

37 Basic DX-NAS Configurations About Readback Method Polices currently support three readback methods: Full--(Celerra or NetApp) The file on secondary storage is recalled, and it replaces the stub on the source. On the Celerra, if attempts to recall data produce an error related to insufficient space or quotas, the Celerra will attempt to re-read the data using the passthrough mode. Full is the only readback method supported on NetApp. Note: Except when using a NetApp source, EMC strongly recommends that you do not use the full readback method. However, if you do use the full readback method with Celerra, then also use the checksum feature. Partial--(Celerra only) The Celerra reads the file on secondary storage and restores only the parts of the file that are being accessed by the requesting client. There is no benefit to be gained by using the partial readback method unless the application recalling the file has the ability to read file using byte offsets. Passthrough--(Celerra only) The Celerra reads the file on secondary storage and passes the data to the requesting client without affecting the stub. Note: On Celerra, the read policy override parameter for the FileMover (or DHSM) connection takes precedence over the value set in the policy. About File Migration 37

38 Basic DX-NAS Configurations Figure 12 illustrates how the readback methods work. Full Readback Method Client Celerra or NetApp Secondary Storage Online Passthrough Readback Method Client Celerra Only Secondary Storage Offline Partial Readback Method Client Celerra Only Secondary Storage Offline Figure 12 Readback methods available for the Celerra 38 EMC DiskXtender for NAS Release 3.0 Theory of Operations

39 Basic DX-NAS Configurations About Match Criteria The behavior of a job is determined largely by the match criteria that is established in its policy. The match criteria tells the policy engine what files to search for. DX-NAS can search for files in the following ways: By name--search for files of a specific name or extension By size--search for files that are at or above a specified size By user--search for files belonging to a specified user By group--search for files belonging to a specified group By accessed date (atime)--search for files based on when they were last accessed. By change time or create time (ctime)--on NFS file systems you can search for files based on when they have had their attributes changed. On CIFS file systems, you can search for files based on when they were created. By modified date (mtime)--search for files based on when they were modified. By file permissions (perm)--search for file based on their file permissions. Note: Source file scan policies don t allow search criteria. A source scan policy examines every file on the specified source. DX-NAS will begin its search in the directory specified as the source and searches through each of its subdirectories. Every file that matches the search criteria is acted upon according to the requirements of the policy. Running Jobs When jobs are run they migrate files, delete files, or generate reports. Additionally, in the case of migration jobs and orphan files jobs you can run the jobs in preview mode to see a list of files that will be affected by the job. This allows you to fine tune your match criteria. About File Migration 39

40 Basic DX-NAS Configurations Jobs can be run or previewed from the CLI or the GUI, and jobs can be scheduled to run at specific times or specific intervals. Figure 13 Migration Job Preview in the DX-NAS GUI 40 EMC DiskXtender for NAS Release 3.0 Theory of Operations

41 Basic DX-NAS Configurations Figure 14 Orphan Job Preview in the DX-NAS GUI Monitoring Jobs If you want to monitor the progress of jobs that are running, you can use the DX-NAS GUI. On the Jobs tab, in the tree display pane, there is a heading named Jobs in progress. All running jobs are listed in under this heading, and by clicking on one of these jobs you can see details about the running jobs. About File Migration 41

42 Basic DX-NAS Configurations Note: You cannot monitor job status in the DX-NAS CLI Figure 15 Details of a migration job in progress 42 EMC DiskXtender for NAS Release 3.0 Theory of Operations

43 Basic DX-NAS Configurations Figure 16 Details of an orphan file job in progress About File Migration 43

44 Basic DX-NAS Configurations Figure 17 Details of a source scan job in progress Stopping Jobs Using the DX-NAS GUI you can stop any job that is running by clicking on the running job and then clicking the Stop button. A confirmation dialog will be displayed, and you can confirm that you want to stop the job. Jobs continue to run as long as the confirmation dialog is displayed. Jobs are not stopped until you click Yes. 44 EMC DiskXtender for NAS Release 3.0 Theory of Operations

45 Basic DX-NAS Configurations File Recall When a file is migrated, the file is copied to secondary storage, and a stub is created on Celerra or NetApp. At this point the file is considered offline, and as long as the relationship between the stub and the data file is maintained, or remains valid, the file can be accessed normally. (This process is represented in Figure 19 on the Migrate line.) However, if the file is migrated with a readback method of full, then when any NAS client accesses the stub, Celerra or NetApp deletes the stub and recalls the entire file from secondary storage. This file recall is transparent to the NAS client. It may take longer to open the file, but it is seen and accessed from the same location with the same name. The only difference may be that, in Microsoft Windows, the user may see that the file has an offline icon and/or offline attribute. Note: As the information necessary to recall the file is contained in the stub on the Celerra, migrated files can be recalled even if DX-NAS is uninstalled. However, there are two exceptions: 1) If you have used DX-NAS to migrate directly to EMC Centera. If this is the case, you will need to recall all your files from the EMC Centera before uninstalling DX-NAS. 2) If you have migrated from a NetApp source. The DX-NAS server is required to recall files from secondary storage on NetApp. Recalling All Files from Celerra If you want to recall all the files from secondary storage that is, un-migrate all your files from secondary storage you can accomplish this by deleting the secondary storage connection string on the control station of the Celerra. The connection string was created on the control station when you configured the Filemover protocol. This connection string contained the information about where to retrieve data for migrated stubs. For example, connection strings might have looked something like this: fs_dhsm -connection CIFSFilesystem1 -create -type cifs -admin administrator -secondary '\\server227.lab.local\winstor' -local_server server227.lab.local fs_dhsm -connection NFSFilesystem1 -create -type nfsv3 -secondary lava2113:/solstor File Recall 45

46 Basic DX-NAS Configurations fs_dhsm -connection myfilesystem -create -type http -secondary -httpport cgi y Connections created with the commands above would have been assigned connection IDs that might have looked like those listed in Figure 18. cid = 2 type = HTTP secondary = state = enabled read policy override = none write policy = full user = options = httpport=8080 cgi=y cid = 3 type = CIFS secondary = \\server227.lab.local\winstor state = enabled read policy override = none write policy = full local_server = server229.lab.local admin = lab.local\wtaylor wins = cid = 4 type = NFSV3 secondary = lava2113:/solstor state = enabled read policy override = none write policy = full options = userootcred=true proto=udp Figure 18 Connection string listing from Celerra You can force a recall of all migrated files from secondary storage by deleting the connection string using the connection ID (CID) and the -recall_policy option. 46 EMC DiskXtender for NAS Release 3.0 Theory of Operations

47 Basic DX-NAS Configurations For example, to force a recall of all files from the CIFS connection shown in the example above, you could enter fs_dhsm -connection CIFSFilesystem1 -delete 8 -recall_policy yes When yes is specified with the -recall_policy option, the Celerra attempts to recall all the files on secondary storage. Depending on how many files have been migrated, this can take some time. While recalling the files the state of the connection will be Migration: ON_GOING. For example, during the time the files are being recalled, if you were to query the connection strings for the file system CIFSFilesystem1 by entering the following command fs_dhsm -connection CIFSFilesystem1 -info you might see something like this: cid = 8 type = CIFS secondary = \\server227.lab.local\winstor\ state = recallonly[ Migration: ON_GOING ] read policy override = none write policy = full local_server = server227.lab.local admin = administrator wins = When the files have all been recalled, the connection is deleted or the state of the connection changes to recallonly. cid = 8 type = CIFS secondary = \\server227.lab.local\winstor\ state = recallonly read policy override = none write policy = full local_server = server227.lab.local admin = administrator wins = You can delete recallonly policies by deleting the connection ID again while by specifying no with the -recall_policy option. For example, you could enter fs_dhsm -connection CIFSFilesystem1 -delete 8 -recall_policy no File Recall 47

48 Basic DX-NAS Configurations With this command, the secondary storage connection ID is permanently deleted. Note: When deleting a Centera connection string, the Celerra wants to have a user associated with the CID before it will delete the connection string using -recall_policy yes. For example, in Figure 18 above, you would need to change CID 2 to user administrator by using a command similar to this: fs_dhsm -connection myfilesystem -modify 2 -user administrator Supply the administrator password usually the password you used to log in to the control station with and the Celerra will update the connection string. Then you can delete the connection string using -recall_policy yes as described above. Recalling All Files from NetApp Unlike Celerra, NetApp does not have a built-in mechanism for recalling all migrated files. In order to recall all files you would need to write a script that opens and reads at least one byte from each file in the directory and sub-directories where you have migrated files. As all NetApp migrations are done with a recall method of full, this will have the effect of recalling all migrated files. 48 EMC DiskXtender for NAS Release 3.0 Theory of Operations

49 Basic DX-NAS Configurations About Orphan Files When DX-NAS migrates a file from primary storage to secondary storage, it essentially breaks the file into two parts: 1. a stub on primary storage 2. the data portion of the file on secondary storage And whenever the relationship between the stub on primary storage and the data file on secondary storage is broken, this constitutes an orphan file. The most common way orphan files are created is when a NAS client deletes or overwrites a stub on primary storage. However, if DX-NAS has migrated the files using the full readback method, then simply reading a file will result in an orphan file being created as well. (See the Migrate and Recall lines in Figure 19.) 1 If for some reason the data file on secondary storage is deleted, then the stub file itself on primary storage is the orphan. (See the Data File Delete line in Figure 19.) 1. Even though NetApp recalls files with full readback, it technically doesn t create an orphan file because all the information needed to recall the data portion of the stub is contained in the DX-NAS database. On NetApp, then, if a migrated file is recalled and then migrated again, DX-NAS will first check to see if the stub information matches what is on secondary storage, and if it does, it simply stubs the file without migrating the data second time. About Orphan Files 49

50 Basic DX-NAS Configurations Primary Storage Secondary Storage Migrate Stub (Offline) Data FIle Recall X Stub (Online) Now-orphaned Data File Data File Delete Now-orphaned Stub X Data File Figure 19 Examples of how orphan files can be created In order, then, to effectively manage orphan files, three mechanisms were introduced into DX-NAS 2.0: source scans orphan file management file versioning 50 EMC DiskXtender for NAS Release 3.0 Theory of Operations

51 Basic DX-NAS Configurations About Source Scans A source scan will inspect every stub or offline file on a defined source and check to see if the information regarding the data portion of the file on secondary storage is still valid. If the data does not exist on secondary storage, the source scan generates an error. If the data exists, but it doesn t have the location information, the source scan will correct the problem. Consequently, running a source scan before an orphan file scan will reduce the number of orphans found. Source Scan on Celerra with File System Backend For Celerras migrating to file system backends, a source scan performs the following steps: 1. It looks at every file on the specified source directory 2. If a file is not a stub, it skips the file. 3. If a file is a stub, the source scan checks to see if the data belonging to the stub is on secondary storage. 4. If the data is not on secondary storage, the source scan generates an alert to notify the user of the error. DX-NAS does not delete stubs or offline files from primary storage; the user needs to make the determination on whether or not to delete these files. 5. If the data is on secondary storage, the source scan looks at the info file. If the info file is wrong that is, it is not pointing to the stub on primary storage correctly the source scan corrects the information in the info file. Source Scan on NetApp with File System Backend For NetApp filers migrating to file system backends, a source scan performs the following steps: 1. It looks at every file on the specified source directory 2. If a file is online, it skips the file. 3. If a file is offline, the source scan looks in the DX-NAS database to see where the data should be on secondary storage. 4. If the data is not on secondary storage, the source scan generates an alert to notify the user of the error. DX-NAS does not delete stubs or offline files from primary storage; the user needs to make the determination on whether or not to delete these files. About Orphan Files 51