FILE ARCHIVING FROM NETAPP TO EMC DATA DOMAIN WITH EMC FILE MANAGEMENT APPLIANCE

Transcription

1 White Paper FILE ARCHIVING FROM NETAPP TO EMC DATA DOMAIN WITH EMC FILE MANAGEMENT APPLIANCE Abstract This white paper is intended to guide administrators through the process of deploying the EMC File Management Appliance technology in environments containing NetApp and EMC Data Domain network storage. November 2010

2 Copyright 2010 EMC Corporation. All Rights Reserved. EMC believes the information in this publication is accurate of its publication date. The information is subject to change without notice. The information in this publication is provided as is. EMC Corporation makes no representations or warranties of any kind with respect to the information in this publication, and specifically disclaims implied warranties of merchantability or fitness for a particular purpose. Use, copying, and distribution of any EMC software described in this publication requires an applicable software license. For the most up-to-date listing of EMC product names, see EMC Corporation Trademarks on EMC.com. VMware is a registered trademark of VMware, Inc. All other trademarks used herein are the property of their respective owners. Part Number h File Archiving from NetApp to EMC Data Domain

3 Table of Contents Executive summary... 5 Audience... 5 Deploying File Management Appliance... 6 File Management Appliance architecture... 6 Network requirements... 7 Configuring FMA... 7 Environment configuration... 7 Preparing NetApp Filers for archiving... 7 Preparing Data Domain appliances for archiving... 9 High availability and load balancing for archiving and recall services HA for archiving services Load balancing for archiving services HA for recall services Load balancing for recall services Backup and disaster recovery Restoring orphan management Archiving and recall architecture FMA software architecture Archiver system architecture Data archived off NetApp systems This section describes the architecture and mechanism for archiving and recall functionality in environments utilizing FMA with NetApp Filers as an archiving source Volume language settings Authentication and authorization Overview of the archiving process Overview of the recall process Data archived to Data Domain systems Authentication and authorization Stub file formats Interoperability with quotas Implementing an archiving strategy Developing archiving policies Classifying datasets Performance requirements Creating File Matching Expressions Developing archiving policies Simulation tasks File Archiving from NetApp to EMC Data Domain

4 Creating and monitoring archiving tasks Monitoring the progress of a running task Scheduling archive tasks Managing archived data Managing Data Domain repositories Stub re-creation Stub scanner Orphan file management The File Management Appliance database Features that utilize the File Management Appliance database User interaction Database maintenance and backup Troubleshooting NetApp file server configuration Reading stub data Performing a packet capture Log files Additional troubleshooting utilities Conclusion File Archiving from NetApp to EMC Data Domain

5 Executive summary EMC File Management Appliance is used to implement a tiered storage strategy through file level archiving, thereby facilitating significant storage savings. This white paper is intended to guide administrators through the process of deploying the technology in environments containing NetApp and EMC Data Domain network storage. There are six sections in this paper: Deploying File Management Appliance Archiving and recall architecture Implementing an archiving strategy Managing archived data The File Management Appliance database Troubleshooting The information contained in this paper applies to EMC File Management Appliance version This paper may be updated periodically. It is always recommended to ensure that you have the latest version prior to applying any information contained herein. File Management Appliance software version is loaded onto a physical appliance called FMA or a virtual appliance called FMA/VE. The general term File Management Appliance or FMA will be used whenever the content in this document applies to both types of appliance. All content discussing NetApp and Data Domain is in reference to the specific software versions listed in the EMC File Management Appliance Interoperability Matrix. FMA will interoperate with any NetApp and Data Domain hardware that runs a supported software version. Audience This white paper is intended for use by EMC File Management Appliance administrators as well as storage administrators responsible for a production environment where FMA will be deployed. This paper does not discuss practices for using EMC File Management Appliance in environments containing EMC Centera or Celerra storage. This paper contains supplemental information to the standard documentation available for EMC File Management Appliance and assumes the reader is familiar with all information contained therein. If you have not reviewed those documents, please do so in addition to reading this paper in order to gain a complete understanding of the technology. 5 File Archiving from NetApp to EMC Data Domain

6 Deploying File Management Appliance This section describes how to deploy EMC File Management Appliance in production environments. File Management Appliance architecture EMC File Management Appliance can be delivered in the form of either a physical or virtual appliance. The capabilities and features available on these appliances are the same and either can be used to archive from NetApp to Data Domain to create a robust solution. The physical File Management Appliance can be deployed with a dedicated highavailability appliance for a fully redundant architecture. The EMC File Management Appliance (FMA) and the File Management Appliance/VE (FMA/VE) are suitable for small and large enterprise environments requiring the capability to archive up to 250 million files per appliance. Figure 1. Example of FMA deployment with NetApp primary NAS In this figure: 1. Clients send read or write operations for files that have been archived. These operations are intercepted by the FPolicy layer prior to being serviced from the 6 File Archiving from NetApp to EMC Data Domain

7 WAFL file system. When clients attempt to read/write a stub file on primary storage it triggers the Celerra to perform a recall of the file data. 2. The NetApp will send FPolicy callbacks to servers registered in the primary group in a round-robin fashion. If a server does not reply to the callback it is removed from its group. If there are no servers in the primary group, the callbacks are distributed in a round-robin fashion among the servers in the secondary group. 3. The FMA or FMA-HA appliance will then connect to the filer using CIFS in order to read the contents of the stub file. The stub file details where the file data is stored. The appliance will then connect to the NFS repository on Data Domain where the data was archived and will read the data back using the NFS protocol. The file data will then be provided back to the NetApp. 4. The filer will then respond back to the client operation as usual if the recall was successful or with an access denied message if the recall failed. Network requirements File Management Appliance relies on IP and Ethernet networking to communicate with NetApp and Data Domain storage systems. FMA must be able to establish IP connectivity to the systems with which it will interact. Specific sites may have performance requirements for archiving and recall speed. In such cases, it is recommended to ensure that FMA has appropriate bandwidth between itself and the NetApp and Data Domain systems that will be involved in archiving and that the network latency is acceptable. As a best practice, File Management Appliance and the file servers that will be involved in archiving should be deployed at the same site. When multiple sites are involved, network performance can have greater impact due to WAN conditions, causing slower archiving or recall speeds. Configuring FMA By default, the rfhsetup utility is launched when logging in to the FMA CLI. This utility will allow you to configure each of the network interfaces of FMA. For specific details on networking configuration, please consult the EMC File Management Appliance and File Management Appliance/VE Getting Started Guide. Environment configuration Data can only be archived and recalled if the configuration steps described in this section have been performed. These steps may need to be repeated when new data movers and/or file systems will be archived. Preparing NetApp Filers for archiving NetApp Filers that will be configured in FMA must have the following options set: options cifs.client.dup-detection off options cifs.nfs_root_ignore_acl on options fpolicy.enable on 7 File Archiving from NetApp to EMC Data Domain

8 options httpd.admin.enable on In order to archive any data from a NetApp Filer, File Management Appliance will require access to: SMB over NetBIOS (TCP port 139) ONTAPI (TCP port 80) In addition, to archive NFS data FMA will require: Portmap v2 RPC service (TCP port 111) Mount v3 RPC service NFS v3 RPC service NLM v4 RPC service Root and read/write export permissions for all NFS data that will be archived When configuring a NetApp Filer in FMA, you will need to supply: All IP addresses used by the filer Credentials for local administrator access through both CIFS and ONTAPI The NetBIOS name of the filer Configuring a Celerra as an archiving source Direct command line access (through telnet/ssh) is not used by FMA; however, ONTAPI access is used to send a variety of API calls and hence the requirement for a local administrators credentials. Note that if a user other than root is specified then the following option must be set: options httpd.admin.hostsequiv.enable on You will also need to ensure that the FMA hostname is resolvable to its IP addresses in the local /etc/hosts file of the NetApp Filer and that the hostname maps to a user given privileges to access the ONTAPI interface in the /etc/hosts.equiv file on the Filer. If the fpolicycallback service on FMA has never been initialized, you should do so using this command: fpsetup init_rffm This command should also be run on any FMA-HA appliances that will be used to service recall requests for high availability purposes. Note that this command does not need to be run on an appliance if the FCD service has already been initialized to provide HA recall service for an FMA. If you are unsure if the FCD is initialized you can check the status also using the command: fpsetup list_rffm The IP address of the FMA should be present in the output of this list (when run on FMA the command should display the address). If there are IP addresses listed in the output but they do not match the FMA IP address then the appliance may 8 File Archiving from NetApp to EMC Data Domain

9 already be providing recall services for an alternate FMA. Appliances can provide recall services for multiple FMA concurrently. In order to register an additional FMA you should not use the initialization command. Instead, use this command: fpsetup add_rffm You should never configure the same NetApp Filer in multiple FMA appliances as an archiving source. Doing so can result in data unavailability and data loss. Note: SnapLock-enabled volumes cannot be used as an archiving source. Preparing Data Domain appliances for archiving In order to archive NFS data to a Data Domain system, File Management Appliance will require access to the Mount v3 and NFS v3 RPC services running on the system. When Data Domain will be the target of archiving from Celerra, File Management Appliance will use the Mount v3 and NFS v3 RPC services to read/write to the storage filesystem. File Management Appliance does not support archiving to a Data Domain system using CIFS. Only NFS metadata (UNIX owner ID and group ID, mode bits) will be written with the file contents during archiving. As a result, only NFS metadata can be recovered with a stub file when leveraging the stub re-creation functionality of FMA. Preparing Data Domain appliances for archiving In order to archive NFS data to a Data Domain system, File Management Appliance will require access to the Mount v3 and NFS v3 RPC services running on the system. When Data Domain will be the target of archiving from NetApp, File Management Appliance will use the Mount v3 and NFS v3 RPC services to read/write to the storage filesystem. File Management Appliance does not support archiving to a Data Domain system using CIFS. Only NFS metadata (UNIX owner ID and group ID, mode bits) will be written with the file contents during archiving and only NFS exports can selected for archiving tasks. Configuring a Data Domain as an archiving destination The following tasks must be completed in order to use File Management Appliance to archive data to a Data Domain system. You will need to repeat these steps for each system that will serve as the target of an archiving task. 1. Create the repository path on Data Domain In order to archive to a Data domain system, a repository needs to be created and exported to one or more File Management Appliances. To create the repository path, mount to the desired repository filesystem from a Linux client and create the desired path using the mkdir command. For example: # mkdir /mnt/datadomain # mount datadomain.domain.prv:/backup /mnt/datadomain # mkdir /mnt/datadomain/repository 2. Create the repository export on Data Domain 9 File Archiving from NetApp to EMC Data Domain

10 There must be an export created on the Data Domain server for the File Management Appliance to mount and subsequently write data to during archiving. The export must also be configured to allow access to any File Management appliance that needs to recall archived data. Create a new export path by running the command from the Data Domain CLI: nfs add <path> <client IP list> [(option list)] As an example: nfs add /backup/repository , When specifying the client IP list, note that any File Management Appliance that will use this repository must be listed. This permits the specified client to access the repository path with read/write permissions. Similarly, any FMA-HA that can be used to connect to the repository to recall archived data must be listed or recalls will fail, resulting in data unavailability. 3. Create a repository for the Data Domain NFS export using the FMA GUI or rffm addnasrepository command High availability and load balancing for archiving and recall services HA for archiving services HA for archiving services is not a feature of EMC File Management Appliance. If a File Management Appliance fails then archiving services will cease until a replacement appliance is implemented. In the event of a failure, the disaster recovery procedure should be followed to implement the replacement appliance. Load balancing for archiving services A single NetApp Filer can be managed by only one EMC File Management Appliance. Multiple FMAs cannot be configured to archive data off of the same NetApp Filer. Load balancing of archiving services is not a feature of FMAs when archiving from NetApp to Data Domain. HA for recall services FMA delivers a simple solution for redundancy, ensuring that clients do not experience data unavailability based on failure of a File Management Appliance. To fulfill requirements for high availability, recall operations can be handled by a group of FMA, and FMA-HA. If FMA/VE is used to archive from NetApp Filers, VMware HA functionality is leveraged to provide high availability for recall services. NetApp file servers allow FPolicy clients (such as FMA and FMA-HA) to register for callbacks in response to user access to files with specific attributes. When using File Management a callback will be generated when a read or write operation occurs to a file with the CIFS offline bit set. For NetApp primary storage, multiple File Management Appliances can register in the primary and/or secondary FPolicy groups of the filer. In the event that a registered server becomes unresponsive, it is removed from its group. Recall requests will be sent by the filer in a round-robin fashion to the IP addresses registered in the primary 10 File Archiving from NetApp to EMC Data Domain

11 group. If there are no responsive IP addresses in the primary group, then the requests are load balanced across the servers in the secondary group. For NetApp primary storage, you will need to run the fpsetup utility on the FMA-HA that will process recall requests. This script allows you to link together multiple appliances that will be able to process recall requests sent from a common set of NetApp Filers. Afterwards, when configuring NetApp Filers you will have the option to select specific FMAs and FMA-HAs that will register in the primary/secondary groups. Note that File Management Appliance is always involved in recall when FMA is used to archive data from NetApp primary storage to Data Domain. NetApp Filers do not recall data directly from Data Domain. Note: A single FMA-HA can provide redundancy for multiple FMAs and a single FMA can have multiple FMA-HAs registered to provide redundancy. An FMA should not be used to provide redundancy for another FMA. Load balancing for recall services Load balancing is achieved by the NetApp Filer by sending recall requests in a roundrobin fashion to the IP addresses registered in the primary group in the FPolicy configuration. This will load balance recall requests between File Management Appliances registered with the NetApp Filer as primary recall agents. Backup and disaster recovery For EMC File Management Appliance, disaster recovery refers to the processes required to ensure that an equivalent functional environment can be created using an alternate appliance in the event that the existing appliance is irrecoverably damaged. The following sections describe major areas of functional equivalence that should be considered in the development of any disaster recovery plan. When recovering from a disaster, the tasks should be performed in the order they are discussed in this guide. The failure of the motherboard or of multiple hard drives of an FMA is an example of a disaster that would require the execution of this procedure. The corruption of the virtual disks associated with an FMA/VE could be another example. The following steps should be taken in the order listed, as needed. Note: If only the File Management Appliance is lost then Celerra blades will still be able to recall data from the Data Domain system and no data unavailability will occur. Step 1. Recover the File Management Appliance If a File Management Appliance is lost, a new appliance should be loaded with the same version of the software that was previously running. An upgrade to the latest software version can be applied later if needed. For FMA, a clean installation of the software should be performed using the fm_clean option when booting from CD. For FMA/VE, a matching version of the FMA/VE virtual appliance should be imported to ESX. The networking should then be configured for the appliance. 11 File Archiving from NetApp to EMC Data Domain

12 Note: New appliances do not need to use the same IP addresses of the old appliances. However, when the IP addresses change there will be additional steps needed to reconfigure the environment. Step 2. Recover FMA-HA If an FMA-HA needs to be rebuilt after a disaster there are no special steps required. A clean installation of the software should be performed. The appliances should be connected to the network and the fpsetup command should be run as needed to reinitialize the FPolicy callback daemon, which will allow the appliance to service recall requests. These are the same steps that would be taken when deploying a new FMA-HA. Note: When an FMA-HA is rebuilt after a disaster it may not have the same IP address as the original appliance. If an appliance IP changes the administrator should check the security used to protect all NFS repositories for archived data to ensure sufficient privileges are granted to perform recall. Step 3. Restore the software configuration The most convenient method to restore the configuration of a File Management Appliance after a disaster is from the most recent backup of the primary File Management Appliance. It is highly recommended to run backup tasks periodically using the automated Backup/Recovery feature of FMA and save the output file to a secure location on NAS or EMC Centera. In the event of a disaster, the latest version of the backup file can be restored to the appliance using the fmrestore command. The EMC File Management Appliance Getting Started Guide provides instructions on how to recover backup files from EMC Centera and NAS after a disaster. The use of these utilities will ensure that primary and secondary servers are defined in the new appliance configuration using identical values from the original appliance. In addition, the contents of the File Management Appliance database are backed up and restored using these commands. Therefore, archived file lists, policies, tasks, and so, will all be preserved and restored with this method. In the event that a backup of the File Management Appliance has not been performed or is not available, an administrator can manually configure the FMA by defining the primary and secondary servers. Particular care should be taken to maintain identical configurations wherever possible. At a minimum, the logical names used to define primary and secondary servers must remain the same. Afterwards, archiving policies and tasks should be defined and run. Each time an archiving task is run, a new stub scanner task is created if it does not already exist. The creation of stub scanning tasks is important to rebuild the orphan management capabilities of the appliance. Note: It is not necessary to define and run policies and tasks if a recent backup of the configuration was restored to the appliance. These actions need to be taken only if no backup was available or if the backup was very outdated. 12 File Archiving from NetApp to EMC Data Domain

13 Step 4. Adjust the environment after appliances have been recovered The file server definition in the File Management Appliance configuration may need to be updated if a NetApp system was lost and the replacement did not maintain identical settings such as IP addresses or local administrator users. It is critical to ensure that any elements defined in the File Management Appliance configuration reflect the state of the new environment. If the IP addresses of the appliance have changed then the export permissions for all NFS NAS repositories should be checked to ensure sufficient privileges have been granted to the new IP addresses. It may also be desirable to remove privileges granted to the old IP addresses. In addition, any local hosts files or DNS records referencing the NetApp Filers or File Management Appliance should be updated. Restoring orphan management The orphan file management feature allows File Management Appliance to clean up unused data on secondary storage. In order to remove a piece of data from secondary storage the appliance must have an entry in its database to indicate that it was the creator of that piece of data. Therefore a File Management Appliance will not delete anything from a secondary storage location unless it has a database entry that references it. Due to this requirement it is very important to preserve the integrity of the File Management Appliance database. It is highly recommended to perform periodic backups of the database using the automated Backup/Recovery functionality mentioned above. However, it should be noted that even when periodic backups are taken there may still be some orphan data on secondary storage that cannot be identified by the appliance. Consider the following sequence of events: The administrator runs a backup task on FMA. An archiving task is launched. A user deletes a stub file created by the archiving task that was just launched. The administrator needs to recover from a disaster by rebuilding a new FMA so the backup file is loaded using the fmrestore command. In this scenario, there will not be a record in the FMA database referencing the object on secondary storage nor will a stub exist on primary storage. FMA will not be able to delete the object on secondary storage because it cannot confirm that it created it. In order to minimize the impact of this scenario an administrator should take frequent backups of the FMA database, especially during periods of heavy archiving activity. If the user did not delete the stub, then the FMA stub scanner would have re-created the FMA database entry when it read the stub contents during its weekly scan. This would restore the ability to perform orphan file management. In a worst-case scenario where no backup of the FMA database was ever taken, the new FMA will be able to perform orphan management for all stubs found by the stub scanner but all 13 File Archiving from NetApp to EMC Data Domain

14 data that was already orphaned before the stub scanner runs cannot be cleaned up by the appliance. Archiving and recall architecture This section describes the architecture and mechanism for archiving and recall functionality in environments utilizing EMC File Management Appliance. FMA software architecture File Management Appliance runs a Linux-based operating system loaded with specialized software. Aside from the operating system, the core component of File Management Appliance is the File Management Daemon (FMD), a process that is part of the group of components referred to as the filemanagement service. The FMD accepts input through an XML-RPC interface from a handful of sources including the CLI, GUI, and other processes running on the system. The FMD does not monitor the CLI and GUI for events. The CLI and GUI components send requests to the XML-RPC interface of the FMD. Therefore, all components run independently. If the FMD is stopped then the CLI and GUI will still be accessible but will not be able to query or command the FMD. As a similar example, the FMD can run with the web GUI shutdown. The FPolicy Callback Daemon (FCD) used to handle NetApp recall requests is a key component of the File Management Appliance. Archiver system architecture The archiver is a component of the filemanagement service that is spawned and controlled by the FMD. The archiver itself is broken down into two major components: the filewalker and a pool of archiving threads. When an archiving task is run, the FMD creates a thread that spawns and manages an archiver process. The archiver will instruct the filewalker to collect and analyze metadata from files within the archiving source. Filewalking threads will use CIFS or NFS operations based upon the protocol specified for the archiving task. Filewalking threads compare the file metadata to the archiving policy and note files that should be archived by creating an entry in an internal queue in the appliance memory. Archiving threads monitor the queue and are responsible for carrying out an archiving process detailed in the following Overview of the archiving process section. Files and data streams smaller than 4 KB on NetApp volumes will not be archived to Data Domain systems by File Management Appliance. This rule prevents file data from being archived to secondary storage if the stub data that will be written to primary storage takes up the same number of file system blocks. Data archived off NetApp systems This section describes the architecture and mechanism for archiving and recall functionality in environments utilizing FMA with NetApp Filers as an archiving source. 14 File Archiving from NetApp to EMC Data Domain

15 NetApp FPolicy NetApp FPolicy provides a network-based interface to receive file system-level callback events in response to specific types of WAFL (the NetApp file system) operations to files meeting a custom set of conditions. Other servers register with FPolicy and inform the NetApp of a particular set of operations for which they will require callbacks. When one of those operations is triggered against a file system, a callback is sent to one of the servers registered with FPolicy and that server is allowed to perform any arbitrary set of actions before instructing the NetApp on how to handle the users operation. Once the server has completed its actions, it will inform the filer to process the operation normally or to return an access denied message to the user. Typical uses for the FPolicy interface are archiving and virus scanning. However, the interface allows a server to perform arbitrary actions in response to file access and thus the actual applications of this technology can be far reaching. As a common example, the FMA and FMA-HA will register with NetApp Filers to receive callback events for read and write operations to the set of files with the offline attribute toggled on. When a user generates an operation that falls into this category, the filer will notify one of the registered appliances and the filer will then wait to be instructed to process the operation normally or return an access denied message. The appliance will then recall the data to primary storage and upon success it will inform the filer that it can process the users operation normally. In some cases, it can take a long time for a server to perform the required actions in response to a callback. As an example, it would take a very long time to recall all of the data for a 500 GB file. In such a scenario, the clients that triggered the recall will have to wait until all of the data has been restored, before they will receive a response to their read/write operation. For CIFS clients or NFS clients using soft mounts, a timeout may occur causing an error to be returned in response to an API call generated by an application running on the client. Those clients can retry their operations, but until the data has been completely recalled they will continue to experience the same behavior. For NFS clients using hard mounts, the application thread that generated the read/write operation will appear to stop responding until the recall operation completes, allowing the application API call to return. In contrast, this is not typically an issue when attempting to access data archived from a Celerra system (due to architectural differences and additional intelligence in the Celerra software). Note that due to the NetApp architecture, CIFS access is required to archive any data, even if it is typically only accessed by NFS clients. Celerra FileMover and NetApp FPolicy differ in the granularity with which you can apply the technology. FPolicy is enabled or disabled for a filer/vfiler, whereas FileMover is controlled per file system. Both architectures allow files from a single share/export to be archived to multiple secondary storage locations. Unlike Celerra FileMover, the NetApp FPolicy architecture in ONTAP 7.2.x and earlier requires that all archived file data be transferred back to primary storage before any user read or write operation can be processed. This may not be necessary or efficient when clients just want to read a small portion of the archived data. As an example, a user who reads the first block of a 1 GB file archived from NetApp will have to wait for 15 File Archiving from NetApp to EMC Data Domain

16 the entire 1 GB to be recalled. This contrasts with Celerra FileMover, which has additional intelligence that makes such behavior unnecessary. NetApp Filers running ONTAP 7.3 and later can leverage the new FPolicy architecture, which does not require a full recall of archived data to process read operations. File Management Appliance will automatically register with filers running ONTAP 7.3 in a pass-through mode, allowing archived content to be recalled from Data Domain systems in response to read operations without writing back to the primary storage. Note that write operations will still trigger a full recall. Volume language settings The NetApp WAFL file system architecture allows characters to be included in filenames that may not be able to be displayed on Windows or UNIX/Linux clients. As an example, Linux clients mounting with NFS can create filenames that end with the period character (. ). In response to CIFS access from a Windows client, the filer would truncate the filename and append a tilde and a number ( ~1 ). In order to archive files off of a NetApp Filer, FMA requires that filenames be displayed the same between NFS and CIFS. To address this requirement, FMA requires UTF-8 encoding to be enabled with the source volume language on the NetApp Filer. If UTF- 8 is not enabled, FMA may not be able to properly recognize Unicode characters within a given file path during archiving. Subsequently the affected files will be skipped by the archiving process. If this requirement is not met then a volume will need to be converted to use compatible settings. The volume language setting can easily be changed from the NetApp CLI but can cause undesirable effects. Changing the language of a volume may prevent the filer from being able to display certain filenames using NFS, causing data unavailability. The NetApp CLI command WAFL_check f will identify files that will become inaccessible after changing the language settings. Appropriate actions will need to be taken manually by an administrator to address any files that will be problematic. Authentication and authorization FMA requires CIFS administrative access over NetApp Filers and vfilers that will be the source of archiving tasks. When configuring a filer/vfiler in FMA, you will need to supply CIFS credentials for a user in the local Administrators group. CIFS administrator access is required regardless of whether you will be archiving data accessed by CIFS or NFS clients. FMA will require CIFS access in order to check and manipulate the offline attribute of files (because NFS does not have operations to interact with offline attributes). FMA uses NTLMv2 for authentication when opening CIFS connections. It does not currently support Kerberos. No special configuration is needed when using FMA-HA to provide redundancy for recall operations of CIFS data. FMA will distribute the user credentials supplied during file server configuration to FMA-HA. To archive data accessed by NFS clients, you will need to configure the IP addresses of the FMA with root and read/write access to the exports that will be archived. The IP 16 File Archiving from NetApp to EMC Data Domain

17 addresses of FMA-HA that will provide redundancy for recall operations of NFS data should also be given root and read/write access to the same exports. Failure to provide authorization for those IP addresses will not prevent the successful completion of archiving operations. However, data unavailability will occur when an FMA-HA attempts to service a recall request, but is not able to restore the data. FMA also requires access to the ONTAPI interface of NetApp Filers and vfilers that will be the source of an archiving task. Specifically FMA requires local administrator access to the ONTAPI interface to perform archiving. If a local administrator role cannot be used, the corresponding role capabilities must be assigned to the role used by FMA. These include the login-* and api-* capabilities that are assigned to the administrator role by default. Some of these requirements are checked when an administrator first attempts to define a server, and an error will be returned to inform the administrator that nothing was added to the FMA configuration. Some problems cannot be detected until an archiving or data collection task is run or until a file needs to be recalled. Best Practice: Before archiving any file from a production file system, create, archive, and recall a test file using the same primary and secondary storage locations. Perform these tests every time you want to archive data off of a new source file system or to a new destination. This will ensure that configuration problems do not affect the availability of real production data. Overview of the archiving process Archiving threads perform the following actions when delayed stubbing is disabled: 1. A file matches an archiving policy and File Management Appliance obtains an exclusive lock on the file to be archived. 2. A temporary file is created in the same directory as the source file. 3. The source filename and NFS owner ID, GID, and mode bits are read. 4. A new file is created in the destination Data Domain repository. The name of the file and its location in the repository are generated by an algorithm; the original name and location are not used. The owner ID, GID, and mode bits are set identically to those of the source file. 5. The file data is read from the source by File Management Appliance and written to the destination. 6. The last modified timestamp of the destination file is set to match that of the original file on the source. 7. The stub data is written into the temporary file. 8. The CIFS offline bit is set for the source file. 9. The contents of the temporary file are used to overwrite the source file. 10. An entry is inserted into the FMA database to record the success of the archiving operation. 17 File Archiving from NetApp to EMC Data Domain

18 11. The temporary file is removed. With delayed stubbing enabled, an archiving thread will query the FMA database to determine if a file matching the archiving policy also matches the stubbing policy. The stubbing policy classifies files into three categories based upon the last modified timestamp, file size, filename, and location within the source share/export. The FMA database is searched for an entry that matches these attributes, indicating the files have already been copied to the secondary storage location. If no matching entry is present in the FMA database, this version of this file has never been copied to the secondary storage location and the following sequence of events will occur: 1. An exclusive lock is obtained on the source file to be archived. 2. The source filename and owner ID/GID/mode bits are read. 3. The source file data is read. 4. The timestamps of the source file are read. 5. The file data and metadata are archived to secondary storage. 6. An entry is inserted into the FMA database to record the date/time when the file data was originally copied to the secondary storage location. If a matching entry exists in the FMA database, it will be associated with a timestamp that indicates the date on which the file was originally copied to the secondary storage location. If the time difference between the system clock of the FMA appliance and the timestamp exceeds the delayed stubbing period, the following sequence of events will occur to stub the file: 1. An exclusive lock is obtained on the source file to be stubbed. 2. A temporary file is created in the same directory as the source file. 3. The stub data is written into the temporary file. 4. The CIFS offline bit is set for the source file. 5. The contents of the temporary file are used to overwrite the source file. 6. An entry is inserted into the FMA database to record the success of the stubbing operation. 7. The temporary file is removed. If a matching entry exists in the FMA database and the time difference does not exceed the delayed stubbing period, then no further action will be taken. After the archiving task completes, a background archiving thread will query the FMA database on a daily basis to determine files where the current timestamp based on the system clock of the FMA exceeds the stubbing time recorded in the FMA database. When a file is found according to these requirements, steps 1-7 above will be executed to stub the file. 18 File Archiving from NetApp to EMC Data Domain

19 Overview of the recall process Recalling archived data to a NetApp Filer or vfiler is dependent upon the existence of the rfpolicy FPolicy profile and the presence of a registered and active FMA or FMA-HA. The rfpolicy FPolicy profile generates callback events to registered appliances when read and write operations occur to files with the offline bit set. Callback events are triggered from the WAFL file system level and can thus be triggered through the use of any network access protocol such as NFSv3, NFSv4, CIFS, and so on. The recall process begins when an FMA or FMA-HA receives a callback from a registered NetApp Filer. The callback informs the appliance of the IP address of the client and will provide the UNC path to the file they are accessing. The appliance will respond back to the callback and allow the client to access the stub file if the client IP address is present in the list of excluded clients. If the client is not an excluded client, then the FMA will attempt to recall the file indicated in the callback. This process begins by parsing the file data of the file on primary storage to determine if it is a valid FMA stub file format. If the file can be successfully parsed, a backup copy of the stub will be made, and then data will be recalled from the secondary storage using NFS. After all of the file data has been restored, the appliance will use CIFS to unset the offline attribute. Depending on the use of the pass-through recall mode in ONTAP 7.3 and later, the filer may not be required to restore the recalled content to primary storage to respond to client read operations. Upon the success of the recall process, the appliance will respond to the callback instructing the filer to respond to the clients operation as usual. If the file could not be recalled successfully then we will instruct the filer to deny access to the user. If a file is being recalled and additional callbacks are received for the same file, the appliance will return the same response to all callbacks to either allow or block access. In the event that the file data on primary storage cannot be parsed as an FMA stub file, the appliance will check for the presence of a temporary stub file within the same directory (left there due to a problem during the original archive operation). If a temporary stub file is found and can be parsed, data will be recalled as normal. If there is no temporary file or it cannot be parsed, the offline bit will be unset on the original file and the client will be allowed to access. Data archived to Data Domain systems This section describes the architecture and mechanism for archiving and recall functionality in environments utilizing FMA with Data Domain systems as an archiving destination. Authentication and authorization In order to archive data to an NFS repository on a Data Domain system, you will need to configure the IP addresses of the FMA with root and read/write access to the export hosting the repository. The steps to accomplish this are described in the Preparing Data Domain appliances for archiving section on page 9. Failure to provide authorization for FMA IP addresses will prevent archiving from completing successfully. 19 File Archiving from NetApp to EMC Data Domain

20 In addition, the IP address of all FMA-HAs must be given root and read/write access to the repository export. Failure to provide authorization for FMA-HA addresses will not prevent the successful completion of archiving operations. However, data unavailability will occur when an appliance fails to service a recall request because it is not authorized to read the data from secondary storage. Stub file formats The NetApp NFS stub file format is as shown below: <?xml version="1.0" encoding="utf-8"?> <RFStubInfo Version="1" LastModifiedTime=" " LastModifiedTimeUTC="Thursday, :11: " ArchiveTime=" " ArchiveTimeUTC="Wednesday, :00: " FileSize="429056" RetentionTimeInSeconds="0" SecondaryStorageProtocol="NFS" SecondaryStorageClass="NAS" SecondaryDeviceType="WINDOWS" URL="nfs://datadomain1.interop.prv/backup/Ali/.rffm_nas/ /0/0/0 /1511" SecondaryServerIP=" " SecondaryServerDnsName="datadomain1.interop.prv" SecondaryServerNetBiosName="datadomain1" SecurityHash="d3ee9027d775f6d3d e4afd4d1" > </RFStubInfo> Interoperability with quotas NetApp servers support the enforcement of quotas that are used to limit the amount of data or number of files a user or group can create. Data archived from a quota directory/quota tree will no longer count toward a user s quota. As an example, a user with a 100 MB quota who creates a 100 MB file will not be able to create any additional data. However, after the file is archived the user will be able to write an additional 100 MB of data. Note that only byte quotas are affected by archiving and file limits remain the same. Implementing an archiving strategy The quality of an archiving strategy is often judged on how it affects users of primary storage tiers since file data reaching the end of its lifecycle is likely to be stored on slower disks. Key factors include the number/percent of files a user must work with that have been archived and the impact to the average service response time for operations the user performs to those files. While the performance aspect is based mostly upon environmental factors, the amount of archived files a user has to deal 20 File Archiving from NetApp to EMC Data Domain

21 with is directly related to the quality of the archiving policies created by storage administrators. Good archiving policies qualify the probability that a file will be used in the future (whereas bad archiving policies result in excessive amounts of data being recalled from secondary storage). Good archiving or tiering strategies ensure that the performance impact associated with accessing the archived data results in the minimal impact to users of primary storage. File data should be kept on storage with appropriate performance levels based upon the stage in the information lifecycle that it has reached. As data transitions through the stages of the lifecycle, the performance requirements decrease. FMA allows administrators to model and then implement archiving strategies to create tiered storage systems and realize lower TCO. It can be used to: Classify files sprawled across a NAS environment into distinct datasets Manage the locations where individual file data is stored Shrink large datasets and file systems by migrating file data to alternate locations (also decreasing backup window requirements) As a result of the fast and transparent archiving and recall architecture, these benefits can be attained without causing significant impact to the performance experienced by end users. Developing archiving policies Classifying datasets The key to designing a good archiving policy is being able to determine the probability that a file will need to be read or written over time. To be able to make this determination, we must define the boundaries of a dataset (the primary storage location) and classify the files it contains. This may be an iterative process wherein while classifying the files in a dataset, it becomes apparent there is more than one distinct dataset, forcing the boundaries to be reset and classification to begin again. Classification of files in a dataset requires knowledge from two sources and goes hand in hand with the creation of archiving policies. First, the metadata maintained on the primary storage tier provides valuable information about the last time files were modified and accessed as well as file sizes, names, and locations in a dataset. Second, the end users of the storage can provide insight into how the dataset is used in general. This information will provide guidance to an administrator when designing archiving policies. As an example, consider the NAS environment of a typical hospital. The first step to designing an archiving policy is to examine the layout of the existing primary storage tier, identify the contacts that have requested storage, and determine how the end users are accessing their storage. During this step, you may find that various departments throughout the hospital are using NAS to store digital images of X-rays, MRIs, and CT scans. Other departments are using it to store copies of patient bills and payments, or medical records. Still others are using NAS to hold general home directory data. 21 File Archiving from NetApp to EMC Data Domain

22 In further discussion with the contacts from the hospital it is discovered that the X- ray, MRI, and CT images are used for very different purposes and are typically needed for different lengths of time. Due to the ways in which the hospital applies X-ray technology, the images are typically required for the treatment of patients for about six months. However, MRI and CT images are only required for treatment over a period of a few days. Due to regulatory requirements, the hospital must keep bills and payments on file for five years. However, once a bill has been paid, the finance department reviews the copy at the end of each month and the copy is not typically accessed again. The stored medical records are not needed unless a patient visits the hospital again and typically, only the last two years of records are required. However, because the immediate availability of patient records is critical, the hospital contact requests that only medical records older than two years be archived. Given these conditions, a good archiving policy will need to account for one or more of the following: X-ray images must not be archived until they are six months old. MRI and CT images must not be archived until they are two weeks old. Copies of bills and payments must not be archived until they are 45 days old. Medical records must not be archived until they are two years old. Data in user home directories should not be archived if it has been accessed recently. Note that the first four conditions are all based upon the length of time since a file was last modified and that the last condition is based upon when a file was last accessed. This is because the images and records affected by the first four conditions are static files that will not be changed over time, but the home directories contain dynamic files. Therefore, in designing an archiving policy for dynamic files, we assume that the more recently a file was last accessed, the more likely it is to be accessed again soon. When designing an archiving policy for static files, we assume that as data ages it becomes less likely to be accessed again soon. Metadata stored on NAS will typically be required in order to translate the conditions listed above into an archiving policy. As an example, assume that the X-ray, MRI, and CT images are stored on an NFS export of file system fs1 inside a directory named repository. Filenames for X-ray images begin with the letter X, MRIs begin with the letter M, and CT images begin with the letter C. Copies of bills and payments are stored in the scans directory in the same export on fs1. Medical records are stored in the mr directory and home directories are stored in the hd directory on file system fs2, which is both shared and exported. The medical records are written and accessed through NFS and the home directories also utilize NFS. In this scenario, an NFS archiving task would be developed for the repository directory on the fs1 file system utilizing an archiving policy with the following rules: 1. Archive a file if the name begins with X and the last modified timestamp is > six months old. 22 File Archiving from NetApp to EMC Data Domain

23 2. Archive a file if the name begins with M or C and the last modified timestamp is > two weeks old. 3. Or else do not archive the file. A second NFS task would be developed for the scans directory on the fs1 file system utilizing an archiving policy with the following rules: 1. Archive a file if the last modified timestamp is > 45 days old. 2. Or else do not archive the file. A third NFS task would be developed for the mr directory on the fs2 file system utilizing an archiving policy with the following rules: 1. Archive a file if the last modified timestamp is > two years old. 2. Or else do not archive the file. In order to prepare a task for the user home directories, we must qualify what it means for a file to be accessed recently. We define recent as a particular length of time since a file was last accessed and we then classify files in a dataset based on whether the last accessed timestamp falls within that length of time. As we increase or decrease the length of time, we will select a larger or smaller percentage of the files in the dataset. With this premise in mind, we would develop an additional NFS task for the hd directory on the fs2 file system using an archiving policy with the following rules: 1. Archive a file if the last accessed timestamp is > some period of time. 2. Or else do not archive the file. Now we are faced with the challenge of quantifying some period of time. A common method for doing this involves choosing a percentage of the total number of files or total amount of file data within a dataset that should be archived, and then adjusting the length of time used in the policy to reach the exact percentage. This method utilizes data collection tasks and previews and is discussed elsewhere in this guide. This method does not necessarily result in optimal performance for users or maximum reduction in TCO. However, for very dynamic datasets (any file data can change or be accessed at any time with no discernable trends) this may be the preferred method for the sake of simplicity. A more advanced method involves the analysis of the last accessed times of all files within a dataset in order to gain a more accurate understanding of how the age of a last accessed timestamp correlates to the likelihood that a file will be used again in the future. For typical datasets, there is an exponential relationship between the age of a last accessed timestamp and the probability of future use of a file. An example of a dataset with this type of relationship would be copies of ISO images generated as part of software development practices. There may be many users accessing the latest build of the software (for instance, to perform QA testing). There may also be many users accessing the various general availability builds that were provided to customers. The general availability builds are all copied to a special directory path in the file system. It is extremely unlikely that anyone will access software images older than 30 days for builds that were not provided to customers as 23 File Archiving from NetApp to EMC Data Domain

24 no testing or development would be occurring off of that code base/image aside from the occasional bug fix. Given such a dataset, we can be very precise and design an archiving policy that balances both the age of a file since it was created and how recently it was accessed. We may design an archiving policy with this logic: 1. Do not archive any files in the specific directory containing the GA releases 2. Archive all files not accessed within the last 45 days 3. Archive all files created more than 45 days ago that have not been accessed in the last seven days This policy will prevent GA software images from being archived while ensuring that data that hasn t been accessed in a long time is moved to secondary storage. In addition, the third rule ensures that if a software image is recalled it will not stick around on primary storage for another 45 days. After one week of not being accessed the data will be moved back to secondary storage. Performance requirements NAS clients may have performance requirements for the speed at which they can access archived data. As an example, a user trying to stream back a video file encoded at 1024 KB/s from NAS is unlikely to accept a data transfer rate of any less. However, service levels can decrease significantly if archiving is not applied correctly. The key to meeting performance requirements is to ensure that data is stored on an appropriate tier of storage based upon its progress through the information lifecycle. Archiving policies should be developed to select files at particular stages of the information lifecycle and place them on appropriate storage tiers. You should determine the performance requirements for data that will be selected as you develop the archiving policy. Each of the following points may affect recall performance. Will the secondary disk storage provide sufficient performance when clients need to access archived data? Will the primary and secondary Celerra blades have sufficient system resources (CPU, RAM) to provide the required performance levels for recall operations? Is there sufficient network bandwidth between the primary and secondary Celerra blades for recall operations? Will the network latency cause a significant effect? What type of recall policy will be used for NetApp FPolicy (ONTAP 7.3 and later)? File Management Appliance will automatically register in pass-through recall mode but can be configured to operate in full recall mode if desired. Creating File Matching Expressions A File Matching Expression (FME) is one or more conditions used by File Management Appliance when data is being archived. A statement in an FME consists of an attribute, an operator, and a value. Supported attributes by the FMA policy engine are the CIFS/NFS last accessed timestamp and last modified timestamp, as well as the size of file data and the format of a filename. 24 File Archiving from NetApp to EMC Data Domain

25 Operators are applied against the specified file attribute and compared against a value in order to return a true or false value. As an example, operators that can be applied to file size are greater than (>), less than (<), greater than or equal to (>=), and less than or equal to (<=). The value supplied when evaluating the file size attribute is a number of bytes. This allows rules to be crafted that select files or varying ranges of sizes. These same operators can be applied to the last accessed and last modified timestamps but the value will be applied as a period of time, allowing rules to be crafted that select files that have not been accessed or modified during a period of time. When using the filename attribute, operators allow exact filenames to be specified ( equals ) or compared to a regular expression ( matches regex ). An FME can consist of one or more statements. When multiple statements are part of a single FME, the statements will be logically combined using the AND operator. Therefore, the evaluation of an FME will be true if and only if all statements within the FME evaluate to be true. Developing archiving policies An archiving policy consists of a collection of ordered rules and their respective archiving destination(s). When creating archiving tasks, an archiving policy is applied against a source dataset to define which files are to be archived and to what location. A building block of an archiving policy, a rule consists of an FME and an indication of whether or not a file that matches the FME should be archived and destination for the archived content. Administrators will define a number of rules and then order them to build an archiving policy. When a policy is applied to a source dataset by running an archiving task, files will be compared to the rules in order. As soon as a file matches the FME of a rule, it will not be checked against any further rules and the indicated action will be taken to either archive the file or skip it. When a file does not match any FME in the rule base, the default action will be taken and the file will not be archived. As an example, consider the following rule base, which illustrates a common method for archiving unstructured data such as user home directories. 1. Archive files that are larger than 5 MB and the last accessed timestamp is > two months old 2. Archive files that are larger than 1 MB and the last accessed timestamp is > three months old. 3. Archive files that are larger than 512 KB and the last accessed timestamp is > four months old. 4. Archive files that are larger than 32 KB and the last accessed timestamp is > six months old. 5. (Implied) Do not archive all files. This policy weighs the bytes of file data that will be saved on primary storage against the length of time since the data was last accessed. When large amounts of space 25 File Archiving from NetApp to EMC Data Domain

26 can be saved (files larger than 5 MB in this example) files will be archived earlier. The benefit to this type of policy is that space can be saved on primary storage earlier in information lifecycle. The drawback is the increased likelihood that the large files will need to be recalled as two-month-old data is more likely to be needed than sixmonth-old data. This type of policy is typically beneficial when applied to unstructured datasets where little can be understood about the nature of the files or their users. This example uses specific values for file sizes and timestamps. Note: When developing an archiving policy it is highly recommended to make use of the simulation feature of File Management to determine the most effective values to use in your own policies. In addition to rules and a destination, an archiving policy allows you to specify a retention period (in days, weeks, months, or years) that will be set when archiving to a NAS repository. Note: The retention feature is not supported when archiving to a Data Domain repository and retention will not be enforced on data archived by FMA. Simulation tasks File Management Appliance allows administrators to simulate the effects of applying an archiving policy against a dataset to determine how many files, how much file data, and the specific files that would have been archived. This allows administrators to judge the effectiveness of an archiving policy and decide whether it will provide the desired results. It is important that the statistics and information provided by this feature be interpreted carefully. A typical goal when implementing an archiving policy is to reduce file data on primary storage by a particular percentage and it would be very simple to design a policy that meets this requirement. However, the quality of an archiving policy is in its ability to meet this requirement by archiving only the least utilized file data. Depending on the design of the archiving policy and the type of data being archived, the overall statistics for number of files and bytes of file data to be archived may not be most relevant for determining effectiveness. For well-classified datasets, administrators will typically want to analyze the list of individual files that would have been archived. This is typically an iterative process in which the archiving policy is modified and a new simulation is run until the desired results are achieved. When using File Management Appliance, previewing the effects of applying an archiving policy to a dataset is referred to as a simulation. A simulation is architecturally very similar to an archiving task. Simulations identify whether a file should be archived in an identical manner to an archiving task; however, a simulation will not archive any data. Policy simulation tasks query NAS servers using CIFS/NFS in the same manner as an archiving task so they provide a very close approximation of the true efficacy of the policy. It is important for administrators to note that policies are being applied against a one-time pass of the file metadata and when a policy is 26 File Archiving from NetApp to EMC Data Domain

27 actually applied it may not produce identical results to the simulation. An example would be that a policy archives files not accessed for three months and after running the simulation a user accessed a very old file that will prevent the file from matching the particular rule within the policy if it is compared again. In order to run a simulation of an archiving task, the archiving task is scheduled in the same manner but the Run Simulation scheduling option is selected rather than a recurring or future scheduling option. A simulation can also be started from a previously scheduled archiving task using the File Management Appliance GUI. This will initiate the simulation task to begin walking the source dataset comparing file metadata to the rules contained in the archiving policy to determine files to be archived. As mentioned above, no file data will be affected by the simulation. However, the simulation task will report similar results to an archiving task in terms of files and bytes archived and stubbed as well as a list of files that would be archived by the task. Creating and monitoring archiving tasks An archiving task in File Management Appliance applies a policy to a source dataset and results in selected files being archived to a secondary storage tier. Creating an archiving task is simple and requires the specification of a source dataset, an archiving policy, and a schedule for running the task. Tasks can be run as soon as they are submitted or run on a periodic schedule. After a task is submitted it can be run at any time without the need to wait for a scheduled time to pass. When archiving from a NetApp system to a Data Domain system, the source dataset must be specified as an NFS export. This instructs FMA to use the NFS protocol to read from the source dataset and write to the Data Domain repository. The dataset can be accessed simultaneously by CIFS, NFS, and other client protocols but FMA will use only NFS to read file data and metadata during archiving. As mentioned in the Overview of the archiving process section, FMA will archive only NFS metadata (UID, GID, mode bits) to Data Domain but any CIFS metadata such as ACLs or alternate datastreams will be unaffected by the archiving process. Monitoring the progress of a running task The status of a running task can be obtained through the GUI or using the rffm gettaskprogress command. In most cases it is not necessary to perform any monitoring of tasks as they are running. Administrators will, however, want to review the final report for archiving tasks that have run, detailing the number of files/bytes that were archived as well as any archiving operations that did not complete successfully. Failed archiving operations will not result in data being unavailable but administrators will want to review log files to determine the root cause of the failure. A log file is created for every execution of every archiving task and the logs are stored in the /var/log/rainfinity/filemanagement/fws/support directory. Administrators can halt running tasks using the rffm stoptask command or using the GUI. An example of when an administrator may choose to stop a task would be when all archiving operations are failing against a large dataset. To save time in such 27 File Archiving from NetApp to EMC Data Domain

28 a scenario an administrator may not want to wait for the task to complete before beginning the troubleshooting process. When running multiple concurrent tasks, an administrator will want to monitor the system performance and resources closely. In particular, the CPU utilization should be monitored using the top command and free RAM should be monitored using the free command. In addition, you should avoid overloading the File Management Appliance database with excessive numbers of connections that can result from running many tasks concurrently. As a general guideline and best practice, you should avoid running more than five tasks concurrently of any type (data collection, stub scanner, policy preview, or archiving). The appliance will automatically limit the number of concurrently running archiving tasks to three, and additional tasks are queued until a running task completes. However, alternate types of tasks are not limited and administrators should avoid overloading the appliance. Scheduling archive tasks When scheduling a recurring task, it is important to set an appropriate frequency. Running an archiving policy too frequently (such as every day) may not be useful as it can require a large amount of scanning and processing time but may result in very few new files being archived. Tasks should be scheduled to run such that the amount of data archived is an acceptable tradeoff for the amount of time and system resources that were required to execute the task (in other words, not too frequently). Managing archived data The management of archived data refers to a handful of activities. Stub re-creation manages stub data on primary storage, orphan file management controls file data stored on secondary storage, and the stub scanner manages the relationship between the two tiers. In addition, data archived to NAS repositories will typically need to be backed up on a regular basis. Managing Data Domain repositories A common reason for implementing tiered storage through file archiving is to shrink the size of a large dataset and in turn to lower required backup windows. The primary storage shrinks and can be backed up more quickly. The secondary storage now requires backup as well. Data in a NAS repository is only modified by certain features of a File Management Appliance. After running an archiving task, new data will be created in the repository and a backup will be required. During orphan file management, data may be deleted from the repository, again requiring backup. In this latter case a backup is less critical as failure to complete the backup would not result in data loss; it would result in excess data being left in the backup copy. Outside of these activities, the data in a NAS repository will not be changed and a backup would be unnecessary. When scheduling archiving tasks it may be desirable to have the expected time of completion coincide with the start of a backup process. A second option is to utilize 28 File Archiving from NetApp to EMC Data Domain

29 delayed stubbing, which will provide a window of time where file data will exist on both primary and secondary storage. This will allow data to be backed up from secondary storage before being removed from primary. Stub re-creation Stub re-creation allows administrators to quickly restore access to archived data through the primary storage tier. When a file is archived, File Management Appliance replaces the file data on primary storage with stub data. It also saves all the information required to re-create the stub file to the File Management Appliance database. File Management Appliance can be used to re-create a stub file at the same location as the original that utilizes the same security settings (UNIX mode bits) and last modified timestamp as the original. This action can only be performed by the File Management Appliance that originally archived the file, or from a cloned appliance created using the Automated Backup/Recovery and fmrestore utilities after a disaster. The process for re-creating a stub file is as follows: 1. An administrator selects a stub to re-create using the File Management Appliance CLI or GUI. 2. The original location of the file on primary storage is checked to determine if there will be a naming conflict, in which case additional strings will be attached to the re-created stub filename. If the path to the original file no longer exists then stub re-creation will fail (for instance, the parent directory was deleted). 3. The new stub file is created and locked. 4. The original security settings for the file (at the time of archiving) are read back from secondary storage and applied to the new file. This includes the original UNIX owner ID and group ID as well as mode bits. 5. The stub data is written into the new file and the file is marked as a stub using using CIFS and FPolicy calls. 6. The last modified timestamp is set for the new file identically to the timestamp from when it was originally archived. Stub scanner Stub scanner tasks detect stub files on all the exports that have been the targets of previously run archiving tasks. The purpose of these tasks is to identify whether file data stored on secondary storage will be needed to service a recall request from a primary storage server. By scanning all exports in an environment that may contain stub files, all of the stub files that can trigger file recalls are identified and recorded. If we have archived a file to secondary storage and fail to find any stub files that reference that data, then that file data on secondary storage is considered to be an orphan file and after 30 days in this state, File Management Appliance will allow administrators to delete the data using the orphan file management feature. The stub scanner task uses filewalking threads to check file metadata and identify stub files. Stub data is then read from any identified stub file. The stub data indicates 29 File Archiving from NetApp to EMC Data Domain

30 the location of associated file data on secondary storage. The File Management Appliance database is then queried for a record that matches the location the stub file was found in, the secondary storage location listed in the stub file, and the last modified time of the stub used to determine the version of the file. If a matching record exists then it will be updated to indicate that at the current date and time the stub file was still present on primary storage. It is also possible that a matching record will not exist. As an example, if a stub file is migrated between Celerra servers, the record in the FMA database will still reference the old primary location. When the stub scanner finds the stub on the new Celerra the FMA record will be updated to indicate the current location. Stub scanning tasks are created and scheduled automatically and do not generally require any configuration by an administrator. Whenever an archiving task is run, the database is checked to ensure a stub scanner task has been scheduled for the source export. If no task exists a new one will be added. By default, stub scanning tasks will start to run at 6 p.m. every week on Friday, though this time can be manually changed by an administrator In some cases, a share or export may contain File Management Appliance stub files even though FMA has not been instructed to archive data from the location, for instance, if a migration tool is used to migrate a NetApp dataset containing stubs. In such scenarios, stub scanning tasks can be manually added to a File Management Appliance by an administrator. If an administrator takes actions to copy stub files to new exports then they should also ensure that a File Management Appliance has a corresponding stub scanning task. Extreme care should be taken to ensure that only one File Management Appliance is used to archive data off of any single NetApp Filer (or manage orphans/stubs). It is not compatible to have multiple appliances archive data off of a single NetApp. This will cause each appliance to have inconsistent database entries when the stub scanner runs. Each appliance will add records to its database for the stub files created by each other appliance. Administrators should also never manually add a stub scanning task to an appliance if an alternate appliance is already being used to archive or stub scan the NetApp volumes. Orphan file management Orphan file management allows administrators to manage the data stored on secondary storage from the File Management Appliance GUI and CLI. Through the use of the stub scanner, file data on secondary storage that is no longer needed is identified and the orphan file management feature allows that data to be removed. This frees up space on secondary storage to allow more data to be archived to it. To consider file data on secondary storage to be orphaned there must not be any stubs on primary storage that reference the data. To verify this condition is true, File Management Appliance scans all primary storage locations from which they have archived data along with any user-defined locations for stub files on a weekly basis. Each time a stub file is found the corresponding entry in the File Management Appliance database is updated to indicate that the data on secondary storage is still needed. If the stub scanner runs for multiple weeks in a row and fails to find stubs 30 File Archiving from NetApp to EMC Data Domain

31 referencing a particular piece of data archived to secondary storage then the database record will contain a timestamp from far in the past. The orphan management features of File Management Appliance will report data on secondary storage as being orphaned if there were no stubs found for at least 30 days. Some companies have compliance requirements where data must be backed up periodically and the backups must be kept for a period of time. In such a scenario it is possible that primary storage no longer has a stub referencing a particular piece of file data but a stub is still present in the backup. If, for instance, backups must be kept for 90 days, a backup was taken of a file system 50 days ago and the user deleted a stub file 45 days ago, then the appliance may report the file as an orphan because the 30-day minimum period has been met. Recall will fail if the file data is deleted from secondary storage and later on the stub is restored to primary storage. Therefore it is important to use an appropriate minimum period when considering files to be orphans. The File Management Appliance CLI and GUI allow users to specify a minimum age for orphan files that will be cleaned up. The value cannot be set to less than 30 days but can be set for longer in order to meet business continuity or compliance requirements. The File Management Appliance database The File Management Appliance database stores information about the tiered storage implementation and archiving activities and it is used extensively within the File Management Appliance software. All major functions of the tiered storage infrastructure other than file recall utilize the File Management Appliance database either by creating, modifying, or querying entries. Features that utilize the File Management Appliance database Most of the features accessed through the GUI utilize the database. This includes: Creating or listing schedules using the Schedule page Generating reports or managing archived files using the Archived Files page Creating or viewing policies using the Policies page Creating/viewing file servers or NAS repositories from the Configuration page When using the command line, nearly all options of the rffm command make extensive use of the database. In addition, the scheduler component of File Management Appliance triggers tasks to run at specific times. When a task runs (regardless of the type) it reads or modifies the contents of the database. User interaction Users interact with the database indirectly by using the GUI and CLI. At no point will the underlying database implementation be exposed to the appliance administrator. Interacting directly with the database can be extremely risky or disruptive. If an administrator needs to access the database then EMC Support should be contacted to provide assistance. 31 File Archiving from NetApp to EMC Data Domain

32 Database maintenance and backup The database does not require any specific maintenance to be performed by administrators. However, depending on the amount of activity that heavily uses the database such as archiving and orphan management the File Management Appliance database performance can degrade. This requires a process known as vacuuming, which cleans up empty or leftover rows from database activity in addition to other steps to optimize the performance of database processes. This process should be run with the assistance of EMC Support personnel to ensure it is run correctly if it is required. If degraded performance of database-intensive tasks is observed, please contact EMC Support to investigate further. Note that when you are upgrading File Management Appliance software, the database tables may be re-indexed or rebuilt. This process can take a few minutes, a few hours, or up to a few days depending upon the number of entries in the database. The database is not an integral part of recalling archived data and thus the loss of the database will not inherently cause data unavailability. However, the database stores significant amounts of metadata describing the states of files on primary and secondary storage as well as their relationships. The loss of this information will affect most features of the appliance as previously described. It is therefore desirable to protect the database through periodic backups. There is no specific recommendation for backup strategies, but specific consideration should be given to the types of events that modify the contents of the database. Specifically, archiving, data collection, and stub scanning tasks have the potential to generate large amounts of database changes. It would typically be advised to schedule backups after these types of tasks have completed. The entire configuration of File Management Appliance, including the database, is backed up using the Automated Backup/Recovery feature of FMA. This generates an output file (.tgz) that is automatically copied to a safe location on NAS or EMC Centera for use in the event that a disaster occurs. The output file from the automated backup can be restored on an appliance using the fmrestore utility from the CLI. This will erase any existing configuration on the appliance and replace it with the configuration and database contents stored in the backup file. The EMC File Management Appliance and File Management Appliance/VE Getting Started Guide provides instructions on how to recover backup files from Centera and NAS after a disaster. Troubleshooting NetApp file server configuration If an error message is returned when attempting to define a NetApp Filer, the following items should be verified: The logical name of the NetApp matches the CIFS NetBIOS name of the filer/vfiler being configured. 32 File Archiving from NetApp to EMC Data Domain

33 The ONTAP version is supported as per the EMC File Management Appliance and File Management Appliance/VE Interoperability Matrix. The CIFS user credentials have been specified correctly and the user is authorized as a local administrator over the source server. There are other problems for which an error message will not be returned. If you are having problems with a NetApp Filer you should also check: If the file server is a vfiler, ensure that the host IP address is supplied under the file server definition in FMA. Ensure all the IP addresses of all the network interfaces assigned to the filer/vfiler are present in the file server definition in FMA. If the filer/vfiler will be used as a source server for archiving, you will also want to verify that: The credentials supplied under NetApp Local Admin are correct and the indicated user has root privileges to the ONTAPI interface. The filer/vfiler is not already configured as a source server in any other FMA in the environment. Reading stub data The contents of a stub file on a NetApp Filer can be read using any protocol such as CIFS or NFS by any client in the Excluded Client configuration of FMA. When an FMA or FMA-HA receives a callback generated by a client read operation from an IP address in this list it will reply back to the filer instructing it to allow the client to read the stub data directly. Performing a packet capture During the troubleshooting process, it may become necessary to perform a packet capture in order to analyze the network traffic being received and generated by an appliance. The tcpdump utility is included by default. The utility is used to capture all of the bytes being sent and received on a particular network interface of the appliance. In addition, it is also a very useful tool allowing administrators to filter and manipulate traffic being captured live, or being replayed from an existing capture file. File Management Appliance may have multiple network interfaces and in order to fully capture all traffic entering and leaving an appliance, multiple instances of the tcpdump utility need to run concurrently. To assist with this process, the fmtcpdumpall utility is included. When run, the single command triggers the tcpdump utility to run against each configured network interface. When the script is stopped, it terminates the tcpdump processes and compresses the resulting output files into a single tar+gzip file. Log files File Management Appliance generates a number of log files; each is described below. /var/log/rfalert.log 33 File Archiving from NetApp to EMC Data Domain

34 Any alerts generated by the appliance are recorded in this log file. This includes events such as logins, changes to system configuration, and hardware failures. /var/log/messages This is a standard Linux log file that is controlled through the /etc/syslog.conf file. The log contains standard operating system messages. /var/log/secure This log file contains messages about system security events such as user authentication and the availability of RADIUS or LDAP services. /var/log/rainfinity/histogram This directory contains log files detailing the appliance hardware performance and load. Statistics similar to those that can be gathered using the netstat command are kept for each network interface. The sys1.log file contains CPU, disk, and memory performance statistics. The proc_*.log files contain memory and CPU requirements imposed by specific processes such as the FPolicy Callback Daemon and the Celerra Callback Daemon. /var/log/rainfinity/filemanagement/debug.log The debug.log file contains log messages from the running filemanagement service at the DEBUG log level. This file contains messages of types DEBUG, INFO, NOTICE, WARNING, and ERROR. /var/log/rainfinity/filemanagement/system.log The system.log file contains log messages from the running filemanagement service at the INFO log level. This file contains messages of types INFO, NOTICE, WARNING, and ERROR. /var/log/rainfinity/filemanagement/deleteorphan.log This log file contains messages detailing orphan file management activities. The success or failure deleting orphan files is recorded in this log. /var/log/rainfinity/filemanagement/stubrecovery.log This log file contains messages detailing stub re-creation activities. The success or failure of re-creating stub files on primary storage is recorded in this log. /var/log/rainfinity/filemanagement/stubscanner.log This log file is created by the stub scanner task and provides general statistics for the number of files scanned on particular servers and shares/exports as well as the number of stub files detected. /var/log/rainfinity/filemanagement/fws/support/ This directory contains a log file for every time a task was run. The log file details the actions taken by the task. 34 File Archiving from NetApp to EMC Data Domain

35 Additional troubleshooting utilities File Management Appliance is based on Linux and supports a wide range of standard commands and utilities such as ping, netstat, arp, nslookup, tcpdump, top, ps, ifconfig, ifdown, ifup, service, and so on. In addition, FMA can be controlled using the rffm command from the command line. This command provides similar functionality to what is available in the GUI. When troubleshooting, it is often more convenient to use the fmsupportdump utility to gather system information, as opposed to using a collection of rffm and Linuxbased commands. This utility gathers the running system status (using rffm) into a text file called raininfo.txt, the startup config files, and the system log files, and compresses these into a single tar+gzip file that can be copied from the appliance and analyzed elsewhere. Conclusion EMC File Management Appliance can be used to implement a tiered storage strategy that can result in significant storage savings. This document covers the key aspects of deploying a tiered architecture with FMA and Data Domain storage. 35 File Archiving from NetApp to EMC Data Domain