Archiving File Data with Snap Enterprise Data Replicator (Snap EDR): Technical Overview

Transcription

1 Archiving File Data with Snap Enterprise Data Replicator (Snap EDR): Technical Overview Contents 1. Abstract Introduction to Snap EDR Product Architecture System Setup and Software Installation Installing the Snap EDR Management Console Software Installing Snap EDR Agents Archive Data Management Module Default Functionality Configuring an Archive Job Remote Archive Move Remote Archive ShortCut Remote Archive Details File Transfer Options How Files are Archived With Windows ShortCut Archived File Layout in Target Storage Location Disk Layout for Remote Archive Move Module Best Practices - Archive Windows ShortCut vs. Versioning Solution Agent Selection Source Directories Specified Open File Support in Windows Debugging Creating Archive Policies How Solutions Work Together Sample Archiving Policies Use Case Parameters Monitoring Performance Viewing Job Statistics Viewing the Job Log Files Verifying Performance Report Types Conclusion Abstract This white paper provides an overview of the Snap EDR product, and details the components and configuration of the Snap EDR Archive data management module. The Archive module allows users to remove selected data such as rarely used files from remote sites and store them at a central location. This frees up additional disk space on the remote agents. The Remote Archive module allows for two methods of archiving data: Remote Archive ShortCut (creates Windows shortcuts to the archived file for easy retrieval) Remote Archive Move (gives users the option to create version changes of an archived file) These two methods are discussed in more detail later in this document. Snap EDR runs on the following platforms: Snap Servers powered by GuardianOS, Windows clients, and UNIX/Linux clients. Common business needs for archiving file data include, but are not limited to, the following: Removing rarely-used files from the system and store them elsewhere Creating additional storage space by removing rarely-used files from the system 2. Introduction to Snap EDR Snap EDR includes three Data Management tools to automate the secure flow of data between systems: The Aggregate tool allows jobs to be scheduled to transfer files from multiple agents to a single target agent The Distribute tool allows jobs to be scheduled to transfer files from one source machine to one or more target machines The Replicate tool allows jobs to be scheduled to transfer files between two systems Snap EDR also includes the Advanced Data Manage-

2 2 ment (ADM) Suite that groups together the functionality of Remote Inventory, Remote Archiving, and Remote Backup and Restore. These solutions provide exceptional functionality to create and deploy policies for solving remote data management problems such as variability, dissimilar computing platforms, security needs, and data integrity. The Advanced Data Management Suite performs the following tasks: Inventory, to determine storage utilization Archive, to move selected data for permanent archive Backup, to protect data Restore, to recover backup data The following describes a typical use case of the Advanced Data Management Suite: Remote Locations Restore Inventory Restore SNAP SERVER 4500 Secondary DR Sites SNAP SERVER Disk-Based Backup Figure 1: ADM Configuration Central Sites Archive Off-Line Backup 1. The user has a set of locations (with Snap EDR Agents) that are part of the enterprise but connected over the wide area network (WAN). (Note: agents can be anywhere on the network.) 2. The user runs a remote inventory job to report on the remote location file sets. It shows that over 15 gigabytes of data have not been accessed in over 270 days. The user decides to create an archiving policy based on the information provided by the remote inventory job. 3. The user runs a remote archive job with the policy created in step 2 to archive unused files. This frees more disk space on the remote locations, and decreases the amount of data that needs to be backed up. Archiving also shortens the time a backup takes because less data needs to be moved. 4. The user wants to run backups of the remote locations. Using a simple Web UI, s/he schedules a remote backup job to run on a regular basis. The backed up data from the remote locations is stored on a central location where it can be restored quickly, if required, using the Remote Restore tool. 5. It is assumed that this location backup data would then be backed up by any central backup software such as: BakBone NetVault (which comes standard on GuardianOS-based Snap Servers,) EMC s Legato Networker or Veritas NetBackup, and so on. Separate white papers provide details on the Aggregate, Distribute and Replicate tools and each module in the ADM suite, and are available from the following location: itepapers 2.1. Product Architecture Snap EDR consists of a Management Console that is installed on a Snap Server powered by GuardianOS, and a collection of agents running on the other servers/workstations involved in the transfer. The Management Console coordinates and logs the data transfer activities carried out by the distributed agents. The Snap EDR agents are responsible for the actual transfer of data. Users access the Management Console using a web browser that allows them to configure Snap EDR and to create data management and transfer jobs from any location. The following diagram illustrates the Remote Archive module using Windows ShortCuts in a typical customer environment: Inventory Report File MANAGEMENT CONSOLE Remote Locations Source Agents Inventory Report File SNAP EDR AGENT SNAP EDR AGENT SNAP EDR AGENT Target Agent SNAP SERVER Inventory Report File Figure 2: Sample Archive Shortcut Deployment Inventory Report File

3 3 In the above diagram, the Snap EDR agent is deployed on all of the machines involved in the Remote Archive job. All the machines involved in this deployment (except the storage host) are Windows machines. In this deployment, a common Universal Naming Convention (UNC) share is required between source and target agent machines to archive the files, since the files must be retrievable when recalled by the user. The following diagram shows a common deployment of the Remote Archive module using the file versioning option: Target Agent AGENT NAS/SAN STORAGE MANAGER Public Network Figure 3: Sample Archive Versioning Deployment In the above diagram, the Snap EDR agent is deployed on all of the machines involved in the remote archive job. The machines involved in this deployment can be running any supported platform of Snap EDR. In this deployment, the files archived from the source agent machines are moved to the target agent. Once moved, the files are removed from the source agent machines (optional). 3. System Setup and Software Installation The Snap EDR system requires users to install the Management Console that coordinates and logs the data transfer activities of the distributed agents. The Management Console should be installed on a Snap Server running GuardianOS. Users can install the Agent software on a GuardianOS-based Snap Server, Windows, or Unix/Linux system involved in the data transfers. Installing and configuring the Snap EDR software involves the following procedures: Installing the Management Console software on a Snap Server Installing Snap EDR agent software on a Snap T1 T1 T1 Source Agents AGENT AGENT AGENT Server, Windows, and/or UNIX/Linux system Note that Snap EDR installs only on GuardianOS or higher. Once the software is installed, users can login to the Management Console and configure and schedule archiving jobs Installing the Snap EDR Management Console Software Note: Only one Management Console should be configured to handle all of the agents in the Snap EDR replication environment. Install the Management Console through the browserbased Administration Tool for the Snap Server. For detailed procedures on how to install the Management Console software, refer to the product documentation Installing Snap EDR Agents Snap EDR agents are responsible for the actual transfer of data. An agent is installed automatically on the Snap Server that is designated as the Management Console. The Snap EDR agent software must be installed on each remote node in the system that will be part of the Snap EDR data replication environment. Users can install a Snap EDR agent on any Snap Server, Windows, or UNIX/Linux system that meets the following requirements: Item Operating System System Memory Disk Space Network Connection Description Snap Server with GuardianOS v or higher AIX 4.3, 5.1, 5.2 Solaris 5.7, 5.8 HP-UX 11.x, 11i Fedora Linux, RedHat Linux 7.2/7.3, Enterprise Linux 2.1 or 3.0 or SUSE LINUX 8.2 and 9.0 Windows 2000 (with SP2 or higher), Windows 2003, Windows Server Appliance Kit (SAK) 2.01, or Windows XP 128 MB or more Windows systems: 5 MB or more UNIX systems: 5 MB or more Installation directory: 100 MB, 20 MB free in /tmp on all platforms 100 Mbps Ethernet or faster OR a WAN connection capable of supporting the throughput required to meet the volume and window of time requirements For detailed procedures on installing the Snap EDR agent software on a Snap Server, Windows, or UNIX/Linux system, refer to the product documentation.

4 4 4. Archive Data Management Module The Remote Archive module provides a method for archiving files that satisfies a user-defined set of criteria. Archiving the files reduces the amount of disk usage at remote sites, which results in faster backups, less storage space required for backups, and increased available disk space. The Remote Archive module allows for two methods of archiving files. The first method uses Windows shortcuts. With this module, files selected for archiving are replaced with a Windows shortcut. If the file is required at a later point in time, the user simply clicks on the Windows shortcut file to retrieve the contents of the original file. Archive module to schedule the type of archive job desired. Upon login to the Management Console, the following screen appears, displaying a menu on the left side that allows users to schedule a variety of jobs (Aggregate, Distribute, Backup, Replicate, and so on), install agents, change user preferences, run maintenance tasks, generate reports, and view transfer logs: The second module allows for versioning of the archived files. With this module, the files selected for archiving are copied to the target disk storage. The user has the option to remove the file from the source machine once the file is archived on the target, which would be the default behavior. This module can also be used to keep version changes to a file. Multiple versions of the file can be stored on the target storage Default Functionality The following are the features of the Remote Archive module: Archive files from multiple agents to a common location Archive files from heterogeneous host platforms to a single common location Archive files maintaining Windows shortcut Archive files using file versioning, with a configurable number of versions to maintain Specify files to archive based on user-defined archiving rules Store files in common archive location in a flat name space or on a per host basis Full scheduling functionality for automating frequency of jobs Centralized control for viewing job status, controlling job execution, and managing agents involved Manage the amount of bandwidth the backup job will utilize Network fault tolerance during backups 5. Configuring an Archive Job Once the Snap EDR software has been installed, login to the Management Console to configure the Remote Figure 4: Management Console Snap EDR provides a straightforward interface to schedule an archive job by clicking on the Remote Archive Move or Remote Archive ShortCut menu items on the left side of the screen Remote Archive Move The Remote Archive Move module allows for versioning of the archived files. With this module, the files selected for archiving are copied to the target disk storage. The user has the option to remove the file from the source machine once the file is archived on the target, which would be the default behavior. This module can also be used to keep version changes to a file. Multiple versions of the file can be stored on the target storage. The Remote Archive Move configuration and scheduling screen is similar to the following:

5 5 Remote Archive module. The machines involved in this deployment can be running any supported platform of Snap EDR. In this deployment, the files archived from the source agent machines are moved to the target agent. With the archive move job, once moved, the files are removed from the source agent machines (optional). With the Archive ShortCut job, the files are removed and replaced with a Windows ShortCut. When creating an Archive job, users can specify the agent to which the data should be archived, the source agents and types of files to archive (by date, size, etc.), the number of archive versions to keep (with Archive Move), and so on. For detailed instructions on creating a remote archive move or a Remote Archive ShortCut job, see the product documentation. Figure 5: Remote Archive Move Job Creation Screen 5.2. Remote Archive ShortCut The Remote Archive ShortCut module uses Windows shortcuts to select files for archiving and replace them with a Windows ShortCut. If the file is required at a later point in time, the user simply clicks on the Windows shortcut link to retrieve the contents of the original file. The Remote Archive ShortCut configuration and scheduling screen is similar to the following: Figure 6: Remote Archive ShortCut Job Creation Screen In the above diagrams, the Snap EDR agent is deployed on all of the machines involved in the 5.3. Remote Archive Details This section describes how archived files are stored on the target archive directory using the Remote Archive module File Transfer Options The files are transferred between the source agents and target host using the following file transfer options: 1. Files are considered for transfer based on the userdefined criteria. If no criteria are specified, all files will be transferred. 2. Files in use during a transfer are skipped How Files are Archived With Windows ShortCut Files selected for archiving using the Windows ShortCut method are archived in the following manner. This information does not apply to the Archive Move module. File meeting the selected criteria is transferred to common UNC target directory On the source host, the transferred file is used as the target of the Windows ShortCut The shortcut name will be the file name with a ".lnk" extension added The Archive Base directory will be used as the location of the real file The original file name with the ".lnk" extension will be used to name the shortcut file Once the file is transferred and the shortcut is created, the original source file is deleted on the source machine The figure below shows the resulting shortcut files after the Remote Archive module has archived with

6 6 Windows ShortCuts the files in the directory "f:\basicdataset1\dir1\subdir1". In this case, all the files in the directory were archived. By hovering the mouse over a file, the resulting popup window displays the details of the Windows ShortCut. Figure 7: Windows ShortCut Details Archived File Layout in Target Storage Location The files to archive from the source agents are stored in the specified target archive directory. The files may be stored on a per-host basis, or in one flat namespace, depending on the user configuration. The following depicts archived files stored on a perhost basis. User specified target storage location as "\\share1\archivedata" User specified source agents in company.com, host1, host2, and host3. \\share1\archivedata\ host1.company.com\users\user1\file.doc host2.company.com\users\user2\file.doc host3.company.com\users\user3\file.doc If the user specified not to store the files on a per-host basis, but instead in one flat namespace, the same data set above would look like the following in the target archive directory: \\share1\archivedata\ users\user1\file.doc users\user2\file.doc users\user3\file.doc With the flat namespace option (not storing data on a per-host basis) the user runs the risk of having same named files from one source host being overwritten by another source host Disk Layout for Remote Archive Move Module The disk layout for the Remote Archive Move module is the same as the Windows ShortCut module when dealing with the base target archive directory. The differences occur when the user has specified to keep multiple versions of the files on the target archive directory and does not remove the files from the source after transferring to the target. In this case, when file changes are detected on the source machine and the file exists on the target, a copy of the existing file on the target is made before being overwritten by the new version from the source. The copy maintained on the target machine is referred to as a version of the file. When a version of the target file must be created, the following steps are performed: 1. Below the root target directory, a new directory is created called "archiveversions" (if it does not already exist). 2. The directory structure leading to the file for which a version will be created is set up under the "archiveversions" directory (if it does not already exist). 3. A directory is created with the same file name as the file for which a version must be created (if it does not already exist). 4. The file that will become the archived version is moved into this directory and the name is prefixed with the current date timestamp. 5. The new copy of the file is transferred from the source machine into the original target archive directory location. 6. Best Practices - Archive This section describes the recommended configuration options and best practices to deploy the Remote Archive Module Windows ShortCut vs. Versioning The Archive module provides two options for archiving files. One option archives files using Windows ShortCuts, and the other option moves the files to the target storage. Consider using the Windows ShortCut option to archive files under the following circumstances: The user wants to recall data quickly All machines involved are Windows agents All machines involved have access to the target storage location

7 7 The number of files should probably not be excessive either. Although the module does not limit the number of ShortCuts that can be created, keep in mind a ShortCut file is created for each archived file. Consider using the archive versioning solution under the following circumstances: Non-Windows agents involved All the agents involved do not have access to a common share drive Files do not have to be recalled back to source machines with link file 6.2. Solution Agent Selection When using the Remote Archive Windows ShortCut solution, all the agents involved in the transfer must be Windows machines since Windows shortcuts are used. The Remote Archive Move solution can use any agents running any of the supported Snap EDR platforms Source Directories Specified The source directories to search for files to archive should be consistent across all source agents of a scheduled remote archive job. If the source agents to archive data from have different source directories, the source agents with the common directories should be grouped into the same job. For example, agents 1 through 5 must be part of an archive job. Agents 1 and 3 have common directories to archive, c:\appdata\userfiles. Agents 2, 4, and 5 have the common directories to archive, e:\documentrepository. Create one job for host 1 and 3 specifying the source directory c:\appdata\userfiles, and another job for host 2, 4, and 5 specifying the source directories e:\documentrepository. Only one root directory can be specified for a remote archive job. From this root directory the search criteria and exclusions will be applied to all subdirectories. To archive files in more than one root directory, specify each root directory in a separate job, with the target directory or common UNC share drive being the same Open File Support in Windows When open files are encountered during a remote archive (unlikely, since it is probably archiving files that have not been used in a long time), the files will be skipped by default. If open file support is required, users can run either of the Snap EDR Remote Archive modules in conjunction with Open File Manger (OFM) from St. Bernard Systems to transfer open files encountered during an archive. OFM enables Snap EDR to successfully capture and transfer files that are open and in use during the transfer process. OFM runs as a service and watches the process list. When the transfer portion of the agent is launched, OFM becomes active, performing certain actions to ensure that open and in-use files can be transferred. NOTE: Transferring open files is not generally recommended, as the data in these files may not be in a consistent state if archived while open, as data may still be in transit buffers and not yet flushed Debugging Users can set the job log detail level option when scheduling a job, to produce agent and job logs useful for debugging. For more information, see the product documentation. 7. Creating Archive Policies This section describes, through a use case, how to create data archiving policies based on the results of a file scan inventory report generated from the Remote Data Inventory module How Solutions Work Together The remote data inventory solution file scan feature can be used to create archiving policies. Based on file scan criteria specified, the remote data inventory solution reports the resulting file information and disk size usage of the files. Running various remote data inventory jobs with different file scan criteria lets the user experiment with different archiving policies. Users can apply the same file scan criteria used in the remote data inventory job to a remote data archiving job. The directories and files reported in the remote data inventory job can be specified in the Archive module job to indicate which files and directories and from which agents to archive. With this process, the remote data inventory solution is used to create archiving policies and the remote data archiving solution is used to implement the archiving policy Sample Archiving Policies This section runs through an example of how the remote data inventory report is used to create an archiving policy.

8 Use Case Parameters In this use case, a corporation has deployed Snap EDR in the enterprise and is working with the Remote Data Inventory and Remote Data Archiving modules. File Servers The user has three Windows 2000 servers where data resides: Host1.company.com Host2.company.com Host3.company.com File Systems Each server has two file systems from which the user wants to archive data. The user suspects there is a lot of data which has not been modified in a long time taking up significant disk space, resulting in slower backup times. The file systems to perform archiving from are: F:\applicationdata E:\userdata Each server has the same drive names for ease of maintenance, but different users accessing each server. Requirement The user wants to develop an archiving policy that will archive the following files: Files that have not been modified in the last 90 days Do not include ".lnk" files (Windows shortcuts) Do not include subdirectory "finance" Remote Data Inventory Job The user schedules a Remote Data Inventory job to run with the appropriate parameters: Source Agents (Host1.company.com, Host2.company.com, Host3.company.com) Files that have not been modified in the last 90 days Do not include ".lnk" files (Windows shortcuts) Do not include the "finance" subdirectory When the remote data inventory job runs with these parameters, the following results appear: Host Number of Files Total Disk Space Used Host1.company.com gigabytes Host2.company.com megabytes Host3.company.com megabytes The user decides to create a remote data archive job to archive the files that meet the specified criteria. This will free disk space and speed up the backup jobs that run on the specified directories. The user has also scheduled this remote data inventory job to run every two weeks to monitor the unmodified file count on the servers in question. Remote Data Archiving Policy The user can take the parameters specified in the remote data inventory job and use them to create an archiving policy. In this case, the user wants to archive the unused files that are taking up disk space on the source host machines where the user ran the remote data inventory job. The user is going to schedule a remote data archive job with the same file scan criteria used in the remote data inventory job. The job will run every two weeks and archive the applicable files from the source host machines. The schedule, along with the file scan criteria constitutes the file archiving policy. The archive job will do the following: Archive files from the source agents (Host1.company.com, Host2.company.com, Host3.company.com) Archive files that have not been modified in the last 90 days Do not Archive ".lnk" files (Windows shortcuts) Do not Archive the "finance" subdirectory These parameters apply to either the Windows ShortCut or the Archive Move modules of the Remote Data Archive module. By using the criteria from the remote data inventory solution, the user has pre-determined the number of files and disk space that will be moved from the source host machines. 8. Monitoring Performance There are a number of ways a user can monitor the performance of a job, including the following: Viewing Job Statistics Viewing the Job Log Files 8.1. Viewing Job Statistics When a job that has run to completion is selected, the screen displays the job's statistics, including information such as the start and end time of the job and whether or not the job ran successfully. The Scheduled Job View screen also allows access to Console functions where other tasks can be performed with the scheduled job. These tasks include manually running a job and viewing past log files. The following is a sample of the Job View screen:

9 9 This summary displays messages about the data transfer, and is generated while the job is running by the Management Console and the Aaents. These messages vary in severity, from simple information messages (e.g., starting group: RemoteArchive (sequential)), to warnings (e.g., Preserve ownership may fail for target 'snap01' because the system types are different. ) to errors and debug messages. These messages are useful for evaluating the job run. Click the Filter button to change the type of log information displayed or to download the job log file to the local machine. Downloading the file allows it to be loaded into different programs (for searching, printing and so on). The log file is a plain text file. Figure 8: Job Statistics Screen Exit codes appear in the Exit Code column in the list of job runs. An exit code indicates whether or not a job completed successfully, and if there was an error, the type of error that occurred. A 0 exit code indicates that the job completed successfully. For a description of what each non-zero exit code means, click more info beside the exit code in the Exit Code column of the job screen, or refer to the product documentation Viewing the Job Log Files Whenever users run a scheduled job, the Management Console generates a log file. If the job is clicked on, the job's display area shows log files for the most recent run and up to ten previous job runs. If there are more than ten job runs, the oldest job log is deleted to allow space for the latest job log. Clicking on the word view beside any past job run displays a detailed log summary for that particular job run: Figure 9: Job Log Screen 9. Verifying Performance The Management Console creates statistical records for each completed job run and stores them in the rules database. Using the Management Console, users can generate a number of different types of reports to evaluate the operation of each job Report Types Using the Management Console, the following types of reports can be generated: Report Type Stats Summary Report Detail Report Custom Query Report Description Provides a summary and totals of the data from all jobs in the selected job group (Archive jobs are in the RemoteArchiveMove and RemoteArchiveShortCut groups), based on how the user chooses to group the data (by month, day, etc.). Provides a detailed record of the data from the selected job group (Archive jobs are in the RemoteArchiveMove and RemoteArchiveShortCut groups). Note that the number of records the report displays depends on the number of agents on which the job runs. The report displays a record for every Agent on which the job runs. Provides a customized query of detail records from the job name within a selected job group (Archive jobs are in the RemoteArchiveMove and RemoteArchiveShortCut groups). Note that the number of records the report displays depends on the number of agents on which the job runs. The report displays a record for every agent on which the job runs. Template names can be assigned to reports that are created. The template appears in the Report view. Report templates allow you to generate reports without having to change the initial parameters. Templates are especially useful for generating the same type of report on a regular basis. Run, Edit, Remove, or Schedule a report by clicking on the appropriate word beside the report template. Click Schedule to run

10 10 reports daily, weekly or monthly at specific times. The following diagram is a sample of a Stats Summary Report: For detailed instructions on how to generate reports, see the product documentation. 10. Conclusion The Snap EDR Archive data management modules are valuable and easy-to-use modules to manage storage utilization. The Archive modules are comprehensive solutions that can assist users in removing rarely used files from remote agents to free up disk space. Figure 10: Sample Stats Summary Report Adaptec, Inc. 691 South Milpitas Boulevard Milpitas, California Tel: (408) Fax: (408) Literature Requests: US and Canada: 1 (800) or (408) World Wide Web: Pre-Sales Support: US and Canada: 1 (800) or (408) Pre-Sales Support: Europe: Tel: (32) or Fax: (32) Copyright 2005 Adaptec Inc. All rights reserved. Adaptec and the Adaptec logo are trademarks of Adaptec, Inc., which may be registered in some jurisdictions. All other trademarks used are owned by their respective owners. Information supplied by Adaptec Inc., is believed to be accurate and reliable at the time of printing, but Adaptec Inc., assumes no responsibility for any errors that may appear in this document. Adaptec, Inc., reserves the right, without notice, to make changes in product design or specifications. Information is subject to change without notice. P/N Printed in USA 03/ _1.3