Aggregating File Data with the Snap Enterprise Data Replicator

Transcription

1 SERVER STORAGE SOLUTION BEST PRACTICES Aggregating File Data with the Snap Enterprise Data Replicator 1. Abstract Introduction to Snap EDR Product Architecture Aggregate Data Management Tool Default Functionality System Setup and Software Installation Installing the Snap EDR Management Console Software Installing Snap EDR Agents Configuring Schedules Monitoring Performance Viewing Job Statistics Viewing the Job Log Files Verifying Performance Report Types Best Practices Bandwidth Requirements Directory Configuration on Remote Nodes Permissions Selection Platform Choice Use of Compression Delivery Mode Conclusion Abstract This whitepaper provides an overview of the Snap Enterprise Data Replicator (Snap EDR) product, and details the components and configuration of a Snap EDR Aggregate solution. This solution allows users to aggregate file data to a common location from multiple Adaptec GuardianOS Snap Servers, Windows clients and UNIX/Linux clients. Common business needs for aggregation of file data include: Retrieval of daily sales files for sales reporting Collection of server logs to support monitoring Collection of Point of Sale transaction data 2. Introduction to Snap EDR Snap EDR includes three Data Management Tools that automate the secure flow of data between systems: The Aggregate tool allows job-based scheduling for transferring files from multiple hosts to a single target host, and storing the files on a local disk The Distribute tool allows job-based scheduling for transferring files from one source machine to one or more target machines The Replicate tool job-based scheduling of file transfers between two systems Also available for Snap EDR is the Advanced Data Management (ADM) Suite that groups together the functionality of Remote Inventory, Remote Archiving, and Remote Backup and Restore. These solutions provide exceptional functionality to create and deploy policies for solving remote data management problems such as variability, dissimilar computing platforms, security needs, and data integrity. The Advanced Data Management Suite simplifies the following tasks: Inventory to determine storage utilization Archive to move selected data to central sites Backup to protect data Restore to reinstall backup data in a recovery situation Separate white papers provide details on the Distribute and Replicate tools and the tools in the ADM suite, and are available at English+US&docType=Whitepapers&source=subnav

2 2 3. Product Architecture The Snap EDR consists of a Management Console that is installed on a single Snap Server, and a collection of Agents running on the various clients involved in the replication. The Management Console coordinates and logs the data transfer activities carried out by the distributed Agents. The Agents are responsible for the actual transfer of data. A web-based administration browser gives users access to the Management Console, allowing them to configure Snap EDR components and to create data management and transfer jobs from any location. The following diagram shows an example of the Snap EDR solution: SNAP EDR Snap EDR Manager REMOTE UNIX SERVER SNAP EDR SNAP EDR SNAP SERVER IP Network REMOTE LINUX SERVER SNAP EDR SNAP EDR REMOTE WINDOWS SERVER REMOTE SNAP SERVER 4500 MANAGEMENT CONSOLE INTERFACE Data Movement Control The Administrator logs into the Management Console via the Web-based administration browser. The administrator then creates and configures jobs through this interface. The Snap EDR Management Console then communicates with the agents involved in the job, and the agents transfer the data to the appropriate machine(s). The Management Console logs the transfer activity, and the user can view these log files and reports through the Management Console interface. Note that the flow of control and data movement illustrated above differs depending on the type of transfer that has been created. In some cases, users may want to transfer data from the central Management Console to remote agents (called a push transfer). In other cases, users may want to transfer data from remote agents to the central Management Console (called a pull transfer). Users may also choose to transfer data among the remote agents (e.g., data from a remote Windows server to a remote Unix Server). The role of the Management Console is analogous to that of an air traffic controller. Just as an air traffic controller informs, monitors, and supervises the movement of airplanes in a specified airspace, so the Management Console initiates, monitors and supervises the flow of data through specified areas of the network. The aggregation of file data using Snap EDR is a straightforward process involving the activation of the Snap EDR agents on two or more host machines, configuring the parameters of the aggregation (e.g., source and target directories, etc.) and scheduling the aggregation task.

3 3 4. Aggregate Data Management Tool The Aggregate data management tool retrieves files from multiple source Agents to a single target Agent, and stores the files on the target s local disk or device. The files transferred from the source Agents may be stored on a per Agent basis, or in a single flat namespace on the target Agent s storage device. This is referred to as a Pull file transfer Default Functionality The Aggregate data management tool has the following default functionality: File transfer from one or many Agents to a single source Agent Cross-platform support (transfers can go from one or more Snap Servers, running GuardianOS 3.1 or later, Windows, or UNIX/Linux Agents to a single Snap, Windows, or UNIX/Linux Agent) Store files from source Agents in a single flat namespace or in a separate namespace on a per source Agent basis Option to exclude files from the transfer Option to exclude directories Option to remove files transferred from source Agents after sending to the target Agent Users can set information in the following categories: Source Agent selection Target Agent selection Directory and file selection File transfer options (such as file compression, certified delivery, source and target synchronization, etc.) Schedule notification 5. System Setup and Software Installation The architecture of the Snap EDR system requires users to install a central Management Console that coordinates and logs the data transfer activities of the distributed Agents. This Management Console should be installed on only one Snap Server. In addition, users can install Agent software on any node that will be involved in the data transfers. Users can install the Agent software on Snap Server, Windows, or UNIX/Linux systems. Installing and configuring the Snap EDR software involves the following procedures: Installing the Management Console software on a Snap Server Installing Snap EDR Agent software on Snap Server, Windows, and/or UNIX/Linux nodes Snap EDR installs only on GuardianOS or higher Once the software is installed, the browser-based Administration tool can be used to login to the Management Console and configure and schedule aggregation jobs Installing the Snap EDR Management Console Software The Management Console software is the engine that drives Snap EDR. The Management Console software is installed on a single Snap Server from which it coordinates and logs the data transfer activities carried out by the distributed Agents. Users can login to the Management Console through a web-based administration browser in order to configure Snap EDR components and to create data management and transfer jobs. Note: Only one Management Console should be configured to handle all of the Agents in the Snap EDR replication environment. Install the Management Console software through the browser-based Administration Tool for the Snap Server. For detailed procedures on how to install the Management Console software, refer to the product documentation.

4 Installing Snap EDR Agents Snap EDR agents are responsible for the actual transfer of data. An agent is installed automatically on the Snap Server that is designated as the Management Console. The Snap EDR Agent software must be installed on each remote node in the system that will be part of the Snap EDR data transfer environment. Users can install a Snap EDR Agent on any Snap Server, Windows, or UNIX/Linux system that meets the following requirements: Item For detailed procedures on installing the Snap EDR Agent software on a Snap Server, Windows or UNIX/Linux system, refer to the product documentation 6. Configuring Schedules Description Operating System AIX 4.3, 5.1, 5.2 Solaris 5.7, 5.8 HP-UX 11.x, 11i Fedora Linux, RedHat Linux 7.2/7.3, Enterprise Linux 2.1 or 3.0 or SUSE LINUX 8.2 and 9.0 Windows 2000 (with SP2 or higher), Windows Server Appliance Kit (SAK) 2.01, or Windows XP System Memory 128 MB or more Disk Space Windows systems: 5 MB or more UNIX systems: 5 MB or more Installation directory: 100 MB, 20MB free in/tmp on all platforms Network Connection 100 Mb/sec Ethernet or faster After installing the Snap EDR software, login to the Management Console to configure the Aggregate tool to schedule the type of aggregation job you want to run. When you login to the Management Console, the following screen appears, displaying a menu on the left-side that allows users to schedule a variety of jobs (aggregate, distribute, backup, replicate, and so on), install agents, change user preferences, run maintenance tasks, generate reports, and view transfer logs:

5 5 To schedule an aggregate job click on the Aggregate menu item on the left side of the screen. The Aggregate configuration and scheduling screen is similar to the following: When creating a job, users can specify the source and target agents, which directories to exclude, file ownership preservation and so on. For detailed instructions on creating an aggregate job, see the product documentation. 7. Monitoring Performance There are a number of ways a user can monitor the performance of a job, including the following: Viewing Job Statistics Viewing the Job Log Files 7.1. Viewing Job Statistics A scheduled job s statistics provide information such as the start and end time of the job and whether or not the job ran successfully. The Scheduled Job View screen also allows access to Console functions where other tasks can be performed with the scheduled job. These tasks include manually running a job and viewing past log files. The following is a sample of the Job View screen:

6 Viewing the Job Log Files Whenever a scheduled job is run, the Management Console generates a log file. The display area shows log files for the most recent run and up to ten previous job runs. If there are more than ten job runs, the oldest job log is deleted to allow space for the newest job log. Clicking on the word view beside any past job run displays a detailed log summary screen for that particular job run. This screen displays messages about the transfer generated during the job run by the Management Console and the Agents. These messages vary in severity, from simple information messages (e.g., starting group: Aggregate (sequential)), to warnings (e.g., Preserve ownership may fail for target snap01 because the system types are different., to errors and debug messages. These messages are useful for evaluating the job run. Exit codes appear in the Exit Code column in the list of job runs. An exit code indicates whether or not a job completed successfully, and if there was an error, the type of error that occurred. A 0 exit code indicates the job completed successfully. For a description of what each non-zero exit code means, click more info beside the exit code in the Exit Code column of the job screen, or refer to the product documentation. Click Filter to change the type of log information that appears or download the job log file to your local machine. Downloading the file allows it to be loaded into different programs (for searching, printing and so on). The log file is a plain text file.

7 7 8. Verifying Performance The Management Console creates at least one statistics record for each completed job run and stores it in the Rules Database. Using the Management Console UI, users can generate a number of different types of reports to evaluate the operation of each job Report Types The following types of reports can be generated using the Management Console: Report Type Stats Summary Report Detail Report Custom Query Report Description Provides a summary and totals of the data from all jobs in the selected job group (all Distribute, Aggregate and Replicate jobs are in the Published_Solutions job group), based on how the user chooses to group the data (by month, day, etc.). Provides a detailed record of the data from the selected job group (all Distribute, Aggregate and Replicate jobs are in the Published_Solutions job group). Note that the number of records the report displays depends on the number of Agents on which the job runs. The report displays a record for every Agent on which the job runs. Provides a customized query of detail records from the job name within a selected job group (all Distribute, Aggregate and Replicate jobs are in the Published_Solutions job group). Note that the number of records the report displays depends on the number of Agents on which the job runs. The report displays a record for every Agent on which the job runs. When a report is created, a template name can be assigned to it. The template appears in the Report view. Report templates allows reports to be generated without having to change the existing parameters. Templates are especially useful for generating the same type of report on a regular basis. Run, Edit, Remove, or Schedule a report by clicking on the appropriate word beside the report template. Click Schedule to run reports daily, weekly or monthly at specific times. The following diagram is a sample of a Stats Summary Report: For detailed instructions on how to generate reports, see the product documentation.

8 8 9. Best Practices To obtain the best performance from the Aggregate job, consider these issues: bandwidth requirements directory configuration on remote nodes permissions selection platform choice use of compression delivery mode 9.1. Bandwidth Requirements To determine performance requirements for the aggregation job, it is important to have the following information: Amount of data to be retrieved Amount of time available for the aggregation Bandwidth of the most constrained segment of the network (can be either the speed of the network or a limit imposed by bandwidth throttling) Use the following equations to manually calculate bandwidth requirements: Sender bandwidth Mbps=(MB to transfer*8)/(avail. Hrs for Aggregation *3600) Receiver bandwidth Mbps=Sender bandwidth Mbps * Number of agents Example: A user needs to aggregate webserver logs nightly (between 6pm and 6am) from 100 locations. The maximum logsize is 1GB and each node being retrieving from has a T-1 (1.544 Mbps) network connection. The receiving (target) machine has a T-3 connection (44 Mbps) It has to be determined if the sender has enough bandwidth to send the data in the time required AND if the receiver has enough bandwidth to receive the data from ALL of the sources. Sender bandwidth Mbps=(MB to transfer*8)/(avail Hrs*3600) Sender bandwidth Mbps = 1000 * 8 / 12 * 3600 Sender bandwidth Mbps = 8,000 / 43,200 Sender bandwidth Mbps = Mbps (Each sender has sufficient bandwidth: Mbps < Mbps) Receiver bandwidth required Mbps=Sender bandwidth Mbps * Number of agents Receiver bandwidth required Mbps=0.185 x 100 Receiver bandwidth required Mbps=18.5 Mbps (The receive sufficient bandwidth Mbps < 44 Mbps) To determine the maximum network performance to the receiver, perform a transfer of a single large file (i.e., 20MB or more) across the same network using the same equipment and storage that will be used in production to the receiver. Ensure that network performance exceeds the required performance. Several network factors will impact the ability of the Snap EDR agents to use all of the available bandwidth: network latency, packet drop rates, network congestion, etc. For these reasons, it is useful to perform test data transfers over the actual networks that will be used. IMPORTANT: This calculation would be an estimate of WORST CASE bandwidth needs. The SNAP EDR software is optimized to make the best use of network bandwidth through features like in-stream data compression and byte-level incremental transfers. Therefore, if you are repeatedly transferring files that change only by a small amount, the bandwidth required might be very small for transfers that are not the first time transfer Directory Configuration on Remote Nodes When running an Aggregate job using hosts on multiple platforms, users must ensure that the same directory is being used on each host, or run separate jobs where the directories are different.

9 Permissions Selection The File Ownership Transfer option allows users to specify whether files maintain the source user ID and group ownership after they are transferred, and what method is used. Users can choose from the following options to preserve ownership: GuardianOS - When transferring between two GuardianOS machines, ACLs are extracted and set using the GuardianOS routines. When transferring between Windows, UNIX, Linux, or GuardianOS, user name matching is always done for the owner and group regardless of the source and target OS type, and for ACLs if the source and target are both Windows. If no match is found, a warning is generated. ACLs without matches are dropped. Owners and groups without matches are replaced with the transfer user (root, system or UID 1). Inherited ACLs are not explicitly copied between Windows machines. For example, a file that inherits its ACLs from a folder on the source is transferred to a folder on the target that has different ACLs. However, the source folder is not transferred. In this case, the file's inherited ACLs on the target will come from the target folder into which it is transferred and not from the source folder. GuardianOS mode is the default mode of transfer. Windows - Preserves Windows SIDs in homogenous Windows environments. A security identifier (SID) is a unique value of variable length that is used to identify a security principal (e.g., user or security group) in Windows Well-known SIDs are a group of SIDs that identify generic users or generic groups - these do not change from system to system. In this mode, the security stream, and all other alternate data streams, are copied; inherited ACLs are explicitly copied. Use when transferring data between Windows hosts only. UNIX - Tries to match the usernames/groupnames found on the source and target systems. Use when transferring between Windows and UNIX hosts. The file ownership transfer capability might not work on some UNIX systems such as HPUX 11 and Solaris 7 that use Orange Book security standards. Off - Do not preserve ownership. The user ID specified in the target User ID field of this job template owns files written to the target. The file will be owned by root on UNIX, NT Authority/System on Windows and UID1 on GuardianOS. The following table provides a matrix for transfer of Ownership, Permission and Attributes with Snap EDR: Transfer Type Permissions Transferred EDR Optimal Permissions Transfer Setting (c) Replicate, Aggregate, Distribute Tool Windows to Windows All Windows Windows to UNIX RWX, Owner, Group UNIX Windows to GuardianOS RWX(a),Owner, Group, Others GuardianOS GuardianOS to Windows RWX(b), Owner, Group, Others Windows UNIX to Windows RWX, Owner, Group GuardianOS UNIX to UNIX RWX, Owner, Group UNIX UNIX to GuardianOS RWX, Owner, Group UNIX GuardianOS to GuardianOS All GuardianOS (a) The read-only file attribute is mapped by setting permission bits to r--r--r-- for read only files and rw-rw-rw- for all others. The execute bit is also set for folders and executable files (e.g., file suffix.exe,.com, etc). The system, hidden, temporary, encrypted and compressed attributes are dropped. The modified and accessed dates are preserved from the source. The create date is not preserved from the source ACE entries for Owning user, group, and other users (b) File owner, group, and other users are mapped. If the write permission mode bit is off for "all" on the source, the read only attribute is set on the target. The system, hidden, temporary, encrypted and compressed attributes are set to off on the target. The modified date and accessed date are preserved from the source The create date is not preserved from the source (c) Optimal file ownership and permissions transfer is achieved when users are consistent across the servers involved in the data transfer, such as being in the same Domain. If not, EDR will try to match users based on the permissions transfer method selected.

10 Platform Choice Cross-platform transfer of file data is supported natively; however, to preserve permissions it is recommended that data be aggregated between similar platforms (e.g., Windows to Windows and UNIX/Linux to UNIX/Linux) Use of Compression If the compression option is set to Yes, the source Agent(s) compresses each file in stream before sending it. The files will be uncompressed automatically on the target Agent(s). The degree of compression depends on the type of data you being transferred. The following are typical rates of compression for different types of data: Plain text: 70-95% TIFF images: 20-40% Binary files: 0-5% Generally this feature provides maximum benefit in WAN data transfers where bandwidth is limited (e.g., less than 3 mbits per second) Using this feature may increase CPU usage, and is not recommended on a LAN, since it may slow down the transfer Delivery Mode In considering the selection of a delivery mode, the first question to ask is whether a signed receipt of the delivery is required. If so, the certified delivery option is the only mode that should be considered. The fast mode option will perform best when bandwidth is high-speed and highly available; however, check-pointing of file data will not be used (i.e., if a transfer is interrupted the entire file will be retransmitted). In most local area networks, the fast mode option is the best choice. Fast mode should be used when writing to a NAS share and preserving ownership. To summarize, in order to obtain the best performance from the Aggregate job, consider the following issues: bandwidth requirements directory configuration on remote nodes permissions selection platform choice use of compression delivery mode 10. Conclusion The Snap EDR Aggregate solution is a valuable and easy-to-use tool to aggregate file data to a common location from multiple GuardianOS Snap Servers, Windows or UNIX/Linux servers. It provides a comprehensive solution for aggregating file data from one or many remote sources to assist with common business needs for aggregation in a variety of business scenarios. Adaptec, Inc. 691 South Milpitas Boulevard Milpitas, California Tel: (408) Fax: (408) US and Canada: 1 (800) or (408) World Wide Web: Copyright 2005 Adaptec Inc. All rights reserved. Adaptec and the Adaptec logo are trademarks of Adaptec, Inc., which may be registered in some jurisdictions. All other trademarks used are owned by their respective owners. Information supplied by Adaptec Inc., is believed to be accurate and reliable at the time of printing, but Adaptec Inc., assumes no responsibility for any errors that may appear in this document. Adaptec, Inc., reserves the right, without notice, to make changes in product design or specifications. Information is subject to change without notice. P/N: Printed in USA 3662_1.3