XenData White Paper XenData Archive Series Software Technical Overview Advanced and Video Editions, Version 4.0 December 2006 XenData Archive Series software manages digital assets on data tape and magnetic disk cache on a Windows server. It creates a secure digital archive that is ideal for digital video, medical images, seismic files and a wide range of other engineering and scientific data. The software supports a broad range of tape drives and robotic tape libraries and is highly scalable. It is optimized for archiving and retrieving high volumes of files - from terabytes to multiple petabytes. Contents Page 1. The Basics 2 2. Standard File System Interface 4 3. Continuous Data Protection and Tape Cartridge Replication 5 4. Hierarchical Storage Management 6 5. Unalterable WORM Tape 7 6. Additional Features 8 7. Glossary 8 20061231 Copyright 2005-2006 XenData Limited. All rights reserved
1. The Basics XenData Archive Series software manages data tapes and magnetic disk on a Windows file server to create a highly scalable digital archive. The solution may be configured with a wide range of data tape libraries, stand alone data tape drives and magnetic storage. It is ideal for archiving professional digital video, medical images, scientific and engineering data and any application which requires a high performance data repository ranging from a few terabytes to multiple petabytes. Key Functionality The software provides the following key functionality: Standard Windows File System Interface The magnetic disk storage and managed tapes appear as a single standard Windows logical drive which means one or more software applications can write to and read from the archive as though it were a standard disk-based logical drive letter. Strong Data Protection The system combines continuous data protection and automatic tape cartridge replication. Hierarchical Storage Management The solution supports online, nearline and offline files. A file is online when it is available from magnetic disk. Nearline refers to a file that is not on magnetic disk but is available from one or more tape cartridges held in the library or within stand-alone tape drives. Offline refers to a file that is not on magnetic disk but is written to one or more tape cartridges all of which have been exported from the library and/or tape drives. Supports WORM Tape The solution supports both standard rewritable tape and unalterable WORM tape. WORM tape is particularly useful for compliance applications. Network Configuration The Windows file server is always configured with a dedicated logical drive on magnetic disk which is managed by the XenData software together with a data tape library and / or data tape drives. A typical system configuration with a single tape library is shown in the diagram below. XenData Archive Series software is hosted on a server running Microsoft Windows 2000 Server or Server 2003. The equivalent Microsoft NAS operating systems are also supported which are termed Windows Powered and Storage Server 2003, respectively. File Storage Policies When writing to and reading from tape, the system always uses the dedicated magnetic disk logical drive for caching purposes. On writing to tape, a file is first written to magnetic disk and after it has been successfully written to disk, the file goes into a queue for writing to tape. On reading a file from tape, it is streamed onto disk cache and simultaneously transferred over the network to fulfill the read request. 2
The system administrator defines policies that determine where data files are physically stored. A single server may have many different policies, tailored to the needs of the different file types that are being archived. The file management policies are defined via the XenData Management Console which is a Microsoft Management Console (MMC) snap-in. This provides a familiar administration interface which can be easily integrated with other MMC snap-ins. The policies determine whether a file will be written to tape and, if so, a specific group of tapes is allocated. The policies also define whether the tapes will be replicated and the schedule for replica updates. The online retention period is also set by policy and can range from permanent retention on magnetic disk to flushing from disk as soon as the last tape replica has been successfully written at which time the file is available nearline. A typical policy is illustrated below. With the file management policy illustrated above, a file would be written to both magnetic disk and a specified group of tape cartridges. This immediately provides two copies of the file for high data integrity. The system administrator will have predefined a replication schedule and the number of tape copies for replication. After all copies of the file have been produced, the file may be flushed (deleted) from magnetic disk according to the administrator-defined policy. On reading the file, it is always retrieved from the fastest physical storage device. If it is stored on magnetic disk and tape, the file will be read from disk. If the file is available only on tape, it will be retrieved from tape. WORM Tape Support The software supports both rewritable and Write-Once-Read-Many (WORM) tape cartridges. WORM tape provides an unalterable record of the data written to it and provides best practice when authenticity of the digital content is important. XenData Archive Series software supports the simultaneous use of WORM and rewritable media within the library. The software presents the magnetic disk storage and tape library as a standard rewritable logical drive, even when unalterable WORM tape is being used within the library. The ability to delete and overwrite files is required by many standard applications. Consequently, the rewritable nature of the archive allows standard applications to work with the XenData storage solution without need to write any special interface software. When a file is deleted, it is no longer available using the standard file system interface. Similarly, when a new version of a file is written, only the latest version is available using the standard interface. However, the XenData software includes a browser-based utility, XenData History Explorer, which makes it easy to identify, access and restore all old file versions and deleted files from tape. Advanced and Video Editions Archive Series software is available in two editions - Advanced and Video - which have identical user interfaces. The Video Edition is optimized for FTP and CIFS transfer of large files. It requires a higher specification server with more RAM and provides higher transfer rates for large files. The extra RAM requirement is used for additional caching of each file to smooth data transfer to and from RAID. It is the same software code for the two editions but the system is configured differently on licensing. 3
2. Standard File System Interface XenData software presents tape and disk as a single rewritable logical drive letter. The use of a standard rewritable file system provides a major benefit; it means that the XenData archive works with standard applications without modification. Example of how the XenData logical drive letter appears to Windows Explorer. In this case, drive letter E is controlled by the XenData software. Even when using WORM tape cartridges, files written to the XenData logical drive may be deleted or modified as though they are being written to a standard rewritable magnetic disk. However, all versions and all deleted files are retained on tape and can be accessed via a XenData utility. Microsoft Offline Attribute XenData Archive Series software uses the standard Windows offline file attribute to identify files that are no longer online on magnetic disk storage. After a file has been successfully written to tape and flushed from magnetic disk, the offline attribute is set. This Microsoft file attribute changes network timeout periods to allow retrieval of files from storage media with long access times. It also changes the appearance of a file within Windows Explorer - a small clock is added to the bottom left of the file icon as shown below. Standard Microsoft File Security XenData Archive Series software integrates fully with the Microsoft Windows security model, based on Active Directory. Files and directories have user-definable security attributes just as they do with standard Microsoft file systems and access control checks are performed in the same way. 4
3. Continuous Data Protection and Tape Cartridge Replication The XenData storage solution replaces conventional backup because it automatically produces multiple instances of files and metadata, as defined by administrator policies. In case of magnetic disk failure, the complete file system can be easily restored from one of the tape sets. The multiple instances of a file are created in two steps: 1. Continuous Data Protection Files are always initially written to magnetic disk. Immediately after the file is written to disk, it goes into a queue to be written to the primary tape cartridge. This provides Continuous Data Protection without any backup window! It overcomes the scalability limits of conventional backup. 2. Tape Cartridge Replication Updating of tape cartridge replicas for disaster recovery purposes is automatic and follows the policies defined by the Administrator. This creates tape cartridge duplicates for off-site retention. Overcoming the Backup Window Problem The problem with conventional backup is that it does not scale well above about a terabyte of data. The backup window eventually becomes so large that there is never enough down time to complete a full backup. This is illustrated below. Volume of Data Time for Full Backup (at 20 MB/s) 1 TB 14 hours 5 TB 70 hours 10 TB 140 hours 50 TB 4 weeks Taking only incremental backups is not the answer because it leads to a shelf of incremental tapes that soon become unmanageable. XenData Archive Series software provides continuous data protection, writing to tape immediately after a file is stored on disk. This approach avoids a backup window and the associated scalability limitations. A further advantage of using continuous data protection is that all data is restored unlike conventional backup where changes to the file system since the last backup are lost. Tape Cartridge Replication The Administrator defines replication policies for any group of tapes. An unlimited number of tape cartridge replicas can be produced automatically by the system. In addition to defining the number of replicas, the administrator defines when replication will occur. This can be immediately, periodically or at the same time each day, as illustrated in the figure opposite. The XenData replication mechanism supports the rotation of partially written tape cartridges between an off-site location and the tape library. A tape cartridge replica can be exported from the tape library at any time, including when it is only partially written. When a tape cartridge replica is reintroduced into the library after being retained off-site, it is automatically brought up to date by the system. 5
When exporting any tape cartridge from the system, including replicas for offsite retention, the location of the tape can be recorded in the XenData Management Console, as illustrated below. The XenData Management Console allows the Administrator to identify the location of offline tapes Data Recovery Data recovery is very easy to accomplish whether it is restoration of the complete file system or the recovery of an individual file that was accidentally deleted. Also, it is easy to test that files have really been written to tape, unlike conventional backup. An individual file can be quickly retrieved from a tape cartridge and opened using the XenData History Explorer utility which provides confidence that the system is functioning correctly. In case of failure of the RAID, the complete file system can be easily restored from tape using the XenData Management Console, illustrated to the right. In case of accidental deletion of a file, the XenData History Explorer utility allows recovery from tape. This browser-based utility is shown opposite. It displays all instances of all versions of all files, including deleted and renamed files. Any instance of a file may be opened or saved to another location. 6
4. Hierarchical Storage Management Online Retention Policy The administrator defines rules for online retention periods. After a file has been safely written to both magnetic disk and tape, the file becomes eligible for deletion from disk. This releases the disk space that was occupied by the file but does not delete the file from file system because it is still available from tape. The process of deleting from disk but maintaining file access from tape is termed flushing. Flushed files have the Microsoft offline attribute set which prevents network timeouts when they are read. If a flushed file is on a tape within an attached tape library or is on a tape within an attached stand-alone tape drive, it is available nearline. When a nearline file is read, it is simply automatically retrieved from tape as though from disk. Of course, the time to start reading the file is longer than if it were on disk. Configuring Nearline Storage A high capacity nearline archive can be configured with an attached tape library and relatively smaller capacity magnetic disk which is used as a cache. The administrator sets a policy to flush files from disk immediately after they have been successfully written to tape. By keeping at least one tape from each replicas set within the library, all files are automatically available to the network. This solution is highly scalable and very cost effective.. Offline File Management After a file has been flushed from magnetic disk and all tapes containing that file have been exported, it becomes offline. The file still appears within the file system. By using the XenData Alert Module, an on-screen and/or e-mail notification will be sent whenever an offline file is accessed. This identifies which file was requested and it s tape barcode, as shown opposite. Combining XenData Archive Series software and the XenData Alert Module, provides very effective management of large offline archives. 5. Unalterable WORM Tape WORM Formats Most of today s higher performance tape drives are WORM enabled. The table opposite shows the WORM enabled drives that are supported by XenData software. They are widely used by the leading tape library manufacturers. All of these drives could be considered as multi-function in that they will work with both standard rewritable tapes and with unalterable WORM tape. Using XenData software, the system can be configured to write to only WORM tapes, or to only rewritable tapes or to a combination of both. Drive manufacturer Tape Format Native Capacity HP LTO-3 400 GB IBM LTO-3 400 GB Quantum SDLT 600 300 GB Sony AIT-2 50 GB Sony AIT-3 100 GB Sony AIT-4 200 GB Sony SAIT-1 500 GB File System Audit Trail The XenData software includes a utility, XenData History Explorer, which provides a file system audit trail. It lists all versions of all files written to WORM tape and their media locations. This includes any deleted and renamed files. It allows retrieval of all file versions and instances directly from tape. The XenData History Utility means that if you have a file written to WORM tape, it can be retrieved even if the file has been deleted from the file system or is an old version of a file. This avoids the unacceptable situation where you know that a file has been written to WORM media but you do not have the software to access it. 7
6. Additional Functionality Additional features of an archive server based on XenData software include: Multiple Tape Sets - the Administrator can group related files together on the same set of tapes. Partial File Restore useful when reading a small clip from a very large video file. Dynamic Expansion of Tape Sets - the system will dynamically expand tape sets using blank media within a tape library to meet capacity demands. Cartridge Spanning - the system supports storage of large files across multiple tape cartridges. Standard Tape Format - open standard TAR file format is used on the tape, allowing the tape cartridges to be read using standard third party utilities.. Comprehensive diagnostics tightly integrated with the Windows operating system. In addition to extensive functionality and adherence to industry standards, XenData software has a strong reputation for ease of use. 7. Glossary Terms used in this paper and other XenData documentation are described in this section. Activation Code An Activation Code is required to run XenData Archive Series software and this enables the chosen library and drive configuration. AIT An acronym for 'Advanced Intelligent Tape', a popular mid-range tape format developed by Sony. API An acronym for 'Application Program Interface'. Most installations of XenData Archive Series software do not use the XenData API. However, a software development kit is available to developers that allows them to use the XenData API to tightly integrate their application with XenData Archive Series software. Autoloader A term for a tape library which has only one tape drive and cannot be expanded to include more. Blank Media Set The set of tapes shown in the XenData Management Console which consist of new (unused) tapes or rewritable tapes that have been reformatted by the system administrator. CIFS An acronym for 'Common Internet File System'. It is the standard protocol used by Windows computers to communicate on a network. DLTIce Quantum brand name for WORM functionality which is applicable to the SDLT 600 drive and media. Note that the XenData Management Console may be used to format an unused blank SDLT 600 tape as WORM. Dynamic Disks From Windows 2000 on, Microsoft introduced the option to configure magnetic disk storage as either Dynamic Disks or Basic Disks. The magnetic disk logical drive that is managed by XenData Archive Series software must be configured as a Dynamic Disk. File Fragmentation The way in which computer systems break large files into smaller, more manageable units for transfer to or from storage devices. Enabling file fragmentation for a File Group allows partial reading of large files and cartridge spanning. File Group A File Group is a collection of files that have the same XenData file management policy and consequently are all treated in the same way by the system. Every file that is handled by the system belongs to exactly one File Group. Files are assigned to a File Group on the basis of their names. This assignment can be based on the name of the directory that contains a file, the name of the file or both. When the 'Save files to tape' option is enabled for a File Group, all files in that File Group will be saved to a designated Volume Set. Flush Files are flushed (or de-migrated) from a storage device when they are deleted to free space. XenData Archive Series software can be configured to automatically flush files from magnetic disk, once they are securely stored on tape. After flushing, the file is still present in the file system and is available from tape. FTP An acronym for 'File Transfer Protocol'. FTP is a protocol commonly used to copy files between two computers on the Internet. Both computers must support their respective FTP roles - one must be an FTP client and the other an FTP server. History Explorer See XenData History Explorer. HSM An acronym for 'Hierarchical Storage Management'. XenData Archive Series software supports three main levels of storage hierarchy: 8
Online with one instance of a file on magnetic disk and, in addition, there may be one or more instances on tape. In this case the file will be retrieved from disk when accessed over the network. Nearline with at least one instance of a file on tape within a tape library or tape drive and no instance on magnetic disk. Often the file will be written to more than one tape cartridge using XenData s automatic tape cartridge replication capabilities. Offline with no instance on magnetic disk and one or more instances of a file on tape, all of which have been exported from the tape libraries and ejected from stand-alone drives. LTO An acronym for 'Linear Tape Open', a popular mid-range tape format which is also known as Ultrium. MMC An acronym for 'Microsoft Management Console'. The MMC can be used to create, save, and open administrative tools that manage the hardware, software and network components of a Windows system. The XenData Management Console is an example of such a tool. NAS An acronym for 'Network Attached Storage'. XenData Archive Series software may be installed on Windows based NAS file servers. (The applicable XenData specification defines which NAS operating systems are supported for a particular version and edition of XenData software.) Nearline See HSM NTFS Microsoft's advanced file system used to store and manage files on a storage medium. It is the preferred Windows file system when storing files on magnetic disk. NFS An acronym for 'Network File System'. It is an open standard protocol used by computers to communicate on a network. It was developed by Sun Microsystems and is commonly used by UNIX computers. Offline See HSM Offline File Attribute XenData Archive Series software uses the standard Windows offline file attribute to identify files that are no longer online on magnetic disk storage. After a file has been successfully written to tape and is no longer on magnetic disk, the offline attribute is set. This Microsoft file attribute changes network timeout periods to allow retrieval of files from storage media with long access times. Online See HSM Petabyte One thousand million megabytes Primary Tape When new files are written to a Volume Set, they are first written to the tape that is designated as the Primary. For any given Volume Set, the system may change the physical tape that is designated as the Primary. This occurs, for example, when a Primary Tape is exported and replaced with one of its replicas. The Primary Tape is identified within the XenData Management Console. Quarantined Media Set This is a set of tapes shown in the XenData Management Console containing tape cartridges that have been imported into the library but for some reason cannot currently be used by the system. Typically, this will be because a cartridge has previously been used by a different application (such as a backup application) or because an error occurred while the system was trying to identify the contents of the cartridge. RAID An acronym for 'Redundant Array of Independent Disks'. RAM An acronym for 'Random Access Memory'. SAIT An acronym for 'Super Advanced Intelligent Tape', a popular mid-range tape format developed by Sony. SAN An acronym for 'Storage Area Network'. The magnetic disk logical drive managed by XenData software may be provided from SAN. Note that it must be formatted with NTFS and configured as a dynamic disk. SDLT An acronym for 'Super Digital Linear Tape', a popular mid-range tape format developed by Quantum. Stand-alone Tape Drive Data tape drive that is not internal to a tape library. Tape Alert Tape drives and other hardware employ a mechanism called Tape Alert to send maintenance information to external devices. XenData Archive Series software periodically monitors Tape Alert information and informs the user or takes other appropriate action where necessary. 9
TAR A term derived from 'Tape ARchive'. It is a popular open standard data format optimized for archiving to tape and is the format used by XenData Archive Series software. Terabyte One million megabytes. Ultrium A popular mid-range tape format, also known as LTO. Volume If replication is not enabled, the XenData term 'Volume' refers to an individual tape. If replication is enabled, a Volume consists of a complete set of replica tapes which, when up-to-date, all contain the same data. For example, when replication is enabled and one additional copy of each cartridge has been specified by the administrator, a Volume will consist of two tapes. Volume Set This is a set of one or more Volumes which stores files from designated File Groups. As more data is written to a Volume Set, the initial Volume will eventually become full. At a preset threshold, defined by the administrator, the system will automatically add another Volume by taking the appropriate number of tapes from the blank media set and will extend the Volume Set. WORM An acronym for 'Write Once Read Many'. WORM tape cannot be reformatted and after data is written to the tape, it cannot be changed. However, further data can be appended. XenData software manages this non-rewritable medium in such a way that files can be deleted within the file system in the normal way. However, they are really being hidden from the normal file system interface and can be retrieved using the XenData History Explorer utility. The History Explorer utility can be used to identify and restore all old file versions and all deleted files. When using WORM media, this capability to identify and retrieve all deleted files and old versions is vital when legal compliance is an important issue. XenData Alert Module This is a companion product which is purchased separately. It provides e-mail and on-screen alerts that are tailored to the needs of archive system operators, systems administrators and IT support personnel. The alerts are derived by filtering and categorizing events recorded in the XenData Event Log. XenData Event Log This is integrated into the Windows Event Viewer within the Computer Management Console. The XenData Event Log is used for XenData specific messages. If the system is not behaving as expected, the Event Log is the first place that a system administrator should look. XenData History Explorer This is a XenData utility used to obtain the complete history and status of any file that the system knows about. It can list all available versions of all files, including deleted and renamed files. It allows retrieval of all file versions from magnetic disk or tape. XenData Management Console This is used to configure all File Group, Volume Set and tape replication options, and to view diagnostic information about the system. For more information, visit www.xendata.com or contact XenData: USA +1 925 472 6522 UK +44 1223 370114 Germany +49 89 99216 422 10