Tivoli Data Protection for NDMP

Tivoli Data Protection for NDMP Dave Cannon Tivoli Storage Management Development Agenda and NDMP Overview of Tivoli Data Protection (TDP) for NDMP Planning the server configuration Server setup Performing backup/restore operations Potential future enhancements and NDMP What is a Device? Network Attached Storage () device Dedicated file server Storage is network-attached via TCP/IP Also known as appliance filer Provides support for file-access protocols such as Network File System (NFS) Common Internet File System (CIFS) HyperText Transfer Protocol (HTTP) File Transfer Protocol (FTP) and NDMP What is a Device? (cont.) Specialized operating system optimized for file-serving Typically does not support third-party software (such as backup applications) device can be configured either as appliance with integrated storage gateway which connects to external storage via direct attachment or SAN and NDMP Benefits of Devices Easy installation and maintenance Exploitation of existing Ethernet network Pooling of storage resources High-performance file serving Cross-platform file sharing (CIFS, NFS) Reliability/Availability Total cost of ownership and NDMP Tivoli Data Protection for NDMP -6 Dave Cannon

Vendors Network Appliance F85 F700 series F800 series EMC Celerra CLARiiON IP4700 Auspex NS2000 IBM TotalStorage 200 and 300 TotalStorage 300G gateway and NDMP NDMP Network Data Management Protocol is an open standard network protocol (SNIA) Allows storage-management application to backup and recovery operations on heterogeneous Partitions responsibilities between vendors of Separates and flow NDMP specifies protocol Data format and transfer protocol is not specified by NDMP and NDMP Benefits of NDMP High-performance, scalable backups and restores for /restore operations do not require LAN traffic Third-party software not required on the filer Minimizes porting effort for backup software vendors NDMP Client-Server Architecture Data Management Application (DMA) was known as NDMP client prior to NDMP V4 Typically a backup application Initiates, s, and monitors NDMP sessions for backup/recovery of Provides media management Tracks backed-up NDMP server performs one or more services Data Service transfers to/from primary storage (typically Tape Service transfers to/from secondary storage SCSI Service passes low-level commands from the DMA to a SCSI device (typically a media changer) and NDMP and NDMP NDMP Session Simple NDMP Configuration Bi-directional TCP/IP connection between DMA and Allows a DMA to and monitor NDMP services NDMP NDMP messages are encoded using External Data Representation (XDR) NDMP Server (Data and Tape Services) and NDMP and NDMP file system Tape library Tivoli Data Protection for NDMP 7-2 Dave Cannon

NDMP 3-Way Configuration NDMP Filer-to-Server Configuration NDMP NDMP Server (Data Service) Data Management Application NDMP NDMP Server 2 (Tape Service) Data Management Application (Tape Service) NDMP NDMP Server (Data Service) Tape library Note: NDMP Server 2 can be integrated within tape library Tape library file system and NDMP and NDMP NDMP Milestones 996 NDMP V developed by Network Appliance and PDC Software (now owned by Legato Systems) First NDMP-compliant products NDMP V2 997 NDMP V3 2000 NDMP workgroup organized as part of Storage Networking Industry Association (SNIA) 200 Over 25 NDMP-compliant products NDMP V4 NDMP use in a tape library and NDMP NDMP-Compliant Products Vendor IBM Workstation Solutions and NDMP Product Tivoli Data Protection for NDMP Traditional TSM Support TSM Client File Server NFS/CIFS Access file system Benefits File-level granularity Progressive incremental backup TSM storage hierarchy Limitations File-access overhead Data flow through TSM client/server Data flow over the network Limited backup of file attributes Tivoli Data Protection for NDMP 3-8 Dave Cannon

Traditional TSM Support (cont.) TSM Client TDP for NDMP Topology TSM Client Accepts requests from client Provides backup/restore commands Displays information User interface for backup/restore Request Runs backup/restore as TSM process Monitors backup/restore TCP/IP Initiates and monitors NDMP sessions Can cancel backup/restore Controls library/tape operations Stores meta- for stored images File Server file system Benefits File-level granularity Progressive incremental backup TSM storage hierarchy Limitations must support TSM client Data flow through TSM client/server Data flow over the network Limited backup of file attributes Device () Accepts requests from TSM server Performs tape/library operations Transfers during backup/restore Reports results to TSM server Control Data Flow Path NDMP Control File System Paths to Drives (optional) Paths to Drives Data Format is NetApp Dump Data Transfer (SCSI/FC) Tape Library (can be shared) Role of Device Accepts information requests (file server operating system level, file systems) from TSM server Passes SCSI commands to the library robotics Performs tape operations (read/write label, positioning) at request of TSM server Transfers from file system to output tape Notifies TSM server of state changes (end of volume, Reports information during backup (number of bytes Role of Provides server commands for backup/restore Accepts client requests for backup/restore operations Initiates and s NDMP sessions with device Executes backup/restore operations as background Performs library operations (mount/dismount) directly or Maintains tape library inventory Assigns drives for transfer operations Stores and uses meta- regarding stored images (full-differential relationships, storage location) Manages policy for file-system images Role of TSM Client Displays information regarding devices and their Displays information about existing file-system images stored by TSM server Provides user interface for initiating backup/restore Monitors progress and outcome of backup/restore Can be used to cancel backup/restore operations Functional Description /restore of Network Appliance file servers with Data ONTAP 6.. or higher NDMP-led backup Full file-system image Differential file-system image (files that have changed since last full backup) NDMP-led restore Full file-system image Full file-system image plus one differential file-system image Policy-based management of file-system images Tivoli Data Protection for NDMP 9-24 Dave Cannon

Functional Description (cont.) Data flow for backup/restore is LAN-free and outboard of TSM client and server Parallel backup/restore operations when multiple file systems are processed SCSI-attached libraries led via Direct attachment to TSM server Passing of SCSI commands through device Sharing of tape drives Functional Description (cont.) Windows NT/2000 servers (UNIX servers planned for Choice of user interfaces for initiating, monitoring, and canceling backup and restore operations Server console or administrative command-line client Administrative web interface Windows NT/2000, AIX, or 32-bit Sun Solaris client Web client Scheduling of backup/restore operations using the administrative command scheduler Not Supported Server platforms other than Windows NT/2000 devices other than Network Appliance Tracking or restore of individual files within file system Non-SCSI libraries (e.g., 3494) 3-way NDMP configuration Filer-to-server NDMP configurations Not Supported (cont.) NDMP V4 Detection of damaged files Data-transfer operations for stored by TSM Migration Reclamation Storage pool backup/restore Move Data Export set generation Nodes nodes represent a new node type node name uniquely identifies filer and its to TSM (similar to client nodes) Required for backup/restore operations of that filer's file spaces belong to a node QUERY OCCUPANCY and QUERY CONTENT of Some node attributes have different meanings for Platform and client OS level refer to device Session statistics are not applicable Tivoli Data Protection for NDMP 25-30 Dave Cannon

Outboard TSM Client Request Control Data Flow Request Data Source Device TSM mover Named device, external to TSM client or server Accepts request from TSM to transfer Reduces CPU cycles on TSM client and server Avoids movement over the LAN Examples: device, SCSI-3 device Data Target Device Dual Nature of Devices device has characteristics of a client Stores its own in one or more file systems Backs up and restores its device has characteristics of a mover Performs outboard transfer on request Formats during transfer node and mover must both be defined, and they must have the same name Node : relationship Data Format SCSI-3 Device Data Format (cont.) Prior to V4.2., all within the TSM storage pool hierarchy was stored using the "native" TSM format For backups using NDMP, the appliance transfers and formats the during backup to the Native format TAPEPOOL NBH format SCSIPOOL Dump format POOL TSM server stores in its own "native" format or in NBH Outboard movers may store in other formats Each storage pool and mover will have a designated Certain operations may be restricted for non-native storage Within its storage hierarchy, TSM segregates by Storage pools will have a format attribute NATIVE (the default) NETAPPDUMP (used for dumping Network Appliance filers) Other formats will be introduced as support is extended to additional devices and other mover types Path Supported Path Definitions in V4.2. Storage Agent Path Path Tape Drive Path Path Device SCSI-3 Device Source Entity Server (Storage Agent) Device (See note) Device Target Entity Drive (in library) A TSM path consists of Source and target Method by which source can access target Paths will replace Device parameter on library and drive definitions Drivemapping definitions for storage agents Allows sharing of the target device for improved resource utilization and scalability Data Mover Device Library (SCSI) Note: Prior to V4.2., access from a storage agent to a drive was specified using a drive mapping. In V4.2., DRIVEMAPPING and PATH commands will both be supported. Beginning with V5., DRIVEMAPPING commands Tivoli Data Protection for NDMP 3-36 Dave Cannon

Full and Differential Images Full backup image includes all files within a file Differential backup image includes all files that have changed since the most recent full backup No incremental NDMP backups Differential is from last full backup, rather than last backup No deactivation based on file or file-system deletion Administrator can specify either Full backup Differential backup (if full backup not found, full backup is Full and Differential Images (cont.) During differential backup, server links new differential image with correct full image Restore of differential image requires restore of full image followed by restore of differential image Expiration does not delete full images with dependent differential images Other operations remove dependent differential images if full image is deleted Policy Considerations file system images are managed according to backup copy group attributes for assigned management class Policy Constructs Policy Domain Copy group destination must be storage pool whose format is identical to that of the mover Following copy group attributes are ignored for images, because they apply only to incremental backups FREQUENCY (applies only to incremental file backup) MODE (applies only to file-level, incremental backup) SERIALIZATION (TSM depends on snapshot capability) VERDELETED (TSM does not recognize deleted file system) RETONLY (TSM does not recognize deleted file system) Active Policy Set Management Class Copy Group Management Class Copy Group Management Class Copy Group Storage Pool Storage Pool Entity Relationships Differential N Matching N Names Node Node Registration Matching N Data Formats Policy File space Domain Policy Set Full Activation Active N Policy Set Full Copy Image Mgmt Class Group N Definition Destination Differential Management Class N Differential Copy Group Image Definition Copy Group Path Definition Path Definition Storage Pool N Stg Pool Definition Device Class N Dev Class Definition Library Drive N Definition Drive M Planning the Server Configuration Tivoli Data Protection for NDMP 37-42 Dave Cannon

Supported Libraries and Drives Direct Library Control by Tape libraries must satisfy both of the following Supported by TSM device code SCSI-attached (defined with LIBTYPE=SCSI) Drives must be supported by file server hardware and Library robotics mechanism is led by direct attachment to the TSM server. device, tape library, and TSM server must be in close proximity. NDMP (SCSI) Library must support the tape drives For SAN attachment, user should also verify interoperablity of, tape, and SAN devices device file system Tape library Planning the Server Configuration Planning the Server Configuration Library Control via File Server Library robotics mechanism is led by passing SCSI commands through the device to the library. device and tape library can be physically distant from the TSM NDMP device file system Tape library Library Control Configurations Direct of library by the TSM server TSM Server Storage Agent SCSI tape library Control of library via device TSM Server SCSI tape library Library Client Drive access Planning the Server Configuration Planning the Server Configuration Example Drive Access Configurations TSM Server TSM Server TSM Server Drive Drive Drive Drive 2 Drive 2 Drive 2 Drive 3 Drive 3 Drive 3 Drive 4 Tape Library Drive 4 Tape Library Drive 4 Tape Library Planning the Server Configuration Tivoli Data Protection for NDMP 43-48 Dave Cannon

Summary of Steps. Define tape library 2. Define a device class 3. Define storage pool 4. Define policy constructs 5. Define option sets (optional) 6. Register node(s) 7. Define mover(s) 8. Define path to library (optional) 9. 0. Define paths to drives Define Tape Library Library definition for NDMP operations LIBTYPE=SCSI SHARED=NO (default) must be used for library via Specify DEVICE if and only if library will be led directly Example (library via file server) define library naslib libtype=scsi shared=no Define a Device Class device classes Sequential access Drive access is only through the file server TSM does not distinguish real device types (3590, DLT, Cannot be used for export/import, backup set generation, DB Defining a device class DEVTYPE= MOUNTRETENTION=0 ESTCAP=size Example def dev nasclass devtype=nas library=naslib mountret=0 Define a Storage Pool for Data Device class must have DEVTYPE= POOLTYPE=PRIMARY (default) DATAFORMAT=NETAPPDUMP (other formats will be added in the future) Collocation is supported (optional) def stg naspool nasclass maxscr=0 format=netappdump Define Policy for Images Consider defining a new domain for nodes Management class options Provide special management classes for nodes Provide new domain for nodes with a particular format and use default management class copy group(s) Destination pools must meet criteria for Some attributes do not apply for images FREQUENCY (applies only to incremental file backup) MODE (applies only to file-level, incremental backup) SERIALIZATION (TSM depends on snapshot capability) VERDELETED (TSM does not recognize deleted file system) RETONLY (TSM does not recognize deleted file system) Define Policy (cont.) Example def domain nasdomain def po nasdomain standard def mgmt nasdomain standard mc assign defmg nasdomain standard mc def co nasdomain standard mc dest=naspool act po nasdomain standard Tivoli Data Protection for NDMP 49-54 Dave Cannon

Define Option Sets An option set is useful if backups will be Define client option set for one or more nodes Define client options belonging to the option set include.fs.nas (includes a file system and assigns a management class) exclude.fs.nas (excludes matching file systems) domain.nas (specifies file systems to be backed up) File specification can be prefixed by node name to restrict the specification to that node Register Nodes node name should be chosen carefully because it cannot easily be changed once a corresponding Attributes Consider assigning node to a special domain TYPE= Specify client option set, if applicable Example reg node nas nas domain=nasdomain type=nas Define a Data mover name must have same name as an existing Attributes Type= IP address of filer IP port for NDMP sessions with filer (default is 0000) Userid for NDMP session with appliance Password for authenticating during initiation of NDMP Define a (cont.) Example def m nas type=nas hla=netapp2 lla=0000 userid=root passw=adsm3fun f=netappdump Data format to be used for storing on target device (NETAPPDUMP is only choice) Define a Path to the Library (Optional) Obtaining Library Information If library will be led via the filer, define a path from the mover to the library Data mover name Library name SRCTYPE=DATAMOVER DESTTYPE=LIBRARY DEVICE=device_name To obtain information about libraries that are connected to a NetApp filer, telnet to the filer and use the sysconfig sysconfig -m Medium changer (0b.0) IBM 7337 mc0 - medium changer device Example def path nas naslib srctype=mover desttype=library Tivoli Data Protection for NDMP 55-60 Dave Cannon

Define Tape Drives Define Paths to Drives Drive definitions for NDMP operations Drive must reside in a library with LIBTYPE=SCSI, Specify DEVICE if and only if drive can be accessed by TSM server for traditional backup/archive operations Example: def drive naslib drive elem=7 Define a path from the mover to each drive that the mover will use for NDMP backup/restore Data mover name Drive name SRCTYPE=DATAMOVER DESTYPE=DRIVE LIBRARY=library_name DEVICE=device_name Example def path nas drive library=naslib srctype=mover desttype=drive device=rst0l Obtaining Drive Information To obtain information about drives that are connected to a NetApp filer, telnet to the filer and use the sysconfig sysconfig -t Tape drive (0b.) Quantum DLT7000 rst0l - rewind device, format is: 8633 bpi 40 GB (w/comp) nrst0l - no rewind device, format is: 8633 bpi 40 GB (w/comp) urst0l - unload/reload device, format is: 8633 bpi 40 GB (w/comp) rst0m - rewind device, format is: 85937 bpi 35 GB nrst0m - no rewind device, format is: 85937 bpi 35 GB urst0m - unload/reload device, format is: 85937 bpi 35 GB rst0h - rewind device, format is: 85937 bpi 50 GB (w/comp) nrst0h - no rewind device, format is: 85937 bpi 50 GB (w/comp) urst0h - unload/reload device, format is: 85937 bpi 50 GB (w/comp) rst0a - rewind device, format is: 85937 bpi 70 GB (w/comp) nrst0a - no rewind device, format is: 85937 bpi 70 GB (w/comp) urst0a - unload/reload device, format is: 85937 bpi 70 GB (w/comp) Performing /Restore Operations User Interfaces TDP for NDMP provides both server and client interfaces for backing up and restoring nodes Server interfaces Server console Administrative command-line client Web administrative interface Client interfaces Command-line client Web client Performing /Restore Operations User Interfaces (cont.) The server interface may be preferred by TSM administrators who are accustomed to server Customers who want to schedule backup (or restore) The client interface may be preferred by administrators who are not familiar with TSM server People who prefer a graphical, interactive interface or who are accustomed to performing TSM client backups images can be restored using either interface, regardless of the interface used for the backup Performing /Restore Operations Tivoli Data Protection for NDMP 6-66 Dave Cannon

Scheduling of NDMP Operations NDMP-led backup/restore operations can be scheduled using the administrative command scheduler Recommended setup Test the server command for the desired operation Define an administrative command schedule for this Scheduling of operations via the client scheduler is Performing /Restore Operations /Restore Processes backup and restore operations are performed as server processes that (usually) run in the background If multiple file spaces are specified for backup/restore, a separate server process is used for Ongoing processes can be checked using Server QUERY PROCESS command Client monitoring processes can be canceled using Server CANCEL PROCESS command Command-line or web client Performing /Restore Operations /Restore Processes (cont.) Final operation results are Displayed on the server console when the process ends Stored in the server's activity log Displayed by a client that is monitoring the process Server BACKUP NODE Command Backs up specified file systems on a device using node name Names of file systems to back up Management class to which file system images will be bound Mode (full or differential) Wait (background or foreground process) BACKUP NODE ignores any options in client options backup node nas /vol/vol,/vol/vol2 mgmt=mc mode=full Performing /Restore Operations Performing /Restore Operations Server RESTORE NODE Command Client Options for Operations Restores a file-system image to a device using Parameters node name Name of file system to restore Destination file system PIT date/time Wait (background or foreground process) Example: restore node netapp /vol/vol /vol/vol2 Client options for operations can be specified in Client option set on the server (recommended) Client option file Client options include.fs.nas (includes a file system and assigns a management class) exclude.fs.nas (excludes matching file systems) domain.nas (specifies file systems to be backed up) Performing /Restore Operations Performing /Restore Operations Tivoli Data Protection for NDMP 67-72 Dave Cannon

Client BACKUP Command Backs up specified file systems on a device using Parameters node name Names of file systems to back up Monitor the background process? Mode (full or differential) Verbose/Quiet Example: dsmc backup nas -nasnode=netapp -mode=full /vol/vol /vol/vol2 Performing /Restore Operations Client RESTORE Command Restores a file system image to a device using Parameters node name Name of file system to restore Destination file system PIT date/time Monitor the background process? Verbose/Quiet Inactive/Pick Example: dsmc restore nas -nasnodename=netapp /vol/vol /vol/vol2 Performing /Restore Operations Other Client Commands Web Client Frame QUERY FILESPACE -CLASS= DELETE FILESPACE -CLASS= QUERY BACKUP -CLASS= QUERY NODE MONITOR PROCESS CANCEL PROCESS Performing /Restore Operations Performing /Restore Operations Web Client Restore Frame TSM Activities Window Performing /Restore Operations Performing /Restore Operations Tivoli Data Protection for NDMP 73-78 Dave Cannon

TDP for NDMP: Single-File Restore Restore of individual files or directories from file-system Potential Future Enhancements Basic restore Administrator specifies file name and directory File server scans backup to locate specified file Direct-access restore TSM collects and stores file information during backup of file-system image Client GUI displays image contents and provides interface for specification of file(s) to be restored File server positions to and restores selected files Potential Future Enhancements TDP for NDMP: 3-Way Configuration device does not need access to tape drives Exploits tape libraries with integrated NDMP-server Data flow over the LAN Device Data Transfer (SCSI/FC) NDMP Control Data Transfer Control Data Flow Device 2 (or tape library with integrated NDMP support) Data Transfer (SCSI/FC) TDP for NDMP: Filer to Server device does not need access to tape drives stored in TSM's storage hierarchy with native NDMP Control Device Data Transfer Storage Hierarchy Control Data Flow Tape Library Data flow over the LAN File System File System Potential Future Enhancements Potential Future Enhancements Other Potential Enhancements Additional vendors IBM EMC Celerra EMC CLARiiON IP4700 Auspex Additional tape libraries (V4.2. supports SCSI only) 3494 ACSLS of images to copy storage pools (V4.2. supports duplication by backing up to multiple primary pools under different node names) Potential Future Enhancements Tivoli Data Protection for NDMP 79-84 Dave Cannon