WHITE PAPER VERITAS Bare Metal Restore AUTOMATED SYSTEM RECOVERY WITH VERITAS NETBACKUP 1
INTRODUCTION...3 CURRENT METHODS FALL SHORT...3 THE REINSTALL-AND-RESTORE METHOD...3 Uncertain Recovery State...3 Lengthy and Error-Prone Recovery...4 The System Backup Method...4 Wasted Storage, Bandwidth and Effort...4 Out-of-Date Backups...4 The Failure of the Current Methods...4 VERITAS BARE METAL RESTORE...5 Bare Metal Restore Overview...5 Components of Bare Metal Restore...5 The Initial Implementation...7 Daily Operations...7 The Restoration Process...7 Bare Metal Restore Speeds Effective and Efficient System Recovery...8 Summary...9 2
INTRODUCTION VERITAS NetBackup is the industry s leading enterprise data protection solution providing customers with fast, reliable, data center strength backup and recovery. NetBackup provides a variety of recovery capabilities for single files, groups of files, entire volumes and specialized applications such as databases. In scenarios where a complete machine restoration must be performed because the root volume or operating system is inoperable, a specific manual procedure must first be executed to rebuild the machine configuration and provide the necessary operating system components to allow NetBackup to restore the appropriate applications and data. This paper describes VERITAS Bare Metal Restore, a complementary product to NetBackup that eliminates this manual procedure and greatly improves the speed and simplicity of system recovery. First, we will assess the common methods used by companies for system recovery and examine the several key problems of these methods. We will then describe the Bare Metal Restore product and how it solves the overall system recovery problem, working in conjunction with VERITAS NetBackup in a fully automated fashion and with ensured integrity. CURRENT METHODS FALL SHORT Reinstall-and-restore and system image backup are the two most common ways to recover a downed server. As we will see, both methods have serious challenges in providing overall integrity, consistency and efficiency in the restoration process. THE REINSTALL-AND-RESTORE METHOD The reinstall-and-restore method involves installing the operating system and applications from scratch and then restoring the machine s data from its latest NetBackup backup. There are several significant problems with this method. Uncertain Recovery State Since the machine is reinstalled with the base operating system and applications, the user data and other files restored from NetBackup may not match the software that the machine is now running. For example, if a machine was running a more recent patch level of an application than what is reinstalled, the application may not function properly with its data files that are restored from NetBackup. Some systems have installation applications that can potentially reinstall the machine with the same level of operating system and applications it had before a disaster e.g., JumpStart on Solaris, Ignite-UX on HP-UX and NIM on AIX. However, these all require that the installation application be constantly in sync with any of the changes to the client machine s configuration and software revision levels. If an application has been upgraded or an OS patch applied and the installation application s corresponding machine definition hasn t been updated, then a machine won t be able to be reinstalled with the same software it had before the failure. Given the nature of system administration in complex environments, it is not surprising that the installation application s view of a client machine often diverges from the machine s actual state. Another problem with these installation applications is that they are unique to a given operating system and differ from platform to platform. These differences typically mean that valuable administration skills and processes developed for one platform cannot easily be adapted to other platforms. In this environment, an administrator experienced in the given platform must usually perform the system recovery. It is not feasible to have a lessskilled employee use a common tool for system recovery on any platform. Because of the intense nature of system recovery, this significantly limits the number of restorations that can be done simultaneously. The reinstall-and-restore process requires specialized skills, and even then there is no guarantee that the machine will function properly. The final result is a machine that has questionable integrity since it was pieced together from potentially disparate elements. 3
Lengthy and Error-Prone Recovery It can take a long time to reinstall a machine s operating system and applications, restore its data and then fix its configuration and reconcile any differences between the reinstalled and restored pieces. It can take days to try to recover a machine this way, and in the end it is often impossible to completely recover the machine. In addition, there are many opportunities for error along the way. Many of these errors are subtle and could take additional days to discover, troubleshoot and rectify. THE SYSTEM BACKUP METHOD On some operating systems it is possible to take a system backup that is a snapshot of all the data on some or all of the system s disks. Using the mksysb and savevg commands on IBM AIX or a product like Symantec Ghost on Windows are some examples. Until recently, these solutions represented the only method of coherently recovering a machine after a catastrophic failure. Wasted Storage, Bandwidth and Effort These processes require a full backup, so they use a huge amount of storage space and network bandwidth. A typical boot disk backup is at least several hundred megabytes, and a full system backup can be many gigabytes. Multiplied by tens or hundreds of machines, this represents a significant amount of data to back up on a regular basis. These backups can cause performance problems on local networks and take up a large amount of dedicated disk and tape storage. System backups also require a lot of effort since administrators and operators must often manually initiate the system backups and mediate conflicts between them and the normal NetBackup backups. As with the installation applications sometimes used with the reinstall-and-restore method, these tools are platform-specific, requiring specialized skills and preventing the use of common tools and processes to lower costs and reduce human error. Out-of-Date Backups Because system backups are expensive in terms of storage, network usage and human effort, they are usually done less frequently than incremental NetBackup backups. Organizations often back up systems only once a week or even once a month. As a result, when restoring a machine from the system backup, that backup may be completely out of sync with the data on the NetBackup server. The problem is similar to that of the reinstall-and-restore method: There is no way to ensure a coherent recovery of the system when you are trying to patch together data that was backed up at different times. THE FAILURE OF THE CURRENT METHODS Here s an example illustrating the weaknesses of the current system-recovery practices. A data center takes mksysb system backups of an AIX server every weekend. This machine acts as a server for NIS, domain name system (DNS) and some applications. During a typical workweek, a system administrator did the following: Added two new users to NIS Added a new application server s IP address to DNS Installed a new application Changed the configuration of another application There was a huge storm on Friday, and the roof leaked water on the AIX server, damaging it beyond repair. The company eventually found a replacement machine with a similar configuration. The administrator restored the new machine from the original server s mksysb image, which brought its system configuration back to its state of the previous weekend. The administrator then installed the NetBackup client package and restored the server s applications and data from NetBackup, which she hoped would bring the machine back to its state as of Thursday night. 4
However, all did not go smoothly. Far from being back in business, the administrator now had to grapple with the following problems: 1. The newly created users home directories and files were restored successfully, but the users couldn t access them. The system didn t recognize the users because the NIS database regressed to last weekend s state. 2. Users were unable to communicate with the new application server, since the DNS configuration files no longer contained the server s IP address. 3. The newly installed application had its executables and data files restored from NetBackup, but the system s configuration database didn t know the application existed since it was installed after the last system backup. Worse still, the application s server process was no longer automatically started at boot time, since the entry it added to the /etc/inittab file at installation no longer existed. 4. The application whose configuration was changed regressed back to its old behavior since its configuration files were restored from the out-of-date mksysb. After many hours of puzzling over the system s behavior and fielding user complaints, the administrator finally pieced together enough of the missing files and configuration to get everything back to where it was on Thursday night. Of course, it took all weekend and part of the next week to get back to this state. And this company was lucky. First, it was able to find its most recent system backup not always a sure thing given the complexity of juggling two different backup methods and the inevitability of Murphy's Law, which predicts that everything that can go wrong, will go wrong. Second, the company was using an operating system that had a system backup solution. If it was using an operating system like Sun Solaris, the administrator would have had to reinstall the machine from scratch, and then restore data from NetBackup and try to make it all work together. Finally, the failed server wasn t being used for critical production purposes. If it was, the company could have been out of business by the time the server was recovered. VERITAS BARE METAL RESTORE As our example shows, even minor system outages can be extremely disruptive to the enterprise. Consider what would happen in the scenario if multiple servers required restoration at the same time. Few organizations have enough talented system administrators to pull off a recovery process like that. And even if they do, every minute counts in those situations, and the common system recovery practices squander precious time and provide little integrity in the process. VERITAS Bare Metal Restore was created to allow NetBackup customers to completely recover their machines from their normal backups, without requiring separate system backups or reinstalls. In the event that a client loses their boot disk or suffers some other catastrophic failure, Bare Metal Restore will restore the machine to the state at which it was last backed up to NetBackup. The recovery is swift and sure, with nothing left to chance or human error. BARE METAL RESTORE OVERVIEW Bare Metal Restore is integrated with NetBackup, providing a system recovery feature that all enterprises need. With Bare Metal Restore, your files are backed up normally to their NetBackup server(s). In addition to this scheduled backup, a Bare Metal Restore program automatically discovers and saves the state of the machine configuration. This information allows Bare Metal Restore to completely recover a machine using only the NetBackup backup thereby leveraging the investment made in enterprise backup solutions. Restoring a machine with Bare Metal Restore is easy and fully automated. The entire process consists of running one command on the Bare Metal Restore server and rebooting the client. The following sections describe the Bare Metal Restore components and the restoration process. Components of Bare Metal Restore Bare Metal Restore uses the existing NetBackup server(s) and adds a Bare Metal Restore main server component, one or more file server components and a Bare Metal Restore client component on every machine 5
that requires the bare metal restore capability. If you use Bare Metal Restore s network-boot capability then one or more boot server components are also necessary, as shown in Figure 1. The Bare Metal Restore server components may be located on the same physical machine; the exact configuration of servers depends on various factors, including Bare Metal Restore client platforms and network topography. The Bare Metal Restore main server manages the clients supported by the Bare Metal Restore system, the process of preparing for the client restoration, and the post-processing after the client(s) has been restored. It makes the appropriate boot image and file systems available to the client, ensures that the boot server and file server are properly configured and generates a customized client-boot procedure. The boot server provides the boot image to the client when it performs a network boot before restoration. Bare Metal Restore uses standard protocols to network-boot the client from the boot server. The file server provides the client with the programs, libraries and configuration data it needs to perform the recovery procedure. These include the necessary operating system commands and libraries, the Bare Metal Restore client package, the NetBackup client package and any other software necessary to recover the machine (e.g., VERITAS Volume Manager for restoring encapsulated root volumes on Solaris machines). The file systems provided by the file server are called Shared Resource Trees (SRTs); a Bare Metal Restore file server may contain one or more SRTs. In the network-boot configuration the file server makes the SRT available to the client over the network (using network file system for UNIX clients and SMB for Windows clients), while in the media-boot configuration it is used to burn a CD that contains the SRT. The Bare Metal Restore client is any machine being protected by Bare Metal Restore. The client continues to back up to its normal NetBackup server. To recover the machine using Bare Metal Restore, all of the client s files must be backed up regularly. Figure 1: Bare Metal Restore components in a network-boot configuration 6
The Initial Implementation Together, the Bare Metal Restore components provide an environment that allows the backup data stored in NetBackup to be used to completely restore the client without first having to install the operating system. The basic process for installing and configuring Bare Metal Restore into your environment is outlined below: 1. Install the Bare Metal Restore server software on the Bare Metal Restore main server. The Bare Metal Restore software is installed using the appropriate method for installing products for each platform, as shown below: o IBM AIX: installp o HP-UX: swinstall o Sun Solaris: pkgadd 2. Configure the Bare Metal Restore main server. 3. Install and configure the Bare Metal Restore file servers and boot servers (only required if utilizing network boot). 4. Install and configure the Bare Metal Restore clients. By default, the client configuration procedure creates a bpstart_notify command that automatically saves the client s configuration before each scheduled backup. If a bpstart_notify command already exists, then you must manually add the command that saves the client machine s Bare Metal Restore configuration, known as SaveConfig. 5. Create the SRTs for each operating system type and version. The SRTs are derived from the installation media of the operating systems and third-party applications. 6. Create the boot images for each operating system type and version. 7. Associate the Bare Metal Restore clients with the appropriate SRTs and boot images. The Bare Metal Restore main server is used to associate a client with a boot image and a SRT prior to restoration. This procedure can also be performed at restoration time, since the client does not have to be operational at the time. Daily Operations Once the Bare Metal Restore environment has been configured, the day-to-day operation is completely automated. Bare Metal Restore allows you to use your normal NetBackup backups to replace the disparate, costly and time-consuming point solutions that were previously required to support server restoration. Before each backup, NetBackup automatically runs the Bare Metal Restore SaveConfig procedure, which discovers the necessary system information required to rebuild your server to its exact current configuration. The information gathered by SaveConfig is saved on the client and is backed up during the NetBackup backup that immediately follows. In this way, Bare Metal Restore ensures that the client s configuration data is synchronized with the corresponding NetBackup backup. Elements of this information are also stored on the Bare Metal Restore main server as required. The Bare Metal Restore main server uses the data saved by SaveConfig during the restoration process for that particular client. The SaveConfig program logs its success or failure to the central Bare Metal Restore main server so an administrator can monitor the central Bare Metal Restore log to determine if all Bare Metal Restore clients have successfully saved their configurations. The Restoration Process The Bare Metal Restore recovery process is highly automated and efficient. Bare Metal Restore is a common solution that performs system restoration across the major enterprise-server platforms. Even though all platforms are administered using a single graphical or command line interface, Bare Metal Restore exploits each platform s capabilities to provide the most effective and efficient system recovery possible. The following are the steps to restore a UNIX Bare Metal Restore client using Bare Metal Restore network-boot capability: 1. The administrator issues a single command or uses the graphical user interface to tell the Bare Metal Restore server to prepare to restore the client. This initiates the following actions: 7
a. The Bare Metal Restore server retrieves the client s configuration data from the NetBackup server. This is the data created by the SaveConfig procedure during the client s normal backups (as described above in Daily Operations ). b. The Bare Metal Restore server creates a customized client-boot procedure and makes the appropriate boot image and SRT available to the client. c. 2. The client boots from the Bare Metal Restore -designated boot server and starts running its customized boot procedure. This procedure performs the following tasks: a. The client mounts the necessary SRT file systems from the file server. b. The client configures its disks, volume groups, logical volumes and file systems, re-encapsulates the root volumes and performs any other tasks necessary to restore its configuration to the state of its most recent backup. c. The client uses the standard NetBackup client to restore all its files from the NetBackup server, including the operating system, applications, configuration data and user files. d. The client configures its boot record and configuration database. e. The client reboots itself. f. The client performs any required post-boot cleanup, including de-allocating the Bare Metal Restore resources allocated to it. After this process is complete, the client machine has been fully restored to the state at which it was last backed up. Bare Metal Restore also supports a media-boot restoration process, which eliminates the need for the boot and file servers. Instead, a CD can be burned from a file server at any time, including in advance of the client restoration. For UNIX Bare Metal Restore clients, this CD takes the place of both the boot server and file server, providing the boot image and the SRT. Windows NT Bare Metal Restore clients always boot using a single floppy disk, and then use either a file server or the media-boot CD to provide the SRT. Other than this distinction, the restoration procedure is identical to the UNIX case described above. In addition, the Dissimilar Disk Support feature allows restoration to a machine with a different disk configuration than the client had at the time of its last backup. At the time that the administrator prepares the client for restoration, it is possible to specify which logical volumes and/or file systems to restore, and to map the previous disk configuration to the current one, even if the number and size of disks have changed. BARE METAL RESTORE SPEEDS EFFECTIVE AND EFFICIENT SYSTEM RECOVERY The entire Bare Metal Restore process can be done in minutes. No manual intervention is required, other than the initial boot, so many machines can be recovered by a small number of administrators and/or operators. Since the majority of the time is spent restoring the client s data from NetBackup, the time required for the restoration is largely determined by network speed, NetBackup server performance, tape access times and other environmental factors. With proper design of the network and NetBackup server configuration, Bare Metal Restore can scale up to completely restore very large sites in one or two days. A large-site recovery could otherwise take many days and administrators, and there would be little chance of properly recovering all the machines. Bare Metal Restore imposes no requirements for additional network bandwidth beyond normal backup requirements. Bare Metal Restore relies on the normal NetBackup incremental backups, as opposed to system backups that take a snapshot of the entire machine (or at least its boot disk) on a regular basis. 8
For the same reason, Bare Metal Restore also imposes little or no additional storage requirements. If the entire machine was not backed up before Bare Metal Restore installation then more storage will be required to back up the additional files, but these typically represent a small percentage of the total machine storage. Perhaps most important, Bare Metal Restore eliminates the need to manage multiple backup and restore methods. With Bare Metal Restore there is no need to perform redundant system backups or maintain client configuration definitions. As long as the normal NetBackup backups are taken, any Bare Metal Restore client can be completely recovered without additional effort, which results in a substantial savings of an administrator s time. SUMMARY Bare Metal Restore provides the key element in extending NetBackup to provide complete system recovery. By allowing any machine to be completely recovered from only its NetBackup backup, Bare Metal Restore eliminates redundant network and storage usage, saves labor and provides users with unsurpassed confidence in their system s recovery capabilities. 9
VERITAS Software Corporation Corporate Headquarters 350 Ellis Street Mountain View, CA 94043 650-527-8000 or 866-837-4827 For additional information about VERITAS Software, its products, or the location of an office near you, please call our corporate headquarters or visit our Web site at www.veritas.com. 10