Methods of Linux Bare Metal Restore and Disaster Recovery www.storix.net 1-877-786-7491
STORIX WHITE PAPER 2 We don t think much about health insurance when we re feeling well, but even the least prudent planner wouldn t neglect such a necessity. As most of us know, problems occur when you least expect them. It is not a matter of if, it is a matter of when. A computer illness can sometimes be easily cured, while others are more serious. Think of disaster recovery software as more of an insurance policy-one that goes beyond recovering lost data. Most companies think they are prepared for disaster by backing up critical data, which they can quickly restore if a loss occurs. Few companies, however, are prepared for a catastrophe-total system failure-in which they lose all their data as well as the entire operating system, applications, settings, patches and more, reducing their system to bare metal. The META Group stated that, in 2002, system downtime cost American businesses an average of $1 million an hour. Unlike most UNIX flavors, many Linux distributions do not include disaster recovery (DR) tools. As a result, Linux users must often go back to their original distribution media and start building from scratch. Some third parties have developed disaster recovery utilities that have ways of recovering an entire Linux system. In the following discussion, two methods of system disaster recovery are examined: disk image recovery and file-level recovery using Storix System Backup Administrator (SBAdmin).
STORIX WHITE PAPER 3 The cost of downtime Rebuilding even a single system from scratch can take a few hours or several days time in which management, employees and customers have no data access. As a result there are no sales, no customer service, no product shipments, no supply chain management, no call centers... no revenue. How much does a single hour of downtime cost? The META Group stated that, in 2002, system downtime cost American businesses an average of $1 million an hour. A survey done by Contingency Planning Research and published in the April 3, 2000 issue of InternetWeek details hourly downtime losses for several industries: Industry Cost per hour Brokerage operations $6,450,000 Credit card authorization $2,600,000 Package shipping services $150,000 Catalog sales center $90,000 Airline reservation center $89,000 According to an Infonetics study on the effects of downtime, published the June 15, 2000 issue of CIO Magazine: 42% of companies lose $1,000 an hour 26% lose $10,000 an hour 5% lose up to $50,000 an hour 4% lose over $50,000 an hour 23% is unknown In today s 24x7 world, even if downtime is only $1,000 an hour, a three day shutdown would cost a company $72,000. At $10,000 an hour it would be $720,000. Few companies can survive such a hit. In fact, a chilling study from Contingency Planning and Management magazine discovered that 40% of companies that were shut down for three days failed within the following 36 months.
STORIX WHITE PAPER 4 Starting over Recovering from a bare metal disaster is a highly complex operation that can take hours, days or even weeks to return the system to its prior state. To understand the enormity of rebuilding an entire system from scratch, consider what is known (and not known) before restoration can commence. For example, does the company have: A list of user accounts and passwords handy? The names, types and attributes of all filesystems and how large they must be created to restore the data? Details of how all printers and display temrinals are configured? List of all network settings? Instructions for reinstalling all third-party applications? Assuming the administrator knows everything about a system at the time it crashed, he must still complete an arduous, step-by-step process to prepare the system before the actual application data can be restored. The most common tasks include: Repartition hard drives Recreate volume groups and logical volumes Remake software RAID devices Remake file systems Reinstall the Linux OS from the original distribution media Recreate users and groups Reinstall hardware drivers Reinstall additional packages and third party software Reinstall all software updates Reconfigure the network and firewall Redefine and reconfigure printers and other peripherals Reinstall and reconfigure web, FTP and email servers Reinstall backup software Reinstall database software
STORIX WHITE PAPER 5 Linux system recovery tools? Some UNIX administrators, who routinely use recovery tools like AIX mksysb and HP/UX make_recovery, are reluctant to move to Linux because that OS has no similar standard tools for full-system recovery. In addition, most Linux distributions are fairly unsophisticated about how they install themselves. In most instances they use a single hard disk that is partitioned with a single root filesystem containing all system files as well as other applications and user data. In more sophisticated configurations, an administrator might take advantage of performance, flexibility and availability benefits by spending considerable time reorganizing the system to use LVM, software RAID, enhanced filesystem types and other advanced techniques. Unfortunately, all this time-consuming and costly hard work, which may evolve over several years, must be repeated if the system crashes or new hardware arrives. Fortunately, third party vendors are making tools available for Linux systems to ease this pain. Often referred to as system recovery, disaster recovery (DR) or bare-metal restore tools, they are capable of restoring a typical Linux system to its previous hardware configuration. Currently there are two methods of accomplishing this: disk image recovery and file level recovery. Each has its own benefits and pitfalls. Disk image recovery A disk image backup program views the hard drive simply as a group of sectors. It backs up all sectors (not individual files) on the hard drive containing the data and creates an exact image. By operating on the disk sector level, the program backs up everything-system files, master boot record (MBA), partition tables and all user data-with no regard to their content or structure. Some disk image backup tools, which are a bit more sophisticated, back the partition table information, then back up each partition independently as if it were a disk image. This adds a bit more flexibility by allowing IT to restore a single partition instead of the entire disk. A disk image restore begins by booting the system from media (typically a CD-ROM) that includes a simple OS, device drivers for accessing the disks and backup media, and a utility for rewriting the data. If the entire disk was backed up, then the entire disk s data is restored. If individual partitions were backed up, then the disk partition table is first re-created, then the data is restored to each partition. While in some instances this can provide the fastest means of backing up and restoring data, the method s inflexibility may cause considerable grief during a system restore. A disk image backup is like a block of ice that will only fit properly into its original container. If the container has changed (such as a different disk size, type or location), the data will no longer fit, and the entire backup is rendered unusable.
STORIX WHITE PAPER 6 DISK IMAGE BACKUP/RECOVERY Pros Fast backup if most of the sectors on the disk contain real data Simple, non-os-specific backup and restore process Cons Backup will take longer backing up sectors of the disk not used by real data Backup cannot be restored to different disk types, sizes or locations Cannot restore partial disk backup no selective file restore Restored data files retain same fragmentation as before the backup File-level system recovery In contrast to a disk image backup, a few backup tools and products are available that better understand the operating system they are running on. These tools generally record the system, disk and filesystem configuration, then use this information to rebuild the system from the ground up. The backup usually consists of a file-by-file backup of either the entire system or of each individual filesystem. Borrowing the previous analogy, if a disk image backup is like a block of ice, a file-level backup is like water; it can be shaped to fit into any container. As a result, disks, partitions, filesystems and other storage tools can be reconfigured for the new hardware, then data files can be simply restored onto the new configuration. It s like pouring water into a different glass; the data doesn t care about the storage configuration it is restored onto. This process of altering, then building a new storage configuration can be a sophisticated task in which most DR products show their limitations. Though most will allow alterations of disk partition tables, they fail to understand more advanced and increasingly common disk configuration tools such as the Logical Volume Manager (LVM) or Software RAID (meta-disks). These logical disks must also be altered to match the new hard disk configuration before the data can be restored. If the DR software is not capable of understanding and guiding the user through the process, it forces the user to manually alter and build the configuration, usually through command-line utilities and manual editing of configuration files. If all is well, the data files are restored and the system disk is made bootable once again. The new system, even with a new disk configuration, should operate the same (and often better) than the original.
STORIX WHITE PAPER 7 FILE-LEVEL SYSTEM BACKUP/RECOVERY Pros Faster backup when disk or filesystems contain free space Backup can be restored to different disk types, sizes and locations Faster system I/O after a recovery since all files are de-fragmented Cons Can take longer to read individual files from a full filesystem than image backup Depending on the utility, reconfiguration to new hardware may be complex. Takes longer to restore files and directories Selective restore of individual files and directories without complete recovery The best of both: Storix System Backup Administrator Storix, Inc. provides a daily backup management product with many features for both standalone systems and networked systems that some other backup products provide. The system backup and recovery features, however, set it aside from the rest. System Backup Administrator (SBAdmin) performs a file-level system backup capable of being restored onto different hardware while supporting all popular Linux storage configuration tools (e.g., LVM and Software Raid). SBAdmin backs up and restores virtually any Linux distribution regardless of disk complexity and I/O configuration. It fully understands both the original and new devices and adapters, and it automatically reconfigures the operating system to support them. The SBAdmin system installation process makes it easy for even the most inexperienced Linux administrator to reconfigure the system to match new hardware. The easy-to-use menu interface warns if the backup cannot be installed onto the new hardware without change. It then guides the administrator through the process of altering the configuration as needed, simplifying such complex tasks such as: Adding new disks to the configuration Adding or changing disk partition tables Adding disks or partitions (or software RAID devices) to volume groups Changing sizes, locations and striping of logical volumes Changing sizes, locations, striping and mirroring of software RAID devices
STORIX WHITE PAPER 8 Since many systems contain raw data that is not in a filesystem, SBAdmin will optionally backup any partition, logical volume or Software RAID device that does not contain filesystem data (or if filesystem is not mounted). In such an instance data is backed up and restored to the entire device, just as with other disk image backup tools. Since a Storix system backup is a file-level backup, it can also be used to restore individual files, filesystems or directories from the backup. And because the backup may also contain raw device data, the software can restore data to individual partitions, logical volumes or software RAID devices. Any selective data restore can be performed from the easy-to-use GUI interface. This eliminates the need to have separate backups for full system recovery and restoration of user files. Since SBAdmin recreates and restores the system from the ground up, there is virtually no fragmentation when the restore is completed. This often results in a considerable increase in I/O performance, even when no changes are made to the configuration. In addition, SBAdmin doesn t limit restoring a backup to the same hardware. Users can easily restore ( clone ) a single backup to different systems, even if the disks, adapters and other devices differ. Because SBAdmin allows reconfiguration to match new or different hardware, the same process can be used to configure the system for better performance and availability, even when no hardware change has occurred. This includes such tasks as: Converting a single root filesystem to multiple filesystems Resizing filesystems to add or free unused space Converting from partition-based file systems to LVM or Software RAID Extending or striping filesystems across disks Implementing RAID 1+0 (mirroring+striping) using both RAID and LVM Creating new volume groups and moving logical volumes between VGs The SBAdmin system installation process makes it easy for even the most inexperienced Linux administrator to reconfigure the system to match new hardware. Once the system is configured, the next SBAdmin System Backup performed will ensure the entire configuration is recorded and can be restored the next time with no user interaction.
STORIX WHITE PAPER 9 Conclusion When a system is down, regardless of whether it s a single disk, server or an entire IT center, business comes to a halt. Workers sit idle, sales aren t made, orders aren t filled, customers can t access the company website. Using a disk image restore program is a fast, easy method for bare metal recovery if everything is restored exactly as the original system was before it went down. However, if there are any unexpected changes (such as disk or system alterations), the backup may be rendered useless. If there is no guarantee that a system replacement will exactly match the original, a system backup such as Storix System Backup Administrator is a better solution. Regardless of the product selected, users must be absolutely sure it will recognize, support and rebuild all of the storage options that have been implemented. Whenever possible, a disaster recovery test should be fully performed. Businesses shouldn t wait until disaster strikes to find out whether or not they made the right choice. Storix, Inc. 7801 Mission Center Ct Suite 320 San Diego, CA 92108 619-543-0200 sba@storix.com www.storix.com