Protecting your Data Protecting your data is a critical necessity of having your DemandBridge Software and data programs loaded on a computer that has the ability to integrate redundant devices such as hard disk, power supplies, etc. In this day and time of cheaper, faster, and more reliable computer hardware it only makes good business sense to make the right choice when upgrading your server equipment. If it becomes your job to recommend a new server to replace your existing machine there are several features that need your attention to make your business run 24/7 and make your life less stressful. Choose a vendor that has nationwide support Ask our technical support staff for suggestions Choose a server that will be sufficient for your companies growth 5 to 10 years This is when a server versus a workstation is desirable because it has the capability to have: Hot swappable hard drives Hard disk RAID Redundant power supplies More space for expansion Better cooling capacity Designed for 24x7 operation Mother board design accommodates built-in SCSI controllers and NICs Mother board bus has greater speed and design The cost of a server versus a workstation will be more, but in the long run the money is well spent. Time saved in loss of business and frustration will far offset the extra expense. What is RAID? RAID stands for Redundant Array of Inexpensive (or sometimes "Independent") Disks. RAID is a method of creating one or more pools of data storage space from several hard drives. It can offer fault tolerance and higher throughput levels than a single hard drive or group of independent hard drives. The Benefits of RAID Reliability Provides real-time data recovery with uninterrupted access when a hard drive fails Increases system uptime and network availability Protects against data loss Performance Multiple drives working in parallel increase system performance (RAID 5)
RAID Is it Software or Hardware? Software Software based RAID is included in certain operating systems such as Windows NT, IBM AIX, and Linux. All Raid functions are handled by the host CPU which a severely tax its ability to perform other computations. If cost is a factor this would be the choice. Hardware RAID requires additional controller with its own processor and cache memory. This card functions independent of the hosts CPU. This would not degrade but enhance system performance and has no limitation in regards to the operating system. This would be the preferred choice since it has a more robust fault-tolerant features and increased performance versus a software RAID. RAID Types RAID level 1 and RAID level 5 will be covered since they are the most popular in the server configuration. RAID 1 RAID 1 requires 2 hard drives of the same capacity; this is commonly known as a disk mirror. If there are 2 18GB hard drives installed and configured as RAID 1 there would be a total disk capacity of 18GB since the 2 nd drive would be constantly updated on every write and be an exact duplicate of drive one. If either hard drive failed the other drive would take over and the system would perform with no interruption. With a hardware RAID controller and a hot swap drive bay the system administrator would replace the failed drive and a rebuild would occur automatically with no system down time. Raid 1 will impact some system performance simply because it takes additional time to write data to 2 drives. When reading data the controller will decide which of the two drives a read can perform faster and read from that drive. RAID 5 RAID 5 is the most common configuration because it provides good overall performance and data protections with a minimum loss of storage capacity. RAID 5 requires a minimum of 3 hard drives of the same capacity and the capacity of 1 drive will be reserved to store redundant data. In this case, suppose there are three 18GB hard drives in the disk array, the total usable space would be 36GB. This level is commonly referred to as striping with distributed parity. RAID l 5 distributes parity among the drives. No single disk is devoted to parity. RAID 5 will out perform RAID 1 for read operations but write performance will be slower than RAID 1. In a RAID 5 there is no data loss or system interruption due to disk failure, because if one disk fails, data can be rebuilt. 2
Redundant Power Supply When choosing a new server you can add even more reliability and less down time by selecting a server that will accommodate and additional power supply. The basis for this is simple, if the main power supply fails the system will notify with a audible alert and the second power supply will handle the load until the primary supply and be replaced. Uninterruptible Power Supply or more commonly known as a UPS is a must for the server. Choose one that is capable to handle the server load for at least 20-30 minutes or more so that an orderly shutdown of the system can be performed in the event of complete A/C power loss. Data Backup Don t let the fact that your server is equipped with a fault tolerant RAID sub-system lead you into a fault sense of security. While the data and operating system are preserved for the moment it does not help if someone accidentally removes files from your server by mistake or you loose the server due to fire or water damage. One of the most vital functions as the system administrator is to manage tape backup policies. The backup routine should be setup on the system to be done everyday automatically. The only operator intervention required is to insert the daily backup tape and monitor root mail on a daily basis. The principals of data backup are fundamentally the same for all server platforms, Frequency How many tapes When the backup starts Tape storage Data verification System bootable tapes Since DemandBridge runs on a variety of operating systems like IBM AIX, SCO Unix, Red Hat Linux And Windows Server 2000, the tape backup scripts, programs and device names will be different. The majority of DemandBridge application servers are presently using UNIX, so this will be focused on UNIX cpio command and the use of a third party tape backup program Lone-Tar. Frequency DemandBridge Programs and Data Files Tape for every business day Settings should be automatic Backup starts after midnight Tuesday through Saturday How Many Copies of Each? 2 weekly backup sets Rotate sets weekly Note: Remember tapes do wear out; make provisions to rotate new tapes in the rotation every 12 months or earlier. Tapes are cheap compared to not having a reliable backup 3
When does the backup start? All of the supported UNIX operating systems have a cron table that schedule tasks to be executed at given times of the day, week and monthly. This is where the backup scripts get scheduled to run. If you don t know when and how often your system is backing up, this is the way to find out. This command will allow you to look at the crontab only. crontab l The following is an example of the crontab file: # (C) COPYRIGHT International Business Machines Corp. 1989,1994 # All Rights Reserved # Licensed Materials - Property of IBM # # US Government Users Restricted Rights - Use, duplication or # disclosure restricted by GSA ADP Schedule Contract with IBM Corp. # #0 3 * * * /usr/sbin/skulker #45 2 * * 0 /usr/lib/spell/compress #45 23 * * * ulimit 5000; /usr/lib/smdemon.cleanu > /dev/null 0 11 * * * /usr/bin/errclear -d S,O 30 0 12 * * * /usr/bin/errclear -d H 90 0 * * * * /usr/lpp/servdir/servdir.timer 1>/dev/null 2>/dev/null 0 3 * * 2-6 /usr/bin/tapeout.sys 0 4 * * 1 /etc/reboot This states that a complete system backup is scheduled to run Tuesday through Saturday at 3 A.M. in the morning. As the system administrator with root privilege you can alter the schedule if necessary. This requires the use of the vi editor. Tape Storage Store Offsite 1 set of daily backup tapes 1 complete system backup 1 bootable backup (this could be floppy diskettes or a mksysb tape) A good location for off site storage would be a trusted employee, business owner, etc. that lives closest to the office. While safety deposit boxes are secure, they are not always available 4
Backup Verification UNIX, AIX and Linux If the backup was performed from a backup script such as tapeout.sys, tapeout.tf which would be done via a crontab entry there is a script in /usr/bin called tapeout.ck To run this script, login as root cd /usr/bin./tapeout.ck This script will list the contents of the tape to the screen. Listing the contents of the tape will take the same amount of time as it did to create the tape so be prepared to see files scroll across the screen for the next hour or so. To cancel the screen listing: SCO Unix system: use the delete key IBM AIX system: Ctrl c Read root mail The system administrator should read root mail daily. The system will mail a message daily to root from cron and it should be time stamped at the time the script runs every night. If it encountered errors there will be some detailed information included. After the mail is read it should be deleted. All unread root mail will remain until it is deleted, over time this could fill up a file system. Reading root mail is a daily event. DemandBridge Application & Data Backup Using Unix cpio cpio is a command that can be found on all flavors of UNIX. If the server does not have a 3rd party backup software installed then tape backup will be accomplished with scripts that use the cpio command. The following script will be used to backup DemandBridge programs and data files. /usr/bin/tapeout.tf Below are the commands used in the tapeout.tf script. There can be differences in the tapeout.tf script, for instance the tape device in AIX would be rmt0 instead of rstp0. umask 0 cd /usr/lib/pvx find. print cpio ovcb >/dev/rstp0 UNIX, AIX and Linux Systems DemandBridge using Thoroughbred Basic would backup everything in /usr/lib/basic DemandBridge using ProvideX would backup everything in /usr/lib/pvx For systems that were converted from Basic to ProvideX the DemandBridge data directory may be found in /usr/lib/basic/pvx/tf2d 5
Complete System Backup Backup of the entire system is suggested. With the large capacity tapes and the ever-increasing processor speeds of computer servers a full system backup is the best solution. If tape capacity is an issue, then a backup of programs and data files should be done daily and a complete system backup should be done monthly. While it is possible to span multiple tapes when doing a backup, the restore of data could be tricky and therefore not recommended. On Unix based servers the cpio script for a full system backup is: /usr/bin/tapeout.sys The tapeout.sys script will start from the root directory and backup the entire hard disk(s). Verify the crontab entry if you are unsure as to which backup routine it calls. Bootable System Backup Bootable system backups can be created in numerous ways depending on the Operating System and tape backup programs. SCO and Red Hat Linux Lone-Tar is a tape backup program that has utilities for creating a set of emergency boot diskettes. This can be performed using a menu system that guides the user through a series of questions about the system. As a rule DemandBridge Software will create these disks when a new server is purchased and configured. It is up to the end user to keep these diskettes current as new hardware and system changes are made. These diskettes must be kept in a secure place along with a master backup tape. For systems with large kernels it may not be possible to create bootable diskettes due to the limitation of the diskette capacity. Lone-Tar has the option of creating an image file on the server which can be transferred to a workstation that has a CD-ROM burner. This image file can be burned to a CD-ROM which can be used to boot the server to a Rescue Ranger menu which can be used to restore the last master backup tape. 6
IBM AIX AIX mksysb (system image) Option #1 (non-production) The mksysb tape is a life saver for the company in the event of a catastrophic server failure. This tape should only be used by the system administrator who is under the direction of either IBM or DemandBridge support. This is a must tape and the administrator should know the status and whereabouts of this tape at all times. The following is a procedure for making a mksysb tape in a non-production state, that is no users are logged in and webec servers are down. (Saturday or Sunday) login as root Verify that all users are logged off (who) Stop webec servers umount /usr/lib/basic (do this from the / directory) or umount /usr/lib/pvx (do this from the / directory) mkszfile && mksysb /dev/rmt0 When backup is complete mount /usr/lib/basic or mount /usr/lib/pvx restart webec servers Note: A mksysb only backs up data and system programs in the root volume group. Your system may have multiple volume groups. To display all volume groups, enter: lsvg A list of volume groups will be listed: rootvg datavg specialvg On this system there are 2 additional volume groups and the data in these volume groups will not be backed up with a mksysb. The data in these volume groups should be backed up on a daily basis AIX mksysb (system image) Option #2 (during production) This method will allow an mksysb tape to be created while users are logged in and the system is in a production or multi-user mode. A file will have to be created with the directory or directories you wish to exclude during this process. login as root cd /etc and create a file called exclude.rootvg Add these lines: #exclude these directories when doing a mksysb tape /usr/lib/basic # (tbred basic) or /usr/lib/pvx #(Providex) save and exit the editor smit mksysb (this is the fast path to the system backup menu) 7
Back Up This System to Tape/File Type or select values in entry fields. Press Enter AFTER making all desired changes. WARNING: Execution of the mksysb command will result in the loss of all material previously stored on the selected output medium. This command backs up only rootvg volume group. * Backup DEVICE or FILE [/dev/rmt0] Create MAP files? no * EXCLUDE files? yes **(toggle this from no to yes) List files as they are backed up? no Generate new /image.data file? yes EXPAND /tmp if needed? no Disable software packing of backup? no Things to consider If it takes 1 hour to create a mksysb tape It will take 1 hour + to restore your system This does not include /usr/lib/basic or /usr/lib/pvx mksysb will only backup data in rootvg 3 rd Party Backup Software In the past 3 or more years DemandBridge has installed 3rd party backup software on all new and upgraded UNIX servers running SCO OpenServer or Red Hat Linux. This package is easy to install and has menu levels that allow the system administrator to restore files and schedule backups without having to know the particulars of the cpio command and the vi editor. Another plus to this package is the ability to create the bootable diskettes that would allow a complete system restore in the event that there was a catastrophic disk or hardware failure. This package cost between $450 to $500 and takes about 30 minutes to install and configure. 8
Lone-Tar Menu Disk Management Get into the habit of checking free space on the server s hard disk. Check at least once a week, keep all file systems less than 85% and pay special attention to file systems allocated for DemandBridge data. UNIX Directories /usr/lib/pvx DemandBridge programs using ProvideX tf2d DemandBridge data directory / root filesystem Typical display of SCO Unix disk usage # dfspace / : Disk space: 1399.46 MB of 1999.99 MB available (69.97%). /stand : Disk space: 6.17 MB of 14.99 MB available (41.15%). /usr/lib/pvx : Disk space: 13993.58 MB of 14816.60 MB available (94.45%). Total Disk Space: 15399.22 MB of 16831.60 MB available (91.49%). Typical display of IBM AIX disk usage $ df Filesystem 512-blocks Free %Used Iused %Iused Mounted on /dev/hd4 32768 16360 51% 1023 13% / /dev/hd2 3325952 357112 90% 24503 6% /usr /dev/hd9var 376832 247824 35% 409 1% /var /dev/hd3 376832 155832 59% 906 2% /tmp /dev/hd1 1507328 941648 38% 7484 4% /home /dev/lv02 16056320 9335704 42% 76095 1% /usr/lib/basic 9
Display of Red Hat Linux disk usage $df filesystem 1k-blocks Used Available Use% Mounted on /dev/sda8 4032092 976644 2850624 26% / /dev/sda2 101107 5829 90057 6% /boot /dev/sda7 20161172 4789772 14347260 25% /tfconv /dev/sda5 24193508 10435716 12528824 45% /usr/lib/basic /dev/sda6 20477104 2182276 17254624 11% /usr/lib/pvx As you can see the file system layout varies depending on the operating system. In AIX there are separate file systems for /usr, /home, /tmp and /var, these are created on system install with default sizes. AIX allows the expansion of file system size on the fly. In SCO Unix and Red Hat Linux, the file systems are defined during the install process and cannot be changed. If DemandBridge configures the server there will be a separate file system for DemandBridge programs and data (/usr/lib/pvx). If you are having a new server configured by a party other than DemandBridge have them call the technical services department for this guideline. When creating new file systems, rule #1 allow plenty of room, because once the file system has been created it cannot be reduced or expanded. The only exception is that IBM AIX allows the expansion of a file system if there is sufficient room in the volume group. NOTE: For AIX systems Never allocate all of the space in the root volume group. This would prevent the addition of additional swap space if necessary it can also prevent the addition of jfs logs, etc. Run the df -v and be familiar with the output. There are system logs, Web.ec start logs, smit logs, etc that can grow over a period of time that will consume disk space. These vary depending on the Operating system. The best policy is to observe disk usage on a weekly basis. If you see a file system increasing in space within a weeks time, this might be an indicator of a problem in regards to the print queuing system, system logs or the impact of a virus that attacks samba shares and creates files with extensions like.txt or.eml or.nws until it runs out of space. 10