A Backup Strategy for Informatics Craig Strachan Version /01/2011

Transcription

1 A Backup Strategy for Informatics Craig Strachan Version /01/2011 This document identifies the various types of backup being carried out in the School of Informatics, briefly considers the characteristics of each type of backup, considers whether it is essential that the School carried out these activities and suggests a policy for each type based on the School's requirements and the need to reduce costs wherever possible. Facts and Figures the amount of storage currently being regularly backed up by the school is as follows: Type of Space Total backed up (TB) AFS user space 3.3 AFS group space 8.4 Other group/mirror space Media 1.5 Total 13.2 There are various media options available for the storage of backup data. These can be broadly characterised as disk, tape and optical media. A rough comparison of costs of each of these (using examples of each type already in use in the School) looks like this: Medium Cost per TB ( ) Notes FibreChannel Disk Array x 1TB disk, RAID 5 Single SATA Disk 65 1TB no enclosure LTO4 Tape 31 Blu-ray Disk 80 Assuming a unit cost of 2/disk For the purposes of this document, we consider three different reasons for performing backups : Disaster Recovery - the ability to quickly restore to service all of the data being provided by an entire disk array, file server or in extremis an entire site in the event of hardware failure or other unexpected event making this data unavailable. User Support - the ability to restore to users within a reasonable time frame relatively small amounts of data which have become unavailable through user error or other circumstances Archiving - The ability to retain access to data which has been removed from the main commodity file servers.

2 Though there are overlaps between these categories user data can be recovered (in the short term at least) from disaster recovery backups and User Support backups can act as an inferior kind of archive, their characteristics are essentially different and the methods used should be appropriate. Disaster Recovery When we talk about a disaster, we are encompassing everything from the failure of a single disk drive to the loss (temporarily or permanently) of an entire site and all the data served from it. Backups made for the purposes of disaster recovery have three essential characteristics. Age - The value of the data restored falls off with age far more steeply than for any other kind of backup. Accessibility - Data should be immediately accessible and restoration of data should be as streamlined as possible. Location - Backup copies should be geographically separated from the original data. Medium These requirements suggest that disk is by far the best medium for these sorts of backups. The data is immediately available with no need to swap tapes between drives (or worse still fetch tapes from fire safes) and as the data has a very short retention period, the amount of storage required (and hence the expense) is not excessive. Current Practise All data which cannot be easily recreated from alternative sources is mirrored daily to disk space located at a remote site (KB for data stored at IF and AT, IF for data stored at KB. For AFS data, the built in AFS mirroring mechanism is used. This has two advantages, each user has direct access to their own mirrored data allowing them to recover data lost since the last mirror was taken and should it become necessary to promote mirrored data to live data, this can be done with a single command. Non-AFS data is mirrored using the rsync based mechanisms developed within the School. Recommendations In general, the current mechanisms the School has in place for disaster recovery backups seem sufficient and appropriate and have already been put to the test several times. The intention is to move all live data to the central area in the near future with all DR backups being performed at KB. Work is currently under way to improve the reliability and accountability of the non-afs backup mechanism and this should be completed by the end of It may be that for some classes of date, that of the finance area for instance, it would be appropriate to mirror the data more frequently. This means that there is less likelihood of recently created data being lost in the event of a disaster but it also means that the window of opportunity for users to recover their own data is diminished. An ideal compromise would be to retain more than one generation of mirrored data but this is not possible with AFS as it stands though it may become so in the future. Currently, when mirror AFS data is promoted, this is done by hand using individual commands or shell scripts hastily crafted on the spot. It would be well worth investing a little effort (I would estimate less than a week) in producing scripts which would carry this work out automatically. This

3 may be done as part of devproj 135 Further AFS development and automation. User Support User support backups provide the ability to recover, usually at the request of a user, relatively small amounts of data which have become unavailable through user error or data corruption. Characteristics To minimise the cost of restoration, the data must be as accessible as possible. Ideally, the backed up data should be accessible by the user without intervention by computing staff. There should be a wide range of data retained. The version of the data required by the user will not necessarily be the last backed up. Balanced against this must be the cost of performing these backups which will increase proportionally to the length of time the data is retained. If data is retained over a long period, we may also hit issues of hardware compatibility which are covered in more detail in the archive section. In theory, we do not have to carry out this type of backup at all since it is purely a user server and one which can be expensive both in monitory terms and in computing staff effort. However I believe that most members of the School have an expectation that this service will be available to them and that any attempt to withdraw this service would be very poorly received. Medium The desire to have this data available to users without intervention by computing staff means that in many ways the ideal medium for this kind of backup is once again disk. The need to retain and organise large amounts of data over a long period is better suited to a tape based mechanism. Ultimately, the best solution is probably one in which the most up to date data is available on-line with older data being stored on tape and retrieved by computing staff. The need to identify and locate different versions of data means that some kind of backup software which maintains a searchable database of the location (tape or disk) of data versions must be used. Current Practice User support backups are currently carried out using the True Incremental Backup System from Teradactyl Software. We have recently purchased an Overland Neo 8000 tape library with 6 LTO-4 drives and 500 slots to support this software. Our Tibs backups go back to the beginning of Until June of 2010, backups were carried out on LTO-2 tapes. This has implications for the recycling of tapes since LTO-4 drives cannot write to LTO-2 tapes (though they can read them). A full dump is performed every 28 days with incremental dumps being performed daily. The latest incremental dump is held in a disk cache as well as being written to tape. Recommendations Analysis of restoration requests in the RT database produces the following results:

4 Period Number of requests Last Day 30 Last Week 37 Last Month 12 Last 6 months 11 Last Year 4 Older than 1 year 5 Note that requests in the Older than 1 year category were in most cases for data between 5 and 10 years old. At our current levels of data usage, we are using roughly 12 LTO-4 tapes per month for backup, a figure which corresponds to a compression ratio of 1.375:1. Since an LTO-4 tape currently costs roughly 25, this means that the media cost for a monthly backup is 300 (all figures exclude VAT) and for a year's worth of backups (144 tapes) Similarly, since the move to LTO-4, we have used roughly 5.5 tapes per month for incremental backups. Looking at the above restoration figures, it is clear keeping a year's worth of backups would allow us to satisfy roughly 95% of all restoration requests. My view is that we should retain full backup tapes for 1 year and incremental tapes (which allow for the restoration of data which may have been created and deleted between two full backups) for 6 months. Any data which users require to be retained for longer periods should be treated as archive data. Archiving Archiving of data, when it has been done at all, has traditionally been performed by removing a month's set of tapes from the standard backup tape reuse cycle and retaining them indefinitely. There are many reasons why this is no longer a viable approach to take to archiving, some of which are listed below: Using this method, recovery of the archived data is dependent on still having the software used to create the backup in the first place available. Hardware changes, OS changes or the demise of the company providing the software may all mean that this may not be possible. This problem can be alleviated by making use of open source software or by using backup software which does not use a proprietary format to store data. Amanda is an example of backup software which meets both of these criteria. The hardware needed to read the medium on which the data is stored may no longer be available. Regularly copying of archive data to more modern media can address this problem but is very expensive in terms of effort. Even if the hardware and software still exist to allow access to the archived data, it may still be difficult to actually locate the data as the link between its location on the file system as it appears to the user and its actual location on disk (and hence in the backups) may well have

5 been lost. For example, the recovery of /group/project/archive/foo may be requested but unless the amd map linking /group/project/archive/foo to a given partition on a given server has also been retained, the recovery of the data may entail searching the entire archive. There are other problems with archiving which are not directly related to how the archives are made: Increasingly, much of the date we may wish to archive no longer exists as plain files. For example, many School websites now are stored in a WCMS's database. Even if the database itself can be archived in some format, we are still reliant on having a working version of the application to retrieve the data from the database. There are many different factors controlling how long the data should be retained for. Legal constraints, contractual obligations and University policy may all have a bearing on how long data should be retained for (and indeed when data must be deleted). Since different constraints will apply for different types of data, it is impossible to come up with one retention period which will satisfy all requirements. It is obviously impractical when using a serial medium such as tape to delete some data in a backup whilst retaining the rest. Given all of the above, I believe that it is now impractical for the School in isolation to meet its archiving requirements. Though certain items of data (for example home directories of ex-users) may still be archived internally, the School should for the most part act instead as a source of advice and assistance to data owners on the most appropriate methods of arranging for the external archiving of their data. Such an approach requires that each item of data in the School's file system has a clearly defined ownership (including a budget to which costs could be charged if appropriate) and backup/archive policy associated with it. At the moment we have no means of recording such an association. It is my belief that one should be developed as a matter of urgency. This need not be a heavyweight system (in fact a comma separated file would do at a pinch) and we are fortunate in that our use of AFS gives us a natural unit of data, namely the volume, to which these assignations should be made. The initial assigning of an owner and a suitable retention period to each volume would require a considerable amount of effort but without this information, it is my belief that it will be impossible for the School to feel secure that all of its data is being archived in an appropriate way.