Backup and recovery best practices for Microsoft SQL Server 2005

Transcription

1 Backup and recovery best practices for Microsoft SQL Server 2005 Overview Solution configuration SQL 2005 database servers Storage configurations Tape libraries HP Data Protector cell Storage management SAN infrastructure Domain controllers Application load simulation Database storage layout Performance collection and monitoring Testing Testing objectives Backing up SQL Server data with Data Protector Impact of multiple backup streams Baseline performance Disk-to-tape library backup performance Disk-to-disk backup performance Online backup performance testing summary Effects of online backup on SQL transactions performance Transaction log considerations System databases considerations Restoring SQL Server data with Data Protector Restore performance Database layout and backup performance considerations Backup types and strategy Best practices and results Database administrators Location of SQL Server binaries Location of system databases Planning for file groups Setup transaction log Server administrators Monitor server workload Windows Instant File Initialization Storage administrators Monitor EVA performance Backup administrators Select the right target backup device Perform incremental backups to save time and space Perform transaction log backup frequently

2 Additional backup considerations Conclusion Appendix A. Bill of materials Appendix B. SQL Server backup key performance indicators Appendix C. HP StorageWorks Storage System Scripting Utility Appendix D. Test database layout For more information

3 Overview With databases regarded as mission critical in most companies, the ability to back up Microsoft SQL Server 2005 data when required and within the specified backup window is vital. HP OpenView Storage Data Protector 6.0 (Data Protector) is integrated with SQL Server 2005 via the Virtual Backup Device Interface (VDI). This paper discusses the operation of Data Protector with various online streaming backup and recovery methodologies. Microsoft is encouraging SQL Server administrators to do backup by using backup software that integrates directly with SQL Server 2005 via VDI. The integration of Data Protector and SQL Server 2005 provides the following benefits: Central management of all backup operations Media management Scheduling Parallel backups and restores To help you choose from among the available configuration options and backup and recovery procedures, HP has conducted extensive laboratory tests to determine best practices. This paper discusses those test results so that users can understand the options and the limitations of implementing backup and recovery using Linear Tape-Open (LTO) tape, disk-to-disk, and virtual tape devices. The audience for this paper is HP users in an enterprise environment currently running or planning to run SQL Server The paper discusses: Best practice recommendations encompassing configuration, design, and deployment Backup and recovery recommendations for the integration of Data Protector Software and Microsoft SQL Server 2005 Backup and recovery recommendations for the LTO tape, the disk-to-disk, and the virtual tape methodologies The impact on database performance and throughput for each of the methodologies: LTO tape, disk-to-disk, and virtual tape General recommendations for selecting which backup and recovery method to use Supporting configuration recommendations for HP servers and for HP StorageWorks 8100 Enterprise Virtual Array (EVA8100) disk array Recommendations for the use of parallel backup and restore operations, and the impact of multiple streams and device concurrency on overall backup performance By leveraging the recommendations and best practices, administrators can shorten backup windows and efficiently load the server, thereby reducing cost and maximizing the use of hardware and personnel resources. 3

4 Solution configuration The testing environment consisted of a Microsoft SQL Server 2005 running on HP ProLiant BL480c BladeServer systems and the HP StorageWorks EVA8100. The HP StorageWorks 6510 Virtual Library System (VLS6510) with Serial Advanced Technology Attachment (SATA) drives, and the HP StorageWorks Enterprise Systems Library (ESL) E-Series 712e tape library were used as dedicated backup and restore devices. Configuration of the servers and the devices attached to the storage area network (SAN) was based on: HP StorageWorks 4x00/6x00/8x00 Enterprise Virtual Array configuration best practices white paper available on 4AA0-2787ENW.pdf?jumpid=reg_R1002_USEN HP StorageWorks Enterprise Backup Solution design guide available on h20000.www2.hp.com/bc/docs/support/supportmanual/c /c pdf For a list of hardware and software used in this project, see Bill of materials on page 35. During this test cycle, Data Protector 6.0 was used for online SQL backups with the goal of reducing backup and recovery windows. Reducing restore times allows more time for recovery operations, including transaction log roll forward, and increases the potential to meet tighter service-level agreements and recovery time objectives (RTOs). SQL 2005 database servers The database servers consisted of HP ProLiant BL480c G1 BladeServer systems running Microsoft Windows Server 2003 R2 x64 Enterprise Edition. Each server was configured with one dual-port, Fibre Channel mezzanine card. The Fibre Channel cards were based on Emulex LP1105 Series adapters and were installed using the HP host bus adapter (HBA) Smart Component drivers. These drivers can be downloaded from the HP website on TheHPStorportmini-port driversand the Microsoft Windows2003Storportdriverwereusedto create a compatible configuration in whichdiskand tape couldbeaccessedthrough thesamehba, thus decreasing the cost of implementing SAN-based backups. Note For more tape compatibility information, see the Enterprise Backup Solution (EBS) compatibility matrix and the HP StorageWorks Enterprise Backup Solution design guide at c /c pdf. Figure 1 provides a graphic of the hardware configuration. 4

5 Figure 1. Configuration diagram 1 SQL Server 2005 database server #1 HP ProLiant BL480c, 16-GB RAM 2 SQL Server 2005 database server #2 (recovery server) HP ProLiant BL480c, 16-GB RAM 3 Management server HP ProLiant BL460c, 4-GB RAM Command View EVA and tape library Data Protector cell manager The c7000 blade enclosure provided Fibre Channel and Ethernet connections to the ProLiant c-class BladeServer systems. The enclosure also provided redundant power and cooling features along with HP Integrated Lights-Out (ilo) connectivity. The ilo connectivity made it possible to deploy and manage remote servers from their consoles. All management features in the c-class enclosure were interfaced through the c-class Onboard Administrator (OA). 5

6 Storage configurations The storage subsystem consisted of an EVA8100 with two HSV210-B controllers and twelve Fibre Channel (FC) drive enclosures (2C12D configuration). The disks were arranged in an inline layout and striped vertically, in keeping with the default behavior on the EVA. The database storage was designed using two different virtual disk and disk group layouts as shown in Figure 2. This allowed a backup and restore performance comparison between a multiple-file andasingle-file database layout. For each layout, the transaction log file is isolated in a dedicated disk group consisting of 16 x 146-GB 15K RPM FC disk drives. The disk group housing the database files consists of GB 15K RPM FC disk drives. All virtual disks are configured with a redundancy of Vraid 1. All SQL Server data files, including the system databases and transaction log data, were stored on the EVA. See Figure 2 for a graphic of this layout. The Microsoft SQL Server test database is 1 TB. The user load on the database was simulated using an online transaction processing (OLTP) workload and a no-wait or constant 3:2 read-to-write ratio of transactions. The workload parameters were modified to maintain a response time of < = 15 ms across database volumes while sustaining a request rate of at least 10,000 IOPS. This same load scenario was used in every test case. 6

7 Figure 2. EVA storage layout Test # Disk groups Virtual disks Physical disks, Physical disks, Physical disks, data disk group log disk group total Tape libraries Two nearline devices were used as backup targets in our testing: The HP StorageWorks ESL E-Series 712e tape library was configured with four HP StorageWorks Ultrium 960 (LTO3) Fibre Channel tape drives. The HP StorageWorks VLS6510 virtual tape library was configured as a single library featuring four virtual LTO3 tape drives (without data compression), GB SATA back-end disk drives and four Fibre Channel ports. The virtual library was set for HP StorageWorks ESL emulation. HP Data Protector cell The Data Protector cell is a network environment that includes a cell manager and client systems that run agents. The cell manager is the control tower managing the activities and internal database within the Data Protector cell. It is not necessary to administer the backup and restore activities 7

8 directly from the cell manager; any client within the cell can connect to the cell manager over the network and be used to administer the activities of the cell. Client systems run agents that are defined at installation according to the following requirements: The media agent is installed on a server if that server is going to have direct access to a tape device for backup and restore. The tape devices either can be directly attached or can communicate over a SAN. The disk agent is allocated to a server if that server is going to read data from a disk device, whether local or remote. In our test environment, SQL database servers were installed with a media agent, a disk agent, and the SQL agent. Figure 3 depicts the logical Data Protector network; all backup-related traffic was conveyed over the SAN. Figure 3. Data Protector cell configuration diagram Storage management HP Command View was running on a general-purpose HP ProLiant BL460c server and was configured with Microsoft Windows Server 2003 Enterprise Edition SP1. HP Command View EVA 7.01 andcommand View TL 2.2wereusedto managethe EVAand thetapedevices,respectively. 8

9 SAN infrastructure SAN interconnect consisted of two independent fabrics based on two Brocade 4-Gb SAN switches for HP c-class running firmware 5.1.0b. All servers and storage devices connected at 4 Gb/s. Domain controllers A single Active Directory (AD) Enterprise using a domain controller and backup domain controller consisted of HP ProLiant DL380 servers running Microsoft Windows Server 2003 x64 R2 Enterprise Edition and were configured as Active Directory (AD), domain name server (DNS), dynamic host configuration protocol (DHCP), and global catalog servers. Application load simulation TheMicrosoft SQLIOSim utilitywas used to stress testthe environmentand to determinethe EVA performance characteristics for the configured solution. The Benchcraft load generation tool developed by Microsoft for SQL Server was used to place the SQL Server solution under a nominal load of approximately 800 transactions per second. Note For more information on using SQLIOSim, see Database storage layout For this testing, an SQL Server housed an OLTP database with 1 TB of data. The database was spread across four data files in two file groups and one transaction log file. Thislayoutfollows themicrosoft guideline of one file per processor core. Each data file was stored on a dedicated EVA virtual disk; transaction log and database virtual disks were isolated in different EVA disk groups. Total server storage was approximately 1.5 TB. For more details, see Test database layout on page 41. Performance collection and monitoring Performance metrics were collected using Microsoft Windows performance monitor for Windows (Windows performance monitor). Performance metrics from Microsoft SQL Server 2005 and the EVA8100 integrate directly into the Windows performance monitor utility. To view a detailed description of Microsoft SQL Server 2005 and EVA performance metrics, see SQL Server backup key performance indicators on page 37. 9

10 Testing Testing objectives OLTP applications tend to grow at an explosive pace, driving customer requirements for increasingly larger methods of backing up and restoring data. In addition, it has become more important that the time required for data backup and restore be reduced to a minimum so as not to interrupt the day-to-day running of company applications. It is important to consider ways to accelerate the process and minimize the impact on the production systems. The goals of the testing were to develop a set of backup and recovery best practices to minimize the backup windows through the use of high-performance, streaming backups. Backing up SQL ServerdatawithDataProtector Microsoft SQL Server 2005 provides the Virtual Backup Device Interface (VDI) for backup. The VDI enables third-party backup solutions such as Data Protector 6.0 to integrate directly with SQL Server, providing support for application-aware backup and restore operations. Such Application Programming Interfaces (APIs) are engineered to provide maximum reliability and performance and to support the full range of SQL Server backup and restore functionality. Data Protector 6.0 makes full use of this API: All backups are performed online; the database remains accessible to users. The entire database is backed up: selecting or excluding a particular table is not possible. However, only the used portion of the database, containing valid pages, is backed up. This is an advantage compared to offline file backups. Different types of backups are supported by the integration: full (database), differential, transaction log or snapshot. Note that Snapshot mode is only possible when used in conjunction with the Zero Downtime Backup (ZDB) option for Data Protector and requires HP StorageWorks Business Copy EVA software to be installed on the array. This testing was performed on a single SQL Server 2005 server instance and aimed at characterizing online backup with Data Protector in an SQL Server environment. Impact of multiple backup streams Parallel backup and restore operations can improve the capability of Microsoft SQL Server 2005 to manage high-performance backup of very large databases. The BACKUP and RESTORE statements use parallel I/O in a number of ways: If a database has files on several logical unit numbers (LUNs), the BACKUP statement uses one thread per disk device to read the extents from the database. If a backup set is stored on multiple backup devices, both the BACKUP and the RESTORE statement use one thread each per backup device. However, SQL Server 2005 supports only a single data stream per targeted backup device. Therefore, the backup hardware configuration is the primary factor for determining the number of streams that can be backed up in parallel. To understand the effect of streams and device concurrency on overall backup performance, we performed a series of backups, increasing the number of targeted backup devices linearly for 10

11 each test iteration until either the point of diminishing return was met or a maximum hardware configuration was attained. We evaluated the following backup devices: LTO-3 tape drive in ESL 712e LTO-3 virtual tape drive in VLS6510 Data Protector file library using 500-GB Fibre Channel Advanced Technology Attachment (FATA) disk drives Baseline performance Data Protector allows for the creation of null devices for baseline testing purposes. This is a very useful feature for assessing the I/O path performance and detecting any possible bottlenecks in the EVA orthe SQLServerconfiguration. Before starting the performance evaluation, we checked the baseline EVA throughput and performance characteristics when subjected to a heavy SQL Server backup workload. As noted in Figure 4, throughput rose from 380 MB/s using one stream to 540 MB/s, nearly 2 TB per hour, when four concurrent streams were used. Figure 4. Throughput using concurrent streams Note The number of parallel streams could have been further increased; however, with four streams, we reached the point of diminishing return for this configuration. For this test we used null devices that do not generate data transfer over the SAN. More streams can generate a load that saturates the available aggregated SAN bandwidth (On the server in this configuration our limit was 2-GB x 4-GB HBAs or about 800 MB/s.) leaving no room for the data flow to thetapedrives attached tothe SAN. Figure 5 shows that adding streams also increases SQL Server processor utilization. 11

12 Figure 5. Impact of stream concurrency on CPU utilization Disk-to-tape library backup performance HP StorageWorks ESL 712e tape libraries were tested, using LTO-3 drives as the target. As the data in Table 1 and Figure 6 point out, linear performance improvements were observed as the number of drives increased. To reach the throughput target of 500 MB/s, which was set during baseline testing with null devices, two additional HBAs and additional Fibre Channel LTO-3 drives would be needed. Table 1. SQL backup performance with LTO-3 tape drives Device 1 drive 2 drives 3 drives 4 drives ESL 712e with LTO-3 92 MB/s 187 MB/s 280 MB/s 374 MB/s 12

13 Figure 6. SQL backup performance with LTO-3 tape drives Disk-to-disk backup performance Backup using a Virtual Tape Library A Virtual Tape Library (VTL) emulates the drives of a physical tape library and stores backup images to disk. Backup applications use the VTL-emulated tape and library devices for disk-to-disk backups. When using HP StorageWorks VLS6510 virtual tape library to test concurrency, performance began to level off after two parallel data streams, because the backup throughput demand exceeded the aggregate performance capabilities of the 24 back-end disk drives configured in the solution. Base VTL throughput can be improved by adding more capacity (disk drives), disk controllers, and Fibre Channel (FC) ports. However, newer tape drives such as LTO-3 and LTO-4 are capable of backing up data, with compression, at a rate greater than 80 MB/s. Therefore, backing up large amounts of multi-streamed data to physical tape can be faster than when using a single VTL. Backup using Data Protector advanced backup to disk The Advanced Backup to Disk functionality in Data Protector also allows tape virtualization with basic backup resource sharing through a new device type called a file library. This new feature complements the Data Protector backup-to-disk solutions portfolio and allows the use of one or more Windows volumes (NTFS) to be used as a backup and recovery repository. The device is conceptually similar to a tape stack in that it consists of one or more files in container directories, which are the equivalent of slots in a tape stack where data is stored, and one or more writer instances, which are the equivalent of tape drives in a library. The backup data is stored in a series of files called file depots, which are created each time a backup to the device is made. Data Protector file libraries should not be confused with the VLS. Both are disk-based backup solutions. The VLS is a hardware solution in which backups are virtualized at the device; the Data Protector file library is a software solution. 13

14 To house the file library repository, we configured an additional disk group on the EVA8100 with GB FATA disk drives. This solution provides a low-cost alternative to the standard, high-performance FC disk drives. Table 2 and Figure 7 compare the backup performance of an HP StorageWorks Virtual Tape Library VLS6510 to that of the Data Protector file library. Table 2. Data Protector file library versus virtual tape performance Device 1 drive 2 drives 3 drives 4 drives Data Protector file library 114 MB/s 126 MB/s 138 MB/s 150 MB/s VLS6510/virtualLTO-3 126MB/s 178 MB/s 190MB/s 193 MB/s Figure 7. Data Protector file library versus virtual tape performance Although using multiple concurrent streams yields benefits in both cases, the VLS6510 provided higher throughput than the Data Protector file library. The FATA disk drives housing the Data Protector file library provided sufficientthroughputinthis configuration. The VLS front end was optimized for sequential I/O and therefore provided better backup performance than the NTFS volumes used with the file library option. Note Beware of the effect of the first write when using EVA virtual disks to house the Data Protector file library The virtual disk will provide its maximum performance only when most disk blocks have been written once. This means that the performance characteristics of your file library will improve over time as more backups are performed. Online backup performance testing summary The test data shows that, irrespective of the device type, the use of more targeted devices for higher backup concurrency improves backup performance. Figure 8 and Table 3 summarize the results obtained with each of the tested backup devices. 14

15 Figure 8. Backup performance Table 3. Backup performance results Configuration 1 drive 2 drives 3 drives 4 drives ESL 712e with LTO GB/hour 676 GB/hour 1 TB/hour 1.3 TB/hour VLS6510 virtual LTO GB/hour 641 GB/hour 687 GB/hour 696 GB/hour Data Protector file library 413 GB/hour 456 GB/hour 498 GB/hour 542 GB/hour The test data shows that using multiple database files and more than one backup device improves backup performance by increasing concurrency. In order to realize the potential performance improvements that concurrency provides, both the server and the disk subsystem must be able to handle the higher throughput. For large SQL Server databases, the use of threaded backups to disk (VTL or file library) does not necessarily improve performance. In fact, threaded backups to disk may be slower than physical tape because the hardware compression and the high-performance sequential access provided by the LTO-3 tape drive outperforms the quasi-sequential access disk-to-disk backup. Effects of online backup on SQL transactions performance Up to this point, backup performance results were based on testing systems with no transactional load. To assess the impact of a database backup executed during production hours while SQL Server was processing user queries, we used an ESL 712e tape library configured with four LTO-3 tape drives. The transactional load applied to the database was an OLTP-like workload with queries performed across the entire database seek range (1 TB). The benchmark parameters were adjusted to generate a nominal load of approximately 800 SQL transactions per second. Server processor 15

16 utilization, SQL transactions per second, disk reads and disk writes request rates and latencies, as well as the backup throughput metrics were recorded during the backup window. Table 4 and Figure 9 highlight the key performance indicators for the disk subsystem. Table 4. Disk performance statistics for backup under load Counter 95 th Percentile Average Maximum LogicalDisk\_Total\Disk Reads/sec 10,408 8,644 13,294 LogicalDisk\_Total\Disk Writes/sec 4,428 3,796 24,528 LogicalDisk\_Total\Disk Transfers/sec 14,212 12,440 27,734 LogicalDisk\_Total\Avg. Disk sec/read LogicalDisk\_Total\Avg. Disk sec/write Figure 9. Read/write disk response times and latencies for backup under load Table 5, Table 6, and Figure 10 highlight the key performance indicators for the SQL Servers. Table5. SQL Server performanceindicators Counter 95 th Percentile Average Maximum Processor\_Total\% Processor Time Databases\_total\Transactions/sec 2, ,154 Buffer Manager\Page reads/sec 9,573 6,815 32,403 Buffer Manager\Page writes/sec 5,105 3, ,231 Buffer Manager\Buffer cache hit ratio

17 Figure 10. Server processor utilization and throughput for backup under load Table 6. Backup performance indicators Counter 95 th Percentile Average Maximum Databases\_total\ 233,352, ,337, ,003,328 Backup/Restore Throughput/sec LogicalDisk\_Total\Disk Bytes/sec 315,210, ,291, ,526,688 As shown in Figure 11, the transaction rate is moderately affected by the full database backup operation. Area 1 shows the SQL Server activity before the backup job started. Area 2 is the transaction rate drop caused by the database checkpoint activity before starting the backup data movement. The transactions that were not yet committed to disk are flushed to the data file. The checkpoint duration is proportional to the amount of dirty pages that need to flush. The database checkpoint frequency and duration can be customized with the command CHECKPOINT (transact SQL). For more information, see The transactions rate valley in area 3 is workload-dependent and depicts an expected pattern for the mix of queries used for load generation in this testing. The backup throughput declines over time as threads terminate each time a data file backup completes. We used a four-file layout in this testing. After approximately one hour, two out of the four data files were backed up. At this stage, data is streamed out of only two files or, more importantly, two LUNs. 17

18 Figure 11. Effect of online database backup on SQL transaction rate The SQL transaction performance is not adversely affected; however, the I/O activity generated by the execution of user queries throttles the backup throughput, extending the required backup window and reducing throughput by 50% when compared to a backup without load. Note Results may vary depending on factors such as the workload on the SQL Server and the underlying storage configuration. Transaction log considerations Every Microsoft SQL Server 2005 database has a transaction log that records all transactions and all database modifications made by each transaction. This record supports three operations: Recovery of individual transactions Rollback recovery of all incomplete transactions when SQL Server is restarted Rolling arestoreddatabaseforward to thepoint of failure When the transaction log has been backed up successfully, the inactive portion of the transaction log is truncated. Because the inactive portion contains completed transactions, it is not needed during a recovery process. Conversely, the active portion of the transaction log contains transactions that have not yet been committed to the database file. SQL Server reuses the truncated, inactive space in the transaction log to log upcoming transactions instead of allowing the transaction log to continue to grow. Note Manually truncating the transaction log breaks the log backup semantic and may impact your recovery point objective. When logs are manually truncated, the database is not protected from media failure until a full database backup is created. In addition, the most recent recovery point is limited to the last known good database backup. 18

19 The ending point of the inactive portion of the transaction log, and hence the truncation point, is the earliest of the following events: The most recent database checkpoint The start of the oldest active transaction, that is, a transaction that has not yet been committed or rolled back This represents the earliest point to which SQL Server would have to roll back transactions during recovery. The start of the oldest transaction that involves objects published for replication whose changes have not yet been replicated This represents the earliest point that SQL Server still has to replicate. The transaction log can be implemented on several files. The files can be definedtoexpandas required to reduce both administrative overhead and the likelihood of running out of space for the transaction log. Truncating the transaction log has minimal effect on transaction throughput. Transaction log backups usually are much quicker than full and differential jobs. Performing frequent transaction log backups throughout the production day allows for more point-in-time recovery options. This lengthens and complicates the recovery process, however, because it will be necessary to restore all transaction log data since the last full or differential backup. Note Full and differential backup jobs do not back up the transaction logs. If you wish to protect that critical data, it is necessary to schedule separate jobs to back up the transaction log between each full or incremental backup. The impact of a transaction log backup on an SQL Server can be summarized as: Log file capacity: 50 GB Used transaction log space before backup: 99% Transaction log backup throughput with four LTO-3 drives: ~280 MB/s Used transaction log space after backup and log truncation: 6% Frequent transaction log backups may be required to enhance recoverability and meet your Service Level Agreement (SLA) requirements. Figure 12 highlights the SQL Server processor utilization caused by a high-performance transaction log backup. Knowing how hard the server is working to perform backups is a key consideration in determining the transaction log backup frequency. Server utilization must be balanced with recovery point objective. With 30% processor utilization for a backup workload of 280 MB/s, this server has reasonable space for SQL transaction processing and would support frequent transaction log backups throughout the production day. Best practice To avoid a bottleneck, the processor utilization induced by the backup workload must be factored in when estimating the server s processing power requirements. 19

20 Figure 12. Server processor percentage utilization and backup throughput When the transaction log has been successfully backed up, the inactive portion of the transaction log is truncated. You cannot see in Windows File Manager that the transaction log has been truncated. The transaction log file does not automatically shrink. To display your SQL Server databases transaction log file usage, use the following Microsoft SQL Server Database Consistency Checker (DBCC) command: DBCC SQLPERF (LOGSPACE) Alternatively, the log-file usage can be monitored using the Windows performance monitor counter Databases\*\LogFile(s) Used Size(KB) as shown in Figure 13. Figure 13. Transaction log truncation 20