How Routine Data Center Operations Put Your HA/DR Plans at Risk

Transcription

1 How Routine Data Center Operations Put Your HA/DR Plans at Risk Protect your business by closing gaps in your disaster recovery infrastructure Things alter for the worse spontaneously, if they be not altered for the better designedly. Francis Bacon CONTENTS Why HA/DR Risk Grows Over Time Configuration Errors That Lead to HA/DR Failures Testing Is Not a Panacea How an HA/DR Monitoring and Analysis Solution Can Help Introduction To ensure business continuity and protect their critical business services and applications, enterprises spend millions of dollars developing plans for high availability and disaster recovery. Yet in most instances, when put to the test these plans will not work as intended. Why? Because as soon as the infrastructure and plans are complete, the data center environment begins changing to accommodate new applications, new equipment and new users. All this change inevitably introduces configuration gaps and errors in an environment that was previously business continuity plan ensured. These errors accumulate, often at the rate of dozens a month, or hundreds in a year. If an incident occurs, any one of these errors can lead to data loss, service outages or recovery times that violate service-level agreements. To discover such gaps, enterprises run high-availability/disaster recovery (HA/DR) tests. However, these tests are so costly and disruptive that they are usually conducted only once a year. Consequently, infrastructure and plans are fully corrected only on an annual basis. Further, in most organizations the tests do not simulate a complete outage, so some vulnerabilities are not discovered until an actual incident occurs. But there is a way to detect misconfigurations and eliminate vulnerabilities between HA/ DR tests: automated high availability and disaster recovery monitoring and analysis.

2 In this paper, we will discuss why the number of data center configuration errors is growing, review examples of common errors and their consequences, and outline how high availability and disaster recovery monitoring and analysis can help enterprises keep their configurations current and reduce risk to critical business operations. Why HA/DR Risk Grows Over Time To maintain business continuity, enterprises invest in advanced high-availability and disaster recovery software. This software provides data replication to non-primary systems in remote locations, failover of business applications to backup systems in the event of failures, and failback of the applications when the primary systems are brought back online. Detailed plans list the exact sequence of steps needed to recover critical systems and data in the event of outages caused by equipment failures, human error, malware, power outages and severe weather damage. However, this elaborate HA/DR infrastructure will perform successfully only if: Data is backed up or replicated with complete accuracy. Backup equipment is available with the same capabilities and capacities as production systems, and with the same software configurations, including the same release levels. Recovery plans exactly reflect the current state of hardware and software configurations, so they provide accurate information on how to move applications and data to backup systems, restart them the right way, and connect them to the right storage and network devices, all in the proper order. Meeting these conditions is becoming increasingly difficult. Today s enterprise production IT environments are continually growing in complexity. They include a myriad of critical business services running across thousands of servers (usually a mix of physical and virtual), databases, storage devices and pieces of networking equipment. Three-tier applications and multi-dependency service-oriented architectures are common, as are cloud-based services. In such a complex data center environment, production systems may be reconfigured many times each day. For example: Hardware devices are brought down for maintenance, upgraded and expanded. Software is upgraded and new software is installed. Storage and networking resources are re-allocated between systems and applications to improve performance and meet surges in transaction levels. Applications and desktop environments are moved from physical to virtual machines, and migrated to different virtual machine clusters. Users are added, regrouped and assigned new resources. Every one of these changes can introduce errors that will lead to data loss and service outages if not caught and corrected in time. Such configuration errors accumulate over time in most enterprises, and are caught only by the occasional (usually annual) HA/DR test. This situation is illustrated in Figure

3 ./01!%&(#2% HA/DR TEST MAX. ERRORS AVG. Configuration Errors That Lead to HA/DR Failures There are literally thousands of types of configuration errors that can lead to failures in highavailability measures and disaster recovery plans. The following are a few representative examples: Human errors: Many human errors can result in lost data or system failures in the event of an incident. An error can be as simple as an application owner bringing an application down for maintenance and forgetting to restart replication after the application is back online. If the primary storage system fails, current data is lost. Other human errors are very difficult to detect. For example, a storage administrator could allocate a backup storage device to a server running a new application, but forget to remove mappings to the server running the old application. In the event of a disaster, both servers could start writing to the storage device, degrading performance and possibly corrupting data and causing the system to crash. REPLICATION inconsistencies: Complex environments typically include different consistency groups for replication. Each group may use different replication adapters and different network infrastructures. Storage volumes in different consistency groups can be provisioned to the same database (Figure 2). If these groups fail at different times, the database at the disaster recovery site will be corrupted, and the data will not be recoverable. TIME Figure 1: Errors accumulate until caught by annual HA/DR tests. PRODUCTION DB GROUP A GROUP B Figure 2: consistency errors can cause data corruption. DISASTER RECOVERY Consistency groups A and B use different replication adapters and network infrastructures Mismatches between production and standby systems: If production systems are upgraded or reconfigured without corresponding changes made to standby systems, then failing over to the standby systems might cause performance degradation or application crashes. could be as simple as software upgraded on one side but not the other, disks added to production systems with more capacity or higher performance, or reconfigured paths to storage. 3

4 Production systems, reconfigured to use three storage volumes Standby systems, configured to use two storage volumes DB DB DB DB DB DB DB Error: Data on this volume invisible to standby systems Figure 3: Failing to change the configuration of standby systems can prevent application recovery. AREA CATEGORY POSSIBLE IMPACT HA Setting SAN SAN SAN SLA Violations Partial replication Inconsistent I/O capacity across nodes Replicas unmapped to DR servers Inconsistent database backup Unauthorized storage access Incorrect r edundancy configuration Insufficient number of hot-spare drives HA/DR Service Level Agreement breach Incorrect storage consistency group configuration Data loss in the event of a site outage Downtime Unexpected performance degradation as cluster service fails over (or re-assigned) to a standby node Extended downtime in the event of site outage Potential data loss Extensive data loss in the event of a physical or logical fault in the primary copy Risk of massive data corruption affecting multiple Virtual Machines Single point of failure in a system designed for redundancy Insufficient redundancy might result in massive data loss affecting multiple servers No compliance with data retention requirement Insufficient performance and redundancy expected after fail-over Data loss in the event of a site outage Downtime Figure 4: Examples of configuration errors. Subtle configuration errors: Some configuration errors can be extremely hard to detect. For example, an administrator might reconfigure a database in the production environment to use three storage volumes but leave standby nodes configured to see only two of these volumes (Figure 3). In the event of a failover, when the application is restarted on the standby systems, a third of the data will be unavailable and the application will be unable to go online. Enterprises moving to virtualization may need to learn a whole new set of configuration best practices. To cite one, they would want to avoid configuring a large number of clustered virtual disk files to use the same storage array. A failure of this array could cause multiple applications to stop running and prevent failover. Figure 4 lists a few out of hundreds of common configuration errors, and the possible impact of each. Testing Is Not a Panacea High-availability/disaster recovery tests are an important tool for discovering data center configuration errors, but they are hardly a panacea. They are so costly and disruptive that most enterprises run them only once a year, allowing configuration errors to accumulate during the intervening 12 months. 4 In addition, tests often fail to uncover critical errors. That is because few organizations are willing to incur the cost and the risk of simulating a disaster by completely shutting down all applications and services in the data center, and then trying to restore and recover everything. Instead, they

5 shut down only part of their infrastructure, or test the failover of servers but not applications, or recover applications but test them at only low-transaction volumes, or don t fully test dependencies among applications, databases, middleware and storage devices. How an HA/DR Monitoring and Analysis Solution Can Help The goal of an HA/DR monitoring and analysis solution is to collect and analyze configuration information from storage, servers, databases, virtual machine clusters and replication infrastructures in order to identify configuration errors and HA/DR risks. For example, Symantec Disaster Recovery Advisor monitors and detects potentially threatening data protection and availability configuration risks in the data center. It: Gathers detailed configuration information from the entire data center environment, including local and remote sites, using non-intrusive, agentless software. Builds a detailed high-availability and disaster recovery topology map that captures the dependencies of devices and applications throughout the environment. Utilizes a knowledge base of more than 5,000 configuration risk signatures and a powerful gap detection engine to detect best practices violations in HA/DR configurations. Generates a ticket for each gap identified, providing a detailed description of the risk, describing its potential impact, and suggesting a remediation approach. With this information, administrators can take action on a regular basis to eliminate misconfigurations. As a result, enterprises that use Disaster Recovery Advisor can keep the average and maximum levels of risk far lower than if they relied on annual HA/DR tests, as shown by a comparison of Figure 1 and Figure 5. ERRORS Analyze/Correct./01!%&(#2% TIME Analyze/Correct Figure 5: Regular monitoring and analysis dramatically reduce the average and maximum number of configuration errors and the associated risks. In summary, an HA/DR monitoring and analysis solution such as Symantec Disaster Recover Advisor can help administrators: Prevent unwanted downtime, by identifying configuration errors and single points of failure in production environments. Improve the effectiveness of high-availability systems, by increasing the chances that failovers and failbacks will work smoothly. Increase the likelihood that critical business applications can be successfully restarted within the limits of service-level agreements, by preventing replication errors and maintaining consistency between production and backup systems. Improve performance and consistently implement best practices by identifying bottlenecks and suboptimal configurations throughout the data center environment. MAX. AVG. CASE STUDY For a financial services company, Disaster Recovery Advisor scanned 46 servers and 12 storage devices and: Uncovered 112 risks of data, availability and performance. Generated 41 tickets related to immediate risks. 5

6 For more information on best practices for meeting critical service-level agreements and additional information on Symantec Disaster Recovery Advisor, please visit IT Business Continuity on Symantec.com. Appendix Sample Ticket Created by Disaster Recovery Advisor 6

7 Selected Hardware and Software Supported by Disaster Recovery Advisor 77