Disaster Recovery and Business Continuity Considerations

Transcription

1 2012 Disaster Recovery and Business Continuity Considerations ANDY BRETT, CHANNEL 3 CONSULTING CTO CHANNEL 3 CONSULTING Manor House, 1 The Crescent, Leatherhead Surrey, KT22 8DH

2 Introduction When it comes to provision of business systems and the supporting IT, much is said about disaster recovery (DR) but actually achieving it can be more elusive than initially anticipated. And that assumes a comprehensive understanding from the outset, both in terms of what is actually meant by a 'disaster' and how 'recovery' might be achieved. The starting point should actually be a view on Business Continuity (BC). Any business would prefer to have no system failure or downtime at all, but this of course comes at a cost. Taking an informed view of business need and properly understanding the potential impact of such failures will enable a pragmatic, cost effective and maintainable approach to business continuity and disaster recovery. Business Continuity is essentially about ensuring that the business processes can continue to operate, and staff can continue to work, in the event that one or more of the technology components relied upon to follow those processes becomes unavailable for any reason. In such circumstances this will require staff to revert to use of alternate systems or applications where available, or in extreme cases, manual and/or paper-based processes. In either case consideration must be given to the 'recovery' aspects of entering data captured during any outage back into the primary system once it again becomes available. All areas of the business should have comprehensive Business Continuity plans in place, which prescribe processes to be followed in the event that normal operation is interrupted for any reason - and this will include technical as well as non-technical reasons. This can be quite a significant piece of work, especially for larger organisations, so should be planned and executed properly, with all business and technical stakeholders involved throughout the process. The definition of a Disaster in technology terms can also be discussed at length, but is typically considered, from a business perspective, to be the inability to access 'critical' business systems for a period of time, primarily due to failure of one or more technology components. The process by which recovery from that 'Disaster' scenario is achieved technically will depend upon a number of factors, primarily driven by the way in which the technical solution has been architected, designed, implemented and maintained. Achieving Technical Continuity The ultimate objective is to achieve technical continuity such that failure of one or more components of the overall technical solution has minimal impact on the business - in fact does not 'adversely impact' the business would be a better term, and is subject to agreement with the business. The level of continuity that can be achieved by the technology will depend upon the level of 'resilience' of the solution as a whole. Most solutions are comprised of a number of technical components. If not properly architected, designed and implemented, any one of those solution components could be a single point of failure (spof) - a single network cable running between two pieces of equipment being a simple example. Properly designed solutions will therefore typically avoid any single points of failure, although there is a corresponding increase in cost to achieve such an objective. In the end it comes September 2012 Page 1 of 8

3 down to an assessment of cost versus risk, which should be assessed from the business level downwards. To a certain extent, a prudent level of technical resilience will be built into any solution, irrespective of business need, if only because this has a positive impact on the cost of maintaining and supporting that solution - i.e. it is simply more cost effective to operate and manage a solution that does not keep failing. It is worth noting at this point that from the users' perspective the perception will be at the level of 'service' or 'application' resilience, not technical resilience. However, for the purposes of this discussion we will focus on technical resilience, the options for which can then be assessed separately in determining an appropriate level of service/application resilience. Technology 'Layers' In order to discuss the methods by which Business and Technical Continuity can be achieved, and ultimately provide support for Disaster Recovery, it is probably easiest to consider a generic technical environment described in 'layers' as a starting point (whilst recognising that there are a number of more advanced technical solutions that collapse or combine layers into a single service provision more of which later). The diagram below provides a simplified representation of such layers. The network layer is the first (lowest) layer in the diagram and provides connectivity and communication between all components, systems, locations and users (essentially providing connectivity between layers). Other layers are 'built on' this lower layer and include disc storage, computer servers, applications, end user devices, etc. The user experiences the technical environment from the 'top down' - i.e. from the presentation layer - and is often unaware of the type and composition of layers of technology that underpin delivery of the service they are using, which is the way it should be. Services can be thought of as overlaying vertically up and down the layers, utilising various components within each of the layers. When represented in this manner it can be seen that a failure in any one layer has the potential to directly impact the user experience if that layer is a key component of the service being used (clearly not all components in all layers necessarily contribute to all services accessed by end users). For example, failure of a component at the storage layer could mean that an application layer component cannot access data and freezes (stops working), which in turn means that the user experiences loss of the application service at the presentation layer. It should therefore be clear that technical resilience needs to be built into each layer, as well as between layers. Introducing resilience at one layer may improve the level of availability of components within September 2012 Page 2 of 8

4 that layer, but ultimately does little to ensure continuity of the service being used by the end user community and only partially enables disaster recovery. For a truly resilient technology environment, and therefore business service, consideration must be given to all components in all layers that contribute to or support those services. These measures will minimise the likelihood of a disaster occurring and should facilitate improved speed of recovery should a disaster occur. Key Concepts In defining the type of technical solution to be implement in support of disaster recovery and business continuity there are two primary concepts to be considered from a business service perspective: 1. The first is the point in time to which the system/service should be recoverable as a minimum. For example, if you take a copy (a backup) of the system data and its state every evening at 6pm, then in the event of a failure and loss of data that system can only be recovered to the last time a copy (backup) was taken i.e. 6pm on the previous day. Any data entered after this time up until the next time a copy is taken will have been lost and must be re-entered manually. This is known as the Recovery Point Objective (RPO) and can vary from days to minutes generally the more recent the RPO, the greater the cost. 2. The second is how long it should take to recover a system/service back to full operational state from the point at which it is recognised to have failed. This will largely depend upon how long the business is able to cope without the system/service (criticality) and how long it actually takes to restore that system/service (e.g. the time it takes to enter any data captured manually during the system/service outage as part of the business continuity process that is created after the RPO). This is known as the Recovery Time Objective (RTO). Typically the more critical the system/service is to the business, the shorter the desired RTO. Once again the shorter the RTO the greater the cost, generally speaking. Each system/service used by the business will usually have an RTO and RPO defined and agreed with the business as part of the overall Service Level Agreement (SLA) for that system/service. These RPO and RTO figures are then used to determine the most appropriate level of technical resilience and capability for each of the above technology layers often needing to consider these aspects across as well as within layers. The following sections outline considerations when implementing each of the above layers in order to achieve the defined RPO and RTO criteria for each service. Network As mentioned earlier, the network underpins all layers. Without connectivity it does not matter whether all components within all the other layers are operational or not. No network connection means that the user will be unable to logon to their PC and access the required application(s) across the network. (There may be levels of contingency that allow the user to log on to their PC locally and use installed September 2012 Page 3 of 8

5 applications such as Microsoft Word and Excel, or a cut down version of a business application, but most business services rely on applications and data hosted centrally within the data centre.) Resilience at the network layer is achieved through diverse routing of primary, secondary and even tertiary links, alternate routes between locations, multiple instances of network switches and other equipment. Modern network technologies use a combination of physical hardware and complex software appliances to improve network resilience, capacity and throughput, combined with increased levels of monitoring to ensure optimal service is available all the time. Investment in a robust and secure network infrastructure is a fundamental element of any technical continuity strategy. Data Centre The data centre is another key component in ensuring resilience and security of service provision. A properly constructed data centre not only provides physical security for expensive equipment and confidential data, it also provides an optimal environment within which to host and run the core technology components comprising the other layers (primarily network, storage and servers). Air conditioning within the data centre environment will ensure that the correct temperature and humidity levels are maintained at all times which prevent equipment overheating and failing. Appropriate levels of clean power feed ensure that systems are always on and not subjected to spikes in power levels. No power means no systems/service. Power provision will include an Uninterruptable Power Supply (UPS essentially a collection of batteries) which can power the systems for a short period of time (usually minutes) in the event of a failure of the main power supply. This short period of time is sufficient for locally sited generators (where installed) to start up and take over the power provision for several days until the main power supply is restored. If no generators are installed, the short period of time allows systems to be shutdown in a controlled manner, thus ensuring data is not corrupted or lost. Other features of the data centre include its location (away from hazards and risks), environmental monitoring and alerting, additional capacity for growth, etc. There are recognised standards ( tiers ) of data centre construction and provision that provide increasing levels of security and resilience Tier 1 through to Tier 4. Of course a single data centre itself can be viewed as a single point of failure, which is why most high availability services are implemented across two, and sometimes more, data centres. This enables further operation of systems across multiple data centres in a coordinated manner, thus improving resilience and ease of recovery in the event of a disaster. However, there is still much work to do within the other layers to realise coherent operation across dual data centres. All of the above features are critical to ensuring continuity of technical and therefore business services. Investing significant amounts of time and money in technology for the other layers without investing in data centre provision is a false economy, and whilst many organisations recognise this and make the investment they do not always realise the full benefits. September 2012 Page 4 of 8

6 Storage The storage layer is probably one of the most commonly mentioned components when it comes to discussions about business continuity, disaster recovery and resilience. There is often the misconception that ensuring a secure and resilient storage layer will resolve the majority of concerns about disaster recovery. Whilst the storage layer is most certainly a key component of service resilience and DR, it cannot achieve the required levels of continuity alone. The most common technology used within this layer is the Storage Area Network (SAN). A SAN is essentially a large number of traditional computer discs all connected together and held centrally, usually within a purpose built data centre. Access to the collection of discs is provided via specialist high speed connections. An individual SAN is then able to provide a good level of resilience by applying various approaches to storing the business data on the collection of discs (e.g. writing two copies of the data onto two separate discs at the same time known as mirroring). However, whilst a single SAN provides a level of resilience it does not prevent outages occurring as a result of loss of that SAN for any reason (e.g. failure of the mechanism that coordinates the writing of the data to the discs, multiple disc failures, etc.). To increase the level of resilience at the storage layer further technology components and processes are required. One method is to add a second SAN and make a copy of all of the data from the first SAN onto the second again this is referred to as mirroring and can take several forms. This second SAN would ideally be hosted at a different, geographically remote, location from the first SAN to prevent both SANs being lost in the event of a disaster at one location (e.g. fire, flood, failure of the air conditioning within the data centre causing equipment to overheat, etc.). Copying all data to the second SAN usually requires both SANs to be of the same specification and capacity (although there are technologies that allow heterogeneous storage arrays to be interconnected). It also requires high specification connectivity between the SAN locations to ensure the volume of data can be managed consistently and in a sufficiently timely manner. Copying of data to the second SAN in real time for example places a limit on the physical distance between two SANs, although recent technical solutions at the network layer are now offering alternative approaches to mirroring of data. How soon the data is mirrored across to a second SAN is a key factor in enabling the RPO, but comes at a price. Features within the SAN equipment itself enable mirroring or snapping of images of data and systems from one SAN to the other, but the frequency with which this can be achieved will depend upon a number of factors including the number of systems, the amount of data involved, the capacity of the connection between the SANs, the capacity of the SAN equipment itself to actually read and write the data being copied, etc. Whilst mirroring of data ensures copies of that data are more likely to be available in the event that one source fails, it does not protect the data from corruption. Incorrect modification or deletion of data in one location will be mirrored to the copy instantly if real time mirroring is enabled. And that corruption may not be detected for some time. September 2012 Page 5 of 8

7 The risk of loss or corruption of data is mitigated using a backup procedure, which takes a copy of the data at specific points in time e.g. daily, every 12 hours, etc. In the instance of corruption, the hope is that any issue is detected before the next backup is taken, at which point a known good copy of the data can be restored from the last backup. If a backup has already been taken which includes the corruption it may be necessary to go back to previous backups to obtain a known good copy of the data. This of course means that any other updates since the backup was taken will have been lost and data may need to be re-entered manually following restoration of that data. There are a number of mechanisms and approaches to enabling data backup, which are too detailed for this paper, but needless to say a sufficiently robust backup regime is key to continuity and disaster recovery. The frequency with which backups are conducted will also be a factor in determining an appropriate level of RPO. And don t forget data restoration. It is no good backing up data if it cannot be read back from the backup medium, which is why a good backup strategy and plan will conduct validation and test restoration processes on backed up data on a regular basis. One point on storage which may seem obvious but is worth stating: applications must be configured to store their data at the storage layer in order to benefit from the increased levels of resilience and security. Holding data on the local disc drives of desktop machines or servers sitting under a desk may be a cheap solution, but does not provide the levels of resilience or security typically required by the business. Once again, having a resilient and highly available storage layer is a key component of continuity and disaster recovery, but does not achieve a fully resilient service alone. It requires coordinated implementation of technologies and configuration across all the layers. Computer Servers This layer includes the main computer systems that host the business applications and tools. These applications will include more generic systems such as database servers (Microsoft SQLServer, Oracle, etc.), communication and collaboration tools ( servers, document management systems, etc.) and business applications (patient administration systems, departmental systems, etc.). In order to ensure that these applications and tools are available to the business as and when required (usually within defined and agreed Service Level Agreement (SLA) thresholds) the computer server layer, often referred to as the compute layer, needs to have a level of resilience designed and built into it as well. Improved levels of resilience can be achieved by using more than one physical computer server to support a single application. In this way failure of a server or a component within a server does not prevent the hosted application from running. This is typically known as clustering where two or more physical servers are connected together to run the application processes required. Failure of one component of the cluster results in the workload for that component being distributed across the remaining physical computers. September 2012 Page 6 of 8

8 However the latest trend is to move to a virtual server environment often referred to as virtual machines or VMs. This model is similar to clustering in that multiple physical servers are used to host a single, or indeed multiple, applications with failure of any one component not preventing failure of the applications. With VMs the physical servers are reduced to small form factor devices (printed circuit boards referred to as blades ) which plug into a shelf in a rack. Each device is a powerful computer with its own CPU and memory, and multiple blades can be plugged into a single shelf on a rack. Virtual server software then runs across all the blades and can selectively allocate resources (CPU, memory, etc.) to applications dynamically based on various rules and software configuration. Further devices can be added as required and will simply be included in the pool of resources managed by the virtual server software. Examples of such software are VMWare s vsphere and Microsoft s Hyper-V. Applications running within VM environments are typically not aware that the computer server on which they are running is actually software. However, application vendors do need to test and warranty their applications to run within a VM environment in order for them to commit to provide maintenance and support, and this should be verified with the vendor before moving applications to a VM environment. There are instances where vendors will not support, or do not recommend, use of VMs for hosting of their applications. One example is the Microsoft SQLServer database, which tends to split opinion of whether it should be hosted within a VM environment or not, depending upon the level of processing it must carry out. Whilst many organisations are moving towards use of VMs, it is not unusual for a number of applications to still be running on non-vm (physical server) environments for various reasons. However it is probably true to say that within the health community current VM deployments operate wholly within a single data centre, and whilst the level of resilience within that environment is good, no crossdata-centre failover capability tends to be enabled. Failure of the VM compute environment in one data centre will require a level of manual effort to restore systems and services at the other data centre. The time it takes to achieve this restoration will directly impact upon the Recovery Time Object (RTO) for any applications hosted in that environment. There are tools available that can reduce this time, but without them it can take hours or even days to restore from a complete failure in one data centre. As with the storage layer, having a resilient and highly available compute layer is a key component of continuity and disaster recovery, but does not achieve a fully resilient service alone. It requires coordinated implementation of technologies and configuration across all the layers. Applications As alluded to in the preceding sections, implementation of a highly resilient, highly available compute and storage layer does not necessarily mean that the applications and business services are themselves resilient or highly available. September 2012 Page 7 of 8

9 Whilst simple applications can comprise a single executable file which runs on one server and accesses data from a single location/database, more complex applications are comprised of a number of components, each of which may require its own dedicated server with communications between those components. Hosting applications within a VM environment and storing data on a SAN will improve the availability of that application, but unless the application is architected and design to be resilient to a failure of any one component (e.g. due to a software bug, excess load, performance degradation, etc.) can cause the whole application to fail even though the underlying servers and storage continue to run normally. This is exactly the same single point of failure concept as described in the network layer above. Applications can be designed such that there are multiple processes sharing a common workload, which can be distributed across a number of (virtual) servers, which in turn could be located across multiple data centres. Failure of any one component does not affect the overall operation of the application and therefore service. Add in a resilient storage layer which replicates data to an alternative location, combined with intelligence in the compute and/or application layer which knows which is the primary source of data and switches to use it automatically and there is a significantly increased level of resilience and availability of the service. In reality most applications are not architected and designed to operate in this fashion and primarily rely on the underlying server and storage infrastructure to ensure that they are able to continue running. In the event of a failure/disaster manual intervention is required to achieve recovery, and typically involves bringing up another instance of the application at an alternate location, ensuring the data is up to date and then making the application available to end users. There are a number of tools available to assist in this recovery process, but they mostly require levels of capability to have been implemented at the storage and compute layers in order for them to operate successfully. An example of one such tool from the VM suite is Site Recovery Manager (SRM). Whilst a number of these aspects will be addressed as part of a strategic IT roadmap, much of the responsibility in this layer lies with the business applications themselves. No amount of technical resilience can mitigate a poorly architected and designed application, which is why preference should always be given to deploying enterprise level applications where high availability and continuity of service is a critical business requirement. Integration This is the glue that enables the flow of information between application layer components, which in turn enables business process. Failure of the integration layer or a component therein can have serious consequences for the business. Within a healthcare environment for example, failure of the integration interface between two critical business systems can mean that the whole business must revert to paperbased processes. No small undertaking for a large healthcare organisation. Mitigating the risk of failure at this layer is similar in approach to that for the application layer. Whilst a level of resilience can be achieved through the underlying host infrastructure, the integration solution itself must be architected and designed to withstand failure of any one component. This will be September 2012 Page 8 of 8

10 especially important for an environment that has a large number of interfaces, all of which are dependent upon other systems for information flow. Implementation of the integration layer is typically performed using an off-the-shelf integration product such as Intersystems Ensemble or Microsoft s BizTalk. These products are usually referred to as an Integration Engine and enable the interoperability between business systems. Although largely invisible to the business (unless there is a failure) the integration engine is increasingly becoming one of the critical systems upon which business process flow, and therefore continuity, is dependent. Presentation and Access Devices There is a limited amount that can be done at this level. The presentation layer (the application screens that the user accesses) will mostly be driven by the application layer, so resilience at this level will be wholly dependent upon lower layers of the environment as discussed above. The access devices themselves are susceptible to wear and tear especially when considering portable devices such as laptops, tablets or smartphones. The most cost effective mitigation for failure at this layer is a comprehensive Service Desk function offering a well-managed end user device estate that includes maintenance, support, repair and replacement on a timely basis (again within SLAs agreed with the business for each service). In Conclusion There are a wide variety of existing and new technologies that enable and inform business and technical continuity capability for an organisation. Technological innovations in the area of network, storage and compute such as Cisco s Unified Computing System (UCS) which tightly integrates all three layers into a homogenous service makes it easier to procure and deploy high levels of infrastructure resilience and availability. Discussions with technology partners will help inform decisions in this area, ensuring that an increased reliance on IT systems for the efficient and cost effective delivery of healthcare services are supported through improved levels of resilience and available of those systems and services. Partnering for improved delivery and management of those services also offers opportunities for increased availability and resilience, as well as reducing total cost of ownership, in both on premise and off premise models. Within the current economic climate investment in technology plays a critical part in ensuring cost effective continuity of services across all areas and geographies of the business, ultimately reducing operational costs and overall cost of ownership, whilst increasing performance, resilience, scalability and availability. September 2012 Page 9 of 8