Protecting PSAP Applications from Downtime

Transcription

1 W H I T E P A P E R Protecting PSAP lications from Downtime Strategies for safeguarding mission-critical applications that public safety call takers, dispatchers, and first responders depend upon Uptime. All the time.

2 Protecting PSAP lications from Downtime 1 Contents Introduction... 2 All PSAPs are not alike... 3 Understanding the nines... 3 Standalone conventional servers... 4 Hardened servers... 5 Cold standby... 5 Server sprawl... 6 The bottom line on stand-alone servers... 6 Failover clusters... 6 Active-passive and active-active clusters... 7 Planned maintenance with less downtime... 7 Geographic clustering... 7 Cluster configurations... 7 Limitations... 7 The bottom line on failover clusters... 8 Virtualization... 8 Why virtualization?... 8 What is virtualization?... 9 Less planned downtime Centralized management What about availability? Virtualization clusters High availability for virtualized infrastructures Software fault tolerance Protecting the management console Limitations The bottom line on virtualization Uptime assurance A review of what we have learned Failure prevention for mission-critical applications About Stratus Technologies ftserver Avance software Conclusion... 20

3 Protecting PSAP lications from Downtime 2 Introduction If you are considering a technology refresh for legacy software or you are about to publish an RFP for updating all or part of your PSAP operations, it is important to understand what it takes to achieve the highest levels of uptime from your applications and systems. The nation s first call was placed in 1968 and since then the term has become synonymous with public safety. According to the National Emergency Number Association, (NENA), there are over 6000 Public Safety Answering Points (PSAPs) in the United States and it is estimated that more than 240 million calls are made each year. Advances in computer technology and the proliferation of software applications have helped to increase the efficiency and effectiveness of PSAPs and first responders. This includes: increasingly sophisticated Computer Aided Dispatch (CAD) applications, computer-based telephony, computerized radio control; ancillary capabilities such as automatic number verification, location mapping, automatic vehicle location and real-time access to dispatch logs; and police, fire, rescue, motor vehicle, and court records. Perhaps the greatest challenge facing PSAP organizations is the continuing evolution of the Internet and wireless technology. The Next Generation (NG 9-1-1) initiative addresses the need for PSAPs to be able to receive, capture, and route digital communications including voice, text, picture, video and caller location from virtually any communications device. The FCC estimates that over one-third of calls now originate from wireless devices and the expectation is that the percentage will continue to grow with the transition away from wireline telephone service. Call takers, dispatchers, and first responders all depend on the applications and information controlled by the computer servers in the backroom of the PSAP or communications center. These servers capture emergency dispatch information from multiple sources and make it available on the dispatcher s desktop, mobile devices in response vehicles, and PDAs in the hands of officers. Computer-based applications provide call takers access to wired and wireless telephone networks. They give dispatchers control over their radio communications systems. They host notification applications such as fire station alerting, reverse 9-1-1, and paging. They also enable access to regional, statewide, and national criminal justice systems. It is no longer just the CAD application that is mission critical; it is also the applications and records management systems that provide the information that assures the efficiency, effectiveness, and safety of emergency response operations. The failure of a computer system not only impacts the ability to capture and disseminate vital information. It can jeopardize the safety of first responders when information such as prior location history and fire inspection records is not accessible. In this age of litigation, any system failure that impedes the response to an emergency may also result in potential law suits. This white paper addresses different strategies to improve the availability of mission-critical applications and minimize the possibility of disruptions caused by server failure. It describes the advantages and disadvantages of dedicated servers, cluster technology, and virtualization.

4 Protecting PSAP lications from Downtime 3 All PSAPs are not alike There is no such thing as a one-size-fits-all solution for improving system availability that meets the needs of all Public Safety Answering Points. Budgets, staffing, application requirements, availability of IT resources, and populations served can be vastly different. However, what all dispatch centers share in common is the mission-critical nature of their work and the need to assure the highest levels of systems and application availability. When it comes to protecting lives and property, public safety executives must take every step possible to minimize the chance of computer downtime. Systems must be up and running 24x7, 365 days a year. A major metropolitan area, serving a population of over one million, and handling over 1000 emergency call per day, is going to have different requirements than a town or county serving a population of 50,000 and handling less than 100 calls per day. For a major metropolitan PSAP, a one hour server outage during peak hours could impact the ability to respond effectively to 50 to 75 emergency calls. For a small PSAP the same outage may only impact a handful of incoming calls. When choosing CAD (Computer Aided Dispatch) technology, PSAPs have a variety of options. Dispatch Magazine s CAD Software Resources Directory lists over one hundred CAD software providers. Solutions vary significantly in terms of cost, functionality and dependability. Some CAD applications provide a higher degree of resiliency to application crashes and server outages, typically by replicating active call data across workstations. However, if the central server goes down, users may be unable to access archived historical and reference data. For PSAPs with less robust technology, call capture and dispatch capabilities are lost when the server goes down. The information that follows will help dispatch centers of all sizes reduce the risk of server downtime and the vital applications that run on them not being available. Understanding the nines Any discussion of server availability will eventually get around to talking about the nines or, put another way, the percentage of uptime that can be expected from a server environment. Figure 1: Understanding the Nines Availability Uptime Level Average Yearly Downtime Conventional Server 99% 87 hours, 40 minutes Hardened Server with RAID storage hours, 48 minutes High Availability Fault Tolerance "Continuous Availability" 99.9% 99.95% 99.99% % % 8 hours, 46 minutes 4 hours, 23 minutes 52 minutes, 36 seconds 5 minutes, 16 seconds 31.6 seconds A typical off-the-shelf x86 server provides an uptime level of 99% which sounds great until you do the math. 99% availability works out to be an average downtime of 87 hours and 40 minutes. And, you don t get to choose those hours. Based on your average hourly incoming call rate, how many calls would be impacted?

5 Protecting PSAP lications from Downtime 4 It s relatively easy to go to 99.5% availability. All it takes is a hardened server with redundant power supplies, fans, and a RAID storage array coupled with good system administration practices. While this might seem like a big improvement, consider the number of emergency calls affected and potential response delays caused by almost nine hours of downtime. Climbing up the scale to 99.95% uptime often involves using failover cluster technology. If a server or operating system fails, the affected applications will automatically failover and restart on a healthy system. Some cluster solutions claim up to 99.99% availability, but achieving only 52 minutes of downtime a year requires a lot of specialized IT expertise and effort. There are other considerations in addition to availability. For example, if a server goes down, any information that has been captured by the call taker or dispatcher that has not yet been saved to disk, either automatically or manually, is lost. A server crash can result in lost call intake records, dispatch information and, in a worst case scenario, data corruption. Equally important is understanding the IT resources and skill levels needed to maintain different system architectures, and if routine maintenance requires taking applications offline. This is a greater issue for PSAPs than the typical business operation because in a 24x7 365 day operation, there are no off hours or convenient times for applications to be down, planned or otherwise. For mission-critical applications such as public safety, the goal should be 100% availability. Realistically, % often called continuous availability is considered the gold standard. This level of availability is within the reach of all PSAPs, and it can be achieved at lower cost and can be easier to administer than systems providing lower levels of availability. To achieve %, you need to go beyond simply reacting to operating system and hardware failures; you need a means to detect and prevent failures before they happen. Regardless of size, any PSAP should be satisfied with nothing less than % availability for all mission-critical applications. It is achievable even for a small dispatch center with limited staff and budget. The following information should help you understand the options and how to costeffectively achieve at least one or two more nines of availability. Standalone conventional servers Figure 2: Conventional Server PCI CPU Chipset Embedded I/O Memory A conventional x86-based server delivers an average uptime of 99%. Data is typically stored on RAID (redundant arrays of independent disks) storage devices. However, a server failure will stop all processing and users will lose access to their applications and information.

6 Protecting PSAP lications from Downtime 5 Figure 3: Calls Potentially Affected at 99% Availability) Annual Call Volume Calls Affected by Outage 50, ,000 1, ,000 2, ,000 5,000 Hardened servers Priced aggressively, some servers offer reliability features such as redundant fans, redundant power supplies, and hot-plug PCI cards to achieve improved reliability over standard servers. Hardened servers provide limited protection against catastrophic hardware failures; however, they do not prevent system crashes resulting from application failures, or transient (randomly occurring) hardware errors such as memory failure or disk drive crashes. Any data not yet written to disk at the time the server fails is lost. Routine maintenance typically requires taking the server offline. Cold standby A cold standby server provides a backup if the primary server fails. The cold standby server is loaded with the same operating system and applications as the primary server. Software upgrades and patches performed on the primary serve must also be performed on the cold standby server to assure it functions properly when needed. Once configured, the cold standby server is turned off until needed. Microsoft as well as some other software providers do not charge an additional license fee for loading software applications onto a standby server as long as it s only powered up if the primary server fails. Figure 4: Calls Potentially Affected at 99.5 % Availability) Annual Call Volume Calls Affected by Outage 50, , ,000 1, ,000 2,500 If the primary server fails, you need a skilled administrator to either connect the second server to a shared disk array or move disks from the primary server to the standby server. Using a backup cold standby server may improve availability by reducing downtime but it also can pose risks, especially for PSAP sites that lack the continuous presence of skilled IT personnel. A cold standby server may get you up to 99.5% availability. This equals close to 44 hours of down time per year, which is well below continuous availability levels and well short of acceptable performance for your mission-critical public safety applications.

7 Protecting PSAP lications from Downtime 6 Server sprawl A large PSAP is likely to have its own data center with racks of servers. With the proliferation of low-cost x86 servers, the historical approach to adding new applications or supporting additional users has been to add more servers. Each server is dedicated to supporting one application, which helps to minimize potential resource conflicts. If that server goes down, only one application goes down with it. This approach does nothing, however, to improve the availability of mission-critical applications. The one application/one server approach commonly resulted in a proliferation of hardware, called server sprawl. Each server typically operates well below capacity because it s only supporting one application, and because of continual dramatic increases in CPU processing power. There are also higher operating costs incurred. It s expensive to operate large server farms. The cost to power and cool each server in the data center can exceed the original cost of the server. The bottom line on stand-alone servers The downside of running applications on a dedicated server is that there is a single point of failure if the server goes down. The use of a cold standby server helps to reduce the amount of time applications are unavailable. Both approaches, however, fall short of what should be considered best practices for maximizing availability. Failover clusters High-availability clusters, also known as failover clusters, provide a means of connecting two or more servers (nodes) in a single network. If a server fails, another server in the cluster will automatically take over, ideally with minimal disruption to users. This process is called failover. In comparison, if a stand-alone server crashes, the applications running on it will be unavailable until the server is fixed. Figure 5: Failover Cluster Heartbeat Storage Area Network Computers in a cluster are connected by a local-area network (LAN) or a wide-area network (WAN) and managed by cluster software. Failover clusters require a Storage Area Network (SAN) to provide the shared access to data required to enable failover capabilities.

8 Protecting PSAP lications from Downtime 7 Servers (nodes) in a cluster communicate with each other by continually checking for a heartbeat that confirms other servers in the cluster are up and running. If one of the cluster nodes becomes unavailable because of a hardware or software failure, another node takes over and access to network resources is transferred to that active server. Figure 6: Calls Potentially Affected at 99.95% Availability Annual Call Volume Calls Affected by Outage 50, , , , At least one other server in the cluster must be assigned as the failover server and be loaded with the same applications as the primary server. Windows Server, Linux, and VMware all support some form of clustering. Predicted availability for a failover server cluster is 99.95% or about 4 hours, 23 minutes of downtime per year. Active-passive and active-active clusters There are two basic types of failover clusters. In an active-passive (asymmetric) cluster, the only function of a standby server is to take over for another server in the event of a failure. It functions as a hot backup, ready to automatically take over if the primary serve goes down. In an active-active (symmetric) cluster, typically, each server is a primary server for one or more applications. If the active server fails, the standby server takes over and continues to process its assigned applications as well as the applications from the failed server. In both scenarios, software must be licensed on all servers where applications reside. Planned maintenance with less downtime Another advantage over standalone servers is that clusters reduce downtime required for routine planned maintenance such as installing patches and upgrades because each application resides on more than one server. Geographic clustering Geographic clustering extends failover capabilities to servers at a different location. It is often used as part of a disaster recovery or contingency plan to enable continuation of operations at a backup dispatch center. The backup location is typically located far enough away from the primary PSAP to limit the likelihood of the same disaster disabling both sites. For example, the backup location should be on a different power grid to minimize the likelihood of both locations being affected by a power outage. Cluster configurations The most common configuration for a failover cluster is two servers (nodes), the minimum required to provide backup. However, clusters can incorporate many servers. For example, Microsoft Windows Server 2008 supports up to 16 server nodes in a cluster. Limitations Failover clusters are designed to reduce unplanned downtime through automated recovery from failure; however, there is still typically some disruption of service. The disruption that a user experiences can range from seconds to minutes or longer for an extensive application restart. Users can also experience slower performance because the failover server is now performing

9 Protecting PSAP lications from Downtime 8 the work of two servers. Databases are particularly susceptible to long recovery and data corruption. Another significant limitation of this approach is that in-flight data information captured by call takers and dispatchers but not yet written to disk can be lost. In public safety, loss of in-flight data can impact the ability to respond to an emergency and could have subsequent legal ramifications if information related to an emergency response is lost. The biggest impediments to clustering are that it requires a high degree of technical skill to configure a cluster, create and test failover scripts, test functionality, and properly manage the cluster on an ongoing basis. Because they are complex and challenging to manage, clusters can be susceptible to failure. Whenever hardware or software changes are made, failover scripts must be updated and re-tested, requiring planned downtime. Support and management requirements should be considered when evaluating the total cost of deploying and maintaining failover clusters. In addition, applications may need special coding to ensure that when a failure occurs, the failover process is transparent to the user and the application remains available. Some applications may not be able to support clustering, even with code changes. The requirement for a Storage Area Network also adds to the cost and complexity of implementation. For all the above reasons, failover clusters are an impractical choice for PSAPs with limited or no dedicated IT resources. The bottom line on failover clusters Predicted availability for a failover server cluster is around 99.95%, or about 4 hours, 38 minutes of downtime per year. That s at least one nine better than a stand-alone server. However, this still falls short of what should be acceptable to meet the mission-critical requirements of a Public Safety Answering Point, where any disruption of service can have tragic consequences. Once again the question is how many calls would be impacted during four hours of downtime? There are alternative approaches that provide superior availability and are much easier to manage. Virtualization Why virtualization? Before diving into what virtualization is, it s worth talking about why server virtualization is becoming increasingly prevalent in data centers, and for hosting web-based services. It provides two significant benefits: it saves money and it s easier to administer. For these reasons, if you are planning to update aging hardware or a major upgrade to your dispatch center applications, you should consider virtualization. As discussed earlier, the need to add more applications or system users was answered with more servers. Having fewer applications per server minimized potential resource conflicts, as well as user impact if that server failed.

10 Protecting PSAP lications from Downtime 9 However, as x86 servers become increasing more powerful, this approach became increasing inefficient and expensive. Without virtualization, the average standalone server is only being used to about 15% of its capacity and data centers have experienced significant increases in the cost of energy required to power and cool these servers. As energy costs continue to climb, this has become a major expense. Additionally, more servers require more IT resources to maintain them. Failover clusters were often implemented to help reduce the likelihood of applications going down; however, this did not reduce server proliferation and added even more complexity to maintaining systems and applications. Government organizations, in particular PSAPs, are caught in the middle by conflicting mandates. On one hand, as a result of reduced funding and budgets, they must try to squeeze more life out of aging systems and applications. On the other hand, they need to incorporate new technologies and applications to respond to the increasing volume of emergency calls from the ever increasing variety of digital devices that are now mainstream. And, information sharing between agencies requires more advanced technologies. What is virtualization? Virtualization is not a new technology. It was first introduced in the 1960s by IBM to boost utilization of large, expensive, mainframe systems by partitioning them into logical, separate virtual machines that could run multiple applications and processes at the same time. Figure 7: Server Virtualization Hypervisor Virtualization is the partitioning of a physical server into more than one virtual server. The software used to partition the physical server into multiple virtual servers is called a virtualization manager, or more commonly, a hypervisor. Each virtual machine (VM) can interact with other devices, applications, data, and users as though it were a stand-alone server. The hypervisor emulates the server hardware allowing more than one operating system to share the resources of one physical server. The operating systems running under the hypervisor are called guest operating systems. The hypervisor controls how much access the operating systems and applications have to the physical server (also called a bare-metal server) and other hardware resources such as memory and disk.

11 Protecting PSAP lications from Downtime 10 Each virtual machine (VM) is isolated from the other virtualized machines. If one VM crashes, it can be independently rebooted without affecting the others. However, if a physical server crashes, all of the VMs will go down, as well. In addition to virtualizing servers, virtualization techniques can also be used to combine multiple physical resources into a single virtual resource. Storage virtualization enables network storage resources to be pooled into what appears as a single storage device simplifying access and management of these resources. VMware is currently the virtualization market leader with an estimated market share of 80%. Other virtualization solution providers include Citrix, Microsoft, IBM, and Red Hat. Less planned downtime Live migration, using products such as VMware vmotion, lets you move VMs from one physical server to another, with users perceiving little or no impact on application availability or performance. This allows routine maintenance and upgrading of the host server or the hypervisor, but does not allow servicing the guest operating system or applications without taking them off line. Centralized management Some virtualization solutions offer a means for centralized management or resource pools. For example, the VMware vcenter Server and Microsoft System Center Virtual Machine Manager provide the capability to monitor and manage host servers, virtual machines, storage, and networks from one central server. Some virtualization solutions enable point-in-time rollbacks, which are useful if data corruption has occurred. Rollback lets an administrator revert the virtual machine to an earlier known state before the data corruption occurred. What about availability? So far, we have talked about how virtualization can save money, optimize the use of resources and simplify administration, but what hasn t been discussed is the impact on availability. The paradox of virtualization is that it removes hardware dependencies while making hardware more important. A collection of virtual machines relies even more heavily on reliable hardware, because more applications are running on each physical server. When a physical server supports many workloads, even less critical applications become critical when viewed as part of the collective PSAP operation. If a physical server goes down, all the virtual machines and applications running on them go down as well. Therefore, steps must be taken to increase the reliability and availability of virtualized applications. Virtualization clusters A typical datacenter environment will utilize a solution such as VMware vsphere Cluster Architecture to bring together multiple servers, storage arrays, and networking components into a resource pool that can be shared by virtualized applications. This provides the foundation for implementing high-availability, fault-tolerance, and centralized management features.

12 Protecting PSAP lications from Downtime 11 Figure 8: Virtualization Cluster Hypervisor Hypervisor Hypervisor Storage Area Network High availability for virtualized infrastructures VMware High Availability (HA) is an example of a virtualization solution with failover and restart capabilities that protects applications against unplanned downtime caused by failure of a guest operating system or physical server. Upon detecting a guest failure, VMware HA will automatically restart virtual machines after a pre-determined interval. In cases where VMware HA detects a server hardware failure, it automatically restarts the affected virtual machines on another server in the resource pool. Figure 9: Calls Potentially Affected at 99.99% Availability) Annual Call Volume Calls Affected by Outage 50, , , , When a physical server shuts down, the virtualization software uses the storage area network (SAN) to retrieve an image of the virtual machine or machines affected. A restart of the applications is then initiated on another server or servers. However, when a virtual machine reboot is necessary, the information about the application state, as well as in-flight data not yet written to disk, will be lost. Virtualization HA has two advantages over failover clusters. It does not require duplicate copies of the guest operating systems and applications on more than one machine. And, it does not require creating and maintaining complex failover scripts. Virtualization failover capabilities can also be used for disaster recovery. Virtual machine images can be replicated and restarted on a server in a protected location. Data can also be replicated to storage devices elsewhere on the network.

13 Protecting PSAP lications from Downtime 12 Software fault tolerance Figure 10: Calls Potentially Affected at % Availability Annual Call Volume Calls Affected by Outage 50, , , ,000 5 A software-based fault tolerant solution, such as VMware Fault Tolerant software, depends on software lockstep to achieve fault tolerance. Availability can be improved by using software lockstep; however, it has its drawbacks. Software lockstep uses replication and a dedicated Ethernet connection to provide a heartbeat that keeps the primary and secondary virtual machines in sync. This technique imposes additional overhead on the CPU, network and I/O processing that can slow down application response time. It also requires at least one duplicate physical server and a duplicate copy of any guest operating systems and applications on the second system. Ironically, this requires adding physical servers - not consolidating them. A lesser-known drawback of VMware Fault Tolerant software lockstep is that it limits a virtual machine to the use of a single processor core. Almost all servers today utilize multi-core processors. Virtual Machines can t use the symmetric multiprocessing (SMP) or multi-core capacity of a CPU. This limitation restricts the application from scaling up, if needed, to provide higher performance, thus sacrificing one of the primary benefits of implementing virtualization to assure availability. Protecting the management console Figure 11: 9 Virtualization Management Console Virtualization Management Console Hypervisor Hypervisor Hypervisor Storage Area Network

14 Protecting PSAP lications from Downtime 13 Lastly, the management platform used for monitoring, managing and provisioning the virtualized resource pool should be protected against hardware failure. If, for example, the VMware vcenter Server crashed, individual servers and virtual machines in the virtualization pool will continue to operate, but centralized management functions will be lost. This can greatly impair the ability to manage a large data center such as those supporting PSAPs in large metropolitan areas. To mitigate the risk of this potential single point of failure VMware offers vcenter Server Heartbeat. The vcenter Server Heartbeat architecture requires a primary and a secondary server. A dedicated network channel is used to maintain a heartbeat between the paired servers and to maintain an up-to-date copy of the primary server on the backup. lication management software identifies and monitors the key components of both servers, alerting the administrator to any performance issues that may impact availability and allowing the secondary server to take over if the primary Heartbeat server fails. Limitations The primary purpose of virtualization is to reduce the costs and the complexity of managing data centers. The failover capabilities provided by virtualization software are a useful defense against failure for non-mission-critical applications where some downtime is acceptable. Failover and restart can require precious minutes. In addition, problems that caused a virtual machine to fail can be propagated to the secondary virtual machine upon restart. Software-based high-availability or fault-tolerant solutions are also not designed to deal with operating system and application bugs, or transient (randomly occurring) hardware errors. This includes device driver malfunctions, which may result in downtime and data corruption when left uncorrected. The source of failure that required a virtual machine to be restarted on the same server or on another server could be replicated along with the same threats to uptime and data integrity. Lastly, the ease of creating new virtual machines has led to the coining of a new term, virtual sprawl. This refers to the fact that virtualization makes it so easy to create new virtual machines, that they can proliferate quickly. That s why it s important to use an appropriate virtualization management tool, and configure it on a platform that ensures management capabilities are always available. The bottom line on virtualization Virtualization is becoming the default approach for new data center server deployments. Server consolidation through virtualization will enable PSAPs to dramatically boost hardware utilization and reduce costs. These savings are not limited to hardware, but also extend to support costs and reduction in expenses such as space, power, and cooling. However, even high-availability virtualization solutions fall short of providing the % availability required by Public Safety Answering Points for mission-critical applications because they are designed to react to failures rather than identify and prevent potential causes of failures. Software-based fault tolerance can provide up to % availability, but the techniques used by solutions such as VMware Fault Tolerant software are resource intensive using up to 20% of processing resources. This can noticeably impact application performance experienced by users.

15 Protecting PSAP lications from Downtime 14 If you are a larger PSAP with a data center full of servers, it s probably only a matter of time until you consolidate servers and virtualize applications. You must take additional steps, however, to assure the highest levels of system availability. If you are a small PSAP with only two or three servers in a data closet, you may be thinking that you have neither the budget nor the IT resources required to improve the availability of essential applications. It is easier and less expensive than you may think. The next section discusses cost-saving ways to improve the reliability and availability of virtualized applications and to assure the continuous availability that mission-critical public safety applications require. Uptime assurance In this section, we will discuss how organizations across the globe, including hundreds of Public Safety Answering Points in North America, are achieving the gold standard of % availability or higher, and how they have simultaneously reduced the complexity of system administration and lowered the total cost of ownership for the systems supporting their missioncritical applications. A review of what we have learned Stand-alone conventional servers offer little protection against failure. When the server goes down, applications go down and data is lost. Failover clusters offer improved availability, but still fall short of providing the availability required for mission-critical applications. Additionally, they are complex to set up and maintain, may require the purchase of duplicate copies of software, and they are designed to react after a system failure has occurred, not to prevent it. Virtualization enables much more efficient use of computer resources and simplifies system administration. High-availability options are less cumbersome to administer than clusters, and do not require purchasing additional software licenses. Software-based fault-tolerant solutions have technical limitations and are resource-intensive, which can result in noticeable degradation of application performance. Failure prevention for mission-critical applications The key to achieving the uptime required for mission-critical applications is to not simply react to hardware and operating system failures but, instead, to detect and prevent failures from happening in the first place. About Stratus Technologies For over thirty years, customers around the world have come to rely on Stratus Technologies to assure the continuous availability of their mission-critical applications. Stratus solutions are designed to prevent downtime and data loss. Advanced, patented, uptime assurance technologies, combined with proactive availability management and monitoring services, provide the highest levels of availability for mission-critical applications. We call it uptime assurance.

16 Protecting PSAP lications from Downtime 15 ftserver The Stratus ftserver family of systems are engineered with fault-tolerant hardware, fail-safe software, and self-monitoring capabilities that prevent unplanned downtime and assure % uptime or higher. That s approximately 5 minutes or less of down time per year based on actual field measurements. They practically bullet-proof your mission-critical application and infrastructure against unplanned downtime. Figure 12: Calls Potentially Affected at % Availability Annual Call Volume Calls Affected by Outage 50, , , ,000 5 The ftserver family of systems leverages off-the-shelf technologies, including Intel highperformance multi-core processors, in a modular physical design. ftserver systems are certified to run Microsoft Windows Server (including Hyper V virtualization software), Red Hat Enterprise Linux, and VMware vsphere. Figure 13: Stratus Uptime Meter The Stratus Uptime Meter (SM) measures the actual availability of supported Stratus ftserver systems worldwide and is re-calculated daily based on actual identified service incidents during the preceding six-month period. The uptime meter is displayed on the Stratus website: Fault tolerant hardware An ftserver has two of everything working in synchronization in a single enclosure, including CPUs, memory, I/O, disks, and power supplies. In the event of a component malfunction, the partner component acts as an active spare. There is no performance degradation, no rebooting, no failover, no lost data, and no downtime. And ftserver systems are simple to set up and maintain, even without onsite IT support. They practically take care of themselves.

17 Protecting PSAP lications from Downtime 16 Figure 13: Duplex Hardware Components Single system view The ftserver operates as a single server. The network, operating system, and applications see a single system image so you don t have to modify your applications or license multiple copies of software. This makes ftservers easy to install and maintain, especially when compared to clusters, a major advantage for public safety organizations with budget and staffing constraints. Stratus Uptime Assurance software Stratus builds uptime assurance capabilities into every ftserver system through a set of tightly integrated technologies that detect and prevent many causes of downtime and data loss. It also simplifies monitoring and management of the server and enables remote service and support. A unique capability provided by the Stratus Automated Uptime Layer is the ability to diagnose potential problems down to the operating system level. This is made possible because Stratus Technologies is one of a small circle of companies provided Windows Server source code by Microsoft Corporation. The Stratus Automated Uptime Layer monitors and analyses more than 500 system events. This comprehensive monitoring and analysis is always active, not just after an event has already caused trouble. It prevents hardware and software failures from causing downtime. Unlike typical servers or clusters, the Automated Uptime Layer software handles most errors transparently. Automated uptime features enable the ftserver to ride through transient errors, protect against device driver failures, and prevent hardware and software failures from causing downtime. Lockstep technology Stratus lockstep technology assures data integrity if a server component fails or there is an operating system failure. The two halves of the server are so tightly coupled that memory is replicated in the same clock cycle so even data not yet written to disk is protected and preserved. Other high-availability solutions can t prevent data loss if there is system failure.

18 Protecting PSAP lications from Downtime 17 Planned maintenance without downtime Planned maintenance such as upgrades and software patching can be performed without downtime or slowing application performance when running Windows environments on an ftserver. Stratus Active Upgrade technology can logically split the fully redundant ftserver system into two independently running servers. While one server continues to run the application without interruption, software updates are applied to the other server. The two sides are then automatically resynchronized and returned to fault- tolerant operation as one logical server. Support Stratus is your single source of accountability for complex inter-related platform, system software, and operating system issues. Stratus supports Microsoft enterprise applications including Microsoft Windows Server and Hyper-V (we are a Microsoft Gold Certified Partner), Red Hat Enterprise Linux, and VMware vsphere. If the Automated Uptime Layer can t fully resolve a problem, the Stratus ftserver system automatically alerts the Stratus Support Center. Stratus uptime assurance experts proactively monitor Stratus systems 24x7 to help resolve problems before they impact system operations. Figure 14: Core Elements of Stratus Uptime Assurance Proactive Availability Management 24/7 monitoring: people and practices Detects, isolates and handles faults before they cause downtime Automated Uptime Layer Lockstep hardware withstands faults that would cause other servers to crash The Stratus ActiveService Network provides a secure, continuous link that enables online, around-the-clock monitoring and remote troubleshooting of systems regardless of their location. Our highly trained technicians can resolve most issues within 30 minutes. Less than 1% of system issues require an onsite visit. Virtually all issues are resolved without downtime. The Automated Uptime Layer s deep diagnostics capabilities automatically capture the information needed to pinpoint the root cause of problems down to the individual hardware component or software line of code. Then, Stratus support engineers work to provide a permanent fix that prevents recurrence of the issue.

19 Protecting PSAP lications from Downtime 18 Should a component fail, a replacement unit (one half of the ftserver) is automatically dispatched for next day arrival. The ftserver system s hot-swappable Customer Replaceable Units (CRUs) can be easily replaced without tools or computer skills. It s as simple as sliding one unit out and sliding the new unit in. No special knowledge is required. There is no disruption in application processing or loss of data. Why virtualized environments need ftserver It s easy to understand the advantages offered by ftserver systems compared to stand-alone conventional servers and high-availability clusters. Stratus ftserver systems provide higher levels of availability and are easier to maintain. But, what about virtualized infrastructures? Virtualization enables more efficient use of computer resources and simplifies system administration. But software-based fault-tolerant solutions have technical limitations, require the addition of more servers, and are resource intensive, which can result in noticeable degradation of performance. Lastly, the virtualization console manager is a potential single point of failure. Figure 15: Hardening Virtual Environments with Stratus Fault-Tolerance Public safety organizations considering virtualization should review their applications to identify the role each application has in assuring the efficiency and effectiveness of emergency response operations. lications should then be classified into three groups: Not critical to dispatch and first responder operations Some level of down time can be tolerated Mission critical any outage can negatively impact the ability to respond to an emergency. For mission-critical applications including the vcenter Server, Stratus ftserver systems assure the highest levels of uptime, are easier to install and maintain than software fault tolerant solutions, do not degrade performance, and are more cost effective.

20 Protecting PSAP lications from Downtime 19 Avance software Stand-alone conventional servers and cold standby servers have often been considered the only affordable option for many smaller PSAPs leaving dispatch centers vulnerable to system outages that impact their ability to respond to emergencies. While failover clusters offer increased protection from outages, they are costly to implement and maintain, and do not prevent loss of data if a server has to failover putting information captured during call intake at risk. Failover clusters also require a Storage Area Network (SAN) to provide the shared access to data required to enable failover capabilities. Stratus Avance software addresses the needs of smaller PSAPs with limited budgets and support resources. It provides a cost-effective solution that assures the availability of missioncritical applications. Avance software transforms a pair of ordinary servers into a high-availability solution that practically manages itself and delivers greater than 99.99% availability. That s less than one hour of downtime per year compared to an average of close to 88 hours of downtime for a stand-alone conventional server. Stratus Avance software manages two x86 servers as if they were one system. The software predicts and prevents downtime, data loss, and disruption of dispatch and emergency response operations. It delivers uptime that is far superior in performance to standby solutions. And, it provides operational simplicity and higher availability at a fraction of the cost of conventional high-availability clusters. No shared storage (SAN) is needed. Avance Automated Uptime Layer The Avance software s Automated Uptime Layer automatically detects, reports, and handles faults before they become a problem. Upon predicting a problem in one server, Avance software migrates applications to the unaffected server where processing continues uninterrupted. lications ride through most causes of system interruption, including disk and network subsystem failure. In the event of a complete server failure or natural disaster, the software generates an automatic rapid restart that takes just seconds. Disaster recovery To ensure minimal downtime and rapid recovery from a localized disaster, the Avance software split-site deployment capability allows you to locate servers up to three miles apart from each other. Unlike traditional disaster recovery solutions that simply keep a copy of data in backup, Avance software continuously replicates data to the second location in real time. Server consolidation Another cost-saving feature is the server consolidation provided by Avance software s built-in virtualization. Avance supports server consolidation by enabling multiple VMs to run on one physical server. Each VM is isolated from the others and can be assigned dedicated resources based on the requirements of the application. This protects other applications and users from being affected if one VM should need to be rebooted. Notifications and alerts Avance automated alerts and notification tools keep your IT staff - even if remotely located - and Stratus global support organization constantly informed with up-to-the minute information. This enables problem prevention and resolution of issues that can t be resolved by Avance software. Most potential problems are resolved without downtime.

21 Protecting PSAP lications from Downtime 20 24x7 software support coverage Stratus 24x7 support services are second to none for uptime protection. You can count on Stratus for immediate identification and resolution of potential issues, prompt attention to support requests, and installation of Avance software updates. Easy to set up and use Avance software makes it easy and economical to achieve the high availability required by dispatch center applications. It requires no special skills to set up and is designed to manage its own operation, making it ideal for dispatch centers with little or no on-site IT expertise. There are no clusters to configure and test, no failover scripts to write, no changes to make applications cluster-aware, or standby servers to buy and store. With just a few mouse clicks, Avance software transforms any pair of standard x86 servers into a high-availability platform. Conclusion Annual Call Volume Figure 16: Calls Potentially Affected at by Server Outage Stand-alone Server (99%) Hardened Server (99.9%) Clusters & VM (99.95%) Stratus Avance (99.99%) Stratus ft Server (99.999%) 50, ,000 1, ,000 2, ,000 5, Public safety call takers, dispatchers, and first responders all depend on the applications and information managed by computer systems to protect lives and property. It is no longer just the CAD application that it is mission-critical; it is also the supporting applications and records management systems that provide the information needed to assure the efficiency, effectiveness, and safety of emergency response operations. Stand-alone conventional servers offer little protection against failure. Failover clusters offer improved availability, but still fall short of providing the availability required for mission-critical applications. And, they are complex to set up and maintain. Virtualization enables much more efficient use of computer resources and simplifies system administration. Virtualized high-availability solutions are easier to administer than clusters, but still fall short of providing mission critical availability. Software-based fault-tolerant solutions have technical limitations, can degrade application performance, and are costly to implement. The key to achieving the uptime required for mission-critical public safety applications is to detect and prevent failures before they happen, not simply to react after the failure already has occurred. Stratus fault-tolerant ftserver systems deliver guaranteed service levels that exceed %, the highest in the industry for mission-critical applications running on Microsoft Windows Server and Hyper-V, Red Hat Enterprise Linux, and VMware vsphere.

22 Protecting PSAP lications from Downtime 21 For small PSAPs with limited budgets and with little or no on-site IT expertise, Avance software makes it easy and economical to achieve % availability without the cost or complexity of clusters. Every aspect of ftserver systems and Avance software is designed to prevent downtime, not simply allow for quick recovery. Unmatched uptime assurance and operational simplicity, combined with around the clock proactive support services, provide unmatched systems and application availability for Public Safety Answer Points. That s what we mean by uptime, all the time. Specifications and descriptions are summary in nature and subject to change without notice. Stratus and the Stratus logo, ftserver and the ftserver logo, Stratus Avance and the Stratus Avance logo are registered trademarks and Uptime Meter is a service mark of Stratus Technologies Bermuda Ltd. Intel and the Intel Inside logo are registered trademarks of Intel Corporation or its subsidiaries in the United States and/or other countries/regions. Microsoft, Windows, and Windows Server are either registered trademarks or trademarks of Microsoft Corporation in the United States and/or other countries/regions. VMware, VMotion, vsphere, and ESX are registered trademarks or trademarks of VMware, Inc. in the United States and/or other jurisdictions. The registered trademark Linux is used pursuant to a sublicense from the Linux Mark Institute, the exclusive licensee of Linus Torvalds, owner of the mark on a worldwide basis. Red Hat, Enterprise Linux, and the Red Hat Shadowman logo are registered trademarks of Red Hat, Inc. in the United States and other countries. All other trademarks and registered trademarks are the property of their respective holders Stratus Technologies Bermuda Ltd. All rights reserved. X1440