SQUIZ SOLUTIONS. Disaster Recovery and Security October 13. Zetland House Clifton Street London EC2A 4LD

Transcription

1 SQUIZ SOLUTIONS. Disaster Recovery and Security October 13 Zetland House Clifton Street London EC2A 4LD

2 Squiz UK Ltd All Rights Reserved. No part of this publication may be reproduced by any process, electronic or otherwise, in any material form or transmitted to any other person or stored electronically in any form, without the prior written consent of Squiz UK Ltd.

3 CONTENTS PAGE SQUIZ GLOBAL GRID NETWORK 1! PHYSICAL RESILIENCE THE BUNKER 1! Kent Facility 1! DISASTER RECOVERY PROCESS DNS FAILOVER 2! Backup to DR Failover 2! Technical Overview 2! Detailed Timeline of a Disaster Recovery Incident 3! Incident Identification 3! Incident Diagnosis 3! Decision to Fail Over 3! DNS Modification 4! Restoration of service in Disaster Recovery mode 4! Optional Escalation of DR to Read-Write 4! DNS Failback On Restoration 5! MTRS Considerations With DNS Failover 5! Disaster Recovery Option - Fully replicated 5!

4 SQUIZ GLOBAL GRID NETWORK Squiz is a global company, with nine offices across the world. We operate a number of hosting grids, supplemented by our full-time 24/7 follow-the-sun support teams. The Squiz grids are currently located: 2x United Kingdom Kent & Newbury 2x Australia Sydney & Canberra Additional grids are located in Germany and Dallas, USA. The primary datacentre in the United Kingdom is The Bunker facility located in Ash, where the core Squiz Cloud infrastructure is located. Your service will be hosted at this grid. PHYSICAL RESILIENCE THE BUNKER Squiz partner with The Bunker ( for physical infrastructure. It is a co-location agreement, and Squiz with Squiz staff running, monitoring and managing the Squiz grid. Kent Facility The Kent bunker was built by the MOD as a command and control centre. The construction and security standards are what you might expect of such a purpose built fortress. Patrolled and monitored 24x7x365 by our specialist on-site security teams, located 30 metres beneath a perimeter fenced site with reinforced concrete walls, steel doors weighing several tons and EMP and tempest shielding. Bunker onsite security includes 3m thick walls Solid steel blast protection doors 24 hour on-site technical and security team On-site MOD trained guard dogs Infrared CCTV Military electromagnetic pulse protection Tempest RFI intrusion protection Redundant UPS conditioned N+1 Dual HV redundant power supplies from physically diverse paths Dual 1700 KVA auto generator back-up sets Dual HV sub stations SCADA Environmental and Power Monitoring system Very Early Smoke Detection Apparatus Fire suppression system 24x7 video recording and monitoring Visual verification of all persons entering the data floor 24x7 manned guardroom (fitted with bullet-proof glass) 24x7 on-site security staff (ex-military & police) Guard dog patrols Multiple fibre providers Multiple Internet Service Providers Diverse independent telco risers from public highway to data floor Private MPLS network 1

5 The diagram below outlines the Bunker core network, showing the multiple connections to the data centre to negate any single point of network failure. The Bunkers Core Network Diagram 1.4 DISASTER RECOVERY PROCESS DNS FAILOVER The option to DR includes providing dedicated hardware to act as DR infrastructure in an additional grid to restore to back-up from. Backup to DR Failover A complete backup is made to the DR infrastructure nightly at 10pm. There is no data retention, as the backup replaces the existing data. The snapshot process is the equivalent of a fully automated Squiz backup and restore, with the exception that the Matrix filesystem is updated with any changes since the previous snapshot rather than copied completely. On the disaster recovery system a new database is created, the Matrix database contents are transferred to the new database, and the databases are renamed. Technical Overview In Disaster recovery via DNS failover, failover is a manual process. When a decision is made to switch to the DR server, a manual change to the site s DNS record will be made. After this change, browsers which request DNS details for the website will receive the IP address of the DR server. DNS changes do not propagate to the Internet immediately. 2

6 To reduce traffic, ISP s will cache DNS entries. It is possible to specify a TTL or Time To Live for DNS entries (e.g. 300 seconds for a 5 minute TTL), and DNS servers should refresh their cache after this period. However, there is no guarantee that a given ISP will observe this protocol. It is general practice to consider the time for a DNS change to propagate to all hosts on the Internet to be between 24 and 72 hours, although the majority of hosts will see a change within 8 hours. Detailed Timeline of a Disaster Recovery Incident Incident Identification The website is monitored using a 3rd party uptime monitoring service. This service is configured to poll a number of pages at one minute intervals. In the event that the website returns an error or fails to return results within a reasonable time, Squiz s Service Desk will be notified via and SMS. Using our 3rd party uptime monitoring service, the time for incident identification can be between 2 and 5 minutes. Incident Diagnosis Investigation will commence immediately to determine the cause of the failure or slowness. If this investigation finds the incident is a result of a known error that is quickly resolvable, service will be restored immediately on the primary instance and an incident report will be completed. If the diagnosis is undetermined or will require an extended outage, focus will move to making a decision to fail over. The time from incident identification to diagnosis can be between 5 and 15 minutes, depending on the cause of the issue. Decision to Fail Over If a decision to fail over must be made, Squiz staff will request authorisation to fail over from a designated client staff member. The Squiz staff member will provide details of the incident and an expected time to resolution. The client staff member will authorise and/or initiate failover to the DR server (depending on whether Squiz or the client have control of the relevant DNS record). Note that subject to agreement from the client and provisions for control of the DNS record, the following process is possible: If the Squiz staff member is unable to contact any designated client staff members, and the Squiz staff member determines that the service should be failed over, and Squiz have the ability to modify the DNS settings, the Squiz staff member will initiate a DNS change. The time from incident diagnosis to a decision to fail over can be from 3 to 15 minutes, depending on availability of designated client emergency contacts. 3

7 DNS Modification Once the decision has been made to make a DNS change, the client staff member or the Squiz staff member (depending on who has ability to modify the DNS entry) will make the necessary DNS changes. Time from decision to fail over to DNS modification can be between 3 and 6 minutes. Restoration of service in Disaster Recovery mode Once the DNS modification has been made, the updated DNS entry will begin to propagate around the Internet. Well-behaved DNS servers will reflect the change within 5 minutes (assuming TTL is set to 600s). When browser DNS caching and badly-behaved DNS servers are taken into consideration, the majority of the Internet will have service provided by the DR server within 3-8 hours, and the change should be consistent globally within hours. When the service is running in Disaster Recovery mode, the service is referred to as degraded. This does not reflect the performance of the system, but the fact that there is no further avenue for disaster recovery should the service fail. Time from DNS modification to restoration of service on DR server can be between 5 minutes and 72 hours. Optional Escalation of DR to Read-Write If restoration of services on the primary server is expected to take a significant amount of time, it is possible to disable the nightly snapshot and enable editing on the DR server at the client s request. This enables site editing to continue and avoids productivity losses due to the incident. This also enables placement of notices about maintenance work on the website if required. Note that if production (not emergency notices) authoring continues on the DR site, time to restore on the primary server will be extended as a snapshot will need to be made of the DR server and restored on the primary server. Time to escalate DR to read-write can be between 5 and 20 minutes. Time to restore DR content to the primary server can be between 45 and 120 minutes or longer, depending on the size of the system at the time of the incident. 4

8 DNS Failback On Restoration When service has been restored on the primary server, a coordinated decision between Squiz and the client can be made to reverse the DNS change. This has the same implications for and above. Time from DNS modification to restoration of service on the primary server can be between 5 minutes and 72 hours. MTRS Considerations With DNS Failover The mean time to restore service using DNS failover can be lengthy, depending on the definition of restoration of service. Time from incident identification to DNS modification can be as little as 13 minutes if all goes smoothly. This does not mean that a person using a web browser in New York (for example) will see the site restored at that time. Using the figures provided here, a person on the Internet will see restored service within around 15 minutes at best. At worst, this figure could be greater than 3 days. A realistic expectation is for a wide range of outcomes of between 30 minutes and 8 hours, depending on geographic location and ISP. Disaster Recovery Option - Fully replicated Clients can choose to deploy additional infrastructure replicated across an additional grid, creating a replica to failover in case of disaster. The infrastructure requirements for this would be the same as for your initial cloud offering, as this option will provide a complete, always ready failover option. Additional DNS switching can be provided to resolve delays with DNS changes on the system. 5