REPORT TO CABINET TO BE HELD ON 15 SEPTEMBER 2015 Key Decision No Forward Plan Ref No 23K Corporate Priority The proposals in this report contribute to the delivery of all the Council s priorities Cabinet Portfolio Holder Councillor Andrew Jenkinson REPORT OF: DEPUTY CHIEF EXECUTIVE 15/250 WARDS AFFECTED: ALL SUBJECT: REPLACEMENT OF CORPORATE ELECTRONIC DATA STORAGE, BACKUP AND DISASTER RECOVERY SOLUTIONS RECOMMENDATION (S): Cabinet Members are asked to: 1. Note the contents of the report and the urgent need to improve our corporate electronic data storage and back up and Disaster Recovery capability 2. Authorise Officers to undertake an exercise to procure the technology necessary to: i. Increase the disk capacity to improve performance and reliability in the Town Hall data centre ii. iii. Establish a replicated data centre with new disk hardware at an second independent site Undertake a programme of work to implement the improved disk capacity and new replicated data centre 3. Note that the capital costs required to procure and implement the proposed solutions are identified within the current ICT development budget. The annual revenue cost will be paid for through the decommissioning of the current solution. Therefore the costs will be at worst cost neutral, with the potential for some revenue savings to be delivered through the procurement process.
4. Subject to the costs being within current capital and revenue budgets, as identified in the approved financial strategy, delegate authority to the Director (LD) to enter into the contracts necessary to implement the recommendations above in paragraphs 2.i 2.iii. REASON FOR RECOMMENDATION (S): To highlight to members the risks inherent with the current data storage and disaster recovery solutions. Some of the elements are old, out of warranty, no longer fit for purpose and command expensive third party support. Therefore they pose a considerable risk to the Council s ongoing service delivery and critical business continuity requirements. To recommend a programme of work that will replace major elements of the Council s corporate electronic data storage, data backup and recovery infrastructure systems, providing modern, fit for purpose data storage solutions and a resilient disaster recovery solution. HIGHLIGHTED RISKS: See attached risk matrix. 1. INTRODUCTION 1.1 The Council currently manages ICT service across twenty-six distinct sites. These services include the provision and support of key ICT solutions such as laptops, PCs, corporate IT servers, corporate IT application systems, electronic data storage and telephony. 1.2 Data links connecting the sites enable cross-organisation access to the corporate IT application systems, electronic data storage, email and the internet. These technologies are critical to the Council s service delivery. 1.3 Corporate electronic data storage, housed in the Council s data centre at Scarborough Town Hall, comprises physical disk arrays that host and serve all electronic data in use across the organisation. 1.4 Data backup facilities are also managed from the data centre. These are critical to ongoing service delivery as they enable data files to be recovered quickly in the event of accidental or malicious loss. These backups are stored securely off-site on a daily basis and provide a comprehensive copy of the Council s entire electronic data in the event of a disaster (e.g. a fire in the Town Hall). 1.5 To recover from such a disaster, a managed Disaster Recovery ( DR ) solution is also in place. This is critical to successfully recovering IT application systems and data when a major incident may damage or destroy the Council s single data centre. 1.6 The current electronic data storage, backup and Disaster Recovery solutions are no longer reliable or fit for purpose, and present significant risks to the
Council s business operation and the ability to ensure ongoing service delivery. 2. CORPORATE AIMS/PRIORITIES AND THE COMMUNITY PLAN 2.1 This programme of improvements contributes to all corporate priorities and to the overarching Council Aim of, Improving the Performance of our Services and ensuring they provide value for money. 3. BACKGROUND AND ISSUES 3.1 Corporate Electronic Data Storage 3.1.1 The Council s current electronic data storage solution is aged (over six years old), unreliable and does not provide sufficient capacity for the data demands of the organisation. It is out of main vendor warranty and is currently serviced by a third party contract that is no longer cost-effective. 3.1.2 The technology is end of life it is no longer in production so further development of the solution is not possible and at this time any replacement components are second-hand, with a risk that replacement may be no longer available. 3.1.3 Given the reliance on modern electronic service delivery, if the electronic storage solution fails, Council business will suffer greatly. Staff will not be able to access electronic files or IT application systems until the service is reinstated. 3.1.4 Further to the reliability issues, the ongoing deployment of a modern corporate Electronic Document and Records Management (EDRM) system and the resultant need to digitise vast volumes of paper files significantly increases the demand for more disk space capacity than is currently available. 3.2 Backup 3.2.1 The backup data storage solution has already suffered failures of components and age-related issues. The ICT service recently had to invoke the Disaster Recovery contract in response to a serious backup appliance hardware failure. The recovery provided temporary storage resources and the incident necessitated the urgent procurement of a new backup appliance unit. Despite this recovery there remains a substantial risk that data may not be able to be recovered if a further loss occurs. 3.2.2 Data backups are scheduled to run once a day (overnight, when impact on IT system availability is not an issue). Current backup procedures involve ICT staff physically removing the data tapes from Scarborough Town Hall on a daily basis, and storing them off-site at a separate, secure location. This is time-consuming and data restored from these tapes can be up to twenty-four hours earlier, meaning that the organisation can lose up to a day s worth of data in the event of a data recovery exercise (or DR scenario). This is no
longer acceptable to a modern business whose customers are increasingly reliant on real-time, digital service delivery. 3.3 Disaster Recovery 3.3.1 The current third party agreement involves a mobile (truck-mounted) computer room being deployed to a public car park at Whitby Harbour. See Appendix 1. This can take up to twenty-four hours following initial contact. Following its arrival, IT application systems and data are then recovered in the mobile room over a period of several days or weeks, and are then made available to a small number of users. 3.3.2 The current DR solution is no longer fit for purpose. The mobile computer room is restricted (due to size) to where it can be deployed. It is reliant on a wired connection to the users, so there is a restriction on the number of users who can be accommodated in the Whitby Harbour office and this number is significantly less than is required by the Council s Business Continuity plans. 3.3.3 As the Council modernises and more of its services utilise digital service channels such as the web and mobile devices, the reliance on the Council s ability to provide continuous access to electronic data, IT applications and digital services becomes increasingly critical. 3.3.4 The current DR solution recovers IT systems and data over a period of a number of days, but as all recovery requires building the applications and restoring their databases from the ground up, it is acknowledged that some systems may not be accessible (to staff or our customers) for over a week. This is no longer acceptable to the business or our customers. 3.4 General 3.4.1 The current technology solutions are inflexible and reactive. They do not readily support modern business needs to expand/contract storage requirements as the organisation reshapes. They create delays in recovery times, and in a DR scenario, the reputation of the Council may suffer as critical services will not be reinstated in a timeframe acceptable to customers or staff. 4. CONSULTATION 4.1 In formalising the programme and its deployment, technical solution advice has been taken from the Council s ICT specialists and manufacturers / suppliers of data storage solutions, backup / DR technologies and hardware. 4.2 Procurement advice has been taken from the Council s procurement team. 4.3 The experience of other Councils and public sector organisations has also been sought.
5. ASSESSMENT 5.1 The Council s current corporate electronic data storage, backup and DR infrastructure carries serious risk to ongoing service delivery. Key components are aged, out of warranty, unsupported and failing. The overall solutions lack capacity, do not meet performance requirements, cannot provide flexibility or scalability and are no longer cost-effective. 5.2 If approved the recommendations in this report will initiate a programme of work that builds on recent infrastructure and data connectivity investments made by the Council and will deliver equipment that will be in warranty and with suitable support agreements in place. 5.3 The deployment plan will be designed to minimise any service interruption to our customers, the Council and our partners. The intended solution suite will deliver the following: 5.4 Corporate Electronic Storage 5.4.1 Increased disk capacity with improved performance and reliability in the Town Hall data centre, whilst significantly reducing the physical and environmental footprint. 5.4.2 A replicated data centre will be established at a second Scarborough Borough Council site (beyond the Town Hall). This will involve similar new disk hardware being hosted at the second site. Electronic data will be automatically saved to both the Town Hall (primary) data centre and the secondary data centre (a process known as replication ). The secondary data centre will be a secure facility and will provide a failover resilience not currently offered, ensuring business continuity if the primary data centre suffers service interruption. 5.5 Backup 5.5.1 The current reliance on physically removing tape backup media off-site on a daily basis will end. All data will be backed up in real-time (through replication) to disks at the secondary data centre. This means that any lost data will be recovered digitally with minimal officer effort. 5.5.2 As tapes are taken once a day (overnight), the current solution means that some data, once recovered, can be up to twenty-four hours out of date. The new solution will reduce this to a few minutes at most, further supporting continuous service delivery. 5.6 Disaster Recovery 5.6.1 The third party truck-mounted mobile computer room agreement will be terminated. The replicated secondary data centre will be the DR technology backup site and this will enable the Council s ICT specialists to ensure that systems and data are available in a small number of hours following a
disaster, rather than the current days and weeks commitment. See Appendix 2 5.6.2 The new DR solution will make the Council s disaster recovery solution location-independent, meaning that the Council s Business Continuity leaders can re-house staff in any number of locations and still be confident they will have reliable and high-performing access to IT systems and data. 5.7 Information Governance 5.7.1 All information governance responsibilities, including physical environment security and upholding all data governance laws (e.g. Data Protection Act 1998) will be taken seriously and upheld responsibly throughout the programme. 5.7.2 It is intended that the proposed solution be audited by the Council s Internal Audit function to ensure compliance with good practice. 5.7.3 It is worth noting that, whilst the Council is establishing a data centre beyond the Town Hall for the first time, it will be in a secure and monitored environment and the risk of data loss or theft will be far lower than currently moving backup tapes manually away from the Town Hall. 6. IMPLICATIONS 6.1 Policy 6.1.1 There are no policy implications arising directly from this report, although Business Continuity and Disaster Recovery policy and plans will have to be updated to reflect the new solutions. 6.2 Legal 6.2.1 All procurement actions will be dealt with in accordance with EU Procurement regulations and the Council s Procurement policy and procedures. 6.3 Financial 6.3.1 Capital funding for the programme is identified in the Council s approved financial strategy. 6.3.2 Based on indicative figures it is intended that annual Revenue savings for the term of the contract of circa 6,000 will be realised. However, this will only be confirmed during the procurement process. Based on a five year agreement, it is anticipated:
Description Cost Capital costs (one off) 186,000 Revenue costs (per annum) 13,500 Total (for 5yr investment) 253,500 6.4 Equalities and Diversity 6.4.1 There are no equality and diversity issues arising directly from this report. However the implementation of this work programme will support the Council s inclusive approach to service delivery. 6.5 Staffing 6.5.1 Due to the specialist nature of the work, it is anticipated that some third party professional services will be required to successfully deploy and configure the solutions. 6.5.2 However, it is intended that training will be provided to the Council s ICT team as part of the programme of work and this will embed knowledge in-house and enable the Council to self-serve elements of the support and further development of the technologies. 6.6 Planning/Crime and Disorder/Environmental Health 6.6.1 There are no direct planning, crime and disorder or sustainability issues arising out of this report. 7. ACTION PLAN 7.1 The timetable for the delivery of the work programme is: Phase Dates Procurement Oct 2015 Dec 2015 Deployment Jan 2016 Mar 2016 Hilary Jones Deputy Chief Executive
Author: Greg Harper, ICT Delivery Manager, ICT Services, Town Hall, Scarborough Telephone No: 01723 384333 E-mail address: greg.harper@scarborough.gov.uk Background Papers: None. IF YOU HAVE ANY QUERIES ABOUT THIS REPORT OR WISH TO INSPECT ANY OF THE BACKGROUND PAPERS, PLEASE CONTACT Greg Harper ON 01723 384333 e-mail greg.harper@scarborough.gov.uk
RISK REGISTER Ref Date Risk Consequence(s) Mitigation 1 15/09/15 Corporate electronic data storage solution is not replaced Corporate disk store fills and corporate electronic data and IT systems fail Electronic service delivery is halted Customer access to services is severely impacted Risk of permanent data loss Current Risk Score Target Risk Score Service Unit Manager / Responsible Officer None D4 A2 ICT Delivery Manager Action Plan Replace corporate electronic data storage solution with new, supported, futureproofed technology of sufficient capacity to ensure current business processes and support future, more flexible, locationindependent ways of working Significant reputational damage for the Council 2 15/09/15 Corporate data backup solution is not replaced Data cannot be restored in a timeframe acceptable to the business or the Council s customers Hardware components will gradually fail, further increasing the risk of more permanent failure Business Continuity expectations become unachievable The business accepts that data may be permanently lost or recovery times will be significantly longer than is acceptable to the business, its customers and its Business Continuity requirements D4 A2 ICT Delivery Manager Upgrade corporate data backup and recovery solutions to ensure they utilise replication and successfully deliver business, customer and Business Continuity technology requirements Significant reputational damage for the Council
Ref Date Risk Consequence(s) Mitigation 3 15/09/15 Corporate Disaster Recovery (DR) solution is not replaced DR solution will add risk to data and IT systems recovery (e.g. reliance on third party; truck-mounted vehicle in transit) Limited staff access to recovered systems from Whitby Harbour Office Significant delays in recovering data and IT systems due to technology and logistical restrictions Electronic service delivery is severely impacted for a period of days / weeks Customer access to services is severely impacted for a period of days / weeks Higher risk of permanent data loss as recovery processes rely on significant human intervention Significant reputational damage for the Council if electronic services are unavailable for a significant period Annual cost savings will not be realised The business accepts that data and IT systems recovery times will be significantly longer than is acceptable to the business, its customers its Business Continuity and DR requirements Current Risk Score Target Risk Score Service Unit Manager / Responsible Officer E4 A2 ICT Delivery Manager Action Plan Upgrade corporate DR technology solution to ensure that it utilises replication and successfully delivers the Council s corporate Business Continuity and DR technology requirements
Glossary of Terms Risk An event which may prevent the Council achieving its objectives. Consequences The outcome if the risk materialised. Mitigation The processes and procedures that are in place to reduce the risk. Current Risk Score The likelihood and impact score with the current mitigation measures in place. Corporate Objectives An assessment of the Corporate Objectives that are affected by the risk identified. Target Risk Score The likelihood and impact score that the Council is aiming to achieve. Service Unit Manager The Service Unit or Officer responsible for managing the risk. Action Plan The proposed actions to be implemented in order to reduce the risk to the target score. Risk Scoring 5 4 3 Impact 2 1 A B C D E Likelihood Likelihood: A = Very Low B = Not Likely C = Likely D = Very Likely E = Almost Certain Impact 1 = Low 2 = Minor 3 = Medium 4 = Major 5 = Disaster