THE BUSINESS IMPLICATIONS OF AN ALWAYS ON TECHNOLOGY CULTURE. DEALING WITH A NEW ERA OF IT OUTAGES

Transcription

1 TMT Insight Summer 2013 TMT DISCUSSION DOC DEALING WITH A NEW ERA OF IT OUTAGES THE BUSINESS IMPLICATIONS OF AN ALWAYS ON TECHNOLOGY CULTURE. DEALING WITH A NEW ERA OF IT OUTAGES 0001

2 The Cost of Failure T he price of unplanned IT downtime can be seen from a variety of high-profile outages across both the private and public sectors. These include: I n October 2011, BlackBerry suffered a global outage that meant data was cut off from customers phones. The outage took four days to rectify, and cost BlackBerry maker Research in Motion around $54 million. I n July 2012, Twitter blamed an infrastructural double whammy within its data centre for an outage that took its entire service down for around two hours. I n February 2013, the Bank of America s online banking operations crashed, cutting customers off from their bank accounts and disrupting the customer service phone line. It was the first day of the month, when customers often check their accounts to pay bills and ensure paycheques have been deposited. Customers took to social media to air their frustrations. I n July 2012, France Telecom customers were unable to make calls following an outage which the firm s CEO said cost the operator several dozen million euros. I n November 2012, IT failures at Adur and Worthing councils cost the local authority at least 26,000, of which 1,800 was overtime, and had a knock-on effect on local private sector businesses, including funeral directors and local leisure facilities providers. I n January 2013, online retailer Amazon s homepage was down for 49 minutes, which industry estimates suggest would have potentially cost the firm nearly $5.7 million in lost sales.

3 DEALING WITH A NEW ERA OF IT OUTAGES INTRODUCTION CONTRIBUTORS David Isaac, Head of Advanced Manufacturing & Technology Services, Clive Seddon, Head of TMT & Sourcing, David Barker, Partner, TMT Litigation, Iain Monaghan, Partner, TMT, Andrew Marks, CIO, Tullow Oil David Bickerton, Vice-President EMEA ITO Portfolio, HP Enterprise Services Damian Saunders, Director, cloud networking group, Citrix Neil Bayles, Head of Governance and IT Performance Management, QBE Simon Acott, Director, Exponential-e Serious IT failures HAVE A NEGATIVE EFFECT ON CUSTOMER LOYALTY, REPUTATIONS AND REVENUE. Major outages today make big and unwelcome news stories for those involved. These seem a relatively new phenomenon but one that looks unlikely to abate. Many large enterprises are built on a foundation of aging IT systems, held together by bespoke software designed many years ago the proverbial sticking plaster. Their IT estates have been built up over many years, with legacy systems often out-living the IT managers and teams that developed and installed them. Boards driven by intense M&A activity during the 1990 s and 2000 s have left a legacy of further IT sprawl. In these cases it is often only a matter of time before the sticking plaster comes loose and a major IT outage causes tides and floods through the business affecting customers and counterparties and, worse still, reputation and stock price. In tough economic times, many boards have little desire to invest in integrating and rationalising these sprawling IT systems, even seeking comfort in the hope that siloed systems will have limited repercussions if they do fail. However, as recent news headlines have highlighted, out-of-date estates and broader networks can fail, and have serious implications when they do. The way a business responds to an outage is critical to system recovery, the financial outlay and the rebuilding of its reputation. The right strategy, plan, processes and people can mitigate the impact and determine whether there is a ripple, tidal or flood impact. ON THE 27 JUNE, AT PINSENT MASONS LONDON HEADQUARTERS, IT EXECUTIVES FROM BOTH CUSTOMER AND SUPPLY SIDE CAME TOGETHER TO DEBATE THE ISSUES AND SHARE THEIR INSIGHTS. This document is a summary of the key points covered in that discussion. We hope you ll agree the challenges identified and insights shared are thought-provoking and useful. To find out more please contact: Clive Seddon Head of TMT & Sourcing E: clive.seddon@pinsentmasons.com David Isaac Head of Advanced Manufacturing and Technology Services E: david.isaac@pinsentmasons.com 01

4 In the always-on age, organisations face an enormous challenge Customers expect to be able to do business with chosen organisations 24 hours a day, 7 days a week. They want to bank or shop or simply engage with their preferred providers whenever they want to and if they can t do so then they ll take their business elsewhere. And they won t go quietly either. With the explosion of social media networks, the unhappy customer has been empowered with a method of mass communication that can cause considerable reputational and brand damage just at a time when customer loyalties are notoriously fickle. There is now a tight correlation between the effectiveness of the technology underpinning the customer experience and the brand reputation, so that technology needs to work. The problem of course is that the technology that underpins 21st century business is not 100% reliable. Failures will happen in the form of downtime or unplanned outages of critical business systems and there is very little that can be done to prevent these without incurring substantial, often uneconomical, overheads. The uncomfortable reality is that it is almost impossible for businesses to put systems and mechanisms in place to prevent IT outages completely. It is however possible to mitigate against the risk of unplanned outages and to plan for what needs to happen when they do occur so as to minimise the impact on the business bottom line and the business brand. CIO front and centre With the almost complete dependence on technology powering the business, IT is a board level agenda item, and the CIO is now on the front line to tackle an issue that could threaten the wider business. It was not always like this. Once, IT was regarded as a back office matter, left to be dealt with by the people down in the data centre or by the outsourcing provider chosen to run the technology infrastructure. As the increasing number of IT outages takes its toll on businesses across multiple sectors, including retail, banking and telcos, such neglect of a critical foundation of the business came with a high price tag when the systems went down. This is no longer a viable approach. When IT failures emerge in any industry the subsequent investigation typically reveals some common themes, including lack of consistent governance, issues around The majority of problem contracts that we see are problematic because of poor contract arrangements, lack of money, poor governance and poor contract management. David Isaac, compliance, poor contingency and resource planning as well as basic problems with contractual terms geared around punitive measures rather than incentivisation. Addressing all, or some, of these provides organisations with a better prospect of mitigating against the risk of failure. But each in turn presents challenges that need to be met not only by the CIO, but the business members of the C-Suite. Good governance Good governance is critical, but also the area most commonly overlooked by too many organisations. Poor governance is what we see in the contracts that go wrong poor governance, probable bad planning and bad communications, says David Isaac, partner and head of the Advanced Manufacturing Technology Services sector at. The majority of problem contracts that we see are problematic because of poor contract arrangements, lack of money, poor governance and poor contract management. In fact the signing of a contract with an IT provider is only the first step on an ongoing journey. Signing a contract, putting it in the metaphorical drawer and forgetting about it is not good practice. There is little point to Service Level Agreements (SLAs) and tough contract terms if no one is ensuring that agreed obligations are met. Governance is about more than just enforcement of terms and conditions. It needs to encompass compliance (at both operational and strategic levels) as well as supplier relationship management, risk assessment and strategic contingency planning for presumed failure. So with this checklist in mind, who is responsible for good governance and compliance practice? The answer is that there are multiple shifting roles and responsibilities at play with different parties taking on key obligations. All 02

5 DEALING WITH A NEW ERA OF IT OUTAGES Neil Bayles, Head of Governance and IT performance Management, QBE Andrew Marks, CIO, Tullow Oil The CIO is now about reaching across the organisation and pulling together the supply chains. Andrew Marks, Tullow Oil The IT team should take responsibility in the event of an IT outage. However, in order to manage an outage as efficiently as possible, supply chains, contracts (both internal and external) and lines of responsibility need to be transparent. Neil Bayles, QBE parties need to learn to work with and across one another. Damian Saunders, director of the cloud platforms & networking group at Citrix, argues that a critical factor in mitigating against unplanned outages is the maturity of the management and communication matrix between the technology department and the various lines of business. IT outages are rarely just to do with technology, he suggests. A business can be caught out by the speed of change which occurs in the technology industry, leaving them exposed to risk. Having the right management matrix in place could better protect businesses against an unforeseen IT outage. The presence of an IT executive in the boardroom is critical to this. It s not acceptable to pin all the responsibility on the CIO, adds Andrew Marks, CIO at Tullow Oil. I cannot do it on my own, he states. There is a clear line of accountability that is shared with specialists in the organisation. Ultimately it is the CIO. If they are a board member, it sits with them. If like me they report to an executive director, it sits with him or her. The role of CIO has evolved, he adds. Perhaps it is now becoming more of an advisor to different parts of the organisation, suggests Marks. The CIO is now about reaching across the organisation and pulling together the supply chains. There clearly needs to be a clear understanding across the organisation of lines of responsibility for good governance to work appropriate. The IT team should take responsibility in the event of an IT outage. However, in order to manage an outage as efficiently as possible, supply chains, contracts (both internal and external) and lines of responsibility need to be transparent, advises Neil Bayles, head of IT, governance and performance management at insurance firm QBE Europe. Tracing the line of responsibility in order to find a cause is easiest if clear contractual boundaries are in place. All too often, when there are gaps in understanding about who needs to do what, a blame culture takes over only exacerbating the issue. Getting the right processes in place should be a priority, but in order to do this a business needs to be more integrated with its IT department. > 03

6 Appetite for RISK 04

7 DEALING WITH A NEW ERA OF IT OUTAGES 100% Simon Acott Director, Exponential-e UPTIME is theoretically possible, but unlikely to be achieved in the real world and will come with a significant cost burden Transparency of responsibilities facilitates a more productive assessment and management of the appetite for risk across an organisation. How much risk is the business able and willing to tolerate? While providers may offer % uptime, does your business need to pay for that? Could it survive with a lower percentage and lower outlay? You need to understand the appetite of your organisation for risk. It may choose not to plan any mitigation but so long as they are aware of the risk, says Bayles. The difficulty to overcome as a CIO is to ensure that the business becomes accountable for its own issues. It comes back to transparency and the ability to articulate that. The business needs to understand what risks they are taking. One hundred percent uptime is theoretically possible, but unlikely to be achieved in the real world and will come with a significant cost burden. Businesses need to find the balance between risk and reassurance that is appropriate to their operational model. What do you really need to achieve and how much money are you prepared to spend to do that based on the relative risk of what happens if you don t? asks Simon Acott, director at network infrastructure provider Exponential-E. How much are you prepared to find the balance between risk and reward? What do you really need to achieve and how much money are you prepared to spend to do that based on the relative risk of what happens if you don t? Simon Acott Exponential-e > 05

8 Iain Monaghan, Partner, TMT, David Barker, Partner, TMT Litigation, Damian Saunders, Director, cloud networking group, Citrix Plan for failure Finding that balance comes down to effective planning across both business and IT to understand the needs of the organisation and meet their requirements. Acott advises: If you don t plan something properly you re going to end up missing something. Servers don t sit there and say I m not having a particularly good day. They do what you ask them to do. Tullow Oil s Marks concurs, adding that the advisory nature of the CIO role should assist in achieving the right level of risk for the business as a whole. Judgements tend to be made around how much we are willing to invest, he notes. Inevitably corners get cut that s the negative view or we ve made a risk-based judgement and we made a mistake. I think the CIO is in a position to support and also in a position to warn and advise on the harder aspects of implementing. Having a balanced argument is something that the head of IT has done for years. A key element of effective planning is typically to have robust contractual boundaries in place so as to prevent a blame culture springing up in the event of unplanned outages. Everyone in the supply chain inside and outside the organisation needs to bear responsibilities transparently. Fix the problem first, then deal with the rest afterwards. Any organisation has to be really clearly on who is responsible for the control environment around it, says David Bickerton, vice-president EMEA for the ITO Portfolio at HP Enterprise Services. I don t like the notion of blame. It s the wrong kind of conversation. The CIO has a really interesting challenge, especially in the economic world we live in today when there is forever that cost pressure. I think that the CIO s challenge is to make sure that accountability is discharged all the way through the supply chain, probably with a greater degree of accuracy than has been done in the past. Thorough documentation of systems and processes will assist in planning, Bickerton adds. Have you got clarity Accountability has much to do with staffing policy. If you bring in a lot of contractors or do a lot of offshoring then there s no long term sense of accountability in place. Damian Saunders, Citrix over your supply chain, around your technology? Is it really clear where all accountabilities and responsibilities lie? he asks. But accountability is not always obvious, suggests Citrix s Saunders, partly due to poor change management and continuity. Accountability has much to do with staffing policy, he argues. If you bring in a lot of contractors or do a lot of offshoring then there s no long term sense of accountability in place. Contractual concerns One reason for poor governance in practice is the misguided idea that by signing a contract or an SLA, governance and compliance issues are taken care of. In reality a contract is simply the first step on a longer journey and should be regarded as a living document demanding of ongoing attention. You can t actually just assume a risk has gone away by ticking the box in relation to assurance and governance. That isn t sufficient, warns Pinsent Masons Isaac. Typically our experience is that most people don t look at their contracts. The SLAs are often over the top, and honoured in the breach rather than the observance. Things tick along quite nicely until there s a major outage and then things unravel. The point at which things unravel is 06

9 DEALING WITH A NEW ERA OF IT OUTAGES David Bickerton, Vice-President EMEA ITO Portfolio, HP Enterprise Services when the blame game starts up. While a great deal of attention is paid to setting up SLAs and to having penalties built into contractual arrangements, in reality few organisations are genuinely willing to embark on a punitive course of action. For their part, suppliers will seek to protect themselves from legal action. partner and head of TMT and sourcing Clive Seddon notes: IT contracts often have reasonably standard provisions concerning limitation of liability for losses flowing from a breach and exclusion of liability clauses. It would be extremely rare to find a contract where the supplier indemnifies an organisation be it bank, telecom provider, or airline for their whole business going down, he adds. Given recent failures however, it is perhaps not surprising that major customers with strong negotiating positions are seeking to hedge their positions by seeking to extend their supplier s remedial obligations and their potential exposure to customer claims for losses arising from a major breakdown. A better long-term solution would be a more incentives-driven approach to terms and conditions, demanding in turn a shift in attitude from both buy and sell side of the contract. This would bring more accountability to the fore from all parties as the threat element of a traditional SLA approach is minimised. This would change the fundamental structure of the way suppliers are What s happening rewarded. So, for example, an SLA may in practice on include goals whereby additional income deals is much is earned from achieving defined levels more focus on of service rather than fines or penalties incentivisation being imposed if they are not met. and payment by What s happening in practice results. Clients on deals is much more focus on are focusing incentivisation and payment by results. on proactive Clients are focusing on proactive remedies, rather remedies, rather than a more penalistic than a more and negative approach to failures, says penalistic and Isaac from. negative approach Pursuance of these new approaches to failures. can pay dividends for the business, David Isaac concludes Isaac. Output contracts that incentivise, good governance, proper risk analysis and regular communication must help resolve problems. They are not going to stop outages, but better contracts can minimise their impact, he says. > 07

10 Guidance Organisations need to start making concrete plans to prevent and mitigate the effects of IT failure. Getting the right technology in place is only half the story. Ensuring process clarity and a distinct chain of decisionmaking is equally as crucial, as is expert advice and support. With these factors in place, a business can create a strong foundation for contingency planning and a strong recovery without substantial loss. Crisis management protocols and a positive PR campaign will always take priority in an outage situation. However, care needs to be taken during the recovery period to ensure reasonable positions are maintained which align with contractual obligations and responsibilities. The way a business responds to an outage is critical to system recovery, the financial outlay and the rebuilding of its reputation. The right strategy, plan, processes and people can mitigate the impact. Capturing evidence, identifying the chain of causation and understanding the losses and costs involved are crucial. With regular technological advancements to consider it is critical that IT customers should regularly review their contracts with suppliers to ensure that the contract is still relevant for meeting the needs of their business. It is wrong to see IT as a cost to business rather than an intrinsic part of it. One reason for poor governance in practice is the misguided idea that by signing a contract or an SLA, governance and compliance issues are taken care of. In reality a contract is simply the first step on a longer journey and should be treated as a living document demanding ongoing attention. You can t actually just assume the risk has gone away and you tick the box in relation to assurance and governance. A better long-term solution would be a more incentives-driven approach to terms and conditions, demanding a shift in attitude from both buy and sell sides of the contract. This would ensure more accountability for all parties as the threat element of a traditional blanket SLA approach is minimised and would change the fundamental approach to the way suppliers are rewarded. So, for example, an SLA may include goals whereby additional income is earned from achieving defined levels of service rather than fines or penalties being imposed if they are not met. What s happening in the contractual landscape is much more focus on incentivisation and payment by results on proactive remedies, rather than a penalistic and negative approach to entering those sorts of arrangements. The TMT team 08

11 DEALING WITH A NEW ERA OF IT OUTAGES II 09

12 LLP is a limited liability partnership registered in England & Wales (registered number: OC333653) authorised and regulated by the Solicitors Regulation Authority and the appropriate regulatory body in the other jurisdictions in which it operates. The word partner, used in relation to the LLP, refers to a member of the LLP or an employee or consultant of the LLP or any affiliated firm of equivalent standing. A list of the members of the LLP, and of those non-members who are designated as partners, is displayed at the LLP s registered office: 30 Crown Place, London EC2A 4ES, United Kingdom. We use to refer to LLP and affiliated entities that practise under the name or a name that incorporates those words. Reference to is to LLP and/or one or more of those affiliated entities as the context requires. LLP For a full list of our locations around the globe please visit our websites: