Sia Partners US SIFMA. Business Recovery The Climate is Changing

Transcription

1 New York Office 641 Lexington Avenue, Suite 1322 New York, NY Tel : (212) Internet : Paris New York Rome Milan Casablanca Dubai Amsterdam Brussels Sia Partners US SIFMA Business Recovery The Climate is Changing December 12, 2012 Presenters Daniel H. Connor Chief Executive Officer Tel : (862) daniel.connor@sia-partners.com Gus Moreno IT Risk Specialist Tel : (917) gus.moreno@sia-partners.com

2 Table of Contents 1 Introduction 2 Hurricane Sandy: BCP/DR In the News 3 Regulatory Viewpoints & Initiatives 4 Lessons Learned & Key Points to Consider 5 Discussion Topics 6 Contacts at Sia Partners US 7 Appendix 2

3 Introduction Companies Experience with Hurricane Sandy The October 29, 2012 Super Storm Sandy wreaked havoc on the East coast, causing massive physical damage and leaving millions of people and businesses without power and communication. The New York Stock Exchange (NYSE) was closed for two days, the first such weather-related closure in 120 years. The following are some real-life examples of experiences businesses had weathering Hurricane Sandy. Companies in the Press & Coping with Sandy NYSE Knight Capital Citigroup Goldman Sachs Verizon Communications Other War Stories We consider the regulatory perspective, in particular the Finra review of securities firms business continuity planning (BCP) during Sandy. Within the context of BCP, we offer some lessons learned and suggestions to help ensure that a company s disaster recovery and business continuity strategies and plans prevent or reduce the impact of a significant interruption in business processes. We present possible Internal Audit responses & the way forward. 3

4 Hurricane Sandy: BCP / DR In the News NYSE Context: The stock exchange maintains its specialist system of market makers physically on the trading floor. Located in a Flood Zone A near Wall Street. Backup plan is the Arca electronic platform, but it did not have a physical backup location as a feature of its DR Plan. Impact: The exchange was closed for two consecutive days, the first such weather-related closure since The NYSE had initially planned to switch to its backup plan to have trading be on Arca, but ultimately decided to shut down operations due to employee safety concerns and the possibility of technological issues affecting market participants, especially given that it would have been operating in an untested environment. Other theories reason they did not actually pursue operating on the Arca electronic platform is because it would demonstrate that the specialist system is archaic and could possibly prompt lawsuits (denied by the NYSE spokesperson). Result: While the stock market was closed for two days, it reopened on Wednesday to smooth trading running off of backup power. Bottom Line: Financial institutions contingency plans should be comprehensive and include physical location backup sites. In delaying resumption by two days, the NYSE factored in human resource considerations and the risk of potential technical problems if Arca operations resumed before proven readiness. 4

5 Hurricane Sandy: BCP / DR In the News Guess Who? Memo to clients on October 31, 2012 Dear Clients: Due to a building emergency (power issues), is asking you to seek an alternate destination for the order handling and execution of your OTC, Options and Listed orders until further notice. All computer interfaces with will be shutdown with no new orders, both by phone or electronic, being accepted at this time. Please continue to Help@.com with any questions. Thanks, Managing Director 5

6 Hurricane Sandy: BCP / DR In the News Knight Capital Context: Brokerage firm, one of the largest market makers in the US, headquartered in Jersey City, NJ. Significant amount of trading done on electronic platforms. In August, Knight Capital lost $475 million and required a bailout when a technology glitch caused the submission of incorrect orders. Impact: On October 31st at around noon, the firm s backup generators failed, causing a halt in trading operations and forcing them to cease accepting new orders. Critical trading systems never actually lost power due to double redundancies battery power was maintained. The generator was back up and running, allowing for limited trading by 2 pm. Regional offices, such as the fixed-income business in Greenwich, continued to trade. Trading operations were back to running on the normal power supply the following day. Result: The company informed clients of the power issues via a memo, asking them to reroute trades to an alternative source of execution. The firm s stock fell 8%, before recovering. Bottom Line: A takeaway for many financial institutions is that financial and reputation risk may suffer. It may also be preferable to shut down operations than risk technical problems prompting erroneous trades, possible lawsuits etc. 6

7 Hurricane Sandy: BCP / DR In the News Citigroup Context: Third-largest US bank with a main office in a Zone A at 111 Wall Street. 1,800 employees work out of the building, predominantly performing operations, technology and administrative roles. Additional offices on Greenwich St. lie less than a mile from the Hudson River. Impact: The building experienced severe flooding and will be out of commission for several weeks CEO Michael Corbat. 388 and 390 Greenwich St. offices, which house senior executives and trading & underwriting floors, respectively, had power failures and some flooding damage, rendering them inaccessible. Trading, which nevertheless resumed on October 31 st, had to relocate operations. On the retail banking side, hundreds of branches were closed. Bottom Line: Through the use of a backup site and remote access for employees to work from home, trading operations continued once markets reopened. However, the physical and logistical problems beg the question of having major banking offices located in a prime flood zone in lower Manhattan, especially in light of rising sea levels. 7

8 Hurricane Sandy: BCP / DR In the News Context: Goldman Sachs Top financial services institution with its global headquarters in a low elevation area lower Manhattan. Impact: Seems to demonstrate the necessity of investing adequately in DR planning. With regard to BCP: recognizes the positive correlation between events that pose a risk to business operations and those that affect the value of its investment portfolios. Fortressed its downtown headquarters with sandbags and installed backup generators. Result: Due to preparatory measures including backup generators, its headquarters were one of only a few buildings in the Wall Street area that were not flooded and had power. Bottom Line: To further improve its BCP/DR strategy, Goldman is reconsidering having two buildings within such close proximity of one another Wall Street and Jersey City according to COO Gary Cohn. 8

9 Hurricane Sandy: BCP / DR In the News Context: Verizon Communications Verizon is a major provider of communications services. Impact: Offices were flooded, in some places with three feet of water, causing power failures in Lower Manhattan, Queens and Long Island. This included some backup power systems. This caused the loss of service as a provider in voice, internet and television to customers. Verizon was able to reroute traffic through other network areas, but service to local areas was suspended. Result: Prioritized financial district given its status as the primary provider in the area. Worked with electricity companies to pump out underground water and dry equipment. Employees worked remotely or from alternate office sites. Additional employees were brought to New York to assist in recovery efforts. Bottom Line: If a company were solely reliant on Verizon for voice, internet and TV, these systems would not have been functional during and post-sandy. 9

10 Hurricane Sandy: BCP / DR In the News Other Effects of Sandy NASDAQ OMX s data center in Carteret, NJ operated on backup power with capacity of 72 hours of fuel (refilled every evening). There have been reports of contaminated fuel used to power generators. There are also second-hand reports of fuel tanks not being configured with pipe connections allowing all fuel to flow to generators, or some combination of UPS, generators or pumps supplying fuel to generators placed below water levels (although other components might have been elevated). UBS Wealth Management Americas closed more than 50 branches on October 29 th and 30 th out of concern for the safety of its staff. Client communication was maintained by Financial Advisors, and a buddy branch system was initiated in which calls made to a closed branch were automatically rerouted to an open branch. 10

11 Regulatory Viewpoints & Initiatives Going Back to 9/11 Interagency (FED, OCC & SEC) Paper on Sound Practices to Strengthen the Resilience of the US Financial System. Final paper was issued approximately one year after 9/11. Directed at core clearing & settlement organizations or firms playing significant roles in the critical markets. Among the points made regarding sound practice were: The sound practices focus on the appropriate back-up capacity necessary for recovery and resumption of clearance and settlement activities for material open transactions in wholesale financial markets. They (sound practices) do not address the recovery or resumption of trading operations or retail financial services. There are important business and internal control reasons for financial firms to maintain processing sites near financial markets and their own headquarters. While there was discussion of a required minimum distance (e.g. 200 miles) between recovery sites and primary locations no quantitative required mileage distance was retained. All plans should provide for ongoing consideration of the costs and benefits of achieving greater geographic diversification of back-up facilities. 11

12 Regulatory Viewpoints & Initiatives Finra Review Finra, in collaboration with the SEC and CFTC, initiated a review in November 2012 of the effectiveness of financial firms disaster recovery plans during Sandy. Key lessons learned from Sandy were solicited, and Finra may issue recommendations for revamping BCP/DR plans as a result of this review. Finra sent a letter to its securities firm constituency outlining its review, and asking for comment to be submitted by December 16 th in response to questions, emphasizing the following: Distance between main and backup site, and staffing (notably trading staff). Issues experienced with vendors backup plans (exchanges, clearing facilities, pricing feeds). Testing of BCP Plan, notably whether testing of electronic platforms was included (Arca) and whether a firm was ready to trade during/after the storm on Arca, the NYSE s backup plan. Remote access to electronic trading platforms and employees ability to connect remotely. Per the Wall Street Journal, Finra s review comes as the industry has yet to take action to substantially alter its preparations for future emergencies. NYSE, for example, said it intends to keeping its current BCP/DR Plan intact (from 2009) and expects only to make some minor changes. However, the onus is largely on regulators to initiate changes as, for example, the NYSE does not have the authority to mandate that brokers test the exchange s backup plan. This was underscored in the Former SEC Chairman s criticism of the NYSE s two-day Sandy closure, particularly for the fact that its DR plan does not feature a physical backup data center. Arthur Levitt said, To see the exchange go down for two days without an adequate backup plan is very, very unfortunate If you re going to have a stock exchange, it should have a backup facility of some sort so that regional events don t cause its closure. 12

13 Lessons Learned & Key Points to Consider In the wake of the storm, it is important to analyze how to be better equipped and prepared in order to avoid impacts to business continuity. The following BCP/DR Areas are Critical to Assess: BCP / DR Planning CoLocation The Cloud Communication Vendor Management / Service Providers Preparing Facilities 13

14 Lessons Learned & Key Points to Consider Business Continuity / Disaster Recovery Plans BCP / DR Planning 1 Include Business Continuity and Disaster Recovery in the design of a company s business model. Create a board committee dedicated to IT and network risks. Highlight BCP and DR as crucial components of an IT Risk Assessment. BCP and DR plans need to be highly detailed and include multiple scenarios and corresponding contingency plans. Account for short, medium and long-term conditions. Factor different degrees of a disaster s effects. Ensure plans are robust, regularly updated and approved by an organization s Executive Management. Should include a broad range of areas such as pandemic crisis management, media communication, hardware recovery and security measures. Designate clear chains of decision-making so that business is not at a standstill if key members of management are not able to communicate. Plan for possible use of alternate work sites (consider renting on an hourly/daily/weekly basis e.g. through insurance and real estate companies). Consider transportation and hotel accommodations for employees having to work at disaster recovery sites. Test the BCP/DR Plan on a comprehensive level: All data centers, data recovery, restoration of dependent applications, systems synchronization. Monitor and analyze the results of testing: Identify areas requiring special attention. Personnel that could benefit from additional training. Consider hiring experts to identify how the company s operations could be brought to a standstill, as well as vulnerabilities in the DR plan. 14

15 Lessons Learned & Key Points to Consider CoLocation CoLocation 2 Spread out data centers in different geographical locations. Establish two DR sites if possible one nearby and one more distant to head off issues caused by different types of wide area disasters. Close DR Site: To allow commuting to site if airlines are down. Distant DR Site: To mitigate wider range of infrastructure damage. Even pre-sandy, this has been a significant focus for regulators, who have required that enough critical staff be available for principal trading applications, especially for market makers. Post-9/11, the Interagency Paper did not take a stand on distance between sites. It did say that ongoing planning should consider geographic diversification. However, this might not always be practical, as larger distances may not be ideal or allow for real-time mirroring of data. DR plan should factor in individual conditions such as the likelihood and scale of a natural disaster (such as within the same flood plain), necessity to move a large portion of staff, and the possibility of having / needing multiple means of transportation. Ensure that employees are able to access their business IT environment with an outside IP address (e.g. not their normal internal IP address). 15

16 Lessons Learned & Key Points to Consider The Cloud The Cloud Consider putting key applications on the Cloud and utilize multiple service providers (such as Amazon Web services) to keep critical systems running. 3 However, do not assume that Cloud storage and hosting are infallible. This has single point of failure risk as a service provider may go down. Consider using a hosting environment for telephone system, network infrastructure and workspace. Research and understand the service provider s DR plans, as some operations are more geographically concentrated than others. 16

17 Lessons Learned & Key Points to Consider Communication Communication 4 During Hurricane Sandy, telephone and internet lines were decimated. 25% of cell phone towers lost power according to the FCC. Turn to alternate forms of communication: Satellite phones. Set up a Wireless Hotspot using a smart phone. Use Social Media such as Twitter and Google+ to communicate. Establish an alternate address (Gmail) for employees to use in an emergency in case of server failure and / or create a website or blog for employees to follow company announcements. Publicize the information so that employees are aware of its existence. Acquire a hosted phone / PBX which enables the control of phone systems with a smart phone in the event on-site communications infrastructure is demolished. Ensure current employee, management and client contact information is stored centrally / in the Cloud and is readily available. 17

18 Lessons Learned & Key Points to Consider Vendor Management / Service Providers Vendor Management / Service Providers 5 Review the BCP and DR Plans of vendors and third-party service providers. Ensure that plans include testing Consider including integrated testing with your firm. Include assurance in service level agreements that the vendor alerts the client when: The BCP/DR Plan is amended. There is an incident requiring the plans to go into effect. Have redundancy alternatives for a provider whose services can kick in if there is a failure of the main vendor. 18

19 Lessons Learned & Key Points to Consider Preparing Facilities Preparing Facilities for Impending Natural Disaster 6 Pre-disaster: Back up all data. Take snapshots of servers. Shut down non-mission critical servers. Shut down workstations, disconnect hardware and place on higher floors / above ground. Connect mission-critical servers to a UPS unit. Test the UPS functionality to ensure that it operates properly (critical servers continue to run). Proper shut down of servers if UPS fails. Ascertain whether buildings have backup generators available. Place generators, fuel and pumps in higher elevated areas off the ground. 19

20 Discussion Topics The Next Big Event New York City Has Experienced Terrorist Attacks Blackouts Anthrax Scare Hurricanes/Storms Questions for the Group What will be the next big disaster / security event? Is declaring a disaster annually at financial services firms the new normal? If climate change means rising sea levels, will banks be able to stay in Lower Manhattan (a Zone A flood zone)? Pandemic Risk (e.g. Avian Flu)? 20

21 Discussion Topics BCP Plan Amendments Questions for the Group As auditors, what adjustments may be necessary to your company s BCP / DR Plan given issues encountered during Sandy? When was the last comprehensive review of BCP / DR? Have certain BCP / DR program tools been leveraged in developing a tailored audit program (IAD, Finra & FFIEC have publications of programs / booklets which can be referred to last updates were likely to be pre-sandy ). Are regulatory examinations, results and experiences being factored into your company s BCP / DR plan? 21

22 Contacts at Sia Partners US Who is Sia Partners? Sia Partners is a leading global strategic and operational management consultancy. As the largest independent management consulting firm in France, it has approximately 400 consultants in eight countries with a strong presence in Europe, Dubai and Morocco. Sia Partners services major clients in Financial Services, as well as other industry sectors such as Energy/Utilities, the Public Sector and Telecommunications. Sia Partners US is expanding its practice and champions high-quality customized client service with senior-level consultants and subject matter experts playing a hands-on role. Our range of services to international financial institutions includes a specialty in risk management and control across the risk spectrum (operational, IT, market, investment, credit). We have significant expertise in project management and regulatory compliance, advising and assisting clients with Dodd-Frank, FATCA, AML, and Basel III rules, among others. Our IT Risk practice leverages experience and methodology to conduct IT Risk Assessments and enhance business continuity planning. Gus Moreno IT Risk Specialist Sia Partners US 641 Lexington Ave. Suite 1322 New York, NY Office :(212) Cell: (917) gus.moreno@sia-partners.com Daniel H. Connor Chief Executive Officer Sia Partners US 641 Lexington Ave. Suite 1322 New York, NY Office :(212) Cell: (862) daniel.connor@sia-partners.com 22

23 Appendix Finra: Targeted Examination Letter Targeted Examination Letters, November 2012 Re: Business Continuity Plans In coordination with the SEC and the CFTC, we are conducting a review of the impact of Hurricane Sandy on firms' operations and their ability to conduct business at a time when business continuity plans were enacted. Please be mindful to address both the securities and futures side of your business in your response. In connection with our review, we are requesting that responses to the following questions be provided in writing on or before December 16, 2012: General Briefly describe the portions of your Business Continuity Plan ("BCP") that were implemented in connection with Hurricane Sandy, if any. Did you utilize locations or take steps that were not part of the firm's BCP? Which business lines were deemed critical? For each of these business lines, how did you prepare staff before the storm? What obstacles did the staff have to overcome during and after the storm, including accessing alternative sites? If you planned to relocate employees to an alternate site, how many miles away was the back-up site from your main office? Was the location of this site far enough away from your primary location to avoid the same physical infrastructure problems? Was it on a different power grid, different central telecommunication circuit? If staff worked from home, were they able to do so effectively, including adequate system access? Did your alternative site have current data available and the necessary equipment and systems to recover and maintain critical operations or services? If not, what critical operation or services could not be maintained? Did you have sufficient and adequately trained staff at your alternative site? Is your alternate site able to be used effectively during a prolonged period in which your main facility cannot be used? When did your firm last test its BCP plan? Please state what was tested as part of the BCP plan. Was the BCP plan test a partial or full test? What were the results of the test? Describe any significant problems experienced with vendors or outsourcing providers, including exchanges, clearing facilities, pricing feeds, service bureaus or other regulated or unregulated entities. Please identify the date when these problems occurred and were resolved. Where were Compliance personnel located and what limitations, if any, were encountered in carrying out compliance responsibilities? How dependent were you on any single telecommunication system or other provider to perform? What functions does the single telecommunication system or other provider perform? What were the main lessons learned from the recent event regarding the effectiveness of your BCP? Will your BCP assumptions change? What updates do you plan to make to your BCP given the recent event and lessons learned? Is there anything the industry can do jointly to be better prepared for any future event? 23

24 Appendix Finra: Targeted Examination Letter Trading Where are the firm's primary trading locations? Where are the firm's alternate trading locations? How many trading staff are located at the primary locations? Does the alternate trading location require the physical presence of trading staff from the primary location? How was the alternate location staffed? What processes are performed to transfer trading activities from the primary to alternate trading location? Where/how are these processes performed? How did you last test trading at the firm's alternate location? Was testing performed in conjunction with market centers? What were the results of these tests? Does your firm have connectivity to the backup sites of the exchanges or other trading markets to which you send orders? If so how does such connectivity to the backup sites differ, if at all, from your firm's connectivity to the markets' primary sites? Did your firm test the various futures markets electronic trading platforms during BCP testing? If so, please state which markets you tested with, when, the scope of the test and the results of the test. Was your firm ready to trade on A90RCA as the primary NYSE market at the opening of business on Monday morning, October 29, 2012? If not, what issues would have prevented you from trading? Did you previously participate in testing of the "Print as N" initiative? If so, when and what were the results? Has your firm participated previously in the testing of BCP plans for exchanges or other trading markets (e.g., ATSs) other than NYSE? If so, when and what were the results? If not, has your firm previously been offered the chance to participate in the testing of such BCP plans but decided not to participate? What testing does the firm conduct to ensure that BCP systems (including Order Management Systems and Execution Management Systems) function to mirror all regular trading workflows? What is the firm's capability to trade products remotely? To what extent can your firm operate from the firm's alternate trading location (e.g. percentage of the firm's trading staff)? Full operation or partial resumption of business? Please describe how the firm defines a partial resumption of business (would the firm be limited in the products it could trade, the percentage of markets that could be made, etc.)? Prior to Hurricane Sandy, when was the last time trading occurred from the alternate trading location? 24

25 Appendix Finra: Targeted Examination Letter Customers How were communications with the firm's customers impacted during the contingency event? If alternative communication efforts were utilized, what were they and were they effective? Did customers have access to their funds at all times? Were customers able to fully engage in transactions when the securities and futures markets reopened? How does the firm generally receive orders from its customers? Did the firm experience issues receiving orders from its customers? If so, when was this problem rectified? Financial/Regulatory What if any regulatory relief was required and requested? Please include when relief was requested, the entity to whom relief was requested (SEC, FINRA, CFTC, NFA), and whether such relief was granted. Did the firm experience settlement issues, securities or futures, during the week of October 29 th? What issues did the firm have with its banks, DTCC, or futures clearing exchanges if any (e.g. clearance of securities and futures, margin, pricing, etc.) and how were they resolved? Did the firm experience issues with its books and records during the event or thereafter and how were they resolved? How is compliance with financial responsibility rules factored into the firm's BCP? Please note that your responses will be shared with the SEC, who may share them with the CFTC. Relevant Finra Links