1 FUTURES INDUSTRY BUSINESS CONTINUITY AND DISASTER RECOVERY 2009 INDUSTRY TEST RESULTS DR VI October 2009 Compiled and Developed by Tellefsen and Company, L.L.C.
2 TABLE OF CONTENTS Page Number: I. Background 3 II. III. IV. Executive Summary 7 Overall Test Results 10 Problems Encountered 23 V. Lessons Learned 25 VI. Suggested Next Steps 29
3 I. BACKGROUND The FIA Information Technology Division conducted the first annual Disaster Recovery (DR) Test in October Starting in Q2, 2009, the FIA Business Continuity Committee began detailed preparations for this year s sixth annual industry-wide test. A Working Group was convened to discuss and agree on a date, goals, objectives, etc.
4 BACKGROUND (Cont d) Two futures Industry BC/DR symposiums were held in September in New York and Chicago. Panelists included representatives from the major exchanges and clearinghouses: CBOE Futures Exchange CME Group (CBOT, CME, NYMEX, COMEX) CME Clearing The Clearing Corp. ICE Exchanges ICE Futures US ICE Clear US International Derivatives Clearing Group Kansas City Board of Trade KCBT Clearing Corp. MEFF NASDAQ Futures Exchange Options Clearing Corp.
5 BACKGROUND (CONT D) The scope of this year s initiative was designed to test disaster recovery back-up connectivity and functionality between exchanges, clearinghouses and member firms: Test firm back-up to exchange back-up sites (DR-DR). Verify connectivity. Test round-trip communications capabilities. The symposium sessions were well attended by numerous operations managers and BC/DR representatives from various clearing and non-clearing firms. Regular Committee conference calls were held between May and October (bi-weekly and weekly).
6 BACKGROUND (Cont d) The Committee including representatives from: Clearinghouses Exchanges FCMs Clearing firms Non-clearing firms Key service providers Independent software vendors Post-test interviews with firms and survey forms were used to fine tune the results.
7 II. EXECUTIVE SUMMARY The sixth annual industry-wide disaster recovery test in the U.S. financial services sector was highly successful, largely in part to the good working relationship between exchanges and the firms. Major U.S. and international futures exchanges, clearinghouses, FCMs and clearing firms participated in this year s test: 62 FCMs, clearing firms and non-clearing firms and 210 trading participants participated - an increase over last year ~96% of firms tested successfully
8 EXECUTIVE SUMMARY (CONT D) This year s test initiative was expanded to include several domestic and international exchanges: ICE Clear Europe International Derivatives Clearing Group Exchanges and clearinghouses demonstrated that their systems, processes and procedures simultaneously worked very well, communicating from back up systems/sites.
9 EXECUTIVE SUMMARY (CONT D) Feedback from the firms, exchanges and clearing houses indicated that this was a valuable and worthwhile exercise and that the industry should continue to do more common testing in the future Firms indicated that the test helped them: Exercise their Business Continuity/Disaster Recovery plans Identify internal single points of failure Tighten up and document their business continuity procedures Better understand the need for cross-training Test connectivity with exchanges DR sites
10 III. OVERALL TEST RESULTS 12 domestic and international futures exchanges and clearinghouses, 62 clearing/non-clearing firms* and 210 trading firms participated** in the test Test participants included clearing firms, nonclearing firms and non-clearing trading participants 66% of all futures clearing firms participated Participant firms represent a significant critical mass of futures order flow and liquidity: ~74% of overall futures exchanges volume. * Some firms are common members of multiple exchanges ** Represents non-clearing trading participants
11 CBOE FUTURES EXCHANGE Tested via member firms back up connectivity to production CBOEdirect trading platform Scripted trade entry for VIX futures contracts Transmitted trades to the OCC s back up systems via MQ Transmitted regulatory data to NFA via FTP.
12 CME GROUP / CME CLEARING Tested member firms back-up connectivity to production GLOBEX trading platform via CME remote back-up site. Entered a script of orders for various products. Received ex-pit, block trade information via CME remote site portal URL. Generated trade reporting and SPAN files from 10/23 trade date. Received PCS and large trader information from member firms. Transmitted trade registry data and SPAN files via FTP.
13 THE CLEARING CORP. Tested member firms back-up connectivity to back-up clearing system Confirmed system connectivity via successful system login message no scripted trade entry PCS, ex-pit and large trader information was not accepted Previous day s clearing reports was made available for firms to retrieve Clients tested MQM, FTP and Report Distribution processes Scripted trades were sent via MQM to clients that requested them.
14 INTERNATIONAL DERIVATIVES CLEARING GROUP Used the IDEX XT trading system from the back up site. Trade orders were received by IDEX Market Operations. Trades were reported and automatically matched and executed with a trade date of Saturday, October 24, Generated trade confirmations and transmitted to firms. NFX/IDEX trades were automatically processed for clearing and settlement to the IDCH.
15 ICE FUTURES US / ICE CLEAR US Tested member firms back-up connectivity to the ICE Trader electronic trading system. Scripted order entry for Cotton and Sugar futures contracts. Trade entry via TIPS system for test trades, ex- pit, large trader and PCS information. ICE Clear produced end-of-day files: Match off files, report files and SPAN files.
16 ICE Tested member firms back-up connectivity to the ICE Exchange electronic trading system DR site. Scripted order entry for Cotton and Sugar futures contracts. Tested ICE Block and other non trading functionality from DR site. ICE Web reporting via Internet portal for deal reporting, position reports, etc. Futures trades submitted to the ICE Clear DR site.
17 ICE CLEAR EU Tested member firms back-up connectivity to the ICE Exchange electronic trading system DR site Tested ECS, AFTS, Crystal Reports, ICE Block and other non trading functionality from DR site ICE Web reporting via Internet portal for deal reporting, position reports, etc. CMs submitted Large Trader files via AFTS Futures trades submitted to the ICE Clear DR site Completed a clearing run and generated regular clearing data files - banking and delivery reports and SPAN files.
18 KANSAS CITY BOARD OF TRADE Members tested connectivity to CME GLOBEX trading system Participated in post-trade testing - generated a subset of October 23rd trades to member firms Transmitted trade confirmations from KCBT Clearing Corp. Accepted EFP, change records, PCS and large trader records.
19 MEFF Tested via the MEFF production ETS environment Simulated a failure of their London POP and successfully failed over to the backup system during the test Members entered trades and received reports Transfer files were delivered from clearing Clearing data was restricted and not sent to member firms back office systems.
20 MINNEAPOLIS GRAIN EXCHANGE Trades were entered for MGEX products into the CME GLOBEX platform and cleared by MGEX Clearing using the MGEX DR Site TREX files were generated by the MGEX DR Clearing Server and placed on the MGEX DR FTP server The MGEX DR FTP server was accessible with the same logins and passwords as the production system.
21 NASDAQ FUTURES EXCHANGE Used the NFX/XL trading system from the primary production site Orders were received at NFX/XL Help Desk Trades were automatically matched and executed with a trade date of Friday, October 23, 2009 Fill reports were generated and transmitted to firms The NFX Trade File was processed for clearing and settlement.
22 OPTIONS CLEARING CORP. OCC successfully tested from their back up site Supported FTP+, Pull/Push, NDM and MQ file connectivity IP addresses and TCP Ports were unchanged, as they were in production for this test Firms submitted file transmissions and received output files.
23 IV. PROBLEMS ENCOUNTERED A number of problems were encountered, including: MQ connectivity and message output issues Delay in starting MQ channels, could not generate MQ messages Trades were not being passed to a trade management system Incorrect IP addresses/incorrect network configuration Firewall configuration parameters Failure of a firm gateway server Member firm help desk line was not working initially Exchange web portal connectivity problems/incorrect URLs Most were resolved quickly, although some caused an unexpected delay to test progress.
24 PROBLEMS ENCOUNTERED (CONT D) Order entry problem with test script used a symbol not set up on DR trading system Incorrect SSL certificate to access order entry website Clearing house end of day processing delayed as a result of change in the trade date Some firms did not verify that their back up servers and equipment was properly functioning prior to the test.
25 V. LESSONS LEARNED The futures industry proved that it is capable of successfully orchestrating an industry-wide disaster recovery test Most problems that were encountered were rectified quickly, although some caused an unexpected delay to the restoration of full processing capabilities: Under real life situations, most problems could probably be resolved within hours or by the next business day The exchanges and clearinghouses internal support processes and procedures worked well.
26 LESSONS LEARNED (CONT D) Firms and exchanges indicated that the test helped them: Identify some internal single points of failure Tighten up and document their business continuity and system fail over procedures Better understand the need for cross-training Test connectivity to/from DR sites Pre-test connectivity testing was valuable and should be a pre-requisite for future tests
27 LESSONS LEARNED (CONT D) Firms must be aware of any environmental impact on their network fire walls caused by test requirements: Pre-register via the FIA registration portal/participate in ping testing Have proper network staffing support actively engaged during the test Be aware of any firewall impact and make changes accordingly to accommodate testing
28 LESSONS LEARNED (CONT D) A number of firms apparently did not pre-register and showed up for the test at several exchanges: They were set up and most tested successfully, however, this placed an additional demand on exchange and clearing house support staffs Participation in pre-test ping testing should be mandatory for future test initiatives
29 VI. SUGGESTED NEXT STEPS Survey firms and exchanges for input to future testing Announce the details, goals and objectives of the next DR test - Saturday October 23, 2010 Broadly market future test initiatives to firms IT, operations, trading and senior management, to ensure a wider dissemination of test information within all firms Participation in pre-test ping testing should become a pre-requisite for all firms Coordinate the October 2010 date with SIFMA Consider conducting a regional scenario outage.