How To Recover From A Trading System Failure

Transcription

1 Millennium Exchange - Oslo Børs cash equities and fixed income markets OSLMIT 601 Guide to Trading Services Disaster Recovery Issue April 2015

2 Important note This document has been produced by Oslo Børs to assist customers in the use of the Millennium platform on the Oslo Børs market place. Part of the documentation is based on documentation from Millennium IT and the London Stock Exchange Group. Where the document title includes a number (i.e. OSLMIT 201), the number corresponds to similar documentation from The London Stock Exchange and/or from Borsa Italiana. For more details, please see references in the appendix. If you have any general queries relating to this document, please technicalsupport@oslobors.no Further copies of this document can also be downloaded from the Oslo Børs website Disclaimer This document has been prepared on the basis of the best information available. Oslo Børs has taken reasonable efforts to ensure that the information in this publication is correct at the time of publication, but shall not be liable for decisions made in reliance on it. Oslo Børs will seek to provide notice to customers of changes being made to this document, but this notice cannot be guaranteed. Therefore, please note that this publication may be updated at any time. The information contained is therefore for guidance only. This document does not form part of the contractual documentation between the Oslo Børs and its customers. Oslo Børs OSLMIT 601 Guide to Trading Services Disaster Recovery 2

3 Change log This document can be updated at any time, and has been through the following iterations: Issue Date Description September 2012 Initial document release October 2012 First update, please refer to Appendix 1 for details November 2012 New release of the document as the Millennium platform is launched and the migration project is finalized. Minor wording changes to remove references to project and document links updated May 2012 Updated to include the Burgundy markets December 2014 Updated the name of the low latency market data channels and protocol April 2015 Removed references to the Burgundy market. Please note that only the latest issue of this document will be available from the Oslo Børs website. Details of the changes made in each issue of the document are described in the appendix. Changes from the previous issue of the document are indicated by a left margin bar. Oslo Børs OSLMIT 601 Guide to Trading Services Disaster Recovery 3

4 Content Important note... 2 Disclaimer... 2 Change log Introduction Purpose Readership Document series Trading Gateway Failure Loss of a Single Gateway General FIX Gateways (Trading, Post Trade and Drop Copy) Native Trading Gateway No Connection Possible Loss of Both Gateways Recovery Market Data Gateway Failure Other Component Failure Loss of Primary Site Approximate Schedule for a Site Failover Appendix 1 Issue updates Issue 1.0 Released 25 September Issue 1.1 Released 22 October Issue 2.0 Released 12 November Issue 2.1 Released 24 May Oslo Børs OSLMIT 601 Guide to Trading Services Disaster Recovery 4

5 1. Introduction In November 2012, Oslo Børs migrated its equity and fixed income trading to the Millennium platform, a flexible, highly scalable trading platform with ultra-low latency from MillenniumIT, a company in the London Stock Exchange Group. Oslo Børs aims to strengthen its competitive position by migrating to this new technology, and is pleased to offer customers a trading system of the highest standard that offers all the functions one might expect from a state-of-the-art trading platform. 1.1 Purpose This document provides an overview of how a client should handle unexpected downtime and failures from the Millennium trading platform, and how both Oslo Børs and the client should recover when the system is back up. 1.2 Readership This document should be read in conjunction with the other technical documentation. Messages from any of the interfaces are highlighted in bold text in this document. Fields within messages are indicated by use of italic letters. References to other documents in the series are indicated by use of underlined italic letters. 1.3 Document series This document is a part of a series of documents providing a holistic view of full trading and information services available from Oslo Børs post the migration to the Millennium Exchange platform. The current series of documents are outlined below: General OSLMIT Oslo Børs Market Model Equities OSLMIT Oslo Børs Business Parameters - Equities OSLMIT Oslo Børs and Nordic ABM Market Model Fixed Income OSLMIT Oslo Børs and Nordic ABM Business Parameters Fixed Income Trading OSLMIT201 Guide to New Trading System OSLMIT202 FIX Trading Gateway (5.0 SP2) OSLMIT203 Native Trading Gateway OSLMIT204 Post Trade Gateway (FIX 5.0 SP2) OSLMIT205 Drop Copy Gateway (FIX 5.0 SP2) Market Data OSLMIT302 FIX/FAST Gateway OSLMIT303 MITCH Gateway OSLMIT306 FIX/FAST News and Indices Gateway OSLMIT401 Reference Data OSLMIT Derived Information Guidelines Other OSLMIT 501 Guide to Testing Services OSLMIT 502 Guide to Application Certification Oslo Børs OSLMIT 601 Guide to Trading Services Disaster Recovery 5

6 OSLMIT 504 Guide to Dress Rehearsals (only relevant to the migration project) OSLMIT 505 Guide to Go-live (only relevant to the migration project) OSLMIT 601 Guide to Trading Services Disaster Recovery OSLMIT 602 Network Guide OSLMIT 604 Technical Parameters OSLMIT 605 Live Environment Connectivity OSLMIT 606 CDS Environment Connectivity OSLMIT 801 Reject Codes The latest version of this documentation series can be found on: Oslo Børs OSLMIT 601 Guide to Trading Services Disaster Recovery 6

7 2. Trading Gateway Failure Each participant connection (identified by a CompID) will be enabled for access to the trading system via the primary and secondary gateways for each trading interface; Native Trading, FIX Trading, Post Trade and Drop Copy. 2.1 Loss of a Single Gateway In case of an unexpected disconnection from a primary gateway, participants should attempt to re-connect to the primary gateway again, before failing over to the secondary gateway. 2.2 General Participants with Cancel on Disconnect enabled should be aware that any orders sent by CompIDs connecting through a failed Primary Native Trading or FIX Trading Gateway will be deleted subject to the configuration adopted. 2.3 FIX Gateways (Trading, Post Trade and Drop Copy) On re-connection after an application failover (not site failover), the FIX gateways will increment outbound (to the participant) message sequence numbers by a configurable amount to ensure that any messages in-flight at the time of failure can be recovered. You can find this configurable amount in OSLMIT 604 Technical Parameters. A configurable amount of messages may be cached while the participant is logged of. Following log-on to the gateway, the cached Execution Reports or Trade Capture Reports generated by the system while participants are disconnected (including deletions due to Cancel on Disconnect), will automatically be re-sent. 2.4 Native Trading Gateway Participants should first connect to the real-time channel before connecting to the recovery channel. Following this any missed messages or Execution Reports generated by the system whilst participants are disconnected can be recovered via a Missed Message Request. 2.5 No Connection Possible Failed connection attempts on both the primary and secondary gateways may indicate a serious issue, and Oslo Børs should be contacted for guidance. See also; Chapter 5 Loss of Primary Site. 2.6 Loss of Both Gateways If both the primary and the secondary gateways fail, no connection will be possible to the Millennium Exchange platform. While the issue is investigated and resolved by Oslo Børs any market control actions and regular updates will be communicated via announcements from Oslo Børs. Oslo Børs OSLMIT 601 Guide to Trading Services Disaster Recovery 7

8 2.7 Recovery Oslo Børs will generally recover the primary gateway first. Dependent on the nature of the recovery, FIX sequence numbers will either be maintained from the previous session or be re-set to 1. This will be communicated to participants prior to re-connection. Once participants have re-connected, recovery actions should be identical to that for loss of a single gateway. In addition, after recovery it is recommended that all participants: - Carry out an Own Trade Download to confirm which trades have been sent to clearing and settlement. - Carry out an Own Order Book Download to reconcile their view of the order book with the order book from the Millennium Exchange platform. 3. Market Data Gateway Failure In normal circumstances, both the market data A- and B-feeds will have upper case market data group names, for example A for MITCH Equities Full Depth. However, if one of the gateways fails (or a full site failover is carried out), only one feed will continue, and the market data group name will be changed to lower case, for example a. The sequence number will be reset to 1 as well. Even if the primary gateway recovers and gets active again, the name will not change back to upper case until the next trading day. 4. Other Component Failure Millennium Exchange is made up of a number of components, with the exact system response to failure varying from component to component. Participants should be able to handle the following situations that may occur in the event of failure. - Temporary increase in message latency as a primary process fails over to its secondary - Complete loss of both primary and secondary major components (e.g. Matching Engine) resulting in an automatic Session Halt for impacted instruments, and any messages for these instruments rejected at the gateways Recovery procedures and market control actions will vary according to the nature of the failure. Guidance will be communicated via announcements from Oslo Børs, but participants should be prepared to: - Carry out an Own Trade Download to confirm which trades have been sent to clearing and settlement. - Carry out an Own Order Book Download to reconcile their view of the order book with that of the Millennium Exchange platform or to confirm that no orders are currently active. Oslo Børs OSLMIT 601 Guide to Trading Services Disaster Recovery 8

9 5. Loss of Primary Site Millennium Exchange will operate in cold standby mode on the secondary data center (SDC) and in the event of total loss of the primary site (primary data center, PDC) Oslo Børs will activate the Disaster Recovery Site on the secondary data center. This procedure is expected to take in the order of 2 hours. Any market control actions and regular updates will be communicated via announcements from Oslo Børs. Once the Disaster Recovery Site is active, all order books will be cleared down and the trading system is restarted. Following this, participants will be asked to connect to the Disaster Recovery Gateways. Participants should re-connect to their designated primary gateways using the same IP addressing as for the primary site. All FIX message sequence numbers will be reset to 1 on the FIX Gateways. Customers must also reset their outgoing sequence numbers to 1 before reconnecting. Oslo Børs cannot guarantee that any re-sent messages from before the disaster will not be processed if customer outgoing sequence numbers are not reset. In the event of disaster, only those trades that have been sent to participants via a Trade Capture Report from the Post Trade Gateway can be guaranteed to have been sent to clearing and settlement (if applicable). Participants should disregard any trades for which only an Execution Report has been sent prior to the disaster. Participants should note that no updated Execution Reports will be sent identifying those orders that have been deleted due to either Cancel on Disconnect or Oslo Børs clearing down order books as part of the recovery procedure. Any requests for missed messages on the Native Recovery channel will not return any messages from the trading system. Since all order books will be cleared, you will receive a number of OrderBookClear messages on MITCH, and Market Data Snapshot with MDEntryType(269) set to Empty Order Book (J) on FIX/FAST. Oslo Børs OSLMIT 601 Guide to Trading Services Disaster Recovery 9

10 5.1 Approximate Schedule for a Site Failover Time Event - X Primary Data Centre (PDC) becomes unavailable 0 Oslo Børs decides to do a failover to the Secondary Data Centre (SDC), and notifies the market. At this stage both PDC and SDC are unavailable for the market participants, who should await further notice from Oslo Børs. +45 min SDC is available for clients to connect. Oslo Børs will inform the market participants when this happens. At this stage, the market data gateways will send heartbeats from the secondary multicast feed only, however the sequence number will be reset to 1, and the participants can log on to the trading gateways. +50 min Symbol Directory / Security Definition messages are disseminated from the market data gateways. Market data participants should clear the order books, either manually or by doing snapshot requests for the entire market. Trading participants should also do an Own Order Book Download and Own Trade Download. All order books will be empty, but trades executed prior to the failure will be available, including unpublished delayed dissemination trades. +60 min The markets are moved into the Halt state. OrderBookClear messages will be disseminated on MITCH, and Market Data Snapshot with MDEntryType(269) set to Empty Order Book (J) on FIX/FAST. Oslo Børs will inform the market of the time of the opening auction. Oslo Børs will use the period between the Halt state and the Re-Opening Auction Call to monitor customer connectivity, and will be able to provide both technical support and market control actions min Markets move into Re-opening Auction Call and regular trading commences. Oslo Børs OSLMIT 601 Guide to Trading Services Disaster Recovery 10

11 Appendix 1 Issue updates This appendix describes the details of the changes made in each issue of this document. Issue 1.0 Released 25 September 2012 Intial release of document. Issue 1.1 Released 22 October 2012 Section 2.3 FIX Gateways (Trading, Post Trade and Drop Copy) Description Clarified that this chapters is about application failover, and not site failover. 3. Market Data Gateway Failure The sequence number after a failover will be reset to Loss of Primary Site 5.1 Approximate Schedule for a Site Failover OrderBookClear messages will be disseminated on MITCH, and Market Data Snapshot with MDEntryType(269)=Empty Order Book (J) will be disseminated on FIX/FAST. Issue 2.0 Released 12 November 2012 New issue of the document post migration to the Millennium platform. No changes to content, except minor wording changes to remove references to migration project and updated document links. Issue 2.1 Released 24 May 2013 Section General Description New issue of the document to include the Burgundy markets. 5. Loss of Primary Site Since Burgundy shares the same infrastructure, matching engine and trading gateways as Oslo Børs, both Oslo Børs and Burgundy will fail over to SDC in the event of total loss of the PDC. Issue 2.2 Released 10 November 2014 Updated the name of the low latency market data channels and protocol. Issue 2.3 Released 28 April 2015 Removed references to the Burgundy market. Oslo Børs OSLMIT 601 Guide to Trading Services Disaster Recovery 11