Nine Use Cases for Endace Systems in a Modern Trading Environment

Transcription

1 FINANCIAL SERVICES OVERVIEW Nine Use Cases for Endace Systems in a Modern Trading Environment Introduction High-frequency trading (HFT) accounts for as much as 75% of equity trades in the US. As capital markets evolve and the influence of HFT spreads it seems inevitable that more and more asset classes will ultimately be influenced by high-speed algorithmic trading practices. The race for ultra-low latency trading has already had a huge influence on the shape of the global trading landscape and, despite calls by regulators and commentators to regulate it, we see it as a permanent fixture. Accurate and timely data has always been at the very heart of the global trading system and over the past 10 years the trading community has made huge advances in both system and network design in its drive to acquire more information more quickly. As markets have become more competitive and the race to the lowest possible levels of network and system latency has played out, high-performance traders have started to look towards data analysis as the next competitive frontier. In any trading firm there are at least three different groups who are critically reliant on data to perform their specific functions. They typically include algorithm developers, traders and network operations teams who are responsible for network performance. Historically, these groups have all gone about gathering the data they need from different sources and have, to a greater or lesser degree, worked in silos with individual, often home-grown traffic capture solutions. Over time, organizations have discovered that the different data sources they use don t enable data to be correlated between the various sources, and even with contributions from each group, they are still unable to construct an accurate picture of network activity. To remedy the situation, traders have started to demand dedicated, purpose-built data capture systems that can provide everyone in the organization with the exact data they need. From this requirement, the concept of Network Monitoring and Recording Fabrics was born. Critical Infrastructure Endace Monitoring and Recording Systems have become critical-infrastructure network elements for many of the world s leading HFTs. Firms that have fully embraced the practice of capturing and monitoring all their network traffic have deployed fabrics of Endace systems across their entire networks to monitor all their trading paths, market data feeds and their general LAN/WAN traffic. The fact that Endace Systems can be remotely managed, configured, and controlled makes them ideally suited for remote deployment in co-location, proximity and service provider facilities that are critical, but often largely inaccessible. Endace Systems are regarded as critical systems for traders because: they deliver continuous 100% accurate packet capture at any line rate to 100Gb/s they write captured traffic to disk at 10Gb/s without loss they add an ultra-high-resolution timestamp to every packet as it is captured they provide truly open access with the ability to apply filters to all captured packets to extract just the traffic relevant to a specific need power to see all page 1

2 they support full global clock synchronization to ensure captured traffic is accurately timestamped wherever it is captured across the network they allow multiple applications to be run simultaneously on the same system. Capture Once, Use Many Endace Systems are designed from the ground up to enable multiple applications to be fuelled by a single source of 100% accurate captured traffic. It s what we call our capture-once, use many design philosophy and has influenced much of our technology strategy over the past few years. The ability to use the same source of traffic to fuel multiple different applications from the same hardware platform within a trading environment enables: Application performance improvements through improved quality / accuracy of packet feed Heat, space, power and CAPEX savings through hardware consolidation Application agility (the ability to deploy new applications without requiring additional hardware deployment) Workflow / MTTR improvements through application level event sharing. Use Cases Below are nine examples of how firms are using Endace Systems to address a range of different data-related issues. The multi-function nature of Endace Systems means that organizations can exploit multiple capabilities simultaneously. 1. Data Mining Within a trading firm a number of groups typically need to access traffic captured from the trading path and/or the market data path. Captured traffic is used by algorithm developers to refine, test, and tune algorithms, by traders for latency monitoring, and by network operations teams who use it for troubleshooting and packet level interrogation. The use cases for the data are quite different, but the requirement for the data to be highly accurate is common to everyone. By ensuring each group can access filtered packets from a common source of accurate data taken from various points in the network, the needs of all groups can be met. More importantly, each group can compare and contrast its insights and understanding of network issues and events in the knowledge that everyone is basing their analysis on the same set of accurate data. To facilitate data mining, Endace Systems support a Soap XML interface that can be programmed to capture trace files in.pcap or.erf format. Captures can be triggered based on system clock time, and multiple destinations can be defined to facilitate automated processing of trace files by remote applications. A wide range of different filters can be set up on the system to ensure that only the required packets are exported. Where the system is recording to its local disk (it s an EndaceProbe rather than an EndaceSensor) raw packets can be mined directly from the system s local store using the Packet Access web GUI. 2. Microburst Detection Microbursts are far more common than most people think; they are a feature of every modern network and there s almost nothing that can be done to avoid them. Microbursts present very particular challenges to traders as they have the potential to cause ticks and trades to get lost or delayed and this can have a profound impact on trading performance. Being able to identify and interrogate the impact of microbursts is essential. However, until the introduction (by Endace) of ultra-high-resolution timestamping, the ability to actually visualize and interrogate these microbursts was missing. Endace Analytics is a tool that is included on every EndaceProbe as standard. It delivers microburst detection and analysis as one of its many capabilities. Two things are unique about Endace Anaytics in respect to microbursts first, its ability to go down to a sub-microsecond view (which is where you start to see the real microbursts), and second, it enables rapid drill-down to the raw packets impacted (which can then be compared to the information seen by other systems in the path). In addition to providing visibility into microbursts, Endace Analytics enables network operations teams to be alerted of everything from bandwidth (over-) utilization to protocol distributions and visualise the impact of these using a sophisticated GUI. 3. Network Latency Monitoring Network latency has traditionally been a real problem for high-frequency traders and has driven the development of co-location and proximity hosting. Many different factors power to see all page 2

3 can cause latency, including router and switch performance and physical distance. For traders, having an accurate, realtime view of network latency across the trading network is of real value because with the right information they can factor network latency into their algorithms and compensate for it in real time. Latency measurement is relevant in the trading path, market data feeds and across the general LAN environment wherever time-sensitive information is generated and distributed. Endace Latency Monitoring is native to every Endace System. The application enables organizations to generate microsecond accurate Layer 3/4 latency measurements between any two points on their network. The application works by deploying GPS-synchronized systems at either end of a link and doing sophisticated pattern matching of traffic at both ends. A correlation engine, either hosted centrally (for large deployments) or on one of the two endpoints, generates minimum, maximum, and average latency measurements once-per-second. These are made available as a UDP feed to which trading systems can be connected using a simple API. Latency measurements are accurate to within 100 microseconds. 4. System Stress Testing Every feed handler, trading system, or network security system has a breaking point a point where its performance starts to degrade due to packet loss or saturation. Having an accurate understanding of exactly when a system hits its limits, and how it behaves as it gets close to those limits, is increasingly important for traders as the volume of data being processed continues to climb. Endace Capture Replay enables traders to record up to 16TB of real traffic from their network and replay it with absolute accuracy (including packet placement and timing intervals) at any speed up to 10Gb/s. When used in a controlled environment it s possible to determine exactly when a system fails and how that failure manifests itself. With this information network management teams can plan to ensure their systems stay within well-understood performance limits. 5. Back Testing From time to time things go wrong, and it s useful to be able to go back in time and analyze events in minute detail to find out exactly what happened. Traditionally this has been challenging for traders as it requires traffic to be captured with very high levels of accuracy, and the ability to replay that traffic packet by packet in what is effect HD slow motion (high-resolution replay). Endace Systems support the ability to capture real traffic and replay it in a variety of different ways, which can help traders to understand what happens under specific conditions. 6. Algorithm Benchmarking Replay is frequently used by traders to accurately benchmark the performance of one algorithm against another. To do this properly, firms need to build hardware-based test environments that remove (or at least control) unpredictable external hardware influences such as routers and switches. Unfortunately, pure software-based testing reveals only 90% of the performance picture. It s only when an algorithm is tested using controlled replay in a real-world environment with existing hardware that its true performance can be accurately determined. 7. Market Testing The ability to figure out whether a new market or asset class is going to be profitable for a firm is critically important. Traditionally it s been hard to accomplish this because there are many different variables to consider. Two variables need to be considered: 1) are the systems in the ground going to be able to handle the additional load? 2) are the algorithms that are available going to perform? Traders can use Endace s Capture Replay functionality to test these two variables. A key feature of Capture Replay is the ability to blend data sources with very high levels of accuracy (slotting the packets into the precise order they would have been recorded in had they been captured at the same time). By taking a data feed captured from a new market, blending it with standard feeds and then playing it through the algorithm (as per use case 6) firms can quickly determine whether a particular market is going to be profitable or not. application dock 8. Trade Latency Monitoring Because of their high degree of capture accuracy, Endace Systems have long been power to see all page 3

4 popular with third-party software vendors offering trade latency monitoring applications. Monitoring flow-based transactions is immensely challenging because of the number of protocols involved in a single transaction as it passes from trader to venue and back again. Full visibility into complete flow-based transactions is of immense value to traders who are then able to pinpoint and address latency issues across every part of the network. Every Endace System provides the Endace Application Dock a high-performance virtual hosting environment that allows customers to run up to six simultaneous applications on a single System, each leveraging the same source of 100% accurate traffic. This enables traders to deploy their chosen mix of custom, third-party, or open-source tools as well as using the built-in tools such as Endace Analytics and Capture Replay that we ve discussed in the use cases above. We ve partnered with the leading vendors in this market to provide trading firms with flow-based trade latency monitoring applications that can run in the Endace Application Dock. Current trade latency monitoring partners include Correlix and SeaNet. 9. Consolidated Tape There s a lot of discussion under the auspices of MiFID2 about the requirement for traders to be able to produce a highly accurate record of every transaction they are involved in. The May 6 Flash Crash exposed the lack of visibility and standardization of reporting and the SEC s inability to rapidly pinpoint a root cause undoubtedly caused confidence in the market to decline. Whether it s ultimately mandated by law, or generally accepted as industry best practice, we believe traders should be recording and storing a carbon copy of every transaction they make during any trading period. With up to 32TB of storage and a SOAP interface, Endace Systems are ideal for recording and archiving trading data. power to see all page 4

5 Deployment Scenarios Endace multi-function systems are deployed in data centre environments using passive optical taps. The exact use case determines the precise physical deployment. The two deployment diagrams below are designed to broadly show, at a very high level, where systems are deployed for specific use cases. The numbers on the diagrams pertain to the use cases described above. Diagram 1: Deployment in a co-lo, proximity or service provider facility. TRADE VENUE CO-LOCATION FACILITY MATCHING ENGINE 10Gb/s VLAN SANDBOX TRADER n CORE DESK ORDER ALGORITHMIC FLOOR NETWORK OPERATIONS OTHERS CORPORATE LAN ENDACE SERVER power to see all page 5

6 Diagram 2: Deployment in a more traditional corporate data center environment SANDBOX MATCHING ENGINE SANDBOX TRADE VENUE 2 MATCHING ENGINE 10Gb/s VLAN TRADE VENUE 1 REMOTE DATA CENTRE CORE OFFICE NETWORK ORDER ALGORITHMIC FLOOR CORPORATE LAN ENDACE SERVER NETWORK OPERATIONS OTHERS For more information on Endace products visit: For enquiries enquiries@ power to see all page 6