Powerful Management of Financial Big Data

Transcription

1 Powerful Management of Financial Big Data TickSmith s solutions are the first to apply the processing power, speed, and capacity of cutting-edge Big Data technology to financial data. We combine open source software with our proprietary software and customized hardware to manage Petabytes of data at a low cost and with powerful indexing, mining and delivery capabilities. Our platforms are resilient, scalable, easy to manage and flexible. We have variants for specific uses. TickSmith s platforms, such as TickVault, can be extended to offer client-specific data storage for retention and compliance. They can also be monetized by firms that manage cross-connect networks by up selling retention/management/delivery of data to internal and external users. For trade and quote data, and more. For FIX messages and trading history. For back-office data. Global trade & quote data available as a service. TickVault TickVault is our big data platform for storing, managing and delivering financial trading data such as trade & quote history. It is ideal for exchanges, trading networks and data vendors who need to cost effectively manage and extract value from mountains of financial data. TickVault is a comprehensive repository for both structured and unstructured data and is ideally suited as the central repository of content in any Quant Cloud application that a financial institution might want to provide to its users. Many suppliers of analytical applications, both server and cloud-based, are already partnering with TickSmith to ensure their analytical applications can source their data from TickVault.

2 TickVault - Flexible Deployment TickVault is provided as a complete platform as a service or managed and installed on site on Client premises. In the latter case, TickSmith provides and configures all the hardware based on agreed sizings and capacity requirements. When accessed as a service, TickVault is accessed from the Montreal-area Tier-3 data center where it is currently hosted. Access to and from is through the internet, the latter being connected through a dedicated 1 Gbps Internet link. VPN, dedicated links or any other secure mean can also be provided. Multiple deployment options are available: As a managed service, offered from TickSmith s infrastructure As a dedicated managed platform hosted at one or multiple Client data centers As a combination service and managed platform TickVault provides flexible delivery of data through its REST API. A web interface already exists and can be used as the basis for web delivery of data. TickVault s powerful architecture ensures rapid response to small and large data requests and queries and is also ideal for data mining.

3 TickVault - How It Works TickVault is built around a core of Big Data technology, including the Hadoop Ecosystem. As such it is ideally suited to manage, index, analyze and deliver huge amounts of structured and unstructured data. Data loading and storage The first step is to load the various data sets that Client produces and want to manage with our platforms. Early access to sample data helps establish an accurate sizing of the platform and the effort needed to put in place the various loading processes. TickVault supports both structured and unstructured data and loads data from files and, in the case of real-time feeds, snaps dropped to files. As described in the diagram, the data is first stored as is in a big data friendly format similar to the original format in which the data was received. This ensures that we can retrieve and regenerate files that are true to the originals. The platform has built-in compression and data is stored and managed compressed. Compression ratios for binary data are typically 3 to 1. As part of the storage process, 3 copies of the data are created and stored on distinct hardware within the platform to ensure redundancy and leverage the parallel processing that is inherent to TickVault. The platform has built-in self-healing and replication capabilities meaning that any hardware failure simply requires a replacement of the affected hard disk or server and TickVault will regenerate the missing data from the other copies.

4 Once loaded the data is readily available to be delivered in its original format through TickVault s delivery mechanisms. TickVault can return data as it was received, and ensure that the data returned is the exact same as what was originally processed. In addition to delivering data as-is, TickVault can be configured to create value-add versions of the data such as roll-ups, adjusted data, statistics and more. Data transformation A powerful feature of TickVault is its ability to transform or "process" data to ensure that the data that is delivered to end-users is accessible in formats that make it readily usable for their specific applications. Examples of processing include: Creation of trade and quote roll-ups/bars at specific intervals (1 minute, 5 minutes, 1 hour, etc) Creation of trade and quote data adjusted for splits, ticker changes and dividends Creation of corrected trade and quote data (i.e. correction messages applied) Creation of derived data such as VWAP fields or currency converted trade and quote data Creation of "snap files" with a trade and quote at a specific interval (every 5 minutes) Creation of files combining data from multiple data sets Creation of format specific files, including combination of fields and output formats Extraction of specific instruments or lists of instruments Extraction of complete exchanges for a specific period (1 day or 5 years, for example) Creation of reports such as market or instrument statistics Data mining and searches, with different levels of complexity, can also be configured as processes The configuration of transformation processes is based on requirements communicated by Client and done by TickSmith staff. Once a process is in place it can be invoked to run on the fly. For access to complete universes of data or to transformed content that is requested often, transformation processes can be set-up to completely transform existing data and store the results as a separate data source. This is referred to as the transformed data cache in the diagram. Monitoring All processes, from loading to delivery, are monitored using an enterprise-class monitoring solution. Alerts are forwarded in real-time to our team for immediate action. Entitlement and control Data and user entitlements are managed by config. files on the TickVault platform, LDAP is also supported. Both can securely be extended to integrate with a client s entitlement system or DB.

5 Data Delivery Core delivery is done through a fully documented and easy to use REST API which has the ability to deliver data as files and as streams. TickSmith has also built a basic Web interface that allows easy access to files. The Web (HTTPS) interface is built on top of the REST API. An FTP mechanism also allows access to files. Secure FTP (SFTP) and SSH File Transfer Protocol (SCP) will be supported shortly. Example web interface delivery of HiTS data About TickSmith TickSmith offers cloud-based and on-site solutions to exchanges, trading networks and financial institutions that must accumulate huge quantities of financial data. Based on disruptive "big data" technology, our platforms and powerful indexing and retrieval tools now make it possible to cost effectively manage and extract value from mountains of financial data. TickSmith is already managing 1 Petabyte+ of data. Our background is in financial technology. We have technical resources that understand financial institutions environments, its specificities and how to efficiently code corresponding Big Data solutions. We make financial big data usable for exchanges, trading networks and financial institutions Contact us at Info@ticksmith.com for more information. TickSmith is a member of