High Performance Time-Series Analysis Powered by Cutting-Edge Database Technology

Transcription

1 High Performance Time-Series Analysis Powered by Cutting-Edge Database Technology Overview Country or Region: United Kingdom Industry: Financial Services Customer Profile builds data and analytics management solutions used by trading, risk, IT, and back-office staff at major financial institutions worldwide. Business Situation Financial trading institutions need high performance analysis of tick and time-series data. wanted to provide these performance levels using standards-based technology. Solution Microsoft SQL Server 2008 will support s proprietary time-series technology. Developers used Microsoft.NET Framework Common Language Runtime to create a high performance data repository. Benefits Performance enhanced. Management simplified. Productivity increased. Operational risk reduced. Compliance needs met. A database that handles half a trillion ticks is large enough to meet many of our customers needs and we are confident that they will be impressed by the solution we re offering on SQL Server Financial technology provider built its TimeScape data and analytics management suite on proprietary time-series database technology. This was to meet the industry s extensive performance needs in the management and analysis of market data. But given the expertise needed and cost involved in managing proprietary database solutions for many institutions, the company sought to find a way to deliver the same performance using a standards-based technology approach. The company chose to support its time-series technology on Microsoft SQL Server 2008 database software. This hosts Microsoft.NET Framework Common Language Runtime, helping developers to optimise SQL Server to create a repository for time-ordered data. SQL Server 2008 not only offers the proven performance levels needed to support trading and risk management, but also boosts IT productivity with advanced inhouse management tools.

2 The management tools that come with database systems such as Microsoft SQL Server are very mature. Situation delivers real-time analytics and data management solutions to financial markets. With offices in London and New York, its customers are investment banks, hedge funds, and asset management institutions across the world s financial centres. They use s sophisticated TimeScape technology, a real-time analytics and data management suite, to accelerate trading decisions and reduce risk. This technology manages growing volumes of increasingly complex data within trading and risk management, making it immediately available wherever and however they wish to analyse it. Trading institutions in particular, need to store time-ordered data the series of prices of a stock every time it is traded. Regulations, such as the European Union s Markets in Financial Instruments Directive (MiFID), demand complete transparency into trades that are executed to increase competition and consumer protection in investment services. In short, transparency requires information such as best bid and offer, trading day and time, instrument identification, and value to be easily presented. The challenge for financial organisations is to ensure their IT systems can provide immediate and continuous availability of this data., explains that, traditionally, proprietary solutions have best handled this type of data, because standards-based, relational databases work in different ways. Tables make up the basic structure of relational databases, with information represented in columns and rows. If you were to put financial time-ordered data in a standard relational database, the table would have many billions of rows, he says. As that table gets larger, it becomes harder for a relational database to quickly extract subsets of records from the table, which is something that trading tools generally need to do. But, today, financial technology vendors such as are experiencing a push from their customers to provide solutions on standards-based systems. Often, the knowledge within IT departments is based on standards, and this type of technology is easier for them to manage. The management tools that come with database systems such as Microsoft SQL Server are very mature, says Sentance. IT teams are looking to avoid proprietary systems that require them to learn a new set of technologies. The main problem is that key users such as traders and risk managers demand the high performance and speed of execution often associated with specialist database systems. Historically, this has been extremely difficult to achieve in relational databases. But, as a leading financial technology provider, was determined to close the gap between manageability and performance, and help its customers boost productivity. Solution Working closely with the Microsoft Developer and Partner Evangelist team, chose to develop TimeScape on Microsoft SQL Server 2005 database software, and transfer the solution to SQL Server 2008 in time for its release. A full proof of concept project followed an architectural design session, to identify the best approach for creating a solution with the performance of its previous specialist system. The answer to the challenge lay in the technologies within SQL Server, which made it possible to extend the database s performance to trading data standards. SQL

3 SQL Server 2008 is a more extensive enterprise offering, and provides many benefits that help boost and extend performance, particularly in scenarios where customers are dealing with large user numbers. Server 2005 hosts the Microsoft.NET Framework Common Language Runtime (CLR). This helps to ensure that programmers can work more efficiently by using an enhanced database programming model. They can write new procedures and functions in any of the CLR languages, such as the Microsoft Visual C#.NET development tool and the Microsoft Visual Basic.NET development system. Using these tools, was able to store complex instrument and market data in a highly compressed format within binary large objects (blobs), while making it available transparently to its existing toolset. In this way, column/vector-based storage has been implemented within SQL Server. Sentance says: Not only can we effectively host our database inside SQL Server, it becomes a very high performance repository for time-ordered, time-series data. It means the data is exposed in a manner that our customers are familiar with and can easily manage in-house, which is something that would be more difficult with competitor solutions. While TimeScape is due to launch initially on SQL Server 2005, is transferring the technology to SQL Server 2008 to further enhance its performance and scalability. SQL Server 2008 offers services well beyond simple relational database storage, and meets the needs of the next generation of data-driven applications. It provides a robust database engine that supports large relational databases and complex query processing. is taking advantage of its many enhancements. For example, administrators can configure pools of resources by using the Resource Governor component, helping IT teams prevent bottlenecks and performance slowdowns. Sentance says: SQL Server 2008 is a more extensive enterprise offering, and provides many benefits that help boost and extend performance, particularly in scenarios where customers are dealing with large user numbers. FILESTREAM support in SQL Server 2008 links data stored on the file system to the database. This data is managed and accessed directly from the database, but not stored there, offering even greater system performance. We are particularly excited about the new FILESTREAM features within SQL Server 2008, which show great promise in our initial tests, says Sentance. We hope to use this feature in a future version to further enhance our time-series solution. Benefits is confident that the TimeScape solution running on SQL Server 2008 offers more than just the high performance traders, risk managers, and IT staff require. The new features in SQL Server 2008 also provide extensive database management functionality, expected to increase productivity within IT departments, as well as advanced features that help the business to comply with strict data transparency regulations. Even before the product s release on SQL Server, customers have expressed significant interest in the solution, which has the potential to reverse preconceptions of standards-based databases in the financial industry. Significant Performance Enhancement To prove its performance levels, and Microsoft recently conducted tests to benchmark the SQL Server 2008 solution on databases of up to 8.5 terabytes of market data, numbering 500,000,000,000 (half a trillion) individual ticks of data. Sentance explains: With data

4 SQL Server 2008 fits with the standardsbased approach that the IT departments demand, while still delivering the tick and time-series performance that business users expect. volumes rising, we need to come to market with credible figures to gain the confidence of the industry and our existing customers. A database that handles half a trillion ticks is large enough to meet many of our customers needs and we are confident that they will be impressed by the solution we re offering on SQL Server He says the issue of database performance measurement is a contentious one in the industry, because many database vendors offer results that are difficult for the market to understand. Our customers want very clear information about the performance of our solution, and in this regard we decided for this first benchmark to test random access to tick data physically stored on disk and not held in memory. We aimed for time-series performance that was at least an order of magnitude better than traditional row-based solutions and comparable to our existing proprietary database solution. I am pleased to say we achieved this with our SQL Server 2008 implementation, which additionally offered very low levels of contention for multiuser access, and no material degradation of performance with increasing database size. clients have already expressed significant enthusiasm for the TimeScape solution based on SQL Server Almost every client and prospect that we mention it to says they d like to move away from proprietary solutions towards a standards-based approach, says Sentance. They re extremely keen to work with the SQL Server 2008 solution, and our results show that in no way is performance compromised. Simplified Management Users such as traders and the risk managers will still see the same financial instruments, performance levels, and execution speeds to which they re accustomed, but customers IT teams can expect to experience a significant difference. The standards-based technology helps to ensure that company administrators can now handle management of the database without having to rely on specialist development teams. Our clients feel that their developers will be able to spend more time focusing on value-added activities, such as creating more advanced user functionality, says Sentance. SQL Server 2008 brings numerous other advantages, including sophisticated system health management tools such as SQL Server 2008 Performance Studio. This provides complete performance monitoring with analytical reporting of data. In addition, SQL Server Replication Services replicate and synchronise information for greater reliability and accessibility of information. Sentance explains that features such as these simply aren t available with proprietary solutions. These tools will help to ensure that larger institutions will be able to share the administrative load, because their database administrators are comfortable with SQL Server databases. SQL Server 2008 fits with the standardsbased approach that the IT departments demand, while still delivering the tick and time-series performance that business users expect. he says. In fact, we expect they ll see some performance improvements, particularly when they have a large number of concurrent users of the system. Increased Productivity Traders and risk managers often use familiar Microsoft Office Excel spreadsheets to analyse statistics and trading opportunities. Typically, they ve relied on IT departments, which have the skills to manipulate database information

5 Advanced data querying and auditing within SQL Server 2008 ensure that market data is completely transparent". to produce daily reports. Now, data is often analysed in real time, so businesses can snapshot price levels across multiple markets and time zones. Consequently, it s increasingly important for users to generate ad-hoc reports at any time, without having to wait, but at the same time avoid the operational risk that can result from unstructured spreadsheet development. SQL Server 2008 offers seamless integration with Office Excel 2007, giving traders familiar tools to manipulate and examine database content. This way, financial organisations can extend business insight to these users, so they can render reports by using highly sophisticated data mining within SQL Server Analysis Services. While the spreadsheet remains as the end-user viewing tool, operational risk is reduced through using Office Excel 2007 as a reporting and analysis tool, and not as a database for storing data. TimeScape also offers tight integration between the database and the end-user through more than 300 easy-touse add-in functions to Office Excel 2007, and access to TimeScape QL+, s market data-aware query language. Being able to offer standardsbased SQL Server as well as optimised QL+ languages to access and analyse data, was another key design consideration, making use of user-defined functions within SQL Server 2008 and the.net Framework CLR. Reduced Operational Risk In addition to tight integration with Office Excel 2007, has developed a solution called TimeScape SpreadSheet Inside, which allows SQL Server 2008 to store, control, and run spreadsheet operations centrally. This means that extremely complex procedures can be defined simply and quickly using a spreadsheet graphical user interface to edit the calculation within SQL Server As a result, complex expressions involving tick/time series, market data preferences, and pricing models can be defined and used throughout the solution, significantly reducing operational risk and increasing productivity. is currently working on the use of Excel Services, part of Microsoft Office SharePoint Server 2007, to further enhance this approach. SQL Server 2008 stores the description of spreadsheet calculations, says Sentance. It means that we have a high performance, time-series database, with the storage and the calculations for spreadsheet operation within it. From a user point of view, the tools provide similar functions to Excel, but avoid the operational risks of having the calculations run on desktops as opposed to running centrally and being controlled. Alignment with Regulatory Compliance Needs With compliance issues such as MiFID placing strict demands on businesses, and audit trail and data validation a key requirement for effective risk management, is confident that its TimeScape technology will stand up to current and future market and regulatory needs. SQL Server 2008 helps to ensure businesses can easily audit activity on data, including information about when data has been read, as well as any modifications. Sentance says: Advanced data querying and auditing within SQL Server 2008 ensure that market data is completely transparent. It s possible to see what amendments have been made and who approved the amendments, which is a regulatory request that a number of our clients are implementing. In this industry, any information that they can provide is important.

6 For More Information For further information about Microsoft products and services, please visit or call * For hearing impaired customers with a Minicom, contact: * *Lines are open 8am 6pm, Monday to Friday. Please note, numbers prefixed 0870 will be charged at national call rates. For details of national call rate charges, please contact your telecommunications provider. For more information about products and services, visit the Web site at: Microsoft SQL Server 2008 SQL Server 2008 is a secure, reliable and scalable platform for business critical applications with transparent encryption, comprehensive auditing and enhanced mirroring, while reducing the time and cost of managing data with enterprise wide policies. SQL Server simplifies development of data driven applications with integration of Visual Studio and.net Framework, support for any type of data, and consistent storage from the device to the data centre. SQL Server delivers the right information to all users with advanced visualizations, powerful reports and integrated analysis. For more information about SQL Server 2008, please visit: Software and Services Microsoft Server Product Portfolio Microsoft SQL Server 2005 Microsoft SQL Server 2008 Microsoft Office Microsoft Office SharePoint Server 2007 Microsoft Office Excel 2007 Technologies Microsoft.NET Framework This case study is for informational purposes only. MICROSOFT MAKES NO WARRANTIES, EXPRESS OR IMPLIED, IN THIS SUMMARY. Document published March 2008