A Fully Standards-Based Approach to Logging High-Throughput Distributed Real-Time Data. Leveraging the DDS and SQL Standards

Transcription

1 A Fully Standards-Based Approach to Logging High-Throughput Distributed Real-Time Data Leveraging the and SQL Standards Mark A. Hamilton And Edwin de Jong Real-Time Innovations 3975 Freedom Circle Santa Clara, CA (408) Abstract The ability to capture run-time activity for future analysis and playback is vital to being able to debug, integrate, test, and support complex distributed systems. Unfortunately, implementing a logging capability is often very difficult and expensive because data must be captured from disparate sources, likely consists of many different native data structures, and often has a higher aggregate throughput (burst or steady-state) than can be supported by traditional commercial databases or even hard disks in some cases. As a result, distributed application developers traditionally have had to implement their own custom logging capabilities and tools to go with them. By integrating widely-used standards and high-performance database technology, a fully standards-based and off-the-shelf approach to logging high-throughput distributed real-time data is now possible. This solution uses the Object Management Group (OMG) Data Distribution Service () standard to distribute real-time data to one or more instances of a Structured Query Language (SQL) database that provides persistent storage. When used with a memory-optimized SQL database, optionally as a cache to a traditional disk-centric database, throughput rates can be supported and sustained that are an order of magnitude higher than most legacy databases support. A significant benefit of this standards-based logging approach is that applications and tools already using the or SQL standards can take advantage of this solution with little or no change. An off-the-shelf implementation of this logging solution is now provided by RTI. RTI Real-Time Connect enables the development of systems that can: efficiently bridge real-time traffic to/from SQL and other enterprise applications, route high-performance traffic between multiple domains, and capture/log message flows persistently. In conjunction with its logging capability, the RTI Real-Time Connect can be leveraged to support the design of a message replay system as well. 1

2 In this paper, we will focus on applying the capabilities provided by RTI Real-Time Connect in the design of a generic message logging subsystem. In addition, we ll also address performance issues related to logging with respect to sustainable rates at which real-time messages or data produced by RTI s solution, RTI Data Distribution Service, can be captured in Oracle TimesTen In-Memory Database and Oracle 10g. We ll also touch on how TimesTen Cache Connect to Oracle can be employed to handle burst message rates during logging activity. Design of a Generic Message Logging Subsystem Being able to persistently log specified messaging activity from various systems and subsystems within a complex distributed system provides a key role in being able to monitor, maintain, and debug system behavior. Once captured, these message logs can be further utilized to support post processing, as well as message replay for further system testing and maintenance within a controlled environment. The standard is particularly well-suited to distributing real-time data for logging as well as for general distributed application development and system integration. The standard has been widely adopted for real-time systems, including in industrial automation, aerospace, and defense applications. specifies an API designed for enabling real-time data distribution. It uses a publish-subscribe communication model and supports both messaging and data-object centric data models. offers several enhanced capabilities with respect to content-based filtering and transformation, per dataflow connectivity monitoring, redundancy, replication, delivery effort and ordering, as well as spontaneous discovery. Furthermore, offers new capabilities with respect to data-object lifecycle management, best-effort and predictable delivery, delivery ordering, resource management, and status notifications. System Requirements Our design objective is a generic Message Logging subsystem. Specifically, we will design a facility that supports the ability to: - log messages from one or more domains, - log messages persistently, - introduce minimal change and negligible performance impact on existing enabled systems, - provide high-performance logging on low-cost off-the-shelf hardware, - provide support for runtime logging configuration, - discover message formats dynamically at runtime, - provide individual message stream logging configuration/control, - provide bounded and unlimited message logging duration. Figure 1 illustrates the fundamental components of a generalized message logging subsystem. The subsystem must interface with applications as well as one or more database management systems. The design will leverage a relational storage model. One of the most compelling reasons for leveraging a relational database is its self describing capability and standards based compliance. In addition, storing information within a relational model opens a vast array of post-processing tools as well. 2

3 Logging Subsystem Message Discovery & Capture Runtime Configuration Engine Relational Database Interface Relational Database Figure 1. Generic Message Logging Subsystem Runtime Configuration The Runtime Configuration Engine controls which messages will be logged to the database. In addition it provides various Quality-of-Service (QoS) settings : - Whether to sub-sample (based on time) the message stream, or simply capture all samples, - the depth of the amount of logging to capture for each message stream, - From which network (i.e. domain) to log a particular message stream from, - The ability to re-configure during runtime. This includes adding a new message to log, indicating which message stream to stop logging, or altering the way in which a current message stream is being logged. Automatic Discovery and Storage Once configured via the Runtime Configuration Engine, the logging subsystem must automatically discover both topics (i.e. unique messages) and their associated message formats (i.e. IDL types). If the message is keyed, then all instances must also be captured. For each message stream that must be logged, as indicated by the Runtime Configuration Engine, the Message Discovery & Capture module simply creates a subscriber for the corresponding topic and data type. The Runtime Configuration Engine controls the actual QoS settings of the subscriber. By using subscribers, the design will introduce minimal performance impact on the existing systems 1. Once the messages have been discovered and are being captured, they will be stored in a relational database. The Relational Database 1 It s important to realize that if every message of a specific topic stream must be logged, then the system that publishes that topic must be configured for Reliable QoS with appropriate History QoS policy. 3

4 Interface module is responsible for inserting each of the message samples into their respective relational tables within the database. Thus if the logging subsystem is configured to capture 50 message types (i.e. topics), then 50 relational database tables (within one database) will be created to persistently store each distinct message stream. Applying RTI Real-Time Connect Next we discuss how RTI Real-Time Connect implements the design of the generic logging subsystem as laid out in the previous section. RTI Real-Time Connect seamlessly integrates applications with enterprise applications by using a relational database as the interface. The database also serves as a buffer giving enterprise applications direct access to real-time data in a way that doesn t negatively impact the performance of real-time applications. Furthermore, RTI Real-Time Connect is designed to integrate existing enabled real-time applications with enterprise based systems with minimal modification to either working application. Figure 2 illustrates the RTI Real-Time Connect architecture. SQL RTI Real-Time Connect Relational Database Figure 2. RTI Real-Time Connect Architecture. RTI Real-Time Connect operationally consists of a configurable number of publishers and subscribers, which act as a bridge between the networks and the relational database. It further enables automatic logging of messages by mapping the Topic to a table in the database and storing an 4

5 instance of a Topic as a row in that table. To be complete, RTI Real-Time Connect can automatically manage the publication of database table changes out to a specified network, but that capability won t be utilized within the logging subsystem design. RTI Real-Time Connect Salient Features In order to better understand how RTI Real-Time Connect can be used as the basis of our logging design, let s briefly list key features that we ll build upon: Automatic discovery of messages being published, Determines message format automatically via typecodes, Automatic storage of received messages within DBMS tables, IDL/SQL Data Mapping, Runtime configuration via meta-tables and QoS, Message History support both bounded and unbounded. Automatic discovery and mapping A key RTI Real-Time Connect feature is that once configured and connected to a network, it will automatically discover the messages (and their type format) being published. Once discovered, message format information is automatically mapped to SQL table schema, and a corresponding database table is created in order to allow the logging of the incoming messages. So, without any user interaction, RTI Real-Time Connect can create, and populate relational tables that will log messages. Each discovered topic will result in a relational database table. Let s explore how RTI Real-Time Connect handles each of these functional steps. Automatic Topic and Type Discovery RTI Real-Time Connect can be configured to listen on one or more domains for message traffic and auto-discover the topic and its data type. RTI Data Distribution Service provides built-in topics that can be used to discover remote entities. By creating a subscriber to the built-in topic DCPSPublication, RTI Real-Time Connect can automatically discover topics. Once the topics have been discovered, their format can be determined by parsing the typecode information. Typecodes provide a standards-based mechanism to represent arbitrarily complex IDL types at run time. They are used to send information about the application defined data format on the wire, thus enabling applications to publish or subscribe to topics of arbitrary types. Since typecode information is automatically propagated during the discovery phase of applications, RTI Real-Time Connect is able to discover and store all of the available typecodes associated with existing Topics. IDL/SQL Data Mapping Once the topic and message format (or type) is known, the relational table can be created in the database (if it doesn t already exist). In order to create the table, the table schema must be decided upon. RTI Real-Time Connect maps the discovered IDL type information to table schema. Fields in IDL types are mapped to columns of a table, and the key fields, if any, form the primary key of the table. Each row in the table represents a different value of the IDL type. Unless RTI Real-Time Connect is configured to keep history data, which will be discussed later, the last value of a instance is stored in the table. So a nonkeyed topic will be stored in a single row whereas for keyed topics, there will be as many rows as there are topic instances. The table below shows how basic IDL types are mapped to SQL data types. Simple IDL structures containing only basic or primitive types directly map to SQL schemas with fields in the structure becoming columns in the table. IDL structures that contain more complex fields, fields that are structures, unions, or sequences and arrays of types other than octet, char or wchar are mapped to SQL tables by flattening the embedded structures so that their fields are all at the top (and only) level. IDL type Char char[n] SQL type CHAR(1) CHAR(n) 5

6 string<n> Boolean Short Int Long long long Float Double Octet octet[n] sequence<octet,n> VARCHAR(n) TINYINT SMALLINT BIGINT REAL DOUBLE BINARY(1)/RAW BINARY(n) VARBINARY(n) RTI Real-Time Connect Runtime configuration Meta-tables and QoS policies Upon startup, RTI Real-Time Connect creates rtidds_subscriptions and rtidds_publications meta-tables in the database. These two relational tables provide the application or user the ability to configure the RTI Real-Time Connect runtime behavior. For the sake of our logging subsystem use case, we ll only focus on the rtidds_subscriptions table. The rtidds_subscriptions table is created empty. The table schema associated with this table is as follows: Table RTI_SUBSCRIPTIONS: *DOMAIN_ID NOT NULL *TOPIC_NAME VARCHAR (200) NOT NULL TABLE_OWNER VARCHAR (128) NOT NULL TABLE_NAME VARCHAR (128) NOT NULL TYPE_NAME VARCHAR (200) TABLE_HISTORY_DEPTH PROCESS_BATCH PROCESS_PERIOD.SEC PROCESS_PERIOD.NSEC COMMIT_TYPE VARCHAR (17) CACHE_MAXIMUM_SIZE CACHE_INITIAL_SIZE IDL_MEMBER_PREFIX_MAX_LENGTH IDL_MEMBER_SUFFIX_MAX_LENGTH SUB.PRESENT.ACCESS_SCOPE VARCHAR (25) SUB.PRESENT.ORDERED_ACCESS TINYINT DR.TIME_FILTER.MIN_SEP.SEC DR.TIME_FILTER.MIN_SEP.NSEC DR.DURABILITY.KIND VARCHAR (30) DR.RELIABILITY.KIND VARCHAR (27) DR.DESTINATION_ORDER.KIND VARCHAR (43) DR.LIVELINESS.LEASE_DUR.SEC DR.LIVELINESS.LEASE_DUR.NSEC DR.DEADLINE.PERIOD.SEC DR.DEADLINE.PERIOD.NSEC DR.HISTORY.KIND VARCHAR (21) DR.HISTORY.DEPTH DR.OWNERSHIP.KIND VARCHAR (23) DR.RES_LIMITS.MAX_SAMPLES DR.RES_LIMITS.MAX_INSTANCES DR.MULTICAST.RECEIVE_ADDRESS VARCHAR (15) 6

7 (primary key columns are indicated with *) Each row in the rtidds_subscriptions table represents a topic in a domain to be logged. The fields of the table expose many of the QoS attributes defined by the standard thus providing the user control over the quality of service when capturing real-time data from the network. More specifically, when a row (corresponding to a specific topic) is inserted into the rtidds_subscriptions meta-table, the RTI Real-Time Connect Daemon creates a DataReader (and Subscriber) according to the provided QoS parameters thus controlling how each unique message is ultimately subscribed to, and logged. As additional records are inserted into the meta table, additional DataReaders are instantiated. By inserting rows, we expand the logging systems ability to capture additional message types. It s not required to specify every column when a row is inserted into the table. In fact, if a row s column is not specified, then a default value will be used. Generally by specifying column-values in the rtidds_subscriptions table the application can configure: Which message or data streams to log; Whether a message stream is captured completely or sub-sampled by time-based filter; How many samples are kept, either all samples (unbounded) or a fixed-length history; Whether samples are committed to the database separately (for better response time) or in batches (for higher throughput); Whether messages are received reliably or best-effort for minimizing the effect on publishers. Once a message stream is being logged, it is possible to reconfigure the logging QoS or to cancel logging the stream by updating or deleting the corresponding row in the rtidds_subscriptions table. For example, to configure RTI Real-Time Connect to log four unique topics (i.e. message streams), the following four SQL statements would be executed on the rtidds_subscriptions table: insert into rtidds_subscriptions values (7, AirCraftPos,'tableowner','AircraftPositionTable','PlanePosType',-1); insert into rtidds_subscriptions (DOMAIN_ID, TOPIC_NAME, TABLE_OWNER, TABLE_NAME, TYPE_NAME, TABLE_HISTORY_DEPTH, DR.TIME_FILTER.MIN_SEP.SEC) values (25, FuseTmp,'tableowner ','FuseTempTable', 'FuseTempType', 20,); insert into rtidds_subscriptions (DOMAIN_ID, TOPIC_NAME, TABLE_OWNER, TABLE_NAME, TYPE_NAME, TABLE_HISTORY_DEPTH, DR.TIME_FILTER.MIN_SEP.NSEC) values (43, Humidity,'tableowner','HumidityTable','HumidType',5, ); insert into rtidds_subscriptions values (43, Pressure,'tableowner ','PressureTable','PressureType',0); As a result, four topics will be logged, each with differing QoS and table history depths. Let s examine each message to be logged: Aircraft Position Message - the topic AirCraftPos of IDL type AircraftPosType will be captured from Domain 7 and stored in a table named AircraftPositionTable. The history column is set to -1 indicating that all samples of the topic will be captured. If the topic has key fields, all instances will be logged in separate rows. Temperature Message - the topic FuseTmp of type FuseTempType will be captured in Domain 25 and stored in a table named FuseTempTable. The history column is set to 20, which indicates that the most recent 20 samples of the topic are kept. In this case the log table acts as a circular buffer. The time based filter QoS policy is set to 1 indicating that only one fuse temperature message per second will be logged. 7

8 Humidity Message - the topic Humidity of type HumidType will be captured from Domain 43 and stored in a table named HumidityTable. The history column is set such that the most recent 5 samples of the topic are kept. The time based filter QoS policy is set such that messages will be logged once per 300ms. Pressure Message - the topic Pressure of type PressureType will be captured in Domain 43 and stored in a table named PressureTable. The history column is set to 0, which indicates that only the most recent sample of the topic is stored. Runtime re-configuration of the Logging Subsystem: Inserting rows into the rtidds_subscriptions meta-table provides the ability to start logging messages in accordance to specified QoS. At some point, it may be desirable to discontinue the capture of a specific message stream. In this case, the row representing that specific message type can simply be deleted from the table. At that point, RTI Real-Time Connect will remove the DataReader, and the message will no longer be logged. The table containing all previously recorded messages continues to be available, for instance for data analysis or simulation/replay. Secondly, if for operational reasons the QoS of the DataReader instantiated by the RTI Real-Time Connect daemon must be altered, one can simply perform an update on the appropriate row within the meta-table, and the existing DataReader will be removed, and a new DataReader (with modified QoS) will be instantiated. This capability can be used, for instance, to dynamically adjust the sample-rate or history depth. For increased flexibility, the rtidds_subscriptions meta-table can be configured and managed either by a dedicated user application or graphical enterprise tools to configure a message logging subsystem with the ability to alter the systems logging functionality during runtime. Performance Considerations: In-memory versus disk-based database Depending on the volume and frequency of messages, RTI Real-Time Connect can be configured to utilize either in-memory, or disk-based relational databases. More specifically, RTI Real Time Connect supports Oracle TimesTen In-Memory Database which, being memory-optimized, delivers exceptionally high performance, enabling it to easily support the high data throughput of real-time applications. Real Time Connect can also be configured to use Oracle Database 10g, for storing large volumes of data. High Performance Logging When high performance logging is required, the TimesTen in-memory database provides the highest transaction rate on commodity hardware because insert and update operations are executed using memoryoptimized data structures. Figure 3 shows the throughput that can be achieved when logging a single message stream into a TimesTen data store. Note that for a single message size of 64 bytes, RTI Real-Time Connect configured with TimesTen can sustain 28,000 messages per second. If RTI Real-Time Connect is configured with Oracle 10g (on commodity hardware), we would achieve a sustainable message capture rate of roughly 8,000 messages per second. 8

9 Message or data rate versus payload 30,000 25,000 20,000 15,000 Oracle Timesten (sustainable rate) Oracle 10g (sustainable rate) TimesTen Cache Connect (burst rate) 10,000 5, ,024 4,096 Payload size (Bytes) Figure 3. RTI Real-Time Connect & Oracle TimesTen Performance (Transactions/Sec). Due to RTI Real-Time Connects scalable design, performance can scale when running on multi-processor or multi-core systems. Figure 4 illustrates RTI Real-Time Connects scalable architecture when logging four message types on a two-processor/dual core system. The results are due to the fact that RTI Real-Time Connect employs a multi-threaded architecture coupled with TimesTen extremely highperformance. Aggregate message or data rate versus payload 90,000 80,000 70,000 60,000 50,000 40,000 30,000 1 topic 2 topics 3 topics 4 topics 20,000 10, ,024 4,096 16,384 Payload size (Bytes) Figure 4. RTI Real-Time Connect & Oracle TimesTen Scalability When Capturing Multiple Messages. 9

10 It s important to note that there are architectural decisions that must be addressed when configuring RTI Real-Time Connect with TimesTen. If your logging system will be running on a 32 bit operating system, you ll be limited to a TimesTen database of roughly 2Gbytes due to the fact that the entire TimesTen database must reside in RAM. In order to support endless logging functionality, the TimesTen tables will need to be persisted to disk, and resized in order to continue to log incoming message samples. This can be achieved by deploying both TimesTen and Oracle 10g with the addition of Cache-Connect to Oracle which allows the seamless flushing of TimesTen tables to Oracle 10g. When TimesTen tables are associated with a Cache-Connect Cache Group, once configured - the Cache Connect daemon will seamlessly flush the designated TimesTen tables to Oracle 10g, and then automatically resize the TimesTen tables in order to allow them to continue to grow with logging. The criteria utilized to determine when to flush the tables, and how the resizing operation occurs is all configurable for each Cache Group managed by the Cache-Connect to Oracle daemon. By leveraging Cache-Connect to Oracle, we can achieve and support endless persistent message logging, with a reasonable amount of system RAM (on a 32 bit operating system). From a performance perspective, it s important to realize that the TimesTen in-memory database serves as a buffer isolating the enterprise infrastructure from potentially overwhelming high-frequency real-time messages. Even though RTI Real-Time Connect with TimesTen can support greater than 80,000 transactions when capturing multiple message streams, this throughput rate is not perpetually sustainable. Oracle TimesTen keeps transactions in an in-memory log buffer before TimesTen Cache Connect propagates these to Oracle 10g. The results for TimesTen Cache Connect shown in Figure 3 were generated using a transaction log buffer size large enough to store all transactions for the duration of a burst. Should the burst length exceed this setting, the data rate would drop to the level as shown by the previous benchmark. By employing TimesTen as an in-memory logging buffer, RTI Real-Time Connect can successfully absorb high message rate spikes without loosing messages. Alternatively, a 64-bit operating system can be deployed with sufficient RAM necessary to support the total volume of messages to be logged. The TimesTen data store can be configured as Persistent, which enables message capture to disk without the use of Oracle 10g. Lastly, as mentioned earlier, if the performance of TimesTen isn t required, then RTI Real-Time Connect can be configured to log directly to Oracle 10g thus allowing virtually unbounded storage of message traffic. In order to increase message throughput when logging to disk, message aggregation or subsampling can be applied if each individual message isn t required to be uniquely logged. Conclusion RTI Real-Time Connect is fully standards-based and is a solution that facilitates the design of a logging capability. From our previously stated design objectives, we have been able to see that RTI Real- Time Connect addresses each of the logging requirements with little or no application specific design or software development. In summary, the RTI Real-Time Connect architecture provides the basis for developing an efficient distributed message auditing subsystem. For more information about RTI Data Distribution Service,, or RTI Real-Time Connect, please visit 10