Technical Report about Active Databases & Data Stream Processing

Transcription

1 Technical Report about Active Databases & Data Stream Processing Advanced Topics in Information Systems: Event Stream Processing By Prof. Nesime Tatbul Marco Stöckli ETH Zurich 1. Introduction In this report the Active Databases, which are well known from the traditional relational Databases are compared to the upcoming topic of Data Stream Processing. This report is organized as follows: Section 2 presents the single topics of Active Databases and Data Stream Processing and their main focus is summarized. Section 3 discusses how these two topics are related to each other. Section 4 discusses how each topic is related to event processing.

2 Technical Report about Active Databases & Data Stream Processing Page 2/6 2. Description of topics 2.1. Active Databases Conventional database systems are passive in the sense that they only change their state or respond to users if they receive an external impuls. An active database system extends this behavior by additionally integrate an event-based rule system, which is mostly in the form of an ECA-Rule (Event-Condition-Action- Rule). Applying these ECA-Rules means to wait for the occurrence of the appropriate event, which is a happening of interest. If then the condition is satisfied, the specified action is executed automatically. ECA-Rules can have different purposes like enforcing integrity constraints, gather statistics, compute derived date or monitor a system. In Fig. 1 you see an example of how an ECA- Rule is structured and how it may look like. Fig. 1 Example of ECA-Rule Different types of events are generated through database operations, time signalizations and user-defined events. The condition usually consists of a predicate on the database state. In the action part of the rule, all active database systems allow to use the data manipulation language. Since the incorporation of the rule system is intended to add functionality to the DBMS, the traditional features of database systems should be preserved. Besides primitive events which are pre-defined and whose operations are provided by the Database Manipulation Language (DML) of the underlying database system, there are also complex events. These are used to react to more sophisticated situations. Complex events are constructed from simpler ones by using operators, like disjunction, conjunction and sequence, of an event algebra. In general, complex events are triggered by a number of primitive and/or complex component events, which, in addition, may have to occur in a predefined order. To shortly summarize and look at the major components for the implementation of active databases we look here at two points: - Rule management The rule management component is the heart of an active system. Those parts of the process model that should be handled automatically are implemented by rules within the active database. Rules are specified in the form of the earlier described ECA-Rules.

3 Technical Report about Active Databases & Data Stream Processing Page 3/6 - Event detection This component registers basic events that occur as update request on the database, result from attaining defined time stamps or from elapsing predefined time intervals. It tests whenever such a basic event encounters whether a complex situation described by the rules has occurred. If positive, it triggers the corresponding action. The conditions are normally tested by querying the database. Actions in turn may result in additional events, which again are monitored by this component Data Stream Processing Data Stream Processing is responsible for handling incoming- and outgoing-data streams. Part of the outgoing stream processing is stream generating. They can be built from different data sources like sensors, cameras or even from the traditional relational databases. Part of the incoming stream processing is to make queries on the incoming stream. The stream can be stored in a relational database or even transformed into another stream. But within Data Stream Processing it s not everything about streams. An important part is also to include the existing relational data. This can be done by storing streams into a relational database or create a stream out of a database. Even when you would like to make a stream-to-stream transformation you need to go over the relational part by first transforming the stream into a relation and than back to a stream. To make this possible Data Stream Management Systems (DSMS) are needed and Query-languages which are based on these systems. The most famous Data Stream Management Systems are Aurora, STREAM and PIPES. All these systems support query languages to process streams. With the help of these systems, the queries can be organized and transformed into a query plan. So the task of theses systems is also to optimize these query plans and store the data of streams, relations and intermediate results internally. To query the streams a language is needed. The most known query languages are SQuAl, which is provided by Aurora, TelegraphQL, PIPES and CQL which is based on the STREAM system. CQL in example divides it s operators into three main classes of stream-to-relation, relation-to-stream and relation-to-relation, as it is shown in Fig. 2. Stream-to-relation operators produce a relation from a stream based on the concept of a sliding window over the stream: a window that at any point of time contains a historical snapshot of a finite portion of the stream. Relation-to-stream operators take a relation as input and produces a stream as output with the same schema as the relation. The relation-to-relation operators take one or more relations as input and produce a relation as output. In CQL they are derived from traditional relational queries expressed in SQL.

4 Technical Report about Active Databases & Data Stream Processing Page 4/6 Fig. 2 CQL: Operator classes and mappings used in abstract semantics Most of these systems and languages are quite new and are developed as a research topic at universities. But data stream processing also gains a lot of interest outside the research projects. This is because we have to handle with more and more data which comes in a stream. 3. Active Databases compared to Data Stream Processing Even when the description of these two topics sounds very different, there are also some similarities between them. Active Databases and Data Stream Processing want to simplify and help to keep data in the way we want it. So we don t need to check every few minutes what s going on with the data or manually execute queries on new arrived data. For that to achieve, both topics need some prework. In the Active Databases this can be done with rules, like which conditions need to hold or what data mixture triggers an event over the data. In a Data Stream Processing System it s quite similar. But instead of defining rules, queries are defined and attached to the system. After this prework, both systems will perform the needed action automatically without an external interaction. The Active Database checks constantly the status of the data in the database. Whenever something has changed that affect the rules or conditions, the corresponding adjustment is done. After the queries are registered to the Data Stream Processing System, the systems execute these queries continuously on the incoming data stream. Everything works automatically and nobody needs to look at the incoming data or execute queries. To change or extend the defined rules or queries, this can be done on the fly. In the Active Databases these rules are applied to all the stored data, even when they are older, immediately and the data is updated. With Data Stream Processing it depends on the system whether the old stream data is queried or not. In the STREAM system for example, there is just this stream data stored which is used for the defined queries. So when a new query is made, the system needs to wait for new incoming data elements to query them.

5 Technical Report about Active Databases & Data Stream Processing Page 5/6 The main differences are that the active databases rely on the well known traditional relational databases and the data stream processing relies on data streams. Active Databases and Data Stream Processing also rely both on data and especially on changes over them. For data stream processing it s every new incoming element sort of a change. So before an element is stored in a relational table, sent to a new stream, edited or triggers an event, the data is first evaluated. In the active database it s almost the opposite. Because first the data gets stored in the database and then, depending on the defined rules an event is triggered. 4. Comparison to Event Processing 4.1. Relation of Active Databases to Event Processing Active Databases are the oldest and most well known topic of event stream processing. They also provided the first event specification language. By comparing Active Databases with Event Processing the similarities can be shown by finding where Events occur in Active Databases and how they are triggered and processed. So it is all about events which stand for the occurrence of significance interest. They occur in an active database whenever an introduced rule is satisfied with the existing data. And from there the event triggers the following action, which cans itself also lead to a new event and trigger further executions. That is like in event processing engines, which try to extend the previous systems in the term of event complexity. As Event Processing is about computations on data representations of events and about the language issues versus the system issues, we see another similarity to the Active Databases the need for system and language. Active Databases rely on Database Management Systems (DBMS) which are used to store the date, execute queries and to recognize when an event situation occurs of the active database rules. These rules are written in SQL, which is the standard language for active databases Relation of Data Stream Processing to Event Processing In this part, Data Stream Processing is compared to Event Processing. It is like in the previous section, where Active Databases are compared to Event Processing, that Event Processing is a subpart of it. Because Event Processing occurs also in Data Stream Processing. For that, we look first at events and their processing in Data Stream Processing. How they are defined, triggered and at the end executed. In a Data Stream Processing System we got an event with every new entering element through the incoming DataStream. Because the

6 Technical Report about Active Databases & Data Stream Processing Page 6/6 registered Queries are executed on all the incoming data. All the attached queries are compiled in the system to a query-plan, where the incoming data is stored temporary to make computations on it. In the STREAM system for example, when two queries rely on the same data source, they can be merged together. Like that they can safe temporary memory or disk space and optimize the query execution performance. As event processing engines try to extend the previous systems in terms of event complexity and processing performance, we can also recognize these to aspects in Data Stream Processing. Also in Data Stream Processing a language is needed beside the system. Mostly they are developed together relying on each other like Aurora with SQuAl or STREAM with CQL does.