Introduction to the Event Analysis and Retention Dilemma

Transcription

1 Introduction to the Event Analysis and Retention Dilemma

2 Introduction Companies today are encountering a number of business imperatives that involve storing, managing and analyzing large volumes of event data. These imperatives include: compliance with government regulations; corporate governance; managing growth; and internal and external threats to the security of data, particularly from hard-to-detect advanced persistent threats (APTs). The good news about event data logs is that they can be incredibly valuable repositories of information when, for example, IT staffs are trying to identify the source of unauthorized data transfers or finance teams are trying to meet regulatory reporting requirements. The challenge is that the volume of event data can be truly staggering as much as petabytes, in the case of large corporations or managed service providers. As a result, companies or service providers may not even try to track the source or progress of APTs, a regrettable decision because event logs can provide important clues and evidence. Mining event data for valuable insights is much more difficult than looking for a needle in a haystack. It s more like looking for a grain of sand. It requires highly specialized approaches and tools. This paper will briefly touch on three areas of technology where recent advances by Sensage have improved the ability to glean actionable insights from mountains of event data 1) understanding the unique nature of event data, 2) managing storage, and 3) performing complex correlation across large data sets. Greater detail about all these issues can be found in the white paper The Event Analysis and Retention Dilemma, available at default/files/sensage_event_analysis_dilemma.pdf. Understanding Event Data In order to design systems that provide the greatest benefit, it is important to understand the nature of event data. Among the key characteristics of event data are the following: Non-transactional: Once event data is stored, it will never be updated. In fact, for compliance purposes, altering and deleting event data should be strictly prohibited. Time-based: Event data is a collection of data about a particular event, at a specific point in time. This means that every event will have a time stamp associated with it. Repetitiveness: Event data is generally highly repetitive (e.g., company employees logging on and off), so finding anomalies involves querying high volumes of repetitive events. Near real-time: Event data is created in real time and must be loaded at least as fast as it is created. Although all event data does not need to be available in real time, load rate must keep pace with creation rate in the long run. Managing Storage Relational Database Management Systems (RDBMS) have become the de facto standard for almost all data management problems. So, it is not surprising that when Security Information and Event Management (SIEM) vendors discovered the need to manage non-real time event data beyond real-time requirements, they used the relational model to store and analyze it. Initially, RDBMS technology was adequate, but as the demand to manage greater volumes of security-related event data emerged, the limitations of RDBMSs became apparent. For a detailed comparison of event data and RDBMS technology, see the white paper referenced earlier. But the quick summary is: event data and RDBMS technology are, at best, poorly matched. The mismatches can be roughly categorized as either a) RDBMS overhead not required to manage event data or b) lack of technology to support the unique requirements of event data management. 2

3 3 Performing Complex Analysis Querying on event data can be precise or patternoriented. Most log data is unstructured, however, and must be analyzed using some form of pattern matching. For that reason, event data storage must not be organized to optimize precision searches at the expense of sub-optimizing pattern-based queries. The unstructured nature of event data is resistant to query optimization via the creation of indices. The specifics of event data analysis reflect an organization s evolving security and systems management landscape. New queries must be created to detect newly discovered security threats or to monitor new system components. This can happen on a daily or weekly basis. Because of the unpredictable nature of rapidly evolving threat detection, event storage systems must be flexible enough to address analytic requirements today and in the future. Mitigating RDBMS Barriers to Event Data Management Faced with the limitations of RDBMS-based solutions for event data management, SIEM vendors and their customers have adopted a number of strategies in a valiant and costly effort to deal with the inherent failings of RDBMS technology. Regrettably, each strategy is either insufficient, risky or both. Among these flawed strategies are the following: Data Filtering: To reduce storage requirements, filters are created to reduce both the number of events and the amount of data stored for each event. But filtering pre-supposes that the nature of queries needed in the future is known in advance, which is not always the case. Additionally, government compliance requires that all original event data be available. Limited Time Range Queries: Typical security analytics are artificially limited in time scope based on storage capacity. While successful threat detection requires analysis of data over longer time horizons, most SIEM solutions can only keep days of data on line and the rest off line. This is due to technical limitations of indexed based systems, which experience performance degradation over time, getting slower and slower as more data is added to their index. This approach ignores business imperatives that require longer-term querying capability. Limited Component Monitoring: Storage requirements can be curtailed by reducing the number of system components being monitored. But like data filtering, this tactic pre-supposes the nature of future event analysis. Discovery of new security and system management scenarios may expose the need to have captured and stored event data from non-monitored components. Two-tier Storage Architecture: To alleviate the high cost of RDBMS storage, aged events can be removed from the database and archived into lower-cost compressed storage. Should events from the archive be needed, they must be uncompressed and restored to the database. But removal and restoration of event data are time-consuming operations that often require database and system administration resources. A two-tier strategy is not a substitute for adequate on-line event data storage. Reducing storage requirements is a worthwhile goal. No one wants to spend more on storage than necessary. But tactics like filtering and reduced time ranges can compromise the integrity and effectiveness of efforts to combat APTs. Because APTs are long-term subversive efforts, the key to discovering and mitigating them is taking a long-range, broad-based approach to collecting, managing and analyzing event data. RDBMS techniques are just not suited for this task. Sensage: Pioneering A New Approach Sensage delivers a high-performance, scalable means for organizations to centrally aggregate, costeffectively store, dynamically monitor and efficiently analyze massive volumes of security-related event log data over long periods of time while retaining the original source data. The Sensage approach eliminates the unnecessary overhead imposed by standard RDBMS technology, and materially increases the performance and capacity to manage massively large volumes of security event data.

4 Sensage s core technology is a combination of: Server clustering (MPP architecture) Data compression A non-transactional model, and Seamless access to online and archived data in a single query Server Clustering Sensage leverages clustered server architecture to distribute workloads and achieve parallel computing on a massive scale. Key elements of this approach include: Near Real-time Loading: Event data is created in real time and must be loaded as fast as it is created. To address this requirement, Sensage has a trickle-feed load feature, loading and making data available for querying near real-time. This is done through special data structures that capture the near real-time data and make it available for querying before it is merged into the actual columnar data store. Distributed Search: Query requests are evenly distributed across servers. Each server conducts its portion of a search in parallel with others. The final results from each server are aggregated and returned to the user. Data Redundancy: Every event is recorded twice in the Sensage server cluster. Each copy is stored on a separate server. Should a server fail, the server that holds the copy of the failed server s event data automatically takes over all query operations for the failed server. Unlimited Scalability: Many massively parallel processing databases can only scale to a single cluster. Sensage has overcome this limitation by distributing data evenly across multiple clusters and returning results from a single query that spans multiple clusters. Sensage users can deploy federated deployments (multiple clusters) as needed and access the data across multiple clusters without compromising on load and query speed. Data Compression Sensage s storage organization is specific to the unique nature of event data, and produces significant advantages when managing that data. Key features in this regard include the following: Time-based organization: The data on each server is partitioned in time ranges. This creates advantages at load time and querying time. For load time, since event data generally has increasing time stamps, the likelihood of combining new load data with data already loaded is small. This dramatically reduces data reorganization needs. For queries with time constraints, the Sensage query engine quickly eliminates the need for scanning data that does not meet the time constraints. Column-based Compression: Data is placed into columnar storage and then compressed when written to disk. High compression ratios are achieved because of the repetitive nature of event data within a column. Compared to the volume of data stored in an RDBMS database, Sensage achieves up to a 40:1 compression ratio, depending on the complexity of the source. No Indices: Because of the unstructured nature of event data, indices render little value. Sensage delivers dramatic improvements in query response time through distributed parallel querying, and event-specific data organization. Unlike an RDBMS, the Sensage architecture requires no indices, meaning there is no need for a database administrative resource to create and drop indices to balance between query and load performance. There is also no overhead of index maintenance during loading. This means the event data load rate will remain constant, no matter how much data has already been loaded. 4

5 Non-transactional Model Sensage delivers unparalleled performance versus RDBMS-based SIM products, largely because of its non-transactional model that minimizes overhead and optimizes the use of computing resources. Key features of this approach include the following: No Concurrency and Locking Overhead: Because event data is never updated, the Sensage solution has no RDBMS overhead of row and table locking. Queries never need to wait for updates. No Transaction Log: Because the commit/ rollback model used in RDBMSs is not meaningful for event data, the Sensage solution avoids CPU, I/O and storage capacity overhead required to maintain a transaction log. Access to Data via Any 3rd Party Business Intelligence Tool Sensage provides the first and only SIEM solution that supports an open access interface to event data using database connectivity (ODBC/JDBC) APIs. These APIs enable any third-party Business Intelligence tools to easily integrate with the Sensage SIEM data warehouse solution. The Sensage solution provides significant benefits to customers requiring advanced event data management. Opening Sensage s security data warehouse to established BI tools enables faster, better, deeper analysis, enabling customers to extend the investment and knowledge they have in their BI tools to gain additional insight and knowledge about their security environment and broader IT infrastructure. The Sensage Advantage The Sensage solution provides significant benefits to customers needing advanced event data management. High Performance Queries: Execute queries in minutes or hours, where RDBMS searches often take hours or days. High-Volume Loading: Data loading keeps pace with enterprise-wide event collection for gigabit class networks, with no degradation based on the volume of data stored. High-Volume, Low-Cost Storage: Sensage uses low-cost Linux-based physical or virtual servers to store highly compressed data. No expensive RDBMS licenses are required. Servers are more efficiently utilized due to the elimination of RDBMS overhead. Sensage Storage Efficiency 68.8 TB (variable bloat amount) RDBMS s require up to 86 TB 2-10X indexing boat 8.6 TB 8.6 TB 17.2 TB (min bloat) For the same 8.6 TB* of raw logs 10X compression.86 TB 2X redundancy 1.7 TB Sensage requires only 1.7 TB *40 million recordes per day, at 300 bytes per record, produces 12GB per day or 8.6 TB for a two year storage period. 5

6 Enhanced Data Security: The Sensage architecture is uniquely suited to the specific demands of discovering, tracking and mitigating high-risk intrusions such as advanced persistent threats. Low Cost of Ownership: This solution requires no database administration resources. Data organization is simple and self-tuning. (Average cost of DBA?) Incremental Scalability: Additional servers can be scaled incrementally to provide increased capacity and throughput to match business growth. High Availability: Built-in redundancy allows continued operation even with a server failure. Data Protection: Event data is protected against any modifications by outside sources. Data redundancy protects against loss of data in the event of component failure. Conclusion Sensage delivers a high-performance, scalable solution for organizations to centrally aggregate, costeffectively store, dynamically monitor and efficiently analyze massive volumes of events over long periods of time, while retaining the complete original source data. This empowers organizations to respond to business threats, conduct thorough investigations, perform complex correlations on large data sets, and fortify broad audit compliance processes. Sensage, Inc Campus Drive Suite 150 San Mateo CA Copyright 2013 KEYW. All rights reserved. Sensage is a trademark of KEYW Corp. in the United States. All trademarks, logos, and service marks (collectively Trademarks ) are the property of KEYW Corp or other parties. Gryphon may not be used in connection with any product or service that is not a KEYW Corp product or in any manner that is likely to either cause confusion among customers, or in any manner that disparages or discredits KEYW Corp. CJ0115_0513