EventTracker Architecture Handling Millions of Events Each Day

Transcription

1 The importance of consolidation, correlation, and detection Enterprise Security Series White Paper 8815 Centre Park Drive Publication Date: August 14, 2008 Columbia MD

2 Abstract The purpose of this paper is to describe at a high level the EventTracker technical architecture and how it enables users to easily process, store and gain actionable intelligence from the millions of events that the servers, workstations, applications and network devices in an organization s IT infrastructure generate each day. EventTracker features a highly flexible, component-based architecture that enables distributed processing and configurable collection and storage methodologies. This paper also introduces some of the more common implementation setups for the product. Event data contains a wealth of valuable information for IT controls and compliance, and in many cases, company directives require event information be kept for multiple years. Collecting and storing event logs offers significant challenges however. Each device type has unique events and event logs are voluminous. A single Windows server can generate over 100,000 events per day. When the auditing feature is in use, Windows servers, like UNIX systems, firewalls and Solaris BSM can generate over a million events per day. As a result even a relatively modest-sized organization can easily generate well over 20 million events each day. EventTracker was designed to automate the efficient collection, storage and analysis of these events. The information contained in this document represents the current view of Prism Microsystems Inc. on the issues discussed as of the date of publication. Because Prism Microsystems must respond to changing market conditions, it should not be interpreted to be a commitment on the part of Prism Microsystems, and Prism Microsystems cannot guarantee the accuracy of any information presented after the date of publication. This document is for informational purposes only. Prism Microsystems MAKES NO WARRANTIES, EXPRESS OR IMPLIED, AS TO THE INFORMATION IN THIS DOCUMENT. Complying with all applicable copyright laws is the responsibility of the user. Without limiting the rights under copyright, this paper may be freely distributed without permission from Prism, as long as its content is unaltered, nothing is added to the content and credit to Prism is provided. Prism Microsystems may have patents, patent applications, trademarks, copyrights, or other intellectual property rights covering subject matter in this document. Except as expressly provided in any written license agreement from Prism Microsystems, the furnishing of this document does not give you any license to these patents, trademarks, copyrights, or other intellectual property. The example companies, organizations, products, people and events depicted herein are fictitious. No association with any real company, organization, product, person or event is intended or should be inferred Prism Microsystems Incorporated. All rights reserved. The names of actual companies and products mentioned herein may be the trademarks of their respective owners. Prism Microsystems 2

3 Introduction This White Paper provides an overview of the EventTracker architecture and introduces eight fairly representative implementations of EventTracker. EventTracker is built utilizing a highly flexible and scalable component architecture that enables support for a wide range of deployments The example deployments range from a relatively simple, straight-forward deployment to a variety of larger, more complex ones. Deployment requirements have two principle drivers that dictate the implementation: the size and topology of the customer s physical IT infrastructure and the way an organization wants to access and leverage the collected data the management, viewing and reporting on the collected data. In the case of IT Infrastructure, questions such as how many and what type of devices, what audit levels are required, physical locations and event volumes drive the deployment of the EventTracker Consoles and Collection Points that receive and process the data. The second organizational driver also affects the number of EventTracker Consoles and Collection Points, but also drives questions such as who, and how many, are going to use the data, what are the real-time correlation requirements and what data needs to be retained. These define the retention schemes, where events are displayed, deployment of correlation engines and the reporting capability. All IT infrastructures and organizations vary, and the EventTracker architecture is flexible enough to handle any enterprise topology. With over 650 implementations to date, EventTracker personnel have wide experience in advising on the right deployment strategy. In this paper we focus on these eight as our experience has taught us that most deployments are some variation or combination of one of the nine. Prism Microsystems 3

4 The Log Management Challenge The term event log management sometimes seems like an oxymoron. When you add the sheer quantity of arcane event data generated by network systems and IT infrastructures to the requirement to meet complex compliance regulations and then add the pressing mandate to guarantee information security in an increasingly dangerous cyber-world, the ability to successfully manage events logs seems a distant and, often, a very lofty goal. Insuring IT compliance and enforcing security policies is no longer optional for companies today. Windows, UNIX, network devices and database systems, as well as critical applications record a substantial number of security and error events into local logs. At a bare minimum these logs should be collected and archived to meet compliance. Many companies are still undertaking this task manually and the collection of log data from even as few as 10 systems is time consuming and tedious. Further, the reality is that in many businesses the number of devices that generate event logs that need to be collected and archived is often in the hundreds or thousands. These logs contain valuable information that, if accessible, can detect serious system problems and security violations before they impact users. It is a challenge to view event logs one system at a time and make sense of them. Message formats vary widely from system to system, and many of the conditions that indicate potential problems can only be detected when events are correlated or associated with events happening on other systems and devices. Overall, the process of reviewing event logs is so expensive, inefficient and time consuming that many companies do so only after something has gone wrong, despite the fact the information that could have enabled them to prevent the problem to begin with was usually there well in advance. Even storing event log data is a challenge. Event logs are extremely voluminous. Normally, a single Windows server can generate over 100,000 events every day without using the auditing feature. With the audit feature in operation, Windows servers, like many UNIX systems, SNMP devices and firewalls, can produce over one million events per day. Domain Controllers are extremely chatty as well, and it is not unusual for even a small organization to generate well over 20 million events every day. This information needs to be securely archived for IT controls and compliance. Many companies directives, as well as regulations like Sarbanes/Oxley and HIPAA, require event log information be kept for years. One hundred Windows servers with an average number of 100,000 events each means a total of 10 million events per day and that is without auditing! If these events are kept for 90 days, it is necessary to manage and store 900 million events. Kept for three years, the archive would contain over 10 billion separate logs. Even with the logs collected, the problem of analysis is difficult. There is no such thing as a standard log entry and every vendor provides different information in their logs. The Prism Microsystems online Knowledge Base (kb.prismmicrosys.com) contains detailed information for more than 20,000 events, which is still a subset of the total events that devices generate today. Relying on human expert knowledge is impossible no expert can know but a very small subset, so knowledge of what all those logs mean is critical to success. Finally security and compliance auditors not only want to see reports proving that the data is securely retained, but that it is also examined and that security policies are operational. It is no wonder that even IT managers and administrators who grasp the importance of the event log data still find the entire task of event log management a difficult challenge. An efficient automated Prism Microsystems 4

5 collection and archival method is absolutely critical. Powerful reporting and analytics capabilities make the data easy to access and built-in knowledge of the logs is mandatory to make the data meaningful. All are necessary to make event log management a reality. IT Professionals must ask themselves the following questions: - What is the easiest way to automate the collection of events? - How can I store all that data securely and efficiently so it is still accessible? - How can I gain actionable intelligence from all that data in real-time? - How do I generate reports out of consolidated data? - Can the solution handle my unique requirements without expensive customization? - How long will it take me to get a solution up and running, and what are my ongoing costs? Prism Microsystems 5

6 EventTracker Overview EventTracker automates the secure collection and consolidation of all enterprise events to a central point and makes them readily available to IT personnel for analysis. The EventTracker architecture is designed with scalability in mind and is highly configurable while still being easy to install and quick to implement. EventTracker features an extremely efficient, secure, tamper-proof event archive for reporting and compliance requirements, a powerful real-time correlation engine that operates on the event stream, and a Windows or web-based reporting and analytics engine for ad-hoc and scheduled querying. EventTracker Agents To initiate the collection process, EventTracker provides optional EventTracker Agents for Windows, Solaris BSM and legacy operating systems such as IBM z/os and the iseries. EventTracker Agents on Windows go well beyond simple Windows log monitoring with the capability to monitor, for example, system thresholds such as CPU, disk usage and memory, the introduction of memory devices such as Flash drives and even monitoring and logging of files copied to the device. The EventTracker Agents are centrally configured, managed and distributed from the EventTracker Management Console. The Agents then monitor the event log and process and forward events as they occur to up to five EventTracker Consoles or Collection Points. In addition, EventTracker can monitor logs from sources Prism Microsystems 6

7 such as applications like IIS and Exchange or databases like Oracle and SQL Server. EventTracker agents can perform sophisticated filtering of the event logs prior to transmission to the central collection point, so if reduction of the event stream is possible, it can be easily accomplished. In Agent-less mode, data is simply periodically collected from the host systems and brought to the EventTracker Console for processing. For Linux, UNIX and network devices the EventTracker Console is also able to receive and process Syslog, Syslog NG and SNMP V1/V2. EventTracker Console And The Virtual Collection Point Architecture Although EventTracker supports multiple, distributed Consoles for scalability, a single Console instance can process in excess of 300,000 events (steady state) per minute using the concept of Virtual Collection Points. A Console is hosted on a Microsoft Windows platform and can be configured to contain multiple Virtual Collection Points. Each Virtual Collection Point is a complete virtualized event processing stack and consists of a Receiver component that processes the incoming event stream, a Policy Engine that routes the events for further processing if required and an Archiver that writes the events into EventVault. Using multiple VCP s EventTracker can fully take advantage of multi-cpu, multi-core and 64-bit operating systems. The VCP also enables grouping of events in EventVault for more efficient and faster reporting. Each Console also includes a UI for administration, configuration and event viewing, reporting and analysis; and the EventVault event archiver. Optional components of EventTracker consist of a real-time Correlation Engine, the Change Management Module and Event Log Central, a role-based, secure web interface. All of these optional components can be deployed on a single machine with a Console instance, or over multiple machines to maximize performance. Each EventTracker Console can also forward events in real time to other EventTracker Consoles allowing a hierarchical management structure for larger corporations. EventVault EventTracker uses a proprietary event storage mechanism called EventVault to archive the original log in a compressed and secured event warehouse for reporting and compliance purposes. EventVault is optimized for the write-once/read many times nature of event log information. In EventVault log data is compressed to less than 10% of the original size, sealed with a SHA-1 checksum and stored in CAB files. If 100 million events are archived, a traditional database can grow to 400 GB while EventVault would require just 10 GB. When a report is generated, EventTracker automatically selects the required archived data, decompresses and unseals it, and then generates the necessary report. Despite the decompression step, reports via EventVault are still generated faster than using a standard RDBMS, and sophisticated caching of the event data, once opened, enables subsequent report generation to be very fast. The EventVault archives can be stored on any storage device that can be accessed from the EventTracker Console. With millions of events generated daily, a database can be an expensive and slow medium for archiving data. One million events can easily consume over 5 GB of storage, and storing even a small time period of event data can require a huge database, a big database server machine and additional expensive database licenses. Databases are also not guaranteed secured storage and event log data can be tampered with. Some organizations, however, still prefer to archive collected events in traditional databases and as a result EventTracker optionally supports SQL Server, Oracle and Microsoft Access for storing events. The database can be installed on the same server or a separate dedicated database server. EventTracker Correlation Engine An EventTracker Correlation Engine can be configured to correlate events coming from multiple EventTracker Virtual Collection Points or Consoles. The Correlation Engine enables powerful realtime monitoring and rules-based alerting on the event stream. Rules can watch for multiple, seemingly minor unrelated events occurring on multiple systems across time that together represent clear Prism Microsystems 7

8 indications of an impending system problem or security breach. Detecting these problems in real-time prevents or minimizes costly impact on the business. IT staff can be notified of triggered alerts through the EventTracker Console or Event Log Central; or, alternately, an notification, SNMP trap, or pager alert can be generated. With the EventTracker correlation engine the entire contents of the event can be examined. EventTracker comes packaged with over 500 predefined rules of the most common conditions. The combination of Rule wizards and a simple Rule grammar enables the creation of custom rules. Change Monitoring EventTracker provides complete change monitoring capability on Windows Servers and Workstations. WhatChanged, the Change Management component of EventTracker, periodically takes a snapshot of a systems state and does a comparison against either a golden master configuration or simply a previous retained snapshot to detect drift over time. WhatChanged provides a powerful browser capability that enables a user to analyze those hard to find changes that occur on the Windows file system and registry. All changes detected are logged into EventVault for subsequent reporting and analysis. EventTracker Reporting and Analytics Engine EventTracker contains a powerful report generator for custom ad-hoc and scheduled reporting on the data. Reports can be generated in Html, Microsoft Word or PDF formats. The product also comes with over 1000 predefined report templates that enable a business to quickly comply with the regulatory standards applicable to them. The Analytics Engine allows sophisticated custom searching of the event archives with powerful search within search and customizable output formats. Event Log Central Event Log Central is EventTracker s secure web-based user interface that provides EventTracker s Reporting and Analytics capability in a web UI. Event Log Central comes with multiple pre-defined roles such as Help Desk, System Administrator or IT Manager, and custom roles can also be created by the Administrator. User authentication is integrated with Active Directory for single sign-on support and https is used as a secure transport between browser client and server. Reports can be configured through a reporting wizard on either the Windows UI or Event Log Central. Event Log Central also enables users to schedule reports that are regularly generated on the off-hours and distributed to subscriber lists, or published to users in Event Log Central. Collection Points The Collection Point model is designed for larger organizations that have multiple sites or are organized into multiple units within the same site. In many cases, the event log data must all be consolidated and archived in a single place for compliance purposes, but the real-time correlation and day to day management can be the responsibility of different, distinct IT groups. In these instances, real-time roll-up of the events is unnecessary, and the Collection Point model allows an organization to collect and stage event logs in EventVault archives at a location or business unit level, and then automatically transmit these compressed and secure archives to a central enterprise-wide report server on a periodic basis. The business units can access either their local archives for analysis or access the enterprise store. In addition, each Collection Point can be configured to range from a simple storage mechanism to a fully functional EventTracker Console. In large organizations, Collection Points are extremely useful in making analysis of the event data quicker by enabling events to be segmented and collected by type. A fairly common use case is to setup a collection point for all Domain Controllers so that security analysts can quickly run queries about user logon/logoff activities without incurring the overhead of querying all the enterprise event data. Another common use case is to have the networking group with a collection point for network data, and the Windows group to have a Collection Point for system events. These Collection Points are Prism Microsystems 8

9 generally unique views of data, but as EventTracker agents can send events to multiple Consoles they can also be redundant views of the data as well. An added advantage with the Collection Point architecture is that the EventVault data is transmitted via TCP, and delivery is guaranteed. The Event Data is also encrypted prior to transmission and the combination of the two enables a company with multiple locales to use the internet for transmission without resorting to VPN tunneling. EventTracker Knowledgebase In order that EventTracker can support the thousands of event types, Prism has developed the EventTracker Knowledgebase which is updated constantly as new events are defined. The Knowledgebase is hosted by Prism Microsystems and provides detailed descriptions of event meanings. These definitions can be used to configure rules or as a convenient look-up for unknown event types. In the case where a new event is not already cataloged in the Knowledgebase, or if the event is a custom type (for example, an event from a custom application), rules can still be configured easily by the user. Prism Microsystems 9

10 Sample Deployment Models The following pages illustrate a number of the more common configurations of EventTracker. The deployment model design is driven by 2 main factors: the volume and type of event data, and the business requirements of the personnel that need to analyze the data. With flexible component architecture and the potential of each EventTracker Agent, Console or Collection Point to communicate with multiple other Consoles and Collection Point Masters, the potential configurations are nearly limitless. These configurations are illustrative of organizations from fairly modest in size with a single console to large multi-location enterprises with multiple IT groups needing access to the event log data. Prism Microsystems 10

11 1. Basic Installation For smaller organizations that have a relatively modest amount of systems to monitor and have compliance requirements that only call for the generation and forwarding of basic reports, a basic installation is all that is generally required. In this case, all events are forwarded to a single machine running Windows Server 2003 or 2008 and a single EventTracker Console. In higher volume installations multiple Virtual Collection Points can be configured. EventVault and the Correlation Engine all run on the same machine. A single individual in these cases is often the only frequent user and it is unnecessary to install the web interface to enable multi-user capability. The native Windows UI is used to view and manage events and configure and schedule reports. One advantage of the EventTracker architecture is that it is able to collect events in real-time from Windows systems that reside outside the domain with the deployment of the EventTracker Agent. This basic implementation is generally very easy to set-up and is deployed in a matter of hours. It is well suited for installations of up to several hundred servers, and can be hosted on a relatively modest workstation-class machine. Prism Microsystems 11

12 2. Basic Installation with multi-user This is the same basic installation as in Example 1, but with a number of different users wanting to view the collected data. Here the web-based user interface Event Log Central has been deployed. Event Log Central is fully integrated with LDAP for secure access and the browser connection can be over https so the information is securely transmitted. All processing is still done on a single server. Generally in this case a dual CPU machine with 2GB of RAM is recommended. Prism Microsystems 12

13 3. Multiple Departmental Consoles In this example, management of the IT infrastructure is decentralized with different organizations having responsibility for the admin functions on local machines. Corporate Compliance is handled in a single location, and there is a centralized IT departmental as well. This example has each managed device reporting to 2 EventTracker Consoles, a local windows-based Console for the local IT staff, and also a centralized Console with Event Log Central installed for enterprise wide analysis and reporting. Event Log Central can also be installed at the departmental or site level as well. Prism Microsystems 13

14 4. Large Enterprise This example is a variation of example 3 and is typically found at a larger enterprise that has IT management divided by device type. Here there is a distinct UNIX administration functions, so all UNIX machines report to a single EventTracker Console, as well as distinct Windows and network administration Consoles. Compliance reporting is still necessary at the enterprise level. In this example there is no need for real-time event correlation at the enterprise level so this deployment strategy introduces the concept of Collection Points. Events are rolled up by admin group, then stored and eventually consolidated to a central reporting instance. Prism Microsystems 14

15 5. Hierarchical Roll-up This example is typically found in larger businesses with multiple locations or departments. Primary management of the infrastructure and compliance reporting is at the local level. There is a corporate EventTracker Console that receives correlated events from the local Console instances. Prism Microsystems 15

16 6. Events Managed by Function In this scenario, events are processed at a central Console then forwarded to Consoles for distinct job functions. Database events, for example, are sent to a console used by the database group. Alerts are also forwarded as SNMP traps to the central enterprise console. Prism Microsystems 16

17 7. Multiple Large Teams This example is a variation of the large enterprise illustrated in example 4. In this case there are multiple groups that want their own dedicated reporting capability but do not need all the same data. They each have a Collection Point master that is collecting from some of the same collection points and also some unique ones. Each Collection Point is similar to the EventTracker Agent as it can transmit to as many as 5 Collection Point Masters. Prism Microsystems 17

18 8. Active Directory OU Implementation In this example, there is a single Active Directory forest that supports a large quantity of users that are all members of local Organizational Units (OUs). In this case due to security requirements the local OU managers cannot see any activities of users that are not part of their OU. Here an EventTracker Console is collecting events from multiple Domain Controllers (DCs). The Console then does an LDAP lookup, ascertains which OU owns the event and forwards the event to the local OU EventTracker Console. The OU EventTracker Console is also collecting system events from the local machines and consequently has complete visibility into all activity happening within the OU. Prism Microsystems 18

19 Summary EventTracker represents an investment of over 100 man years of development and is the most advanced scalable and flexible event log management solution available on the market. Sophisticated Agents, flexible event routing and event collection enables EventTracker to successfully meet the requirements of customers ranging in size from 50 to thousands of managed devices. With EventTracker manual collection and slow and tedious analysis of individual event logs are things of the past. With the component architecture, event processing can be split over multiple machines for the highest degree of scalability, and customers can define consolidation, roll-ups and management views based around their business structure and requirements. Once collected, the events are securely and efficiently stored to ensure complete regulatory compliance while still remaining available on-line for sophisticated analysis using the EventTracker Reporting and Analytics Engine. Real-time event correlation represents a powerful real-time tool to prevent system failures and security breaches and integrated change monitoring provides capability unmatched in any other Log Management or Security Information and Event Management solution. With EventTracker you can meet compliance requirements with ease, improve information security and improve service levels by reducing infrastructure downtime. Studies by our customers show that using EventTracker saves $100 per server per month in maintenance costs, and EventTracker returns positive ROI in a matter of months. Prism Microsystems 19

20 About Prism Microsystems Prism Microsystems, Inc. delivers business-critical solutions to consolidate, correlate and detect changes that could impact the performance, availability and security of your IT infrastructure. With a proven history of innovation and leadership, Prism provides easy-to-deploy products and solutions for integrated Security Management, Change Management and Intrusion Detection. EventTracker, Prism s market leading enterprise log management solution, enables commercial enterprises, educational institutions and government organizations to increase the security of their environments and reduce risk to their enterprise. Customers span multiple sectors including financial, communications, scientific, healthcare, banking and consulting. Prism Microsystems was formed in 1999 and is a privately held corporation with corporate headquarters in the Baltimore-Washington high tech corridor. Research and development facilities are located in both Maryland and India. For additional information, please visit Prism Microsystems 20