Management of Very Large Security Event Logs

Transcription

1 Management of Very Large Security Event Logs Balasubramanian Ramaiah Myungsook Klassen Computer Science Department, California Lutheran University 60 West Olsen Rd, Thousand Oaks, CA 91360, USA Abstract - Modern computing systems generate huge amounts of system event log data which are used to address a wide range of important issues such as auditing network status. The traditional method of analyzing event log data is labor intensive and error-prone. In this paper, we will describe our efforts on architecting an integrated log data mining system for automatic management. The system includes a SQL server as a central log server to store historic event log data from other monitored computers. The relational database offers many features desirable for conducting network management: indexes, query optimizer, and powerful query language. We evaluate SQL server hardware performance and query execution speeds with two different sizes of real data. Keywords: network, security, authentication, event log, management 1. INTRODUCTION Data in the log files describe the status of each component and record system operational changes, such as starting and stopping of services, detection of network applications, software configuration changes, and software execution errors. The log analysis problems in general can be broadly divided into the following 3 major tasks: Log data collection and storage in a central repository. Event categorization and event mining : Ability to identify the various log types and categorize them based on type, application, and operating system etc. Reporting and auditing: generate historic reports or perform investigations, audits these logs for compliance and regulatory audits. The time required to do log file data analysis is often high and subject to many errors, due to the fact that traditionally most network management activities have been performed with direct human involvement. As a network becomes increasingly large, those activities become more demanding and data intensive. At large corporations, log files generated by computing systems for different functions and different processes are quite large. When network management is performed with extensive and direct human involvement, it presents some limitations: complexity, difficulty and errors in the operation of network management systems. For these reasons, the automation of network management has become important, as it provides for improved quality of service offered to the customers, while at the same time minimizing potential human errors and reducing network operational costs (Parulkar). The desirable framework is an integrated and automated network maintenance system which incorporates all 3 major tasks mentioned above. Papavassiliou(2000) described methods and tools to provide an efficient and proactive network maintenance process. It was designed using AT&T transaction access service network which is a hybrid POTS-and data wide area network. The system uses an Oracle 7 relational database running on HP K series servers running HP-UX The developed system includes the following tools: (1) the information tracking system that

2 provides a common framework for the storage and retrieval of provisioning, capacity management and maintenance data; (2) the transaction event viewer system that generates, filters, and presents diagnostic events that indicate system occurrences or conditions that may cause a degradation of the service. Tao (2005) proposed an integrated framework to mine system log files for automatic management. The log files were collected from several different machines with different operating systems and Windows NT data was collected using a tool logdump2td developed by Event Mining Team at IBM. The raw data file has 14 columns and text mining techniques were used to categorize messages. One of the important factors for success of an integrated network administration system is a fast response time. Being able to access data and extract necessary information from it at a fast speed to determine if there are any security breaches (Yamanishi, Smyth) or other issues is very important. A robust, fast data analysis system will help companies not only satisfy most audit and regulatory requirements but also boost security and system stability. One way to improve the system response time is to use an efficient data storage and query system for a large log data. Today network management systems use custom storage and retrieval solution to store log file data. They might be enhanced with basic fixed indexes. While they have advanced to quickly process incoming data and generate queries on-the-fly, storing and efficient querying hundreds of gigabytes of historical log file data remain unchanged. While such custom databases can be made to support large flow rates, they are expensive to build, provide a very narrow set of features, and offer limited API(Snyder et al). In contrast, relational databases offer powerful query optimizers, indexes, and a flexible query language. Geambasu examined how and when relational database can be used to store and query historical network flow information in a network intrusion detection system. They reported that Postgres relational database supports relatively large deployments very well with 3 indexes. The purpose of this paper is to present an integrated event log system to automate data gathering, data analysis and report generation using relational database as a log data file storage format. It addresses an issue if a relational database can support a network log data query in a timely manner, especially for a large number of monitored local machines. It also briefly touches the hardware scalability of the proposed system. Operating systems like Linux do not have security auditing functionality and therefore are at a disadvantage in providing data for event queries (Reynolds). Reynolds pointed out that ironically Window NT and its successors provide much of necessary data for security events. So he (2003) used the Windows event log and developed a host-based intrusion detection system. Our system was implemented with the Identity Management system like the authentication systems in Microsoft Window Servers since it provides data required for network activity analysis. But the developed system is easily extensible and pluggable to other operating system platform and to other log formats. In this document, in section 2 we present our methodology including system architecture, data preprocessing, and database design. Section 3 shows query results and hardware performances on this important problem. We continue with query locking issue in Section 4 and conclude in Section 5.

3 2. METHODOLOGY 2.1 System Architecture Figure 1 shows the deployment architecture for the event log system. The system was designed to be vertically and horizontally scalable and therefore was divided into multiple components. A server in a local site logs its event data in its designated local site log collection server. Then it in turn will replicate data to a central log repository using replication at scheduled intervals. Adiscon software (Adiscon, 2007) was used, along with some customized code written in.net platform to collect and send data to a SQL Server, which is the central log server. Adiscon was chosen because of the flexibility of the software: it can send data simultaneously to multiple destinations including syslog server and can write data in a relational database format. The software also provided features for intelligent event notification based on the rules that can be set on the servers. Figure 1: Deployment architecture The companies running servers across multiple sites can take advantage of the compartmentalized model, not worrying about bandwidth consumption by the log data during the peak business hours. The solution will also provide web servers which will host the web front end and custom reports to query the data from the central repository. The system is comprised of a NetApp filer with 3 TB of storage and SQL 2000 Server Database built on a 4way - 8 GB HP DL 585 G1 with 8 GB RAM. More storage can be added as required without changing the central log server as the storage system is decoupled from it. The DL 585 server is capable of handling 8 processors and up to 32 GB of memory.

4 2.2 Data Collection and Preprocessing In the site where the project was implemented, there were about 2000 Window based systems. For prototyping, log data was collected for 1 day from 200 domain name controller servers which generate more data than the rest of machines. One domain controller generates about 25GB per day. Initially 23 attributes, shown in Table 1, were logged by the Adiscon tool from every server into the central log server. The goodness of each attribute was measured from its data distribution for all possible attribute values. If it was the same or has little variation for possible attribute values, it was considered not very useful. Through this process, attributes were reduced from 23 to 10. Retained attributes are ID, ReceivedAt, DevieReportedTime, facility, message, EventSource, EventUser, EventCategory,EventID, and EventLogType. They are shown in Table 1 in bold and Italic. As a result, each server sends only ten attributes to the central log. A lower number of attributes not only reduces the overall storage space requirements, but also improves the performance of SQL query execution. Table 1:Data fields (attributes) captured in Adiscon log data. Attributes Data Function Type ID Unique Identifier of the log entry CustomerID ReceivedAt Datetime The Time when the log entry was received by the loghost service DeviceReportedTime Datetime The time when the event was reported to the OS service. Facility Syslog facility name an integer value Priority FromHost The system name reporting the event name Message Text The actual data in the log entry NTSeverity Importance Reserved for future use EventSource EventUser EventCategory EventID The Event ID number corresponding to the Windows event EventBinaryData Text Event entry binary information MaxAvailable Disk space available CurrUsage Current disk usage MinUsage Minimum disk space used MaxUsage Maximum disk space used InfoUnitID SysLogTag EventLogType GenericFileName Any file name configuration specified. SystemID 2.3 SQL Server Database design There is one main table server which contains 10 attributes selected through preprocessing. This table has a relationship directly or indirectly to 14 auxiliary tables which can provide detail information about the server under question. For instance, its operating system and its service pack installed can be queried. This design stores all dynamic and real time log data from one

5 server in one table, thus allows a fast and efficient query for a particular server. The schema is extensible for different log type application and for different data size. 3. EXPERIMENTAL RESULTS 3.1 Central Log Server Processor & Memory Utilization Processor utilization was evaluated with a 85GB database with 6 simultaneous connections running a query with a single join between 2 tables. As shown in, processors are running at about 24% of the time. The actual production data is expected to be 10 times this sample size and as a result, execution will slow down. Efficient indexing or more than a 4 concurrent processors should be considered as a possible solution. Table 2: Processor Utilization with 85 GB in joining 2 tables Total Processor 0 Processor 1 Processor 2 Processor 3 % Privileged Time % Processor Time % user time Memory Utilization The SQL server showed a 66% Cache hit ratio on a 3 GB RAM system with 85 GB Database. Since system will be scaled up to around 1 to 1.5TB, the memory usage is expected to be around 6-7 GB. 3.2 SQL Query Results When a system administrator investigates the event, he/she issues queries against database. We bench marked the performance of three specific queries against two different size data sets: 2.5GB(40 million records) and 7.5GB(100 million records). Our focus was to investigate effectiveness of the system in terms of system security stability. One particular case is to identify a cause of account lockouts, and following query results show that a source data of the cause in the logs can be easily queried against historical relational database. EventId 642 is Use Account Management Events for the status of Changed. Usamdc01am is a server table and Devicereportedtime is time when the event took place. Three types of queries: 1. Query all records of particular event(s) of interest select * from usamdc01am where EventID='642' 2. Query all records of particular event(s) of interest with a given time range select * from usamdc01am where Devicereportedtime>='12/01/2006' and EventID='642' 3. Query all events with in a time range. select * from usamdc01am where Devicereportedtime >='12/01/2006' Experiments were conducted to measure a system performance with indexes. Initially queries were executed without any indexes and later indexes were created on various attributes and narrowed down to one index of EventID on all log tables.

6 The experimental results in Table 3 demonstrate that indexing the attribute used in the query dramatically improved the performance. In the case of query 3, indexing EventId didn t improve the performance at all and it is expected since the query is on DeviceReportedTime. Table 3:query performances with index EventId for 25GB data and 75GB data. Query type Data Size Time with No EventID Index Time with EventID Index Columns Queried query1 2.5 GB 7.42 minutes 20 Seconds Event ID query2 2.5 GB 6.58 minutes 8 Seconds Event ID + DeviceReportedTime query3 2.5 GB 1 hr 37 minutes 1 hr 37 minutes DeviceReportedTime query GB Not available 39 seconds Event ID query 2 7.5GB Not available 13 seconds Event ID + DeviceReportedTime query GB 5 hr 40 minutes 5 hr 40 minutes DeviceReportedTime Query performances with two different data sizes are consistent: when a data size increases, a total amount of query time increases. For instance, for query1 with indexing, it took 20 seconds with 25 GB while it took 39 seconds with 75GB. 4. QUERY CONNECTION TIMEOUT AND LOCKING The front-end web application written to query data in a single user environment worked well without problems when database is small, and when there are not very many simultaneous insert operations into the SQL database. However when the number of entries in the database grew to be over 10GB and query times increased, queries against a table would create locks, causing problems with an insert operation and other queries running concurrently. The problem became magnified when the number of connections to a table querying for data increased: users started getting timeouts/never ending queries till the SQL server dropped one of the connections to resolve the locks. When the application code was modified to explicitly not use any locking and to allow dirty reads, this issue was solved. The following SQL code snippet show forcing the SQL server not to put any type of locks being executed against a server log data table. Sample query with explicit no lock option: Select message from ustodc01am with (NOLOCK) where Devicereportedtime>= 10/01/2006 and Devicereportedtime<= 10/15/2006 and EventID='644' 5. CONCLUSION We have created an integrated network monitoring system with a web front end application and a commercial relational database as a central log server. Handling large amounts of historical data is not a new problem in databases. However, most work on historical data is data warehousing where data is archived, indexed, and summarized offline with plenty of time to optimize. In our work, handling and querying a large amount of data is on-line, while having to optimize for reading.

7 We have demonstrated the validity of our system using real log file data collected from 20 Windows Identity Management systems. The central log server processor utilization rates and memory cache ratios were monitored for system stability with the prototype 20GB and the suggestion was made for the real production data size of over 1 terabytes. Our system and its methodology can be straightforwardly applied to the analysis of a wide range event log files. In this paper, we have focused on use log lockout queries. It is left for future study how to extend our work in order to find a method to choose the right indexes by examining databases and queries. 6. REFERENCES [1] Adiscon event log support manual. [2] Geambasu, R. and Bragin, T Exploiting History in a network intrusion Detection Ststem. NDSI 2007, NetDB Workshop [3] Papavassiliou, S. and Pace. M From service configuration through performance monitoring to fault detection: implementing an integrated and automated network maintenance platform for enhancing wide area transaction access services. ernational Journal of Network management Vo:10. p [1] Parulkar, G., Schmidt, D., Kraemer, E., Turner, J. and Kantawala, J Architecture for monitoring, visualization, and control of gigabit networks. IEEE Networks. Sept/Oct, Vol:34. [5] Reynolds, J., and Clough, L Continual Repair for Windows using the Event Log. SSR 03, October 31, [2] Smyth. P Markov monitoring with unknown states. IEEE journal on selected areas in Communications (JSAC), special issue on elligent Signal processing for Communications. [7] Snyder, D., and Thayer, R or Fetched on March 23, [3] Tao, L. Liang, S., Sheng, M., and Peng, W An integrated framework on mining logs files for computing system management. KDD 05. p776-p781. [9] Yamanishi, K. and Maruyama.Y Dynamic Syslog Mining for network Failure Monitoring. KDD 05. p