Development of a System Log Analyzer

Development of a System Log Analyzer A Thesis submitted in partial fulfillment of the requirements for the degree of Master of Computer Application Department of Computer Science and Engineering Jadavpur University, Kolkata By Sushobhan Chakrabarti Examination Roll No. MCA-3213014 University Registration No. 113600 of 2010-13 Under the guidance of Prof. Chandan Mazumdar Department of Computer Science and Engineering Faculty of Engineering and Technology Jadavpur University, Kolkata May, 2013

Department of Computer Science and Engineering, Faculty of Engineering and Technology, Jadavpur University, Kolkata 700032 Certificate of Approval This is to certify that the thesis entitled Development of a System Log Analyzer is a bona-fide record of work carried out by Sushobhan Chakrabarti in partial fulfillment of the requirements for the award of the degree of Master of Computer Application, in the Department of Computer Science and Engineering, Jadavpur University during thr period December 2012 to May 2013. It is understood that by this approval, the undersigned do not necessarily endorse or approve any statement made, opinion expressed or conclusion drawn therein, but approve the thesis only for the purpose for which it is submitted. Examiners: (Signature of the Examiner) (Signature of the Supervisor)

To whom it may concern This is to certify that the work in this thesis entitled Development of a System Log Analyzer has been satisfactorily completed by Sushobhan Chakrabarti. It is a bona-fide piece of work carried out under my supervision and guidance for partial fulfillment of the requirements for the awarding of the Master of Computer Application degree of the Department of Computer Science & Engineering, Faculty of Engineering & Technology, Jadavpur University, during the academic year 2012-13. Prof. Chandan Mazumdar Department of Computer Science and Engineering Jadavpur University Countersigned: Prof. Sivaji Bandyopadhyay Head of the Department of Computer Science & Engineering Jadavpur University

Declaration of Originality and Compliance of Academic Ethics I hereby declare that this thesis contains literature survey and original research work by the undersigned candidate, as part of my Master of Computer Application studies. All information in this document has been obtained and presented in accordance with academic rules and ethical conduct. I also declare that, as required by these rules and conduct, I have fully cited and referenced all material and results that are not original to this work. Name : Roll Number : Thesis Title : Signature with date:

Acknowledgement With my sincere respect and gratitude, I would like to thank my thesis guide Prof. Chandan Mazumdar for his continuous support for this thesis work, for his patience, motivation and enthusiasm. His guidance helped me a lot throughout the duration of the work. His valuable suggestions inspired me a lot. I feel deeply honored that I got the opportunity to work under his guidance. I would also wish to thank Prof. Sivaji Bandyopadhyay, Head of the Department of Computer Science & Engineering, Jadavpur University for providing me all the facilities and for his support to the activities of this project. I would like to thank Dr. Anirban Sengupta, Principal Research Engineer, CDCJU, Souvik Sarkar, Research Engineer, CDCJU and Chiranjeet Dutta, Research Engineer, CDCJU for sharing their knowledge and experience with me and also their immense support and co-operation. Last, but not the least, I would like to thank all my family members and my class mates of Master of Computer Application batch of 2010-2013, for their co-operation and support. Their wealth of experience has been a source of strength for me throughout the duration of my work. Sushobhan Chakrabarti Exam Roll No: MCA-3213014 Registration No: 113600 of 2010-13 Master of Computer Application

Contents 1 Introduction... 9 1.1 Problem Area... 9 1.2 Motivation... 9 1.3 Contribution... 10 1.4 Outline... 11 2 Study of System Logs... 12 2.1 System logs... 12 2.2 Auditing and Logging for Windows Hosts... 12 2.3 Event Logs... 19 2.4 Event Viewer... 20 2.5 Configuring logging parameters... 22 2.6 Auditing and Logging for Linux Hosts... 24 2.7 Utility Of Log Management... 29 3 Design and Tools Used For Developing The Analyzer... 30 3.1 Overview Of Log Parser... 30 3.2 Log Surfer Overview... 33 4 Analysis of the logs... 41 4.1 The Front page of my Analysis is given in the screenshot below:... 41 4.2 Comparison... 44 4.3 Datewise_view... 46 5 Conclusion... 53 6 References... 54

List of tables: Table 2.1: Audit Settings options... 14 Table 2.2 : Event Header Information Types... 21 Table 2.3: Event Types... 21 Table 2.4: Event Log Options... 24 Table 2.5: Supported Facilities in Linux... 26 Table 2.6: Priority Levels... 26 Table 3.1: Configuring File Structure... 34 Table 3.2: Use of Quotes... 34 Table 3.3: Action available for Rules... 35 Table 3.4: Action available for Rules... 36 Table 3.5: Concept of Dynamic Rules... 36 Table 3.6: Position available for Rules... 36 Table 3.7: Schema for Linux Logtable... 40 Page 7

List of Figures: Figure 2.1: Group Policy Editor Settings for Audit Policy... 14 Figure 2.2: Application Log Properties... 22 Figure 3.1: LOG ANLYZER ARCHITECTURE... 30 Figure 3.2: Dtabase Schema of Windows Table... 33 Figure 3.3: Contents of Logsurfer.sh File... 38 Figure 3.4: How to Start Logsurfer... 39 Figure 4.1: View Of the Front Page... 41 Figure 4.2: Result of the submit button... 42 Figure 4.3: Page that specifies the IP Addresses For Further Classification... 42 Figure 4.4: Output Of the submit query.... 43 Figure 4.5: Screen which compares the successful and failure events of both windows and Linux... 44 Figure 4.6: Output Showing The Comparison of two Host IP s where event type is successful... 44 Figure 4.7: Page That Contains The Datewise View... 46 Figure 4.9: Page That Contains The Datewise View.... 47 Figure 4.10: Output that shows all the Failure events between a particular date.... 47 Figure 4.11: For Monitoring Host at a Particular Date... 48 Figure 4.12: Output that shows all the Failure events at a particular date of Windows Host machine.... 48 Figure 4.13: Specify the IP Addresses For Further Classification of Linux... 49 Figure 4.14: Output that shows Further Classification of Linux host... 49 Figure 4.15: Page for Comparison of two Linux host... 50 Figure 4.16: Output that shows Comparison of two Linux host... 50 Figure 4.17: Page That Contains The Datewise View.... 51 Figure 4.18: Page That Contains The Datewise View.... 51 Figure 4.19: Page That Contains Datewise monitoring of Linux Host.... 52 Figure 4.20: Page That Contains The Particular Date Failed events of Linux Host.... 52

1 Introduction 1.1 Problem Area Current software application often produces some auxiliary text files known as log files. Such files are used during various stages of software development, mainly for debugging purpose. Use of log files helps testing by making debugging easier. Log files are commonly for the purpose of permanent software monitoring and are essential in operating systems, computer networks and distributed systems. Log files are often very large and can have complex structure. Although the process of generating log files is quite simple and straightforward, log file analysis could be a tremendous task that requires enormous computational resources, long time and sophisticated procedures. This often leads to a common situation, when log files are continuously generated and occupy valuable space on storage devices, but nobody uses them and utilizes enclosed information. Event logging and event logs play an important role in modern IT systems. Today, many applications, operating systems, network devices, and other system components are able to log their events to a local or remote log server. For this reason, event logs are an excellent source for determining the health status of the system, and a number of tools have been developed over the past 10 15 years for monitoring event logs in real time. Event is a change in the system state, with some events corresponding to system faults (e.g., a disk failure) and some reflecting normal system activity (e.g., a successful user login). When a system component encounters an event, the component could emit an event message that describes the event. For example, when a user gives wrong password, then under security log, under event category it is logged as FAILURE AUDIT and message it gives is authentication failure. since in most cases event messages are appended to event logs in real time as they are emitted by system components, event logs are an excellent source of information for monitoring the system, information that is stored to the event log can be useful for analysis at a later time, e.g., for audit procedures or for incident analysis. So, log monitoring can help us to understand the health of a system and also keep track of both intra network and internetwork intrusions. 1.2 Motivation Log files get accumulated each and every day in our system. We never utilize the logs for checking our system health.log file analysis is an important task, by manual checking our system log files it is very difficult to understand the health of our system. So, some tool needs to be developed which would show our system health in a human friendly manner. The information reported in log files are the information that programmers consider important or useful for program monitoring and/or locating faults. Each record (i.e. a Page 9

line or a group of lines) in a log files is caused by a given event in the program, like user interaction, function call, input or output Procedure etc. Log files helps us to detect security related events, such as successful login or failed login.we can also compare the total number of successful and unsuccessful events. Track all changes when files and folders are created, accessed, viewed, deleted, modified, renamed, etc. Log files helps to mitigate internal threats, monitor file integrity, conduct log forensics analysis, monitor privileged users etc. In an organization, there are many possible signs of incidents which may go unnoticed each day. These events can be studied mainly by analyzing network behavior or by reviewing computer security event logs. In order to avoid or minimize the losses from an incident outcome, the events need to be analyzed as close to realtime as possible. Logging and intrusion detection systems have the potential to produce very large amount of data, and all that data must be managed, filtered and analyzed. Having a single approach and a unified platform helps with this very difficult and challenging task to monitor and report in near real time. Automation is needed to perform an initial analysis of the data and to alert on select events of interest for human review. Event correlation software and centralized logging can be of great value in automating the analysis process. However, the effectiveness of the process depends on the quality of the data and the data rules that goes into it. 1.3 Contribution System Log Analyzer collects, normalizes, and aggregates security, systems, and application log data from enterprise wide Windows, Linux, and UNIX hosts. The logs are collected in real time and send to server to store in a central database for further analysis. Before sending log records to the server, the log records have been converted into XML format for sending logs.for sending log records Transmission Control Protocol is used, though it is slow compared to User Datagram Protocol but it is reliable and packet losses chances are very low here. The format of XML record will be described later. The main challenge was that different host had different software. Some System uses LINUX while other Windows. Converting the logs of these two in XML format and then sending to the server was a challenging task. The XML tags have been used as such, that the data within the tags can be easily understood and make the line look structured. Also, if there would have been any data loss that can also be detected.windows log keeps record of various attributes, but here only the essential attributes have been taken into consideration for syslog analysis.in case of Linux all the attributes have been considered.inconsistent log formats also present Page 10

challenges to people reviewing logs, who need to understand the meaning of each data field in each log to perform a thorough review. Log management infrastructure typically comprises of three tiers: log generation, log storage, and log monitoring and analysis. The log generation tier involves hosts making their logs available to log servers in the second tier. This is performed in two different ways. The exact method depends on the log type, and, on the host and network controls. In one way hosts run some services to send their log data over the network to log collection servers. Alternatively, hosts allow the log servers to pull the log data from them. The logs are often transferred to the log receivers either in a real time or near real time manner, or in occasional batches based on a schedule. The log storage tier is composed of one or more log servers receiving log data from the hosts. These log receivers are also called collectors or aggregators. The log monitoring and analysis tier contains consoles that are used for monitoring and reviewing of log data. Analyses in the form of graphs are shown. 1.4 Outline This thesis describes how to build a system that checks Log Based Intrusion Detection to create a comprehensive security monitoring platform. Chapter 2 provides an overview of essential terminology in the field of Security Information, Event Monitoring, Study of System Logs, how logs are generated, how system maintain the logs and also about utility of logs. Chapter 3 builds on the terminology by proposing a technical architecture and by providing configuration guidance, comparative study of log formats of Windows And Linux, design of xml format, sending logs in XML format, design of the system Log Analyzer, components of Log Analyzer and finally implementation. Chapter 4 discusses about further studies and improvements the can be done and components that can be added. Chapter 5 gives the detailed list of the references and bibliography. Page 11

2 Study of System Logs 2.1 System logs System logs contain the local events logged by the system or host such as Windows XP or Linux system. It helps in identifying normal and abnormal entries. It s an important resource for computer system management. Collection of important logs on host system would enable the administrators to: Look for performance issues of operating system and applications. Detect the malicious and unauthorized activities such as Failed Login Attempts, data theft, malicious activities performed by virus, worms. Collection and securing the important logs from the host is viral in the security process.methodologies for enabling auditing, generation of logs and managing the same are discussed in the following sections. In order to provide individual user accountability, the computing system must be able to correct identify and authenticate each user, this is then distinguishing factor between system log data and user audit data. Log data captured by syslogd for example, is typically generated by system processes and daemons the report significant events or formation. It neither corresponds to specific user actions nor directly traceable to a specific user. However, audit data generated by the system corresponds directly to recorded actions taken by identifiable and authenticated users. All the audit information gathered must be sufficient for an after the fact investigation. In a sense, audit data is the complete recorded history of a system user. Administrators that review the audit data must watch for events that may signify misuse or abuse of the system and user privileges or intrusions. Some examples include: Accessing files requiring higher privilege Killing system processes Opening a different user's files, mail, etc. Probing the system Installing of unauthorized, potentially damaging software (backdoors, Trojan Horses etc.) Exploiting a security vulnerability to gain higher or different privileges Modifying or deleting sensitive information 2.2 Auditing and Logging for Windows Hosts Audit policy pertains to specific categories of security related events that an administrator wants to audit. By default, all auditing policy settings are not defined. On domain controllers, auditing is turned off by default. By turning ON various auditing event categories, administrator can implement audit policy that suits the security needs of the organization. An audit log records an entry whenever user Page 12

performs certain specified actions. For example, the modification of a file or a policy can trigger an audit entry that shows the action that was performed, the associated user account, and the date and time of the action. Both successful and failed attempts action can be audited. Security audits are extremely important for any enterprise network, as audit logs may provide an indication that a security breach has occurred. If the breach is discovered some other way, proper audit settings will generate an audit log that contains important information about the breach. If no audit settings are configured, it will be difficult or impossible to determine what occurred during a security incident. However, if audit settings are configured so that too many authorized activities generate events, the Security Event Log will be filled with useless data. Regular security analyses enable administrators to track and determine that adequate security measures are in effect for each computer as part of an enterprise risk management program. Such analyses focus on highly specific information about all aspects of a computer related to security, which administrators can use to adjust the security levels. More importantly, this information can help detect any security flaws that may occur in the computer over time.before any audit processes are implemented, an organization should determine how they will collect, organize, and analyze the data. There is little value in large volumes of audit data if there is no underlying plan to utilize it. Also, audit settings can affect computer performance. The effect of a given combination of settings may be negligible on an end user computer but quite noticeable on a busy server. Therefore, it is recommended to perform some performance tests before deploying new audit settings in the production environment. Audit policy settings can be configured at following location within the Group Policy Object Editor: Computer Configuration\Windows Settings\Security Settings\Local Policies\Audit Policy Page 13

Figure 2.1: Group Policy Editor Settings for Audit Policy The options available for each of the audit settings are: Security Setting Success Failure No Auditing Description An audit entry is generated when the requested action succeeds An audit entry is generated when the requested action fails No audit entry is generated for the associated action Table 2.1: Audit Settings options Page 14

2.2.1 Audit account logon events This security setting determines whether to audit each instance of a user logging on to or logging off from another computer in which this computer is used to validate the account. Account logon events are generated when a domain user account is authenticated on a domain controller. The event is logged in the domain controller's security log. Logon events are generated when a local user is authenticated on a local computer. The event is logged in the local security log. Account logoff events are not generated. If an administrator configures this policy setting, he can specify whether to audit successes, audit failures, or do not audit the event type at all. Success audits generate an audit entry when an account logon attempt succeeds, which is useful information for accounting purposes and for post incident forensics so that who successfully logged into which computer can be determined. Failure audits generate an audit entry when an account logon attempt fails, which is useful for intrusion detection. However, this policy setting also creates the potential for a Denial of Service (DoS) attack.when the "Audit: Shut down system immediately if unable to log security audits " setting is enabled; an attacker could generate millions of logon failures, fill the Security event log, and force the computer to shut down. If an administrator configures the Audit account logon events setting to Success on a domain controller, an entry is logged for each user who is validated against that domain controller, even though the user actually logs on to a workstation or server that is joined to the domain. 2.2.2 Audit account management This policy setting determines whether to audit each account management event on a computer. Examples of account management events include the following: A user account or group is created, changed, or deleted A user account is renamed, disabled, or enabled A password is set or changed If an administrator configures the Audit account management setting, he can specify whether to audit successes, audit failures, or do not audit the event type at all. Success audits generate an audit entry when any account management event succeeds, and user should enable them on all computers in the enterprise. When an organization responds to security incidents, it is critical that they should be able to track who created, changed, or deleted an account. Failure audits generate an audit entry when any account management event fails. To set the value to No auditing, click Properties dialog box for this policy setting, select the Define these policy settings check box and clear the Success and Failure check boxes. Page 15

2.2.3 Audit directory service access This policy setting determines whether to audit user access of an "Active Directory" directory service object that has an associated System Access Control List which is a is list of users and groups for which actions on an object are to be audited on a Microsoft Windows based network. If an administrator configures the Audit directory service access setting, he can specify whether to audit successes, audit failures, or do not audit the event type at all. Success audits generate an audit entry when a user successfully accesses an Active Directory object that has a SACL that indicates that the user should be audited for the requested action. Failure audits generate an audit entry when a user unsuccessfully attempts to access an Active Directory object that has a SACL that requires auditing. (Both types of audit entries are created before the user is notified that the request succeeded or failed.) To set value to No auditing, click Properties dialog box for this policy setting, select the Define these policy settings check box and clear the Success and Failure check boxes. If an administrator enables this policy setting and configure SACLs on directory objects, a large volume of entries can be generated in the Security logs on domain controllers. An Administrator should only enable these settings if actually intend to use the information that is created.note: Administrator can configure a SACL on an Active Directory object through the Security tab in that object's Proper1ies dialog box. This method is analogous to Audit object access, except that it applies only to Active Directory objects and not to file system and registry objects. 2.2.4 Audit logon events This policy setting determines whether to audit each instance of user logon, logoff, or network connection to the computer that records the audit event. Account logon events are generated on domain controllers for domain account activity and on local computers for local account activity. If system administrator logs successful account logon audit events on a domain controller, workstation logon attempts do not generate logon audits. Only interactive and network logon attempts to the domain controller itself generate logon events on the domain controller. To summarize, account logon events are generated where the account lives, and logon events are generated where the logon attempt occurs. If an administrator configures the Audit logon events setting, he can specify whether to audit successes, audit failures, or do not audit the event type at all Success audits generate an audit entry when a logon attempt succeeds, which is useful information for accounting purposes and for post incident forensics so that administrator can determine who successfully logged on to which computer. Failure audits generate an audit entry when a logon attempt fails, which is useful for intrusion detection. To set value to No auditing, click Proper1ies dialog box for this policy setting, select the Define these policy settings check box and clear the Success and Failure check boxes. This configuration also creates a potential DoS condition, because an attacker could generate millions of logon failures, fill the Security event log, and force the server to shut down. Hence suitable log rotation methodology should support the auditing. Page 16

2.2.5 Audit object access This policy setting determines whether to audit the event of a user who accesses an object for example, a file, folder, registry key, or printer that has a SACL that specifies a requirement for auditing. If an administrator configures the Audit object access setting, he can specify whether to audit successes, audit failures, or do not audit the event type at all. Success audits generate an audit entry when a user successfully accesses an object that has a SACL. Failure audits generate an audit entry when a user unsuccessfully attempts to access an object that has a SACL (some failure events are to be expected during normal computer operations). For example, many applications (such as Microsoft Word) always attempt to open files with both read and write privileges. If the applications are unable to do so, they then try to open the files with read only privileges. If user enables failure auditing and the appropriate SACL on the file, a failure event will be recorded when such an event occurs. To set the value to No auditing, click Propel1ies dialog box for this policy setting, select the Define these policy settings check box and clear the Success and Failure check boxes. In Microsoft Windows Server 2003 with Service Pack! (SP 1), an administrator can audit access to objects that are stored in the Internet Information server (TIS) metabase. To enable metabase object auditing, administrator must enable Audit object access on the target computer, and then set SACLs on the specific metabase objects whose access he wants to audit. If an administrator configures the Audit object access policy setting and configure SACLs on objects, a large volume of entries can be generated in the Security logs on computers in the organization. Therefore, administrator should only enable these settings if actually intend to use the information that is logged. Note: Administrator must perform a two step process to enable the capability to audit an object, such as a file, folder, printer, or registry key, in Windows Server 2003. After administrator enables the audit object access policy, he must determine the objects whose access he wants to monitor, and then modify their SACLs accordingly. For example, if administrator wants to audit any attempts by users to open a particular file, he can configure a Success or Failure audit attribute directly on the file that he wants to monitor for that particular event with Windows Explorer or Group Policy. 2.2.6 Audit policy change This policy setting detel1nines whether to audit every incidence of a change to user lights assignment policies, Windows Firewall policies, Audit policies, or trust policies. If an administrator configures the Audit policy change setting, he can specify whether to audit successes, audit failures, or do not audit the event type at all. Page 17

Success audits generate an audit entry when a change to user rights assignment policies, Audit policies, or tl1lst policies is successful. This audit information is useful for accounting purposes and can help to determine who successfully modified policies in the domain or on individual computers. Failure audits generate an audit entry when a change to user rights assignment policies, Audit policies, or trust policies fails. To set the value to No auditing, click Properties dialog box for this policy setting, select the Define these policy settings check box and clear the Success and Failure check boxes. If an administrator enables the Audit policy change setting in Windows XP with SP2 and Windows Server 2003 with SP1, logging of configuration changes for the Windows Firewall component is also enabled. 2.2.7 Audit privilege use This policy setting determines whether to audit each instance of a user who exercises a user right. If an administrator configures the Audit privilege use setting, he can specify whether to audit successes, audit failures, or do not audit the event type at all. Success audits generate an audit entry when the exercise of a user right succeeds. Failure audits generate an audit entry when the exercise of a user right fails. To set tins value to No auditing, in tile Proper1ies dialog box for this policy setting, select the Define these policy settings check box and clear the Success and Failure check boxes. If an administrator enables this policy setting, the volume of events that is generated can be very large and the events may be difficult to sor1 through. This setting should only be enabled when there is a plan to use the information that is generated. Audit events are NOT generated for use of the following user rights, even if success audits or failure audits are specified for this policy setting: Bypass traverse checking Debug programs Create a token object Replace process level token Generate security audits Backup files and directories Restore files and directories 2.2.8 Audit process tracking This policy setting determines whether to audit detailed tracking information for events such as program activation, process exit, handle duplication, and indirect object access. If an administrator configures the Audit process tracking setting, he can specify whether to audit successes, audit failures, or do not audit the event type at all. Page 18

Success audits generate an audit entry when the process being tracked succeeds. Failure audits generate an audit entry when the process being tracked fails. To set the value to No auditing, click Proper1ies dialog box for this policy setting, select the Define these policy settings check box and clear the Success and Failure check boxes. If an administrator enables Audit process tracking in Windows XP with SP2 and Windows Server 2003 with SP1, Windows will also log information about the operating mode and status of the Windows Firewall component. When enabled, the Audit process tracking setting generates a large number of events. This policy setting is typically configured to No Auditing. However, the information that this policy setting generates can be very beneficial during an incident response because it provides a detailed log of the processes that were stai1ed and when they were launched. 2.2.9 Audit system events This policy setting determines whether to audit when a user restal1s or shuts down their computer, or when an event occurs that affects either computer security or the Security log. If an administrator configures the Audit system events setting, he can specify whether to audit successes, audit failures, or do not audit the event type at all. Success audits generate an audit entry when an event executes successfully. Failure audits generate an audit entry when an event is unsuccessful. To set the value to No auditing, click PropeI1ies dialog box for this policy setting, select the Define these policy settings check box and clear the Success and Failure checkboxes. As few additional events are recorded if both failure and success audits are enabled for system events, and because all such events are very significant, administrator should configure this policy setting to Enabled on all computers in the organization. 2.3 Event Logs The Event Log records events on the computer and the Security log records audit events. There are three basic logs in Windows environment, these are: Application Logs: The application log contains events logged by applications or Programs. For example, a database program might record a file error in the application log. Security Logs: The security log records events such as valid and invalid logon attempts, as well as events related to resource use such as creating, opening, or deleting files or other objects. For example, if logon auditing is enabled, attempts to log on to the system are recorded in the security log. Page 19

System Logs: The system log contains events logged by Windows system components. For example, the failure of a driver or other system component to load during startup is recorded in the system log. The event types logged by system components are predetermined by the server. Application and system logging start automatically when the computer starts. Security log starts logging as per the audit policy defined either locally or domainwide (in Active Directory implementations). Apart from the logs mentioned above, Microsoft Windows also make other "custom log records" according to the applications or services installed on the system such as Internet Explorer logs.windows PowerShell logs, DNS logs, Directory Service logs, file replication Service logs etc. Again, the connotation of all these types of logs are to record all the events occurring on the system which can be application specific or system specific. 2.4 Event Viewer Using Event Viewer, user can view and set logging options for event logs in order to gather information about hardware, software, and system problems. The event header contains the following information: Information Type Date Time User Computer Source Event Description The date the event occurred. The date and time of an event are stored in Universal Time Coordinate (UTC) but always display in the viewer's locale The time the event occurred. The date and time of an event are stored in UTC but always display in the viewer's locale. The name of the user on whose behalf the event occurred. This name is the client ID if the event was actually caused by a server process or the primary ID if impersonation is not taking place. Where applicable, a security log entry contains both the primary and impersonation IDs. Impersonation occurs when the server allows one process to take on the security attributes of another. The name of the computer where the event occurred. This is usually the name of user's own computer, unless user is viewing an event log on another computer. The software that logged the event, which can be either a program name such as SQL Server, or a component of the system (such as a driver name) or of a large program. For example, "Elnkii" indicates an EtherLink II driver. The Source always remains in its original language. A number identifying the pal1icular event type for this Page 20

Type Category source. The first line of the description usually contains the name of the event type. For example, 6005 is the ID of the event that occurs when the Event log service is started. The first line of the description of such an event is "The Event log service was started." Using the values of Source and Event together, product support representatives can troubleshoot system problems. A classification of the event severity: Error, Information, or Warning in the system and application logs; Success Audit or Failure Audit in the security log.in the Event Viewer normal list view, these are represented by a symbol. A classification of the event by the event source. This information is primarily used in the security log. For example, for security audits, this corresponds to one of the event types for which success or failure auditing can be enabled in Group Policy by a member of the Administrators group. Table 2.2 : Event Header Information Types Different types of events: Event type Description Error A significant problem, such as loss of data or loss of functionality. For example, if a service fails to load during startup, an Error will be logged. Warning An event that is not necessarily significant, but might indicate a possible future problem. For example, when disk space is low, a Warning might be logged. Information An event that describes the successful operation of an application, driver, or service. For example, when a network driver loads successfully, an Information event will be logged. Success Audit Any audited security event that succeeds. For example, a user's successful attempt to log on the system will be logged as a Success Audit event. Failure Audit Any audited security event that fails. For example, if a user tries to access a network dri.ve and fails, the attempt will be logged as a Failure Audit event. Table 2.3: Event Types Page 21

2.5 Configuring logging parameters Administrator can use Group Policy to set the maximum log sizes and log wrapping options, as well as set access permissions on event logs. Configure the event log settings in the following location within the Group Policy Object Editor: Computer Configuration\Windows Settings\Security Settings\Event Log\Settings for Event Logs These event log settings do not appear in the Local Computer Policy object. Another way to define logging parameters for each kind of event log is by using Event Viewer.To define parameters, click start, type "eventvwr" in the RUN window, "Event Viewer "Management console will be opened, right click a log in the console tree, and then click Properties. On the General tab, administrator can set the maximum size of the log and specify whether the events are overwritten or stored for a certain period of time (see Figure 2.2). Figure 2.2: Application Log Properties Page 22

2.5.1 Settings for "Maximum log size" This security setting specifies the maximum size of the event log of any type application, security or system which has a maximum limit of 4 GB. Log file sizes must be a multiple of 64 KB. If an administrator enters a value that is not a multiple of 64 KB, Event Viewer will set the log file size to a multiple of 64 KB. Even though the theoretical limit for the Event Viewer and Group Policy Object Editor UIs allows to specify as much as 4GB per log Microsoft has verified that the practical limit is approximately 300 MB on most servers that is, 300 MB for all of the event logs combined. On Microsoft Windows XP, member servers, and standalone servers, the combined size of the Application, Security, and System event logs should not exceed 300 MB. On domain controllers, the combined size of these three logs plus the Active Directory directory service, DNS, and Replication logs should not exceed 300 MB. These limitations can be prevailed over by using central logging servers like syslog or by using SIEM (Security Information and Event Management) products. Event Log size and log wrapping should be defined to match the business and security requirements, determined at the time of designing the enterprise security plan. Although there is no simple equation to determine the best log size for a particular server, administrator can calculate a reasonable size. The average event takes up about 500 bytes within each log, and the log file sizes must be a multiple of 64 KB. If administrator can estimate the average number of events that are generated each day for each type of log in organization, user can determine a good size for each type of log file. For example, if a file server generates 5,000 events per day in its Security log and user wants to ensure that he should have at least 4 weeks of data available at all times, then user would want to configure the size of that log to about 70 ME. (500 bytes * 5000 events/day * 28 days = 70,000,000 bytes.). Then, check the servers occasionally over the following four weeks to verify calculations and that the logs retain enough events for user's needs. 2.5.2 Retaining event logs This policy setting determines the number of days of event log data to retain for the Application, Security, and System logs; if the retention method that is specified for the log is By Days. Configure this setting only if administrator archive the log at scheduled intervals and make sure that the maximum log size is large enough to accommodate the interval. The possible values for the "Retain event logs" settings are: A user defined number in days between 1 and 365 Page 23

Not Defined (in Group Policy Object Editor) A user must be assigned the "Manage auditing and security log" user right to access the Security log. 2.5.3 Retention method for event log The default logging policy is that if a log is full, the oldest events are deleted to make room for new events, provided events are at least seven days old. Administrator can customize this policy, or use event log wrapping options, for different logs. Event log wrapping options include the following: Use Overwrite events as needed Overwrite events older than [x] days Do not overwrite events To Have new events continue to be written when the log is full. Each new event replaces the oldest event in the log. This option is a good choice for low maintenance systems. Retain the log for the number of days administrator specifies before overwriting events. The default is seven days. This option is the best choice if administrator wants to archive log files weekly. This strategy minimizes the chance of losing important log entries and at the same time keeps log sizes reasonable. Clear or archive the log manually rather than automatically. Select this option only if administrator call1lot afford to miss an event (for example, for the security log at a site where security is extremely important). Table 2.4: Event Log Options In addition to the above options available for the event log settings in Group Policy Object Editor, one more option is available for "Retention method for event log", that is "Not Defined". 2.6 Auditing and Logging for Linux Hosts 2.6.1 Linux System Logging By default Linux system logs are contained in the directory /var/log. Some applications such as httpd and samba have a directory within /var/log for their log files. Most log files are in plain text format. Some log files are readable by all users on the system; however, root privileges are required to read most log files. The log files being in text format is readable by any text editor. Page 24

2.6.2 Syslog daemon The syslog program is a distributed logging interface providing a standardized framework under which programs (both operating system and applications) can issue messages to be stored either on the local system or sent to a remote host. Logging on Linux system can be done through the syslogd, syslog daemon. Syslogd listens for messages from either port 514 (UDP) or through /dev/log. Once running, almost any part of the system, including applications, drivers, as well as other daemons can make log entries. Syslog accepts log data from the kernel, from all local processes and even from processes on remote machines. 2.6.3 Configuring syslogd, syslog daemon Syslog configuration consists of routing error messages from various facilities (and at various severity levels to one or more of the following destinations: Log files anywhere on the system Another computer running syslog with its own syslog configuration Active users on the system Default configuration file for syslogd is located at /etc/syslog.conf The configuration file contains one rule per line. Each rule consists of selector portion, which determines the events to react to and the action portion, which determines what is to be done. Thus Syslog's mapping of actions to facilities and priorities are specified in /etc/syslog.conf. The selector has the general syntax: facility.priority/level The facility part says what aspect of the system is to be recorded and the priority says what level of messages to react to. 2.6.3.1 Facility The facility argument is used to specify what type of program is logging the message. The developer of a program decides which facility a program will utilize. In some cases, it may be configurable by the end user. Messages from different facilities can be handled differently by editing the syslog configuration file. Supported facilities in Linux Platform: LOG_AUTH security/authorization messages (DEPRECATED Use LOG_AUTHPRIV instead) LOG _AUTHPRIV security/authorization messages (private) Page 25

LOG _CRON LOG _DAEMON LOG _FTP LOG _KERN LOGLOCAL_O through LOG LOCAL_ 7 LOG _LPR LOG _MAlL LOG _NEWS LOG _SYSLOG LOG_ USER (default) LOG _UUCP clock daemon (cron and at) system daemons without separate facility value ftp daemon kernel messages reserved for local use line printer subsystem mail subsystem USENET news subsystem messages generated internally by syslogd generic user level messages UUCP subsystem Table 2.5: Supported Facilities in Linux 2.6.3.2 Priority Level The log level or priority determines the importance of the message. The levels are in order of decreasing importance. LOG_ EMERG LOG _ALERT LOG_ CRIT LOG _ERR LOG _WARNING LOG _NOTICE LOG _INFO LOG _DEBUG system is unusable action must be taken immediately critical conditions error conditions warning conditions normal, but significant, condition Informational message debug-level message Table 2.6: Priority Levels 2.6.3.3 An example /etc/syslog.conf An example syslog configuration file /etc/syslog.conf with description of each setting therein is given below. kern.* *. info ;mail.none ;news.none;authpriv.none; cron. None authpriv.* mail. * cron. * /dev/console var/log/messages var/log/secure var/log/maillog var/log/cron Page 26

*.emerg @loghost *.emerg * uucp,news.crit local7.* news.=crit var/log/spooler var/log/boot.log var/log/news/news.crit *.emerg;kernel.critical root Description: kern.* dev/console Kernel messages are sent to the console. *. info ;mail.none;news.none; authpri.none; cron.none var/log/messages All messages of priority "info" and above are logged in the var/log/messages file, but none from the mail, news, cron or authentication facilities/subsystems. authpriv. * var/log/secure Messages from authentication facility are logged to file var/log/secure *.emerg @loghost Sends all emergency messages to the remote machine "loghost" Note: The IP address of loghost is defined in /etc/hosts file so as to enable the system to send the logs to the remote machine. Also the remote host's syslogd process will need to be enabled for remote logging described in section. uucp,news.crit varl/log/spooler Messages of priority "info" and above for uucp and news are logged to file var/log/spooler. news.=crit var/log/news/news.crit Messages of priority "crit" only for news are logged to file /var/log/news/news.crit file. *.emerg;kernel.critical root Page 27

This is a multiple selector on a single line. In this case all emergency messages as well as critical messages from the kernel facility would be notified to the user root. *.emerg * Messages of priority emergency and above are notified to all logged in users. 2.6.4 Running syslogd By default syslog daemon is run at the system start up. On most of the Linux systems the start up script for syslogd is located at /etc/init.d/syslog. 2.6.5 Remote logging Almost all distributions of Linux servers have a syslog daemon by default. However the default configuration of the syslog daemon of Linux doesn't accept syslog messages from the network. In order to tell syslogd to accept remote logs, remote logging must be turned on in two configuration files. The two configuration file settings are similar, so the same change will be performed on both files. Modify the files /etc/init.d/syslog and /etc/sysconfig/syslog. Add r in the SYSLOGD OPTIONS so that the lines are as shown below. SYSLOGD OPTIONS= m 0 r x Save and exit the file and restart syslog using the command service syslog restart. Page 28

2.7 Utility Of Log Management Log events are the primary records of system and network activity. Log is a record of actions and events that takes place on a computer system. Logs are the primary record keepers of system and network activity. When security controls experience failures, logs would be particularly helpful. Some of the utilities are: Detect/Prevent Unauthorized Access and insider Abuse Meet Regulatory Requirement Forensic Analysis and Correlation Track Suspicious Behavior IT Troubleshooting and Network Operation Monitor User Activity Measure Application Performance Achieve Cost Reduction in System Maintenance Page 29

3 Design and Tools Used For Developing The Analyzer Log management infrastructure typically comprises of three tiers: log generation, log storage, and log monitoring and analysis. The log generation tier involves hosts making their logs available to log servers in the second tier. This is performed in two different ways. The exact method depends on the log type, and, on the host and network controls. In one way hosts run some services to send their log data over the network to log collection servers. Alternatively, hosts allow the log servers to pull the log data from them. The logs are often transferred to the log receivers either in a realtime or near real time manner, or in occasional batches based on a schedule. The log storage tier is composed of one or more log servers receiving log data from the hosts. These log receivers are also called collectors or aggregators. The log monitoring and analysis tier contains consoles that are used for monitoring and reviewing of log data. Analysis in the form of graphs is shown. LOG ANALYSIS LOG STORAGE XML ENGINE LOG GENERATION 3.1 Overview Of Log Parser Figure 3.1: LOG ANLYZER ARCHITECTURE Log parser is a powerful, versatile tool that provides universal query access to textbased data such as log files, XML files and CSV files, as well as key data sources on the Windows operating system such as the Event Log, the Registry, the file system, and Active Directory. You tell Log Parser what information you need and how you want it processed. The results of your query can be custom formatted in text based output, or they can be persisted to more specialty targets like SQL, SYSLOG, or a chart. Page 30

Features: This version,can parse the following files very easily from the Log Parser tool: IISW3C, NCSA, IIS, IISODBC, BIN, IISMSID, HTTPERR, URLSCAN, CSV, TSV, W3C, XML, EVT, ETW, NETMON, REG, ADS, TEXTLINE, TEXTWORD, FS and COM. Benefits: Log Parser provides a generic SQL like language on top of many types of data like IIS Logs, Event Viewer entries, XML files, CSV files, File System and others; and it allows to export the result of the queries to many output formats such as CSV (Comma Separated Values, etc), XML, SQL Server, Charts and others. Log parser is a powerful, versatile tool that provides universal query access to textbased data such as log files, XML files and CSV files, as well as key data sources on the Windows operating system such as the Event Log, the Registry, the file system, and Active Directory. You tell Log Parser what information you need and how you want it processed. The results of your query can be custom formatted in text based output, or they can be persisted to more specialty targets like SQL, SYSLOG, or a chart. Most software is designed to accomplish a limited number of specific tasks. Log Parser is different... the number of ways it can be used is limited only by the needs and imagination of the user. So,Log Parser generates the logs and sends it to the client program. The client program then puts it in a XML Format and sends it to the server for storage at a central database. The contain of the batch file that is executed from the client application to gathers the security log is C:\"Program Files\Log Parser 2.2"\LogParser "SELECT TOP 10 EventLog, TimeGenerated, EventID, EventTypeName, EventCategoryName, SourceName,ComputerName,Message FROM Security WHERE EventID = 680" We can change the query according to our needs. The xml format the the client application generates as output is: <?xml version="1.0" <loginformation> encoding="utf8" standalone="no"?> <log> <EventLog>Security</EventLog> <TimeGenerated>20130511 13:24:55</TimeGenerated> <EventID>680</EventID> Page 31

</log> <EventTypeName>Failure Audit event</eventtypename> <EventCategoryName>Account Logon</EventCategoryName> <SourceName>Security</SourceName> <ComputerName>WORKSTATION-3</ComputerName> <Message>Logon attempt by: MICROSOFT_AUTHENTICATION_PACKAGE_V1_0 Logon account: Soma Hazra Source Workstation: WORKSTATION-3 Error Code: 0xC000006A </Message> </loginformation> Now, the server accepts the xml output and parses it before sending to the database. The server can accept multiple clients output. The database connectivity is done in the following manner: try{ Class.forName("com.microsoft.sqlserver.jdbc.SQLServerDriv er"); System.out.println("JDBC driver loaded"); Connection connect = DriverManager.getConnection("jdbc:sqlserver://localhost:1 433;DatabaseName=LogTable","sa","abcdefgh"); System.out.println("JDBC driver loaded:"+connect); Statement statement = connect.createstatement(); String command = "INSERT INTO windowslogtable (Hostip,EventLog,TimeGenerated,EventID,EventTypeName,Even tcategoryname,sourcename,computername,message) VALUES('"+HOSTIP+"','"+s1+"','"+s2+"','"+s3+"','"+s4+"',' "+s5+"','"+s6+"','"+s7+"','"+s8+"')"; statement.executeupdate(command); connect.close(); }catch (ClassNotFoundException e){ System.err.println(e); } catch (SQLException e1){ System.err.println(e1); } Here s1,s2,s3,s4,s5,s6,s7,s8 are the parsed xml fields which are being send into the database. Page 32

NAME TYPE DESCRIPTION Hostip nvarchar(50) For storing ip of the client machine Eventlog nvarchar(50) For storing Eventlog TimeGenerated smalldatetime For storing time at which the log was generated EventID nchar(10) For storing the event id EventTypeName nvarchar(50) For storing the event type name EventCategoryName nvarchar(50) For storing event category name SorceName nvarchar(50) For storing source name ComputerName nvarchar(50) For storing the name of computer Message nvarchar(max) For storing the message generated by the operating system. Figure 3.2: Dtabase Schema of Windows Table 3.2 Log Surfer Overview 3.2.1 Introduction Logsurfer is a program for monitoring system logs in real time, and reporting on the occurrence of events. It is similar to the well known swatch program on which it is based, but offers a number of advanced features which swatch does not support. Logsurfer is capable of grouping related log entries together for instance, when a system boots it usually creates a high number of log messages. In this case, Logsurfer can be setup to group boot time messages together and forward them in a single Email message to the system administrator under the subject line "Host xxx has just booted". Swatch just couldn't do this properly. Logsurfer is written in C this makes it extremely efficient, an important factor when sites generate a high amount of log traffic. It has been reported that Logsurfer could record more than 500,000 events per day and Logsurfer had no trouble keeping up with this load. Swatch, on the other hand, is based on Perl and runs into trouble even when dealing with a much smaller rate of log traffic. So, Logsurfer is both economic and efficient, and thus this tool for monitoring Linux logs in real time. Configuration of file structure Each line starting with the character "#" is considered a comment and is not processed. Each line starting with a white space (TAB or space) is considered as a continuation of the previous line. All other lines are interpreted as the beginning of a new rule. Each configuration rule consists of six mandatory fields and one optional field. Fields are separated by white spaces. More details are explained in the man page logsurfer(4): Page 33

Field Explanation 1. match_regex This field determines what lines match the rule. Use POSIX regular expressions, as defined withinegrep (1), to describe the pattern. 2. not_match_regex Unless this field is specified as " ", it is considered to be a regular expression that excludes any line matched by match_regex. 3. stop_regex Unless this field is specified as " ", the rule is deleted from the list of active rules if the line matches this regular expression. 4. not_stop_regex If the line matches stop_regex and the fourth field is not specified as " ", the rule is not deleted. 5. timeout Time duration in seconds. The rule does not timeout if "0" is specified. 6. continue(optional) If the keyword "continue" is specified, the remaining rules in the configuration files are considered instead of terminating the matching process with the current line (the default condition). 7. action This field specifies the action of a rule by one keyword (explained below) along with optional arguments. Table 3.1: Configuring File Structure Use of quotes You can use sub matches of regular expressions to define new contexts or to invoke other actions. These are defined in egrep (1). You need to use quotes correctly to ensure the action is executed as intended. There are three ways to use quotes: Quotes Explanation string The string is terminated at the first whitespace (TAB or space). 'string' The string is terminated at the first single quote (') following the initial one. The content between opening and closing quotes is matched. "string" The string is terminated at the corresponding double quote ("). The backslash character \ is used as an escape character. \" specifies a double quote without ending the quoted string. Table 3.2: Use of Quotes Actions available for rules The following actions are available: Action Explanation ignore No further processing is done. Matching lines are ignored from that rule on. exec At least one argument must be specified that identifies the program to be called. More arguments can be passed. Variables describing submatches of the match_regex are also allowed. Page 34

pipe open delete report rule Similar to the exec action, this calls an external program to process the matching lines but passes these lines via stdin. This action opens a new context it already exists. See below for more information regarding the use of contexts. This deletes an existing context without forcing the action related to this context to be executing. The specified contexts are fed as standard input to the program specified by the first argument. New rules can be created "on the fly." See below for more information about the concept of creating new rules. Table 3.3: Action available for Rules Contexts Single log messages and sets of log messages can relate to a single event or a set of related events. Sending single messages to an administrator by email is not efficient and makes it more difficult for the administrator to know what is happening. Most of the time, the administrator can only understand the situation by inspecting the log file and analyzing messages in context. With this being the case, logsurfer was developed to collect a set of log messages based on regular expressions. Default actions are associated with each context. These are executed after a specified time period or after a specified number of log messages are collected. The opening of a context ensures that there is a defined end to a context. The context is also available for further processing such as sending it as a report to an email address. The regular pattern that specifies the match pattern is used as the "name" for a context and is used to reference it within other rules. Each context consists of six mandatory fields. Fields are separated by white spaces. More details are explained in the man page of logsurfer: Field Explanation 1. match_regex This field determines what lines match the context. 2. not_match_regex Unless this field is specified as " ", it is considered to be a regular expression that excludes any line matched by match_regex. 3. line_limit This value defines the maximum number of lines that are to be stored within the context. It is always useful to define a limit to avoid denial of service attacks. 4. timeout_abs This value in seconds defines how long the context is considered active before the default action is executed. 5. timeout_rel In addition to the absolute timeout, the default action is executed if a relative timeout (in seconds) is defined and no new log message is added to the context within that time period. Page 35

6. default_action The specified action is carried out whenever the maximum number of lines, or the absolute or relative timeout is reached. All actions except open, delete, and rule are available. Table 3.4: Action available for Rules Concept of dynamic rules One of the most powerful features of logsurfer is the ability to create new rules as needed. This is especially useful when a specific recurring message (such as "file system full") needs immediate attention but only requires an alert to be sent once per hour. This can be achieved by sending mail to the administrator and including a new rule to ignore the same message over a specified time period. The advantage of dynamic rules is that you can redefine at run time which events are "of interest." For example, if a telnetconnection from a given host is denied, you might want to review all other events related to this host. To initiate a new rule, use the following action syntax Field Explanation 1. The keyword rule initiates the creation of a new rule. rule_keyword 2. position This specifies the position of a rule (see below). 3. new_rule This specifies the new rule in the same syntax as explained above for all rules. Table 3.5: Concept of Dynamic Rules Position before behind top bottom Explanation The new rule is added before the actual rule. The new rule is added behind the actual rule. The new rule is added as the first rule. The new rule is added as the last rule. Table 3.6: Position available for Rules Handling of a single log message For each log message, sequences of procedures are executed: the message is stored within each context when it matches match_regex unless it matches not_match_regex. the message is compared to the rules in their actual order. When the first rule matching the match_regex line occurs, the given action is executed unless the message matches not_match_regex. No other rules are compared unless the keyword continue is specified. the message is compared to all rules. If the message matches the stop_regex rule but not the not_stop_regex rule, the rule is deleted. Page 36

the timeout value of all rules is analyzed. If a timeout is reached, the rule is deleted. the relative and absolute timeout values of all contexts are analyzed. If a timeout is reached, the context is deleted and the default action is processed. By default, logsurfer stops searching for another matching rule after the first matching rule is found, so the order of rules is important. Rules that specify messages to ignore and rules that address frequently occurring messages should appear early in the configuration definition. Start the Logsurfer by writing the command /etc/init.d/logsurfer start result kern.log, where result is the dump file and kern.log is the log file that LogSurfer will monitor in real time.here, in this application the dump file is not required as the output of Logsurfer is redirected to a java program which converted it into XML and sending it to the server.if dump file is not required you can change the Logsurfer code slightly according to your need. The logsurfer.sh file must be inside /etc/init.d/. The LogSurfer.sh file must be modified slightly in order to redirect the output of it in the java client program. The screenshot of the modification is given below. This will help you to understand as to where make the modification. The EXE file must be in the path specified in Logsurfer.sh. SYSLOG_FILE = /var/log and then depending on the log file that you specify while starting the LogSurfer,it monitors that specific kind of log file. Then, if you want to redirect the output to another program then you have to comment the dumping part in the LogSurfer.sh file, which is being shown in the last line of the screenshot below. This is done to redirect the output of LogSurfer to another application. If you want to dump the result to a file in /usr/local/log/ then don t comment the last line. Page 37

Figure 3.3: Contents of Logsurfer.sh File Now, the command that you need to type for starting the Logsurfer is given below: /etc/init.d/logsurfer start result kern.log Now, to redirect the output to the java application, type Java jar /root/netbeansprojects/client_side/dist/client_side.jar with a (PIPE)Sign at front,to redirect the output to the java application. Page 38

Figure 3.4: How to Start Logsurfer The output that is generated using the XML format. <?xml version="1.0" encoding="utf-8" standalone="no"?><loginformation><log> <datetime>20130527 01:29:28</datetime> <username>ubuntu</username> <programname>kernel:</programname> <information>[ 122.680779] usbcore: registered new interface driver hiddev</information></log></loginformation> Now, the server accepts the xml output and parses it before sending to the database. The server can accept multiple clients output. The database connectivity is done in the following manner: try{ Class.forName("com.microsoft.sqlserver.jdbc. SQLServerDriver"); System.out.println("JDBC driver loaded"); Page 39

Connection connect = DriverManager.getConnection("jdbc:sqlserver://localh ost:1433;databasename=logtable","sa","abcdefgh"); System.out.println("JDBC driver loaded:"+connect); Statement statement = connect.createstatement(); String command = "INSERT INTO logtable (Datetime, Username, Programname, Information, hostip) VALUES('"+s1+"','"+s2+"','"+s3+"','"+s4+"','"+HOSTIP +"')"; statement.executeupdate(command); connect.close(); }catch (ClassNotFoundException e){ System.err.println(e); } catch (SQLException e1){ System.err.println(e1); } Here s1,s2,s3,s4,hostip are the parsed xml fields which are being send into the database. The database table schema is as follows: Name Type Description Datetime smalldatetime Stores the datetime at which the log was generated Username nvarchar(50) Name of the user Programname nvarchar(50) The application which is being executed Information nvarchar(max) Contains the entire message hostip varchar(50) Stores the IP of the host Table 3.7: Schema for Linux Logtable Page 40

4 Analysis of the logs In Analysis, the logs are classified into 3 parts: i) It contains the count of all successful and failure events of all HOST IP that are there in the database. a) Under (i) there is further classification of HOST IP s that use windows operating system. b) Under (i) there is also further classification of HOST IP s that use Linux operating system. ii) It contains the comparison of Two HOST IP s that use same operating system. iii) It contains Datewise view of Successful and Failure events of HOST IP s. And it also contains Particular Day Events Analysis of different hosts. 4.1 The Front page of my Analysis is given in the screenshot below: Figure 4.1: View Of the Front Page Page 41

This resulted in the following image generation. Figure 4.2: Result of the submit button This shows, all the successful events that took place till date, of all the Host IP s of both Windows and Linux machines. Figure 4.3: Page that specifies the IP Addresses For Further Classification Page 42

For Further Classification of Windows and Linux Host enter the IP Address And after clicking submit button it generates the following result Figure 4.4: Output Of the submit query. This shows IP 192.168.148.66 has 4 policy change,5 object accesses,10 privilege use and 10 system event. Page 43

4.2 Comparison Now, the next button in my menu bar is Comparison, which compares Successful and Failed events of both Windows and Linux. The screen is shown below: Figure 4.5: Screen which compares the successful and failure events of both windows and Linux This results into another page which is shown below: Figure 4.6: Output Showing The Comparison of two Host IP s where event type is successful Page 44

IP 192.168.148.66 shows that it has 3 policy changes,6 object accesses, no account logon, no logon, no logon/logoff,10 privilege use and 10 system events. IP 192.168.148.1 shows that it has 6 policy change and 4 object access and no account logon, logon/logoff, privilege, system event. Now, similarly you can do failure event comparison of windows and also both failure and successful event comparison of Linux. Now, the next option in the menu bar is DATEWISE_VIEW which gives the graph of all the events between two dates of particular Host. One can also view the events at a Particular Date, corresponding to a particular Host IP. This gives graphically all the events that took place on that particular date. Here, you can specify two dates, first date and second date and the hostip, against which you want to see the events. Page 45

4.3 Datewise_view Below, the screenshot displays the page that contains the DATEWISE_VIEW. Figure 4.7: Page That Contains The Datewise View After clicking submit it gives the result as follows : Figure 4.8: Page That Shows the events between two particular dates. Page 46

Figure 4.8: Page That Contains The Datewise View. Here It shows all the Failure events between that particular date. Figure 4.9: Output that shows all the Failure events between a particular date. Figure 4.8 shows, all the successful events that took place between Apr 20 2013 and May 20 2013 for IP address 192.168.148.66 Figure 4.10 shows, all the failure events that took place between Apr 20 2013 and May 20 2013 for IP address 192.168.148.66 Page 47

Now, for viewing all the events at a particular day, for a particular Host IP, see the following screenshot. Figure 4.10: For Monitoring Host at a Particular Date Now you will get the following output: Figure 4.11: Output that shows all the Failure events at a particular date of Windows Host machine. It shows that On May 11 2013 there are 5 account logon and 5 logon/logoff events for IP address 192.168.148.66 Page 48

Now similarly all the task that have been shown for Host using windows operating sytem can also be done on Host using Linux operating system. For example in Figure 4.3, if you would like to see the further classification of Linux. Figure 4.12: Specify the IP Addresses For Further Classification of Linux It gives the following result: Figure 4.13: Output that shows Further Classification of Linux host It shows it has 1 account change, 1 cron jobs and 1 link state change and no Authentication Failures, configuration change for IP address 192.168.148.35. Page 49

Now, from Figure 4.5 we can also compare two Linux Host as follows. Figure 4.14: Page for Comparison of two Linux host Now, we get the following output : Figure 4.15: Output that shows Comparison of two Linux host Here 192.168.148.35 has 1 account change,1 cron jobs,no authentication failures,1 link state change and no configuration change, whereas 192.168.148.12 has only 1 cron job and no other events. Page 50

Now from Figure 4.7, we can also view Datewise events of a particular LINUX Host as follows. Figure 4.16: Page That Contains The Datewise View. Now, after clicking submit button we get the following output, Figure 4.17: Page That Contains The Datewise View. This shows all the successful events between May 10 2013 and May 22 2013 for the IP address 192.168.148.35 Page 51

Now, from Figure 4.10, which shows all the Failure events at a particular date of windows host we can do the same for Linux also. Figure 4.18: Page That Contains Datewise monitoring of Linux Host. Now, we get the following output: Figure 4.19: Page That Contains The Particular Date Failed events of Linux Host. Here there are 3 authentication failures and no account changes, cron jobs, link state changes and configuration changes. Page 52

5 Conclusion This project contributes to design and development of a system log analyser, which collects logs in real time. It also does analysis based on the data stored in database. The graphical user interface will help users to analyse and also compare different host. This will help users to prevent unauthorized access, track suspicious behaviour, monitor user activity, and check the system health and also reduction of cost. In comparison to present day Log Analyzer, the features that can be added to make this analyzer much more efficient are Analyze Application specific Logs, MS IIS Web server, DHCP Windows server, DNCP Linux server, MS SQL database, Oracle database, Apache Web server, Print server. Real time alerting, privileged user monitoring, Universal Log Parsing and Indexing, Generate pre define compliance reports for Event logs & Syslogs. Future Work: Collecting server logs, Switches and router logs. Generating Alerts when some emergency take place Agentless log collection The search results can be exported to CSV and PDF format reports. More search criteria can be provided for more specialized kind of analysis Page 53

6 References General References and Resources 1. System Security guidelines http://www.cert-in.org.in/knowledgebase/guidelineslcisg-2004-04.htm 2. Web server Security guidelines 3. http://www.cert-in.org.in/knowledgebase/guidelinesl cisg-2004-04.htm 4. NIST SP - 800-92 Guideline for Log Management www.csrc.nist.gov/publications/nistpubs/800-92/sp800-92. pdf 5. http://www.loganalysis.org 6. http://www.syslog.org/ 7. http://www.cert-in.org.in/knowledgebase/presentation/logs-forensics. pdf 8. http://www.cert-in.org.in/knowledgebase/presentation /16thapril04/loganalysis.pdf 9. http://www.crypt.gen.nz/logsurfer/#links 10. http://www.microsoft.com/en-in/download/details.aspx?id=24659 Windows Auditing and Logging 10. Threats and Countermeasures Guide http://www.microsoft.com/technet/securitylguidance/serversecurity/tcg/tc gch00.mspx 11. http://technet2.microsoft.com/windowsserver/en/library/6847e72b-9c47-42 ab-b3e3-691addac9f331033.mspx?mfr=true 12. http://technet2.microsoft.com/windowsserver/en/library/d8fc798c-1e77-4043-b59c-97b4961d85a1033.mspx?mfr=1rue 13. http://technet2.microsoft.com/windowsserver/en/library/42c66475-3346- 428f-8faf-47a6611655ee1033.mspx?mfr=true 14. http://technet2.microsoft.com/windowsserver/e/llibrary/20068d03-6473 - 4e00-84d4-fb1c7cce57d21033.mspx?mfr=true 15. http://technet2.microsoft.com/willdowsserver/en/library/ e104c96f-e243-41c5-aaead0465 5 5a079d1033.mspx?mfr=true 16. http://technet2.microsoft.com/windowsserver/en/library/50fdb7bc-7dae- 4dcd-8591-382aeft2ea791033.mspx?mfr=true Page 54

16. http://technet2.microsoft.com/willdowsserver/en/library/962f5863-l5df- 427l-9ae0-4b0412e2974 91033.mspx?mfr=true 1 7. http://technet2.microsoft.com/windowsserver/en/libraryl ee2f85ac-e3fb- 4a24-b133-8c7fdfc5cee81033.mspx?mfr=true 18. http://technet2.microso ft. com/willdowsserverl enllibrary/oa 64 2 coc-3 8 7 a-44 f5 -bfd9-95 1 b87fd 13 80 1 033.mspx?mfr=true 19. http://techllet2. microsoft. com/willdowsserver/en/library/a 8 297bc2 -d5 3 a -4a2 f-94c5-8e4 l2ae4e386 l 03 3.mspx?mfr=tllle 20. http://www.windowslog.com/en/ 21. http://www.eventid.net 22. http://www.monilog.com/en 23. http://www.eventreporter.com/en Event Viewer 24. http://teehllet2.microsoft.com/windowsserver/en/ library/905 f2 SI 0-a4S 2-4040-S2S-1280d5ddd2d51033.mspx?mfr =true 25. http://teehnet2.microsoft.com/windowsselver/en/ library/ad4 2d67 d -fl f6-4aef-a 13 0-dd4eaOdc93491 033.mspx?mfr=true 26. http://technet2.microsoft. com/windowsserver/ en/ library/6 fcff 4d4-5624- 43fc-aed 1-68808d17d0391033.mspx?mfr=true Windows Event Logs 27. http://support.microsoft.com/308427 28.http://technet2.microsoft.com/windowsserver/en/library/9930c8f1-54ed- 4d07-afa6-bc3e597bbege 1 03 3.mspx?mfr=true Configuring Logging Parameters in Windows 29. http://technet2.microsoft.com/windowsserver/en/librarylbb9fa8 5 8-a8bb-4 769-93f3-65f2fbc04e3e1033.mspx?mfr=true 30. http://technet2.microsoft.com/willdowsserver/en/ library/62e90 135-7a56-417b-a9S3-SdeeflOSa9a91 033.mspx?mfr=true 31. http://teehllet2.microsoft.com/windowsserver/en/library/2ge9aaso-sb6f- 464e-9b bf-7f50b7034dfsl033.mspx?mfr=true Page 55

32. http://technet2.microsoft.com/windowsserver/en/library!63 efis37-2172-4 996-a94ddb62023194151033.mspx?mfr true 33. http://technet2.microsoft.com/windowsserver/en/library/1 d92 Ofd 7 -aac5-4 2a5 -b92d-2see7a30be911033.mspx?mfr=true 34. http://technet2. microsoft.com/windowsserver/en/library/07b597 d2-0 1 cc-4663 -Sa9 SeSO594S f3 99fI 033.mspx?mfr=true 35.http://technet2.microsoft.com/windowsserver/en/library l d21s4 3 6b-0 3 ds-4bba-ae80-7ed5eeeod6e91033.mspx?mfr=true Retaining Event Logs 36. http://technet2.microsoft.com/windowsserver/en/library/66 f2f79c-9248-44 9a-811c0a34ad6394751033.mspx?mfr=true 37. http:// http://technet2.microsoft.com/windowsserver/en/library /1903d159- c1ee-486e-93 2fcb8d881b94951 033.mspx?mfr=true 38. http:// http://technet2.microsoft.com/windowsserver/en/library/bs837 f6b4-9252 -40f1-895 2-f5b7fc2d90ab1033.mspx?mfr=true 39. http://technet2.microsoft.com/windowsserver/en/library/56d7a732-5730-44 5 2-a701-27Od4db5 6d961033.mspx?mf=true 40. http:// http://technet2.microsoft.com/windowsserver/en/library /a4b48910-7f82-4d50-b537-cc58a3 e2acbc1033.mspx?mfr=true 41. http://technet2.microsoft.com/windowsserver/en/librarv/fb 7892 8d-0329-4 3df-9700-383b7e53c8f11033.mspx?mfr=true Page 56