Spotlight. Log and Event Management

Transcription

1 Spotlight Log and Event Management A Spotlight Paper by Bloor Research Author : Philip Howard Publish date : December 2009

2 It makes sense to treat event management and log management as two sides of the same coin, with both being part of a single, coherent environment that will support not just compliance, but also real-time security, forensic and operational functions Philip Howard

3 Executive summary Log and event management has been variously referred to as business intelligence for IT and clarity for IT and has been described as what business intelligence is to the CFO, log and event management is for the CIO. We would actually go further than that because log and event management is also business intelligence for security and compliance officers as well as, in some cases (depending on your organisational structure), for risk managers. Regardless of who it is for, log and event management provides business intelligence against your infrastructure. That is, it tells you who did what and when. Just like business intelligence, this may either be in hindsight for forensic purposes or in real-time. In the first case you might use this for a variety of purposes, such as fraud discovery or to detect misuse of company time (accessing social networking sites during working hours, for example), or to monitor compliance with relevant regulations. In the second case you might want to prevent malicious activity or to instigate action if external or internal threats (such as hacking) are detected. You can also use log and event management for operational purposes, in conjunction with network performance optimisation, for example. Basically, the point is that you cannot control your infrastructure if you can t peek inside it to see what is happening. In other words, log and event management is about visibility and control. It is also worth considering the role of log and event management in a world in which a lot of IT resource is outsourced or hosted by third parties. As a customer of one vendor in this area put it, you can outsource responsibility but you cannot outsource accountability. Thus there is a continuing role for log and event management even in such environments. Indeed, we know a number of companies in which log and event management is the only IT function that remains in-house, for precisely this reason. While log and event management is typically concerned with the IT infrastructure it doesn t have to be limited to that area. For example, it is becoming increasingly common for companies to link physical security devices into their log and event management infrastructure, and there is also scope for encompassing such things as sensors, especially where environmental monitoring is required for compliance purposes. The offerings that address these requirements are typically referred to as SIEM (security information and event management) products. We do not like this term as it suggests that log and event management is only about security, whereas even this brief introduction should have made it clear that this is not the case. Thus we prefer the more descriptive log and event management. In this paper we will discuss what events and logs are, when you should care about them, why you should care about them, and the sort of features (at a high level) that you should be looking for in potential solutions. We will also briefly discuss the state of the market for log and event management platforms, as this is somewhat confused at present. Note that we use the word platforms advisedly: our view is that you should be able to collect and analyse all relevant data once, in a single place, which will support the (re)viewing of this information multiple times across the organisation, depending on the use case in question, as opposed to the siloed, separated, piecemeal approach that characterises the approach of many organisations to log and event management, resulting in duplicated technology and effort. We will discuss this further as we proceed. A Bloor Spotlight Paper Bloor Research

4 What are events and logs? A log is a record of events. An event is, literally, something that happens. In business terms it is something that potentially acts as a trigger for something else, such as a process or alert. For example, a stock tick may trigger a buy or sell, an order may trigger a stock reorder process and a customer going over his or her credit limit may trigger an irritating letter (amongst other things). These are what we might define as business events and they are not the subject of this paper. However, there is a second class of events that we can describe as infrastructure events. Examples would include logging into a database, sending an (not the content of the , the act of sending it), visiting a web site, someone making repeated attempts to access data with an incorrect password, details of router activity (packets sent and so on) and so forth. While we will discuss why you might want to record and manage these events in detail in due course, suffice it to say for the moment that it is important for compliance and security reasons in particular as well as for operational purposes. In addition, there is what we might think of as hybrid events that fall somewhere between infrastructure and business events. For example, RFID readers are certainly part of the infrastructure but their readings will normally be about business events. The same applies to automated number plate recognition systems; monitoring equipment of various types on the shop-floor, for environmental purposes and elsewhere; GPS data and, indeed, pretty much any type of sensor reading. When should these types of events be treated as infrastructure events and when as purely business events that are outside the scope of this paper? Again, we will discuss this further but the short answer is that if an event raises compliance or security issues in real-time or if the historical log of those events is required for forensic or evidentiary purposes then we will consider that to be an infrastructure event, otherwise we won t. Note that an event may be both a business event and an infrastructure event. This not only applies to hybrid events but also to things like call detail records (CDRs). Storing these is mandated in the EU by the Data Retention Directive and they are treated for this purpose as infrastructure events. On the other hand, the same information forms the basis of business functions such as billing and traffic analysis, which is used for capacity planning, in which case they would be classed as business events. In general it is good practice to treat hybrid events as separate business and infrastructure events. Often it is the case that some of the data is only relevant in a business setting while other data is only relevant from an infrastructure point of view. Moreover, there are, potentially, evidentiary and legal considerations that apply to infrastructure events that do not apply to business events, so it is common to adopt different processing and storage strategies depending on whether the data is viewed from an infrastructure or business perspective. So, we are clear about the types of events, and the logs of those events, that we are interested in. In this paper we will consider the sorts of functionality that you need to manage these events and logs and, along the way, we will discuss the benefits that can accrue from taking a formalised approach to log and event management Bloor Research A Bloor Spotlight Paper

5 Immediate threats In the first instance, infrastructure events are used to recognise security threats (whether internal or external), such as hacking attempts, cyber attacks and so forth, in real-time, and to support incident reporting and escalation. What you want in these sorts of situations is to recognise the attack as early as possible, flag it to the appropriate manager and activate your incident response processes, aimed at stopping the attack on the one hand and identifying the culprit on the other. The difficulty is that there are a great many events to monitor across a very broad spectrum of hardware and software. Moreover, the vast majority of events that are generated within your infrastructure will be of no particular interest. You therefore need facilities that will recognise anomalous behaviour and that will filter out all events that reflect normal activity. On the other hand, just because there is unusual behaviour in one particular place does not necessarily mean that you are under attack. It may simply mean that a router (say) has gone down, which may be of interest for operational and network management purposes, but which does not reflect a threat. You therefore need more than simply the ability to spot unexpected occurrences, but also the additional facility to prioritise particular events, to recognise correlated events or to detect patterns of suspicious activity. Note that this also applies to real-time fraud detection as well as attacks per se. As we have said, there are a very large number of sources from which you might want to collect event data and logs. You will therefore need an environment that supports a variety of data collection capabilities. There are two issues here: how you collect the data and where you collect it from. To the first point, the process of collecting relevant data should place as little strain as possible, and preferably none at all, on the originating systems. Fundamentally, there are two options: An agentless approach, whereby event data is collected without requiring any software to be installed on the source system. Other issues being equal this non-intrusive approach is to be preferred because it means that there is a reduced (or no) overhead imposed on the source, precisely because it is non-intrusive. Indeed, some companies will forbid the installation of any code on critical servers, thereby requiring an agentless approach. An agent-based approach. In some instances it is not possible to use an agentless approach but agents should be as lean and mean as possible. One advantage of using agents is that you can encrypt the data at source. You can also, potentially, garner additional information when using agents as opposed to not using them. Actually, there are two types of agent-based approach. In the first case, the source creates an event log and the agent is used to forward (and possibly encrypt) the details. The second case is more interesting. Here, the agent actually creates the log data. That is, it instruments the source with a sensor or monitor of some sort. The classic example of this approach is that database management systems typically have their own logging capabilities, but they are very inefficient and are invariably turned off because of their impact on performance. Agents that monitor the database (often sitting on the network) have little or no performance implications and have the further advantage that you can monitor things like stored procedures that are trying to access sensitive data, as well as people trying to do so. Another advantage is that the database administrator cannot turn the logging off, which means that you have a clear segregation of duties. While this sort of approach is most widely used for monitoring databases at present, we expect it to grow in use for monitoring application software such as ERP applications; not so much because of performance implications but because of the segregation of duties and the ability to monitor non-human requests for information from other applications and web services. Having said all this, we should point out that it is not the primary role of log and event management to create logs or trace events but rather to create a metatrace. In other words, as a general principle it is better to leave event capture to relevant devices or applications: for example, anti-virus software is probably the best place to generate event data about viruses, intrusion detection systems about intrusions and so on. However, as we have seen, not all sources are adequate for this purpose. In so far as what event data should be collected. the short answer is everything: ranging from hardware devices; through operating systems, databases, and message buses; to security software such as firewalls, spyware and so on; and to end-user software including A Bloor Spotlight Paper Bloor Research

6 Immediate threats ERP, CRM and similar applications. We will not list the protocols that will need to be, or might be, supported because there is a huge list of them, but standards should be adhered to wherever possible and, where nothing is available, then it will be useful to have some sort of parser that can put the data into a standardised format. In addition, you may want to bear in mind that a really serious hacker will probably start by turning off your event logging. Of course, you should be alerted to the fact that events are not being received, which is anomalous in itself, but that won t help in finding out what is happening. You therefore may want to consider monitoring other types of information such as configuration data about devices (firewalls, servers, routers and so forth), asset data, performance data (disk utilisation, for example), network flow data and known vulnerability information. Such facilities will also assist in using the software for other purposes such as network performance management. You may even wish to extend the environment to include physical security devices so that, for example, if an employee only has permission to log into your database while he is on your premises, then you can be alerted if he tries to log on after he has left the building. In a similar vein, monitoring of USB and similar devices is important to protect against data theft and it will be advisable if security is kept up to date by the human resources department with information about employees who may have been made redundant, who have handed in their notice to go and work for a competitor, or are otherwise disgruntled, as this can lead to malicious damage or theft Bloor Research A Bloor Spotlight Paper

7 Storing events While threat monitoring is a significant requirement, a recent study by a major vendor found that 70% of companies implementing event and log management solutions were doing so either for compliance reasons or because an auditor had appended a note to their accounts. Only 30% had internal or external threats as their primary consideration. We can therefore reasonably assume that everybody will want to support compliance even if they don t need facilities to cater for real-time threats. This will require storing all of the events that have occurred as logs. Note that while you have filtered out events that were not of interest for real-time monitoring purposes, for compliance and log management all events need to be stored. This raises a number of issues. The first is that there are potentially a great many events to record so you need a storage mechanism that is highly compressible so that you do not go overboard on storage capacity. Secondly, depending on the environment, you may have extremely high ingestion requirements. Further, it is not simply a question of loading the data: you will also need to create indexes (at least in some environments; column-based storage methodologies may not require the use of indexes) against the loaded data for subsequent search or query purposes, and with extreme load rates you will not have the luxury of doing this after the fact but you will have to do this as the data is loaded. Other environments may not be so onerous of course, but loading and indexing are factors that must be considered. The next point is that log detail may subsequently be used in court proceedings, as evidence of fraud, for example. You therefore need to be able to prove that the original data could not have been tampered with, in order to ensure its evidentiary weight. This may mean writing the log data to write-once read many (WORM) media or it may mean using other forms of tamper proofing technology. Finally, you also need to have lifecycle management in place. For purely compliance purposes, you may need to store log data for a specified period of time but once that period has expired you will want to have softwarebased policies in place that automatically delete that data. However, for other uses, such as fraud investigation, you may decide that, rather than deleting log data, you wish to archive it, in which case you will need relevant policies that support that. A Bloor Spotlight Paper Bloor Research

8 Formatting log data Log data is notoriously opaque. That is, it is extremely difficult to read or make sense of, unless you have specialist knowledge. In addition, there is no standard log format. Even two different databases may use very different formats for presenting event information, let alone the difference between a router and an ERP application. This is one major reason why monitoring log data separately for separate sources does not make sense: you require multiple sets of expertise to interpret what is happening, and even then it is fiendishly difficult. A much more sensible solution is to have a single repository for all log and event data and to present information in a consistent way, based on standardisation within the software, which can be used by whatever department requires that information for whatever purposes are necessary. Moreover, at the same time, you need to be able to convert the log data into a user-friendly format, so that the data is more easily readable. Different approaches to this are available, with some companies storing a separate copy of the data in user-readable, standardised format, which is what you manipulate when you are running queries (thereby leaving the tamperproofed original untouched); or, some suppliers convert the data on the fly as it is read from tamper-proof storage. The problem with the latter approach is that it means that identifying correlations becomes difficult as this requires normalisation and categorisation in order to support real-time identification of correlations. Thus for handling immediate threats you really need to take the former approach. However, if you are only interested in log management then which method you prefer will largely be a question of price and performance. In either case this is another strong argument in favour of taking a platform-based approach to log and event management because it means having the data presented not just in a user-friendly format but also in one that is consistent across application requirements Bloor Research A Bloor Spotlight Paper

9 Compliance As we have already indicated, the primary use of log management is to support compliance. However, there are three relevant types of compliance. The first is where you are mandated to store event or log data for a specified period of time. For example, the UK government requires, in its GCSx regulation, that parties accessing the government secure gateway keep relevant log data for a period of six months. A second form of compliance is to support corporate governance policies such as a rule forbidding staff to access social media sites during working hours: you can monitor and log this sort of activity using the sorts of facilities under discussion. Thirdly, there are other government or industry requirements that you may be required to enforce and monitor. These include, but are not limited to: Sarbanes-Oxley (and associated regulations such J-SOX). PCI-DSS, for retailers taking credit card payments across the Internet. HIPAA, for electronic health care transactions in the United States. Whether compliance is to meet an internal governance standard or an external regulation, the key requirement is to capture all of the relevant event and log data and then display that in a dashboard, perhaps raising an alert where that is appropriate. Many of the providers of solutions in this area have pre-built monitoring and dashboards for many of these compliance requirements but the vendor should also provide the facility to customise these as necessary, as well as the ability to create your own dashboards where you have requirements that are not supported out-of-the-box. It is also worth bearing in mind that some regulations (such as PCI-DSS) require you to actually look at the data as well as merely storing it. In this case, PCI specifically mandates a daily review of log data and, of course, you would want your software to monitor the fact that you are doing so. Finally, note that simply being compliant with any particular security regulation or framework does not mean that your environment is secure it just means you won t get fined for failing to comply. Compliance should really be a starting point for ensuring your corporate security rather than an end goal in itself. COBIT and ITIL, which are IT governance standards. ISO/IEC 27002, which is an IT security standard. NIST , which is a US government standard for federal security. NERC, which is a US standard for electricity providers. FISMA, the federal information security management act. Data protection legislation of various sorts. A Bloor Spotlight Paper Bloor Research

10 Querying the data While there are exceptions (see later), in most cases it is conventional reporting, query and analytic capabilities that will be required. Most suppliers provide a wide range of pre-built reports and queries that are provided as standard, as well as a range of analytic capability to support functions like root cause analysis. However, it is not always the case that SQL access is provided. This means that you are limited to the facilities that the vendor provides which, while it may be very comprehensive, will always be limiting to some degree. For example, when you are looking for patterns in the data, most providers will offer facilities to determine correlations, but you may want to go deeper than this. You may, in effect, want the sort of facilities that you would get from a third party data mining tool. However, such tools typically require SQL access to the data. In addition, if you do discover patterns of activity using such tools, then you may want to embed these models within your environment for things like real-time fraud detection. In this case, it would be useful to support a standard such as PMML (predictive modelling mark-up language) that will allow you to import these models (patterns) directly from a data mining tool and into your event monitoring environment so that you can automatically check events against these patterns in order to discover suspicious activity. In general the issues to be considered will be the extent of the out-of-the box capabilities provided, the degree to which these can be customised, support for SQL, and the provision of web services or REST (representational state transfer) APIs. These APIs (application programming interfaces) will allow you to extract log data and use this to create new dashboards, portals, mash-ups and so on. This is important because it reflects the multiple use cases for log and event management that typically exist within an organisation, allowing the reuse of the data in appropriate ways for different purposes, as previously discussed. It will be useful if there are developer tools provided in conjunction with these APIs and, even better, if there is also a developer community that can share experience and functions through a relevant portal Bloor Research A Bloor Spotlight Paper

11 The market Having described the basic features you will require from an event and log management solution it is pertinent to discuss the market for such products, since it is by no means homogeneous. As we have already noted, we do not agree with the terminology currently in use in this sector, as there is a direct implication that log and event management is just about security, which it is not. However, in this section we will employ the commonly used acronyms that, in our view, currently divide (rather than synthesise) the market. There are a number of different product categories that address some or all of the requirements discussed in this paper (and there are two more in the next section). These include: Log management: the storage of log data for compliance and analytic purposes but excluding real-time alerting (immediate threats). Log management may also be used for operational purposes such as identifying an (IT) network bottleneck. Security information management (SIM): focused on real-time tracking and alerting of security incidents such as attacks against your firewall. Security information and event management (SIEM): has largely superseded SIM because it also incorporates long-term storage of security event data (logs) for analysis. In theory, log management is distinct from SIEM but in practice most vendors treat SIEM as the combination of SIM with log management. Some vendors have separate products for the two areas while others offer a single product spanning both spaces. Database intrusion detection (DBID): also known as database activity monitoring (DAM). This does what it says on the tin but also monitors any unusual (but authorised) activity so that alerts can be raised. SIM products, in particular, tend to be focused on external attacks but DBID and SIEM are as much focused on insider threats. A number of vendors in the SIEM space also include DBID offerings. Data loss/leak prevention (DLP): this covers a similar requirement to DBID but focuses on data in motion rather than data at rest: monitoring and raising appropriate alerts (or running processes to stop unauthorised activity such as loading data onto a USB stick) when appropriate. It should be clear that there is considerable overlap across these areas. However, we should point out there is a divergence of opinion amongst vendors as to the importance of the overlaps between DBID and DLP on the one hand and the various log and event management technologies on the other. Some suppliers take the position that DBID and DLP solutions are simply providers of information to log and event management solutions. While it is certainly possible to build a coherent solution based on this approach we believe that the market will move away from this position because of the functional overlaps involved. It is our view that the market for log and event management currently parallels that of the business environment prior to the introduction of ERP applications, wherein there were lots of siloed and point solutions that did not work together. While a common platform for supporting these tasks could emerge from anywhere it seems most likely to come from amongst the SIEM vendors. There are encouraging signs that this is the direction that the market is moving in (integration of SIEM with DBID/DAM, for example, because you want to monitor your database in context to other activity rather than in isolation) but there is still some way to go before a complete, integrated platform is available from any single vendor, covering all of the areas outlined. In our view, this must be the direction that the market moves in. Note that most vendors in the SIEM market offer hosted services as an alternative to a software-based solution in case you prefer that option. If you do wish to have your own inhouse solution (and many security officers will prefer that) then many suppliers offer an appliance-based approach in order to minimise implementation and administration. A Bloor Spotlight Paper Bloor Research

12 Complementary markets There are two markets that could or should form part of the overall picture for log and event management, which we will discuss in turn. Data retention We have already mentioned the EU Data Retention Directive, which requires CSPs (communications service providers) to store CDRs and IPDRs (call and IP detail records respectively) for a period of up to two years (depending on the country). In addition, there is a requirement to provide search (as opposed to query and analytic) capabilities either by request from a law enforcement agency or via self-service capabilities. In addition, there may be a requirement to support lawful intercept capabilities. Countries outside the EU are rapidly introducing similar regulations. As a CDR is record of an event one might reasonably conclude that log and event management products could easily meet the requirements of the EU directive. However, there are differences. For example, a CDR is, in effect, a single event rather than a log. When you query the system you are looking for individual events (who called whom from this number?) rather than looking for patterns or correlations. Hence there is a need for search capabilities rather than business intelligence. Secondly, there is a scalability issue. Ingesting CDRs at a large telco can run into billions of records per day. With IPDRs ingestion requirements can be into the tens of billions of records per day. Few log and event management products have been designed to cater for such volumes, though there are, of course, smaller CSPs where this will be less of an issue. Thus, in theory, the data retention market is a subset of the event and log management sector, and if you ask any supplier in that space if they can meet data retention requirements they will say yes. In practice, however, only some vendors have the necessary specialised facilities, including high ingestion rates and a search interface. Sensor-based infrastructure events There are a number of sensor-based environments such as manufacturing, utilities, logistics and so on, where there are many potential hybrid events of interest. For instance, if you are collecting environmental monitoring data from various sensors then there is likely to be both a real-time alerting requirement and a compliance element to that task. Another example is SCADA (supervisory control and data acquisition) systems, where there are major security issues involved. At present the event and log management vendors have not laid claim to this market. On the other hand some complex event processing (CEP) vendors have done so. While CEP is primarily concerned with business events there is an overlap when it comes to hybrid events that fall into the infrastructure category. In practice, most CEP vendors focus on capital markets and algorithmic trading and, while there are compliance issues that have to be monitored in a manner that is similar to the compliance monitoring provided by log and event management, this is not typically the raison d être behind the use of CEP engines. That said, there are a few CEP vendors that focus on sensor-based environments. To date, neither the CEP vendors nor the log and event management suppliers have laid down a clear stake for the ownership of this sensor-based hybrid space and it represents a potential market for log and event management vendors in the future Bloor Research A Bloor Spotlight Paper

13 Conclusion If you don t monitor events then you can t react when they happen. You want to be able to take urgent action when something of interest occurs, whether that it is to fire up an application or process, raise an alert, send an , or whatever. If you don t log events then you can t discover or prove what happened after the fact. Moreover, if you don t have a comprehensive view of information then you cannot easily discover what has happened or do root cause analysis. If you have isolated pockets of log data, which are incomplete and in different formats, then the process of examining the data will be well nigh impossible: you need a central repository of all log data that can be presented to you in a consistent and user-friendly manner. You may want to examine log data to determine if, say, fraud has occurred, or perhaps to determine where there are bottlenecks in a network. To do this, you require a holistic view of the data. Also, if you do detect fraud, then you may need to be able to prove that in court and the log details (assuming they have been tamper-proofed) provide the relevant evidence. Then there is compliance. Proving that you are compliant with relevant regulations is as important as the act of compliance itself, and you can do this by using log-based systems, since these provide the audit trail of your relevant activities. To sum up, it makes sense to treat event management and log management as two sides of the same coin, with both being part of a single, coherent environment that will support not just compliance, but also real-time security, forensic and operational functions. Further Information Further information about this subject is available from A Bloor Spotlight Paper Bloor Research

14 Bloor Research overview Bloor Research is one of Europe s leading IT research, analysis and consultancy organisations. We explain how to bring greater Agility to corporate IT systems through the effective governance, management and leverage of Information. We have built a reputation for telling the right story with independent, intelligent, well-articulated communications content and publications on all aspects of the ICT industry. We believe the objective of telling the right story is to: Describe the technology in context to its business value and the other systems and processes it interacts with. Understand how new and innovative technologies fit in with existing ICT investments. Look at the whole market and explain all the solutions available and how they can be more effectively evaluated. Filter noise and make it easier to find the additional information or news that supports both investment and implementation. Ensure all our content is available through the most appropriate channel. Founded in 1989, we have spent over two decades distributing research and analysis to IT user and vendor organisations throughout the world via online subscriptions, tailored research services, events and consultancy projects. We are committed to turning our knowledge into business value for you. About the author Philip Howard Research Director - Data Philip started in the computer industry way back in 1973 and has variously worked as a systems analyst, programmer and salesperson, as well as in marketing and product management, for a variety of companies including GEC Marconi, GPT, Philips Data Systems, Raytheon and NCR. After a quarter of a century of not being his own boss Philip set up what is now P3ST (Wordsmiths) Ltd in 1992 and his first client was Bloor Research (then ButlerBloor), with Philip working for the company as an associate analyst. His relationship with Bloor Research has continued since that time and he is now Research Director. His practice area encompasses anything to do with data and content and he has five further analysts working with him in this area. While maintaining an overview of the whole space Philip himself specialises in databases, data management, data integration, data quality, data federation, master data management, data governance and data warehousing. He also has an interest in event stream/complex event processing. In addition to the numerous reports Philip has written on behalf of Bloor Research, Philip also contributes regularly to and Analysis.com and was previously the editor of both Application Development News and Operating System News on behalf of Cambridge Market Intelligence (CMI). He has also contributed to various magazines and published a number of reports published by companies such as CMI and The Financial Times. Away from work, Philip s primary leisure activities are canal boats, skiing, playing Bridge (at which he is a Life Master) and walking the dog.

15 Copyright & disclaimer This document is copyright 2009 Bloor Research. No part of this publication may be reproduced by any method whatsoever without the prior consent of Bloor Research. Due to the nature of this material, numerous hardware and software products have been mentioned by name. In the majority, if not all, of the cases, these product names are claimed as trademarks by the companies that manufacture the products. It is not Bloor Research s intent to claim these names or trademarks as our own. Likewise, company logos, graphics or screen shots have been reproduced with the consent of the owner and are subject to that owner s copyright. Whilst every care has been taken in the preparation of this document to ensure that the information is correct, the publishers cannot accept responsibility for any errors or omissions.

16 2nd Floor, St John Street LONDON, EC1V 4PY, United Kingdom Tel: +44 (0) Fax: +44 (0) Web: