White Paper Enabling Robust Logging of Web Applications for SIEM and Log Aggregation Solutions
Executive Summary Enterprises face a growing challenge in complying with regulations that require them to continuously track transactions and user activities for long-term archival, analysis, and, forensics. The regulatory environment increasingly requires the collection, storage, maintenance and review of logs; which in parallel is shifting log management from a best practice recommendation to an absolute mandate. In particular, four United States regulations Federal Information Security Management Act (FISMA), Health Insurance Portability and Accountability Act (HIPAA), Gramm-Leach-Bliley Act (GLBA), and the Payment Card Industry Data Security Standard (PCI-DSS) prescribe the collection and/or the analysis of logs. In addition, enterprises have moved beyond using online applications for just e-commerce. Most now offer customers self-service platforms for online banking, online reservations, quote generation, and a variety of other tasks. As more and more enterprises webify their business processes, the increased number and type of web applications that must be logged will further complicate required collection and analysis. Logging of web applications has traditionally been problematic due to the lack of available metrics including the user s identity, the user s in-session activity, and others details that can not be produced by web application logs alone. Additionally, logging performed on production web applications can adversely impact user performance and response time due to the system overhead required to generate the web logs. With the increase in regulatory drivers and the growth of web applications, the generation, collection, and analysis of web applications logs will continue to pose a vexing problem for IT departments. Radware s Inflight solution significantly simplifies the task of collecting logs from web applications. The Inflight solution delivers a unique approach to the capture and generation of meaningful web logs. When combined with the synergy of Security Information and Event Management (SIEM) or Log Aggregation engines, Inflight plays a central role in delivering detailed, actionable data for web applications needed to address compliance and user activity monitoring. Inflight simplifies the collection of web application logs while reducing OPEX and providing immediate ROI. This solution brief describes the challenges of log management in detail and demonstrates how Inflight addresses each one of these challenges. The Challenges of SIEM and Log Management Implementations in a Web Application Environment A recent survey on the Log Management market (The SANS 2007 Log Management Market Report) indicated that 25% of the 653 IT professionals that were surveyed stated that log data collection is their most critical problem when dealing with web applications. The challenges of collecting log data from web applications can be categorized in the following areas: 1. Generating Meaningful Data Logging and Instrumenting Web Applications 2. Latency 3. Impact to performance and availability.
Generating Meaningful Data from Web Applications The first and foremost problem in generating web application logs is to generate meaningful data to represent what occurred in the web transaction. Traditional web application logs from either IIS or Apache web servers lack relevant details for analyzing what a user did in a given online transaction. Currently, IIS and Apache logs provide an endless volume of cryptic data on what request was made or what response the web server delivered to the user. However, mining through the volumes of data to understand information such as who is the user, what was the business logic in the application, and what did the user do in the web application is not currently possible from web logs. The resulting output provides little meaningful context for tracking user activity or business level events that are occurring in the application. Thus, in terms of sending current web logs to a SIEM or Log Aggregation engine, the concept of garbage in, garbage out comes into play as the analytics and correlation can provide little value without the missing contextual information. An alternative approach is to instrument the online application to provide events directly from the applications in the initial stages of the development cycle. The downside to instrumenting applications is that architecture, development, and/or operations teams have to agree upfront as to what events or activity is required from the applications and then instrument the application code to generate this data in a uniform manner across the application. Due to time-to-market constraints placed on architecture and development teams this is often overlooked or prioritized as less important than delivering a production application on time and on budget. As a result, many organizations rely on the web application server logs to provide the basic functionality required to generate logs. Latency Web applications are designed first and foremost to serve a user s request with a response in the most expeditious manner. Web application server resources are thus given a higher priority for processing user requests. By definition, anything outside of this primary mission is secondary. The generation of logs and the resulting collection / aggregation of logs from multiple web servers are given lower priority so as to not impact web server performance. As a result, web application architects and operations design web applications to prioritize the processing of web requests from users and not the creation of web logs, which can have an impact on the server s resources including CPU. There is also an inherent latency involved with web application logs in two primary areas: 1. Creation of web logs on individual web servers 2. Offloading and collecting web logs across multiple servers. The resulting latency may vary from a few seconds to a considerably longer period of time up to days or even weeks depending on the systems resources to not only to generate the logs but also collect or aggregate the logs into a central point for analysis by the SIEM and Log Aggregation solutions. In e- commerce environments with hundreds or thousands of applications across multiple web server farms, this is particularly problematic given the scale and magnitude of aggregating logs, creating a common time stamping mechanism for each of the servers, and extracting the logs to a centralized analytics repository.
Impact to Performance and Availability Industry figures indicate logging on production web applications can consume on average between 10% - 50 % of web application resources depending on the amount and type of data generated in the web logs. The more data that is logged the worse production applications perform. By logging less the architects can reduce overhead on production applications resulting in better user response time and system availability. However, they may not be adequately meeting the needs of their compliance, risk management, or security governance policies. Thus, systems architects instrument the systems to provide an appropriate level of logging to meet the needs of the both the business owners without compromising the needs of either operations, compliance, security or risk management teams that require the log data for security analytics, performance management, or regulatory compliance. The resulting data from the web applications, as optimized for these considerations, is less than adequate for making meaningful analysis of what activity the user did while in the online session. A New Approach Collecting the appropriate web log data needed for proper analysis of user activity and threat management by SIEM and log management solutions is a complex task. While SIEM and log management vendors have tried to ease the log collection process, the growth and importance of web applications continues to present a significant and vexing challenge. A different approach for integrating web application data is required. According to Gartner 1, A solution that is optimal for the current market will support real-time collection and analysis of log data from host systems, security devices and network devices; will support long-term storage and reporting; will not require extensive customization; and will be easy to support and maintain. An alternative paradigm can be considered one that requires looking at the issue from the network perspective. Since access to all resources passes through the network, user activities and transactions can be captured and analyzed at the network layer and consolidated by the SIEM application. The SIEM application, in turn, can analyze the information with more precision and greater accuracy for correlating activity for a user s activity while in the online application. Radware s Inflight Solution Inflight is the only network-based, pervasive real-time log event generation platform that delivers real-time, meaningful events from online applications to any SIEM or log management application, without requiring application modifications to capture the data. As a network-based appliance, Inflight is deployed within the network and passively monitors all web data including the user s request, the servers response, or the entire round trip transaction. Inflight transforms web traffic into a detailed, identity-based transactional event that can be accessed by all types of SIEM applications and log management solutions. 1 Gartner, Inc. Magic Quadrant for Security Information and Event Management, 1Q07, Mark Nicolett, Kelly M. Kavanagh, May 9, 2007.
Inflight is deployed in the production network as a passive utility either off of a span/mirror port or switch or via a passive network tap. By its nature and architecture, Inflight is transparent to applications and users so there is no integration or coding required. This saves enterprises time and money and results in a quick, out-of-path deployment for rapid results and ROI. Inflight employs a Capture, Transformation and Feed (CTF) architecture that is the cornerstone of the web logging solution. Inflight analyzes the data that it captures and creates enhanced logs. As it captures all requests and responses passing through the network, Inflight can generate logs similar to any web server logs (IIS, Apache, and others). It also delivers enhanced information not found in traditional web logs. Inflight s enhanced log data includes: Application User ID/Name, User Session: Adding the actual user ID/Name and/or user session to each log entry enables the analysis system to correlate all log entries that belong to a specific user transaction. Adding information such as user name (or other user related information) enables the analysis system to correctly identify who the user is performing the transaction. Real geographical location of each user accessing the application Page title: URLs tend to be cryptic to the human eye. Adding page title information per each user action enables better understanding of the end user activity. By analyzing web traffic, Inflight can be deployed extensively to replace web application logs. For example, Inflight generates log entries that describe a money transfer. A simple log entry for a money transfer may look like this: [Time date], User X, money transfer, $1000, account X, account Y, success In this log entry example, User X successfully transferred $1000 from account X to account Y. The resulting output from Inflight supports common integration formats for the SIEM and/or log management application to correlate and report on user activity and suspicious or fraudulent activity. Inflight Answers Log Management Challenges Inflight provides the ability to centralize the creation of logs. By capturing all web traffic, Inflight can produce detailed logs on all user activities from all types of web based applications. As a result, Inflight address the three primary problems outlined prior that are encountered when generating log data for web applications: Collecting meaningful data Zero latency No impact to application performance or user experience As a network based solution that observes the bi-directional flow of web application traffic, Inflight is an ideal solution for capturing events, transforming the HTTP/HTTPS traffic into meaningful events, and sending the output directly to a range of analytics solutions for logging and analysis of activity.
Meaningful Data without Latency Given its vantage point in the network Inflight observes the initial login process and authentication while it occurs. Inflight will capture and monitor the entire session from the point the user logs in all the way until the user logs out. Inflight attributes the user s login with each session request or server response eliminating the problem of understanding who the unique user is for the given session. Inflight also provides an embedded geo location engine that can identify the geographical location of each user. After attributing the user s name to the session Inflight observes where the user goes in the application. With this monitoring architecture, Inflight translates the complex click-stream activity into a clear, meaningful output for each user including the user s request parameters and what content or response the web application provided. All of this data is captured in sub-second timing which allows visibility of a user s activity while the user is still in the online session. This real-time, event driven data is ideal for feeding into a SIEM or logging engine while the user is in-session. Performance and Availability Gains As stated, enterprise, application and systems architects continually seek an optimal balance between the business need for maximizing systems performance/availability and the operational/compliance considerations for logging. They are bound by the constraints and burdens that log generation places on the production web application environment. Inflight eliminates the conflict for achieving a balance between performance optimization and logging by moving the collection and generation of logs from the host systems to an out-of-band, network-based utility. As a passive architecture deployed in the network, Inflight offloads the collection and generation of logs for web applications as a utility in the network. This network-based approach delivers three discrete benefits to production web application performance and availability: 1. No impact to production web application performance 2. No impact to user performance and response time 3. No degradation to the production network when collecting real-time logs These advantages are possible by monitoring and capturing user activity at the network layer as an outof-band (passive) solution. Inflight does not impact or degrade system resources on the production host in order to capture, filter, and generate logs for the web applications. These benefits, when coupled with the enhanced data (user ID, geo-location, and page information) that Inflight yields, collectively provide system and application architects with a new design option to consider: disable logging on the production applications and use the network as a utility. Integration with SIEM and Log Management Applications Inflight makes integration with SIEM and log management applications an easy and straight forward task. Inflight can replace the two most common working methods collecting logs by installing agents and collecting logs by pushing/polling the logs from/by the hosts. Inflight s built-in technology enables it to send a log entry as it is being created to the SIEM or log management system. This behavior simulates the behavior of an agent. Inflight supports a variety of transportation protocols including SysLog, TCP, HTTP, Web Services, JMS, JDBC and SMB to enable integration with the SIEM/log management application. In addition, Inflight supports all types of message formats including comma separated values (CSV), tab separated values, XML and plain text.
As a log generator Inflight can simulate an agent-less use case as well. Inflight technology stores log entries locally or places the logs directly on a shared network drive, storage area network (SAN) or a specialized log archival appliances or servers. These capabilities enable Inflight to feed all types of SIEM and log management applications. Summary Inflight enables IT organizations to address the challenges typically associated with log management. Logs generated by Inflight contain data that is not found in typical web server logs and the logs Inflight produces can replace most application logs. Since those application logs are generated in one centralized place there is no need to understand each application log format or to add more application logs in case information is missing from the original application log. Inflight, which is therefore easy to support and maintain, enables IT organizations to rapidly and easily generate log data, in real-time without the need for extensive customization. With Inflight, IT organizations gain the following benefits: Offloading logging tasks from the different hosts the host resources are 100% dedicated to serving the end user Only one log source no need to manage multiple log sources Consistency between all log content, format and timestamps Central protection of log files Clear, detailed activity from web applications to determine the who, what and when details required for a more holistic SIEM approach to determining fraud and suspicious activity in real-time No need to install agents or provide authentication credentials Rapid deployment and ROI of SIEM, log management, fraud or other security solutions For more information about Inflight or to download free evaluation software, please visit: www.radware.com/inflight 2010 Radware, Ltd. All Rights Reserved. Radware and all other Radware product and service names are registered trademarks or trademarks of Radware in the U.S. and other countries. All other trademarks and names are the property of their respective owners.