Beyond Web Application Log Analysis using Apache TM Hadoop. A Whitepaper by Orzota, Inc.

Transcription

1 Beyond Web Application Log Analysis using Apache TM Hadoop A Whitepaper by Orzota, Inc. 1

2 Web Applications As more and more software moves to a Software as a Service (SaaS) model, the web application has become ubiquitous. Figure 1 below shows the architecture of a typical web application. The simplest applications have the web and application tiers combined while more complex ones may have multiple web tiers to handle static content and security, multiple application tiers (called middleware ) as well as one or more data bases. Database Server Web Browser Request Web Servers Application Servers External Services Static Files Logs File System Figure 1: The Typical Web Application Web Server Logs Web servers typically provide at least two types of log files: access log and error log. The access log records all requests that were made of this server including the client ip address, URL, response code, response size, etc. The error log records all requests that failed and the reason for the failure as recorded by the application. A large site that receives thousands to millions of requests per day may have 100s of web servers, each server logging requests it processes. This data can easily range in the GBs for a day. Application Server Logs Business logic in web applications is handled by the application server tier. In some cases, the business logic is executed as part of the web server tier itself and in more complex applications, there may be several different types of application servers that handle various pieces of the logic. In all cases, application servers also do their own logging. The custom logs used can provide a great level of detail for 2

3 application developers and analysts to understand how the application is used. Since developers can log arbitrary information, application server logs can be even larger than web server logs. Log Processing With logs that are being continuously produced in various tiers with different types of information, the challenge then is to store and process this data in an efficient manner to produce rich insights into the application and customer behavior. Although one might be tempted to throw all data into a relational database system, this is usually not the correct architecture. RDBMS systems can be very expensive and cheaper alternatives like MySQL cannot scale to the volume of data that is continuously being added. A better technology exists to store terabytes of log data and process it efficiently. This is Apache Hadoop. Hadoop Overview Apache Hadoop is an open source project that provides a parallel storage and processing framework that enables customized analytical functions using commodity hardware. It scales out to clusters spanning tens to thousands of server nodes making it possible to process very large amounts of data at a fraction of the cost of enterprise data warehouses. The key is the use of commodity servers Hadoop makes this possible by providing for replication and distribution of the nodes over multiple nodes, racks and even data centers. Hadoop is rapidly being adopted by enterprises from leading banks and retailers to online travel and ecommerce sites in a wide range of applications. Hadoop for Log Processing One of the first applications developed using Hadoop was web log processing. Processing the large volume of logs for web companies such as Yahoo! and Facebook was a challenge. Add to this the textual nature of the content in these log files and Hadoop became the platform of choice for log analysis. Figure 2. shows the architecture of a log processing system using Hadoop. Log files from many different types of servers are fetched via Apache Flume and loaded into a Hadoop cluster. Jobs are scheduled to analyze the logs and generate aggregated summary metrics which are then sent to an external RDBMS for visualization using a BI tool. The data in Hadoop is also available for ad-hoc queries via Hive by developers, product managers, etc. for exploratory analytics. 3

4 Data Sources Logs Web Logs Firewall Logs Server Logs FLUME Collection & Analytics HDFS MapRedu ce Reporting HIVE Schedule r MySQL Big Data Analytics The power of this architecture lies in the fact that we can combine data from arbitrary sources in Hadoop. By applying the correct filtering and analytical processes, new questions can be answered. Some examples are: Ranked list of the most significant trends and page views over the last day/week/month 4

5 Identify the characteristics of the most active users to help promote this behavior across the board Co-relate user sessions with transactional data to better understand how to improve sales Log Analysis Services If you have log files that are growing by the day and are not being analyzed effectively to add business value, you are missing out. Understanding customer behavior can help in defining and optimizing functionality to improve ease of use and performance. Gaining operational insights from system logs can greatly improve the efficiency of your production systems. Orzota can help implement the right architecture and automate log analysis to provide meaningful insights. For more information, contact us at 5