Journal of Electrical Engineering 2 (2014) 29-33 D DAVID PUBLISHING The Automatic HTTP Requests Logging and Replaying Subsystem for CMS Plone Maksym Oleksiv and Taras Poburynnyi Computer Engineering Department, Lviv Polytechnic National University, Lviv 79013, Ukraine Received: November 07, 2013 / Accepted: December 07, 2013 / Published: January 25, 2014. Abstract: In the paper, the problem of storing and replaying of HTTP requests to websites in order to improve their debugging efficiency during development and further support is considered. To solve this problem, the automatic HTTP requests logging and replaying subsystem Dagger that provides storage and playback of the requests sent to a website, which is developed by CMS (content management system) Plone means, is offered. The subsystem consists of the three components: Detector, Logger and Player. Detector detects filters user s requests, creates a log and sends it to Logger. Logger is a daemon program that stores logs of user s work with a website. The logs are stored in the files in one of the supported formats: JSON, CSV, PICKLE. Player replays the GET and POST HTTP requests that users sent to a server. Key words: CMS, plone, logging, HTTP requests. 1. Introduction During websites of any complexity development, the problem of debugging and errors detection during their functioning raises. One of the examples of such errors is errors that occur in a particular type of request. For example, when a server receives some HTTP header (i.e., User-Agent), it generates an error, but when you use other browser, everything works correctly. Another example is the errors caused by a not valid code that a server contains. All web systems have these problems. Nowadays there are several methods that in most cases can be reduced to the following sequence of actions that are performed manually to solve these problems [1]: A program s calls stack and HTTP request are analyzed; Attempt to recreate the request using web interface is performed; If request, that caused an error, recreation is Corresponding author: Maksym Oleksiv, Ph.D., research fields: signal and image processing, heterogeneous computing, embedded systems, web technologies and software engineering. E-mail: maxoleksiv@gmail.com. unsuccessful, then a variety of utilities, such as curl utility [2], are used, but they do not always help to identify the cause of the error; An analysis of the error causes and its solving are performed. The disadvantage of these methods is that software developers need to spend a lot of time to analyze a bug and to determine and playback the full sequence of user actions that led to it and only then the developer can take steps to fix it. But often the developers cannot determine the cause of the error. If we look a little bit to software means that are available for websites developers we can mention about Sentry [3]. Sentry is a software subsystem, which stores errors and shows their details in variety of modes. But a great problem of Sentry, as of the others subsystems of this class is that they cannot replay errors. So a software subsystem, which automates the process of storing and replaying a sequence of user s actions during his or her work with the website, development problem for automation of the errors
30 The Automatic HTTP Requests Logging and Replaying Subsystem for CMS Plone detection during websites functioning arises. Solving this problem will speed up and facilitate the detection and fixing of errors that occur during websites debugging and functioning, as well as the probability of successful errors detection will be increased. This paper considers the automatic HTTP requests logging and replaying subsystem called Dagger whose main task is to ensure storing and correct playback of user s requests to the web server during work with a website that is designed by means of CMS Plone. Plone is a CMS (content management system), which is based on Zope platform [4] that was written using the Python programming language. It is used to deploy internal websites, but can be used for deploying web servers in the internet, which will serve as documents publication systems and tools for employees interaction. Examples of websites designed using Plone are: the NASA s website (http://science.nasa.gov); the Brasil government s website (http://www.brasil.gov.br); others. Plone can be considered as a set of standard packages (eggs) that are overbuilt on the Zope application server, which uses the Python programming language (Fig. 1) [5]. There is an opportunity to develop, install, and configure own packages using Buildout framework in addition to the standard packages. Dagger logging subsystem consists of such packages and connects to CMS Plone using this framework. Fig. 1 CMS Plone structure. Use of the subsystem will simplify the reproduction of errors which happened on the website, and provide the ability to quickly and correctly fix them. 2. The Structure of the Subsystem Dagger The automatic HTTP requests logging and replaying subsystem Dagger consists of the three components: Detector, is installed on the client s side and detects requests that are sent to the server. Its main task is to detect, transform and send a request to the repository, which is implemented by component Logger. Logger, provides storage for information that came to it from Detector. This component performs buffered data accumulation in RAM, and then writes it in the file system in a one write cycle in one of the formats: JSON, CSV or PICKLE. In addition, it logs errors that occurred during the website functioning. In order to be able to gather information from a multiple processes, it is designed as a daemon. Player is installed on the client s side and provides playback of stored GET and POST HTTP requests that were received by the server. Requests replaying are performed on the web server s copy to avoid the possibility of data corruption. The structure of the subsystem is shown on Fig. 2. Detector consists of: an active profile a profile that allows the CMS Plone system to identify a component; socket utilities, which provide a memoization and data transfer; protocol utilities that are used for requests filtering; a test profile that is intended for configuring a test environment, event handlers that perform user s requests interception. Logger consists of: a data repository that provides the user s requests storage; the component s settings which are controlled by developer during a recipe creation; a daemon program that provides the ability to execute component as a resident program in a Unix-like system and the ability to accept data transmitting through a socket; the recipe creation mechanism that allows to generate executable file according to the desired settings.
The Automatic HTTP Requests Logging and Replaying Subsystem for CMS Plone 31 Fig. 2 Structure of the automatic HTTP requests logging and replaying subsystem Dagger. Player consists of: an active profile a profile that allows the CMS Plone system to identify a component; a test profile that is intended for configuring a test environment; web-based interface that provides the user interface for the requests reproduction; the requests replayer, which provides the replaying engine for users requests, which have been received by the server. 3. The Features of Dagger and Its Components Implementation Dagger is implemented using three components (Detector, Logger and Player), which being executed together performs the following functions: User s request identification the subsystem detects all user s requests that were sent to the server, and transmits them into the repository; Data transfers into the repository are performed using optimization, which speeds up the program s performance; The request that led to the error identification; Requests are stored in files in one of the formats: JSON, CSV or PICKLE for further processing; User s requests playback; Results of requests playback execution presentation in a convenient form for further analysis; The ability for the users to create requests through a web interface manually. Detector is presented by Python s package collective.error.detector (Fig. 3). It is installed on the client s side and is designed to identify requests that were sent to the web server. The component is designed so that it provides high performance while using low computational resources. The component was implemented using: Python/Plone/Zope packages, ZCML (Zope configuration markup language), XML, and Buildout framework. Logger is represented by recipe collective.recipe.logger (Fig. 4). Recipe is used as a part of Buildout framework. When Buildout starts, it generates a script logger, which is a software daemon. It is designed to collect information from a variety of processes. The daemon collects data in a cyclic buffer and then in a one write cycle stores it into the repository, providing so high application performance. The daemon program also stores the errors and a standard
32 The Automatic HTTP Requests Logging and Replaying Subsystem for CMS Plone Fig. 3 Architecture of the component Detector. file, but also to open a new file-storage. This possibility is provided by the signal None or by any other False object. After its receiving a data that is kept in a buffer is stored in a file system, a new file-storage is opened immediately. Player is represented by Python package collective.request.player (Fig. 5). It is designed to playback HTTP 1.1 POST and GET requests, which were sent to the server. Requests can be created directly using the web interface or loaded from files, which were created by Logger. The component is implemented using following means: Python/Plone/Zope packages, ZCML, XML, Buildout framework. All components are designed in such a way that they can operate independently of each other. The components are written using Python 2.6 and intended for CMS Plone 4. 4. Dagger Deployment and Using Fig. 4 Architecture of the component Logger. output logs. Furthermore daemon provides an ability to create a file that stores its PID. Logger receives a log from Detector and stores it in the repository. Implementation of the component Logger was made using Python packages and Buildout framework. The component Logger being residently executed is a daemon program that is managed by the following basic commands: start, stop, restart, status:./bin/logger start starts the daemon;./bin/logger stop stops the daemon;./bin/logger restart restarts the daemon;./bin/logger status views the status of the daemon and its PID. The daemon stores logs of the user s interaction with the web site, keeping them in data file-storages in one of the formats: JSON, CSV or PICKLE. The choice of a particular data file-storage s format is made in recipe s settings depending on user s needs. Sometimes there is a need not only to write data into a To deploy and execute Dagger, it is needed to: (1) Download Dagger subsystem from: https://github.com/potar/dagger; (2) Install collective.recipe.logger and collective.error.detector to your buildout configuration; (3) Run the demon and check its status using next commands: $bin/logger start; $bin/logger status; Fig. 5 Architecture of the component Player.
The Automatic HTTP Requests Logging and Replaying Subsystem for CMS Plone 33 (4) Install collective.error.detector to your web site; (5) Choose a log file (parts/logger/storage) and play it on a web site s copy using collective.request.player. Additional Buildout s settings (demo-storage and before-storage) can help to protect database from damage and to replay requests more accurately. 5. Conclusions In the paper the problem of storing and replaying of HTTP requests to web sites in order to improve their debugging efficiency during development and further support is solved. To solve this problem the automatic HTTP requests logging and replaying subsystem Dagger is developed. It provides storage and playback of the requests that were sent to a website, which is developed using CMS Plone. The subsystem consists of the three components: Detector, which detects requests and creates logs; Logger, which stores the requests sent to a web server; Player, which replays the requests that were sent to the web server. Further improvement of the subsystem may lie in: increasement of the number of supported platforms; development of embedded analytic functions to facilitate the process of finding and removing errors in requests to web sites and development of user s personal data protection features. References [1] A Guide to Solving Web Development Problems, http://www.textfixer.com/tutorials/solving-webdev-probl ems.php. [2] curl, http://curl.haxx.se. [3] Sentry, https://github.com/getsentry/sentry. [4] About Plone, http://www.plug.org.ua/documentation/about-plone. [5] Plone is Based on Zope and Python, http://idv.sinica.edu.tw/marr/python/slides/img/systemou tline.png.