Chapter 4 Generic Log Adapter: A Step towards building Parser based Self-healable Autonomic System 4.1 Introduction

Transcription

1 Chapter 4 Generic Log Adapter: A Step towards building Parser based Self-healable Autonomic System 4.1 Introduction Autonomic computing is used to describe the set of concepts, technologies and tools, which enable applications, systems and entire networks to become more self-managing [44]. IBM has proposed a MAPE-K framework for implementing the Autonomic computing into a software product that supports self-healing mechanism. This system can monitor applications and execute appropriate actions to keep them in a stable state. This is achieved by allowing the administrator to create and maintain rules and policies for healthy working of software product. In the previous chapter 2, a simple human-readable way of representing symptoms in the symptom catalog was introduced. In continuation of the previous work, the autonomic system approach of IBM is based on the processing of the log files generated by the various products are of different formats. In order to work together with other heterogeneous log files IBM has 69

2 proposed a mechanism to write Adapters to convert heterogeneous log-formats into a standard format called Common Base Event (CBE) before processing, [23], [45]. The contents of CBE Logs are then used by the Autonomic manager that triggers the Log and Trace analyzer to read the log-events in the Common Base format, to correlate the logs, based on different criteria (for example, time-based or field-based (such as URLs)) and views the correlated log records. However, this approach has a drawback in terms of performance and cost. 4.2 A framework for Generic Log Adapter The domain of this research orients around current day applications which maintain log files for maintenance purposes. The log-files ensure easier analysis and maintenance of the application by creating a record for every event occurring during the working of the application. The applications, although maintains logs, it can neither take any steps to curb the errors occurring nor can they prevent similar errors from occurring in future-runs. The consequences of such types of errors may lead to a system-crash or loss of data and information. Such crashes or data-losses cannot be tolerated in today s conditions where every millisecond lost by the crashed system can lead to enormous losses to the business. The focus is around such systems in which applications create log-files of their executions but do not have the capability to protect themselves from crashes. A computing system has been created which can sustain on its own and prevent errors by taking appropriate actions before they can occur. Although, technologies exist for autonomic behavior of a system [38], [46], the complex nature and need for a huge knowledge base to operate such tools are what keeping a small scale system administrator from implementing Autonomic solutions. The need for an easier way of maintaining symptom rules is addressed in the previous chapter. In continuation of the previous work and in the problem-determination-scenario of autonomic system [15], [41], [47], every software product has its own log-file format and all files are belonging to different vendors which are heterogeneous in nature. This heterogeneity in the log file format makes the administrator s job more difficult in writing management tools that enhances the complexity issues of problem determination. This is a major obstacle in developing the autonomic computing system that identifies symptoms and initiates the corrective measure to fix the problem. The log-file formats of various products are diverse in nature though they belong to the same category of application. In order to take the benefits of autonomic computing and to address the heterogeneity of log-files, IBM has introduced the concept of writing a code called adapter 70

3 for every product to convert its log-event recorded in the log-file to Common Base event (CBE) logs, so that the software product can effectively be monitored by the autonomic manager for runtime critical errors. Figure 4.1 describes the above concept as suggested by the IBM [48], [49]. Alternatively, a better approach to IBM s work, a new mechanism called know-how of product and generic parser is proposed that supports heterogeneous log-files which interprets each logevents and matches with the symptoms and takes the specified actions. This avoids programming effort required for adapters [50] for every log-file. Software product -N Log file-n Adapter-n Common Base Event (CBE) Monitoring Sub component Action sub component Admin Symptom Catalog Symptom Editor IBM s Autonomic System Manager Figure 4.1 IBM s approach of writing Adapter separately for each product Proposed System architecture Software product-n Log file-n KNOW-HOW-N Generic Log Adapter run-time entity Monitoring Sub Component Action Sub Component Admin Symptom Catalog Symptom Editor Proposed Autonomic System Manager Figure 4.2 Proposed Generic System to Interpret Heterogeneous Log Files 71

4 Figure 4.2 shows the proposed architecture of the ASM software that attempts to implement the computing system based on autonomic concepts which can allow the administrator to create and maintain rules and policies as per the requirement using the symptom-editor and presents the self-monitoring and self-healing property of an autonomous system by monitoring a log-file for breach of policies (set by administrator) and alerting the administrator in the case of any breaches (an act healing) of policies. Here each product is allowed to define the log-file format in its own way using generic notations called regular expressions/language grammars used by compilers. These generic notations (know-how) are different for each product, written and supplied by the vendor. The proposed Autonomic System Manager (ASM) focuses on creating an environment for the system administrator where administrator can monitor and maintain applications by creating policies for the working of the application. If any breach of policy is undesired the ASM takes appropriate actions and alerts the administrator via SMS or . The ASM provides the administrator with a user-friendly SYMPTOM CATALOG (SC) which allows the user to create policies for the application. Administrator has to construct the simple rules expressions as combinations of various events and operations between them. The proposed architecture of the ASM software as shown in the figure 4.2, consists of two main sub components along with the product Knowhow, they are: 1. Know-How: This file describes the log file structure to autonomic manager. It avoids converting log file to a standard format known as Common Base Event. It provides detailed information about the log structure and log variables in the form of regular expression/language grammar. Know-how plays very important role in writing policies for the system. As an illustration IIS Web server is used and its log-file structure know-how description are as follows: In order to effectively manage a web server, it is necessary to get feedback about the activity and performance of the server as well as any problems that may be occurring. IIS Server provides very comprehensive and flexible logging capabilities. The ASM supports the W3C Extended log-format that is the default log format of IIS. By default, IIS logs contain only few fields it can be configured to show other fields of IIS events. ASM uses the following fields: Date Time Client_IP_Address Method URI Stem Protocol status Time taken 72

5 Table 4.1 Internet Information Server (IIS) Log Fields Field Appears As Description Default Y/N Date Date The date on which the activity occurred. Y Time Time The time, in coordinated universal time (UTC), at which the activity occurred. Y Client IP cs-ip The IP address of the client that made the Y Address request. User Name csusername The name of the authenticated user who accessed your server. Anonymous users are indicated by a hyphen. Y Method cs-method The requested action, for example, a GET method. Y URI Stem cs-uri- The target of the action, for example, Y stem Default.htm. HTTP Status sc-status The HTTP status code. Y Time Taken time-taken The length of time that the action took, in milliseconds. N The following section illustrates the format log-file, example of log file instances and the corresponding KNOW-HOW structure of the IIS log- file. A. Know-how structure of IIS application: The Know-How is model that describes the structure of log-file in terms of Lex and Yacc rules which uses generic notations called regular expressions and language Grammar. It also allows the corporate user (ASM) to write their own symptoms and associated actions for heterogeneous software products, without bothering much about product specific knowledge. Hence, there is no learning time, as Know-how refers every variable of software products by $1,$2,$3.., and symptoms are to be written with $1,$2,$3.using C-language constructs. Hence, the Autonomic system manager 73

6 writes a symptoms using $1, $2, $3, which are mapped to various log variables of software products by Know-How. A.1 Know-How YACC specification IIS log file: IIS_Log_format f1 {;} f1 header_type {;} other {;} header_type software version date {;} software SOFTWARE : NAME '\n' { /* $1 ->Refers to label software $2 ->Refers to name of the software While writing symptom policy, ASM is hereby informed to use $1,$2 as log-variables along with execution of required actions. call to action_1() */ Action_1(); } version VERSION ':' VER '\n' { Same as above } date DATE ':' DD1 TIME '\n' { Same as above } other TIME CS-IP CS-USERNAME CS-METHOD SC-URL-STEM SC- STATUS TIME-TAKEN \n { /* $1-> Refers to actual time (hr:mm:sec) $2-> Refers to IP address $3-> Refers to username $4-> Refers to method invoked $5-> Refers to URL $6-> Refers to status code $7-> Refers to length of time taken in miliseconds While writing symptom policy, ASM is hereby informed to use $1,$2 as log-variables along with execution of required actions. /* call to Action_4() */ 74

7 } Action_4(); A.2 Know-How LEX specification for IIS log file: %{ /* Definition section to be included */ %} /* Définition of log-variables * / Logd [0-9][0-9][0-9][0-9]\-[0-9][0-9]\-[0-9][0-9] Logtime [0-9][0-9][:][0-9][0-9][:][0-9][0-9] Logip [0-9][0-9][0-9]\.[0-9]\.[0-9]\.[0-9] Logurl \/[a-z]+\.[a-z]+ U-name [a-za-z]* Time-taken [0-9]+ %% Software { return SOFTWARE; } Version { return VERSION; } date { return DATE; } {Logd} { return DD1; } {Logtime} { return TIME; } {Logip} { return CS-IP; } {U-name} { return CS-USERNAME; } GET POST { return CS-METHOD; } {Logurl} { return CS-URL-STEM; } [0-9]+ { return STATUS; } {Time-taken } { return TIME-TAKEN; } %% B. Example of IIS log format: 1. #Software: <Service and instance> 2. #version: <version> 3. #date: <yy-mm-dd> <Hr :min :sec> 75

8 4. #fields: <time> <cs-ip><cs-username><cs-method> <cs-url-stem><sc-status>< time-taken> C. Example of IIS Log-file Instances 1. #Software: Microsoft Internet Information Services #version: #date: :12 : :33: anjana GET /index.html :38: mmm GET /test.html At the time of execution the Know-How matches with instances 1 to 3 and are skipped as there are no actions routine, When Know-how matches with the instances 4 and 5 the symptoms are matched and corresponding actions are executed. The above mentioned Know-how-Yacc and Know-how LEX specification is supplied by the IIS product vendor in the form of Regular expression and language grammar which are generic in nature. The ASM would then edit the Know-How structure and writes symptom policies in the action part of the production rule using C-language constructs. The ASM can also attach required actions routines for each symptom policy as a corrective major whenever the symptom is matched. 2. Monitoring : Monitors the application and fetches its log records. After fetching, the log records are matched against policies. If a match is successful, action sub component is invoked. 3. Action : Once a breach of policy is detected, this action sub-component sends an alert to the administrator (an SMS or or both as prescribed by the administrator). The administrator can reply back with specific action command. The action sub component extracts this command and executes it as remedial step. If no reply is received, a default action set for that policy is executed. 4. Symptom Catalog : The monitoring sub component fetches the policies for matching from a disk file known as the Symptom Catalog. The Symptom Catalog is a file containing the policies stored by the system administrator. This is done with the help of a tool called the Symptom-Editor. The policies in the symptom -catalog are generated as per administrator s inputs in order to keep system working. Figure 4.3 illustrates the structure of the symptom-catalog. The symptom catalog contains a set of rules and their associated actions. The rules are identified by a unique rule-id and this 76

9 rule-id is also helpful in creation of complex rules. Every rule created by the user is validated and written into the symptom catalog. The Symptom catalog has three fields and are as under : R001# cs-method=get && cs-uri-stem=/hello.html$ SendSms.Java R002# cs-uri-stem=/hello.html $ alertmain.javar003# cs-method=get $ SmsHttpSCDesc.java R004# cs-uri-stem=/hello123.html && time-taken > 100$ shutdown.java R005# cs-uri-stem=/git.html $ SendSms.Java R006# cs-uri-stem=/hello.html &&cs-uri-stem=/hello.html$ SendMail.java R007# time-taken=203 $ alertmainlow.java R008# time-taken=203 $ Logoff.java Figure 4.3 Structure of Symptom catalog or Database a. Rule ID: This is a unique identifier for every rule. This is indicated by the first field. In this project, we have provided facility for 999 unique rules. b. Rule Description: This is the description of the rule being stored. It is a symptom which might occur in the log files. Every bracket has an event inside it. Events can be simple consisting of the log fields or can be a rule by itself. A simple event is indicated by the field name and its value separated by a =. Such events are used as operands in the expression and operators like && and etc are used for the operations. c. Action: This field shows the action to be performed for that rule. The options include SendSms.java, Send .java, Logoff.java, Shutdown.java, restart.java and alertmainlow.java action routines. 5. Symptom Editor: The Symptom Editor is a tool which allows the system administrator to create and maintain rules for any application. The system administrator can create simple rule for monitoring a single event or can create complex rules containing sub rules and operators between them to form expressions. These complex rules can be added, modified or deleted any time by using the symptom editor Implementation details and working of the system The proposed system is implemented on windows XP platform and coding is done using Java Developers Kit (JDK), the version used is v1.6. In order to demonstrate the working of ASM software the simulated IIS log file is generated continuously as long as the ASM software is 77

10 running and monitoring function is being performed. Table I shows the structure of simulated IIS log file, used for demonstration Implementation details The ASM software implements three modules and the details are as follows 1. Monitoring This module consists of code to retrieve contents of a log file and monitor it. In this case, the log-file is simulated IIS log-file being generated continuously The simulated IIS log format is shown in Table I. The simulated log is generated continuously by producer program (JAVA code) that runs in the background and produces log records continuously and these records are written into a file. Another java program called Consumer program monitors this file, i.e. it reads the file and fetches predefined number of lines from the file. The fetching takes place continuously as long as the software is running. The pre-condition required for this to happen is that the log file should be written continuously as long as software runs. This parallel working of the producer and consumer programs is achieved by using the concept of serialization. After the log records are extracted, the records are parsed and fields are separated. The figure 4.4 is a sequence diagram that demonstrates the process of monitoring and creation of simulated IIS log file. Figure 4.4 Sequence diagram for monitoring phase 78

11 1. Store normal record format in a String variable. 2. Keep on writing normal records continuously after appending system time and date. 3. At a random time, pause the normal record writing and insert an error record (all possible error records are stored in a file). 4. Keep on repeating the writing process as long as producer and program is running. From the above steps it is observed that the normal records are written continuously and error records are inserted in between at random. This simulates a real application log generation. 2. Symptom Matching This module consists of code to extract the rules from the symptom catalog as well as the code to parse the log-fields records and match them. Figure 4.6 is a sequence diagram that demonstrates the process of symptom matching. Figure 4.5 Sequence diagram for symptom matching process. 1. Extract a set of records from the log file generated. 2. Extract one record out of the fetched records. 3. Separate and parse all the fields from a record. 4. Read the Symptom Catalog and extract rules one by one. 5. Parse the Rule description of the extracted Symptom Catalog record and extract the fields needed from source log record for matching. 79

12 6. Analyze the source log s fields and match against the symptom catalog s rule description. Break down the complex rules if necessary. 7. Call the action module if a match is found. From the above steps it is seen that the log records are matched against symptom records and action module is called if match is found. This simulates monitoring of an application. 3. Action This module consists of performing appropriate actions when a successful match occurs. The action is an alert to the administrator in terms of an or an SMS. The administrator can also reply with a command back to the ASM. The ASM will then parse the command and perform the ordered execution if possible. Figure 4.7 is a sequence diagram that demonstrates the process of action execution. Figure 4.6 sequence diagram for Action execution process. 1. Get the administrators details (like mobile number and address) from the registration details. 2. For , Compute port address. Contact the mail-server through that port. 3. Send a mail using the internet connection. 4. For SMS, connect to the external GSM modem. 5. Send a message using the GSM modem and its SIM card. 6. For receiving and SMS, use the built-in functions in a similar way. The ASM will receive replies both 80

13 7. via and SMS. 8. After receiving the replies, parse it to extract the command. 9. Use the application APIs to execute the command. 10. If no replies are received for a period of time, execute the default actions stored Working of the proposed system The modules namely: monitoring, analyzing and action are combined together as Generic Log Adapter runtime entity (GLA-r). Figure 4.8 and Figure 4.9 demonstrate the complete working of the proposed system and Figure 4.10 is a snapshot that demonstrates the process of action execution namely and SMS to the administration. Figure- 4.7 working of GLA-e-r GLA-e : Generic Log Adapter editor, GLA-r : Generic log adapter run time entity which include monitoring, analyzing and action execution of self-healing process. 81

14 Figure-4.8 GLA-e-r 1 Administrator logs in the GLA-e. 2 GLA-e prompts for log file location and get it from administrator. 3 GLA-e prompts for know-how location and gets it from administrator. 4 GLA-e prompts for the location LEX & YACC tools 5 GLA-e reads know how a regular expression / grammar describing log file format. 6 GLA-e allows administrator to edit Know-How so that administrator can attach action routines to Know How read. 7 Use LEX & YACC to generate run time GLA called GLA- r. 8 GLA-e calls the appropriate routines of GLA-r. 9 The routine GLA-r reads the contents of the log-file continuously and takes an action as per the policies specified till the program is being terminated by GLA-e. 10 GLA-e terminâtes GLA-r. From the above steps it is observed that an action is taken for every match found. The administrator has been given an alert message. This simulates the self-healing process of the autonomous system. 82

15 Figure 4.9 snapshot of execution of action for error match 1.3. Validation of the proposed work The creation of proposed ASM is inspired by the works of IBM on Autonomic Computing. IBM s Autonomic Computing Toolkit also contains many features apart from its own Symptom Catalog. The scope of this section is limited to comparison and validation between the proposed ASM software with IBM s ASM software for self-healing in Autonomic system. Figure 4.1 and Figure 4.2 shows the details of the IBM s Autonomic system Manager and proposed Autonomic System Manager for self-healing in Autonomic system. A. Heterogeneity of product log-files: In case of proposed ASM for self-healing autonomic system as shown in figure 4.2, the heterogeneity of product log-files are handled by defining the log file structure in a generic notation called as regular expression/language grammar which is called as Know-how that provides the detailed information about the log variables which are used to create the policies. Know-how of each product is specified by the product vendor. During runtime the log variables are extracted and error-logs interpreted by the parser which is a part of ASM software and takes the necessary action as desired by the administrator. The IBM s ASM approach when compared with the proposed ASM approach shown in figure 4.1, the heterogeneity of product log-files are handled by writing separate code for every product called Adapter that converts log-events into a standard format called Common Base Event which are interpreted by the ASM sub component and necessary action are executed as per the administrators recommendation. 83

16 The software product business has taken noticeable growth in the last few years. According to the survey details obtained from research report by CII and Boston consulting, it is observed that the expected export growth of Indian market would be increased to INR 34,400 billion, over the years In this scenario, the today s computing environment which comprises of diverse set of software products running on certain hardware platforms require adapter-code to be written for every products log file to convert log events into standard CBE events to take the benefit of autonomic computing. This necessitates the need for extra programming effort to address the heterogeneity of log files which is a major issue in the problem determination [23] [42] [51] of IBM s self-healing autonomic system. The proposed mechanism as shown in figure 4.2 eliminates the programming effort to write adapters for every software product as explained above. Figure 4.10 and 4.11 show the log-file structure of typical IIS and Apache server applications. The figures also show the programming effort required in terms of number of lines for the two software products. The proposed work is simulated in heterogeneity environment by considering IIS and Apache server applications. It observed that the IBM s Autonomic Computing approach requires 35 lines of Adapter code for IIS log files (30k) and 300 lines of Adapter code Apache log-file (30K) to convert log instances into a Common Base event format. This programming effort has been eliminated in the proposed ASM.This helps the IT professionals to focus more on the service that they provide to the customers, suppliers, employees and business partners. Product -1 Name of the product : IIS server Log file name : IIS log file Structure of log file : 1. #Software: <Service and instance> 2. #version: <version> 3. #date: <yy-mm-dd> <Hr :min :sec> 4. #fields:<time><cs-ip><cs-method><cs-url-stem> <cs-code> Example : #Software: Microsoft Internet Information Services 5.1 #Version: 1.0 #Date: :49:14 10:49: GET /iisstart.asp Number of lines of code for Adapter : 35 Figure 4.10 Log file structure of IIS server 84

17 Product -2 Name of the product : Apache server Log file : Apache log file Structure of log file : %h %l %u %t \ %r\ >s %b Where %h Remote IP address, %l Remote log name %u Authenticated remote user, %t- time %r First line of the request, %s Server response code %b Number of bytes transmitted Example : [18/feb/2000:13:33: GET/HTTP/ Number of lines of code required : 300 Figure 4.11 Log file structure of Apache B. Symptom access-time The Know-How supplied by the software product manufacturer contains several segments and in each segment the Autonomic Manager writes multiple (business specific) policies which are usually few in numbers per segment. A log entry generated at runtime is detected by Know- How (Lex and Yacc Routines ) though lexical analyzer and specialized LR parsing technique called canonical sequences of reductions, i.e rightmost derivation in reverse order. The parser routine would map log entries to the appropriate segments of symptom catalog. Each segment is sequentially searched and appropriate user defined action routines are triggered for run-time errors as a part of self-healing capability of a software product. The symptom reference path in this is X+Y+Z, where Z is the sequential path length within the segment, X=1 for segment location and Y=1 for calling action routines. Hence, the performance of the proposed approach in terms of symptom path reference would be 2 + Z. As Z is very small and is negligible the symptom path reference would be approximated to 2. As mentioned in Chapter 3, the traditional symptom access-time in terms of path-length is, it is observed that the proposed ASM software with know-how model enhances significantly, the self-healing time and action execution time [52]. 85

18 Table 4.2 Comparison of proposed ASM software and IBM s ASM software. Features Proposed ASM Software Traditional IBM s ASM Software Heterogeneous Log file Handled by Know-how model Handled by Common Base Event (CBE) Adapter code Not required Required for every product Symptom access time On an Average two path references Table II summarizes the comparison for the Proposed ASM architecture with IBM s architecture for self-healing autonomic system and it is observed that the proposed ASM architecture eliminates the programming effort for writing Adapter as compared to IBM s ASM architecture. It also helps the IT professionals to focus more on the service that they provide to the customers, suppliers, employees and business partners. Summary The autonomic system approach of IBM is based on the processing of the log files generated by various products. Each product has its own log file format. So, each log file is converted into a standard called Common Base Event (CBE), before being processed. This requires writing adapter for every log file for every product. The proposed ASM system eliminates writing adapters for each log file for each product, thereby saving time and cost of code writing. The experimental results show that the proposed ASM software system addresses the heterogeneity of log-files by Know-how mechanism effectively and self-healing time and action execution time would be enhanced significantly, whenever the symptom is matched. 86