RULESET DEFINING MANUAL. Manual Version: Software version: 0.6.0

Transcription

1 RULESET DEFINING MANUAL Manual Version: Software version: Last Revision: June 09, 2011

2 Contents 1 Introduction What is a F-Deets ruleset Where are they located How to modify them Ruleset elements Introduction Parsing rules Rule Detailed Rule Detailed Rule Definition Detailed Rule Reference Rule elements Main regex Check Name Parameters Dummy parameter Special Message parameter Actions Other Color definition Regex definition Load command Comments /26

3 3 Creating rules First rule Using wildcards Categorizing logs Coloring logs Two detailed rules in one rule Removing unwanted log fragments Referencing detailed rule Modifying parameter values Required and defining rules Continue, skip and nomessage actions Using dummy parameters How does the parser work Ruleset loading process Ruleset compilation Parsing process Appendixes Appendix A: A complete list of directives Appendix B: Actions and their behavior Appendix C: Predefined general message parameters Appendix D: Supported date formats /26

4 1 Introduction This document describes the language used to create customized F-Deets parsing rulesets. The rulesets define how the information from log messages transmitted to F-Deets syslog is extracted. While relatively simple and intuitive, the F-Deets ruleset language is a powerful tool that can be utilized to further enhance F-Deets usability. 1.1 What is a F-Deets ruleset F-Deets ruleset is an ordered list of parsing instructions used by the paser to extract information from logs. The rules are processed from top to bottom, one-by-one, until the log message fits one of them. The rule is then applied to this message and the parser proceeds to next log message. There can be more than one ruleset. Parser will process them in the order they are defined in configuration file until one of them recognizes the message. If it is not recognized, the message is stored anyway (and an entry in the application log is created with WARN level informing about unrecognizable event). 1.2 Where are they located All rulesets are stored in {Application directory}/aid/def folder. This is the default location the parser searches. 1.3 How to modify them To modify a ruleset simply use a text editor of your choice. Warning: If there are any syntaxerrors ine the ruleset file the parser service will not start. All errors will be reported at WARN level or greater in parser log file {Application directory}/applog/aid.log. Be sure to keep backup of the original files. In case they get corrupted you may have to reinstall F-Deets completely to restore them. These rulesets may be treated like a simple scripting or markup language. While flexible, there are some restrictions that must be obeyed in order for them to work properly. Even if the parser successfully compiles the rulesets, it may behave in a different way than the rulesets creator intended. Make sure to verify it's behavior after creating rules. 4/26

5 2 Ruleset elements 2.1 Introduction The main element of a ruleset is a rule, which can consist of any number of detailed rules (this includes no detailed rules at all). In a rule or detailed rule actions may appear. They either modify parser's behavior or a value they are related to. Rule and detailed rule may have a check element, which is used to check whether this rule should be applied to given message. Detailed rules may have any number of parameters, which correspond to regex match groups in the order they are defined in detailed rule. The behavior of certain elements may vary depending on where they are defined. The same action may do something for parameter than for a detailed rule. All these elements, their behavior and usage are explained in further sections of this manual. The elements' directives are case insensitive, which means you can either write RD, rd, rd or Rd. Remember that regexes are still case-sensitive! 2.2 Parsing rules There are two types of parsing rules: Rules and Detailed Rules. A rule can consist of any number of detailed rules. Rules contain no parsing information directly, they are used to aggregate detailed rules into one functional entity Rule A rule is the uppermost element of a ruleset. It usually corresponds to one message and is designed to extract all information from this message. Rule should contain an, as simple as possible while still distinctive, check element to increase parsing speed. RD: a name of a rule CH: this regex must be found in the message to match the rule Once a message is matched against a rule, no other rules will be used for this message unless a continue action is encountered Detailed Rule A detailed rule is an element of a rule and it must be declared after a rule. It consists of a regex string, possibly a check instruction, and any number of actions and parameters. The 5/26

6 amount of parameters must be the same as amount of match groups in the regex. If the amounts differ, parser will report error in the log file and will not start. DR: a regex for this rule CH: any check string Detailed Rule Definition A detailed rule definition is a named detailed rule which can be used multiple times. It must contain a name element, and by using this name with Detailed Rule Reference instruction it can be used in any amount of rules. Detailed Rule Definition can occur anywhere, though it is strongly encouraged to define them before all rules. DRD: a regex of the detailed rule definition N: name of the detailed rule Detailed Rule Reference After defining a named detailed rule it can be referenced in any rule. Aprat from having body defined elsewhere, detailed rule references act exactly the same as detailed rules. RD: any rule DRR: an existing detailed rule name 2.3 Rule elements So far only the ruleset logical structure has been covered. The elements of parser rules play a different role they define the data to be extracted and the modifications that will be applied to this data Main regex The main regex is located after the detailed rule directive. It can have any length and any number of matching groups. If the regex is invalid, an error will be reported in the same way. DR: here goes the detailed rule's regex The regex will be matched against the log if any part of the log matches it, i.e. regex cats will be successfully matched against Mike loves cats and dogs. There is no need to 6/26

7 include start/end wildcard (like ^.* or.*$) in the regex. If the regex is to complex (contains to many.* wildcards), the parser might not be able to use it and will quit and report error in the log Check Check is used to verify whether given rule or detailed rule should process a log message. A rule should contain a check element. Detailed rule may, but doesn't have to. Log message is matched against the check regex and the rule/detailed rule is processed if and only if the matching was successful. RD: a traffic rule CH: traffic Name Name element may occur only in detailed rules. By defining a name we get two benefits: the same rule can be used again by calling detailed rule reference command and, in case there are errors in this rule, it's name appears in the application log. Detailed rule definitions must contain a name. DR: traffic shaping N: a simple name Parameters Parameter elements define the names of data extracted from log messages. Each parameter corresponds to one match group exactly in the order they were defined. For regex: IP=(.*) port=(.*) and two parameters IP and port, the first match group would be assigned to IP and the second to port. The name of the parameter will be available as a column in F-Deets gui as soon as it is encountered in a valid ruleset. DR: param1=(\w+) param2=(\w+) P: param1 P: param Dummy parameter Dummy parameter is a special name which can be used to perform an otherwise impossible modification to the log message. The value of this parameter will not be stored in 7/26

8 the database. The usage of dummy parameters is extensively covered in Creating Rules tutorial later in this document. The example below shows a detailed rule which will remove we don't like this part of log from incoming message. DR: (we don't like this part of log) param2=(.*) P: dummy A: delete P: param Special Message parameter A message parameter will automatically be added to the parsed results and it will contain the part of the log remaining after the rule has been applied (some parts of the log may be cut using actions). If you use a parameter called message, it's value will be overwritten automatically Actions Actions can be used to influence parser's behavior or modify either parameter values or the log messages. Some actions have different behavior depending on the preceding element. Actions can be assigned to rules, detailed rules or parameters. They do not need to appear immediately after these elements, they will be applied to the nearest above defined element of any of these types. An example of action-packed rule: RD: start of line CH: NetScreen device_id A: continue A: defining A: required DR: \w+?: NetScreen device_id=(.*?)\s A: delete P: MachineAlias Some actions require additional parameter, some do not. If they do, the parameter should be added after the action's name and be separated from it by a space. Certain actions can not be applied in certain elements. Consult Appendix B for details. 8/26

9 2.4 Other Either of the following elements can occur anywhere in the ruleset file. It is strongly encouraged, however, to keep their locations logical so the ruleset remains easily readable Color definition Color definitions assign string aliases to RGB colors. The defined strings can then be used in combination with color action. The definition looks like this: color: red #FF0000 Colors do not need to be defined before they are used Regex definition A reusable regex can be defined like this: regex: IP \d{1,3}\.\d{1,3}\.\d{1,3}\.\d{1,3} The first string (no spaces allowed) is the regex's name and the remaining characters are the regular expression itself. It can then be used in a detailed rule regex (or check regex) in this way: IP address (##IP##) is assigned This can be especially useful for long and error-prone regexes. It also makes modifying often-occurring regexes much simpler. Regexes do not need to be defined before they are used Load command Load command can be used to include a ruleset from file. You may want to have all your regexes or colors defined in different files. Large parsers can be divided into smaller parts for readability's sake. If the file location is not absolute, the parser searches {Application directory}/aid/def folder. Using subdirectories is also possible: load: regexes/regex_set_1.def will try do open {Application directory}/aid/def/regexes/regex_set_1.def file. 9/26

10 The loaded file must be a valid ruleset file. It is not possible to have half of a rule defined in one file, and half in the other. The loaded file can, however, reference regexes, colors and detailed rules from the file which loaded it. The loading file may also use the objects defined in the loaded file Comments Comments can occur anywhere in the ruleset file. They start with # char and always cover the while line. It is not possible to create a comment after an instruction. 10/26

11 3 Creating rules This sections explains in detail how to write rules for F-Deets. We believe in learning by doing, so we will be preparing real rules for parsing messages from screenos devices. 3.1 First rule We will start with something simple. We want to parse the following message. Traffic shaping is turned on Let's create a rule: RD: traffic shaping CH: traffic shaping is turned on DR: traffic shaping is turned (on) P: state Let's go through it line by line. The RD: directive creates a new rule and assigns it the name traffic shaping. The next line contains a check element. It means that the rule will be applied to log message only if the message contains traffic shaping is turned on string. DR: element defines a detailed rule with one match group. We want to extract one information from it the state of the traffic shaping feature so we surround the state value with match group delimeters ( and ) and add one parameter state. If we applied this rule to the message, in F-Deets we would actually receive an entry with two parameters: state with value on and parameter message containing the whole log. 3.2 Using wildcards What would happen if we received a message: Traffic shaping is turned off The rule we've created before would not parse this message, since the check element would not be fulfilled. Let's modify the rule to match the new message: 11/26

12 RD: traffic shaping CH: traffic shaping is turned DR: traffic shaping is turned (.*) P: state We've done two things: shortened the check string by removing on from it. Now the parser will match both messages with this check. Secondly, we switched matching group contents to wildcard.* which means that any string can fit this match group. Now we've got a rule that actually parses two slightly different messages. The state value will be either on or off, and the whole message will still be saved. 3.3 Categorizing logs Category is an important parameter and it can be set in two ways. First, we can simply define a parameter called category and match it against any part of the message we want. For a rule like this: RD: traffic shaping CH: traffic shaping is turned DR: (traffic) shaping is turned (.*) P: category P: state We would get a traffic category. Notice that the category parameter has been inserted before the state, because the match group for traffic is before the match group for state. The second way is to use an action. Our rule would then look like this: RD: traffic shaping A: category traffic CH: traffic shaping is turned DR: traffic shaping is turned (.*) P: state Notice that the action is defined after the rule definition. In this case it could be defined after detailed rule and it would have the same effect. However, if you have many detailed rules in one rule, each detailed rule may define it's own category. If the detailed rule is matched, it's category will replace the earlier category assigned to this log. 12/26

13 3.4 Coloring logs Coloring logs is achieved through a color action. If we wanted to color our rule blue, it would look like this: RD: traffic shaping A: category traffic A: color #0000FF CH: traffic shaping is turned DR: traffic shaping is turned (.*) P: state We can also define the color earlier as a string, using the color directive: color: blue #0000FF RD: traffic shaping A: category traffic A: color blue CH: traffic shaping is turned DR: traffic shaping is turned (.*) P: state The color name will be converted to the number in while compiling ruleset. If you modify the color's value the change will not propagate to earlier-parsed logs. 3.5 Two detailed rules in one rule One rule may contain more than one detailed rule (or might not contain any at all, the message would only be stored then). To illustrate this we will prepare a rule for such two messages: NTP server is disabled on interface ethernet0/5 NTP server is enabled on interface ethernet0/4, mode: control This message contains two parts. First defines the state of the NTP server, the second defined the server's mode. However, the second part occurs only when the NTP server is enabled. Let's create a rule capable of parsing both of these messages: 13/26

14 RD: NTP server is dis/en abled CH: NTP server is DR: server is (enabled) on interface (.*), mode:?(.*) CH: enabled P: state P: interface P: mode DR: server is (disabled) on interface (.*) CH: disabled P: state P: interface Each of the detailed rules contains a check element. The first is applied only when there is a word enabled inside the message, the second, if disabled occurs. This is a crude way of doing this, but it certainly works. By keeping the checks simple, we ensure that the parser works quickly. Let's try to make a rule doing the same thing, but in a different way: RD: NTP server is dis/en abled CH: NTP server is DR: server is (.*) on interface (.*)(?:$,) CH: enabled disabled P: state P: interface DR: mode:(.*) CH: mode P: mode The first detailed rule gets the state and interface, regardless of whether the server is on or off, and the second detailed rule finds the server's mode if it is specified. Notice the (?:$,) after the interface match group. Since we use a general (and greedy!) wildcard which fits any string, for this message NTP server is enabled on interface ethernet0/4, mode: control the parameter interface would be equal ethernet0/4, mode: control if we hadn't used the (?:$,). It would also be possible to define the interface string in a better way, like this: #regex interface [A-Za-z]+\d/\d Which would allow us to skip the (?:$,) wildcard. 14/26

15 3.6 Removing unwanted log fragments CC Open Computer Systems Ltd. Sometimes we want to remove a part of a message. It can make the log less readable, it can contain redundant information. We received the following message: PASSWORD authentication successful for admin user 'backup' at host ( :01:01) And we want to remove the date time part at the end of the line while saving its value as a parameter. So we prepare a detailed rule to do it: DRD: \((\d{4}-\d{2}-\d{2}\s+\d+:\d+:\d+)\) N: time at the end of line A: delete P: timeserver This is a detailed rule reference, which can be used in any rule. When building the regex we should be as precise as possible to reduce parser's matching time. The most important part of this rule is the delete action. Placing it after the rule definition means, that the part of the log message matched against this regex will be removed from the message. So after applying this rule to our message, the remaining part would be: PASSWORD authentication successful for admin user 'backup' at host So we've gotten rid of the unwanted part. If we didn't want to store the value at the end of the line, we would have to remove the grouping brackets from regular expression and remove the timeserver parameter. 3.7 Referencing detailed rule In the previous example we've defined a named detailed rule which removes a part of a message. How do we use it in a rule? A complete rule parsing the previous message would look like this: 15/26

16 RD: authentication successful for admin CH: authentication successful for admin A: category security DRR: time at the end of line DR: (.*) authentication successful for admin user '(.*)' at host (##IP##)\. P: auth type P: user P: sourceip rule. By using the DRR instruction we include the previously defined detailed rule in this 3.8 Modifying parameter values After each parameter there can be any number of actions that modify it's value. The possible actions are: deletestr <str> - deletes str from the parameter's value replace <str> - replaces the parameter's whole matched value with given string; if used with dummy parameter, will replace the matched string with given string in the message itself insertbefore <str> - inserts the given string before the parameter's value insertafter <str> - inserts the given string after the parameter's value tolowercase converts the parameters value to lowercase characters touppercase converts the parameters value to uppercase characters 3.9 Required and defining rules Some actions can be applied to rules. The required action marks a rule or a detailed rule and causes a parser to finish working when such a rule is not matched. If a message from given device contains some distinguishing elements, such required rule can be used to boost parser's efficiency. Instead of trying to apply all it's rules to a message, it will stop working after not matching a required rule and proceed to next ruleset. 3Defining rules have opposite meaning. If such rule is matched, the parser will permanently assign its ruleset to the IP address of the device that sent this message. It will not try to use other rulesets for this device, therefore improving performance. Great caution must be taken when creating defining and required rules. Making a too restrictive required rule might make some messages unparseable. Making a not restrictive enough defining rule may assign a wrong parser to a device. Here is an example of both defining and required rule: 16/26

17 RD: start of line CH: NetScreen device_id A: continue A: defining A: required DR: \w+?: NetScreen device_id=(.*?)\s A: delete P: MachineAlias 3.10 Continue, skip and nomessage actions A continue action can be defined for either rule or detailed rule. If rule with this action is fulfilled, the parser will not stop processing the ruleset and will continue to next rules. This can be useful for removing constant-syntax parts of the message, like, for example, source IP or time. If a skip action is defined for a rule or detailed rule, then the messages matched against this rule will simply not be saved to F-Deets database. Nomessage action is also defined in rule or detailed rule and, when encountered, will tell the parser not to save the message parameter automatically. By default, the whole message remaining at the end of the parsing will be saved as a message parameter Using dummy parameters A dummy parameter has been mentioned before. It can be used to perform changes on a part of a log message using normal grouping regexes without creating unnecessary parameters. For example, we have such message: Certificate cert1 with expiration date has expired And we want to convert it to: Certificate cert1 has expired on To do this, we can make the following rule: 17/26

18 RD: simple convert rule CH: with expiration date.* has expired DR: (with expiration date).*( has expired) P: dummy A: replace has expired on P: dummy A: delete The first match group will be replaces with has expired on string, and the second (after the date) will be deleted. Neither matching group will be saved as a separate field for this log message. The whole text, after modification, will be saved as message by default. 18/26

19 4 How does the parser work In order to process log messages, the rulesets first need to be loaded. The parser opens devices.cfg file from config directory and loads all the rulesets defined there. Also, it may associate the ruleset with an IP address if one is defined in the config file. After the configration file has been read, all the rulesets are loaded and pre-parsed. If there are any syntax errors, an appropriate message will be appended to aid.log in applog directory and the parser will stop working. When there are no syntax errors the ruleset will be compiled. All regexes will be prepared, regex, color and detailed rule references will be resolved. If any of these stages fails, the parser will append a message to log and quit. After all rulesets have been successfully loaded, the parsing commences. All errors will cause the parser to quit. If you modified a ruleset and the parser process doesn't start, consult aid.log for details. 4.1 Ruleset loading process When the file is successfully opened and read, each line is parsed one by one. Comment lines are silently skipped, so are empty lines. The order of instructions is verified (parameter must be defined after a detailed rule, detailed rule can not be defined without a rule, etc). The actions locations are also validated some of them must be assigned to parameters, some to rules. If there is a misplaced action it will be reported in this stage. The same goes for unrecognized instructions, unrecognized actions and actions lacking required parameters. 4.2 Ruleset compilation In this stage all dynamic elements are resolved. This includes colors, regexes and detailed rules. After resolving regular expressions references, the resulting expressions are also validated. This operation takes place after executing all load commands, so cross-file references are possible and will not cause an error. Unrecognized references and invalid regexes will result in the parser not running. All errors will be reported in aid.log file. When the ruleset is verified, the encountered parameters are synchronized with the database. If you have used your own, new parameter names, they will be available in F-Deets GUI after compiling was successful. 19/26

20 4.3 Parsing process A log message is processed by all rulesets until one of them reports that it recognized the message. Inside one ruleset, the message is matched against all rules. If the message fulfills the rule's check, then all of this rule's detailed rules are applied to the message. Detailed rule may contain a check element and if it does, the message is matched against it. If it passes or there is no such element specified, the detailed rule's main regex is used to extract information from the message, modify the message, etc. If the rule or any of it's detailed rules contains continue action, then parsing is continued in the same ruleset. If there are not non-continue rules that matched against this message, the parser proceeds to the next ruleset. The parsed message is isolated for each ruleset, so even if it has been modified by any rules in one of the rulesets, the message will return to original for next rulesets. If a skip action has been applied, the log message will not be saved into F-Deets database. For a defining action the devices.cfg configuration file will be updated with new IP/ruleset mapping. 20/26

21 5 Appendixes 5.1 Appendix A: A complete list of directives Direct ive RD Name Parameters Location Action Comment Rule Definition Name Anywhere Create a new rule DR Detailed Rule Regex After RD Create a new detailed rule DRD DRR Detailed Rule Definition Detailed Rule Reference Regex Anywhere Create a new, named, detailed rule Name After RD Reference a previously defined detailed rule CH Check Regex After RD or DR N Name Name After DR or DRD P Parameter Name After DR or DRD A Action Name parameter regex Regex definition load Load file instruction color Color definition After RD, DR, DRD or P Check if rule applies to message Associates a detailed rule with a name Associates a matching group with a name Creates an action for given element Required N directive Should be as simple as possible Necessary for DRD, for DR useful for debugging log files Name regex Anywhere Creates a regex alias Preferably defined at the beginning of a file Filename Anywhere Loads a definitions file Preferably defined at the beginning of a file name #RRGGBB Anywhere Creates a color alias Preferably defined at the beginning of a file 21/26

22 5.2 Appendix B: Actions and their behavior defining if a rule with this action is fullfilled, the currently processed log is marked as defining, which means that the IP address of the device will be permanently associated with the parser that contains this action. Applies to: RD, DR, DRD required if a rule with this action is not fulfilled, the parser stops execution and the log message processing proceeds to next parser. Applies to: RD, DR, DRD continue if a rule with this action is fulfilled, the parser will still process next rules. Useful for cutting unchanging elements of the message. Applies to: RD, DR, DRD nomessage if a rule with this action is fulfilled, the parser will not save the message parameter to the database. Applies to: RD, DR, DRD skip - if a rule with this action is fulfilled, the parser will not save any information about this message to the database. Applies to: RD, DR, DRD category - if a rule with this action is fulfilled, the category of the message is set to action's parameter. Applies to: RD, DR, DRD color - if a rule with this action is fulfilled, the color of the message is set to action's parameter. The parameter can be either #RRGGBB code or an alias defined through color instruction Applies to: RD, DR, DRD delete the action depends on the place if was used: DR, DRD: removes the whole matched regex from the log string. For a message: Admin user backup has logged on via SSH from :36933 and a rule: 22/26

23 DR: Admin user (.*) has A: delete P: user The message would be transformed to: logged on via SSH from :36933 P: removes the parameter's value from the log string. The parameter is still assigned the proper value. For a message: Admin user backup has logged on via SSH from :36933 and a rule: DR: Admin user (.*) has P: user A: delete The message would be transformed to: Admin user has logged on via SSH from :36933 Applies to: DR, DRD, P deletestr removes the given string from associated element. DR, DRD: removes a string from a message. For a message: Admin user backup has logged on via SSH from :36933 and a rule: DR: Admin user (.*) has A: deletestr Admin user P: user The message would be transformed to: backup has logged on via SSH from :36933 P: removes a string from a the parameter value message. The log message remains unchanged. For a message: Admin user backup has logged on via SSH from :36933 and a rule: 23/26

24 DR: Admin user (.*) has P: user A: deletestr bac The user parameter's value would be kup The action would change if one used the dummy parameter. Consider this rule: DR: Admin user (.*) has P: dummy A: deletestr bac It would transform the message, in the same way as in the first case, to: Admin user kup has logged on via SSH from :36933 Applies to: DR, DRD, P replace replaces the associated parameter's value with given string. For a message: Admin user backup has logged on via SSH from :36933 and a rule: DR: Admin user (.*) has P: user A: replace AN USER The user parameter value wolud be AN USER. However, for a message: Admin user backup has logged on via SSH from :36933 and a rule: DR: Admin user (.*) has P: dummy A: replace AN USER It would transform the message to: Admin user AN USER has logged on via SSH from :36933 insertbefore insets the given string before the associated element. DR: inserts the string before the whole matched message. For a message: Admin user backup has logged on via SSH from :36933 and a rule: 24/26

25 DR: from ##IP## A: insertbefore _protocol The resulting message would be: Admin user backup has logged on via SSH protocol from :36933 Remember, that for action values the _ character will be translated into space. A space at the start/end of a value will be trimmed. P: inserts the string before the matched parameter's value. Affects only the parameter itself. For a message: Admin user backup has logged on via SSH from :36933 and a rule: DR: Admin user (.*) has P: user A: insertbefore mr. The user parameter's value would be : mr.backup Have we used dummy parameter instead of user, the behavior would be different - the log message would be modified. For exactly the same message and rule like this: DR: Admin user (.*) has P: dummy A: insertbefore mr. The resulting message would be: Admin user mr.backup has logged on via SSH protocol from :36933 And, of course, the dummy parameter would not be stored. insertafter behaves exactly like insertbefore, except that it inserts the value after matched string or at the end of parameter. tolowercase converts the related string value to lowercase Applies to: RD, DR, DRD, P touppercase converts the related string value to uppercase Applies to: RD, DR, DRD, P 25/26

26 5.3 Appendix C: Predefined general message parameters All these names are case-insensitive. They can not, however, contain any spaces. If they do, they will be treated as different parameters. DeviceName DeviceIp TimeSource TimeServer SourceIp SourceDnsName SourcePort DestinationIp DestinationDnsName DestinationPort SourceXlatedIp SourceXlatedPort DestinationXlatedIp DestinationXlatedPort Action Category MessageNumber MessageType MachineAlias Message 5.4 Appendix D: Supported date formats syslogd format: month d HH:MM:SS, examples: Oct 17 14:52:13, Dec 1 01:34:51 ISO 601: YYYY-MM-DDTHH:MM:SS+HH:MM, examples: T21:46:18+02:00, T01:11:34 simple date time format, examples: :59:01, :13:48 26/26