TIBCO LogLogic Unity Quick Reference Guide Concepts Overview TIBCO LogLogic Unity is a sleek, modern and scalable platform enabling technical teams to resolve open issues, which require advanced troubleshooting techniques, complex root cause analysis or deep forensics. LogLogic Unity is a Log processing Search & Alerting tool that takes data from any source and structures that data. This allows for intuitive, fast, and complete interaction with data resulting in faster turn-around from open to close in issue resolution. Taxonomy LogLogic leverages a defined taxonomy for proper field normalization. This enables users to quickly map fields together from different sources into a common schema. Deployment LogLogic Unity architectural view is shown in the following illustration: Highlights: Modular search queries Use all or part of saved search filters to build new search queries using new building block technology Multiple search queries Run multiple searches at the same time Working data sets Work with multiple search results without losing what you are working on. Walk away and come back without losing your search results. Data lookups Enrich your experience with lookup tables enhancing search and alerting capabilities Data at rest correlation Perform advanced correlation against historical data to identify trends Data in motion correlation Maintain advanced correlation in memory to identify key patterns for alerting Comprehensive APIs Leverage core functionality using intuitive APIs built on REST Scalable clustering technology Scale horizontally as needed to maintain performance and storage Parsing A parsing rule relates to the ability to structure data from its original state. By providing structure a user can then perform other functions such as grouping, aggregations or visuals. Parsing also assists in finding information when performing searching. In LogLogic 6 parsing can be performed ad-hoc producing a flexible design to work with. LogLogic 6 also comes pre-loaded with 100 s of rules. It also provides KVP and CSV style parsers to easily extract columns for semi-structured data. Events An event in the system represents a single entry of data. Events may be small single line entries such as syslog or they may consume multiple lines such as JSON outputs or XML outputs. Tags tags are predefined with sys_. These tags are auto defined for each log message and are indexed by the system. This enables faster search queries and access to the data. Field Enrichment Field enrichment provides a way for a user to edit any field in the system to produce outputs that are informative and intuitive. We support lookups, math computations and standard functions to help users define data the way they need to see it.
Filter Bloks Common Search Commands Operations and functions USE Defines event sources including parsing configuration. COLUMNS Defines which columns should appear in the results. GROUP BY Grouping can be used to group values by given column or columns. Grouping requires list of grouping columns and the list of aggregation columns. SORT BY Sorts search results based on the expression. LIMIT Limits the size of search results to be displayed. Common Filter Commands/Statements/Operations Operations and functions AND Narrows your search results by only returning those events where each one of the AND conditions evaluates to true. OR Expands your search results by returning events where either of the OR conditions evaluates to true. Equals (=) A comparison condition compares two expressions using the operator specified in the comparison, which may Not equals (<>), (!=) be one of seven possible comparison operators with well-known meanings. The comparison condition Lower than(<) evaluates to true only if the comparison condition is satisfied. This may be used to narrow search results. Lower or equal (,=) Greater than (>) Greater or equal (>=) Plus (+) The arithmetic (+,-,*,/) and string concatenation ( ) operators can be used to create parts of other conditions. Minus (-) Multiply (asterisk (*)) Divide (forward slash (/)) String concatenation ( ) Function Any set of predefined functions. IS NULL, IS NOT NULL Narrows your search results by accepting or rejecting the event based on whether the evaluated expression is null or not null. An expression most frequently becomes null if a column named in the expression has no value for the current event. LIKE, NOT LIKE Returns true if it matched the supplied pattern (% matches any character, _ matches exactly one character). BETWEEN Supports Timestamps, Long and Integers. IN Checks if value matches anyone of the values in a set. REGEXP, NOT REGEXP Returns true if it matches the supplied pattern. Search Examples Filter Blok Expression sys_eventtime in TUE:WED Use MyParserProfile sys_eventtime in -1h "bob" Use system sys_eventtime in 1d:NOW "bob" and devtype in ('Windows', 'Cisco PIX') sort by sys_eventtime DESC LIMIT 100 "Bob" sys_eventtime BETWEEN '2012-02- 14 14:34:34' and '2012-03-14 12:00:00' sys_eventtime in -1h sort by sys_eventtime DESC sys_eventtime in -1h group by transactionid columns COUNT() sys_eventtime in -1h group by transactionid columns max(sys_eventtime) - min(sys_eventtime) as duration Displays results from the Default data parser profile within defined time range. Displays results from a defined data Parser Profile for a specified time range. Displays results from the default log source with Bob in the text. Displays results from the Data Parser profile within a certain time range. Displays results from a data source with 'bob' in the text, and for a certain set of log sources. Displays sorted first page of results for events ordered by timestamp in descending order. Display results from the log source with Bob in the text Displays sorted events by timestamp in descending order. Displays grouped results based on transaction. Displays grouped results based on the transaction durations. Provides an alias for the expression that will be used as column name. Aggregation Functions Function (*) This will apply the function for any event with no additional constraints. All This will apply the function on all values that are not null. Distinct This will apply the function once per distinct values. Sum This is the total value. Avg This is the average value. Max This is the minimum value. Var This is the variance. Stdev This is the standard deviation function.
Correlation Bloks Functions Arguments Returns String functions lenchar_lengthcharacter_length String Length of string argument 1 lower String Lower case of string 1 upper String Upper case of string 1 trim String Trimmed string 1 (without leading and trailing spaces) substitute String 1, String 2, String 3 Substitute string 2 by string 3 in string 1 left String, Int <int> left characters of string 1 right String, Int <int> right characters of string 1 midsubstrsubstring String, Int 1, Int 2 Characters from string 1 starting at offset <int1> for a length of <int2> findposition String, String Index of the first occurrence of string 2 within string 1, -1 if no occurrence is found concatenate String, String, Concatenation of all strings passed as arguments. List functions size List Size of list Conditional functions IIF Condition, then, else Returns then value if condition else, then returns else valve. Smart List functions lookup String 1, String 2 The value associated with String2 in the smart list named String1. isinlist String 1, String 2 True if the value String2 is defined in smart list named String1. Conversion functions ToTimestamp ToIP ToTimestampString ToInt ToLong ToString ToFloat ToBool ToDouble expression, formatstring or expression, formatstring expression, formatstring or expression, formatstring, The expression, which should evaluate to a string, is interpreted as a time according to the supplied formatstring. If the conversion fails, null is retured, unless a default string is provided, which is interpreted as a time and returned. Same as ToTimestamp, except the conversion is to an IP address (Java InetAddress). Same as ToTimestamp, except the conversion is in the opposite direction to get a printable timestamp. The obvious conversion to integer with default value taken if not convertible. The obvious conversion to Long with default value taken if not convertible. The obvious conversion to String with default value taken if not convertible. The obvious conversion to Float with default value taken if not convertible. The obvious conversion to Boolean with default value taken if not convertible. The obvious conversion to Double with default value taken if not convertible. Columns The system (event metadata) columns are indexed so searching is faster on the system columns. All system columns are displayed with the prefix sys_ and all built-in parsers are displayed with the prefix ll_ in the column list. The following list describes all system columns in the LogLogic Unity event. Name Type Column sys_eventtime Timestamp UTC time of the event in Epoch milliseconds. sys_body String Text of the message. sys_bodysize Integer Size in number of bytes of the body. sys_sourcetype Integer TIBCO LogLogic Log Management Intelligence (LMI) type ID. sys_collectip InetAddress IP from where the event originated. sys_collecttime TimeStamp UTC time of the event when it was ingested into the LogLogic Unit event storage. sys_filename String File name for event collected from a file. sys_filelinenumber Integer Line number in file. sys_tenant String Customer identifier. sys_domain String Customer sub-identifier. sys_partition Long Identifier of the portion of the data on the data node. sys_offset Long Location of the LogLogic Unity event store. sys_eventkey String Unique key that refers to an event in the LogLogic Unity store. sys_lmieventkey String Unique key that refers to an event in the LogLogic LMI event store. sys_applianceid String Identifier for the LMI Appliance. sys_lmidomain Integer, String LMI Domain is a component of the LMI device (source) identifier. sys_sourcedns String DNS name for the event_source_ip.
Rule Structure Statement Rule Structure USE <source identifier> (, <source identifier>)* Within <integer> [ d h m s ] [ Fixed Sliding ] <event group 1> <event group 2> [ Correlation <correlation criteria 1> <correlation criteria 2> ] [ Autofill ] ( Set <expression> As <identifier> )* [ Inject Correlation Event ] [ LIMIT <integer> CORRELATION EVENTS ] Event group structure Event Group <identifier> [ Is ( Required Optional Excluded ) ] [ With Delayed Evaluation ] [ At Least <integer> Events ] [ At Most <integer> Events ] [ <identifier environment> ] [ Where <expression> ] [ With The Same <expression> [ As <identifier> ] (, <expression> [ As <identifier> ] )* ] ( Having <having clause> )* [ Limits <integer> Groups And <integer> Events ] Having Clause At (Least Most) <integer> Distinct <expression> As <identifier> Limit <integer> Count Of <expression> Being <expression> (Greater Lower) Than <integer> Percentage Of <expression> Being <expression> (Greater Lower) Than <integer>%<condition> Correlation Criteria <event_group_identifier1>><field_identifier1> == <event_group_identifier2>><field_identifier2> <event_group_identifier1> (Begins Ends) [At Least <integer> [ d h m s ]] [Up To <integer> [ d h m s ]] (Before After) <event_group_identifier2> (Begins Ends) Search Examples Correlation Blok Example Use system Event Group [My Events] Use my source Event Group [My Events] Operation limit 1000 Direction limit 1000 Use MySource Event Group [Builts] where Operation = "Built" and Direction = "outbound" [Source IP] limit 1000 Use MySource Event Group [Builts] At least 41 events where Operation = "Built" and Direction = "outbound" With the same [Source IP] [Destination IP] limit 1000 Use MySource Event Group [DenyAndBuilt] where Operation = "Deny" OR Operation = "Built" With the same [Source IP] Having at least 2 distinct [Operation] limit 1000 This rule will trigger a new alert at the first event and will accumulate all events during 30 minutes time period. This rule will do the same as rule 1 but the alerts generated will then give information about the number of distinct operation/direction and their values. This rule will filter events which have Operation equals to Built and Direction equals to outbound, and give count of distinct source IP and values up to 1000. This rule is looking for at least 41 events with the same criteria as the previous one, coming from the same Source IP, and giving information about the number of distinct Destination IPs and their value up to 1000. This rule looks for a Source IP which has events from at least 2 distinct operations. 3301 Hillview Avenue Palo Alto CA 94304 Copyright TIBCO Software Inc. ALL RIGHTS RESERVED
Columns The system (event metadata) columns are indexed so searching is faster on the system columns. All system columns are displayed with the prefix sys_ and all built-in parsers are displayed with the prefix ll_ in the column list. The following list describes all system columns in the LogLogic Unity event. Name Type Column sys_eventtime Timestamp UTC time of the event in Epoch milliseconds. sys_body String Text of the message. sys_bodysize Integer Size in number of bytes of the body. sys_sourcetype Integer TIBCO LogLogic Log Management Intelligence (LMI) type ID. sys_collectip InetAddress IP from where the event originated. sys_collecttime TimeStamp UTC time of the event when it was ingested into the LogLogic Unit event storage. sys_filename String File name for event collected from a file. sys_filelinenumber Integer Line number in file. sys_tenant String Customer identifier. sys_domain String Customer sub-identifier. sys_partition Long Identifier of the portion of the data on the data node. sys_offset Long Location of the LogLogic Unity event store. sys_eventkey String Unique key that refers to an event in the LogLogic Unity store. sys_lmieventkey String Unique key that refers to an event in the LogLogic LMI event store. sys_applianceid String Identifier for the LMI Appliance. sys_lmidomain Integer, LMI Domain is a component of the LMI device (source) identifier. String sys_sourcedns String DNS name for the event_source_ip. Log Sources LogLogic Unity supports message body text search for all of the log sources supported by LogLogic LMI and also supports advanced searching of source-specific parsed columns for the following sources via General Parser. For details, see the TIBCO LogLogic Log Source Packages documentation. Note that not all event types supported by LogLogic LMI may be supported by LogLogic Unity. Log Source ADS Microsoft Active Directory Service Apache Web Server Blue Coat ProxySG Syslog BMC Remedy Action Request (AR) CA SiteMinder Access Management Check Point (CP Audit) Cisco ASA Adaptive Security Appliance Cisco ACS for Windows Cisco Content Engine Cisco ESA Cisco Services Module (FWSM) Cisco IOS Cisco IPS Cisco Identity Services Engine (ISE) Cisco NetFlow Cisco (Nexus) NX-OS Cisco Secure ACS Cisco Web Security Appliance (WSA) Fortinet (FortOS) F5 BIGIP traffic Management Operating (TMOS) General Unix GuardiumSQLGuard GuardiumSQLGuard Audit HP NonStop HP-UX Operating Audit IBM AIX Audit IBM AIX Operating IBM DB2 Universal (UDB) IBM Resource Facility (RACF) Device Category Active Directory Apache Web Server WebProxy BMC Remedy ARS UTM Content Engine Mail Security /VPN Router & Switches IPS Router Switch Web Security LoadBalancer DB IDS/IPS DB IDS/IPS Audit
Log Sources - continued Log Source IBM ISS SiteProtector Juniper IDP Juniper RT_Flow Jumper SSL VPN Secure Access Juniper (JunOS) LogLogic Appliance LogLogic Security Manager McAfee epolicy Orchestrator McAfee G2 Sidewinder Microsoft DHCP Microsoft Office Sharepoint Server Microsoft Operations Manager Microsoft Internet Authentication Service (IAS) Microsoft SQL Server Microsoft Microsoft (French) Microsoft (German) Microsoft (Japanese) MySQL Server GDBC NetApp Decru DataFort NetApp Filer Novell edirectory Oracle Serer General Collector for Oracle Palo Alto Networks PanOS RSA ACE/Server Reuters KondorPlus Snort Sourcefire Sensor Sourcefire Defense Center Squid2 Sun Solaris Basic Security Module (BSM) Sybase Adaptive Server Enterprise (ASE) Symantec Symantec SEP TIBCO ActiveMatrix Administartor TIBCO ActiveMatrix BPM TIBCO ActiveMatrix Business Works TIBCO Administrator TIBCO API Exchange Gateway Server TIBCO Hawk Agent TIBCO Enterprise messaging Service Collector (EMSC) TrendMicro Control Manager TrendMicro OfficeScan Tripwire for Server VMware ESX Server VMware vcenter VMware vcenter Orchestrator VMware vcloud Director VMware vshield Edge Device Category IPS IDS/IPS VPN UTM IPS /VPN Microsoft DHCP Application Content Management Decru Datafort NetApp Filer, NetApp Filer Audit LDAP Directory Service UTM Application Intrusion Detection IDS/IPS IDS/IPS Blue Coat Sun Solaris Operating BSM EMS Tripwire Management Station Hypervisor Automation Server