Application for Splunk Enterprise

Transcription

1 Application for Splunk Enterprise User Guide Document Version October EN Rev. A

2 2015 ThreatConnect, Inc. ThreatConnect is a registered trademark of ThreatConnect, Inc. UNIX is a registered trademark of The Open Group. JavaScript is a registered trademark of the Oracle Corporation. Splunk is a registered trademark of Splunk, Inc. Application for Splunk Enterprise 2

3 TABLE OF CONTENTS OVERVIEW... 5 Key Features... 5 GETTING STARTED... 5 Prerequisites... 5 Installation... 6 ThreatConnect API User Creation... 6 THE THREATCONNECT APP FOR SPLUNK... 8 App Setup and Configuration... 8 Owner Configuration... 9 Setting Up Alerts Option Option 2 (Legacy) Basic Alert Scenario The ThreatConnect Dashboard The Indicator Dashboard The Diamond Dashboard The Threat Lookup Screen The Reports Screen The DB Data Menu The Search Screen Workflow: Event Actions Application for Splunk Enterprise 3

4 Workflow: Field Actions The ThreatConnect App Logs Indicator Logs Group Logs Tag Logs Application for Splunk Enterprise 4

5 OVERVIEW The ThreatConnect Application (App) for Splunk Enterprise gives Splunk users the ability to leverage customizable threat intelligence integrated into Splunk from their ThreatConnect accounts. ThreatConnect provides the ability to aggregate threat intelligence from multiple sources (i.e., open source, commercial, communities, and internally created), analyze and track identified adversary infrastructure and capabilities, and put that refined knowledge to work in Splunk, identifying threats targeting organizations. Key Features Multi-source threat intelligence collection (open source, commercial, communities, internal research) Transparent threat intelligence aggregated, confidence weighted, and applied to triggered Splunk alerts Automatic threat intelligence indicator updates Prioritized events based on criticality and confidence scores, relationships to known threat types and groups, past incidents, and tags Insights on a threat s capability, infrastructure, and past incidents affecting users or members of trusted communities represented using the Diamond Model for Intrusion Analysis History of how a Threat has affected users networks using the Diamond Dashboard GETTING STARTED Prerequisites Users will need an active ThreatConnect Application Programming Interface (API) account to leverage the ThreatConnect App for Splunk Enterprise. Users without a current subscription to ThreatConnect, who wish to start a trial of the ThreatConnect App for Splunk with a live connection to the latest customizable threat intelligence data, please inquire at: Once an Organization has been licensed for API access, its users will need to create an API User within the Organization prior to Splunk interfacing with the ThreatConnect API. For detailed steps on creating an API User, please see the API User Creation section in the ThreatConnect Application Programming Interface User Guide. Application for Splunk Enterprise 5

6 Installation Users can download the ThreatConnect App for Splunk Enterprise from or they can directly install the App from the Find more apps online link in Apps Manage Apps. For more information on installing Splunk Apps, reference the Splunk documentation located at: ThreatConnect API User Creation A ThreatConnect API User is created from within the ThreatConnect Web application for the instance being used. For the ThreatConnect Cloud edition this is: To Create an API User: 1. Log in with an Organization Administrator account. 2. On the top navigation bar, click on the Account drop-down menu and select Org Settings (Figure 1). The Organization Settings screen will appear with the MEMBERSHIP tab highlighted (Figure 2). Figure 1 Figure 2 Application for Splunk Enterprise 6

7 3. Click on the + Create API User button, and the API User Administration pop-up screen will appear (Figure 3). Figure 3 4. Fill in the information requested, as shown in Table 1, and copy the Access ID and Secret Key to a safe location. Note: It is critical to save the Secret Key, as this will not be available again. If it is lost, the existing API User will need to be deleted, and a new API User will need to be created. Do not share the Access ID or Secret Key as these are your credentials and will be used to track API usage limits. Table 1 Parameter Description First Name Last Name Pseudonym The First Name of the user that will appear in posts, data modifications, and data creation within ThreatConnect The Last Name of the user that will appear in posts, data modifications, and data creation within ThreatConnect The Pseudonym of the user that will appear in posts, data modifications, and data creation within ThreatConnect Application for Splunk Enterprise 7

8 THE THREATCONNECT APP FOR SPLUNK App Setup and Configuration URL: After installing the ThreatConnect App for Splunk Enterprise, the application setup must be completed before accessing the App. To properly configure the App, fill in each of the form text boxes with the appropriate data from the API User Creation section, as shown in Figure 4. Figure 4 Application for Splunk Enterprise 8

9 Default Organization: This is the user s Organization as defined during account creation within the ThreatConnect application. API Base URL: The ThreatConnect Public Cloud API can be accessed at Users with a Private Cloud or On-Premises edition of ThreatConnect were provided with their instance URL during their initial setup and installation. API Access ID: The API Access ID corresponds with a user s ThreatConnect API account s Access ID. API Secret Key: The API Secret Key corresponds with a user s ThreatConnect API account key, accessible during account creation within the user s ThreatConnect organization. Activity Log: The Activity Log checkbox enables activity logging in the ThreatConnect Platform for any API write actions. Result Limits: These are the maximum value of results that can be returned in one query without pagination. The maximum value currently supported with the ThreatConnect API for returned results without pagination is 500. Unless there is a special requirement on a user s network, it is recommended this value be left at 500. Enable Proxy: This checkbox enables proxy-server support. Enable Debug: This checkbox enables debug logging for the ThreatConnect App for Splunk Enterprise. After the setup is complete, the ThreatConnect App for Splunk Enterprise will be accessible. When accessing the App, the default dashboard will not show any populated results, which is expected, until matched data is available. The ThreatConnect App for Splunk Enterprise will begin matching against a set of defined Common Information Model (CIM) data models upon setup. For custom matching, Alerts must be created to specify which log and event data within Splunk to match against. It is recommended that users create specific Alerts for their environments. Owner Configuration URL: After setting up the app, users may want to specify filters for the intelligence and indicators imported from ThreatConnect. The Owner Configuration page allows users to selectively choose what is imported into Splunk for alerting and context and how often it is updated. To create filters, provide criteria in each of the fields depicted on the Owner Configuration screen (Figure 5). Application for Splunk Enterprise 9

10 Figure 5 Select Owner: Select the Owner (Source, Community, or a user s ThreatConnect Organization s data) in which the download setting needs to be updated. Group Types: Select the ThreatConnect Group types to download for use in the Diamond Dashboard. Available options are Adversary, Incident, and Threat. Indicator Types: Select the Indicator types to download for use by the ThreatConnect App for Splunk Enterprise. Available options are: Address, , File, Host, and URL. Tag Filters: Add any Tags to filter indicators that are available to the ThreatConnect App for Splunk Enterprise. If Tags are provided, only indicators that have those Tags present will be downloaded into the App. Rating Filter: Select an indicator threat-rating threshold. In ThreatConnect, Threat ratings have a value of 0-5 points. Only indicators that meet the filter s threshold will be downloaded into the App. Confidence Filter: Select an indicator confidence threshold. In ThreatConnect, Confidence ratings have a value of 0-100%. Only indicators that meet the filter s threshold will be downloaded into the App. Cron Schedule: The schedule for the Indicator download is defined here. The recommended download period is once every 24 hours. More information on Cron settings can be found at: Application for Splunk Enterprise 10

11 Setting Up Alerts Any search that returns a set of indicators can be saved as an alert that automatically searches for matching indicators from ThreatConnect. The search can return any single indicator type or multiple indicator types that are supported by the ThreatConnect API. There are two options to implement these automated ThreatConnect queries once the search has been created: Option 1 Once the search command is returning valid indicator results, the results need to be sent to the ThreatConnect script; provide the indicator field names in comma-separated format in the ioc_map option (e.g., sourcetype=cisco:asa action=drop stats count by src_ip where count > 500 rename src_ip as indicator tcalert ioc_map= src_ip victim_field= dest_ip ). The victim field argument is optional but highly recommended when possible. Option 2 (Legacy) 1. Use the Splunk built-in rename command to rename the indicator field to indicator (e.g., sourcetype=cisco:asa action=drop stats count by src_ip where count > 500 rename src_ip as indicator). 2. Once the search command is returning valid indicator results with a field name of indicator, the results need to be sent to the ThreatConnect script (e.g., sourcetype=cisco:asa action=drop stats count by src_ip where count > 500 rename src_ip as indicator tcalert). It is recommended that the search be run over the past hour of logs, and that the alert be set to run every hour. If any matches are found, a log entry will be created in the ThreatConnect App log, which can be viewed from the App s ThreatConnect Dashboard or by executing a search on the ThreatConnect sourcetype (e.g., sourcetype=threatconnect match=true). All non-matching calls to the ThreatConnect script will also create a log entry, which can be viewed by searching the ThreatConnect sourcetype (e.g., sourcetype=threatconnect match=false). The non-matching events are not displayed on the dashboard. Basic Alert Scenario In this example, a company has a requirement to monitor Internet traffic sourced from China that is coming into its network. The search could be as simple as: sourcetype=cisco:asa iplocation src_ip search Country=China rename src_ip as indicator. However, this could create a lot of results and trigger too many lookups. We can filter these results down to all traffic sourced from China that is not blocked by the firewall by adding an additional filter. The new search would be: sourcetype=cisco:asa action=permit iplocation src_ip search Country=China. This Application for Splunk Enterprise 11

12 search will now return less results. The search can be filtered further to exclude duplicate IP addresses by using the stats command to get the count of events by src_ip. To limit the results even further, a threshold can be added to the search using the where command. The new search would then be: sourcetype=cisco:asa action=permit iplocation src_ip search Country=China stats count by src_ip where count > 50.Now that the search results have been filtered to a comfortable level, they can be passed to the ThreatConnect script. 1. Ensure that the ' tcalert ioc_map= src_ip victim="dest_ip"' has been appended to the search (Figure 6). Figure 6 2. Click on the Save As button, select Alert from the drop-down menu, and the Save As Alert screen will appear (Figure 7). Note: The alert title can be any value; however, it is recommended to add a meaningful name for ease of recognition. Adding a description will also help others understand the purpose of the search. Figure 7 3. Ensure that the search time period is equal to the frequency of the Alert so that results are not missed. 4. Once the parameters on this screen have been entered, click the Next button to continue configuring the Alert (Figure 8). Application for Splunk Enterprise 12

13 Figure 8 5. Finalize the Alert configuration and click the Save button. This is a basic sample of creating a search to run as an Alert. There are unlimited combinations of searches that can be created to return indicators that are meaningful to threat analysis. These searches can then be configured as an Alert to automatically match indicators from ThreatConnect. All that users require is knowledge of their security logs and an understanding of what type of events can return useful data for threat analysis. The ThreatConnect Dashboard URL: The ThreatConnect Dashboard provides an overview of matches between events in Splunk and indicator data in ThreatConnect. The first row of Single Valued Results provides a count of matched indicators with trending for the past 30 days. These indicators are separated by Indicator Type (Figure 9). Figure 9 Application for Splunk Enterprise 13

14 The drilldown action for these results directs the user to the Splunk search that generated the count. Note that the count deduplicates the indicator by each API lookup, so that the number of events might be different depending on the number of communities to which the indicator belongs. See The Search Screen section for more information on viewing the generated logs from matched indicators. The second row provides a timeline view of Matched Indicator by Type (Figure 10), which offers an oversight of indicator activity in Splunk. Figure 10 The view at the bottom displays the latest matched indicators in a paginated table (Figure 11). This table has a built-in form that allows dynamic filtering on indicator data. Figure 11 Note: A matching event may display several times if the matched indicator has several owners in ThreatConnect. In cases such as these, a row will be displayed for each owner of the indicator. Each column is described below: _time: This column is stored internally in UTC format. It is translated to human-readable UNIX time format when Splunk Enterprise renders the search results (the very last step of search-time event processing). Indicator: This column lists the indicator that matched between local logs and ThreatConnect. This value is a hyperlink that will open a page to the indicator s Details screen on the ThreatConnect website. Application for Splunk Enterprise 14

15 Victim: This column represents the bottom of the Diamond Model. This value is determined by the user while setting up alerts or automatically when searching the Data Models. Trigger: The trigger field is classified into three types: alert, auto, or manual. The alert trigger indicates that the matched event was found using a Splunk search configured as an alert. The auto trigger indicates that the matched event was found using the automatic Data Model search. The manual trigger indicates that the event was either matched using a workflow action, or that the indicator was searched using the manual search option. Rating: This column lists the criticality rating assigned by the indicator s owner within ThreatConnect. This value is on a scale of 0-5 points, with 5 being the most critical. The ratings are color highlighted by their value ( no highlight; 3-4 amber; and > 4 red). Confidence: This column lists the confidence rating assigned by the indicator s owner within ThreatConnect. This value is on a scale of 0-100%. Owner: This column displays the owner of the indicator within ThreatConnect, which is, typically, a particular source, community, or the organization s own private data. Tags: This column displays any tags associated with the matched indicator by the owner. If multiple owners exist for a matched indicator, only tags created by the owner listed in the same row will be displayed in this column. Groups: This column display any groups associated with the matched indicator by the owner, as well as the group type (e.g., Incidents, s, Threats, s, Adversaries, Documents, or Signatures). If multiple owners exist for a matched indicator, only groups created by the owner listed in the same row will be displayed in the Groups (Types) column. Application for Splunk Enterprise 15

16 The Indicator Dashboard URL: The Indicator Dashboard provides an additional view of the matched indicator data. This dashboard focuses on the groupings of the matched indicators themselves and provides some statistics on the most-recent indicators available from ThreatConnect. The first row of this timeline view (Figure 12) displays Matched Indicators by Owners (i.e., Source, Community, and Organization). Figure 12 The second row displays additional paginated tables for matched indicators (Figure 13). From left to right, these are the top matched indicators (with a count of times observed for each indicator), the top matched group names, and the top matched tags. Figure 13 Application for Splunk Enterprise 16

17 The Diamond Dashboard URL: This page allows users to lookup an Adversary, Incident, or Threat group from a specified owner in ThreatConnect. A Diamond Model profile of the group is given to include all known capabilities, infrastructure, and related incidents. Indicators associated to the group are matched using specified CIM Data Models. The first section of the dashboard allows the user to query a group with specified data models (Figure 14). Figure 14 The query parameters are specified by the following fields: Data Models: Specified CIM Data Models to match events against the group s indicators. Available data models include: Alerts, , Intrusion Detection, Malware, Network Resolutions, Network Sessions, Network Traffic, and Web. Owner: Select the Owner (i.e., Source, Community, or Organization) in which the group exists. Type: Select the Group Type (i.e., Adversary, Incident, or Threat). Name: Select the specific Group by name. Application for Splunk Enterprise 17

18 The second section of the dashboard shows a Diamond breakout of the queried group (Figure 15). The Adversary/Threat table shows Adversary and Threat associations to the queried group. The Associations table shows all associated Documents, s, Incidents, and Signatures to the queried group. The Capability table shows all known associated Common Vulnerabilities and Exposures (CVE) and malware files by MD5 hash. The Infrastructure table shows all Address, , Host, and URL indicators associated to the queried group. Figure 15 The third section of the dashboard shows a timeline and table of CIM events matched to the queried group (Figure 16). Figure 16 Application for Splunk Enterprise 18

19 The Threat Lookup Screen URL: This screen allows manual lookup of indicators against the ThreatConnect API (Figure 17). The indicator type will be automatically detected. This manual lookup will be logged to the App logs and will be displayed on the App Dashboard. The Reports Screen Figure 17 URL: A few canned reports are created for monitoring ThreatConnect alert queries and for tracking how many of those queries hit the ThreatConnect API (Figure 18). User-defined custom reports can be added for more detailed views into the ThreatConnect data. Figure 18 Application for Splunk Enterprise 19

20 The DB Data Menu The DB Data drop-down menu provides additional screens for each ThreatConnect indicator type, independent of matches to events or logs within Splunk, allowing users to view statistics for each. The screens are formatted identically, and one screen exists for each indicator type: Address Indicators DB Data Page URL: Addresses DB Data Page URL: File Indicators DB Data Page URL: Host Indicators DB Data Page URL: URL Indicators DB Data Page URL: The first row displays graphical representations for the total number of indicators from ThreatConnect of the specified type (Figure 19). There are two charts: one for Indicator type by owner and another by rating. Figure 19 Application for Splunk Enterprise 20

21 The second row contains two tables that display the last 10 created and updated indicators (Figure 20). Figure 20 The final row displays a paginated table of all the indicators of that type pulled from ThreatConnect (Figure 21). The Search Screen Figure 21 URL: Workflow: Event Actions Using the Search screen while in the ThreatConnect App provides additional features for threat analysis. The built-in Splunk Event Actions feature will display a link for any indicator field that follows the Common Information Model (CIM) standard naming convention. By selecting this link, the indicator displayed will be queried using the ThreatConnect API, and any matching results will be displayed in a results table (Figure 22). Application for Splunk Enterprise 21

22 Figure 22 Application for Splunk Enterprise 22

23 Workflow: Field Actions If the results fields do not follow the CIM naming convention, an indicator can still be queried in the ThreatConnect API by using the Field Actions menu. By selecting this link, the indicator displayed will be queried using the ThreatConnect API, and any matching result will be displayed in a results table (Figure 23). Figure 23 Application for Splunk Enterprise 23

24 The ThreatConnect App Logs The ThreatConnect App logs can be searched just like any other log in Splunk. The App logs are formatted in JavaScript Object Notation (JSON) and are easily readable by humans and machines. Indicator Logs All indicators that are passed to the ThreatConnect script are logged. Figure 24 displays a matching result in its native, collapsed form. Figure 24 Clicking on the plus sign (+) symbol in the Event column will expand the event to display additional data. Figure 25 displays a fully expanded matching-indicator result in the ThreatConnect App log. All the indicator data is nested in the indicator section of the event. All the search information, including the search string that created this log entry, is nested in the search section of the event. A key field in this data is search.method, which indicates if the search was performed manually by using the ThreatConnect Manual Lookup, by Workflow Action, or by an alert. Each field from the log below is described as follows: app: This field notates the ThreatConnect App as the source of the log. Indicator: This field is nested with data on the matched indicator from ThreatConnect s API. indicator.confidence: This field displays the owner-specific confidence value on the indicator provided from ThreatConnect s API. indicator.dateadded: This field displays the date the indicator was added to the owner within ThreatConnect. indicator.description: This field includes the owner-specific default-description attribute of the indicator from ThreatConnect. indicator.id: This field displays the indicator s id value from ThreatConnect. indicator.indicator: This field specifies the actual indicator value from ThreatConnect. Application for Splunk Enterprise 24

25 indicator.lastmodified: This field notates the time the indicator was last modified within ThreatConnect. indicator.owner: This field is nested with data on the indicator s owner from ThreatConnect. indicator.owner.id: This field displays the owner s id value from ThreatConnect. indicator.owner.name: This field includes the name of the indicator s owner within ThreatConnect. indicator.owner.type: This field specifies the type of owner. indicator.rating: This field notates the owner-specific criticality rating of the indicator. indicator.source: This field displays the owner-specific default-source attribute value for the indicator from ThreatConnect. indicator. This field is present if the matched indicator is an address. Its value will be the address indicator. indicator.hostname: This field is present if the matched indicator is a host name. Its value will be the host indicator. indicator.ip: This field is present if the matched indicator is an IP address. Its value will be the IP address indicator. indicator.md5: This field is present if the matched indicator is an MD5. Its value will be the MD5 hash indicator. indicator.sha1: This field is present if the matched indicator is a SHA1. Its value will be the SHA1 hash indicator. indicator.sha256: This field is present if the matched indicator is a SHA256. Its value will be the SHA256 hash indicator. indicator.text: This field is present if the matched indicator is a URL. Its value will be the URL indicator. indicator.type: This field specifies the type of indicator that was automatically determined by the ThreatConnect script. indicator.weblink: This field displays a unique URL link to the indicator s Details screen within the ThreatConnect platform. match: A value of True in this field signifies that the searched indicator matched a known indicator in ThreatConnect. A value of False in this field signifies that the searched indicator did not match a known indicator in ThreatConnect. search: This field is nested with data relevant to the search made within the ThreatConnect App. search.indicator: This field will be identical to the indicator.indicator on an event where match=true. Events that have match=false will not have an indicator section. Application for Splunk Enterprise 25

26 search.method: A value of alert in this field indicates that the log event was triggered by an alert. A value of manual in this field indicates that the log event was generated from a ThreatConnect Manual Lookup or a Workflow Action. search.owners: This field notates the ThreatConnect owners that were searched for the value in search.indicator. search.string: This field includes the search string that created this event. search.type: This field specifies the type of search performed in the ThreatConnect App. The values can be indicators, groups, or tags. timestamp: This field displays the timestamp for this logged event. uuid: This field displays the universally unique id value for this logged event. The uuid field can be used to join indicator events with matching tag and group events for the same search. Figure 25 Application for Splunk Enterprise 26

27 Figure 26 displays a fully expanded indicator event that did not match during the ThreatConnect script execution. The key field to note is match: False. This information is useful in tuning the alert to filter out additional indicators. It also provides a view into what is defined as a relevant indicator to trigger an alert. Figure 26 Group Logs Each time a matching indicator is found, the ThreatConnect script performs a query to find any matching groups that are related to the matching indicator. If matching groups are found, the group data will be written to the App log file. Figure 27 displays a fully expanded log entry for a group. This group can be associated with the matching indicator log entry using the uuid field. Each field unique to the group log is described as follows: group: This field is nested with data relevant to a group associated to a matched indicator. group.dateadded: This field displays the date on which an owner-specific group was added to ThreatConnect. group.id: This field displays the unique id of the group within ThreatConnect. group.name: This field specifies the name of the group within ThreatConnect. group.ownername: This field displays the name of the owner of the group within ThreatConnect. group.type: This field specifies the type of group. group.weblink: This group displays a URL link to the group s Details screen within ThreatConnect. Application for Splunk Enterprise 27

28 Figure 27 Tag Logs Each time a matching indicator is found, the ThreatConnect App performs a query to find any matching tags that are related to the matching indicator. If matching tags are found, the tag data will be written to the App log file. Figure 28 displays a fully expanded log entry for a tag. This tag can be associated with the matching indicator log entry using the uuid field. Each field unique to the group log is described as follows: tag: This field is nested with data relevant to a tag associated to a matched indicator. tag.name: This field displays the name of the tag within ThreatConnect. tag.weblink: This field displays a URL link to the tag s Details screen within ThreatConnect. Application for Splunk Enterprise 28

29 Figure 28 Application for Splunk Enterprise 29