Juniper Secure Analytics

Transcription

1 Juniper Secure Analytics Big Data Management Guide Release Published:

2 Juniper Networks, Inc North Mathilda Avenue Sunnyvale, California USA All rights reserved. Juniper Networks, Junos, Steel-Belted Radius, NetScreen, and ScreenOS are registered trademarks of Juniper Networks, Inc. in the United States and other countries. The Juniper Networks Logo, the Junos logo, and JunosE are trademarks of Juniper Networks, Inc. All other trademarks, service marks, registered trademarks, or registered service marks are the property of their respective owners. Juniper Networks assumes no responsibility for any inaccuracies in this document. Juniper Networks reserves the right to change, modify, transfer, or otherwise revise this publication without notice. Juniper Secure Analytics Big Data Management Guide All rights reserved. The information in this document is current as of the date on the title page. YEAR 2000 NOTICE Juniper Networks hardware and software products are Year 2000 compliant. Junos OS has no known time-related limitations through the year However, the NTP application is known to have some difficulty in the year END USER LICENSE AGREEMENT The Juniper Networks product that is the subject of this technical documentation consists of (or is intended for use with) Juniper Networks software. Use of such software is subject to the terms and conditions of the End User License Agreement ( EULA ) posted at By downloading, installing or using such software, you agree to the terms and conditions of that EULA. ii

3 Table of Contents About the Documentation vii Documentation and Release Notes vii Documentation Conventions vii Documentation Feedback ix Requesting Technical Support x Self-Help Online Tools and Resources x Opening a Case with JTAC x Part 1 Juniper Secure Analytics Big Data Management Chapter 1 JSA and Big Data Integration Configuring JSA and Big Data Integration Integration Components Example: Using Big Data Technologies for Advanced Data Threats Installing the Data Management Application Server Plug-In and Reference Data API Adding Forwarding Destinations Customizing the Forwarding Profile Configuring Routing Rules for Bulk Forwarding Part 2 Index Index iii

4 Juniper Secure Analytics Big Data Management Guide iv

5 List of Tables About the Documentation vii Table 1: Notice Icons viii Table 2: Text and Syntax Conventions viii Part 1 Juniper Secure Analytics Big Data Management Chapter 1 JSA and Big Data Integration Table 3: Integration Components Table 4: Forwarding Destinations Parameters Table 5: Routing Rules Window Parameters v

6 Juniper Secure Analytics Big Data Management Guide vi

7 About the Documentation Documentation and Release Notes Documentation and Release Notes on page vii Documentation Conventions on page vii Documentation Feedback on page ix Requesting Technical Support on page x Documentation Conventions To obtain the most current version of all Juniper Networks technical documentation, see the product documentation page on the Juniper Networks website at If the information in the latest release notes differs from the information in the documentation, follow the product Release Notes. Juniper Networks Books publishes books by Juniper Networks engineers and subject matter experts. These books go beyond the technical documentation to explore the nuances of network architecture, deployment, and administration. The current list can be viewed at Table 1 on page viii defines notice icons used in this guide. vii

8 Juniper Secure Analytics Big Data Management Guide Table 1: Notice Icons Icon Meaning Description Informational note Indicates important features or instructions. Caution Indicates a situation that might result in loss of data or hardware damage. Warning Alerts you to the risk of personal injury or death. Laser warning Alerts you to the risk of personal injury from a laser. Tip Indicates helpful information. Best practice Alerts you to a recommended use or implementation. Table 2: Text and Syntax Conventions Table 2 on page viii defines the text and syntax conventions used in this guide. Convention Description Examples Bold text like this Represents text that you type. To enter configuration mode, type the configure command: user@host> configure Fixed-width text like this Italic text like this Represents output that appears on the terminal screen. Introduces or emphasizes important new terms. Identifies guide names. Identifies RFC and Internet draft titles. user@host> show chassis alarms No alarms currently active A policy term is a named structure that defines match conditions and actions. Junos OS CLI User Guide RFC 1997, BGP Communities Attribute Italic text like this Represents variables (options for which you substitute a value) in commands or configuration statements. Configure the machine s domain name: [edit] root@# set system domain-name domain-name viii

9 About the Documentation Table 2: Text and Syntax Conventions (continued) Convention Description Examples Text like this Represents names of configuration statements, commands, files, and directories; configuration hierarchy levels; or labels on routing platform components. To configure a stub area, include the stub statement at the [edit protocols ospf area area-id] hierarchy level. The console port is labeled CONSOLE. < > (angle brackets) Encloses optional keywords or variables. stub <default-metric metric>; (pipe symbol) Indicates a choice between the mutually exclusive keywords or variables on either side of the symbol. The set of choices is often enclosed in parentheses for clarity. broadcast multicast (string1 string2 string3) # (pound sign) Indicates a comment specified on the same line as the configuration statement to which it applies. rsvp { # Required for dynamic MPLS only [ ] (square brackets) Encloses a variable for which you can substitute one or more values. community name members [ community-ids ] Indention and braces ( { } ) ; (semicolon) Identifies a level in the configuration hierarchy. Identifies a leaf statement at a configuration hierarchy level. [edit] routing-options { static { route default { nexthop address; retain; } } } GUI Conventions Bold text like this Represents graphical user interface (GUI) items you click or select. In the Logical Interfaces box, select All Interfaces. To cancel the configuration, click Cancel. > (bold right angle bracket) Separates levels in a hierarchy of menu selections. In the configuration editor hierarchy, select Protocols>Ospf. Documentation Feedback We encourage you to provide feedback, comments, and suggestions so that we can improve the documentation. You can provide feedback by using either of the following methods: Online feedback rating system On any page at the Juniper Networks Technical Documentation site at simply click the stars to rate the content, and use the pop-up form to provide us with information about your experience. Alternately, you can use the online feedback form at ix

10 Juniper Secure Analytics Big Data Management Guide Send your comments to Include the document or topic name, URL or page number, and software version (if applicable). Requesting Technical Support Technical product support is available through the Juniper Networks Technical Assistance Center (JTAC). If you are a customer with an active J-Care or JNASC support contract, or are covered under warranty, and need post-sales technical support, you can access our tools and resources online or open a case with JTAC. JTAC policies For a complete understanding of our JTAC procedures and policies, review the JTAC User Guide located at Product warranties For product warranty information, visit JTAC hours of operation The JTAC centers have resources available 24 hours a day, 7 days a week, 365 days a year. Self-Help Online Tools and Resources For quick and easy problem resolution, Juniper Networks has designed an online self-service portal called the Customer Support Center (CSC) that provides you with the following features: Find CSC offerings: Search for known bugs: Find product documentation: Find solutions and answer questions using our Knowledge Base: Download the latest versions of software and review release notes: Search technical bulletins for relevant hardware and software notifications: Join and participate in the Juniper Networks Community Forum: Open a case online in the CSC Case Management tool: To verify service entitlement by product serial number, use our Serial Number Entitlement (SNE) Tool: Opening a Case with JTAC You can open a case with JTAC on the Web or by telephone. Use the Case Management tool in the CSC at Call JTAC ( toll-free in the USA, Canada, and Mexico). x

11 About the Documentation For international or direct-dial options in countries without toll-free numbers, see xi

12 Juniper Secure Analytics Big Data Management Guide xii

13 PART 1 Juniper Secure Analytics Big Data Management JSA and Big Data Integration on page 3 1

14 Juniper Secure Analytics Big Data Management Guide 2

15 CHAPTER 1 JSA and Big Data Integration This chapter describes about the following sections: Configuring JSA and Big Data Integration on page 3 Installing the Data Management Application Server Plug-In and Reference Data API on page 4 Adding Forwarding Destinations on page 5 Customizing the Forwarding Profile on page 6 Configuring Routing Rules for Bulk Forwarding on page 7 Configuring JSA and Big Data Integration To build insights from broader data sets, integrate Juniper Secure Analytics (JSA) with vendor data management applications, such as IBM InfoSphere BigInsights. Integration process involves the following steps: 1. Install and configure your data management application on a separate server from JSA. For more information about how to install and configure your application, see your vendor documentation. 2. Configure JSA to forward raw data to the server. 3. Use the data management application to transform and enrich the data for analytic information. 4. Configure the API to send the analytic insights back to JSA. Integration Components Table 3 on page 4 lists the components that are used to integrate the data management application with JSA. 3

16 Juniper Secure Analytics Big Data Management Guide Table 3: Integration Components Components Description Data management application A software application that processes and analyzes the volume, variety, and velocity of data. You must purchase your data management application separately. Although JSA can integrate with any data management application that supports Apache Hadoop, supports IBM InfoSphere BigInsights. Reference data API A utility that JSA uses to retrieve data from a data management application. Forwarding destination A JSA component that specifies the server that hosts the data management application. Forwarding profile A JSA file that is automatically generated when you create a forwarding destination that uses the JSON format. Routing rule A JSA component that you use to create filter-based routing rules to forward large quantities of data. Use routing rules to specify what data is forwarded to the forwarding destination. Example: Using Big Data Technologies for Advanced Data Threats Use JSA and big data management application integration to periodically identify suspicious external domains that your systems or users are connecting to. This suspicious activity is based on inconsistencies in registration info and the correlation between actual data that is associated with the domain and actual data that is associated with some external malware domains. Your data management application receives and analyzes data from JSA and external data sources. The results are then sent back to your JSA system. Related Documentation Configuring Routing Rules for Bulk Forwarding on page 7 Customizing the Forwarding Profile on page 6 Adding Forwarding Destinations on page 5 Installing the Data Management Application Server Plug-In and Reference Data API on page 4 Installing the Data Management Application Server Plug-In and Reference Data API To install the data management application server Plug-In and reference data API: 1. Log in to your Juniper Secure Analytics (JSA) console. 2. On the Admin tab, perform the following steps: a. Click User Roles and create a dedicated user role that includes the Reference Data API role permission. b. Click Authorized Services and create add an authorized service and assign the authorized service to the dedicated user role. 4

17 Chapter 1: JSA and Big Data Integration 3. Download the data management application plug-in and Reference Data API RPM files from Install the plug-in and API on your JSA console. Related Documentation Configuring Routing Rules for Bulk Forwarding on page 7 Customizing the Forwarding Profile on page 6 Adding Forwarding Destinations on page 5 Installing the Data Management Application Server Plug-In and Reference Data API on page 4 Adding Forwarding Destinations Before you can configure bulk or selective data forwarding, you must add forwarding destinations. To add the forwarding destinations: 1. Click the Admin tab. 2. In the navigation pane, click System Configuration. 3. Click the Forwarding Destinations icon. 4. On the toolbar, click Add. 5. In the Forwarding Destinations window, enter values for the parameters. Table 4 on page 5 describes the forwarding destinations parameter. Table 4: Forwarding Destinations Parameters Parameter Description Event Format Payload is the data in the format that the log source or flow source sent. Payload is the data in the format that the log source sent. Normalized is raw data that is parsed and prepared as readable information for the user interface. JSON (Javascript Object Notation) is a data-interchange format. NOTE: JSON data can only be transmitted using the TCP protocol. Destination Address The IP address or host name of the vendor system that you want to forward data to. Protocol TCP Use the TCP protocol to send normalized data by using the TCP protocol, you must create an off-site source at the destination address on port UDP NOTE: You cannot transmit normalized and JSON data by using the UDP protocol. If you select the Normalized Event or JSON options, the UDP option in the Protocol list is disabled. 5

18 Juniper Secure Analytics Big Data Management Guide Table 4: Forwarding Destinations Parameters (continued) Parameter Description Prefix a syslog header if it is missing or invalid If a valid syslog header is not detected on the original syslog message, select this check box. The prefixed syslog header includes the JSA appliance host IP address in the Hostname field of the syslog header. If this check box is not selected, the data is sent unmodified. When JSA forwards syslog messages, the outbound message is verified to ensure that it has a valid syslog header. 6. Click Save. Related Documentation Configuring Routing Rules for Bulk Forwarding on page 7 Customizing the Forwarding Profile on page 6 Installing the Data Management Application Server Plug-In and Reference Data API on page 4 Customizing the Forwarding Profile Customize the forwarding profile that is automatically created when you create a forwarding destination that sends data in the JSON format. When you create a forwarding destination that sends data in JSON format, a forwarding profile is automatically created and stored in the /opt/qradar/conf folder on your system. The file name for each profile is appended with a number that relates to the forwarding destination ID. For example, forwardingprofile_jason.xml.1. You can edit the forwarding profile to transform and enrich the data to provide more information, which supports the advanced analytical processes. The forwarding profile controls what data is forwarded, how default settings are applied, and whether to include custom properties. You can customize the name and version of the profile, and edit the pre-amble element. The preamble contains information that is required for the forwarding profile to interact with multiple receivers. For each attribute listed in the forwarding profile, you can also customize the properties. The following is an example of a flow-specific attribute: <attribute tag="firstpackettimeiso" enabled="true" name="firstpackettimeiso" defaultvalue="" enabledefaultvalue="false"></attribute> Use the enabled property to specify that you want to send the field to the forwarding destination. Update the name property from the automatically generated name. 6

19 Chapter 1: JSA and Big Data Integration Specify the default Value property for the field. Use the enable Default Value property to enable or disable the default Value property. The JSON forwarding mechanism sends only default values for a field if the enable Default Value is set to true. To customize and forward the profile: 1. Log in to your system. 2. Change directories to /opt/qradar/conf. 3. Locate and open the forwarding profile file that you want to customize. 4. Customize the forwarding profile by using the following options: Edit the profile level options, such as the name, version, or pre-amble. To forward custom properties, ensure that the include Custom Properties property is True in the Header section. To configure the time format, edit the iso Time Format attribute in the Header section. In each of the attribute sections, edit the attributes that you want to enhance. 5. Close the file. Related Documentation Configuring Routing Rules for Bulk Forwarding on page 7 Customizing the Forwarding Profile on page 6 Adding Forwarding Destinations on page 5 Installing the Data Management Application Server Plug-In and Reference Data API on page 4 Configuring Routing Rules for Bulk Forwarding After you added one or more forwarding destinations, you can create filter-based routing rules to forward large quantities of data. You can configure routing rules to forward data in either online or offline mode: In Online mode, your data remains current because forwarding is performed in real time. If the forwarding destination becomes unreachable, data can potentially be lost. In Offline mode, all data is stored in the database and then sent to the forwarding destination. This assures that no data is lost, however, there might be delays in data forwarding. Table 5 on page 8 describes the Routing Rules parameter. 7

20 Juniper Secure Analytics Big Data Management Guide Table 5: Routing Rules Window Parameters Parameter Description Forwarding Event Collector This option is displayed when you select the Online option. Specifies the Event Collector that you want this routing rule process data from. Forwarding Event Processor This option is displayed when you select the Offline option. Specifies the Event Processor that you want this routing rule process data from. NOTE: This option is not available if Drop is selected from the Routing Options pane. Routing Options The Forward option specifies that data is forwarded to the specified forwarding destination. Data is also stored in the database and processed by the Custom Rules Engine (CRE). The Drop option specifies that data is not stored in the database and is not processed by the CRE. The data is not forwarded to a forwarding destination, but it is processed by the CRE. This option is not available if you select the Offline option. The Bypass Correlation option specifies that data is not processed by the CRE, but it is stored in the database. This option is not available if you select the Offline option. You can combine two options: Forward and Drop Data is forwarded to the specified forwarding destination. Data is not stored in the database and is processed by the CRE. Forward and Bypass Correlation Data is forwarded to the specified forwarding destination. Data is also stored in the database, but it is not processed by the CRE. The CRE at the forwarded destination processes the data. If data matches multiple rules, the safest routing option is applied. For example, if data that matches a rule that is configured to drop and a rule to bypass CRE processing, the data is not dropped. Instead, the data bypasses the CRE and is stored in the database. All events are counted against the EPS license. To Configure the routing rules for bulk forwarding: 1. Click the Admin tab. 2. In the navigation pane, click System Configuration. 3. Click the Routing Rules icon. 4. On the toolbar, click Add. 5. In the Routing Rules window, enter values for the parameters. a. Type a name and description for your routing rule. b. From the Mode field, select one of the following options: Online or Offline. c. From the Forwarding Event Collector or Forwarding Event Processor list, select the event collector from which you want to forward data. d. From the Data Source field in the Event Filters section, select which data source you want to route: Events or Flows. 8

21 Chapter 1: JSA and Big Data Integration If you select the Flow Filters option, the section title changes to Flow Filters and the Match All Incoming Events check box changes to Match All Flows. e. To forward all incoming data, select the Match All Incoming Events or Match All Incoming Flows check box. NOTE: If you select this check box, you cannot add a filter. f. To add a filter, in the Event Filters or Flow Filters section, select a filter from the first list and an operand from the second list. g. In the text box, type the value that you want to filter for, and then click Add Filter. h. Repeat the previous two steps for each filter that you want to add. i. To forward log data that matches the current filters, select the Forward check box, and then select the check box for each preferred forwarding destination. NOTE: If you select the Forward check box, you can also select either the Drop or Bypass Correlation check boxes, but not both of them. If you want to edit, add, or delete a forwarding destination, click the Manage Destinations link. 6. Click Save. Related Documentation Adding Forwarding Destinations on page 5 Installing the Data Management Application Server Plug-In and Reference Data API on page 4 Customizing the Forwarding Profile on page 6 9

22 Juniper Secure Analytics Big Data Management Guide 10

23 PART 2 Index Index on page 13 11

24 Juniper Secure Analytics Big Data Management Guide 12

25 T technical support contacting JTAC...x Index Symbols #, comments in configuration statements...ix ( ), in syntax descriptions...ix < >, in syntax descriptions...ix [ ], in configuration statements...ix { }, in configuration statements...ix (pipe), in syntax descriptions...ix B braces, in configuration statements...ix brackets angle, in syntax descriptions...ix square, in configuration statements...ix C comments, in configuration statements...ix conventions text and syntax...viii curly braces, in configuration statements...ix customer support...x contacting JTAC...x D documentation comments on...ix F font conventions...viii M manuals comments on...ix P parentheses, in syntax descriptions...ix S support, technical See technical support syntax conventions...viii 13

26 Juniper Secure Analytics Big Data Management Guide 14