QRadar SIEM 7.2 Flows Overview

Transcription

1 QRadar SIEM 7.2 Flows Overview Panelists Dwight Spencer Principal Solutions Architect & Co-founder of Q1 Labs Aaron Breen QRadar World-wide Support Leader Adam Frank Principal Solutions Architect Dale Beresford Support Services Team Lead Jonathan Pechta Support Technical Writer Reminder: You must dial-in to the phone conference to listen to the panelists. The web cast does not include audio. USA: Canada: Participant passcode: QRadar Open Mic Webcast #2 July 29, IBM Corporation

2 Goal: Provide insight on the QRadar components responsible for flow collection. Router sending NetFlow data ETH0 ETH1 ETH2 ETH3 Console Span port on a switch Network tap QFlow Collector (12xx or 13xx) Flow records port ETH0 Flow Processor (FP) (17xx) 2

3 Types of flow data QRadar can collect several types of flow data: QFlow, NetFlow, SFlow, JFlow, and Packeteer. We differentiate these into two categories: Internal flows: packet based collection (QFlow or Packeteer) External flows: sources from routers or switches that generate their own session statistics (NetFlow, SFlow, and JFlow) Data available by flow type: QFlow or Packeteer layer 7 visibility, provides details on application communication, URLs, etc. NetFlow, JFlow, and SFlow layer 3 and layer 4 visibility. 3

4 Placement of devices for flow collection 1. Where do you require visibility? Most volatile If you are to consider the perimeter of the network, where a corporate entity connects to the public internet as the most volatile location, then this is where most users will put the most granular data collection type, which is a packet based collection solution. 2. Available hardware & capabilities Least volatile 4

5 The Event Correlation Service (ECS) The QFlow component is responsible for reading different types of flow data and creates flow records to be processed. Start Flow data (internal or external) QFlow process ECS is the core service responsible for event and flow collection for QRadar. ECS is comprised of three core components: Event Collector component 1. Event Collector 2. Event Processor ECS 5 Event Processor component Magistrate component (Console only) End 3. Magistrate

6 What is an Event Collector component? The Event Collector component completes a number of flow processing functions for ECS. Flow deduplication: Flow deduplication is a process that removes duplicate flows when multiple QFlow collectors are providing data to flow processor appliances. Asymetric recombination: Responsible for combining two sides of each flow when data is provided asymmetrically. This process can recognize flows from each side and combine them in to one record. However, sometimes both sides of the data do not exist. External flow sources such as NetFlow that may only report ingress or egress traffic. Instances where span traffic enters a network from a single point, and exists via another, creating asymmetric reporting of data to flow collectors. Event Collector Flow deduplication Asymetric recombination Throttle Forwarding Throttle: Monitors the number of incoming events & flows to the system to manage input queues and licensing. Forwarding: Applies routing rules for the system, such as sending data to offsite targets, external Syslog systems, JSON systems, other SIEMs, etc. 6

7 What is an Event Processor component? Custom Rules Engine (CRE): The Custom Rules Engine (CRE) is responsible for processing events received by QRadar and comparing them against defined rules, keeping track of systems involved in incidents over time, generating notifications to users and generating offenses. Host profiler: Responsible for resolving asset information from passive flow data. Flows provide detailed information about network activity and allow QRadar to build a passive database on assets, ports, protocols, direction, applications, number of packets, bytes transferred, and even an index of the source and destination payload. Event Processor Custom Rules Engine Host profiler Streaming Storage Streaming: Responsible for sending real-time event data to the Console when a user is viewing events from the Log Activity tab with Real time (streaming). Streamed events are not provided from the database. Event storage (Ariel): A time series database for events and flows where data is stored on a minute by minute basis. Data is stored where the event is processed. 7

8 What is the Magistrate component? (MPC) component? The Magistrate Processing Core (MPC) is responsible for correlating offenses with event notifications from multiple Event Processor (EP) components. Only the Console will have a Magistrate component. Layers Offense rules: Monitors and takes actions on offenses, such as generating notifications. Offense management: Updates active offenses, transitioning inactive offenses to active and provides access to offense information to the user through the Offenses tab. Magistrate Offense rules Offense management Offense storage Offense storage: Writes offense data to a Postgres database. 8

9 ECS, the big picture Start End Protocol Parsing, traffic analysis, and auto detection Event Collector Throttle Coalescing Forwarding Event Processor Custom Rules Engine Host profiler Streaming Storage Magistrate Offense rules Offense management Offense storage Flow deduplication Asymetric recombination Remember: ECS runs on any appliance that processes events, such as 16xx, 17xx, and 18xx appliances. This means that ECS is running simultaneously on a number of appliances in a multi-system deployment. Each ECS is taking in events, processing them, evaluating rules, etc. 9

10 Types of flow records Standard flow: A single standard flow record Type A Superflow (Network scans): One source to many destination IPs This is a unidirectional flow, which has the same source, but multiple destinations. Type B Superflow (DDoS): Multiple sources to a single destination IP This is a unidirectional flow, which has the multiple sources, but has a single destination. Type C Superflow (Port scans): One-to-one source and destination with many ports This is a one-to-one flow with different source or destination ports Over Flow record: Created when license limits are exceeded When a QFlow collector hits its flow license limit, it begins creating over flow records. Over flow records have a source IP of and a destination IP of with one flow created per protocol (icmp, udp, tcp, etc). When the license limit is reached, QFlow rolls the rest of the traffic for the protocol within the interval in to a single record. All bytes and packet counts are totaled up and added to these overflow records. Flow bundle subflow record: Legacy no longer used 10

11 Advanced questions: part 1 The first questions addressed by the panelists will be these that were asked in advance in the QRadar Customer forum. Q1: How can I achieve application layer visibility using QRadar All-in-one appliance? Do I need to connect SPAN port to a NIC which is used by QFlow collector? Q2: How will encryption be dealt with here (I mean whether we need any SSL termination proxy or not )? Q3: How can Layer-7 analysis be made possible in case of VFlow collectors since we don't have any SSL termination mechanism in virtual environments? Q4: After VFlow is configured, to which component will it be forwarded? Q5: What is the packet structure of QFlow records? 11

12 Advanced questions: part 2 Q6: How does QFlow differentiate between applications that are using the same port? (For example, port 443 being used by Facebook and LinkedIn) How is an IRC running on port 80 or 443 identified? Q7: Feeding un-encrypted data to QFlow puts limitations on me as my QRadar administrators/analysts can have my company's s retrievable in their consoles. Does QRadar have capability to re-encrypt its input just like an IPS or IDS or any application layer filtering platform? 12

13 Questions for the panel? Now is your opportunity to ask questions of our panelists. To ask a question now: 1. Type your question into the chat window. 2. When prompted by the operator, you can press *1 to ask a question over the phone. To ask a question after this presentation: If you were unable to attend this webcast or have questions later, we have set aside a forum post specifically for this webcast. See the IBM Security Intelligence QRadar Forum. 13

14 Where do I get more information? Questions on this or other topics can be directed to the QRadar forums: IBM Security Intelligence QRadar Forum. More articles you can review: Article : QRadar Licenses and Flow Data Article : What are flows with the source and destinations of and ? Article : Common message and errors from the QRadar flow pipeline Useful links : Getting Support for IBM Security QRadar products Follow us: IBM Support Portal Open a Service Request Update your PMR Escalate your PMR 14

15 ibm.com/security 15 Copyright IBM Corporation All rights reserved. The information contained in these materials is provided for informational purposes only, and is provided AS IS without warranty of any kind, express or implied. IBM shall not be responsible for any damages arising out of the use of, or otherwise related to, these materials. Nothing contained in these materials is intended to, nor shall have the effect of, creating any warranties or representations from IBM or its suppliers or licensors, or altering the terms and conditions of the applicable license agreement governing the use of IBM software. References in these materials to IBM products, programs, or services do not imply that they will be available in all countries in which IBM operates. Product release dates and/or capabilities referenced in these materials may change at any time at IBM s sole discretion based on market opportunities or other factors, and are not intended to be a commitment to future product or feature availability in any way. IBM, the IBM logo, and other IBM products and services are trademarks of the International Business Machines Corporation, in the United States, other countries or both. Other company, product, or service names may be trademarks or service marks of others.