TUT8173 Best Practices for Security Monitoring in Distributed Environments November 2014 Chris Patzer ZF Norbert Klasen NetIQ
Agenda Sentinel Deployment Scenarios Case Study: ZF Lessons Learned 2
Infrastructure Patterns Single site 3
Distributed Environments So your environment is distributed, does the SIEM infrastructure need to be distributed as well? Required by law/contracts network topology/bandwidth horizontal scale out for performance Desirable for resilience 4
Infrastructure Patterns Large regional sites 5
One-Tier Distributed Architecture 6
Infrastructure Patterns Distributed environments 7
Case Study ZF Group
ZF Group Motion and Mobility 9
Worldwide Presence with Production, Development, Aftermarket Trading and Service 122 production companies Claudia Bucher/HGB 8 main development locations 32 service companies Over 650 service partners worldwide 10
The ZF Group An Overview 2013 Sales $ 21 billion Employees 72,600 Countries 26 In September, ZF agreed to buy TRW Automotive Holdings Corp. (TRW) for $13.5 billion 11
Employees 2012 by Region Western Europe 49,093 thereof in Germany 43,195 Eastern Europe 4,485 North America 6,856 South America 5,235 Asia-Pacific 7,888 Africa 1,218 Total: 74,775 Thereof: Research and Development (R&D) approx. 7,120 Training 1,980 12
SIEM@ZF
Project Definition The project will introduce a SIEM system and integrate systems management ZF was not able to react to external or internal attacks effectively. Log files were not collected centrally and there was no possibility to correlate events or provide forensic information. Representative systems from all zones and regions of the IT Infrastructure to log data to SIEM Operating Systems Special Applications (with out-of-the-box and custom build collectors) 14
Not In Project Scope SIEM will not provide inventory data SIEM is not for compliance check and reporting project will not provide a configuration management system (baseline security, CERT) project will not implement classical intrusion detection system Worldwide Rollout Staffing of SOC 15
Project Targets / Milestones Contents: Milestones End of analysis: Rough overview on the systems that should log to SIEM End of conception: Systems and applications for the pilot are defined and attack scenarios modeled End of realisation: Systems are logging to the SIEM, EM is in place, attack scenarious are defined as correlation End of validation: Pilot systems are running and tests are succeeded Measurability For servers, network devices and applications the amount of world wide used systems is documented A list of systems and applications for the pilot is documented A list of out of scope systems and applications is documented Attack scenarios are defined and priorised A set of EM rules to implement is defined All systems defined are logging to SIEM Attack scenarious are modelled as correlations and anomaly detection All EM rules defined are integrated All project goals are reached 16
Change of Scope Security incident change of scope and priorities 3 steps First, servers with suspicious activity worldwide OS logs from all servers in the EMEA DMZ OS Logs from all servers worldwide Create emergency rules Postpone all application logs except Firewall Proxy AV 17
Sentinel Architecture at ZF 18
Numbers Sources 40 security devices (Firewall, Proxy, IDS, Anti-Virus) 4500 hosts (Linux, Unix, Windows) Event volume Daily world-wide average: 600 million events European servers: 150 GB/day Retention period OS 2 years 2001 1900 3185 6 5 1 Proxy FW AV IDS/IPS Average EPS worldwide Based on ObserverCategory 20
Operational EPS (EMEA) Servers Security Devices 21
Avertage EPS (EMEA) Servers Security Devices 1 hour search 22
Let s Get Physical Virtual servers were not able to handle load New main servers Blade, 16 cores, 128 GB RAM, FC 13 TB primary, 50 TB secondary storage (TBD) all collectors run remotely 6 servers and 26 CMs Virtual systems 23
Context (Maps) Maps to enrich data with different informations Easier handling ZF specifica for searches 750.000 lines of maps so far Enhance data for the use in distributed systems sentinel server name in clear text 24
Internal Informations (Maps) Generated maps Configuration management system (CMS) Asset information like system type (server, desktop, notebook), responsible ZF group, location Used for searches and tickets Global directory service (GDS) Identity tracking with minimal set of mapped fields (only username, ZF group and location) Privacy reasons IP Adress Management (IPAM) Network zones for searches and reports Clear text information with region, country and location information DNS Hostname resolution of internal clients and servers IP Geolocation Countries 25
Threat Intelligence (Maps) Public reputation data OTX, abuse.ch Trackers Used for reports Whitelist ip addresses needed Private blacklists Internal systems that were investigated External sources 26
Operations Monitoring Sentinel System also used for operations monitoring Nagios, HP Openview and syslog Special collectors for that Written by ZF and NetIQ Used for CMS and ticket integration 27
Ticket Generation ZF uses Jira ticket system SOC does not use the Sentinel internal ticket system Tickets are created with a specialized syslog connector On the operating monitoring Sentinel server The tickets are enriched with CMS data Sending a ticket to the correct group Have all information regarding the CI 28
Ticket Generation Tickets are prepared on each Sentinel server Execute command Sent to the operations monitoring Sentinel Server Sent via syslog out of the script A specialized connector creates the ticket request 29
Ticket Generation Tickets are formatted by The executed command formats the ticket request with Wiki language for a better readability Link direct to the found event Information links for SOC and HelpDesk 30
Lessons Learned
Obvious Things Time synchronization Monitor Disk utilization and availability Daily review of internal logs ((st:(a p i) OR evt:collector) AND sev:[3 TO 5] AND NOT (evt:(readevent* IssueSAMLToken*))) 32
Customizations Adhere to a naming convention Common prefix for custom filter, rules, etc Action plugin being used in an action ZF_SM_EC_Windows_Log_Cleared ZF_SM_SE_Notify_SIEM_Admin Custom field for FQDN Dynamic lists are one dimensional and Sentinel splits into host and domain Chop off description from each Windows event 33
Buffering Collector managers Event source server queues Windows Collector 15 MB/file Event raw data buffers Plan enough disk space in a separate volume Sentinel servers Tune queues Tmp directory for reports Autovacuum disabled RDD disabled (not used in the moment) 34
Object UUIDs All Objects in Sentinel have a Universally Unique Identifier (UUID) Event Sources, Connectors, Collectors, Collector Managers, Sentinel Servers, Identities Asset These are different per server They can t be resoved on a different server / are overwritten by Sentinel Link information Persons / hosts would have a different identity / asset on each server 35
ESM Objects Use SQL script to extract all ids and names on all servers Combine into single map Distribute that to all servers Map CustomerVars to names 36
Identities and Assets Identities Customized collector to use UUID from data source Assets Don t use Asset API, DB tables, views so far script on one Sentinel server to generate map file directly Inject data into events Link to CMS (tbd) 37
Multiple Sentinel Servers Avoid them if you can Rather spend more on hardware If you can t, automation will help you 38
Automation Solution Packs Collect, organize and distribute content SQL Scripts to pull data for cross server reports REST API Update plugins Enable no-data alerts on all DCs 39
Automation OS level Run commands on all servers/cms Silent install Distribute Files Map Data (static) Dynamic List Rsync 40
Correlation in Distributed Environments Deviding by geo Some devices will be used cross-geo or centrally managed Those report up into one specific sentinel server symmetric rules could be hard to do Need to think of data locality, message passing, race conditions 41
Don t miss the Identity-Powered Experience in IT Central. Thank you. 42 2014 NetIQ Corporation. All rights reserved.
This document could include technical inaccuracies or typographical errors. Changes are periodically made to the information herein. These changes may be incorporated in new editions of this document. NetIQ Corporation may make improvements in or changes to the software described in this document at any time. Copyright ActiveAudit, ActiveView, Aegis, AppManager, Change Administrator, Change Guardian, Compliance Suite, the cube logo design, Directory and Resource Administrator, Directory Security Administrator, Domain Migration Administrator, Exchange Administrator, File Security Administrator, Group Policy Administrator, Group Policy Guardian, Group Policy Suite, IntelliPolicy, Knowledge Scripts, NetConnect, NetIQ, the NetIQ logo, PSAudit, PSDetect, PSPasswordManager, PSSecure, Secure Configuration Manager, Security Administration Suite, Security Manager, Server Consolidator, VigilEnt, and Vivinet are trademarks or registered trademarks of NetIQ Corporation or its subsidiaries in the United States.