TUT8173 Best Practices for Security Monitoring in Distributed Environments November 2014

Transcription

1 TUT8173 Best Practices for Security Monitoring in Distributed Environments November 2014 Chris Patzer ZF Norbert Klasen NetIQ

2 Agenda Sentinel Deployment Scenarios Case Study: ZF Lessons Learned 2

3 Infrastructure Patterns Single site 3

4 Distributed Environments So your environment is distributed, does the SIEM infrastructure need to be distributed as well? Required by law/contracts network topology/bandwidth horizontal scale out for performance Desirable for resilience 4

5 Infrastructure Patterns Large regional sites 5

6 One-Tier Distributed Architecture 6

7 Infrastructure Patterns Distributed environments 7

8 Case Study ZF Group

9 ZF Group Motion and Mobility 9

10 Worldwide Presence with Production, Development, Aftermarket Trading and Service 122 production companies Claudia Bucher/HGB 8 main development locations 32 service companies Over 650 service partners worldwide 10

11 The ZF Group An Overview 2013 Sales $ 21 billion Employees 72,600 Countries 26 In September, ZF agreed to buy TRW Automotive Holdings Corp. (TRW) for $13.5 billion 11

12 Employees 2012 by Region Western Europe 49,093 thereof in Germany 43,195 Eastern Europe 4,485 North America 6,856 South America 5,235 Asia-Pacific 7,888 Africa 1,218 Total: 74,775 Thereof: Research and Development (R&D) approx. 7,120 Training 1,980 12

13

14 Project Definition The project will introduce a SIEM system and integrate systems management ZF was not able to react to external or internal attacks effectively. Log files were not collected centrally and there was no possibility to correlate events or provide forensic information. Representative systems from all zones and regions of the IT Infrastructure to log data to SIEM Operating Systems Special Applications (with out-of-the-box and custom build collectors) 14

15 Not In Project Scope SIEM will not provide inventory data SIEM is not for compliance check and reporting project will not provide a configuration management system (baseline security, CERT) project will not implement classical intrusion detection system Worldwide Rollout Staffing of SOC 15

16 Project Targets / Milestones Contents: Milestones End of analysis: Rough overview on the systems that should log to SIEM End of conception: Systems and applications for the pilot are defined and attack scenarios modeled End of realisation: Systems are logging to the SIEM, EM is in place, attack scenarious are defined as correlation End of validation: Pilot systems are running and tests are succeeded Measurability For servers, network devices and applications the amount of world wide used systems is documented A list of systems and applications for the pilot is documented A list of out of scope systems and applications is documented Attack scenarios are defined and priorised A set of EM rules to implement is defined All systems defined are logging to SIEM Attack scenarious are modelled as correlations and anomaly detection All EM rules defined are integrated All project goals are reached 16

17 Change of Scope Security incident change of scope and priorities 3 steps First, servers with suspicious activity worldwide OS logs from all servers in the EMEA DMZ OS Logs from all servers worldwide Create emergency rules Postpone all application logs except Firewall Proxy AV 17

18 Sentinel Architecture at ZF 18

19 Numbers Sources 40 security devices (Firewall, Proxy, IDS, Anti-Virus) 4500 hosts (Linux, Unix, Windows) Event volume Daily world-wide average: 600 million events European servers: 150 GB/day Retention period OS 2 years Proxy FW AV IDS/IPS Average EPS worldwide Based on ObserverCategory 20

20 Operational EPS (EMEA) Servers Security Devices 21

21 Avertage EPS (EMEA) Servers Security Devices 1 hour search 22

22 Let s Get Physical Virtual servers were not able to handle load New main servers Blade, 16 cores, 128 GB RAM, FC 13 TB primary, 50 TB secondary storage (TBD) all collectors run remotely 6 servers and 26 CMs Virtual systems 23

23 Context (Maps) Maps to enrich data with different informations Easier handling ZF specifica for searches lines of maps so far Enhance data for the use in distributed systems sentinel server name in clear text 24

24 Internal Informations (Maps) Generated maps Configuration management system (CMS) Asset information like system type (server, desktop, notebook), responsible ZF group, location Used for searches and tickets Global directory service (GDS) Identity tracking with minimal set of mapped fields (only username, ZF group and location) Privacy reasons IP Adress Management (IPAM) Network zones for searches and reports Clear text information with region, country and location information DNS Hostname resolution of internal clients and servers IP Geolocation Countries 25

25 Threat Intelligence (Maps) Public reputation data OTX, abuse.ch Trackers Used for reports Whitelist ip addresses needed Private blacklists Internal systems that were investigated External sources 26

26 Operations Monitoring Sentinel System also used for operations monitoring Nagios, HP Openview and syslog Special collectors for that Written by ZF and NetIQ Used for CMS and ticket integration 27

27 Ticket Generation ZF uses Jira ticket system SOC does not use the Sentinel internal ticket system Tickets are created with a specialized syslog connector On the operating monitoring Sentinel server The tickets are enriched with CMS data Sending a ticket to the correct group Have all information regarding the CI 28

28 Ticket Generation Tickets are prepared on each Sentinel server Execute command Sent to the operations monitoring Sentinel Server Sent via syslog out of the script A specialized connector creates the ticket request 29

29 Ticket Generation Tickets are formatted by The executed command formats the ticket request with Wiki language for a better readability Link direct to the found event Information links for SOC and HelpDesk 30

30 Lessons Learned

31 Obvious Things Time synchronization Monitor Disk utilization and availability Daily review of internal logs ((st:(a p i) OR evt:collector) AND sev:[3 TO 5] AND NOT (evt:(readevent* IssueSAMLToken*))) 32

32 Customizations Adhere to a naming convention Common prefix for custom filter, rules, etc Action plugin being used in an action ZF_SM_EC_Windows_Log_Cleared ZF_SM_SE_Notify_SIEM_Admin Custom field for FQDN Dynamic lists are one dimensional and Sentinel splits into host and domain Chop off description from each Windows event 33

33 Buffering Collector managers Event source server queues Windows Collector 15 MB/file Event raw data buffers Plan enough disk space in a separate volume Sentinel servers Tune queues Tmp directory for reports Autovacuum disabled RDD disabled (not used in the moment) 34

34 Object UUIDs All Objects in Sentinel have a Universally Unique Identifier (UUID) Event Sources, Connectors, Collectors, Collector Managers, Sentinel Servers, Identities Asset These are different per server They can t be resoved on a different server / are overwritten by Sentinel Link information Persons / hosts would have a different identity / asset on each server 35

35 ESM Objects Use SQL script to extract all ids and names on all servers Combine into single map Distribute that to all servers Map CustomerVars to names 36

36 Identities and Assets Identities Customized collector to use UUID from data source Assets Don t use Asset API, DB tables, views so far script on one Sentinel server to generate map file directly Inject data into events Link to CMS (tbd) 37

37 Multiple Sentinel Servers Avoid them if you can Rather spend more on hardware If you can t, automation will help you 38

38 Automation Solution Packs Collect, organize and distribute content SQL Scripts to pull data for cross server reports REST API Update plugins Enable no-data alerts on all DCs 39

39 Automation OS level Run commands on all servers/cms Silent install Distribute Files Map Data (static) Dynamic List Rsync 40

40 Correlation in Distributed Environments Deviding by geo Some devices will be used cross-geo or centrally managed Those report up into one specific sentinel server symmetric rules could be hard to do Need to think of data locality, message passing, race conditions 41

41 Don t miss the Identity-Powered Experience in IT Central. Thank you NetIQ Corporation. All rights reserved.

42

43 This document could include technical inaccuracies or typographical errors. Changes are periodically made to the information herein. These changes may be incorporated in new editions of this document. NetIQ Corporation may make improvements in or changes to the software described in this document at any time. Copyright ActiveAudit, ActiveView, Aegis, AppManager, Change Administrator, Change Guardian, Compliance Suite, the cube logo design, Directory and Resource Administrator, Directory Security Administrator, Domain Migration Administrator, Exchange Administrator, File Security Administrator, Group Policy Administrator, Group Policy Guardian, Group Policy Suite, IntelliPolicy, Knowledge Scripts, NetConnect, NetIQ, the NetIQ logo, PSAudit, PSDetect, PSPasswordManager, PSSecure, Secure Configuration Manager, Security Administration Suite, Security Manager, Server Consolidator, VigilEnt, and Vivinet are trademarks or registered trademarks of NetIQ Corporation or its subsidiaries in the United States.