Best Practices Guide Revision C. McAfee Network Security Platform 8.3

Transcription

1 Best Practices Guide Revision C McAfee Network Security Platform 8.3

2 COPYRIGHT 2016 Intel Corporation TRADEMARK ATTRIBUTIONS Intel and the Intel logo are registered trademarks of the Intel Corporation in the US and/or other countries. McAfee and the McAfee logo, McAfee Active Protection, McAfee DeepSAFE, epolicy Orchestrator, McAfee epo, McAfee EMM, McAfee Evader, Foundscore, Foundstone, Global Threat Intelligence, McAfee LiveSafe, Policy Lab, McAfee QuickClean, Safe Eyes, McAfee SECURE, McAfee Shredder, SiteAdvisor, McAfee Stinger, McAfee TechMaster, McAfee Total Protection, TrustedSource, VirusScan are registered trademarks or trademarks of McAfee, Inc. or its subsidiaries in the US and other countries. Other marks and brands may be claimed as the property of others. LICENSE INFORMATION License Agreement NOTICE TO ALL USERS: CAREFULLY READ THE APPROPRIATE LEGAL AGREEMENT CORRESPONDING TO THE LICENSE YOU PURCHASED, WHICH SETS FORTH THE GENERAL TERMS AND CONDITIONS FOR THE USE OF THE LICENSED SOFTWARE. IF YOU DO NOT KNOW WHICH TYPE OF LICENSE YOU HAVE ACQUIRED, PLEASE CONSULT THE SALES AND OTHER RELATED LICENSE GRANT OR PURCHASE ORDER DOCUMENTS THAT ACCOMPANY YOUR SOFTWARE PACKAGING OR THAT YOU HAVE RECEIVED SEPARATELY AS PART OF THE PURCHASE (AS A BOOKLET, A FILE ON THE PRODUCT CD, OR A FILE AVAILABLE ON THE WEBSITE FROM WHICH YOU DOWNLOADED THE SOFTWARE PACKAGE). IF YOU DO NOT AGREE TO ALL OF THE TERMS SET FORTH IN THE AGREEMENT, DO NOT INSTALL THE SOFTWARE. IF APPLICABLE, YOU MAY RETURN THE PRODUCT TO MCAFEE OR THE PLACE OF PURCHASE FOR A FULL REFUND. 2 McAfee Network Security Platform 8.3 Best Practices Guide

3 Contents Preface 5 About this guide Audience Conventions Find product documentation Introduction 7 Pre-installation checklist Cabling best practices 9 3 Hardening the Manager Server for Windows platform 11 Pre-installation Installation Post-installation Disable non-required services Set system policies Set user policies Set the desktop firewall Configure audit events Hardening the MySQL installation for Windows platform 15 Harden the MySQL installation Remove test database Remove local anonymous users Remove remote anonymous users Secure MySQL remote access How to roll back your changes Removal of debug shell at port Other best practices for securing Manager Large Sensor deployments 19 Staging Sensors prior to deployment Recommendations for large Sensor deployment Using active fail-open kits 21 Considerations Effective policy tuning practices 23 Analyzing high-volume attacks Managing ignore rules Learning profiles in DoS attacks Response management 25 Sensor response actions McAfee Network Security Platform 8.3 Best Practices Guide 3

4 Contents 9 How to create rule sets 27 Best methods for rule set creation Working with firewall policies How to handle asymmetric networks SSL best practices 33 SSL only traffic throughput: I-series Sensors SSL only traffic throughput: M-series Sensors SSL traffic mixed with HTTP 1.1 traffic: I-series Sensors SSL traffic mixed with HTTP 1.1 traffic: M-series Sensors SSL only traffic - throughput: NS-series Sensors SSL traffic mixed with HTTP 1.1 traffic: NS-series Sensors Sensor HTTP response processing deployment 39 Tests for enabling HTTP response traffic HTTP response processing results for I-series Sensors HTTP response processing results for M-series Sensors HTTP response processing results for Virtual IPS Sensor HTTP response processing results for NS-series Sensors Sensor performance with Layer 7 Data Collection I-series Sensor capacity by model number M-series Sensor capacity by model number NS-series Sensor capacity by model number Virtual IPS Sensor capacity by model number Comparison between I-1200/I-1400 and M-1250/M-1450 FE ports 65 Index 67 4 McAfee Network Security Platform 8.3 Best Practices Guide

5 Preface This guide provides the information you need to configure, use, and maintain your McAfee product. Contents About this guide Find product documentation About this guide This information describes the guide's target audience, the typographical conventions and icons used in this guide, and how the guide is organized. Audience McAfee documentation is carefully researched and written for the target audience. The information in this guide is intended primarily for: Administrators People who implement and enforce the company's security program. Users People who use the computer where the software is running and can access some or all of its features. Conventions This guide uses these typographical conventions and icons. Italic Bold Monospace Narrow Bold Title of a book, chapter, or topic; a new term; emphasis Text that is emphasized Commands and other text that the user types; a code sample; a displayed message Words from the product interface like options, menus, buttons, and dialog boxes Hypertext blue A link to a topic or to an external website Note: Extra information to emphasize a point, remind the reader of something, or provide an alternative method Tip: Best practice information Caution: Important advice to protect your computer system, software installation, network, business, or data Warning: Critical advice to prevent bodily harm when using a hardware product McAfee Network Security Platform 8.3 Best Practices Guide 5

6 Preface Find product documentation Find product documentation On the ServicePortal, you can find information about a released product, including product documentation, technical articles, and more. Task 1 Go to the ServicePortal at and click the Knowledge Center tab. 2 In the Knowledge Base pane under Content Source, click Product Documentation. 3 Select a product and version, then click Search to display a list of documents. 6 McAfee Network Security Platform 8.3 Best Practices Guide

7 1 1 Introduction McAfee Network Security Platform [formerly McAfee IntruShield ] is a combination of network appliances and software, built for the accurate detection and prevention of intrusions and network misuse. We recommend that you follow some of the best techniques and tips to use McAfee Network Security Platform most effectively. This can save considerable time during the installation and tuning process of the system. Following chapters outline the best practices for Network Security Platform. Pre-installation checklist There are some important tasks that you should consider before McAfee Network Security Manager [formerly McAfee IntruShield Security Manager] software installation. For more information, see Planning for installation, McAfee Network Security Platform Troubleshooting Guide. McAfee Network Security Platform 8.3 Best Practices Guide 7

8 1 Introduction Pre-installation checklist 8 McAfee Network Security Platform 8.3 Best Practices Guide

9 2 2 Cabling best practices It is a common practice to physically cable the monitoring ports, only after the McAfee Network Security Sensor (Sensor) has been fully configured. In other words, most administrators cable the console and management ports, use those connections to configure the solution, and only physically introduce the Sensor into the scanning process once the proper scanning policies are in place, the monitoring ports have been configured, the latest signature set has been downloaded, and so on. Also, in the most security-conscious environments, or those with congestion problems, a private network is often used to connect the Sensor management ports to the McAfee Network Security Manager (Manager). This is another best practice in terms of cabling the Sensors. McAfee Network Security Platform 8.3 Best Practices Guide 9

10 2 Cabling best practices 10 McAfee Network Security Platform 8.3 Best Practices Guide

11 3 3 Hardening the Manager Server for Windows platform Implementation of Manager varies from environment to environment. The Manager's physical and logical position in the network influences specific remote access and firewall configuration requirements. The following best practices on managing configurable features on Manager impacts the security of Manager. These steps are applicable to Windows Server 2008 and Windows Server 2012 editions. Contents Pre-installation Installation Post-installation Pre-installation Use a dedicated machine for the Manager server and then install Manager and the embedded MySQL database. Other than the host-based firewall, no other software should be installed on the server. Before installation of Manager do the following: Ensure that the server is located in a physically secure environment. Connect the server on a protected or isolated network. If the hard disk is old, use fdisk (a command line utility) to remove all partitions and create new partitions. Installation Installation of Manager should be performed as follows: Install the US version of Windows Server. Use NTFS on all partitions. McAfee Network Security Platform 8.3 Best Practices Guide 11

12 3 Hardening the Manager Server for Windows platform Post-installation Post-installation After installation of Manager perform the following installations: Install the latest Windows Server patches, service packs, and hot fixes from Microsoft. Install a Virus Scanner and update the signatures. Exclude "McAfee Network Security Manager (Manager)" and "MySQL" directories from being scanned. Also keep a check on the following: Minimize the number of Windows roles and features that are installed. Uninstall applications that are not necessary. Disable non-required services Disable the following services: DHCP Client FTP Print spooler Remote access auto connection manager Remote procedure call locator Remote registry Server TCP/IP NetBIOS helper service Telephony service. Enable these services only if it is absolutely required. Set system policies Ensure to set the following system policies: Implement the System key and strong encryption of the password database by running SYSKEY.EXE Use Microsoft security compliance toolkit or set local security policy Display legal notice at during interactive logon window. Do not display username that was earlier used to login. Disable Posix Clear virtual memory page file during shutdown Disable autorun Disable LMHOSTS lookup while setting the advanced TCP/IP settings. 12 McAfee Network Security Platform 8.3 Best Practices Guide

13 Hardening the Manager Server for Windows platform Post-installation 3 Set user policies Make sure to set the following user policies: Rename the administrator account. Disable guest account. Passwords should be at least 8 ASCII characters. Enable locking of screensaver. Set the desktop firewall It is recommended that a desktop firewall operates on the Manager server. The following ports are required for Manager-Sensor communication. Ensure that there are no other open ports using a scanning tool such as McAfee Vulnerability Manager. Port Description Communication 80 HTTP port Client to Manager 443 HTTPS Client to Manager 3306 MySQL database Open only while using external SQL database 8500 Command channel(udp) Manager to Sensor 8501 Install channel (TCP) (1024-bit) Sensor to Manager 8502 Alert channel (TCP) (1024-bit) Sensor to Manager 8503 Packet log channel (TCP) (1024-bit) Sensor to Manager 8504 File transfer channel (TCP) Sensor to Manager 8506 Install channel (TCP) (2048-bit) Sensor to Manager 8507 Alert channel (TCP) (2048-bit) Sensor to Manager 8508 Packet log channel (TCP) (2048-bit) Sensor to Manager 8509 Bulk file transfer channel for 2048-bit certificates Bulk file transfer channel for 1024-bit certificates. Sensor to Manager Sensor to Manager 8555 Alert viewer (TC) Client to Manager When notification or SNMP forwarding is configured on Manager and there is firewall between Manager and SNMP Server, ensure that the following ports are allowed through firewall. Port Description Communication 25 SMTP port Manager to SMTP server 162 SNMP forwarding Manager to SNMP server If you have McAfee epo integration configured on Manager, and there is firewall between Manager and the McAfee epo Server, ensure the following port is also allowed through firewall. Port Description Communication 8443 McAfee epo communication port Manager to McAfee epo server McAfee Network Security Platform 8.3 Best Practices Guide 13

14 3 Hardening the Manager Server for Windows platform Post-installation Configure audit events Set the following events to be audited: Audit account logon events Audit policy change (Success) Audit account management Audit privilege use (Failure) Audit logon events Audit system events (Success) Audit object access (Failure) 14 McAfee Network Security Platform 8.3 Best Practices Guide

15 4 Hardening the MySQL installation for Windows platform This section describes methods for hardening your MySQL installation. Contents Harden the MySQL installation Other best practices for securing Manager Harden the MySQL installation Ensure the cmd window used for making changes to database tables in the "mysql" database stays opened in the mysql shell until validation is completed. This is necessary to enable you to rollback the changes in case you need to. Rollback procedures are shown at the end of this section. Use another cmd window, where necessary, to validate hardening changes you have made. Remove test database Remove the 'test" database from the server. 1. Start My SQL. mysql> use mysql; 2. Backup db table to do dbbackup before changing it. mysql> create table db_backup as select * from db; 3. Validate that the backup table was created and row count matches that of the mysql.db table. mysql> select count(*) from db_backup; 4. Check all the databases on the Manager server. mysql> show databases; 5. Remove the test db. mysql> drop database test; 6. The test database should no longer be listed. mysql> show databases; Remove local anonymous users To remove local anonymous users: 1. Look for blank entries for user. mysql> select host,db,user from db; 2. Remove anonymous access to databases mysql> update db set host="localhost" where user=""; McAfee Network Security Platform 8.3 Best Practices Guide 15

16 4 Hardening the MySQL installation for Windows platform Harden the MySQL installation 3. Remove anonymous/blank accounts mysql> flush privileges; 4. Validate that "localhost" replaced % entry under the host column. You will also notice you will now need to qualify username and password on the local machine to get into mysql shell from the mysql.exe CLI. Remove remote anonymous users To remove remote anonymous users, you harden mysql.exe CLI access by forcing the requirement for a username and password to get into the mysql shell as follows. Start MySQL. Back up the user table to user_backup before changing it. Validate that the backup table was created and row count matches that of the mysql.db table. List all users and hosts. Remove anonymous/blank accounts. Validate that rows with blank user columns have been removed. mysql> use mysql; mysql> create table user_backup as select * from user; mysql> select count(*) from user_backup; mysql> select user,host from user; mysql> delete from user where user=""; mysql> select user,host from user; Secure MySQL remote access This section provides two options for removing remote access. Remove individual users' remote access Remove ALL remote access (Recommended) Removal of individual user's remote access Do ONE of the following: Remove admin (Network Security Platform user) remote access mysql> delete from user where host!='localhost' and user='admin'; (The admin user cannot login remotely; however Manager root can. Use second cmd window to validate.) mysql>flush privileges; Remove root remote access (Recommended minimum action) mysql> delete from user where host!='localhost' and user='root'; This ensures that the root user cannot login remotely; however Manager user can log in remotely. Use second cmd window to validate. mysql>flush privileges; Remove ALL remote access mysql> delete from user where host!='localhost'. ALL user access is disabled including Manager users from remote host(s). 16 McAfee Network Security Platform 8.3 Best Practices Guide

17 Hardening the MySQL installation for Windows platform Harden the MySQL installation 4 Use another cmd window to validate; you can ONLY log in to the MySQL CLI on the Manager server by qualifying username, password and db. For example: mysql -uadmin -pxxx lf. How to roll back your changes If you need to roll back your changes, use the following commands: To roll back changes made to the mysql.db table from the mysql.db_backup table: mysql> rename table db to db_1; mysql> alter table db_backup engine=myisam; mysql> rename table db_backup to db; mysql> flush privileges; To roll back changes made to the "mysql.user" table from mysql.user_backup table: mysql> rename table user to user_1 mysql> rename table user_backup to user; mysql> flush privileges; Removal of debug shell at port 9001 In addition to denying traffic over port 9001 and 9002 (as per Install a desktop firewall), the debugging shell that runs on port 9001 can be disabled by modifying the value of the iv.policymgmt.ruleengine.bsh_diagnostics_port record in the iv_emspropertiestable. To disable the port, set the value in the field called "value" = -1 McAfee Network Security Platform 8.3 Best Practices Guide 17

18 4 Hardening the MySQL installation for Windows platform Other best practices for securing Manager Other best practices for securing Manager Use a clean, dedicated machine for the Manager server and perform a fresh install of the Manager software, including the installation of the embedded MySQL database. No other software should be installed on the server, with the exception of a host-based firewall as described above. Make sure the PC is in an isolated, physically secure environment Disallow access to the installation directory and its sub-directories to anyone other than authorized administrators. Use Microsoft Knowledge Base article # to accomplish this procedure. Disallow the following permissions: Read Modify Write List folder contents Read and Write Full control 18 McAfee Network Security Platform 8.3 Best Practices Guide

19 5 Large 5 Sensor deployments When you consider large McAfee Network Security Sensor (Sensor) deployments, (where the number of Sensors deployed range from 36 to 100) there are some important tasks which should be considered, before deployment. McAfee recommends that you have a good understanding on the best techniques required to accomplish these tasks in your deployment scenario, prior to the deployment. Concurrent Signature Set and Sensor Software downloads In x and above, the Manager provides an option for parallel processing of Sensor software and signature set updates. In earlier releases of 6.0, when multiple Sensors are configured to your Manager, any software update on the Sensors had to be performed individually. If you are using 5.1, then note that this option is available on Manager version and above. This enhancement is applicable only for Sensor updates in the parent domain. The Sensor updates in the child admin domain is performed in the same method as the earlier releases. Sensor Software Updates All Sensor software updates do require a reboot. A reboot can take up to 5 minutes. You can schedule this process though you can't reboot the Sensor automatically. But any update from the Manager Server causes the process to take place sequentially, one Sensor at-a-time. You can instead use the TFTP method for updating the Sensor image, which helps you to load concurrent images on the Sensor via the Sensor's CLI, at a much faster rate. For more information, see Upgrading Sensor software via a TFTP server, McAfee Network Security Platform CLI Guide. Central Manager deployment If you have a large Sensor deployment of 200 Sensors for example, which are deployed across various geographic locations, then consider using a Central Manager at your organization's headquarters and deploy a dedicated Manager for each region. Each Manager will then handle the daily device operations for all Sensors configured to it. Note that when you use a Central Manager, your regional/local Managers can add their own region-specific rules, but cannot modify any configuration established by the Central Manager. Configuration updates to the Sensors must be applied through the local Managers. See McAfee Network Security Platform Manager Administration Guide for details. Usability Depending on the number of VIDS and Admin Domains defined in your deployment, the Manager Resource Tree can become very crowded, which makes it difficult to locate the resource you require at any point of time. It can also lead to confusion if you have not provided unique, recognizable names for your Sensors and any VIDS you create. Note that the resource names appear both in the Resource Tree of the Manager as well as in Alert data and Reports. Your VIDS names should also be clear and easy for everyone maintaining the network to recognize at a glance. For example, compare a worldwide deployment where Sensors are named "4010-1" through " " as opposed to "UK-London-sens1," "India-Bangalore-sens1," and so on. Alert Traffic Take note of the volume of alerting in your Sensors. Depending on the policies deployed on your system, there is potential to starve Manager resources since the resulting alerts are passed to the Manager. As the volume of alerting increases, more data is passed into the Manager. The Manager can handle short bursts of high alert volume but on an average, the Manager can handle a total of 1500 alerts per minute from all the Sensors configured to it. McAfee Network Security Platform 8.3 Best Practices Guide 19

20 5 Large Sensor deployments Staging Sensors prior to deployment Start-up load on the Manager When the Manager starts, establishing connections with all Sensors can be time consuming as Sensors continue to collect alerts. If the communication with the Manager is lost, each Sensor must pass its alert data to the Manager when connectivity is re-established. So, it is required to account for the start-up load on the Manager. Concurrent processes Be aware of the time periods in which your scheduled processes (such as database backup or report generation) occur, and try not to attempt other tasks during that time period, as this can lead to process locking. This includes having many users logged into the system simultaneously. Contents Staging Sensors prior to deployment Recommendations for large Sensor deployment Staging Sensors prior to deployment With large or very large deployments, and/or if you are planning to release Sensors to various geographical regions or remote locations, you will have to consider staging your Sensors before you release them to their final destination. For example, use the McAfee Network Security Manager in a lab environment to push Sensor software to the Sensor, make sure that the Sensor is working as expected, and then box the configured Sensor and send it to its final destination. For more information, see Updating the configuration of a Sensor, McAfee Network Security Platform IPS Administration Guide. Or you might use the TFTP feature to load the Sensor image at one location, before shipping the Sensor to another. For more information, see Upgrading Sensor software via a TFTP server, McAfee Network Security Platform Installation Guide. Very large Sensor deployments mean that the number of Sensors deployed is more than 100. Large Sensor deployments have Sensors numbering between 36 and Recommendations for large Sensor deployment Most McAfee Network Security Platform customers begin their deployment in their lab environment. Here they test the Sensor functionality, familiarize themselves with the Manager, and create an initial policy. Once they are comfortable with the product, they deploy the Sensor in a live environment. McAfee provides a few recommendations for this process: Spend time creating effective policies before actual deployment. Availability of more information makes the tuning process easier. But policies like the McAfee Network Security Platform provided All-Inclusive policy can overwhelm you with data, if every Sensor in a large deployment is running it without any customization. Stagger your Sensor deployment in phases. As each new batch of Sensors provides you with more data points, you can tune your policies more effectively, and become more aggressive in the number of Sensors you deploy in the next phase. 20 McAfee Network Security Platform 8.3 Best Practices Guide

21 6 Using 6 active fail-open kits McAfee supports the following types of passive and active fail-open kits: 10/100/1000 Gigabit Copper Passive Fail-Open Bypass Kit 1 Gigabit Optical Passive Fail-Open Bypass Kit 10 Gigabit Optical Passive Fail-Open Bypass Kit 10/100/1000 Copper Active Fail-Open Bypass Kit 10/100/1000 Copper Active Fail-Open Bypass Kit with SNMP monitoring 1 Gigabit Optical Active Fail-Open Bypass Kit 10 Gigabit Optical Active Fail-Open Bypass Kit Fail-open kits can be deployed in production networks for the following reasons: Reduce the network downtime to seconds during any Sensor reboot or Sensor failure Protect your network during link failure on the Sensor Bypass the traffic when troubleshooting network issues. This will help you identify or eliminate the Sensor as the cause of network issues In the passive fail-open kit, if the Sensor goes down, the link has to be renegotiated again between the peer devices and this causes the link to go down for some time. In case of an active fail-open kit, a physical link will be established between the active fail-open kit and the two peer devices even when the Sensor is active. There would not be any link flap even when the Sensor goes down. McAfee recommends deploying active fail-open kits for protection of mission critical networks. For Virtual IPS Sensors, only 10/100/1000 Copper Active Fail-Open Bypass Kit and 10/100/1000 Copper Active Fail-Open Bypass Kit with SNMP monitoring are supported. For more information, see Virtual IPS Sensor deployment section in the IPS Administration Guide. Passive Fail-open In passive fail-open kits, during normal Sensor in-line, fail-open operation, the Fail-Open Controller or built-in Control port (depending on which controls the Bypass Switch) supplies power and a heartbeat signal to the Bypass Switch. If this signal is not presented within its programmed interval, the Fail-Open Bypass Switch removes the Sensor from the data path, and moves into bypass mode, providing continuous data flow with little network interruption. While the Sensor is in bypass mode, traffic passes directly through the switch, bypassing the Sensor. When normal Sensor operation resumes, you may or may not need to manually re-enable the monitoring ports from the Manager interface, depending on the activity leading up to the Sensor's failure. Active Fail-open McAfee Network Security Platform 8.3 Best Practices Guide 21

22 6 Using active fail-open kits Considerations In case of active fail-open kits, during normal Sensor in-line fail-open operation, the built-in monitoring sends a heartbeat signal (1 every second) to the Bypass Switch. If the Sensor does not receive 3 heart beat signals within its programmed interval, the Fail-Open Bypass Switch removes the Sensor from the data path, and moves it into the bypass mode, providing continuous data flow. When the Bypass Switch loses power, traffic continues to flow through the network link, but is no longer routed through the Bypass Switch. This allows network devices to be removed and replaced without network downtime. Once power is restored to the Bypass Switch, network traffic is seamlessly diverted to the monitoring device, allowing it to resume its critical functions. Considerations This section discusses the generic requirements and notes that you need to consider with respect to active fail-open kits: The currently supported active fail-open kits are not plug and play devices. Initial configuration/ setup is required before you begin. The following default options are fixed in McAfee active fail-open kits and cannot be changed: LFD is set to On Bypass Detection is set to Off Even if you change the configuration for these options using the NetOptics Web Manager or System Manager, the settings of these options on the McAfee active fail-open kit hardware cannot be changed. The management port on the active fail-open bypass kits cannot be configured. The parameters for the monitoring port must be set to Auto-Negotiate based on the speed, that is, 10/100/1000 Mbps. McAfee recommends that you set the Speed to 100 Mbps full Duplex with Auto-Negotiate enabled to improve performance. Unlike passive fail-open kits, an active fail-open kit moves into the bypass mode only when it does not receive the heart beat signals within its programmed interval. When the Sensor monitoring port is manually disabled or the cable is pulled out for example, the Manager displays the port status as AUK (Active Unknown) under Device List / Sensor_Name > Physical Sensor > Port Settings page. If you are planning to use the 10/100/1000 copper active fail-open kit with SNMP monitoring, then note that Network Security Platform currently supports only SNMP v1 on active fail-open kits. You can configure only a single SNMP Manager. The option to configure a secondary SNMP Manager is currently not available. The active fail-open kits do not provide any CLI option to view the serial and model numbers of the kits. If your network architecture is such that it requires you to remotely manage the active fail-open kits in your deployment, then you can consider one of the following options: Use a terminal server to connect to the system console and then connect using a remote login [interoperability issues might be seen while using UPLOGIX Terminal Server] Pre-configure the kit with the required settings before shipping. 22 McAfee Network Security Platform 8.3 Best Practices Guide

23 7 Effective 7 policy tuning practices All Network Security Sensors (Sensors) on initial deployment, have the ' Default Prevention' policy loaded on all interfaces. McAfee recommends that you use the Default Prevention policy as a starting point, then customize the policies based on your organization's requirements. The customized policies can be either cloned versions of the default pre-configured policies or custom-built policies that employ custom rule sets. An appropriately tuned policy will reduce false positives. Though each network environment has unique characteristics, the following best practices can make tuning more efficient and effective. As you interact with Network Security Platform policies, you encounter the term "attack", not "signature." Network Security Platform defines an attack as being comprised of one or more signatures, thresholds, anomaly profiles, or correlation rules, where each method is used to detect an attempt to exploit a particular vulnerability in a system. These signatures and checks may contain very specific means for identifying a specific known exploit of the vulnerability, or more generic detection methods that aid in detecting unknown exploits for the vulnerability. Contents Analyzing high-volume attacks Managing ignore rules Learning profiles in DoS attacks Analyzing high-volume attacks Take attacks that are generating the most alerts (use the Attack Log page) and investigate their legitimacy. For more information, see McAfee Network Security Platform Manager Administration Guide. Many of the top alerts seen on the initial deployment of a Sensor will be common false positives seen in many environments. Typically, at the beginning of the tuning process, it will be evident that your network or security policy will affect the overall level of alerts. If, for instance, AOL IM is allowed traffic on the network, then there might not be a need to alert on AOL IM setup flows. Managing ignore rules When a particular alert is declared as a false positive, the next decision is whether to disable the corresponding attack altogether OR apply a particular ignore rule to that attack that will disable alerting for a particular IP address or range of IP addresses. In almost all cases, it is a best practice to implement the latter. McAfee Network Security Platform 8.3 Best Practices Guide 23

24 7 Effective policy tuning practices Learning profiles in DoS attacks Consider the McAfee Vulnerability Manager. When McAfee Vulnerability Manager scans a network host, some of this traffic might appear as an attack. However, you are aware of the purpose of this traffic and you do not want the Sensor to take any response action on this traffic. However, if similar traffic is generated by any other server, you want the Sensor to treat it as an attack and respond accordingly. Network Security Platform provides various options to handle such situations. Every ignore rule created is globally stored, so that the filter can be applied to any Exploit or Reconnaissance attack. It is also a best practice to document all your tuning activities. The Configuration Report section can be used to assist the documentation process. The Performance Monitoring - Sensor Configuration report will deliver reports that list ignore rules that have been applied and attacks that have been otherwise customized. For more information, see chapter How to create Ignore rules for an applied IPS policy in the McAfee Network Security Platform IPS Administration Guide. Learning profiles in DoS attacks It is a best practice to let the Sensors learn the profiles of the particular segments they are monitoring, before tuning DoS attacks. This is Learning Mode operation. The learning process takes two days. During this period it is not unusual to see DoS alerts associated with normal traffic flows (for example, DoS SYN flood alerts reported outbound on a firewall interface to the Internet). After a profile has been learned, the particulars of the profile (number of SYNS, ACKS, and so on) can be viewed per Sensor. DoS detection can also be implemented using the Threshold Mode. This involves setting thresholds manually for the type of segment characteristics that are learned in Learning Mode. Implementing this mode successfully is critically dependent on detailed knowledge of the segments that the particular Sensors are monitoring. It is a best practice to have the Sensor re-learn the profile when there is a network change (that is, you move the Sensor from a lab or staging environment to a production environment) or a configuration change (that is, you change the CIDR block of a sub-interface) that causes a significant sudden traffic change on an interface. If the Sensor does not re-learn the new environment, it may issue false alarms or fail to detect actual attacks during a time period when it is adapting to the new network traffic conditions. There is no need to re-learn a profile when network traffic increases or decreases naturally over time (for example, an e-commerce site that is getting more and more customers; thus its Web traffic increases in parallel), since the Sensor can automatically adapt to it. For more information, see Managing DoS Learning Mode profiles on a Sensor, McAfee Network Security Platform IPS Administration Guide. 24 McAfee Network Security Platform 8.3 Best Practices Guide

25 8 Response 8 management When McAfee Network Security Sensor (Sensor) detects an activity which violates a configured security policy, a preset response from the Sensor is integral to the protection or prevention process. Proper configuration of responses is crucial to maintaining effective protection. Critical attacks like buffer overflows and DoS attacks require responses in real time, while scans and probes can be logged and researched to determine compromise potential and the source of the attack. Developing a system of actions, alerts, and logs based on specific attacks or attack parameters (such as severity) is recommended for effective network security. For example, since McAfee Network Security Platform can be customized to protect any zone in a network, knowing what needs to be protected can help to determine the response type. If the Sensor is monitoring the network outside of the firewall in inline mode, preventing DoS attacks and attacks against the firewall is crucial. Other suspicious traffic intended for the internal network, such as scans and low-impact well-known exploits, are best logged and analyzed as the impact is not immediate. In this case, a better understanding of the potential attack purpose can be determined. Thus, if you are monitoring outside of a firewall in in-line mode, it is important not to set the policies and responses so fine that they disrupt the flow of traffic and slow down the system. Remember that response actions are decoupled from alerting. Pay particular attention to this with the Recommended For Blocking (RFB) category of attacks, lest you enable blocking for an attack, but disable alerting, causing the attack to be blocked without your knowledge. When there are multiple attempts to login to a specific web server from a client, the Sensor detects a reconnaissance Brute force attack (Attack ID 0x40256b00) and raises an alert. Brute force attacks are used by programs, such as password crackers, to try many different passwords in order to guess the correct one. The alerts raised are threshold based. The Sensor may generate an alert even in scenarios, where a legitimate user keeps on retrying to login to the web server simply because he has forgotten his password. Instances of someone mistyping a password or username on the login are also common. In such cases, valid traffic flow would be blocked or subject to unnecessary responses from the Sensor, leading to a false positive. Consequently, the traffic might be dropped. When such alerts are seen in high volume, there may be multiple reasons for it, like, a dictionary attack against the web server, or network monitoring systems (like WebSense) not updated with a user password change, and so on. McAfee Network Security Platform recommends that while configuring a Reconnaissance policy, you to edit and set optimum threshold values to suit your particular environment. This avoids unnecessary responses from the Sensor and hindrance to the traffic flow. For example, if you have a web-server farm behind the Sensor so there are more HTTP logins seen on this segment, in such a scenario you require to set higher thresholds. The default values are good for most environments. McAfee Network Security Platform 8.3 Best Practices Guide 25

26 8 Response management Sensor response actions Sensor response actions There are multiple Sensor actions that are available for configuration per attack. These include: Dropping Alert Packets Only works in in-line mode. Will drop a detected attack packet and all subsequent packets in the same flow. Quarantine Sensor will quarantine or remediate a host as per the configurations in McAfee Network Security Manager and the Sensor monitoring ports. Quarantine can be enabled per attack in the Policy Editors. For more information, see McAfee Network Security Platform IPS Administration Guide. 26 McAfee Network Security Platform 8.3 Best Practices Guide

27 9 9 How to create rule sets A rule set is configured based on attack category, operating system, protocol, application, severity, and benign trigger probability options. Each rule in a set is either an include rule or an exclude rule. An include rule (which should always start a rule set) is a set of parameters that encompass a broad range of well-known attacks for detection. An exclude rule removes elements from the include rule in order to focus the policy's rule set. Proper creation of rule sets is essential for eliminating false positives and ensuring maximum protection on your network. These best practices can assist while creating rules sets in the McAfee Network Security Manager. Best methods for rule set creation There are two best practice methods employed for creating rule sets. General-to-specific rule creation The first method is general-to-specific. Start with an include rule that covers a broad range of operating systems, applications and protocols. After this, create one or more exclude rules to strip away specific operating systems, protocols, et cetera, thus focusing the rule set on the environment where it will be enforced. For example, start with an include rule for all Exploit category attacks. Follow this with multiple exclusion rules that strip away protocols, applications, severities, et cetera, that are rarely or never seen in a zone of your network. Collaborative rule creation The second method is collaboration: Create multiple include rules within one rule set for each category, operating systems, et cetera, combination that needs to be detected. Each criterion must be matched in order for an alert to be triggered. For example, create the first rule in the set with the Exploit category, Unix as the OS, Sendmail as the application, and SMTP as the protocol. Next, create another include rule for Exploit, Windows 2000, WindMail, and so forth in the same manner. Each include rule added, broadens the scope of the detection. For more information, see Managing Rule Sets, McAfee Network Security Platform IPS Administration Guide. McAfee Network Security Platform 8.3 Best Practices Guide 27

28 9 How to create rule sets Best methods for rule set creation 28 McAfee Network Security Platform 8.3 Best Practices Guide

29 10 Working with firewall policies Review the following points while working with Firewall policies: You cannot set explicit access rules for protocols that negotiate ports dynamically, with the exception of FTP, TFTP, and RPC services. Protocols such as H.323 and Netmeeting, which negotiate the data channel separately from the control channel, or negotiate ports that do not follow a standard, are not supported. However, you can explicitly deny these protocol instances by denying the fixed control port. However, you can configure access rules to explicitly deny these protocol instances by denying the fixed control port. For RPC services, you can configure explicit permit and deny rules for RPC as a whole, but not its constituents, such as statd and mountd. Protocols or services, such as instant messaging and peer-to-peer communication, that use dynamic ports, are not supported. An alternative option for denying protocols that use dynamic ports is to configure IDS policies to drop the attacks that are detected in such transmissions. Network Security Platform detects use of and attacks in such programs as Yahoo Messenger, KaZaA, IRC, and so on. There is a limit on the number of access rules that can be supported by various Sensor models. For more information, see McAfee Network Security Platform IPS Administration Guide McAfee Network Security Platform 8.3 Best Practices Guide 29

30 10 Working with firewall policies 30 McAfee Network Security Platform 8.3 Best Practices Guide

31 11 How to handle asymmetric networks Traffic that uses a different path for the request vs. response is termed as asymmetric traffic. There are chances of having asymmetric traffic within a network, when networks increase in size. If there are chances of asymmetric traffic in your network, consider the following options: Install IPS Sensors at a location where the traffic is symmetric. Implement a port clustering configuration for asymmetric traffic. Port clustering [referred to as Interface groups in the Manager] enables multiple ports on a single Sensor to be grouped together for effective traffic monitoring. Asymmetric networks are common in load balancing and active/ passive configurations, and a complete transmission may be received on one segment, but depart on another. Thus keeping state of asymmetric transmissions is essential for successfully monitoring the traffic. Interface groups normalize the impact of traffic flows split across multiple interfaces, thus maintaining state to avoid information loss. Place an IPS Sensor each on the request and the response path of the asymmetric traffic and create a failover pair to sync up the traffic flow between the two Sensors. If you are using a failover pair to monitor asymmetric traffic where the TCP traffic is going through two geographically different data centers, connect the Sensors using dark fiber. In this option, both the Sensors will have full state. When the distance between the two IPS Sensors is such that a failover pair cannot be created, consider enabling Stateless Inspection. In Stateless Inspection, the Sensor detects attacks without requiring a valid TCP state. This option should be used only when Sensors are placed in a network where the Sensors do not see all packets of a TCP flow like in an asymmetric network configuration. When Stateless Inspection is enabled: - ACLs and syn cookie protection cannot be enabled. - HTTP re to the Remediation Portal may or may not work depending on your network deployment scenario for example, in a setup where SYN+ACK packets cannot be sent from the Sensor to the client The diagram below explains about HTTP traffic flow in an asymmetric network between User A and the University Admin server. The outgoing connection flow from User A is through Switch 1, Switch 2, Network Security Sensor 1, Router 1, Internet Service Provider 1, to the Internet connection. The return path for the packet however, is through Internet Service Provider 2, Router 2 etc. If traffic flows by the Sensor in an asymmetric manner as described above, all packets of a TCP flow are not visible to a single Sensor. In such a scenario, if Stateless Inspection is enabled, the Sensor will inspect packets without having the valid state for the TCP connection. Consequently, it might generate false positives that is, when a single communication flow is divided across paths, each interface will receive and analyze part of the conversation and therefore be susceptible to false positives and false negatives. McAfee Network Security Platform 8.3 Best Practices Guide 31

32 11 How to handle asymmetric networks When you enable Stateless Inspection, there are chances of false positives, and the detection accuracy will be lower compared to when the Sensor sees all traffic. McAfee recommends that you use this feature only when network configuration does not allow the Sensor to be placed in locations where it could see all traffic. 32 McAfee Network Security Platform 8.3 Best Practices Guide

33 12 SSL best practices Note that there is a performance impact when using the SSL decryption feature. If there is a lot of outbound SSL traffic from the client to the internet as well, it consumes SSL flows. Therefore, to enable the Sensor to effectively utilize the SSL decryption feature, it is recommended to bypass these outbound SSL traffic using ACL Ignore rules. Refer to the following sections for the SSL throughput measurements and test methodologies. SSL decryption feature is not supported on Virtual Sensors such as NS3200, NS3100, IPS-VM600, IPS-VM100, and IPS-VM100-VSS. Contents SSL only traffic throughput: I-series Sensors SSL only traffic throughput: M-series Sensors SSL traffic mixed with HTTP 1.1 traffic: I-series Sensors SSL traffic mixed with HTTP 1.1 traffic: M-series Sensors SSL only traffic - throughput: NS-series Sensors SSL traffic mixed with HTTP 1.1 traffic: NS-series Sensors SSL only traffic throughput: I-series Sensors Session resumption for 4 out of 5 TCP connections 128-bit ARC4 5 HTTP 1.1 get page requests per TCP connection with a 5K response each I-2700 I-3000 I-4000 I-4010 Max. SSL Connections / Sec Throughput (Mbps) bit key length 85 Mbps 155 Mbps 200 Mbps 310 Mbps Throughput (Mbps) bit key length 65 Mbps 115 Mbps 125 Mbps 250 Mbps 5 HTTP 1.1 get page requests per TCP connection with a 10K response each I-2700 I-3000 I-4000 I-4010 Max. SSL Connections / Sec Throughput (Mbps) bit key length 150 Mbps 200 Mbps 400 Mbps 400 Mbps McAfee Network Security Platform 8.3 Best Practices Guide 33

34 12 SSL best practices SSL only traffic throughput: M-series Sensors SSL only traffic throughput: M-series Sensors Session resumption for 4 out of 5 TCP connections 5 HTTP 1.1 get page requests per TCP connection with a 10K response each 128-bit ARC4 Max. SSL Connections / Sec. Throughput (Mbps) bit key length Throughput (Mbps) bit key length M-8000 M-6050 M-4050 M-3050 M-2950 M-2850/ M Gbps 2 Gbps 1200 Mbps 600 Mbps 400 Mbps 250 Mbps 1.2 Gbps 600 Mbps 550 Mbps 320 Mbps 320 Mbps 200 Mbps SSL traffic mixed with HTTP 1.1 traffic: I-series Sensors Session resumption for 4 out of 5 TCP connections 5 HTTP 1.1 get page requests per TCP connection with a 5K response each 1024-bit RSA 128-bit ARC4 I-2700 Max. SSL Connections / Sec SSL Throughput 25 Mbps 50 Mbps HTTP 1.1 Throughput 475 Mbps 350 Mbps Total Throughput 500 Mbps 400 Mbps I-3000 Max. SSL Connections / Sec SSL Throughput 50 Mbps 105 Mbps HTTP 1.1 Throughput 860 Mbps 475 Mbps Total Throughput 910 Mbps 580 Mbps I-4000 Max. SSL Connections / Sec SSL Throughput 100 Mbps 200 Mbps HTTP 1.1 Throughput 1550 Mbps 780 Mbps Total Throughput 1650 Mbps 980 Mbps I-4010 Max. SSL Connections / Sec SSL Throughput 100 Mbps 200 Mbps 34 McAfee Network Security Platform 8.3 Best Practices Guide