Enhanced Load Balance Cluster FortiGate-5001A, FortiGate-5001B FortiSwitch-5003A, FortiSwitch-5003B Configuration Guide

Enhanced Load Balance Cluster FortiGate-5001A, FortiGate-5001B FortiSwitch-5003A, FortiSwitch-5003B Configuration Guide

13 July 2015 01-430-117726-20150713 Copyright 2015 Fortinet, Inc. All rights reserved. Fortinet, FortiGate, FortiCare and FortiGuard, and certain other marks are registered trademarks of Fortinet, Inc., in the U.S. and other jurisdictions, and other Fortinet names herein may also be registered and/ or common law trademarks of Fortinet. All other product or company names may be trademarks of their respective owners. Performance and other metrics contained herein were attained in internal lab tests under ideal conditions, and actual performance and other results may vary. Network variables, different network environments and other conditions may affect performance results. Nothing herein represents any binding commitment by Fortinet, and Fortinet disclaims all warranties, whether express or implied, except to the extent Fortinet enters a binding written contract, signed by Fortinet s General Counsel, with a purchaser that expressly warrants that the identified product will perform according to certain expressly-identified performance metrics and, in such event, only the specific performance metrics expressly identified in such binding written contract shall be binding on Fortinet. For absolute clarity, any such warranty will be limited to performance in the same ideal conditions as in Fortinet s internal lab tests. In no event does Fortinet make any commitment related to future deliverables, features, or development, and circumstances may change such that any forward-looking statements herein are not accurate. Fortinet disclaims in full any covenants, representations,and guarantees pursuant hereto, whether express or implied. Fortinet reserves the right to change, modify, transfer, or otherwise revise this publication without notice, and the most current version of the publication shall be applicable. Visit these links for more information and documentation for your Fortinet product: Technical Documentation - http://docs/fortinet.com Fortinet Knowledge Base - http://kb.fortinet.com Technical Support - http://support.fortinet.com Training Services - http://training.fortinet.com

Contents Introduction...1 About Load Balance Clustering... 1 Before you begin... 2 Document Conventions... 3 IP addresses... 3 Notes, Tips and Cautions... 3 Typographical conventions... 3 CLI command syntax conventions... 4 Registering your Fortinet product... 6 Fortinet products End User License Agreement... 6 Customer service and technical support... 6 Training... 7 Fortinet Documentation... 7 Fortinet Tools and Documentation CD... 7 Fortinet Knowledge Base... 7 Comments on Fortinet technical documentation... 7 Setting up the Load Balance Cluster...9 Prerequisites... 9 Build numbers... 9 Topology... 10 Configuring the FortiSwitch-5003A/B... 11 Configuring the Service Group Mode... 12 Setting the Load Balancing Algorithm... 15 Configuring the FortiGate-5001A/B... 16 Verifying the configuration... 18 Testing the configuration... 21 Traffic Calculator... 21 Traffic Monitor... 23 FortiSwitch HA Mode... 24 Logging with FortiAnalyzer... 25 High Availability Clusters...27 To Set Up HA in one chassis... 27 To Setup Cross-Chassis HA... 28 Session Synchronization... 29 Upgrading Firmware in HA Clusters... 29 FortiSwitch cluster... 30 How it works... 30 Why would you want to use it... 30 CLI/GUI configuration... 30 FortiGate cluster... 30 How it works... 30 01-430-117726-20150713 i

Why you would want to use it... 30 CLI/GUI configuration... 31 Appendix A - Upgrading Firmware...33 Upgrading FortiSwitch/B firmware... 33 Upgrading FortiGate/B firmware... 34 ii

1. Introduction Welcome and thank you for selecting Fortinet products for your network protection. This document describes how to configure the FortiGate-5001A/B and FortiSwitch- 5003A/B to work together to distribute traffic and increase performance. The FortiGate-5001A/B is a high-performance security blade that integrates: enterprise firewall, Virtual Private Network (VPN), intrusion prevention, antivirus/anti-malware, Web filtering, antispam, and application control features. The FortiGate-5001A/B offers flexible network interface options, including optional hardware-accelerated Gigabit and 10-Gigabit Ethernet support. The FortiSwitch-500A/B blade provides 10-Gigabit Ethernet switching services to the FortiGate-5001A/B security platform. Combined with the FortiGate-5140 chassis, the FortiGate-5001A/B provides multi-gigabit network security using a 10-Gigabit Ethernet network environment. Note: You cannot mix ELBC, FGCP and SLBC clusters in the same chassis. This chapter contains the following topics: About Load Balance Clustering Before you begin Document Conventions Registering your Fortinet product Customer service and technical support Training Fortinet Documentation About Load Balance Clustering The FortiSwitch-5003A/B and FortiGate-5001A/B together with the FortiGate-5140 chassis provides a clustering technology. It load balances network traffic across the cluster, helping to enhance the scalability, reliability, and availability of mission critical IP-based services, such as firewall, antivirus, web filtering, IPS, and so on. It also provides high availability by detecting host failures and automatically redistributing traffic to the hosts. The way that Enhanced Load Balance Clustering works is that the FortiSwitch- 5003A/B applies a load balancing algorithm against the source and/or destination address of the packet to generate a hash key value. Each FortiGate-5001A/B blade has hash key values assigned to it. If the worker blades are running, then the traffic is forwarded to the worker blade assigned to the hash key. See Figure 1. 01-430-117726-20150713 1

Before you begin Introduction Figure 1: Overview of how the load balance clustering works on the FortiSwitch-5003A/B. Internal Network Packet Packet IPs Source IP: 1.1.1.1 Destination IP: 2.2.2.2 External Network S1 F1/F2 F3/F4 S5 S7 Load balancing algorithm is applied to the packet and generates a Hash Key value of 1. Hash Key 1 is assigned to Slot 7 so traffic is sent to Slot 7. Slot 5 configuration: Source IP: 1.1.1.1 Destination IP: 2.2.2.2 Hash Keys accepted: 2/5/8/9/11/13/14/15/19/25/30 Traffic to: Slot 5 Slot 7 configuration: Source IP: 1.1.1.1 Destination IP: 2.2.2.2 Hash Keys accepted: 0/1/3/4/6/7/10/12/18/21/31 Traffic to: Slot 7 FortiSwitch-5003A FortiGate-5001A FortiGate-5001A The hash key value generated by the algorithm, the hash keys accepted by the worker blades, and the blade the traffic is sent to are automatically calculated by the FortiSwitch. Before you begin Before you begin using this guide, please ensure that: You have the following Fortinet hardware: FortiGate-5001A/B, FortiSwitch- 5003A/B, FortiGate-5140/5050/5060 Chassis, and a FortiAnalyzer. A FortiManager- 5001A is optional, but recommended. The correct software versions. You have administrative access to the web-based manager and/or CLI. The system time, DNS settings, administrator password, and network interfaces on your switch have been configured. Firmware, FortiGuard antivirus and FortiGuard antispam updates are completed. While using the instructions in this guide, note that: Administrators are assumed to be super_admin administrators unless otherwise specified. Some restrictions will apply to other administrators. 2 01-430-117726-20150713

Introduction Document Conventions Document Conventions Fortinet technical documentation uses the conventions described below. IP addresses Notes, Tips and Cautions To avoid publication of public IP addresses that belong to Fortinet or any other organization, the IP addresses used in Fortinet technical documentation are fictional and follow the documentation guidelines specific to Fortinet. The addresses used are from the private IP address ranges defined in RFC 1918: Address Allocation for Private Internets, available at http://ietf.org/rfc/rfc1918.txt?number-1918. Fortinet technical documentation uses the following guidance and styles for notes, tips and cautions. Tip: Highlights useful additional information, often tailored to your workplace activity. Note: Also presents useful information, but usually focused on an alternative, optional method, such as a shortcut, to perform a step. Caution: Warns you about commands or procedures that could have unexpected or undesirable results including loss of data or damage to equipment. Typographical conventions Fortinet documentation uses the following typographical conventions: Table 1: Typographical conventions in Fortinet technical documentation Convention Button, menu, text box, field, or check box label CLI input* CLI output Emphasis File content Hyperlink Keyboard entry Example From Minimum log level, select Notification. config system dns set primary <address_ipv4> end FortiGate-602803030703 # get system settings comments : (null) opmode : nat HTTP connections are not secure and can be intercepted by a third party. <HTML><HEAD><TITLE>Firewall Authentication</TITLE></HEAD> <BODY><H4>You must authenticate to use this service.</h4> Visit the Fortinet Technical Support web site, https://support.fortinet.com. Type a name for the remote VPN peer or client, such as Central_Office_1. 01-430-117726-20150713 3

Document Conventions Introduction Table 1: Typographical conventions in Fortinet technical documentation Navigation Publication Go to VPN > IPSEC > Auto Key (IKE). For details, see the FortiGate Administration Guide. Note: Links typically go to the most recent version. To access earlier releases, go to http://docs.fortinet.com/. This link appears at the bottom of each page of this document. * For conventions used to represent command syntax, see CLI command syntax conventions on page 4. CLI command syntax conventions This guide uses the following conventions to describe the syntax to use when entering commands in the Command Line Interface (CLI). Brackets, braces, and pipes are used to denote valid permutations of the syntax. Constraint notations, such as <address_ipv4>, indicate which data types or string patterns are acceptable value input. For more information, see the FortiGate CLI Reference. Table 2: Command syntax notation Convention Square brackets [ ] Description A non-required word or series of words. For example: [verbose {1 2 3}] indicates that you may either omit or type both the verbose word and its accompanying option, such as: verbose 3 4 01-430-117726-20150713

Introduction Document Conventions Table 2: Command syntax notation (Continued) Angle brackets < > A word constrained by data type. To define acceptable input, the angled brackets contain a descriptive name followed by an underscore ( _ ) and suffix that indicates the valid data type. For example: <retries_int> indicates that you should enter a number of retries, such as 5. Data types include: <xxx_name>: A name referring to another part of the configuration, such as policy_a. <xxx_index>: An index number referring to another part of the configuration, such as 0 for the first static route. <xxx_pattern>: A regular expression or word with wild cards that matches possible variations, such as *@example.com to match all email addresses ending in @example.com. <xxx_fqdn>: A fully qualified domain name (FQDN), such as mail.example.com. <xxx_email>: An email address, such as admin@mail.example.com. <xxx_url>: A uniform resource locator (URL) and its associated protocol and host name prefix, which together form a uniform resource identifier (URI), such as http://www.fortinet./com/. <xxx_ipv4>: An IPv4 address, such as 192.168.1.99. <xxx_v4mask>: A dotted decimal IPv4 netmask, such as 255.255.255.0. <xxx_ipv4mask>: A dotted decimal IPv4 address and netmask separated by a space, such as 192.168.1.99 255.255.255.0. <xxx_ipv4/mask>: A dotted decimal IPv4 address and CIDRnotation netmask separated by a slash, such as such as 192.168.1.99/24. <xxx_ipv6>: A colon( : )-delimited hexadecimal IPv6 address, such as 3f2e:6a8b:78a3:0d82:1725:6a2f:0370:6234. <xxx_v6mask>: An IPv6 netmask, such as /96. <xxx_ipv6mask>: An IPv6 address and netmask separated by a space. <xxx_str>: A string of characters that is not another data type, such as P@ssw0rd. Strings containing spaces or special characters must be surrounded in quotes or use escape sequences. <xxx_int>: An integer number that is not another data type, such as 15 for the number of minutes. 01-430-117726-20150713 5

Registering your Fortinet product Introduction Table 2: Command syntax notation (Continued) Curly braces { } Options delimited by vertical bars Options delimited by spaces A word or series of words that is constrained to a set of options delimited by either vertical bars or spaces. You must enter at least one of the options, unless the set of options is surrounded by square brackets [ ]. Mutually exclusive options. For example: {enable disable} indicates that you must enter either enable or disable, but must not enter both. Non-mutually exclusive options. For example: {http https ping snmp ssh telnet} indicates that you may enter all or a subset of those options, in any order, in a space-delimited list, such as: ping https ssh Note: To change the options, you must re-type the entire list. For example, to add snmp to the previous example, you would type: ping https snmp ssh If the option adds to or subtracts from the existing list of options, instead of replacing it, or if the list is comma-delimited, the exception will be noted. Registering your Fortinet product Before you begin, take a moment to register your Fortinet product at the Fortinet Technical Support web site, https://support.fortinet.com. Many Fortinet customer services, such as firmware updates, technical support, and FortiGuard Antivirus and other FortiGuard services, require product registration. For more information, see the Fortinet Knowledge Base article Registration Frequently Asked Questions. Fortinet products End User License Agreement See the Fortinet products End User License Agreement. Customer service and technical support Fortinet Technical Support provides services designed to make sure that your Fortinet products install quickly, configure easily, and operate reliably in your network. To learn about the technical support services that Fortinet provides, visit the Fortinet Technical Support web site at https://support.fortinet.com. You can dramatically improve the time that it takes to resolve your technical support ticket by providing your configuration file, a network diagram, and other specific information. For a list of required information, see the Fortinet Knowledge Base article Technical support requirements. 6 01-430-117726-20150713

Introduction Training Training Fortinet Training Services provides a variety of training programs to serve the needs of our customers and partners world-wide. Visit the Fortinet Training Services web site at http://campus.training.fortinet.com, or email training@fortinet.com. Fortinet Documentation The Fortinet Technical Documentation web site, http://docs.fortinet.com, provides the most up-to-date versions of Fortinet publications, as well as additional technical documentation such as technical notes. In addition to the Fortinet Technical Documentation web site, you can find Fortinet technical documentation on the Fortinet Tools and Documentation CD, and on the Fortinet Knowledge Base. Fortinet Tools and Documentation CD Fortinet Knowledge Base Comments on Fortinet technical documentation Many Fortinet publications are available on the Fortinet Tools and Documentation CD shipped with your Fortinet product. The documents on this CD are current at shipping time. For current versions of Fortinet documentation, visit the Fortinet Technical Documentation web site, http://docs.fortinet.com. The Fortinet Knowledge Base provides additional Fortinet technical documentation, such as troubleshooting and how-to articles, examples, FAQs, technical notes, and more. Visit the Fortinet Knowledge Base at http://kb.fortinet.com. Please send information about any errors or omissions in this or any Fortinet technical document to techdoc@fortinet.com. 01-430-117726-20150713 7

8

2. Setting up the Load Balance Cluster The FortiGate-5001A/B and FortiSwitch-5003A/B to work together to distribute traffic and increase performance. This section describes how to configure the FortiGate- 5001A/B and FortiSwitch-5003A/B to build up the Enhanced Load Balance Cluster (ELBC). The following topics are included in this section: Prerequisites Configuring the FortiSwitch-5003A/B Configuring the FortiGate-5001A/B Verifying the configuration Testing the configuration Traffic Calculator Traffic Monitor FortiSwitch HA Mode Logging with FortiAnalyzer Prerequisites The following prerequisites are required prior to configuring the FortiGate-5001A/B and FortiSwitch-5003A/B: FortiGate-5140, 5050 or 5060 Chassis 2-12 FortiGate-5001A/B blades 1-2 FortiSwitch-5003A/B blades 2-12 RTM-XD2 modules (not required for the FortiGate-5001B) The correct firmware and software versions for the RTM-XD2. See Appendix B - RTM-XD2 Modules on page 31. The correct build numbers. See Build numbers. The correct topology. See Topology on page 10. Build numbers This document is restricted to specific software and hardware version as following. FortiGate-5001A/B using the latest software: TBD FortiSwitch-5003A/B using the latest software: TBD Caution: Ensure you are using the latest version for the FortiGate-5001A and FortiSwitch- 5003A. For information on upgrading the firmware, see Appendix A - Upgrading Firmware. 01-430-117726-20150713 9

Prerequisites Setting up the Load Balance Cluster Topology The FortiSwitch-5003A/B must be located in slot 1 or 2. The FortiGate-5001A/B units can be placed into any slots except slot 1, 2, and 14. Slot 14 may be used if all other slots are populated. See Figure 2. Figure 2: Basic topology for load balance clustering. Internal Network Internal Switch S1 S5 S7 S9 S11 F2 F1 External Network External Switch Service Group 1 Management VIP 172.18.9.105 MGMT FortiSwitch-5003A FortiGate-5001A FortiGate-5001A FortiGate-5001A FortiGate-5001A Management computer 172.18.9.99 Management port 172.18.9.101 10 01-430-117726-20150713

Setting up the Load Balance Cluster Configuring the FortiSwitch-5003A/B Figure 3: FortiSwitch-5003A/B and FortiGate-5001A/B in FortiGate-5040 chassis. Configuring the FortiSwitch-5003A/B To upgrade the FortiSwitch-5003A/B, you will need to do the following: Factory reset the FortiSwitch and set the management IP address, access, and administrator time-out value. Set the Service Group Mode and configure the Service Groups. See Configuring the Service Group Mode on page 12 Set the type of load balancing algorithm to be used. See Setting the Load Balancing Algorithm on page 15. Add the worker blades (FortiGate-5001A/B). See To add the Service Group (worker blade) members on page 14. To configure the FortiSwitch-5003A/B 1 Factory reset the FortiSwitch-5003A/B blade. 2 Configure the management IP using the interface management. config system interface edit "mgmt" set ip 172.18.9.101 255.255.255.0 set allowaccess ping https ssh snmp telnet http end 01-430-117726-20150713 11

Configuring the FortiSwitch-5003A/B Setting up the Load Balance Cluster 3 Using the CLI console, enter the following commands: config system global set admintimeout 60 set hostname "FS5003A" (or "FS5003B") end Configuring the Service Group Mode You will need to select a Service Group Mode for the FortiSwitch-5001A/B. The Service Group Modes are: 2-Port-LAG Two fabric ports per internal/external interface. Select this option if you need more bandwidth than 10GB. This option aggregates F1 and F2 together into one port to connect to internal (providing 20GB of traffic), and F3 and F4 are connected to external. Only two service groups are available in this mode. 4-Port-LAG Four fabric ports per internal/external interface. Select this option if you need more bandwidth than 10GB. This option aggregates F1, F2, F3, and F4 to connect to internal (providing 40GB of traffic) and F5, F6, F7, and F8 to connect to external. Only one service group is available in this mode. Basic A single fabric port per internal/external interface. Four service groups are available in this mode. Each port provides 10GB of bandwidth. Disabled Regular switch mode. No load balance clustering. In the example below, a 2-Port-LAG is selected for the FortiSwitch-5003A/B. For Service Group 1, the fabric ports F1 and F2 are aggregated to make the internal port and F3 and F4 are aggregated to make the external port. To set the Service Group Mode 1 Go to System > Status > System Information area. In the Service Group Mode field, click Change to select the service group mode: 2-Port-Lag 4-Port-Lag Basic Disabled Figure 4: FortiSwitch-50003A/B using 2-Port-Lag mode. 2 Once the service-group-mode is enabled, the eight front panel ports will be assigned to service groups. The number of service groups depends on the type of Service Group Mode selected. In the example above, a 2-port-lag mode was selected; this means that there will be two service groups. 12 01-430-117726-20150713

Setting up the Load Balance Cluster Configuring the FortiSwitch-5003A/B Figure 5: Service Group 1 assigned ports. F1 and F2 are assigned as internal. F3 and F4 are assigned as external. To configure the Service Groups 1 Go to Service Groups > Service Group number > Config tab. 2 Enter the following information: Enable Service Group Select the check box to enable the Service Group. Link Aggregation Mode (not available in Basic mode) Link Aggregation Control Protocol (LACP) is used to negotiate a dynamic aggregated link between this Switch and the FortiGate units. Active - LACP mode that places a port into an active negotiating state in which the port initiates negotiations with other ports by sending LACP packets. Passive - LACP mode that places a port into a passive negotiating state in which the port responds to LACP packets that it receives but does not initiate LACP packet negotiation. LACP Speed Select the speed of the LACP. The speed determines the time (not available in Basic between heartbeats between LACP peers: mode) Fast - every 2 seconds. Slow - every 30 seconds. External Management IP/Netmask Internal Management Network Administrative Access Enter the IP address for the Service Group management VIP on the FortiSwitch-5003A/B MGMT port. This VIP is used to access the management features (HTTP, HTTPS, SSH/TELNET) of the running worker blades. This network is used internally between the FortiSwitch-5003A/B and FortiGate-5001A/B. Use the default IP address unless it conflicts with another network in your environment. Select the administrative access types that are accessible via the External Management IP. HTTPS PING HTTP SSH SNMP TELNET FGFM 01-430-117726-20150713 13

Configuring the FortiSwitch-5003A/B Setting up the Load Balance Cluster Figure 6: Configuring Service Group 1 in 2-Port-LAG mode. 3 The system will reboot after changing the Service Group mode. You will need to login again. To add the Service Group (worker blade) members 1 Go to Service Groups > Service Group number > Config tab and click Edit in the Group Membership area. Figure 7: Selecting the worker blades. The first blade in the list is the Master blade. 2 Move the blades from the Available Slots area to the Members area using the Arrow buttons. The first slot in the list will typically be be the Master blade, but this can be changed if you require. 3 Click OK. 4 Go to Service Groups > Service Group number > Status to view the current status of the blades. The status should be down, indicating that the blades are not ready to pass traffic. 14 01-430-117726-20150713

Setting up the Load Balance Cluster Configuring the FortiSwitch-5003A/B Figure 8: Worker blade status. Setting the Load Balancing Algorithm The load balancing algorithm is used to determine the Hash Key value of the packet. The algorithm is applied to the source and destination IP address of the packet and converts it to a Hash Key value. If the worker blades are running, then the traffic is forwarded to the worker blade assigned to the hash key. There are three types of load balancing algorithms that can be used: Hash IP Least 3-bits Uses the least 3 bits of both the source and destination addresses. This allows traffic to enter and leave from the same port (single port traffic). However, this algorithm prevents you from using NAT mode. In order to use this load balancing algorithm, the service group hash size must first be set to Expanded. Hash IP Least 5-Bits (service group hash size = normal) Uses the least 5 bits of both the source and destination addresses. This is the default algorithm. It supports up to four service groups. Snat or dnat mode is supported. Snat mode is traffic that is hashed by destination address on the internal port and source address on the external port. Dnat mode is traffic that is hashed by the destination address on the external port and source address on the internal port. Hash IP Least 6-Bits (service group hash size = expanded) Uses the least 6 bits of both the source and destination addresses. Using this algorithm may result in traffic that is better balanced. It supports up to two service groups. Snat, dnat, or nonat is supported. Snat mode is traffic that is hashed by destination address on the internal port and source address on the external port. Dnat mode is traffic that is hashed by the destination address on the external port and source address on the internal port. 01-430-117726-20150713 15

Configuring the FortiGate-5001A/B Setting up the Load Balance Cluster To set the load balancing algorithm to 5-bits (normal) or 6-bits (expanded) 5 Go to System > Status > CLI Console. 6 Type the following commands: config system global set service-group-hash-size <normal or expanded> (Where Normal is the 5-Bits load balancing algorithm and Expanded is the 6-Bits load balancing algorithm, keeping in mind that if you use expanded you are limited to a maximum of two service groups, regardless of mode used.) y (type y at the Do you want to continue? prompt) end 7 The system will reboot. You will need to login again. To set the load balancing algorithm to 3-bits 1 Ensure the service group hash size is set to expanded. See To set the load balancing algorithm to 5-bits (normal) or 6-bits (expanded). 2 In the CLI type the following: config service group edit <service group id number> set status enable set mode nonat end Configuring the FortiGate-5001A/B Use the following steps to configure the FortiGate-5001A/B. To upgrade the FortiGate-5001A/B 1 Factory reset the FortiGate-5001A/B blades. After the blade boot up, you will need to execute the following commands: config system elbc set status enable end y (type y at the Do you want to continue? prompt) 2 The unit will reboot, then automatically join the ELBC cluster. Once the blade has synchronized with the master of the cluster, two logical ports (internal and external) are created. Figure 9: Internal and external interfaces created automatically on the FortiGate-5001A. 16 01-430-117726-20150713

Setting up the Load Balance Cluster Configuring the FortiGate-5001A/B 3 If you wish, you can create a VLAN interface to forward traffic on the VLAN arriving on the FortiSwitch-5003A/B front panel ports. Untagged traffic arriving on the front panel ports of the FortiSwitch-5003A/B will appear untagged on the internal or external devices on the FortiGate-5001A/B. config system interface edit vlan901 set vdom vd_901_902 set ip 91.1.1.8 255.255.255.0 set allowaccess ping https ssh snmp http telnet set interface internal set vlanid 901 next end 4 Go to Firewall > Policy > Policy tab and click Create New to create the policy for the vlan. 5 Enter the following information in the New Policy form and click OK: Source Interface Source Address Destination Interface/Zone Destination Address Schedule Service Action vlan901 All vlan902 All Always Any Accept Figure 10: Configuring a policy for the vlan. 01-430-117726-20150713 17

Verifying the configuration Setting up the Load Balance Cluster Verifying the configuration Using the CLI console, you will need to verify that the settings you configured for the FortiGate-5001A/B and FortiSwitch-5003A/B are working. To verify the configuration for the FortiSwitch-5003A/B 1 Enter the following commands in the CLI: FS5003A # exe get service group status Service Group: 1 ELBC Master Blade: slot-5 Confsync Master Blade: slot-5 Blades: Working: 2 [ 2 Active 0 Standby] Ready: 0 [ 0 Active 0 Standby] Dead: 0 [ 0 Active 0 Standby] Total: 2 [ 2 Active 0 Standby] Slot 5: Status:Working Function:Active Fabric Channel: Link:Up HeartBeat:Good Base Channel: Link:Up HeartBeat:Good Status Message:"Running" Slot 7: Status:Working Function:Active Fabric Channel: Link:Up HeartBeat:Good Base Channel: Link:Up HeartBeat:Good Status Message:"Running" Service Group: 2 Disabled ELBC Master Blade The master blade slot number. Also handles dynamic routing. Working Ready Dead Status Message The number of blades handling the traffic. The number of blades that are available as standby blades but not in use. The number of blades that have failed or are not ready and cannot handle traffic. Blade status. May contain simple information to help you troubleshoot such as Waiting for fabric link or Waiting for confsync. 2 To view the Service Group 1 status, go to Service Group > Status tab in Web Config. The status should be Running and the link should be Up. In the example in Figure 11, Blade 5 is configured to be the master blade. For more information about adding the worker blades, see To add the Service Group (worker blade) members on page 14. 18 01-430-117726-20150713

Setting up the Load Balance Cluster Verifying the configuration Figure 11: Service Group 1 status. Master blade Configure master blade icon To verify the configuration for the FortiGate-5001A/B 1 Enter the following commands in the CLI: ELBCv3-slot5 (global) # d sys fortiswitch-hearbeat status Heartbeat Status: 3 (0 = disabled, 1 = configuring, 2 = waiting for heartbeat from FortiSwitch, 3 = good heartbeat received from FortiSwitch) Heartbeat Packet Interval: 0.2s My Slot: 5 My Chassis: 1 Channel-0: flags(0xf) (Channel-0 = FortiSwitch slot 1, Channel-1 = FortiSwitch slot 2) Status: enabled FSW-HB: good FSW-Active: yes HB-Tx: enabled Heartbeat Packet Sending Device: elbc-ctrl/1 last_rx=19 Traffic Handling Devices: 2 internal/1 external/1 Swdev: base1 Slot Swdev MAC Serial-Number 0 ff:ff:ff:ff:ff:ff N/A 1 ff:ff:ff:ff:ff:ff N/A 2 ff:ff:ff:ff:ff:ff N/A 3 ff:ff:ff:ff:ff:ff N/A 4 ff:ff:ff:ff:ff:ff N/A 5 00:09:0f:8a:05:3a FG5A013408600211 6 ff:ff:ff:ff:ff:ff N/A 01-430-117726-20150713 19

Verifying the configuration Setting up the Load Balance Cluster 7 00:09:0f:8a:02:fa FG5A013E08600125 8 ff:ff:ff:ff:ff:ff N/A 9 ff:ff:ff:ff:ff:ff N/A 10 ff:ff:ff:ff:ff:ff N/A 11 ff:ff:ff:ff:ff:ff N/A 12 ff:ff:ff:ff:ff:ff N/A 13 ff:ff:ff:ff:ff:ff N/A 14 ff:ff:ff:ff:ff:ff N/A 15 ff:ff:ff:ff:ff:ff N/A Service Group: 1 Active Slots: 000000a0(1.5,1.7) Master Slot: 5 Master Chassis: yes Channel-1: flags(0x0) Status: disabled FSW-HB: fail FSW-Active: no HB-Tx: disabled Heartbeat Packet Sending Device: none last_rx=0 Traffic Handling Devices: 0 Swdev: none Slot Swdev MAC Serial-Number 0 ff:ff:ff:ff:ff:ff N/A 1 ff:ff:ff:ff:ff:ff N/A 2 ff:ff:ff:ff:ff:ff N/A 3 ff:ff:ff:ff:ff:ff N/A 4 ff:ff:ff:ff:ff:ff N/A 5 ff:ff:ff:ff:ff:ff N/A 6 ff:ff:ff:ff:ff:ff N/A 7 ff:ff:ff:ff:ff:ff N/A 8 ff:ff:ff:ff:ff:ff N/A 9 ff:ff:ff:ff:ff:ff N/A 10 ff:ff:ff:ff:ff:ff N/A 11 ff:ff:ff:ff:ff:ff N/A 12 ff:ff:ff:ff:ff:ff N/A 13 ff:ff:ff:ff:ff:ff N/A 14 ff:ff:ff:ff:ff:ff N/A 15 ff:ff:ff:ff:ff:ff N/A Service Group: 0 Active Slots: 00000000() Master Slot: 0 Master Chassis: no FSW-Active HB-Tx Active Slots Indicates the connected FortiSwitch is ready to handle traffic, either as an HA Master or as a standalone unit. Indicates that the FortiGate is ready and sending a heartbeat to the FortiSwitch. The slots that the FortiSwitch-5003A/B is sending traffic too. 2 In the FortiGate-5001A Web Config, go to System > Config > HA to see the that the worker blade is in Enhanced Load Balance Cluster mode. You can also see the Service Group ID number, MAC addresses, allow access, and slot ID numbers. 20 01-430-117726-20150713

Setting up the Load Balance Cluster Testing the configuration Figure 12: Enhanced Load Balance Cluster mode on the worker blade. Testing the configuration In order to test the configuration, you will need a load testing appliance such as Spirent Avalanche (www.spirent.com). Using Spirent Avalanche, generate some traffic with big range source IP addresses. This traffic will be evenly distributed to the worker blades. You should be able to see the linear increase for the performance. Before you begin load testing, ensure the topology is set up as such: Avalanche -------(F1)---- 5003A/B===Worker Blades (slot 5, 7, 9, 11) Reflector ---------(F4)--- Traffic Calculator The Traffic Calculator tab is a tool used to understand how a packet will be load balanced based on the current state of the cluster. The only information that is required is the Source and the Destination IPs that correspond to the source and destination IPs of the packet, and the direction of the of the ports (i.e. internal to external or vice versa). The Hash Key value generated by the algorithm, the Hash Keys accepted by the worker blades, and the blade the traffic is sent to are automatically calculated and displayed by the FortiSwitch. The source and destination addresses, combined with the direction of the ports, the runtime state of the load balancing, and configuration of the load balancing (expanded/normal mode setting, and service group mode setting) will be fed into the software emulation of the hashing algorithm and to produce a hash key. The type of load balancing algorithm is displayed. The type of algorithm used cannot be set in the Traffic Calculator tab. See Setting the Load Balancing Algorithm on page 15. 01-430-117726-20150713 21

Traffic Calculator Setting up the Load Balance Cluster Figure 13: Traffic calculator tab. To set the Traffic Calculator 1 Go to Service Groups > Service Group number > Calculator tab. 2 In the Direction field, select the direction for the traffic. Note: If you are using the Hash IP Least 3-bits load balancing algorithm, the direction does not matter since traffic is hashed the same way in all directions. 3 In the Src IP field, enter the source IP address of the packet. 4 In the Dst IP field, enter the destination IP address of the packet. 5 Click Apply. The Hash Key, Traffic To, and the hash keys assigned to the worker blades are calculated automatically by the system. 22 01-430-117726-20150713

Setting up the Load Balance Cluster Traffic Monitor Traffic Monitor The Traffic Monitor tab displays how much traffic is being processed on each port for each blade. Figure 14: Traffic Monitor tab. To view the traffic 1 Go to Go to Service Groups > Service Group number > Traffic Monitor tab. 2 In this tab you can set the following: Time Period Statistics Select: Last hour, Last day, Last 30 days Select: Tx and Rx, Rx only, Tx only Auto Refresh Select: Disabled, Every 10 seconds, Every 30 seconds, Every 60 seconds, Every 5 minutes, Every 10 minutes, Every 30 minutes, Every 1 hour 3 Click Apply to see the changes to the graphs. 01-430-117726-20150713 23

FortiSwitch HA Mode Setting up the Load Balance Cluster FortiSwitch HA Mode You can use two FortiSwitch-5003A/B units to set up high availability (HA) for the load balancing cluster. For HA mode, ensure that the FortiSwitch-5003A/B units are in slot 1 and 2 of the chassis. Figure 15: FortiSwitch-5003A/B units in HA mode. S1 S5 S7 S9 S11 S2 MGMT MGMT FortiSwitch-5003A FortiGate-5001A FortiGate-5001A FortiGate-5001A FortiGate-5001A FortiSwitch-5003A F2 F1 You can set the priority for the HA mode if you want to force a certain blade to be the master blade. To set a priority HA mode 4 Go to System > Status > CLI Console. 5 Type the following commands: config system ha set mode a-p set priority 200 set hbdev <mgmt> (5003A only) or <mgmt/b1/b2> (5003B only) end To verify HA mode 1 Go to System > Status > CLI Console. 2 Type the following command: diag system ha status 24 01-430-117726-20150713

Setting up the Load Balance Cluster Logging with FortiAnalyzer 3 The output should be similar to the following: mode: a-p FS5A033E08000035, Master(priority=0), ip=169.254.0.1, uptime=175.68, in_sync=1 service-group-1, status=enable, worker_failure=0, lag_link_failure=1 service-group-2, status=disable, worker_failure=0, lag_link_failure=2 FS5A033E08000030, Slave(priority=1), ip=169.254.0.2, uptime=20.45, last_hb_time=1953.13, in_sync=0, conn=3(connected) service-group-1, status=enable, worker_failure=0, lag_link_failure=1 service-group-2, status=disable, worker_failure=0, lag_link_failure=2 Logging with FortiAnalyzer You can send logs to the FortiAnalyzer through the b1, b2 or Base-10GB (5003A only) port of the FortiSwitch-5003A/B. Fortinet recommends that you use a FortiAnalyzer- 4000A unit or higher for logging. The VLANs used are 100 plus the service group number. For example, VLAN 101 is Service Group 1, VLAN 104 is Service Group 4, etc. This is the default numbering scheme, but you can change this if you require. The IP address must be with in the mgmt-internal-network for that service group but cannot be the first 3 host addresses. This means if the subnet is: x.x.x.0/24 it cannot be.1,.2,.3, however; If you have a non-boundary subnet like x.x.x.192/26 then you cannot use x.x.x.193, x.x.x.194, and x.x.x.195. 01-430-117726-20150713 25

1 2 A Logging with FortiAnalyzer Setting up the Load Balance Cluster Figure 16: Load balance cluster logging with the FortiAnalyzer-4000A FortiAnalyzer-4000A S1 S5 S7 S9 S11 10.101.10.3 Switch Base-10G MGMT FortiSwitch-5003A FortiGate-5001A FortiGate-5001A FortiGate-5001A FortiGate-5001A Management computer 172.18.9.99 Management port 172.18.9.101 26 01-430-117726-20150713

3. High Availability Clusters To Set Up HA in one chassis 1 Wire it up: Connect the MGMT port of both FSW to an external switch, and allow VLAN 999 tagged on those ports. (VLAN 999 is the default vlan used for HA and FSW confsync traffic. This VLAN can be changed by changing it's value like so: config sys ha set hbdev-vlan-id 737 end Connect the b1 port of all the FSW to an external switch. Allow tagged VLAN 101 (default base-mgmt VLAN for service group 1). 2 Configure basic HA On both FSW: config sys ha set mode a-p set hbdev mgmt end 3 Wait for cluster to form: Check the following output: diagnose sys ha status mode: a-p 5060-Sl1(FS503B3E10700053), Master(priority=0), ip=169.254.12.1, uptime=66114.57, chassis=1(1) sync: conf_sync=1, elbc_sync=1 state: worker_failure=0/2, intf_state=(total/good/down/bad-score)=2/2/0/0 hbdevs: local_interface= mgmt best=yes 5060-Sl2(FS503B3E10700037), Slave(priority=1), ip=169.254.12.2, uptime=66108.44, chassis=1(1) sync: conf_sync=1, elbc_sync=1, conn=3(connected) state: worker_failure=0/2, intf_state=(total/good/down/bad-score)=2/2/0/0 hbdevs: local_interface= mgmt last_hb_time=66143.42 status=alive Give the units some time (~1-2 minutes) to achieve confsync. You can check the confsync status in the diag sys ha status, look for the line sync: conf_sync=1, elbc_sync=1 on all devices listed. (1 means sync, 0 means not yet synced). 01-430-117726-20150713 27

To Setup Cross-Chassis HA High Availability Clusters 4 Configure the base-mgmt network of service group 1 so that base-mgmt traffic can leave/enter the cluster through the b1 port. This connection is so that we can use the base-mgmt-external-ip to access the slave fortiswitch. config service group edit 1 config base-mgmt-interfaces edit b1 get name : b1 vlan-id : 101 <----- make sure this VLAN is end end allowed on your external switch To Setup Cross-Chassis HA To setup cross chassis HA we will build on the in-chassis config. The important change is that we setup the "elbc-base-ctrl" network so that both chassis can talk to each other. This network holds the traffic the FGs use for all of our automatic synchronization (confsync, route sync, session sync (see Session Synchronization on page 29), etc). 1 Wire it up: Wire it up the same as in in-chassis On the external switch ports connected to b1 allow tagged VLAN 301 as well. (default elbc-base-ctrl VLAN for service group 1). 2 Configure HA on the FSW: Configure the same as in chassis enable chassis-redundancy and set the chassis number appropriately (eg. switches in the same chassis should have the same value, and different chassis should have a different value). eg. Set this on the FortiSwitches in chasiss #1: config sys ha set chassis-redundancy enable set chassis-id 1 end Set this on FortiSwitches in chassis #2: config sys ha set chassis-redundancy enable set chassis-id 2 end 3 Configure the base-mgmt network of service group 1 so that base-mgmt traffic can leave/enter the chassis through the b1 port. This connection is just so that we can use the base-mgmt-external-ip to access the slave fortiswitches in both chassis, and the FortiGates in both chassis. Same as above. 28 01-430-117726-20150713

High Availability Clusters Session Synchronization 4 Configure the elbc-base-ctrl network of service group 1 so that elbc-base-ctrl traffic can leave/enter the chassis through the b1 port. This is so that the FortiGates in both chassis can talk to eachother for synchronization purposes. config service group edit 1 config elbc-base-ctrl-interfaces edit b1 get name : b1 vlan-id : 301<----- make sure this VLAN is end end allowed on your external switch Session Synchronization There are two types of session synchronization: manually configured session sync, and a service group configuration option. The manually configured session sync is a FortiOS feature which can be used for custom session synchronizations in ELBCv3 clusters. For more information see the FortiOS Handbook, available at http://docs.fortinet.com/fgt40mr3.html. The service group configuration options are unique to ELBCv3. In a multi-chassis ELBCv3, session synchronization can be enabled using the following CLI command: config service group edit <service group number> set session-sync enable next... end Session synchronization will then automatically be configured between blades in the same slot in different chassis, using the IP addesses assigned to the blades on their elbc-base-ctrl interfaces. The chassis ID must be correctly set on each FSW to support this properly. You must also ensure that the elbc-base-ctrl network is connected between the two chassis. This is done by configuring an elbc-base-ctrl-interface in the service group configuration on all FSW. See To Setup Cross-Chassis HA on page 28. Upgrading Firmware in HA Clusters HA Clusters have a unique upgrade method known as "graceful update". Graceful update is designed to facilitiate upgrading the firmware image of a cluter of blades with little to no disruption of the network. There are two graceful upgrade clusters, the FortiGate cluster, and the FortiSwitch cluster. 01-430-117726-20150713 29

Upgrading Firmware in HA Clusters High Availability Clusters FortiSwitch cluster How it works The image to upgrade to is loaded to the HA master. The HA master FSW will distribute the image to all the slave FortiSwitches. It will wait for them to upgrade, resume the HA cluster, and synchronize ELBC cluster state, and configuration. Once the slaves have returned and synchronized, the master will failover to one of the upgrade slaves, in most cases the slave in the same chassis as the master, and upgrade itself. Upon completing it's own upgrade the former master may become a slave or regain mastership depending on the configuration. Why would you want to use it A graceful upgrade of the FortiSwitches results little to no loss of capacity, especially in a cluster with FortiGate-5001Bs. CLI/GUI configuration The feature is enabled by default, and can be disabled here: config system ha set graceful-upgrade <disable enable> end The only other requirement is that the HA cluster be completely formed, and confsync connections are established. All of this should happen automatically on a properly configured cluster. To use the feature simply update the firmware using either the CLI or the GUI. The master will send out the image to all slaves, then await their return to the cluster. Once they all returned, the master starts waiting for the slaves to complete ELBC and configuration sync. Once all slaves are synced the master automatically updates itself. FortiGate cluster How it works The FortiGate graceful upgrade is only support in dual-chassis HA setups. It will function in a single chassis setup, but is not very useful and results in a huge drop in capacity (as the master upgrades all the other FortiGates, leaving only itself to handle all the clusters traffic). In a dual chassis setup the image should be uploaded to the confsync master via the GUI or CLI commands. The confsync master will then send the image to all the blades in the slave chassis. They will upgrade, reboot, re-join the cluster, confsync to the master in the master chassis, and if configured, synchronize sessions with their peers in the master chassis. Once the slave chassis is upgraded and synchronized the master chassis will wait for the administrator to manually initiate a chassis failover. This is usually done on the FortiSwitch by enabling "override" in the system.ha configuration, and then changing the "priority" setting on a FortiSwitch in the slave chassis to be higher than any other FortiSwitch. Once the failover has ocurred, the FortiGates in the former master chassis will upgrade, reboot, and resync. Why you would want to use it In single chassis setup you would not want to use this feature. In dual chassis setups this feature will result in no loss of capacity. It will result in loss of UTM sessions, although pure-firewall sessions will be retained. 30 01-430-117726-20150713

High Availability Clusters Upgrading Firmware in HA Clusters CLI/GUI configuration There is normally no configuration required. "Graceful-upgrade" is enabled by default once the system is in "service-group" mode. The feature can be disabled, or reenabled using the following CLI: config system elbc set graceful-upgrade <enable disable> end Next connect to the confsync master via the base-mgmt-external-ip or some other means and upload an image through the CLI or GUI. Once the slave chassis has upgraded and restarted, the administrator can take a few minutes to confirm that the config, and other factors, are still correct. When ready to complete the upgrade connect to the CLI of a FortiSwitch in the slave chassis and set the HA priority to a high value: config sys ha set priority 254 end Wait for a failover, (enabling "override" the FortiSwitches if necessary). Once the failover and upgrade of the master chassis is complete it is advisible to restore the "priority" setting to it's previous value. Please note that on the FortiGates there is a 20 minute timeout. If the upgrade, and failover have not taken place within this timeout the cofnsync master performing the upgrade will re-sync it's image to the slave FortiGates and abort the graceful-upgrade. 01-430-117726-20150713 31

32

APPENDIX A - UPGRADING FIRMWARE Upgrading FortiSwitch/B firmware Fortinet periodically updates the FortiSwitch/B FortiOS firmware to include enhancements and address issues. After you have registered your FortiSwitch/B security system (see Registering your Fortinet product on page 6) you can download FortiSwitch/B firmware from the support web site http://support.fortinet.com. To upgrade the firmware using the web-based manager: 1 Copy the firmware image file to your management computer. 2 Log into the web-based manager as the admin administrator. 3 Go to System > Status. 4 Under System Information > Firmware Version, select Update. 5 Type the path and filename of the firmware image file, or select Browse and locate the file. 6 Select OK. The FortiSwitch-5003A/B board uploads the firmware image file, upgrades to the new firmware version, restarts, and displays the FortiSwitch-5003A/B login. This process takes a few minutes. 7 Log into the web-based manager. 8 Go to System > Status and check the Firmware Version to confirm the firmware upgrade is successfully installed. To upgrade the firmware via TFTP: To use the following procedure, you must have a TFTP server the FortiSwitch/B board can connect to. 1 Make sure the TFTP server is running. 2 Copy the new firmware image file to the root directory of the TFTP server. 3 Log into the FortiSwitch/B CLI. 4 Make sure the FortiSwitch board can connect to the TFTP server. You can use the following command to ping the computer running the TFTP server. For example, if the IP address of the TFTP server is 192.168.1.168: execute ping 192.168.1.168 5 Enter the following command to copy the firmware image from the TFTP server to the FortiSwitch/B board: execute restore image tftp <name_str> <tftp_ipv4> Where <name_str> is the name of the firmware image file and <tftp_ipv4> is the IP address of the TFTP server. For example, if the firmware image file name is image.out and the IP address of the TFTP server is 192.168.1.168, enter: execute restore image tftp image.out 192.168.1.168 The FortiSwitch/B board responds with the message: This operation will replace the current firmware version. Do you want to continue? (y/n) 01-430-117726-20150713 33

Upgrading FortiGate/B firmware Appendix A - Upgrading Firmware 6 Type y. The FortiSwitch/B board uploads the firmware image file, upgrades to the new firmware version, and restarts. This process takes a few minutes. Note: if running in HA mode, the FortiSwitch will send the updated image to its peers. 7 Reconnect to the CLI. 8 To confirm the firmware image is successfully installed, enter: get system status Upgrading FortiGate/B firmware Fortinet periodically updates the FortiGate/B FortiOS firmware to include enhancements and address issues. After you have registered your FortiGate/B security system (see Registering your Fortinet product on page 6) you can download FortiGate/B firmware from the support web site http://support.fortinet.com. Only FortiGate/B administrators (whose access profiles contain system read and write privileges) and the FortiGate/B admin user can change the FortiGate/B firmware. For complete details about upgrading and downgrading FortiGate/B firmware using the web-based manager or CLI; and using a USB key, see the FortiGate-5000 Series Firmware and FortiUSB Guide. To upgrade the firmware using the web-based manager: 1 Copy the firmware image file to your management computer. 2 Log into the web-based manager as the admin administrator. 3 Go to System > Status. 4 Under System Information > Firmware Version, select Update. 5 Type the path and filename of the firmware image file, or select Browse and locate the file. 6 Select OK. The FortiGate/B board uploads the firmware image file, upgrades to the new firmware version, restarts, and displays the FortiGate/B login. This process takes a few minutes. 7 Log into the web-based manager. 8 Go to System > Status and check the Firmware Version to confirm the firmware upgrade is successfully installed. 9 Update the FortiGate/B antivirus and attack definitions. See the FortiGate/B online help for details. To upgrade the firmware using TFTP: To use the following procedure, you must have a TFTP server the FortiGate/B board can connect to. 1 Make sure the TFTP server is running. 2 Copy the new firmware image file to the root directory of the TFTP server. 3 Log into the CLI. 34 01-430-117726-20150713