Understanding Virtual Router and Virtual Systems

Understanding Virtual Router and Virtual Systems PAN- OS 6.0 Humair Ali Professional Services

Content Table of Contents VIRTUAL ROUTER... 5 CONNECTED... 8 STATIC ROUTING... 9 OSPF... 11 BGP... 17 IMPORT RULE... 21 REDISTRIBUTE RULE... 28 IBGP... 41 ROUTE REFLECTOR... 45 HIGH AVAILABILITY... 49 ACTIVE- PASSIVE... 49

1 3 2 4 5 7 6 8 MGT CONSOLE USB POWER FAN ALARM STATUS HA TEMP PA-500 1 3 2 4 USB 1 / 2 5 6 CONSOLE 7 8 9 11 10 12 MGT 13 14 15 16 HA1 17 19 21 23 18 20 22 24 HA2 POWER STATUS ALARM FANS HA PWR 1 TEMP PWR 2 PA-4020 Purpose of this Document The purpose of this document is to explain how Virtual Router and Virtual system works and some example on they need to be configured. It is not a complete guide for every scenario but is aimed at giving pointers on how to achieve some of Routing or Virtual System configurations This document is mostly focus on examples where the Palo Alto Networks Firewall is configured in Layer 3 mode and not for Layer 2 or Vwire. The Inter- vsys is inherent as most Virtual Routers are in their own Virtual Systems, so most of the Routing that we will see are setup across Multi- VSYS Lab Setup All the commands and output shown in this document is based on the PA-4020 unless specifically said otherwise PANOS 6.0.10 PA-4020 in HA Active/Passive Physical Setup INTERNET Default GW 10.193.56.1.200 Ubuntu Desktop 10.193.58.x/24 Cisco SG500 Switch.22 PA- VM- 300 10.193.57.x/24.2 PA- 500 Eth1/3 10.193.56.x/24 Default GW to PA- 500 10.193.56.2 Vsys1 Eth1/1 EBGP Vsys2 Eth1/2.200.2.2 PA- 4020

Virtual Routers in Multi-VSYS environment Virtual Router The firewall uses virtual routers to obtain routes to other subnets by manually defining a route (static routes) or through participation in Layer 3 routing protocols (dynamic routes). All routes learned are first place in the routing information base (RIB) The best route obtained through these protocol are used to populate the firewall s Forwarding information base (FIB). When a packet is destined for a different subnet, the Virtual Router obtains the best route from this FIB and forwards the packet to the next hop router defined in the table. You can configure the virtual router to participate with dynamic routing protocols (BGP, OSPF, or RIP) as well as adding static routes. You can also create multiple virtual routers (VR), each maintaining a separate set of routes that are not shared between virtual routers, enabling you to configure different routing behaviors for different interfaces. 1 VSYS can have all multiple VR s as well as multiple VSYS can share the same VR Each Layer 3 interface, loopback interface, and VLAN interface defined on the firewall must be associated with a virtual router. While each interface can belong to only one virtual router, multiple routing protocols and static routes can be configured for a virtual router. The firewall uses Ethernet switching to reach other devices on the same IP subnet. The common command in Virtual Router will start with the following: Ø show routing route Ø show routing fib Ø show routing protocol xxx Ø Show routing route type xxxx Show routing route is essentially the RIB of the Firewall è Routing Table in Management Plane Show routing FIB is essentially the FIB of the Firewall è Best routes selected and installed in the DataPlane, and is the routes use for actual packet forwarding

In my lab, the output on PA- 4020 > show routing route Ø Show routing fib

If you look at the Show routing route, it will display the RIB table from the Firewall All the routes that are active are marked with an A in the flags column. If you look at the show routing fib, all the route entries in it are the one marked with A in show routing route and only those ones. CONNECTED Connected routes are IP Subnet belonging to the Firewall interfaces itself. All interfaces configured in the Firewall will belong to this category. Connected are the most commonly routes used for Redistribution in other Routing Protocols with Static routes.

**Important** any /32 interface (including Loopback) configured on the Firewall are not considered Connected but Host, and therefore cannot be redistributed. As you can see from above screenshot, Host are not define in the Redistribution Profiles and cannot therefore be redistributed into OSPF. The workaround is to advertise those interface define the IP address /32 in Export rules (see ospf section) STATIC ROUTING To configure a static route

NextHop: As you would expect in any static route, you have to define a directly connected next hop for reachability to external network **Important** Palo Alto Device does not support recursive lookups, so all next hop IP must be directly connected.

If a non-directly connected IP has to be define, then a second static route has to be define to advise how to reach that non-directly connected Next hop IP. Next-VR: to forward traffic to a destination that is not in the local VR but another VR Used often when dealing with Multiple VSYS scenario, where each VSYS has its own VR. Discard: often use when needing to blackhole traffic, this is the equivalent next hop Null in other Vendors. **Important*, in 5.x, 6.0 and 6.1, Discard route were not to be redistributed to OSPF because they don t have a valid next hop. This behavior has been modified, where now you can redistribute Next-Hop Discard in OSPF from PAN-OS 6.1.4 (backported to 6.0.10 and above as well). Discard routes can be redistributed in BGP in any version without issues. None: often use for NAT and/or Global Protect Tunnel interface NAT: When doing NAT to a subnet/ip address that is not directly connected ( meaning not part of any Firewall Interface subnets), it allows to install the route so that IP Lookup during NAT evaluation can occur successfully for the NAT translation to be completed. Global Protect: When doing Global Protect, it is often required to have a static route pointing to the GP Subnet allocated with Interface tunnel.x and use the next hop none. This is allows the GP subnet to be installed in the Routing table so FW know where to forward the GP user packet back to, and it knows from the interface configure that it is reachable via tunnel.x interface (and doesn t need next ip for that hence the none). OSPF OSPF is a Link State Routing protocol based on AREA s to segment a customer network. The segmentation is based on: 1. Backbone network (aka Area 0) 2. Area Border router (ABR- Firewall/Routers having at least an interface in backbone and another in a different area),

3. Internal ( Firewall/routers only belong to one area ) OSPF uses IP multicast to send link-state updates. This ensures less processing on routers that are not listening to OSPF packets. Also, updates are only sent in case routing changes occur instead of periodically. This ensures a better use of bandwidth. OSPF has better convergence than RIP. This is because routing changes are propagated instantaneously and not periodically. OSPF allows for a logical definition of networks where routers can be divided into areas. This limits the explosion of link state updates over the whole network. This also provides a mechanism for aggregating routes and cutting down on the unnecessary propagation of subnet information. OSPF allows for routing authentication by using different methods of password authentication. OSPF allows for the transfer and tagging of external routes injected into an Autonomous System. In 7.0, with the addition of ECMP feature, OSPF will allow for better load balancing. The minimum basic item to configure to get OSPF up and running: -Define Router ID -Define Area ID -Define Area Type (normal by default) -Define interface to be part of the OSPF -Define Link Type (Broadcast by Default)

After setting up the neighborship

As you can see in the above output, all /32 address are seen as Host (via the flags H), The 1.1.1.1/32 is a static route (Discard) redistributed into OSPF (we are on 6.0.10), yet we are not seeing our loopback ip 192.168.0.2/32 in the table. This means any loopback required in OSPF cannot be redistributed, The workaround is to Export the /32 IP so it can be advertise in OSPF.

From the above note, since Loopbacks cannot be redistributed into OSPF, It leads to the conclusion that you cannot have OSPF running between different VSYS/VR instances, however within a single VSYS/VR, OSPF can be established. The workaround is to use a Physical cable across 2 physical interfaces assigned in their respective VR in the device or use a 3 rd party switching/routing equipment in between the 2 VR s.

BGP BGP is a Distance vector Protocol based on AS-PATH, It is compose of Attributes, some are transitive (mean are kept across AS-PATH), some are non transitive (mean only relevant in the local AS) Any BGP session between 2 different AS are called External BGP (EBGP) and any BGP session within one AS are called Internal BGP (IBGP) The rule is all IBGP (in the same AS) neighbors must be full mesh to avoid routing loop, The only way to overcome this rule is to create Route Reflector or Confederations. Here we will cover only Route Reflector as Confederation are not as commonly used in the market. In general, BGP is the most flexible routing protocol for inter-vr/vsys routing, it gives you much more granularity in controlling routes than any other protocols. However Palo Alto Networks preferred method for Inter-VR/VSYS communication remains via Static Routes. To configure BGP with the default settings, Graceful Restart is enable by default

The default for the PEER Group configuration is below When doing EBGP,

In my opinion the key thing that should usually be change straight away by the Admin on the BGP configuration is to change the : Import Next Hop: from Original to Use Peer Export Next Hop: from Resolve to Use Self This is always to be double check depending on the customer topology and routing as in some case, the changes of Next Hop may not be required, but in my opinion it should be set as Use PEER and Use Self by default The reason for this By default when a route is advertised to an Ebgp outside of the AS, the router will make sure that the next-hop attribute reflects its original IP address (basically preserving the announcer IP address). As a result when a route is advertised to an Ibgp speaker and sourced into the BGP as-group, all Ibgp routers will have for next-hop the ip address of the Ebgp neighbor. But what happen if the Ebgp speaker is not reachable? What happen if the IBGP peer does not know how to get to the EBGP speaker announcing the route (not IP reachability)?. To prevent this, we can make sure that a route advertised to an Ibgp router; echoes the IP address of the router sourcing that route into the AS to the Ibgp neighbors; and not the IP address of the Ebgp speaker which originally advertised this route.

As you can see, a lot of the information are left by default and those timers should work in most case for your customer. If a faster failure detection is required for BGP, you can set the BGP to be more aggressive by lowering Keep Alive and Hold time respectively to 10 and 30 (instead of default 30 and 90 sec)

Note the row highlight in RED MULTI-HOP By default, the number of Hop is 0, which is usually a TTL of 1, TTL of 1 mean it will establish BGP only with a directly connected peer (it will not do more than 1 hop). In other to have more than 1 hop you can increase this value to 2 (recommended for EBGP) or 255. 2 or more Hop (or TTL if you prefer) can be useful when doing BGP Neighborship (whether EBGP or IBGP) between the PA loopback and the peering router Loopback. The above information should be enough to setup most basic/regular BGP For any advance BGP, you will need to start playing with the features specs. Import Rule So let s create an import rules from Static to BGP (redistribute 2 static route into BGP) in VR- VSYS1 and we will then advertise them to VR-VSYS2

I am importing the 2 subnet 10.194.56.0/24 and 10.193.58.0/24 and I want to tag them with the Community Attribute of 65002:1111 and replacing the default Local Pref from 100 to value of 200

So now let s check our BGP Neighborship (here on VR-VSYS2, but same for VR-VSYS1)

Cool, see that even without manipulating too many options, BGP Neighborship comes up, so the default values are good enough to get us up and running. (the import policy is just to have routes in BGP as this is a lab environment, and see if they are advertise to the BGP peer, they are not mandatory to setup for simple BGP Neighborship) Let us check the BGP routes learned No Routes??!! Anybody figured out why yet? If you go back to the 1 st screenshot in the BGP section The Install Route option is not ticked (it is unticked by default) If I enable the tick next to Install Routes and commit

Voila!!! ;-) You see that the receiving end (VR-VSYS2 BGP is receiving BGP routes) from VR_VSYS1 IP 10.193.57.2 Let us check the BGP route details to ensure how Attributes (Local Pref, Communities) are also correctly advertise So In VR-VSYS1, which is the one who own the routes and will be advertising to VR-VSYS2, I have highlight the key info relative to 10.193.56.0/24 route specifically but that applies to all the other routes

In VR-VSYS2

We verify for the same route 10.193.56.0/24 We now have a next hop à 10.193.57.2 (on VR-VSYS1, it shows local as expected ) We also added an AS-PATH: 65002 (which is VR-VSYS1 AS number to reach that destination) We also received now correctly the Community Attribute tag for this 10.193.56.0/24 define in VSYS1 (65002:1111) so we now will maintain that Attribute and advertise to our BGP Peer However we do not have the same advertised Local Preference: à here we have 100, in our output from VR-VSYS1, we configure it to be 200. Anybody know why? As discuss in intro, in BGP you have transitive and non-transitive attribute, which is essentially meaning which BGP attribute are share with other BGP Peer AS (so which attribute can transit to another AS and which attribute Cannot).

In intro, we specify that Local Pref is a powerful attribute but is non-transitive and has only Local significance, meaning it only is relevant within its own AS (here only within VR-VSYS1 AS) So all in all, it looks like we are doing ok so far. Redistribute Rule Redistribute Rule allows to redistribute route from one protocols into another protocol While we saw some test previously on Communities, Let s see the maximum community can be added when redistributing route into BGP

From the above Screenshot, we can see that the Add button is greyed out after 10 communities added That tell us we cannot have more than 10 Communities redistributed to a route. Again Palo Alto is a firewall and not a router and 10 should be more enough for 99.9% of cases. On VR-VSYS1, we do see correctly the 10 Communities

Let s verify on VR-VSYS2, and Yes we do receive all 10 Communities

Now this shows that when redistributing from one protocol into BGP, we can add up to 10 Communities. But when Advertising from one BGP peer on the PA FW to another BGP Peer, we can only append 1 community

I have tried different ways to append additional Communities but we just cannot. So the main recommendation should all communities tagging required should be done by an upstream/downstream router so can forward it to the next hop. So What if we try to have 2 export rules to see if we can append 2 communities list, Lets change the setup so that the same prefix (let s try with 10.193.56.0/24) be use in 2 different export profile Creating 2 export rule (in below order) in VR- VSYS1 to same neighbour for the same prefix (10.193.56.0/24) to be advertise to VR- VSYS2 10.193.57.120, each with different communities (purposefully change to different communities to validate the redistribution) export rule 1 : 10.193.56.0/24 with community 65010:9876 export rule 2 : 10.193.56.0/24 with community 65010:5432

This is the Export rule for community 65002:9876

This is the Export rule for community 65002:5432 In RED, the advertise communities from the Export Rules,

in GREEN the advertised communities from the Redistribution rules, since we are appending we should see both the Export rules communities and the Communities from the Redistribution Rules. Notice we don t see the 2nd export rule communities 65002:5432 If I swap the rule order to make Export rule community 65002:9876 below 65002:5432

Same behavior, we now see the top order export rule with community 65002:5432 but not the bottom export rule 65002:9876 So the Export rule is matching the first Export rule but not processing the bottom one, Let s see if this is because each are Export rule based on same attributes (Communities) by Prepending AS-PATH which is a different Attribute. From the current output, we only have 1 AS-PATH Let s create our AS-PATH prepending policy

We have place it at the bottom in the Rule order, below the 2 Community export rules We should now expect the AS 65002 to be prepending 5 times

In the output, it looks the export rule did not work as we still only 65002 once. But we still see the Community 65002:5432 in the community list (who is at the top of the order in Export rules) If we place the AS-Path Prepending export rule at the order (so the 2 Community export rules are below)

Now we see the AS-Path Prepending (first rule in the order list) work, but we lost all the other Export List communities (there is no 65002:9876 nor 65002:5432 community anymore in community ). If we create a 4 th filter, this time with a different prefix (10.193.58.0/24) with AS-PATH PREPENDING 5

Here our prefix 10.193.58.0/24 Export rules is at the bottom And when we verify

Now using a different subnet in the match criteria, we are having the AS-PATH prepending happening. That confirms the Export rules do a first match action and stop processing any following export rules. IBGP So now that we have tested BGP successfully, let s try and setup ibgp between the 2 VSYS. We currently have EBGP between VR-VSYS1 and VR-VSYS2, We will now create a 3 rd VSYS with its own VR called VR-VSYS3 and have IBGP between the 2. We create another loopback for VR-VSYS3 with IP 192.168.0.4/32 The config on VR-VSYS2 is still the same (loopback IP is 192.168.0.3/32) and AS is 65120 So VR-VSYS2 config looks like

Created a new Peer Group with BGP type as IBGP As you can see, we define Peer AS 65120, which is the same as our Local AS, This is normal as IBGP is BGP peering between 2 neighbour within the same AS. So the expected behavior is all the EBGP routes learn from VR-VSYS1 by VR-VSYS2, VR- VSYS2 will then advertise all those routes to VR-VSYS3 (as they are IBGP neighbor, they both advertise to each other all their local BGP routes

On VR-VSYS3, same configuration of BGP, but with the IP s other way around

Let s verify the Neighborship On Vr-VSYS2, we see it established On Vr-VSYS3, we see it established as well and receiving the BGP routes

VR- VSYS3 is now receiving the BGP routes from VR- VSYS2 (VR- VSYS2 learned those route from VR- VSYS1). Also we notice that VR- VSYS2 is preserving the AS- PATH prepending done by VR- VSYS1 and is advertising the same AS=PATH length to VR- VSYS3. Route Reflector Now that we are adding VSYS and VR to achieve the RR scenario, our logical topology now changed. As a reminder, the current topology now looks like this

The Route Reflector is a BGP routing components that offer an alternative to having all IBGP peer to be in full Mesh (as full mesh doesn t scale well for large networks ) A fundamental point in Route Reflector (since it is not in full mesh anymore) is the Loop Avoidance and 2 new attributes gets generated. In RR the 2 attributes for this: Originator-ID Cluster ID The Router ID field is for Originator-ID and Reflector Cluster ID is for the Cluster ID Route Reflector Configurations

Here we use the Next-hop Use Self options for the purpose of the lab, but in IBGP you probably want to keep the Original next-hop.

As you can see, above screenshot show the config of the RR itself (192.168.0.3) and it has the Reflector Client option set to Client. It does not mean the RR is a client itself, but that the ibgp peering to him (here 192.168.0.5 ) will be his client

The router ID is referring the RR attribute Originator ID The Reflector Cluster ID is referring the RR Attribute Cluster ID High Availability Active-Passive When doing Active-Passive, it is important to remember that the passive member is in standby mode, not responding to Arp and does not own any IP as the IP address are still owned by the primary. If we take the routing table output of our lab device Active in the HA

We have our 3 Virtual router, with the routes installed in each Virtual Router. If we look at the flags column, we see the Host (H),Directly Connected (C ), Static routes (S), OSPF (Oi), BGP (B) route entries. So that confirms all routes are correctly present in the Routing RIB tables. If we now go to our Passive device, to verify the state of the routing entries

If you look on passive, we do not see the same route entries, No OSPF, no BGP, more interestingly not Directly connected, If you look only Host and Static entries, No dynamic routing (as when the failover occurs, the Firewall will re-establish the Neighborship, learn the routes from its neighbor and recalculate the SPT), The directly connected not being present, actually make sense since the Interface and the IP s are only active on the other Firewall (who is currently the active member and the one who own the IP addresses).