I have covered QinQ tunnels a couple of times now, but for an exceedingly brief recap a QinQ tunnel extends the layer 2 network across a WAN. You can read the first two posts on QinQ tunnels here and here. They are well worth a read to understand what I will be discussing in this post.
For all the flexibility that having a QinQ tunnel across two sites gives us, it can introduce problems. Imagine the scenario. We have Site A and Site B. Site B is a backup (or non-production) site, and services (if required) can be run out of Site B if Site A fails, with minimal changes required.
Some background on both the sites.
- Both sites are configured with a VPN tunnel between them, and a QinQ tunnel between a switch in both sites.
- Both sites have very similar hardware and software configurations. The only differences would be the IP subnets used.
- Both sites are set up to maximise HA where ever possible.
At the top of the stack is a pair of Cisco ASAs with high availability set up (in a failover pair and are tracking the switches on the inside interface).
Connected to the ASAs are a pair of Catalysts, these are connected together using a channel group (ether channel - which you can read about here).
The VLANs that are used in the network are set up using HSRP for HA purposes.
The topology for the sites will look much like this:
What is HSRP?
HSRP stands for Hot Standby Router Protocol, and is Cisco proprietary. Other vendors will probably use VRRP, which is supported on Cisco hardware, and the commands for which are very similar to how we would set up HSRP. With HSRP we assign interfaces (or more frequently) VLAN interfaces to an HSRP group, one will be the active, and one will be the standby, and these virtual IP addresses are used as a default gateway.How to configure HSRP
A basic HSRP configuration for a vlan would look like this:SW1#conf t SW1(config)#int vlan 10 SW1(config-if)#ip address 1.1.10.2 255.255.255.0 SW1(config-if)#standby 10 ip 1.1.10.1 SW1(config-if)#standby 10 priority 90 SW2#conf t SW2(config)#int vlan 10 SW2(config-if)#ip address 1.1.10.3 255.255.255.0 SW2(config-if)#standby 10 ip 1.1.10.1 SW2(config-if)#standby 10 priority 150 SW2(config-if)#standby 10 preempt delay minimum 60We have set up a vlan (10) on both switches. SW1 will have a local IP address of 1.1.10.2 (/24) and SW2 will have a local IP address of 1.1.10.3. The virtual IP address of the VLAN will be 1.1.10.1, which is what we would set the default gateway of our client machines in VLAN 10 to. SW2 would become the active switch for the VLAN as its configured priority (which if not configured would default to 100) is higher than that of SW2. Lastly we set the preempt delay so that the SW1 should wait for one minute before trying to reclaim its active status for the VLAN (useful if its flapping for any reason, you don't want it to try becoming the active member too quickly, only to have it flap again).
HSRP and QinQ tunnels
So there we have a quick over view of HSRP. Now remember when I said that both sites have been configured with very similar hardware and software set ups? Well if Site A has the HSRP set up as above and Site B has the following setup:SW1#conf t SW1(config)#int vlan 10 SW1(config-if)#ip address 1.2.10.2 255.255.255.0 SW1(config-if)#standby 10 ip 10.2.10.1 SW1(config-if)#standby 10 priority 90 SW2#conf t SW2(config)#int vlan 10 SW2(config-if)#ip address 1.2.10.3 255.255.255.0 SW2(config-if)#standby 10 ip 10.2.10.1 SW2(config-if)#standby 10 priority 150 SW2(config-if)#standby 10 preempt delay minimum 60What will happen when we introduce our QinQ link?
Well, the IP addresses are clearly different, there are in totally different subnets, so everything should be fine right?
Well no. Our QinQ link will need a separate subnet (common on both sides to communicate), but will also be used to trunk between our two sides (so we can make failing over to the other site, or file transfers, etc, quicker). so although the separate QinQ interfaces in Site A and Site B can communicate over a separate VLAN these interfaces will also need to be opened up to allow other subnets to use the trunk link between the sites.
The general topology would look a bit like this:
So we have the VPN between ASA2 and ASA4 (actually with the ASAs one would always be the primary so the VPN would always to the the primary external IP address, but you get the general idea), and the QinQ tunnel between SW2 and SW3 (simplified for this example).
Once you start allowing all the VLANs across the link between SW2 and SW3, the HSRP messages will also flow across the switches. In the switches you will see these kind of messages:
Event Description : .Oct 20 10:22:48: %HSRP-4-DIFFVIP1: Vlan10 Grp 10 active routers virtual IP address 1.1.10.1 is different to the locally configured address 1.2.10.1So even if the subnets are completely different HSRP will not mind at all.
HSRP does not care about IP addresses!
The fact is that HSRP really doesn't care what IP addresses you use.Suppose that the first switch is set up as
SW1#conf t SW1(config)#int vlan 10 SW1(config-if)#ip address 1.2.10.2 255.255.255.0 SW1(config-if)#standby 10 ip 10.2.10.1 SW1(config-if)#standby 10 priority 90The second switch can be set up as follows:
SW2#conf t SW2(config)#int vlan 10 SW2(config-if)#ip address 1.2.10.3 255.255.255.0 SW2(config-if)#standby 10 ip SW2(config-if)#standby 10 priority 150 SW2(config-if)#standby 10 preempt delay minimum 60If we do this (and its completely valid according to Cisco), then just so long as one switch has the ip address set, the other switch will learn the virtual interface to use.
It's all about the HSRP group number. Which explains why in the scenario above if the group numbers are the same on both sides, you will have issues.
When HSRP goes bad
Lets actually see this in actionWe have a working HSRP set up between SW1 and SW2:
And also between SW3 and SW4
Above you can see that on SW4 I have omitted the IP address, but we can still see from the messages that HSRP works, and we can confirm it using "sh standby vlan 10":
Now we have two "sites" with working HSRP.
I will now configure the "QinQ" link (although as its not properly supported in GNS3 it is just a trunk link between the two sides).
I started by creating vlan 200, and then moving on to the trunk interfaces between SW2 and SW3. Firstly I shut down the interfaces before applying the configs and then doing a no shut on the interfaces.
The configs look like this:
And we can see that the two sides are talking:
Now if we allow all VLANs over the trunk...
Ooooh, "BADVIP" not good. Clearly we need to stop the HSRP messages from one side influencing the other side.
Lets shut down the link between SW2 and SW3, allow HSRP to recover, and figure out where to go from here.
So how do we fix this?
Blocking HSRP cross subnet traffic
We can try and stop the HSRP messages from going over our trunk, or at least from breaking the HSRP setup(s) in one of three ways:- Implement HSRP authentication
- Change the HSRP group numbers on one side
- Use an ACL
Implementing HSRP authentication
One option open to us is to use HSRP authentication.SW1#conf t SW1(config)#key chain hsrp-key SW1(config-keychain)#key 1 SW1(config-keychain-key)#key-string 0 wtfbbq SW1(config-keychain-key)#exit SW1(config-keychain)#exit SW1(config)#int vlan 10 SW1(config-if)#standby 10 authentication md5 key-chain hsrp-keyNow we can re-enable our SW2 to SW3 link and see what happens
Wireshark is seeing a lot of HSRP traffic:
And our switches are showing a lot of bad authentication attempts:
But at least we don't have our VLANs getting all screwed up.
This still isn't the optimal method, as the traffic is still being generated, and a lot of noise is being logged on our switches. So can we stop the noise on our switches, because if this is getting logged then every few seconds a log entry will be generated.
Re-numbering the standby groups
An obvious choice here is to re-number the HSRP group.I have now changed SW3 and SW4 to use standby group 20 (instead of 10). This is just a case of entering the "no standby 10" and then setting up the HSRP group again with a new group number. This needs to be done on both switches.
Wireshark is still showing a lot of HSRP traffic, but SW1 and SW2 are not logging all the failed authentication attempts. So can we stop the traffic all together? And, showing my age, I really feel like using a picture from Airplane! after that question.
 |
| Stop HSRP traffic altogether |
Using an Access Control List to block HSRP traffic
Well, what do we know about HSRP? Well, it comes in two versions, version 1 and version 2.
Version 1 uses the multicast address 224.0.0.2 and version 2 uses 224.0.0.102. Both versions have a source and destination port of 1985.
We also know that the MAC address used is 0000.0c07.ac0a.
If we try and block using the MAC address and a VACL then we could stop legitimate HSRP traffic between the switches on one site. So the solution needs to be IP based.
So we could use an ACL to block outgoing or incoming traffic to the multicast addresses used by HSRP, or block all traffic on UDP port 1985. An outgoing ACL would be the best option as it will also reduce traffic across the link.
We can see from a wireshark capture that we are using version 1 of HSRP as the multicast address is 224.0.0.2:
The most obvious acl would be "access-list 101 deny udp any any eq 1985" and apply this to the trunk link between SW2 and SW3 outbound. Try as I might this did not work in GNS3. I had heard reports that HSRP didn't work in IOU, but it does (and you can read about how to get HSRP working on both GNS3 and IOU) so decided to give it a shot.
So anyway, the obvious acl didnt work, if I tried a direction of out I got the error:
"error ip acl configuration on 'out' direction is not supported"
And having it in an in direction didn't work as I could still see the HSRP traffic across the wire, and the HSRP was getting messages from one side interfering with the other side.
This is part if the trouble with emulated systems, it's difficult to get all the functionality you need, if you need functionality that's even slightly out of the ordinary, you either have a fight to get it to work, or it just plain won't work.
So after trying numerous ACLs and finding nothing worked I found the command "switchport block multicast" which again isn't supported by my GNS3 topology. This command has been around since at least 12.2, but apparently it's not available on GNS3, I then moved to IOU in the hope that the "switchport block multicast" command would be support, but it isn't there either. Also it looks like this is just for "unknown multicast" traffic, so probably wouldn't fit the bill anyway.
Once I was able to lab this up on real equipment (a 3750 and a 3550 for one side and a 3560 and 3550 for the other side) I was able to get some good progress. The downside of emulated equipment is that its harder (slightly) to get Wireshark sniffing going on - but that's where SPAN and RSPAN come into play. You can have a read of SPAN and RSPAN configuration.
The caveat I found was that this needs to be implemented on both sides of the link between sites.
access-list 101 deny tcp any eq 1985 host 224.0.0.2 access-list 101 deny udp any eq 1985 host 224.0.0.2 access-list 101 permit ip any anyIf it's only implemented on one side only then although HSRP will recover quickly for that side, the other will still be accepting HSRP messages from the other side, and you'll see messages along the lines of:
*Mar 1 02:55:58.883: %HSRP-4-DIFFVIP1: Vlan10 Grp 10 active routers virtual IP address 10.20.1.1 is different to the locally configured address 10.10.1.1
Hence the ACL must exist on both sides, and be assigned to the link interface on the "in" direction. Sadly the out direction is not an option, but that would have been perfect for what we need. The messages still flow across the link, but are dropped according to the ACL, allowing for both sides to have identical HSRP group numbers. It's not a perfect solution as I would like to stop the traffic at source rather than destination, but it would appear to be the only option available.
Now there is the chance that the solution would actually work with emulated equipment, maybe it was down to software versions, but maybe something's just work better on real hardware (not surprising really!).