SPAN and RSPAN (1.50)

Whilst trying to figure out how to block HSRP traffic over a QinQ link I had to set up a SPAN port, so thought it would be a good idea to cover it briefly here, as this is covered under exam topic 1.50.

What is a SPAN port?

A SPAN port or Switched Port ANalyzer offers a way of duplicating traffic from one port to another. This can be local to a switch (SPAN) or sent to a different switch (Remote SPAN or RSPAN). In a nutshell its a "free" network monitor

At the destination end of the port would be a laptop (or workstation) running a program like Wireshark.

Basic SPAN configuration

So imaging this scenario, interface FastEthernet1/0/2 is a trunk port to another switch, and we want to send a copy of all the traffic to a laptop connected to FastEthernet1/0/15 which is running Wireshark.

The commands a very simple.

We start by setting a monitor session, and each of these is numbered, and a source interface:

monitor session 1 source interface Fa1/0/2

We then set a destination interface for the same session number:

monitor session 1 destination interface Fa1/0/15

A SPAN source port can be a routed port, switch port, access port, trunk port, or an EtherChannel port.

Basic RSPAN Configuration

A Remote SPAN is similar to SPAN, but it the destination is on a different switch. RSPAN does require a VLAN to be used to carry the traffic (and therefore must be allowed over any trunk links).

Using this topology:

We have a laptop running Wireshark connected to Fa 1/0/15 on the 3750 switch.

the 3750 and 3550-1 are configured in an HSRP pair using the subnet 10.20.1.0/24 and the 3560 and 3550-2 are also in an HSRP pair using the subnet 10.10.1.0/24.

3560:

interface Vlan10
ip address 10.10.1.2 255.255.255.0
standby 10 ip 10.10.1.1
standby 10 priority 150
standby 10 preempt delay minimum 60

3550-2:

interface Vlan10
ip address 10.10.1.3 255.255.255.0
standby 10 ip 10.10.1.1
standby 10 priority 90

If we set out 3560 and 3750 up as follows:

3560#conf t
3560(config)# vlan 100
3560(config-vlan)# remote-span
3560(config-vlan)# exit
3560(config)# monitor session 2 source vlan 10
3560(config)# monitor session 2 destination remote vlan 100

3750# conf t
3750(config)# vlan 100
3750(config-vlan)# remote-span
3750(config-vlan)# exit
3750(config)# monitor session 2 source remote vlan 100
3750(config)# monitor session 2 destination interface fastethernet 1/0/15

We can then ping from 3550-2 to the 10.10.1.2 interface on the 3560.

Wireshark shows us the following:

And we can see that everything is preserved nicely, including the MAC addresses.

That's the beauty of RSPAN, its invisible to the end user, it perfectly preserves the data, making it ideal for forensic analysis. network monitoring and network performance monitoring, and its quick.

Advanced SPAN and RSPAN concepts

We can be very selective with what we use our SPAN and RSPAN for. We can monitor traffic coming in (rx), or going out (tx) or both ways (which is the default). We can also filter out vlans that we don't want to have monitored.

In the configuration below we are monitoring incoming and outgoing traffic on vlan 10. For vlan 11 we are only monitoring incoming traffic, and only outgoing traffic for vlan 12. All traffic on Fa1/0/2 is being monitored and as this is a trunk port we have then filtered out vlans 1 through to 5 and also vlan 23. Lastly we are sending this to a remote VLAN as we did above.

SW1#conf t
SW1(config)# vlan 100
SW1(config-vlan)# remote span
SW1(config-vlan)# exit
SW1(config)# monitor session 1 source vlan 10
SW1(config)# monitor session 1 source vlan 11 rx
SW1(config)# monitor session 1 source vlan 12 tx
SW1(config)# monitor session 1 source interface fa1/0/2>
SW1(config)# monitor session 1 filter vlan 1-5, 23
SW1(config)# monitor session 1 destination remote vlan 100

Troubleshooting SPAN and RSPAN

To troubleshoot use the commands "show monitor detail" or "show session <session number> detail" if you have more than one monitor session and you want to be selective.

Fun with QinQ tunnels - Part 3 - Why HSRP and QinQ don't play well

I have covered QinQ tunnels a couple of times now, but for an exceedingly brief recap a QinQ tunnel extends the layer 2 network across a WAN. You can read the first two posts on QinQ tunnels here and here. They are well worth a read to understand what I will be discussing in this post.

For all the flexibility that having a QinQ tunnel across two sites gives us, it can introduce problems. Imagine the scenario. We have Site A and Site B. Site B is a backup (or non-production) site, and services (if required) can be run out of Site B if Site A fails, with minimal changes required.

Some background on both the sites.

• Both sites are configured with a VPN tunnel between them, and a QinQ tunnel between a switch in both sites.
• Both sites have very similar hardware and software configurations. The only differences would be the IP subnets used.
• Both sites are set up to maximise HA where ever possible.

The hardware and network

At the top of the stack is a pair of Cisco ASAs with high availability set up (in a failover pair and are tracking the switches on the inside interface).
Connected to the ASAs are a pair of Catalysts, these are connected together using a channel group (ether channel - which you can read about here).
The VLANs that are used in the network are set up using HSRP for HA purposes.

The topology for the sites will look much like this:

What is HSRP?

HSRP stands for Hot Standby Router Protocol, and is Cisco proprietary. Other vendors will probably use VRRP, which is supported on Cisco hardware, and the commands for which are very similar to how we would set up HSRP. With HSRP we assign interfaces (or more frequently) VLAN interfaces to an HSRP group, one will be the active, and one will be the standby, and these virtual IP addresses are used as a default gateway.  

How to configure HSRP

A basic HSRP configuration for a vlan would look like this:

SW1#conf t
SW1(config)#int vlan 10
SW1(config-if)#ip address 1.1.10.2 255.255.255.0
SW1(config-if)#standby 10 ip 1.1.10.1
SW1(config-if)#standby 10 priority 90

SW2#conf t
SW2(config)#int vlan 10
SW2(config-if)#ip address 1.1.10.3 255.255.255.0
SW2(config-if)#standby 10 ip 1.1.10.1
SW2(config-if)#standby 10 priority 150
SW2(config-if)#standby 10 preempt delay minimum 60

We have set up a vlan (10) on both switches. SW1 will have a local IP address of 1.1.10.2 (/24) and SW2 will have a local IP address of 1.1.10.3. The virtual IP address of the VLAN will be 1.1.10.1, which is what we would set the default gateway of our client machines in VLAN 10 to. SW2 would become the active switch for the VLAN as its configured priority (which if not configured would default to 100) is higher than that of SW2. Lastly we set the preempt delay so that the SW1 should wait for one minute before trying to reclaim its active status for the VLAN (useful if its flapping for any reason, you don't want it to try becoming the active member too quickly, only to have it flap again).

HSRP and QinQ tunnels

So there we have a quick over view of HSRP. Now remember when I said that both sites have been configured with very similar hardware and software set ups? Well if Site A has the HSRP set up as above and Site B has the following setup:

SW1#conf t
SW1(config)#int vlan 10
SW1(config-if)#ip address 1.2.10.2 255.255.255.0
SW1(config-if)#standby 10 ip 10.2.10.1
SW1(config-if)#standby 10 priority 90

SW2#conf t
SW2(config)#int vlan 10
SW2(config-if)#ip address 1.2.10.3 255.255.255.0
SW2(config-if)#standby 10 ip 10.2.10.1
SW2(config-if)#standby 10 priority 150
SW2(config-if)#standby 10 preempt delay minimum 60

What will happen when we introduce our QinQ link?

Well, the IP addresses are clearly different, there are in totally different subnets, so everything should be fine right?

Well no. Our QinQ link will need a separate subnet (common on both sides to communicate), but will also be used to trunk between our two sides (so we can make failing over to the other site, or file transfers, etc, quicker). so although the separate QinQ interfaces in Site A and Site B can communicate over a separate VLAN these interfaces will also need to be opened up to allow other subnets to use the trunk link between the sites.

The general topology would look a bit like this:

So we have the VPN between ASA2 and ASA4 (actually with the ASAs one would always be the primary so the VPN would always to the the primary external IP address, but you get the general idea), and the QinQ tunnel between SW2 and SW3 (simplified for this example).

Once you start allowing all the VLANs across the link between SW2 and SW3, the HSRP messages will also flow across the switches. In the switches you will see these kind of messages:

Event Description : .Oct 20 10:22:48:
%HSRP-4-DIFFVIP1: Vlan10 Grp 10 active routers virtual IP address 1.1.10.1
is different to the locally configured address 1.2.10.1

So even if the subnets are completely different HSRP will not mind at all.

HSRP does not care about IP addresses!

The fact is that HSRP really doesn't care what IP addresses you use.

Suppose that the first switch is set up as

SW1#conf t
SW1(config)#int vlan 10
SW1(config-if)#ip address 1.2.10.2 255.255.255.0
SW1(config-if)#standby 10 ip 10.2.10.1
SW1(config-if)#standby 10 priority 90

The second switch can be set up as follows:

SW2#conf t
SW2(config)#int vlan 10
SW2(config-if)#ip address 1.2.10.3 255.255.255.0
SW2(config-if)#standby 10 ip
SW2(config-if)#standby 10 priority 150
SW2(config-if)#standby 10 preempt delay minimum 60

If we do this (and its completely valid according to Cisco), then just so long as one switch has the ip address set, the other switch will learn the virtual interface to use.

It's all about the HSRP group number. Which explains why in the scenario above if the group numbers are the same on both sides, you will have issues.

When HSRP goes bad

Lets actually see this in action

We have a working HSRP set up between SW1 and SW2:

And also between SW3 and SW4

Above you can see that on SW4 I have omitted the IP address, but we can still see from the messages that HSRP works, and we can confirm it using "sh standby vlan 10":

Now we have two "sites" with working HSRP.

I will now configure the "QinQ" link (although as its not properly supported in GNS3 it is just a trunk link between the two sides).

I started by creating vlan 200, and then moving on to the trunk interfaces between SW2 and SW3. Firstly I shut down the interfaces before applying the configs and then doing a no shut on the interfaces.

The configs look like this:

And we can see that the two sides are talking:

Now if we allow all VLANs over the trunk...

Ooooh, "BADVIP" not good. Clearly we need to stop the HSRP messages from one side influencing the other side.

Lets shut down the link between SW2 and SW3, allow HSRP to recover, and figure out where to go from here.

So how do we fix this?

Blocking HSRP cross subnet traffic

We can try and stop the HSRP messages from going over our trunk, or at least from breaking the HSRP setup(s) in one of three ways:

• Implement HSRP authentication
• Change the HSRP group numbers on one side
• Use an ACL

Implementing HSRP authentication

One option open to us is to use HSRP authentication.

On both switches we set up the following

SW1#conf t
SW1(config)#key chain hsrp-key
SW1(config-keychain)#key 1
SW1(config-keychain-key)#key-string 0 wtfbbq
SW1(config-keychain-key)#exit
SW1(config-keychain)#exit
SW1(config)#int vlan 10
SW1(config-if)#standby 10 authentication md5 key-chain hsrp-key

Now we can re-enable our SW2 to SW3 link and see what happens

Wireshark is seeing a lot of HSRP traffic:

And our switches are showing a lot of bad authentication attempts:

But at least we don't have our VLANs getting all screwed up. 

This still isn't the optimal method, as the traffic is still being generated, and a lot of noise is being logged on our switches. So can we stop the noise on our switches, because if this is getting logged then every few seconds a log entry will be generated.

Re-numbering the standby groups

An obvious choice here is to re-number the HSRP group.

I have now changed SW3 and SW4 to use standby group 20 (instead of 10). This is just a case of entering the "no standby 10" and then setting up the HSRP group again with a new group number. This needs to be done on both switches.

Wireshark is still showing a lot of HSRP traffic, but SW1 and SW2 are not logging all the failed authentication attempts. So can we stop the traffic all together? And, showing my age, I really feel like using a picture from Airplane! after that question.

Stop HSRP traffic altogether

Using an Access Control List to block HSRP traffic

Wouldn't it be nice if we could block the multicast traffic completely across the link between the sites?

Well, what do we know about HSRP? Well, it comes in two versions, version 1 and version 2.
Version 1 uses the multicast address 224.0.0.2 and version 2 uses 224.0.0.102. Both versions have a source and destination port of 1985.

We also know that the MAC address used is 0000.0c07.ac0a.

If we try and block using the MAC address and a VACL then we could stop legitimate HSRP traffic between the switches on one site. So the solution needs to be IP based.

So we could use an ACL to block outgoing or incoming traffic to the multicast addresses used by HSRP, or block all traffic on UDP port 1985. An outgoing ACL would be the best option as it will also reduce traffic across the link.

We can see from a wireshark capture that we are using version 1 of HSRP as the multicast address is 224.0.0.2:

The most obvious acl would be "access-list 101 deny udp any any eq 1985" and apply this to the trunk link between SW2 and SW3 outbound. Try as I might this did not work in GNS3. I had heard reports that HSRP didn't work in IOU, but it does (and you can read about how to get HSRP working on both GNS3 and IOU) so decided to give it a shot.

So anyway, the obvious acl didnt work, if I tried a direction of out I got the error:

"error ip acl configuration on 'out' direction is not supported"

And having it in an in direction didn't work as I could still see the HSRP traffic across the wire, and the HSRP was getting messages from one side interfering with the other side.

This is part if the trouble with emulated systems, it's difficult to get all the functionality you need, if you need functionality that's even slightly out of the ordinary, you either have a fight to get it to work, or it just plain won't work.

So after trying numerous ACLs and finding nothing worked I found the command "switchport block multicast" which again isn't supported by my GNS3 topology. This command has been around since at least 12.2, but apparently it's not available on GNS3, I then moved to IOU in the hope that the "switchport block multicast" command would be support, but it isn't there either. Also it looks like this is just for "unknown multicast" traffic, so probably wouldn't fit the bill anyway.

Once I was able to lab this up on real equipment (a 3750 and a 3550 for one side and a 3560 and 3550 for the other side) I was able to get some good progress. The downside of emulated equipment is that its harder (slightly) to get Wireshark sniffing going on - but that's where SPAN and RSPAN come into play. You can have a read of SPAN and RSPAN configuration.

The caveat I found was that this needs to be implemented on both sides of the link between sites.

access-list 101 deny tcp any eq 1985 host 224.0.0.2
access-list 101 deny udp any eq 1985 host 224.0.0.2
access-list 101 permit ip any any

If it's only implemented on one side only then although HSRP will recover quickly for that side, the other will still be accepting HSRP messages from the other side, and you'll see messages along the lines of:

*Mar  1 02:55:58.883: %HSRP-4-DIFFVIP1: Vlan10 Grp 10 active routers virtual IP address 10.20.1.1 is different to the locally configured address 10.10.1.1

Hence the ACL must exist on both sides, and be assigned to the link interface on the "in" direction. Sadly the out direction is not an option, but that would have been perfect for what we need. The messages still flow across the link, but are dropped according to the ACL, allowing for both sides to have identical HSRP group numbers. It's not a perfect solution as I would like to stop the traffic at source rather than destination, but it would appear to be the only option available.

Now there is the chance that the solution would actually work with emulated equipment, maybe it was down to software versions, but maybe something's just work better on real hardware (not surprising really!).

Which is the best option of the three methods?

Actually all of them, if you want a belt-and-braces approach. Just having the authentication is enough to stop the HSRP messages confusing the different sites, but will log a log of noise on the switches. Changing the standby group numbers will also help, it will certainly cut down on the noise generated by failed authentication methods, but still a lot of traffic would be crossing the link. The ACL allows us to have the same HSRP groups on each side, and (if we wanted to) the same authentication method. Due to not being able to specify the ACL in an out direction the traffic will still go across the link, but I guess that that's something I'll havr to live with.

How to implement HSRP on Cisco IOU and GNS3

Lots of people have said that HSRP does not work in either GNS3 or IOU. It does, but both softwares are very particular about what they require for it to work.

HSRP on Cisco IOU

For HSRP on Cisco IOU I have used the IOS image i86bi_linux_l2-ipbasek9-ms.jan24-2013-team_track.

To upload the image fire up IOU web and click on Manage.

Click on Manage IOSes

In the Upload IOS bit at the top of the page enter the filename - this must be the same as the file you are going to upload or else it won't work. The alias can be anything you want it to be. Then browse to the image from the Choose File button.

Your page should look like this:

When you are ready click on Upload.

Once its uploaded your Manage IOSes bit of the page should look like this:

You can see our added IOS at the bottom.

We can now start with a basic lab, using a netmap of 1:0/1 2:0/1.

SW1 is configured as follows:

SW2 is set up similarly (excuse the typing mistakes)

After a few seconds we can start to see HSRP messages coming through:

And using the command "sh standby vlan 10" we can confirm that the HSRP is working as it should:

HSRP on GNS3

For HSRP on GNS3 I am using the image c3660-is-mz.124.25b.

Fire up GNS3 and go into Edit > IOS images and hypervisors, add your image for the c3660 (if you don't already have one, but your mileage may vary with other images), and set it as the default.

Next go into Edit > Symbol Manager, find an image you like, personally I like the route_switch_processor, highlight it and press the arrow button. Change the type to Router c3600 and give it a good name, then click Apply, then click OK.

Now we can drag two of our new HSRP switches onto the canvas:

Now configure both the new switches and add an NM-16ESW to each:

Once you have done that you can cable them together and start them up, you must use the connections on the newly added module though - I have used 1/10 on each.

The configs are the same as above for IOU and are a very simple implementation:

Once you give the switches a few seconds once everything is plumbed in you should see the HSRP messages start to flow:

And there we have two ways to get HSRP working in a home environment without needing to purchase separate hardware.

The BIG lab - Part 4 - RIP

Welcome back to part 4 of the big lab. So far we have covered some trunking and etherchannel technologies, PPP and OSPF.

Today we are going to extend our topology by adding RIP. The exam guide has RIPv2 under part 2.30 and does not have as much listed for is as say OSPF or EIGRP. So what can we do with RIP? Well apart from the obvious basic configuration we'll use it as the intermediate routing protocol for redistribution - with the end goal being that our OSPF network will be able to talk to our EIGRP network.

Out topology has evolved since the first part of this series, and will, in all probability, change again. But at the moment it looks like this:

About RIP

RIP is a Distance Vector protocol, it has a maximum of 15 hops so does not scale well in larger environments.

RIP comes in two flavours - version 1 (RIPv1) and version 2 (RIPv2). 

RIPv1 versus RIPv2

RIPv1 is classful so does not include the subnet mask in its routing updates, therefore it does not support VLSM. Version 2 is classful to overcome this and so does include the subnet mask in the updates, and also supports VLSM. RIPv2 also supports authentication and route tagging. Cisco IOS will default to version 1.

Interoperability between RIPv1 and RIPv2

RIPv1 will only send version 1 packets
RIPv1 can receive V1 and V2 packets
RIPv2 will only send and receive V2 packets

Configuring RIPv2

At the moment we will concentrate on R4, R5, R6 and R7.

I will be using the 10.20.X.X/24 networks in this post:

R4 
s0/1 - 10.20.1.1/24

R5
s0/0 - 10.20.1.2/24
so/1 - 10.20.2.1/24

R6 
s0/0 - 10.20.2.2/24
s0/1 - 10.20.3.1/24

R7
s0/1 - 10.20.3.2/24
lo0 - 10.60.1.1

Lets start by setting up R4:

We start with creating the routing process using the command "router rip". We then specify the version to be v2 with the sub command "version 2"
RIP will by default summarize networks on classfull boundaries so we use the command "no auto-summary", and after that we can specify which of our local networks we want to participate in the RIP process using the command "network 10.20.1.0".

R5 is very similar:

Not at this stage would we expect R4 to see the 10.20.2.0/24 subnet in its routing table? R5 has an interface that is configured for that network, and we have advertised it into the RIP process, so it should be there, right? Well no. R5 does not even have the 10.20.2.0 network in its routed table. It's not until we fire up R6 that we can see the network advertised:

Its easier to see on R5 as we can see the interface coming up:

And on R4 we can now see:

R6 and R7 were started at around the same time so we get the 10.20.3.0 network advertised into our RIP process much quicker:

Although its not shown the command "no auto-summary" has been set, the routing table looks good:

R7 will have a loopback address and will be used to create the tunnel over to R9 in a different post.

Now if we head back over to R4 we can see that out routing table now looks like this:

And we have full reachability

Advanced RIPv2 configuration

So far our RIPv2 implementation is very straight forward, in fact this is the beauty of RIP in that it is very simple, doesn't scale well for larger environments, but its simple.

So what can we do with it now that we have it in place? We can:

• Configure the timers
• Control participation in RIP through passive-interfaces
• Change the default behaviour from multicast to unicast
• Set authentication for RIPv2

Configuring RIP timers

RIP has four timers (default times are in brackets):

Update timer (30 seconds) - how often a router will send a routing table update

Invalid timer (180 seconds) - how long a route remains in the routing table before being declared invalid. A route declared invalid will be placed in the hold-down state and advertised with a metric of 16 (unreachable).

Hold-down timer (180 seconds) - RIP will not accept updates for routes in the hold-down state unti the hold-down timer expires.

Flush timer (240 seconds) - how long a route can remain in the routing table before being flushed. Runs at the same time as the Invalid timer - so a route marked as Invalid will be flushed after 60 seconds.

RIP timers must be identical on all routers in the RIP network, otherwise instability will occur.

We can adjust the timers using the command

Router(config)#router rip

Router(config-router)#timers basic 20 120 120 160

This would set the Update timer to 20 seconds, the Invalid and Hold-down timers to 120 seconds and the Flush timer to 160 seconds.

Passive-interfaces

A passive interface will allow a router to listen to updates from the neighbouring RIP routers, but will not send any updates back.

within the RIP process we use the command "passive-interface <interface>" command:

We can configure all interfaces to be passive using the command "passive-interface default" and then remove particular interfaces from this default behaviour by the command "no passive-interface <interface>"

If we want to have even tighter control we can use the passive-interface command and use unicast instead of multicast for our neighbors.

Controlling RIPv2 updates - from Multicast to Unicast

By default RIPv2 uses multicast to send out updates,with the IP address 224.0.0.9. We can change this to unicast by specifying the neighbor command (similar to BGP).

On R5 and R6 we will change from the default of multicast updates to unicast:

If we turn on debugging for rip we can see that on R5 its using multicast for one connection and unicast for another connection:

Above we can see that it it sending an update over serial 0/1 to R6 using the 10.20.2.2 unicast address but also via multicast. To force unicast only on the specific interfaces we need to include the "passive-interface" option, once we have done this we can look at the debug output on R5 and see:

Now we can see that updates to R4 over serial 0/0 are still using the multicast address, but all updates sent and received between R5 and R6 use the unicast address.

RIPv2 Authentication

The final thing we are going to cover today is RIPv2 authentication.

RIPv2 authentication is very similar to other routing protocol authentication setups. We start by creating a keychain:

Next we go into our respective interfaces and specify that rip will be using a keychain and what mode we are to use (clear text or md5):

Its always worth checking back with another router (R4 in this case) to make sure that we don't lose any routes along the way. But we can also debug the rip database to see what's going on.

Here we can see that I have lost the 10.60.1.0 network:

If I remove the authentication between R6 and R7 the loopback is back:

If I try adding it back with text mode, it still loses the route. If I turn on debugging for rip we can see:

The configs look fine:

So once we remove and then re-add the keychain we can head back to R7 and finally see the authentication working correctly:

Chances are there was an space at the end of the keychain key-string, which although not visible in the config, would still be taken as part of the key-string passed between routers - a handy exam tip!

R4 can again see the 10.60.1.0 network:

RIPv2 debugging

So some useful debugging there! I used "debug ip rip" to see that the authentication was wrong and "debug ip rip database" to see what gets added and removed from the routing table, and I also used "debug ip rip events" to see the flow of updates.

That's it for today. In the next post we'll get EIGRP up and running before we move on to redistributing between the three protocols.