Showing posts with label HSRP. Show all posts
Showing posts with label HSRP. Show all posts
CCIE Security lab: Switching

CCIE Security lab: Switching

It is time to set up the core network.  Nothing new here really, a bit of HSRP and a few niceties (like disabling timeout on the console). The switches need to be given a hostname (obviously), and then we have some basics to do on all of them, and then the HSRP, which will be on SW1 and SW2.

On all switches we create the VLANs, turn off the time out on the console, and set the inter-switch ports to be dot1q trunks:
vlan 4
name Management
vlan 7
name DMZ
vlan 9
name Phones
vlan 20
name Users-1
vlan 21
name Users-2
vlan 55
name Failover
vlan 99
name Data-Phone
exit
line con 0
exec-t 0 0
exit
int ra gi0/1 - 3
swi tru enc dot
swi mo tru
no shu
exit
The next part is to set up the VLAN interfaces and HSRP. I have decided to use .254 as the final octet for the HSRP IPs, as it means I can use the switch numbers (.1 and .2) on the respective switches.

Vlan interfaces and HSRP


int vlan 4
ip address 10.1.4.1 255.255.255.0
standby 4 ip 10.1.4.254
standby 4 pri 90
standby 4 pre del min 60
no sh
int vlan 7
ip address 10.1.7.1 255.255.255.0
standby 7 ip 10.1.7.254
standby 7 pri 110
standby 7 pre del min 60
no sh
int vlan 9
ip address 10.1.9.1 255.255.255.0
standby 9 ip 10.1.9.254
standby 9 pri 90
standby 9 pre del min 60
no sh
int vlan 20
ip address 10.1.20.1 255.255.255.0
standby 20 ip 10.1.20.254
standby 20 pri 110
standby 20 pre del min 60
no sh
int vlan 21
ip address 10.1.21.1 255.255.255.0
standby 21 ip 10.1.21.254
standby 21 pri 90
standby 21 pre del min 60
no sh
int vlan 55
ip address 10.1.55.1 255.255.255.0
standby 55 ip 10.1.55.254
standby 55 pri 110
standby 55 pre del min 60
no sh
int vlan 99
ip address 10.1.99.1 255.255.255.0
standby 99 ip 10.1.99.254
standby 99 pri 90
standby 99 pre del min 60
no sh
int vlan 4
ip address 10.1.4.2 255.255.255.0
standby 4 ip 10.1.4.254
standby 4 pri 110
standby 4 pre del min 60
no sh
int vlan 7
ip address 10.1.7.2 255.255.255.0
standby 7 ip 10.1.7.254
standby 7 pri 90
standby 7 pre del min 60
no sh
int vlan 9
ip address 10.1.9.2 255.255.255.0
standby 9 ip 10.1.9.254
standby 9 pri 110
standby 9 pre del min 60
no sh
int vlan 20
ip address 10.1.20.2 255.255.255.0
standby 20 ip 10.1.20.254
standby 20 pri 90
standby 20 pre del min 60
no sh
int vlan 21
ip address 10.1.21.2 255.255.255.0
standby 21 ip 10.1.21.254
standby 21 pri 110
standby 21 pre del min 60
no sh
int vlan 55
ip address 10.1.55.2 255.255.255.0
standby 55 ip 10.1.55.254
standby 55 pri 90
standby 55 pre del min 60
no sh
int vlan 99
ip address 10.1.99.2 255.255.255.0
standby 99 ip 10.1.99.254
standby 99 pri 110
standby 99 pre del min 60
no sh

Confirmation:

SW1#sh standby bri
                     P indicates configured to preempt.
                     |
Interface   Grp  Pri P State   Active          Standby         Virtual IP
Vl4         4    90  P Standby 10.1.4.2        local           10.1.4.254
Vl7         7    110 P Active  local           10.1.7.2        10.1.7.254
Vl9         9    90  P Standby 10.1.9.2        local           10.1.9.254
Vl20        20   110 P Active  local           10.1.20.2       10.1.20.254
Vl21        21   90  P Standby 10.1.21.2       local           10.1.21.254
Vl55        55   110 P Active  local           10.1.55.2       10.1.55.254
Vl99        99   90  P Standby 10.1.99.2       local           10.1.99.254
SW1#

SW2#sh standby bri
                     P indicates configured to preempt.
                     |
Interface   Grp  Pri P State   Active          Standby         Virtual IP
Vl4         4    110 P Active  local           10.1.4.1        10.1.4.254
Vl7         7    90  P Standby 10.1.7.1        local           10.1.7.254
Vl9         9    110 P Active  local           10.1.9.1        10.1.9.254
Vl20        20   90  P Standby 10.1.20.1       local           10.1.20.254
Vl21        21   110 P Active  local           10.1.21.1       10.1.21.254
Vl55        55   90  P Standby 10.1.55.1       local           10.1.55.254
Vl99        99   110 P Active  local           10.1.99.1       10.1.99.254
SW2#
Looks good now. We have an internal network and can start playing with the really cool toys, like the ISE and WSA, but not before we setup the firewalls for HA and NAT

I am waiting on my (newish) 3750X to be delivered today, its the PoE version, so it'll handle the phone and the Wifi parts of the lab.

IOU - Image is everything


Finding the right image for IOU can be a tricky thing to do, depending on what you are looking for. Things may look correct, but when you dig deeper down into it, you might not get the results you are looking for.

This all came about when discussing HSRP and IOU limitations with Maureen on a post about HSRP and ASICs.

IOU Images for Switches and for Routers

IOU images come in two flavors. We have Layer 2 images for switches, and we have router images. Calling the Layer 2 switch images is a bit of a misnomer, because they can actually perform layer 3 functionality (such as having a layer 3 vlan interface), so don't worry too much about it being called layer 2 (or l2 in the filename). Don't ask me to supply the images though, that's your homework...

IOU Switch images

IOU Switch images are usually named i86bi_linux_l2-<image>-<date> such as:
  • i86bi_linux_l2-ipbasek9-ms.jan24-2013-B
  • i86bi_linux_l2-ipbasek9-ms.jan24-2013-team_track
  • i86bi_linux_l2-ipbasek9-ms.may8-2-13-team_track
  • i86bi_linux_l2-upk9-ms.june20_2012_golden_spike
  • i86bi_linux_l2-adventerprise-ms.nov11-2013-team_track
Most of these are using the ipbasek9 image. But there are others out there.

IOU Router images

Router images follow pretty much the same naming format:
  • i86bi_linux-adventerprisek9-ms.152-2.15.T
  • i86bi_linux-adventerprisek9-ms.152-4.M1
  • i86bi_linux-jk9s-ms.150-1.XJR111.358_120107
  • i86bi_linux-p-ms.june20_2012_golden_spike
Needless to say that a router image is not the same as a switch image.

Not all IOU images are equal

So what's the point of this anyway?

Take the following topology:


We have three switches and one host. The top two switches are set up with HSRP running for vlan 10, with trunk ports between them and the third switch, that just has vlan 10 configured. The VPCS host is on an access port that's in vlan 10. The configuration is as follows:
SW1#sh run int e0/0
Building configuration...

Current configuration : 103 bytes
!
interface Ethernet0/0
 switchport trunk encapsulation dot1q
 switchport mode trunk
 duplex auto
end

SW1#sh run int e0/1
Building configuration...

Current configuration : 103 bytes
!
interface Ethernet0/1
 switchport trunk encapsulation dot1q
 switchport mode trunk
 duplex auto
end

SW1#sh run int vlan 10
Building configuration...

Current configuration : 147 bytes
!
interface Vlan10
 ip address 10.10.1.2 255.255.255.0
 standby 10 ip 10.10.1.1
 standby 10 priority 150
 standby 10 preempt delay minimum 60
end

SW1#

SW2#sh run int e0/0
Building configuration...

Current configuration : 103 bytes
!
interface Ethernet0/0
 switchport trunk encapsulation dot1q
 switchport mode trunk
 duplex auto
end

SW2#sh run int e0/1
Building configuration...

Current configuration : 103 bytes
!
interface Ethernet0/1
 switchport trunk encapsulation dot1q
 switchport mode trunk
 duplex auto
end

SW2#sh run int vlan 10
Building configuration...

Current configuration : 109 bytes
!
interface Vlan10
 ip address 10.10.1.3 255.255.255.0
 standby 10 ip 10.10.1.1
 standby 10 priority 90
end

SW2#

SW3#sh run int e0/0
Building configuration...

Current configuration : 103 bytes
!
interface Ethernet0/0
 switchport trunk encapsulation dot1q
 switchport mode trunk
 duplex auto
end

SW3#sh run int e0/1
Building configuration...

Current configuration : 103 bytes
!
interface Ethernet0/1
 switchport trunk encapsulation dot1q
 switchport mode trunk
 duplex auto
end

SW3#sh run int e0/3
Building configuration...

Current configuration : 93 bytes
!
interface Ethernet0/3
 switchport access vlan 10
 switchport mode access
 duplex auto
end

SW3#
Nothing too hard here, but the image makes all the difference.
SW1#sh ver
Cisco IOS Software, Solaris Software (I86BI_LINUXL2-ADVENTERPRISE-M), Experimental Version 15.1(20131216:211730) [mmen 106]
Copyright (c) 1986-2013 by Cisco Systems, Inc.
Compiled Mon 16-Dec-13 13:50 by mmen

ROM: Bootstrap program is Linux

SW1 uptime is 1 hour, 5 minutes
System returned to ROM by reload at 0
System image file is "unix:/home/gns3/Documents/GNS3/images/i86bi_linux_l2-adventerprise-ms"
With this image we don't get any working HSRP. Both the top two switches think that they are the active switch:
SW1#sh standby vlan 10
Vlan10 - Group 10
  State is Active
    2 state changes, last state change 01:02:08
  Virtual IP address is 10.10.1.1
  Active virtual MAC address is 0000.0c07.ac0a (MAC In Use)
    Local virtual MAC address is 0000.0c07.ac0a (v1 default)
  Hello time 3 sec, hold time 10 sec
    Next hello sent in 1.904 secs
  Preemption enabled, delay min 60 secs
  Active router is local
  Standby router is unknown
  Priority 150 (configured 150)
  Group name is "hsrp-Vl10-10" (default)
SW1#

SW2#sh standby vlan 10
Vlan10 - Group 10
  State is Active
    2 state changes, last state change 01:03:07
  Virtual IP address is 10.10.1.1
  Active virtual MAC address is 0000.0c07.ac0a (MAC In Use)
    Local virtual MAC address is 0000.0c07.ac0a (v1 default)
  Hello time 3 sec, hold time 10 sec
    Next hello sent in 2.032 secs
  Preemption disabled
  Active router is local
  Standby router is unknown
  Priority 90 (configured 90)
  Group name is "hsrp-Vl10-10" (default)
SW2#
Debugging HSRP shows that the two are not talking to each other, even though CDP and ARP look fine:
SW1#sh cdp neigh
Capability Codes: R - Router, T - Trans Bridge, B - Source Route Bridge
                  S - Switch, H - Host, I - IGMP, r - Repeater, P - Phone,
                  D - Remote, C - CVTA, M - Two-port Mac Relay

Device ID        Local Intrfce     Holdtme    Capability  Platform  Port ID
SW2              Eth 0/0           162             R S I  Linux Uni Eth 0/0
SW3              Eth 0/1           122             R S I  Linux Uni Eth 0/0
SW1#sh arp
Protocol  Address          Age (min)  Hardware Addr   Type   Interface
Internet  10.10.1.1               -   0000.0c07.ac0a  ARPA   Vlan10
Internet  10.10.1.2               -   aabb.cc80.0100  ARPA   Vlan10
SW1#debug standby
HSRP debugging is on
SW1#
*May 23 07:39:26.362: HSRP: Vl10 Grp 10 Hello  out 10.10.1.2 Active  pri 150 vIP 10.10.1.1
SW1#
*May 23 07:39:29.211: HSRP: Vl10 Grp 10 Hello  out 10.10.1.2 Active  pri 150 vIP 10.10.1.1
SW1#
*May 23 07:39:31.624: HSRP: Vl10 Grp 10 Hello  out 10.10.1.2 Active  pri 150 vIP 10.10.1.1
SW1#
*May 23 07:39:34.614: HSRP: Vl10 Grp 10 Hello  out 10.10.1.2 Active  pri 150 vIP 10.10.1.1
SW1#
*May 23 07:39:37.568: HSRP: Vl10 Grp 10 Hello  out 10.10.1.2 Active  pri 150 vIP 10.10.1.1
SW1#
*May 23 07:39:40.328: HSRP: Vl10 Grp 10 Hello  out 10.10.1.2 Active  pri 150 vIP 10.10.1.1
SW1#
*May 23 07:39:43.090: HSRP: Vl10 Grp 10 Hello  out 10.10.1.2 Active  pri 150 vIP 10.10.1.1
SW1#
No HSRP traffic gets returned. If we switch images we do get a bit further:
SW1#sh ver
Cisco IOS Software, Solaris Software (I86BI_LINUXL2-IPBASEK9-M), Experimental Version 15.1(20130124:233217) [dstivers-jan24-2013-team_track 101]
Copyright (c) 1986-2013 by Cisco Systems, Inc.
Compiled Thu 24-Jan-13 16:17 by dstivers

ROM: Bootstrap program is Linux

SW2 uptime is 7 minutes
System returned to ROM by reload at 0
System image file is "unix:/home/gns3/Documents/GNS3/images/i86bi_linux_l2-ipbasek9-ms.jan2"

SW1#sh standby vlan 10
Vlan10 - Group 10
  State is Active
    2 state changes, last state change 00:01:36
  Virtual IP address is 10.10.1.1
  Active virtual MAC address is 0000.0c07.ac0a (MAC In Use)
    Local virtual MAC address is 0000.0c07.ac0a (v1 default)
  Hello time 3 sec, hold time 10 sec
    Next hello sent in 0.160 secs
  Preemption enabled, delay min 60 secs
  Active router is local
  Standby router is 10.10.1.3, priority 90 (expires in 10.592 sec)
  Priority 150 (configured 150)
  Group name is "hsrp-Vl10-10" (default)
SW1#

SW2#sh standby vlan 10
Vlan10 - Group 10
  State is Standby
    1 state change, last state change 00:01:21
  Virtual IP address is 10.10.1.1
  Active virtual MAC address is 0000.0c07.ac0a (MAC Not In Use)
    Local virtual MAC address is 0000.0c07.ac0a (v1 default)
  Hello time 3 sec, hold time 10 sec
    Next hello sent in 2.256 secs
  Preemption disabled
  Active router is 10.10.1.2, priority 150 (expires in 10.256 sec)
  Standby router is local
  Priority 90 (configured 90)
  Group name is "hsrp-Vl10-10" (default)
SW2#
So HSRP looks good from the viewpoint of the switches. However, not all is well from the VPCS guest:
VPCS2> ip 10.10.1.254 /24
Checking for duplicate address...
PC1 : 10.10.1.254 255.255.255.0

VPCS2> sh arp

arp table is empty

VPCS2> ping 10.10.1.2
10.10.1.2 icmp_seq=1 ttl=255 time=1.500 ms
10.10.1.2 icmp_seq=2 ttl=255 time=5.000 ms

VPCS2> ping 10.10.1.3
10.10.1.3 icmp_seq=1 ttl=255 time=3.500 ms
10.10.1.3 icmp_seq=2 ttl=255 time=4.000 ms

VPCS2> ping 10.10.1.1
10.10.1.1 icmp_seq=1 timeout
10.10.1.1 icmp_seq=2 timeout

VPCS2> sh arp

aa:bb:cc:80:04:00  10.10.1.2 expires in 65 seconds
aa:bb:cc:80:05:00  10.10.1.3 expires in 68 seconds
00:00:0c:07:ac:0a  10.10.1.1 expires in 70 seconds

VPCS2>
It can see that there is the HSRP VIF, but cannot reach it. SW3 also knows of the VIF:
SW3#sh mac address-table
          Mac Address Table
-------------------------------------------

Vlan    Mac Address       Type        Ports
----    -----------       --------    -----
  10    0000.0c07.ac0a    DYNAMIC     Et0/0
  10    0050.7966.6802    DYNAMIC     Et0/0
  10    aabb.cc80.0400    DYNAMIC     Et0/0
  10    aabb.cc80.0500    DYNAMIC     Et0/0
Total Mac Addresses for this criterion: 4
SW3#
So it goes to show that although things might look like they are working in IOU switches, downstream hosts might have a different idea about the network - so much so that things may not actually be reachable. Getting the correct image is important where IOU is concerned. The people who enable us to get these images do a great job, but obviously cannot replicate 100% a device that relies on hardware to perform complex switching functions. There maybe functions that work in the first image used, that do not work in the second image I used. It can be a bit hit and miss.

If I do find an image that works in this scenario I will update the post, or if you know of one then please leave a comment below.
HSRP, Cisco Emulation software and ASICs

HSRP, Cisco Emulation software and ASICs

I wrote sometime back about how to implement HSRP using Cisco IOU and in GNS3, but it has been pointed out that even though it might look like it works, in fact it's actually a little more hit and miss, with some people having issues, such as not being able to ping the Virtual IP address configured on the HSRP standby group - check out the comments to follow the reference, hopefully you'll understand why I am writing this subsequent post.

A recap on HSRP

HSRP (Hot Standby Routing Protocol) allows you to set up the same SVI (Switched Virtual Interface) on a pair of switches so that you have some redundancy, in the event that one switch is unavailable then the virtual IP address assigned to the standby group will still be available. The virtual IP address is given the same MAC address, which is a combination of the well-known HSRP MAC address (0000:0c07:ac) and the standby group number (in our case this is 10, which is 0a in hexadecimal).

HSRP is Cisco specific, other vendors user the industry standard of VRRP (Virtual Router Redundancy Protocol), there is also GLBP (Gateway Load Balancing Protocol) for routers.

We will be using two different versions of GNS3 for this, the first is the publicly available 0.8.6 version, because that can still use qemu, which means that we can use the IOSv images to try this out on, as well as the image c3660-is-mz.124.25b referenced in the first article. If you haven't checked out how to get the IOSv images from onePK into GNS3 then you might want to have a quick read of it.

So the goals for this post are to prove that we can get HSRP working between two devices. One should show up as the active and one as the standby, and we should be able to ping the virtual IP assigned to the HSRP group. As per the original post we need to create a multi-layer switch to enable the required functions, this still holds true today, so do have a read over the original article.

For each example we will have an HSRP group, number 10, one switch will use the IP address 10.10.1.2/24, the other will use 10.10.1.3/24 and they will have a virtual IP address of 10.10.1.1. The HSRP MAC address will be 0000.0c07.ac0a, so this is what we will be looking for in our ARP table.

Let's check out GNS3 first.

HSRP using native GNS3

Once we have set up our multi-layer switch in GNS3, the configurational steps for our first switch are as follows:
R4#vlan database
R4(vlan)#vlan 10
VLAN 10 added:
    Name: VLAN0010
R4(vlan)#exit
APPLY completed.
Exiting....   
*Mar  1 00:00:35.719: %LINEPROTO-5-UPDOWN: Line protocol on Interface Vlan1, changed state to up
R4#conf t
Enter configuration commands, one per line.  End with CNTL/Z.
R4(config)#int vlan 10
R4(config-if)#ip add 10.10.1.2 255.255.255.0
R4(config-if)#standby 10 ip 10.10.1.1
R4(config-if)#standby 10 pri 150
R4(config-if)#standby 10 preem del min 60
R4(config-if)#no shut
R4(config-if)#int fa 1/10
R4(config-if)#switchport mode trunk
*Mar  1 00:02:00.087: %DTP-5-TRUNKPORTON: Port Fa1/10 has become dot1q trunk
*Mar  1 00:02:00.563: %LINEPROTO-5-UPDOWN: Line protocol on Interface Vlan1, changed state to down
R4(config-if)#no shut
R4(config-if)#
*Mar  1 00:02:30.687: %LINEPROTO-5-UPDOWN: Line protocol on Interface Vlan1, changed state to up
*Mar  1 00:02:30.695: %LINEPROTO-5-UPDOWN: Line protocol on Interface Vlan10, changed state to up
R4(config-if)#
*Mar  1 00:02:51.699: %HSRP-5-STATECHANGE: Vlan10 Grp 10 state Speak -> Standby
*Mar  1 00:02:52.199: %HSRP-5-STATECHANGE: Vlan10 Grp 10 state Standby -> Active
R4(config-if)#exit
R4(config)#exit
R4#
*Mar  1 00:03:34.879: %SYS-5-CONFIG_I: Configured from console by console
R4#
So far our first device is looking healthy for HSRP. The second switch is very similar:
R5#vlan database
R5(vlan)#vlan 10
VLAN 10 added:
    Name: VLAN0010
R5(vlan)#exit
APPLY completed.
Exiting....
R5#conf t
Enter configuration commands, one per line.  End with CNTL/Z.
R5(config)#int fa1/10
R5(config-if)#switchport mode trunk
*Mar  1 00:02:45.687: %DTP-5-TRUNKPORTON: Port Fa1/10 has become dot1q trunk
*Mar  1 00:02:46.159: %LINEPROTO-5-UPDOWN: Line protocol on Interface Vlan1, changed state to down
R5(config-if)#no shut
R5(config-if)#exit
R5(config)#int vlan 10
R5(config-if)#
*Mar  1 00:02:53.711: %LINEPROTO-5-UPDOWN: Line protocol on Interface Vlan10, changed state to up
R5(config-if)#ip add 10.10.1.3 255.255.255.0
R5(config-if)#standby 10 ip 10.10.1.1
R5(config-if)#standby 10 pri 90
R5(config-if)#
*Mar  1 00:03:16.291: %LINEPROTO-5-UPDOWN: Line protocol on Interface Vlan1, changed state to up
R5(config-if)#
*Mar  1 00:03:40.299: %HSRP-5-STATECHANGE: Vlan10 Grp 10 state Speak -> Standby
R5(config-if)#
With our trunk setup to carry the VLAN information, as well as the HSRP multicast, we should be able to check that everything is in order. Using the command "sh standby vlan" with the vlan number we can check the HSRP settings and status. Let's check to see if we have everything we are hoping for, including the correct ARP entries and making sure that the interfaces are pingable:
R4#sh standby vlan 10
Vlan10 - Group 10
  State is Active
    2 state changes, last state change 00:03:19
  Virtual IP address is 10.10.1.1
  Active virtual MAC address is 0000.0c07.ac0a
    Local virtual MAC address is 0000.0c07.ac0a (v1 default)
  Hello time 3 sec, hold time 10 sec
    Next hello sent in 1.724 secs
  Preemption enabled, delay min 60 secs
  Active router is local
  Standby router is 10.10.1.3, priority 90 (expires in 7.344 sec)
  Priority 150 (configured 150)
  IP redundancy name is "hsrp-Vl10-10" (default)
R4#sh arp
Protocol  Address          Age (min)  Hardware Addr   Type   Interface
Internet  10.10.1.1               -   0000.0c07.ac0a  ARPA   Vlan10
Internet  10.10.1.2               -   cc04.38a1.0000  ARPA   Vlan10
R4#ping 10.10.1.1

Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to 10.10.1.1, timeout is 2 seconds:
!!!!!
Success rate is 100 percent (5/5), round-trip min/avg/max = 1/2/4 ms
R4#sh arp
Protocol  Address          Age (min)  Hardware Addr   Type   Interface
Internet  10.10.1.1               -   0000.0c07.ac0a  ARPA   Vlan10
Internet  10.10.1.3               0   cc05.38a1.0000  ARPA   Vlan10
Internet  10.10.1.2               -   cc04.38a1.0000  ARPA   Vlan10
R4#

R5#sh standby vlan 10
Vlan10 - Group 10
  State is Standby
    1 state change, last state change 00:03:22
  Virtual IP address is 10.10.1.1
  Active virtual MAC address is 0000.0c07.ac0a
    Local virtual MAC address is 0000.0c07.ac0a (v1 default)
  Hello time 3 sec, hold time 10 sec
    Next hello sent in 1.992 secs
  Preemption disabled
  Active router is 10.10.1.2, priority 150 (expires in 7.376 sec)
  Standby router is local
  Priority 90 (configured 90)
  IP redundancy name is "hsrp-Vl10-10" (default)
R5#sh arp
Protocol  Address          Age (min)  Hardware Addr   Type   Interface
Internet  10.10.1.3               -   cc05.38a1.0000  ARPA   Vlan10
R5#ping 10.10.1.1

Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to 10.10.1.1, timeout is 2 seconds:
.!!!!
Success rate is 80 percent (4/5), round-trip min/avg/max = 20/21/24 ms
R5#ping 10.10.1.2

Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to 10.10.1.2, timeout is 2 seconds:
.!!!!
Success rate is 80 percent (4/5), round-trip min/avg/max = 20/21/24 ms
R5#sh arp        
Protocol  Address          Age (min)  Hardware Addr   Type   Interface
Internet  10.10.1.1               0   0000.0c07.ac0a  ARPA   Vlan10
Internet  10.10.1.3               -   cc05.38a1.0000  ARPA   Vlan10
Internet  10.10.1.2               0   cc04.38a1.0000  ARPA   Vlan10
R5#
So arp looks good and we can ping the Virtual IP address for the standby group, with no problems here. Let's check out HSRP on IOU next.

HSRP using IOU

With the same configuration (apart from the priority of 150 on the first device) as before running on the IOU routers do we have the same level of visibility?
Switch#sh standby vlan 10
Vlan10 - Group 10
  State is Active
    2 state changes, last state change 00:00:34
  Virtual IP address is 10.10.1.1
  Active virtual MAC address is 0000.0c07.ac0a (MAC In Use)
    Local virtual MAC address is 0000.0c07.ac0a (v1 default)
  Hello time 3 sec, hold time 10 sec
    Next hello sent in 1.984 secs
  Preemption enabled, delay min 60 secs
  Active router is local
  Standby router is 10.10.1.3, priority 90 (expires in 9.904 sec)
  Priority 100 (default 100)
  Group name is "hsrp-Vl10-10" (default)
Switch#sh arp
Protocol  Address          Age (min)  Hardware Addr   Type   Interface
Internet  10.10.1.1               -   0000.0c07.ac0a  ARPA   Vlan10
Internet  10.10.1.2               -   aabb.cc80.0100  ARPA   Vlan10
Internet  10.10.1.3               0   aabb.cc80.0200  ARPA   Vlan10
Switch#ping 10.10.1.1
Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to 10.10.1.1, timeout is 2 seconds:
!!!!!
Success rate is 100 percent (5/5), round-trip min/avg/max = 4/4/5 ms
Switch#sh ver | i IOS
Cisco IOS Software, Solaris Software (I86BI_LINUXL2-IPBASEK9-M), Experimental Version 15.1(20130124:233217) [dstivers-jan24-2013-team_track 101]
Switch#sh ip int bri     
Interface              IP-Address      OK? Method Status                Protocol
Ethernet0/0            unassigned      YES unset  up                    up      
Ethernet0/1            unassigned      YES unset  up                    up      
Ethernet0/2            unassigned      YES unset  up                    up      
Ethernet0/3            unassigned      YES unset  up                    up      
Vlan10                 10.10.1.2       YES manual up                    up      
Switch#

Switch2#sh standby vlan 10
Vlan10 - Group 10
  State is Standby
    1 state change, last state change 00:00:29
  Virtual IP address is 10.10.1.1
  Active virtual MAC address is 0000.0c07.ac0a (MAC Not In Use)
    Local virtual MAC address is 0000.0c07.ac0a (v1 default)
  Hello time 3 sec, hold time 10 sec
    Next hello sent in 0.912 secs
  Preemption disabled
  Active router is 10.10.1.2, priority 100 (expires in 9.696 sec)
  Standby router is local
  Priority 90 (configured 90)
  Group name is "hsrp-Vl10-10" (default)
Switch2#sh arp
Protocol  Address          Age (min)  Hardware Addr   Type   Interface
Internet  10.10.1.3               -   aabb.cc80.0200  ARPA   Vlan10
Switch2#ping 10.10.1.2
Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to 10.10.1.2, timeout is 2 seconds:
.!!!!
Success rate is 80 percent (4/5), round-trip min/avg/max = 1/1/1 ms
Switch2#ping 10.10.1.1
Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to 10.10.1.1, timeout is 2 seconds:
.!!!!
Success rate is 80 percent (4/5), round-trip min/avg/max = 1/1/1 ms
Switch2#sh ver | i IOS
Cisco IOS Software, Solaris Software (I86BI_LINUXL2-IPBASEK9-M), Experimental Version 15.1(20130124:233217) [dstivers-jan24-2013-team_track 101]
Switch2#
Again arp looks good and we can ping the virtual IP address again.

EDIT:-

It just goes to show that image is everything. As Maureen (in the comments below) has pointed out whilst following this blog and my posts about HSRP, nothing is 100% guaranteed when using a virtualized environment.

If you try adding a guest to the mix then it probably won't be able to ping the standby address, and if you do, it probably won't last. I will go into it in more depth in my post about IOU Images and functions.

Finally lets check out IOSv.

HSRP using IOSv on GNS3

HSRP is slightly different on IOSv, there are no vlan commands, and instead everything is performed under the interface, so technically it's not enabling redundancy for a vlan, it's enabling it for the interface, but the principal remains the same, it still requires the same coding within IOS for the feature to work, and multicast for the communication to flow over. I am including it here because of this. 
SW1(config)#int gi 0/0
SW1(config-if)#ip add 10.10.1.2 255.255.255.0
SW1(config-if)#standby 10 ip 10.10.1.1
SW1(config-if)#standby 10 pri 150
SW1(config-if)#standby 10 preem delay min 60
SW1(config-if)#no shut
SW1(config-if)# 
*May 11 16:11:44.818: %LINK-3-UPDOWN: Interface GigabitEthernet0/0, changed state to up
*May 11 16:11:45.818: %LINEPROTO-5-UPDOWN: Line protocol on Interface GigabitEthernet0/0, changed state to up
*May 11 16:12:05.647: %HSRP-5-STATECHANGE: GigabitEthernet0/0 Grp 10 state Standby -> Active
SW1(config-if)#exit
SW1(config)#exit
SW1#sh standby
*May 11 16:12:25.655: %SYS-5-CONFIG_I: Configured from console by consoleand 
GigabitEthernet0/0 - Group 10
  State is Active
    2 state changes, last state change 00:00:27
  Virtual IP address is 10.10.1.1
  Active virtual MAC address is 0000.0c07.ac0a
    Local virtual MAC address is 0000.0c07.ac0a (v1 default)
  Hello time 3 sec, hold time 10 sec
    Next hello sent in 0.272 secs
  Preemption enabled, delay min 60 secs
  Active router is local
  Standby router is 10.10.1.3, priority 90 (expires in 9.232 sec)
  Priority 150 (configured 150)
  Group name is "hsrp-Gi0/0-10" (default)
SW1#

SW2(config)#int gi 0/0
SW2(config-if)#ip add 10.10.1.3 255.255.255.0
SW2(config-if)#standby 10 ip 10.10.1.1
SW2(config-if)#standby 10 pri 90      
SW2(config-if)#no shut
SW2(config-if)#
*May 11 16:11:44.460: %LINK-3-UPDOWN: Interface GigabitEthernet0/0, changed state to up
*May 11 16:11:45.460: %LINEPROTO-5-UPDOWN: Line protocol on Interface GigabitEthernet0/0, changed state to up
*May 11 16:12:29.371: %HSRP-5-STATECHANGE: GigabitEthernet0/0 Grp 10 state Speak -> Standby
SW2(config-if)#
SW2(config-if)#exit
SW2(config)#exit
SW2#sh stand
GigabitEthernet0/0 - Group 10
  State is Standby
    1 state change, last state change 00:00:23
  Virtual IP address is 10.10.1.1
  Active virtual MAC address is 0000.0c07.ac0a
    Local virtual MAC address is 0000.0c07.ac0a (v1 default)
  Hello time 3 sec, hold time 10 sec
    Next hello sent in 1.744 secs
  Preemption disabled
  Active router is 10.10.1.2, priority 150 (expires in 10.480 sec)
  Standby router is local
  Priority 90 (configured 90)
  Group name is "hsrp-Gi0/0-10" (default)
SW2#
So it certainly looks like HSRP works on IOSv, let's make sure by looking at arp, and seeing if we can ping the virtual IP address:
SW1#sh arp
Protocol  Address          Age (min)  Hardware Addr   Type   Interface
Internet  10.10.1.1               -   0000.0c07.ac0a  ARPA   GigabitEthernet0/0
Internet  10.10.1.2               -   00ab.60a2.3000  ARPA   GigabitEthernet0/0
SW1#ping 10.10.1.3
Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to 10.10.1.3, timeout is 2 seconds:
.!!!!
Success rate is 80 percent (4/5), round-trip min/avg/max = 3/8/11 ms
SW1#ping 10.10.1.1
Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to 10.10.1.1, timeout is 2 seconds:
!!!!!
Success rate is 100 percent (5/5), round-trip min/avg/max = 1/1/2 ms
SW1#sh ver | i Version
Cisco IOS Software, IOSv Software (VIOS-ADVENTERPRISEK9-M), Version 15.4(1.24)T0.9, MAINTENANCE INTERIM SOFTWARE
SW1#

SW2#sh arp
Protocol  Address          Age (min)  Hardware Addr   Type   Interface
Internet  10.10.1.2               0   00ab.60a2.3000  ARPA   GigabitEthernet0/0
Internet  10.10.1.3               -   00ab.1061.0700  ARPA   GigabitEthernet0/0
SW2#ping 10.10.1.1
Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to 10.10.1.1, timeout is 2 seconds:
.!!!!
Success rate is 80 percent (4/5), round-trip min/avg/max = 4/7/11 ms
SW2#sh arp
Protocol  Address          Age (min)  Hardware Addr   Type   Interface
Internet  10.10.1.1               0   0000.0c07.ac0a  ARPA   GigabitEthernet0/0
Internet  10.10.1.2               0   00ab.60a2.3000  ARPA   GigabitEthernet0/0
Internet  10.10.1.3               -   00ab.1061.0700  ARPA   GigabitEthernet0/0
SW2#sh ver | i Version
Cisco IOS Software, IOSv Software (VIOS-ADVENTERPRISEK9-M), Version 15.4(1.24)T0.9, MAINTENANCE INTERIM SOFTWARE
SW2#
So again, all good and we are three for three.

So why am I going over this again, if actually I am only proving that it does work? Well, it's all to do with mileage, and your mileage may vary. Remember we are dealing with emulated, or virtualized hardware, so there will be issues, especially when we come across things that the router, or more specifically switch, generally offload to ASICs. With all the benefits that Cisco emulation software offers, the one thing it cannot do is truly emulate the functions contained within the ASIC.

What's a running shoe brand got to do with Cisco emulation software?

An ASIC is an Application-Specific Integrated Circuit, not to be confused with ASICS the running shoe manufacturer. These ASICs are so complex that they can (depending on the type of ASIC) be referred to as a system on a chip. It is this off-loading that makes running a virtualised switch, with the full functionality and speed as a proper hardware one, extremely difficult to implement.

The most common Cisco specific ASIC you will encounter is the port ASIC, and these handle traffic forwarding, QoS and ACL lookups. A number of switches have within these port ASICs their TCAM, which holds the IPv4 and IPV6 addresses, MAC addresses, and Access Control Entries.

For a greater understanding of packet flow within a switch and how this is governed by the ASIC then have a look at this PDF from Cisco Live.

So in short, we can get close, but close is a very relative term, finding the right IOU or GNS3 image can be a bit of a mission.

It will be interesting to see how CML stands up to higher-level switch functionality when it is finally released. One would think that as it is the basis for the new version of the CCIE exam then it will be supported. Time will tell on that!

In an ideal world we would all have the required hardware, it wouldn't cost more than a cup of coffee, and it would fit into your pocket. But for the moment we'll have to see how far our mileage gets us.

Cisco ASA failover, redundant interfaces, Catalyst HSRP and power

Most networks have redundancy (or at least should do). This can be in the form of additional cables cross-connecting devices, dual power supplies going to different feeds and HSRP on switches. All of this is good stuff, but some things can easily be taken for granted.

Consider the following topology:

We have two ASAs in a failover pair (Outside interface 10.30.1.1 with standby ip 10.30.1.2), both connecting to a switch (10.30.1.254), that has a loopback interface assigned to it to simulate the internet (10.250.1.1). The ASAs also connect to a pair of switches, set up in an HSRP group (ASA inside interface 10.40.1.1, standby ip 10.40.1.10, switch vlan 40 ip 10.40.1.254). The topology looks like this:



It looks pretty redundant right? But let's work through the config and see what could go wrong. We'll start by setting up the ASAs in a failover pair.

Cisco ASA failover setup

On the first ASA we set up failover as follows:
ciscoasa(config)# enable password cisco
ciscoasa(config)# hostname ASA1
ASA1(config)# failover
ASA1(config)# failover lan unit primary
ASA1(config)# failover lan interface FAILOVER GigabitEthernet0/3
INFO: Non-failover interface config is cleared on GigabitEthernet0/3 and its sub-interfaces
ASA1(config)# failover key fail0ver
ASA1(config)# failover replication http
ASA1(config)# failover link FAILOVER GigabitEthernet0/3
ASA1(config)# failover interface ip FAILOVER 10.20.1.1 255.255.255.252 standby 10.20.1.1
ASA1(config)# int gi0/3
ASA1(config-if)# no shut
ASA1(config-if)# exit
ASA1(config)# exit
ASA1# copy run start
The second ASA is similar:
ciscoasa(config)# enable password cisco
ciscoasa(config)# hostname ASA2
ASA2(config)# failover
ASA2(config)# failover lan unit secondary
ASA2(config)# failover lan interface FAILOVER GigabitEthernet0/3
INFO: Non-failover interface config is cleared on GigabitEthernet0/3 and its sub-interfaces
ASA2(config)# failover key fail0ver
ASA2(config)# failover replication http
ASA2(config)# failover link FAILOVER GigabitEthernet0/3
ASA2(config)# failover interface ip FAILOVER 10.20.1.1 255.255.255.252 standby 10.20.1.2
ASA2(config)# int gi0/3
ASA2(config-if)# no shut
ASA2(config-if)#exit
ASA2(config)#exit
ASA2# copy run start
We can then save our configuration, so that the partner (or mate) firewall gets sent the active configuration:
ASA1# copy run start
ASA1# Beginning configuration replication: Sending to mate.
End Configuration Replication to mate
I do like to set the prompt to show me easily which firewall I am currently on:
ASA1#
ASA1# conf t
ASA1(config)# prompt hostname state
ASA1/act(config)# exit
ASA1/act# copy run start

Source filename [running-config]?
Cryptochecksum: 9f0c4260 95438d98 2d1439ef 0704bf77

2693 bytes copied in 3.310 secs (897 bytes/sec)
ASA1/act#
Our partner ASA will now change to show its status in the prompt:
ASA1#
ASA1/stby# Cryptochecksum: 0df25109 9f4fecb5 cebb9db5 93664d73

ASA1/stby#
2691 bytes copied in 3.230 secs (897 bytes/sec)
ASA1/stby#
Now we can configure our connection to the "internet" and to the inside network:

ASA Outside (internet) connection

ASA1/act# conf t
ASA1/act(config)# int Gi0/0
ASA1/act(config-if)# nameif OUTSIDE
INFO: Security level for "OUTSIDE" set to 0 by default.
ASA1/act(config-if)# ip address 10.30.1.1 255.255.255.0 standby 10.30.1.2
ASA1/act(config-if)# no shut
ASA1/act(config-if)#exit
ASA1/act(config)# exit
ASA1/act# copy run start

ASA Inside connection

ASA1/act(config-if)# int gi 0/1
ASA1/act(config-if)# nameif INSIDE
INFO: Security level for "INSIDE" set to 100 by default.
ASA1/act(config-if)# ip address 10.40.1.1 255.255.255.0 standby 10.40.1.10
ASA1/act(config-if)# no shut
ASA1/act(config-if)# exit
ASA1/act(config)# exit
ASA1/act# copy run start
Now we have connections on the inside and outside:

Switch HSRP setup

I have covered HSRP a few times now, so won't go into any details, but here is the configuration walkthrough:
Switch(config)#hostname SW1
SW1(config)#vlan 40
SW1(config-vlan)#exit
SW1(config)#int vlan 40
SW1(config-if)#ip address 10.40.1.2 255.255.255.0
00:06:08: %LINEPROTO-5-UPDOWN: Line protocol on Interface Vlan40, changed state to down
SW1(config-if)#standby 40 ip 10.40.1.254
SW1(config-if)#standby 40 pri 150
SW1(config-if)#
00:09:16: %LINEPROTO-5-UPDOWN: Line protocol on Interface FastEthernet1/0/2, changed state to down
SW1(config-if)#exit
00:09:19: %LINEPROTO-5-UPDOWN: Line protocol on Interface FastEthernet1/0/2, changed state to up
SW1(config)#int fastEthernet 1/0/2
SW1(config-if)#switchport trunk encap dot1q
SW1(config-if)#switchport mode trunk
00:09:49: %LINEPROTO-5-UPDOWN: Line protocol on Interface Vlan40, changed state to up
SW1(config-if)#
00:10:10: %HSRP-6-STATECHANGE: Vlan40 Grp 40 state Standby -> Active
SW1(config-if)#

Switch(config)#hostname SW2
SW2(config)#vlan 40
SW2(config-vlan)#exit
SW2(config)#int vlan 40
*Mar  1 00:06:56.997: %LINEPROTO-5-UPDOWN: Line protocol on Interface Vlan40, changed state to down
SW2(config-if)#ip address 10.40.1.3 255.255.255.0
SW2(config-if)#standby 40 ip 10.40.1.254
SW2(config-if)#standby pri 90
SW2(config-if)#exit
SW2(config)#int fa0/2
SW2(config-if)#switchport trunk encap dot1q
SW2(config-if)#switchport mode trunk
SW2(config-if)#
*Mar  1 00:09:22.447: %LINEPROTO-5-UPDOWN: Line protocol on Interface FastEthernet0/2, changed state to down
*Mar  1 00:09:25.517: %LINEPROTO-5-UPDOWN: Line protocol on Interface FastEthernet0/2, changed state to up
*Mar  1 00:09:55.515: %LINEPROTO-5-UPDOWN: Line protocol on Interface Vlan40, changed state to up
*Mar  1 00:10:36.561: %HSRP-5-STATECHANGE: Vlan40 Grp 40 state Speak -> Standby
SW2(config-if)#

Switch to ASA configuration

SW1(config)#int fa1/0/1
SW1(config-if)#switchport mode access
SW1(config-if)#switchport access vlan 40
SW1(config-if)#exit
SW1(config)#exit
SW1#ping 10.40.1.1
Type escape sequence to abort.

Sending 5, 100-byte ICMP Echos to 10.40.1.1, timeout is 2 seconds:
.!!!!
Success rate is 80 percent (4/5), round-trip min/avg/max = 1/1/1 ms

SW2(config)#int fa0/1
SW2(config-if)#switchport mode access
SW2(config-if)#switchport access vlan 40
SW2(config-if)#exit
SW2(config)#exit
SW2#
*Mar  1 01:09:09.651: %SYS-5-CONFIG_I: Configured from console by console
SW2#ping 10.40.1.1
Type escape sequence to abort.

Sending 5, 100-byte ICMP Echos to 10.40.1.1, timeout is 2 seconds:
!!!!!
Success rate is 100 percent (5/5), round-trip min/avg/max = 1/2/8 ms

ASAs to "Internet" Configuration

On the active ASA we need to set a route out to the internet:
ASA1/act# sh run | i route
route OUTSIDE 0.0.0.0 0.0.0.0 10.30.1.254 1

Internet configuration

The internet is very simple, we have a loopback that will be our end-point for testing, and a vlan for the ASAs to communicate with:
3550-SW3#sh run
interface Loopback0
  ip address 10.250.1.1 255.255.255.0
!
interface FastEthernet0/1
  switchport access vlan 10
  switchport mode access
!
interface FastEthernet0/11
  switchport access vlan 10
  switchport mode access
!
interface Vlan10
  ip address 10.30.1.254 255.255.255.0

NAT statements to allow access

To allow the inside to talk to the internet we need to set up NATing on the ASAs:
global (OUTSIDE) 1 interface
nat (INSIDE) 1 10.40.1.0 255.255.255.0

Testing

Let's see how out connectivity looks:
ASA1/act# ping 10.250.1.1

Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to 10.250.1.1, timeout is 2 seconds:
!!!!!
Success rate is 100 percent (5/5), round-trip min/avg/max = 1/2/10 ms

ASA1/stby> ping 10.250.1.1

Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to 10.250.1.1, timeout is 2 seconds:
!!!!!
Success rate is 100 percent (5/5), round-trip min/avg/max = 1/2/10 ms

SW1#ping 10.250.1.1

Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to 10.250.1.1, timeout is 2 seconds:
!!!!!
Success rate is 100 percent (5/5), round-trip min/avg/max = 1/2/8 ms

SW2#ping 10.250.1.1

Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to 10.250.1.1, timeout is 2 seconds:
!!!!!
Success rate is 100 percent (5/5), round-trip min/avg/max = 1/2/8 ms
SW2#
So far so good!

Forcing failover and simulating outages 

We can simulate issues in a number of ways. I started by removing the power to SW1. This made SW2 go active for vlan 40. SW2 has a good connection to ASA2 so pings were 100% successful.

With ASA1 as the standby we can force a failover from ASA2:
ASA1/stby# failover active

    Switching to Active

SW2#ping 10.250.1.1

Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to 10.250.1.1, timeout is 2 seconds:
....!
Success rate is 20 percent (1/5), round-trip min/avg/max = 1/1/1 ms

ASA1/act# Waiting for the earlier webvpn instance to terminate...
Previous instance shut down. Starting a new one.

    Switching to Standby
At this point the firewalls failed back, due to the traffic generated by SW2, so that ASA2 became the active mate again. If ASA2 had the power removed then they would not be able to fail back, and therefore SW2 would have no way to route the traffic over to SW1 (which would be the active firewall), and therefore the inside of the network would lose all internet communication.

So we can survive one ASA power outage and one switch outage (ASA1 and SW1 or ASA2 and SW2) but not if we lose SW2 and ASA1 or SW1 and ASA2.

So how can we get better resiliency? We can do this with a cable from SW1 to ASA2 and from SW2 to ASA1. This does require some reconfiguration on the ASAs, but how can we do this? We certainly can't have another physical interface on the ASAs with the same IP range, as we would get an error informing us of an IP overlap. They key work here is physical interface. We can however set up a different type of interface to achieve this.

Cisco ASA Redundant interfaces

Redundant interfaces are two or more interfaces joined together in a redundancy group. They act like physical interfaces, in that they have an IP address, but with the added bonus of having multiple links.

We start by adding another interface on the switches (fa1/0/3 on SW1 and fa0/3 on SW3) to be access ports on vlan 40, and these interfaces are connected to GigabitEthernet 0/2 on the "other" ASA:



Next we move onto the firewalls and remove the IP information from the existing inside interfaces:
ASA1/act(config)# int GigabitEthernet0/1
ASA1/act(config-if)# no nameif
ASA1/act(config-if)# no ip address
ASA1/act(config-if)# exit
Then we can create a redundant port:
ASA1/act(config)# interface Redundant1
ASA1/act(config-if)# member-interface GigabitEthernet0/1
INFO: security-level and IP address are cleared on GigabitEthernet0/1.
ASA1/act(config-if)# member-interface GigabitEthernet0/2
INFO: security-level and IP address are cleared on GigabitEthernet0/2.
ASA1/act(config-if)# nameif INSIDE
INFO: Security level for "INSIDE" set to 100 by default.
ASA1/act(config-if)# ip address 10.40.1.1 255.255.255.0 standby 10.40.1.10
ASA1/act(config-if)#
I could have saved a bit of typing as adding a member interface to a redundant interface will clear the security-level and IP address automatically, but its useful to do manually.

Once we have saved our config we can check on the standby ASA:
ASA1/stby# sh run
: Saved
<Truncated>
interface GigabitEthernet0/1
  no nameif
  no security-level
  no ip address
!
interface GigabitEthernet0/2
  no nameif
  no security-level
  no ip address
!
interface Redundant1
  member-interface GigabitEthernet0/1
  member-interface GigabitEthernet0/2
  nameif INSIDE
  security-level 100
  ip address 10.40.1.1 255.255.255.0 standby 10.40.1.10
!
At this point we cant ping the loopback of 10.250.1.1 on the "internet", and to do this we need to do some NATing on the ASA, because when we clear the interface on the ASA it removes our NAT statement:
ASA1/act(config)# nat (INSIDE) 1 10.40.1.0 255.255.255.0
Once that is done we can ping the loopback again:
SW1#ping 10.250.1.1

Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to 10.250.1.1, timeout is 2 seconds:
!!!!!
Success rate is 100 percent (5/5), round-trip min/avg/max = 1/2/9 ms

SW2#ping 10.250.1.1

Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to 10.250.1.1, timeout is 2 seconds:
!!!!!
Success rate is 100 percent (5/5), round-trip min/avg/max = 1/2/8 ms
SW2#

Testing the redundant interface with failover

Now if we failover the ASAs (ASA1 being the primary, so testing from SW1) do we lose any pings?
SW1#ping 10.250.1.1 repeat 4000

Type escape sequence to abort.
Sending 4000, 100-byte ICMP Echos to 10.250.1.1, timeout is 2 seconds:
!!!!!
<omitted>
!!!!!
Success rate is 100 percent (4000/4000), round-trip min/avg/max = 1/2/9 ms
No we dont.

If we remove SW1 (simulating a power outage) does SW2 lose any pings?
SW2#ping 10.250.1.1 repeat 4000

Type escape sequence to abort.
Sending 4000, 100-byte ICMP Echos to 10.250.1.1, timeout is 2 seconds:
<omitted>
*Mar  5 02:12:17.681: %LINEPROTO-5-UPDOWN: Line protocol on Interface FastEthernet0/2, changed state to down
<omitted>
*Mar  5 02:12:18.746: %LINK-3-UPDOWN: Interface FastEthernet0/2, changed state to down
Success rate is 100 percent (4000/4000), round-trip min/avg/max = 1/3/34 ms
Again no ping loss. And if we turn off ASA2 (now the primary):
SW2#ping 10.250.1.1 repeat 4000

Type escape sequence to abort.
Sending 4000, 100-byte ICMP Echos to 10.250.1.1, timeout is 2 seconds:
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!.
*Mar  5 02:13:49.133: %LINEPROTO-5-UPDOWN: Line protocol on Interface FastEthernet0/1, changed state to down
*Mar  5 02:13:50.140: %LINK-3-UPDOWN: Interface FastEthernet0/1, changed state to down.......!!!!!!!!!!!!!!!!!!!!!!!
<omitted>
Success rate is 99 percent (3992/4000), round-trip min/avg/max = 1/3/17 ms
Then yes we do lose a few pings, but only in the time it takes ASA 1 to become the active again. But losing a couple of pings is better than losing all communication! Now we can lose one switch and one ASA on different sides of the topology and still have a working (redundant) network.

Fun with QinQ tunnels - Part 3 - Why HSRP and QinQ don't play well

I have covered QinQ tunnels a couple of times now, but for an exceedingly brief recap a QinQ tunnel extends the layer 2 network across a WAN. You can read the first two posts on QinQ tunnels here and here. They are well worth a read to understand what I will be discussing in this post.

For all the flexibility that having a QinQ tunnel across two sites gives us, it can introduce problems. Imagine the scenario. We have Site A and Site B. Site B is a backup (or non-production) site, and services (if required) can be run out of Site B if Site A fails, with minimal changes required.

Some background on both the sites.
  • Both sites are configured with a VPN tunnel between them, and a QinQ tunnel between a switch in both sites.
  • Both sites have very similar hardware and software configurations. The only differences would be the IP subnets used.
  • Both sites are set up to maximise HA where ever possible.
The hardware and network

At the top of the stack is a pair of Cisco ASAs with high availability set up (in a failover pair and are tracking the switches on the inside interface).
Connected to the ASAs are a pair of Catalysts, these are connected together using a channel group (ether channel - which you can read about here).
The VLANs that are used in the network are set up using HSRP for HA purposes.

The topology for the sites will look much like this:

Basic HSRP topology

What is HSRP?

HSRP stands for Hot Standby Router Protocol, and is Cisco proprietary. Other vendors will probably use VRRP, which is supported on Cisco hardware, and the commands for which are very similar to how we would set up HSRP. With HSRP we assign interfaces (or more frequently) VLAN interfaces to an HSRP group, one will be the active, and one will be the standby, and these virtual IP addresses are used as a default gateway.  

How to configure HSRP

A basic HSRP configuration for a vlan would look like this:
SW1#conf t
SW1(config)#int vlan 10
SW1(config-if)#ip address 1.1.10.2 255.255.255.0
SW1(config-if)#standby 10 ip 1.1.10.1
SW1(config-if)#standby 10 priority 90

SW2#conf t
SW2(config)#int vlan 10
SW2(config-if)#ip address 1.1.10.3 255.255.255.0
SW2(config-if)#standby 10 ip 1.1.10.1
SW2(config-if)#standby 10 priority 150
SW2(config-if)#standby 10 preempt delay minimum 60
We have set up a vlan (10) on both switches. SW1 will have a local IP address of 1.1.10.2 (/24) and SW2 will have a local IP address of 1.1.10.3. The virtual IP address of the VLAN will be 1.1.10.1, which is what we would set the default gateway of our client machines in VLAN 10 to. SW2 would become the active switch for the VLAN as its configured priority (which if not configured would default to 100) is higher than that of SW2. Lastly we set the preempt delay so that the SW1 should wait for one minute before trying to reclaim its active status for the VLAN (useful if its flapping for any reason, you don't want it to try becoming the active member too quickly, only to have it flap again).

HSRP and QinQ tunnels

So there we have a quick over view of HSRP. Now remember when I said that both sites have been configured with very similar hardware and software set ups? Well if Site A has the HSRP set up as above and Site B has the following setup:
SW1#conf t
SW1(config)#int vlan 10
SW1(config-if)#ip address 1.2.10.2 255.255.255.0
SW1(config-if)#standby 10 ip 10.2.10.1
SW1(config-if)#standby 10 priority 90

SW2#conf t
SW2(config)#int vlan 10
SW2(config-if)#ip address 1.2.10.3 255.255.255.0
SW2(config-if)#standby 10 ip 10.2.10.1
SW2(config-if)#standby 10 priority 150
SW2(config-if)#standby 10 preempt delay minimum 60
What will happen when we introduce our QinQ link?

Well, the IP addresses are clearly different, there are in totally different subnets, so everything should be fine right?

Well no. Our QinQ link will need a separate subnet (common on both sides to communicate), but will also be used to trunk between our two sides (so we can make failing over to the other site, or file transfers, etc, quicker). so although the separate QinQ interfaces in Site A and Site B can communicate over a separate VLAN these interfaces will also need to be opened up to allow other subnets to use the trunk link between the sites.

The general topology would look a bit like this:

HSRP QinQ topology

So we have the VPN between ASA2 and ASA4 (actually with the ASAs one would always be the primary so the VPN would always to the the primary external IP address, but you get the general idea), and the QinQ tunnel between SW2 and SW3 (simplified for this example).

Once you start allowing all the VLANs across the link between SW2 and SW3, the HSRP messages will also flow across the switches. In the switches you will see these kind of messages:
Event Description : .Oct 20 10:22:48:
%HSRP-4-DIFFVIP1: Vlan10 Grp 10 active routers virtual IP address 1.1.10.1
is different to the locally configured address 1.2.10.1
So even if the subnets are completely different HSRP will not mind at all.

HSRP does not care about IP addresses!

The fact is that HSRP really doesn't care what IP addresses you use.

Suppose that the first switch is set up as
SW1#conf t
SW1(config)#int vlan 10
SW1(config-if)#ip address 1.2.10.2 255.255.255.0
SW1(config-if)#standby 10 ip 10.2.10.1
SW1(config-if)#standby 10 priority 90
The second switch can be set up as follows:
SW2#conf t
SW2(config)#int vlan 10
SW2(config-if)#ip address 1.2.10.3 255.255.255.0
SW2(config-if)#standby 10 ip
SW2(config-if)#standby 10 priority 150
SW2(config-if)#standby 10 preempt delay minimum 60
If we do this (and its completely valid according to Cisco), then just so long as one switch has the ip address set, the other switch will learn the virtual interface to use.

It's all about the HSRP group number. Which explains why in the scenario above if the group numbers are the same on both sides, you will have issues.

When HSRP goes bad

Lets actually see this in action

We have a working HSRP set up between SW1 and SW2:

basic HSRP configuration

basic HSRP configuration

And also between SW3 and SW4

basic HSRP configuration

basic HSRP configuration

Above you can see that on SW4 I have omitted the IP address, but we can still see from the messages that HSRP works, and we can confirm it using "sh standby vlan 10":

sh standby hsrp vlan

Now we have two "sites" with working HSRP.

I will now configure the "QinQ" link (although as its not properly supported in GNS3 it is just a trunk link between the two sides).

I started by creating vlan 200, and then moving on to the trunk interfaces between SW2 and SW3. Firstly I shut down the interfaces before applying the configs and then doing a no shut on the interfaces.

The configs look like this:

configuring a trunk link

And we can see that the two sides are talking:

cisco ping

Now if we allow all VLANs over the trunk...

controlling trunk access cisco

HSRP error messages

HSRP BADVIP

Ooooh, "BADVIP" not good. Clearly we need to stop the HSRP messages from one side influencing the other side.

Lets shut down the link between SW2 and SW3, allow HSRP to recover, and figure out where to go from here.

So how do we fix this?

Blocking HSRP cross subnet traffic

We can try and stop the HSRP messages from going over our trunk, or at least from breaking the HSRP setup(s) in one of three ways:
  • Implement HSRP authentication
  • Change the HSRP group numbers on one side
  • Use an ACL

Implementing HSRP authentication

One option open to us is to use HSRP authentication.

On both switches we set up the following
SW1#conf t
SW1(config)#key chain hsrp-key
SW1(config-keychain)#key 1
SW1(config-keychain-key)#key-string 0 wtfbbq
SW1(config-keychain-key)#exit
SW1(config-keychain)#exit
SW1(config)#int vlan 10
SW1(config-if)#standby 10 authentication md5 key-chain hsrp-key
Now we can re-enable our SW2 to SW3 link and see what happens

Wireshark is seeing a lot of HSRP traffic:

Wireshark HSRP

And our switches are showing a lot of bad authentication attempts:

HSRP authentication error

But at least we don't have our VLANs getting all screwed up. 

This still isn't the optimal method, as the traffic is still being generated, and a lot of noise is being logged on our switches. So can we stop the noise on our switches, because if this is getting logged then every few seconds a log entry will be generated.

Re-numbering the standby groups

An obvious choice here is to re-number the HSRP group.

I have now changed SW3 and SW4 to use standby group 20 (instead of 10). This is just a case of entering the "no standby 10" and then setting up the HSRP group again with a new group number. This needs to be done on both switches.

Wireshark is still showing a lot of HSRP traffic, but SW1 and SW2 are not logging all the failed authentication attempts. So can we stop the traffic all together? And, showing my age, I really feel like using a picture from Airplane! after that question.


Blocking HSRP altogether
Stop HSRP traffic altogether

Using an Access Control List to block HSRP traffic

Wouldn't it be nice if we could block the multicast traffic completely across the link between the sites?

Well, what do we know about HSRP? Well, it comes in two versions, version 1 and version 2.
Version 1 uses the multicast address 224.0.0.2 and version 2 uses 224.0.0.102. Both versions have a source and destination port of 1985.

We also know that the MAC address used is 0000.0c07.ac0a.

If we try and block using the MAC address and a VACL then we could stop legitimate HSRP traffic between the switches on one site. So the solution needs to be IP based.

So we could use an ACL to block outgoing or incoming traffic to the multicast addresses used by HSRP, or block all traffic on UDP port 1985. An outgoing ACL would be the best option as it will also reduce traffic across the link.

We can see from a wireshark capture that we are using version 1 of HSRP as the multicast address is 224.0.0.2:

HSRP version 1 multicast address

The most obvious acl would be "access-list 101 deny udp any any eq 1985" and apply this to the trunk link between SW2 and SW3 outbound. Try as I might this did not work in GNS3. I had heard reports that HSRP didn't work in IOU, but it does (and you can read about how to get HSRP working on both GNS3 and IOU) so decided to give it a shot.

So anyway, the obvious acl didnt work, if I tried a direction of out I got the error:

"error ip acl configuration on 'out' direction is not supported"

And having it in an in direction didn't work as I could still see the HSRP traffic across the wire, and the HSRP was getting messages from one side interfering with the other side.

This is part if the trouble with emulated systems, it's difficult to get all the functionality you need, if you need functionality that's even slightly out of the ordinary, you either have a fight to get it to work, or it just plain won't work.

So after trying numerous ACLs and finding nothing worked I found the command "switchport block multicast" which again isn't supported by my GNS3 topology. This command has been around since at least 12.2, but apparently it's not available on GNS3, I then moved to IOU in the hope that the "switchport block multicast" command would be support, but it isn't there either. Also it looks like this is just for "unknown multicast" traffic, so probably wouldn't fit the bill anyway.

Once I was able to lab this up on real equipment (a 3750 and a 3550 for one side and a 3560 and 3550 for the other side) I was able to get some good progress. The downside of emulated equipment is that its harder (slightly) to get Wireshark sniffing going on - but that's where SPAN and RSPAN come into play. You can have a read of SPAN and RSPAN configuration.

The caveat I found was that this needs to be implemented on both sides of the link between sites.
access-list 101 deny tcp any eq 1985 host 224.0.0.2
access-list 101 deny udp any eq 1985 host 224.0.0.2
access-list 101 permit ip any any
If it's only implemented on one side only then although HSRP will recover quickly for that side, the other will still be accepting HSRP messages from the other side, and you'll see messages along the lines of:

*Mar  1 02:55:58.883: %HSRP-4-DIFFVIP1: Vlan10 Grp 10 active routers virtual IP address 10.20.1.1 is different to the locally configured address 10.10.1.1

Hence the ACL must exist on both sides, and be assigned to the link interface on the "in" direction. Sadly the out direction is not an option, but that would have been perfect for what we need. The messages still flow across the link, but are dropped according to the ACL, allowing for both sides to have identical HSRP group numbers. It's not a perfect solution as I would like to stop the traffic at source rather than destination, but it would appear to be the only option available.

Now there is the chance that the solution would actually work with emulated equipment, maybe it was down to software versions, but maybe something's just work better on real hardware (not surprising really!).

Which is the best option of the three methods?

Actually all of them, if you want a belt-and-braces approach. Just having the authentication is enough to stop the HSRP messages confusing the different sites, but will log a log of noise on the switches. Changing the standby group numbers will also help, it will certainly cut down on the noise generated by failed authentication methods, but still a lot of traffic would be crossing the link. The ACL allows us to have the same HSRP groups on each side, and (if we wanted to) the same authentication method. Due to not being able to specify the ACL in an out direction the traffic will still go across the link, but I guess that that's something I'll havr to live with.

How to implement HSRP on Cisco IOU and GNS3

Lots of people have said that HSRP does not work in either GNS3 or IOU. It does, but both softwares are very particular about what they require for it to work.

HSRP on Cisco IOU

For HSRP on Cisco IOU I have used the IOS image i86bi_linux_l2-ipbasek9-ms.jan24-2013-team_track.

To upload the image fire up IOU web and click on Manage.


Click on Manage IOSes


In the Upload IOS bit at the top of the page enter the filename - this must be the same as the file you are going to upload or else it won't work. The alias can be anything you want it to be. Then browse to the image from the Choose File button.


Your page should look like this:



How to add a Cisco IOU ios image

When you are ready click on Upload.


Once its uploaded your Manage IOSes bit of the page should look like this:


Managing Cisco IOU images

You can see our added IOS at the bottom.


We can now start with a basic lab, using a netmap of 1:0/1 2:0/1.


SW1 is configured as follows:


Cisco IOU HSRP configuration

SW2 is set up similarly (excuse the typing mistakes)


Cisco IOU HSRP Configuration

After a few seconds we can start to see HSRP messages coming through:



HSRP messages

And using the command "sh standby vlan 10" we can confirm that the HSRP is working as it should:



HSRP status

HSRP on GNS3


For HSRP on GNS3 I am using the image c3660-is-mz.124.25b.

Fire up GNS3 and go into Edit > IOS images and hypervisors, add your image for the c3660 (if you don't already have one, but your mileage may vary with other images), and set it as the default.


Add image to GNS3

Next go into Edit > Symbol Manager, find an image you like, personally I like the route_switch_processor, highlight it and press the arrow button. Change the type to Router c3600 and give it a good name, then click Apply, then click OK.


Make custom device GNS3


Now we can drag two of our new HSRP switches onto the canvas:


HSRP on GNS3

Now configure both the new switches and add an NM-16ESW to each:


HSRP on GNS3


Once you have done that you can cable them together and start them up, you must use the connections on the newly added module though - I have used 1/10 on each.

The configs are the same as above for IOU and are a very simple implementation:

HSRP configuration on GNS3

HSRP configuration on GNS3

Once you give the switches a few seconds once everything is plumbed in you should see the HSRP messages start to flow:

Working HSRP on GNS3

Working HSRP on GNS3

And there we have two ways to get HSRP working in a home environment without needing to purchase separate hardware.