Showing posts with label failover. Show all posts
Showing posts with label failover. Show all posts

CCIE Security lab: HQ ASAs - failover and stuff

Starting to make some headway now. In the previous post the LAN was set up within the HQ, and now we need to get the LAN talking to the WAN.

The HQ ASAs will be set up in a failover pair. We will also add a interface for vlan 1 to the switches, and set up a static route to the ASAs. Then we will set up NAT and see how far we can get in the topology.

Failover

First of all, let's set up LON-FW1:
hostname LON-FW1
int e0
ip add 163.4.4.254 255.255.255.0 standby 163.4.4.252
nameif outside
int e1
ip add 10.1.1.254 255.255.255.0 standby 10.1.1.252
nameif inside
int e3
ip add 10.1.55.1 255.255.255.0 standby 10.1.55.2
nameif fover
route outside 0 0 163.4.4.1
exit
failover link fover
failover interface ip fover 10.1.55.1 255.255.255.0 standby 10.1.55.2
failover lan unit pri
failover replication http
failover lan interface fover e3
failover key cisco
failover
We don't need to add much to LON-FW2, pretty much just the failover configuration. However... if you do add things, such as setting the hostname to LON-FW2, then it will be easier to see that failover is working correctly.
ciscoasa(config)# hostn LON-FW2
LON-FW2(config)# int e0
LON-FW2(config-if)# ip add 163.4.4.254 255.255.255.0 standby 163.4.4.252
LON-FW2(config-if)# nameif outside
INFO: Security level for "outside" set to 0 by default.
LON-FW2(config-if)# 
LON-FW2(config-if)# int e1
LON-FW2(config-if)# ip add 10.1.1.254 255.255.255.0 standby 10.1.1.252
LON-FW2(config-if)# nameif inside
INFO: Security level for "inside" set to 100 by default.
LON-FW2(config-if)# 
LON-FW2(config-if)# 
LON-FW2(config-if)# int e3
LON-FW2(config-if)# no sh
LON-FW2(config-if)# nameif fover
INFO: Security level for "fover" set to 0 by default.
LON-FW2(config-if)# exi
LON-FW2(config)# failover link fover
INFO: Non-failover interface config is cleared on Ethernet3 and its sub-interfaces
LON-FW2(config)# failover interface ip fover 10.1.55.1 255.255.255.0 standby 1.1.55.2
LON-FW2(config)# failover lan unit sec
LON-FW2(config)# failover replication http
LON-FW2(config)# failover key cisco
LON-FW2(config)# failover
LON-FW2(config)# failover lan interface fover e3
LON-FW2(config)# ..

        Detected an Active mate
Beginning configuration replication from mate.
ERROR: Password recovery was not changed, unable to access 
the configuration register.
Crashinfo is NOT enabled on Full Distribution Environment
End configuration replication from mate.

LON-FW1(config)# end
LON-FW1# sh interface ip brief 
Interface                  IP-Address      OK? Method Status                Protocol
Ethernet0                  163.4.4.252     YES CONFIG up                    up  
Ethernet1                  10.1.1.252      YES CONFIG up                    up  
Ethernet2                  unassigned      YES unset  administratively down up  
Ethernet3                  10.1.55.2       YES unset  up                    up  
LON-FW1#

A bit of housekeeping

A couple of housekeeping bits now. We need to get some routing from the switches to the firewalls:
SW1(config)#int vlan 1
SW1(config-if)#ip add 10.1.1.3 255.255.255.0
SW1(config-if)#standby 1 ip 10.1.1.1 
SW1(config-if)#standby 1 pre delay min 60
SW1(config-if)#standby 1 pri 110
SW1(config-if)#
SW1(config-if)#ip route 0.0.0.0 0.0.0.0 10.1.1.254

SW2(config)#int vlan 1
SW2(config-if)#ip add 10.1.1.2 255.255.255.0
SW2(config-if)#standby 1 ip 10.1.1.1 
SW2(config-if)#standby 1 pre delay min 60
SW2(config-if)#standby 1 pri 90
SW2(config-if)#no sh
SW2(config-if)#ip route 0.0.0.0 0.0.0.0 10.1.1.254
SW2(config)#
SW2(config)#int ran gi 1/1 - 3
SW2(config-if-range)#swi mo acc
SW2(config-if-range)#swi acc vl 4
SW2(config-if-range)#no sh
SW2(config-if-range)#

SW4(config)#int ra gi1/2 - 3
SW4(config-if-range)#swi mo acc
SW4(config-if-range)#swi acc vl 4
SW4(config-if-range)# 
We also need to set up the switch above the firewalls, and add a little something for the ASAs to talk to:
Switch(config)#ho LON-SW
LON-SW(config)#int ra gi 0/0 - 2
LON-SW(config-if-range)#no sh
LON-SW(config-if-range)#swi mo acc
LON-SW(config-if-range)#swi acc vl 1
LON-SW(config-if-range)#

LON1(config)#int lo1
LON1(config-if)#ip vrf for 802101
% Interface Loopback1 IPv4 disabled and address(es) removed due to disabling VRF 802101
LON1(config-if)#ip add 31.32.33.34 255.255.255.255
LON1(config-if)#

LON-FW1# ping outside 31.32.33.34
Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to 31.32.33.34, timeout is 2 seconds:
!!!!!
Success rate is 100 percent (5/5), round-trip min/avg/max = 1/2/10 ms
LON-FW1# 
I added this because the other devices are currently turned off, we don't really need them at the moment, but we do need to set up NAT on the firewalls, so that the LAN can talk to the WAN.

NAT

LON-FW1(config)# object-group network INSIDE-NAT-SUBNETS
LON-FW1(config-network-object-group)# network-object 10.1.1.0 255.255.255.0
LON-FW1(config-network-object-group)# network-object 10.1.4.0 255.255.255.0
LON-FW1(config-network-object-group)# network-object 10.1.9.0 255.255.255.0
LON-FW1(config-network-object-group)# network-object 10.1.20.0 255.255.255.0
LON-FW1(config-network-object-group)# network-object 10.1.21.0 255.255.255.0
LON-FW1(config-network-object-group)# network-object 10.1.99.0 255.255.255.0
LON-FW1(config-network-object-group)# exi
LON-FW1(config)# nat (inside,outside) after-auto source dynamic INSIDE-NAT-SUB$
LON-FW1(config)# route inside 10.1.0.0 255.255.0.0 10.1.1.1
LON-FW1(config)# 
LON-FW1(config)# access-list outside->in extended permit ip any any 
LON-FW1(config)# access-list outside->in extended permit icmp any any 
LON-FW1(config)# access-group outside->in in interface outside 
LON-FW1(config)# 
LON-FW1(config)# end
LON-FW1# sh run | i nat
LON-FW1# 
Note that I have not included the 10.1.55.0/24 network, really we don't even need this on the switches, and it's kinda messed with the whole IP addressing thing anyway. But there we go.

It is working:
ISE14/admin# ping 31.32.33.34
PING 31.32.33.34 (31.32.33.34) 56(84) bytes of data.
64 bytes from 31.32.33.34: icmp_seq=1 ttl=254 time=19.2 ms
64 bytes from 31.32.33.34: icmp_seq=2 ttl=254 time=20.4 ms
64 bytes from 31.32.33.34: icmp_seq=3 ttl=254 time=19.0 ms
64 bytes from 31.32.33.34: icmp_seq=4 ttl=254 time=14.4 ms

--- 31.32.33.34 ping statistics ---
4 packets transmitted, 4 received, 0% packet loss, time 3019ms
rtt min/avg/max/mdev = 14.425/18.296/20.445/2.300 ms

ISE14/admin#
This is from a Windows host in VLAN 4:
So the Firewalls as set up in Active/Standby HA, we have NAT working and our inside hosts can talk to devices beyond the firewalls.

The new switch has arrived, so I can start to play around with that. I'll need to set up UNL to use a separate NIC for this, and that will be in a later post. Till then, it's really starting to take shape!

Cisco ASA failover, redundant interfaces, Catalyst HSRP and power

Most networks have redundancy (or at least should do). This can be in the form of additional cables cross-connecting devices, dual power supplies going to different feeds and HSRP on switches. All of this is good stuff, but some things can easily be taken for granted.

Consider the following topology:

We have two ASAs in a failover pair (Outside interface 10.30.1.1 with standby ip 10.30.1.2), both connecting to a switch (10.30.1.254), that has a loopback interface assigned to it to simulate the internet (10.250.1.1). The ASAs also connect to a pair of switches, set up in an HSRP group (ASA inside interface 10.40.1.1, standby ip 10.40.1.10, switch vlan 40 ip 10.40.1.254). The topology looks like this:



It looks pretty redundant right? But let's work through the config and see what could go wrong. We'll start by setting up the ASAs in a failover pair.

Cisco ASA failover setup

On the first ASA we set up failover as follows:
ciscoasa(config)# enable password cisco
ciscoasa(config)# hostname ASA1
ASA1(config)# failover
ASA1(config)# failover lan unit primary
ASA1(config)# failover lan interface FAILOVER GigabitEthernet0/3
INFO: Non-failover interface config is cleared on GigabitEthernet0/3 and its sub-interfaces
ASA1(config)# failover key fail0ver
ASA1(config)# failover replication http
ASA1(config)# failover link FAILOVER GigabitEthernet0/3
ASA1(config)# failover interface ip FAILOVER 10.20.1.1 255.255.255.252 standby 10.20.1.1
ASA1(config)# int gi0/3
ASA1(config-if)# no shut
ASA1(config-if)# exit
ASA1(config)# exit
ASA1# copy run start
The second ASA is similar:
ciscoasa(config)# enable password cisco
ciscoasa(config)# hostname ASA2
ASA2(config)# failover
ASA2(config)# failover lan unit secondary
ASA2(config)# failover lan interface FAILOVER GigabitEthernet0/3
INFO: Non-failover interface config is cleared on GigabitEthernet0/3 and its sub-interfaces
ASA2(config)# failover key fail0ver
ASA2(config)# failover replication http
ASA2(config)# failover link FAILOVER GigabitEthernet0/3
ASA2(config)# failover interface ip FAILOVER 10.20.1.1 255.255.255.252 standby 10.20.1.2
ASA2(config)# int gi0/3
ASA2(config-if)# no shut
ASA2(config-if)#exit
ASA2(config)#exit
ASA2# copy run start
We can then save our configuration, so that the partner (or mate) firewall gets sent the active configuration:
ASA1# copy run start
ASA1# Beginning configuration replication: Sending to mate.
End Configuration Replication to mate
I do like to set the prompt to show me easily which firewall I am currently on:
ASA1#
ASA1# conf t
ASA1(config)# prompt hostname state
ASA1/act(config)# exit
ASA1/act# copy run start

Source filename [running-config]?
Cryptochecksum: 9f0c4260 95438d98 2d1439ef 0704bf77

2693 bytes copied in 3.310 secs (897 bytes/sec)
ASA1/act#
Our partner ASA will now change to show its status in the prompt:
ASA1#
ASA1/stby# Cryptochecksum: 0df25109 9f4fecb5 cebb9db5 93664d73

ASA1/stby#
2691 bytes copied in 3.230 secs (897 bytes/sec)
ASA1/stby#
Now we can configure our connection to the "internet" and to the inside network:

ASA Outside (internet) connection

ASA1/act# conf t
ASA1/act(config)# int Gi0/0
ASA1/act(config-if)# nameif OUTSIDE
INFO: Security level for "OUTSIDE" set to 0 by default.
ASA1/act(config-if)# ip address 10.30.1.1 255.255.255.0 standby 10.30.1.2
ASA1/act(config-if)# no shut
ASA1/act(config-if)#exit
ASA1/act(config)# exit
ASA1/act# copy run start

ASA Inside connection

ASA1/act(config-if)# int gi 0/1
ASA1/act(config-if)# nameif INSIDE
INFO: Security level for "INSIDE" set to 100 by default.
ASA1/act(config-if)# ip address 10.40.1.1 255.255.255.0 standby 10.40.1.10
ASA1/act(config-if)# no shut
ASA1/act(config-if)# exit
ASA1/act(config)# exit
ASA1/act# copy run start
Now we have connections on the inside and outside:

Switch HSRP setup

I have covered HSRP a few times now, so won't go into any details, but here is the configuration walkthrough:
Switch(config)#hostname SW1
SW1(config)#vlan 40
SW1(config-vlan)#exit
SW1(config)#int vlan 40
SW1(config-if)#ip address 10.40.1.2 255.255.255.0
00:06:08: %LINEPROTO-5-UPDOWN: Line protocol on Interface Vlan40, changed state to down
SW1(config-if)#standby 40 ip 10.40.1.254
SW1(config-if)#standby 40 pri 150
SW1(config-if)#
00:09:16: %LINEPROTO-5-UPDOWN: Line protocol on Interface FastEthernet1/0/2, changed state to down
SW1(config-if)#exit
00:09:19: %LINEPROTO-5-UPDOWN: Line protocol on Interface FastEthernet1/0/2, changed state to up
SW1(config)#int fastEthernet 1/0/2
SW1(config-if)#switchport trunk encap dot1q
SW1(config-if)#switchport mode trunk
00:09:49: %LINEPROTO-5-UPDOWN: Line protocol on Interface Vlan40, changed state to up
SW1(config-if)#
00:10:10: %HSRP-6-STATECHANGE: Vlan40 Grp 40 state Standby -> Active
SW1(config-if)#

Switch(config)#hostname SW2
SW2(config)#vlan 40
SW2(config-vlan)#exit
SW2(config)#int vlan 40
*Mar  1 00:06:56.997: %LINEPROTO-5-UPDOWN: Line protocol on Interface Vlan40, changed state to down
SW2(config-if)#ip address 10.40.1.3 255.255.255.0
SW2(config-if)#standby 40 ip 10.40.1.254
SW2(config-if)#standby pri 90
SW2(config-if)#exit
SW2(config)#int fa0/2
SW2(config-if)#switchport trunk encap dot1q
SW2(config-if)#switchport mode trunk
SW2(config-if)#
*Mar  1 00:09:22.447: %LINEPROTO-5-UPDOWN: Line protocol on Interface FastEthernet0/2, changed state to down
*Mar  1 00:09:25.517: %LINEPROTO-5-UPDOWN: Line protocol on Interface FastEthernet0/2, changed state to up
*Mar  1 00:09:55.515: %LINEPROTO-5-UPDOWN: Line protocol on Interface Vlan40, changed state to up
*Mar  1 00:10:36.561: %HSRP-5-STATECHANGE: Vlan40 Grp 40 state Speak -> Standby
SW2(config-if)#

Switch to ASA configuration

SW1(config)#int fa1/0/1
SW1(config-if)#switchport mode access
SW1(config-if)#switchport access vlan 40
SW1(config-if)#exit
SW1(config)#exit
SW1#ping 10.40.1.1
Type escape sequence to abort.

Sending 5, 100-byte ICMP Echos to 10.40.1.1, timeout is 2 seconds:
.!!!!
Success rate is 80 percent (4/5), round-trip min/avg/max = 1/1/1 ms

SW2(config)#int fa0/1
SW2(config-if)#switchport mode access
SW2(config-if)#switchport access vlan 40
SW2(config-if)#exit
SW2(config)#exit
SW2#
*Mar  1 01:09:09.651: %SYS-5-CONFIG_I: Configured from console by console
SW2#ping 10.40.1.1
Type escape sequence to abort.

Sending 5, 100-byte ICMP Echos to 10.40.1.1, timeout is 2 seconds:
!!!!!
Success rate is 100 percent (5/5), round-trip min/avg/max = 1/2/8 ms

ASAs to "Internet" Configuration

On the active ASA we need to set a route out to the internet:
ASA1/act# sh run | i route
route OUTSIDE 0.0.0.0 0.0.0.0 10.30.1.254 1

Internet configuration

The internet is very simple, we have a loopback that will be our end-point for testing, and a vlan for the ASAs to communicate with:
3550-SW3#sh run
interface Loopback0
  ip address 10.250.1.1 255.255.255.0
!
interface FastEthernet0/1
  switchport access vlan 10
  switchport mode access
!
interface FastEthernet0/11
  switchport access vlan 10
  switchport mode access
!
interface Vlan10
  ip address 10.30.1.254 255.255.255.0

NAT statements to allow access

To allow the inside to talk to the internet we need to set up NATing on the ASAs:
global (OUTSIDE) 1 interface
nat (INSIDE) 1 10.40.1.0 255.255.255.0

Testing

Let's see how out connectivity looks:
ASA1/act# ping 10.250.1.1

Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to 10.250.1.1, timeout is 2 seconds:
!!!!!
Success rate is 100 percent (5/5), round-trip min/avg/max = 1/2/10 ms

ASA1/stby> ping 10.250.1.1

Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to 10.250.1.1, timeout is 2 seconds:
!!!!!
Success rate is 100 percent (5/5), round-trip min/avg/max = 1/2/10 ms

SW1#ping 10.250.1.1

Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to 10.250.1.1, timeout is 2 seconds:
!!!!!
Success rate is 100 percent (5/5), round-trip min/avg/max = 1/2/8 ms

SW2#ping 10.250.1.1

Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to 10.250.1.1, timeout is 2 seconds:
!!!!!
Success rate is 100 percent (5/5), round-trip min/avg/max = 1/2/8 ms
SW2#
So far so good!

Forcing failover and simulating outages 

We can simulate issues in a number of ways. I started by removing the power to SW1. This made SW2 go active for vlan 40. SW2 has a good connection to ASA2 so pings were 100% successful.

With ASA1 as the standby we can force a failover from ASA2:
ASA1/stby# failover active

    Switching to Active

SW2#ping 10.250.1.1

Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to 10.250.1.1, timeout is 2 seconds:
....!
Success rate is 20 percent (1/5), round-trip min/avg/max = 1/1/1 ms

ASA1/act# Waiting for the earlier webvpn instance to terminate...
Previous instance shut down. Starting a new one.

    Switching to Standby
At this point the firewalls failed back, due to the traffic generated by SW2, so that ASA2 became the active mate again. If ASA2 had the power removed then they would not be able to fail back, and therefore SW2 would have no way to route the traffic over to SW1 (which would be the active firewall), and therefore the inside of the network would lose all internet communication.

So we can survive one ASA power outage and one switch outage (ASA1 and SW1 or ASA2 and SW2) but not if we lose SW2 and ASA1 or SW1 and ASA2.

So how can we get better resiliency? We can do this with a cable from SW1 to ASA2 and from SW2 to ASA1. This does require some reconfiguration on the ASAs, but how can we do this? We certainly can't have another physical interface on the ASAs with the same IP range, as we would get an error informing us of an IP overlap. They key work here is physical interface. We can however set up a different type of interface to achieve this.

Cisco ASA Redundant interfaces

Redundant interfaces are two or more interfaces joined together in a redundancy group. They act like physical interfaces, in that they have an IP address, but with the added bonus of having multiple links.

We start by adding another interface on the switches (fa1/0/3 on SW1 and fa0/3 on SW3) to be access ports on vlan 40, and these interfaces are connected to GigabitEthernet 0/2 on the "other" ASA:



Next we move onto the firewalls and remove the IP information from the existing inside interfaces:
ASA1/act(config)# int GigabitEthernet0/1
ASA1/act(config-if)# no nameif
ASA1/act(config-if)# no ip address
ASA1/act(config-if)# exit
Then we can create a redundant port:
ASA1/act(config)# interface Redundant1
ASA1/act(config-if)# member-interface GigabitEthernet0/1
INFO: security-level and IP address are cleared on GigabitEthernet0/1.
ASA1/act(config-if)# member-interface GigabitEthernet0/2
INFO: security-level and IP address are cleared on GigabitEthernet0/2.
ASA1/act(config-if)# nameif INSIDE
INFO: Security level for "INSIDE" set to 100 by default.
ASA1/act(config-if)# ip address 10.40.1.1 255.255.255.0 standby 10.40.1.10
ASA1/act(config-if)#
I could have saved a bit of typing as adding a member interface to a redundant interface will clear the security-level and IP address automatically, but its useful to do manually.

Once we have saved our config we can check on the standby ASA:
ASA1/stby# sh run
: Saved
<Truncated>
interface GigabitEthernet0/1
  no nameif
  no security-level
  no ip address
!
interface GigabitEthernet0/2
  no nameif
  no security-level
  no ip address
!
interface Redundant1
  member-interface GigabitEthernet0/1
  member-interface GigabitEthernet0/2
  nameif INSIDE
  security-level 100
  ip address 10.40.1.1 255.255.255.0 standby 10.40.1.10
!
At this point we cant ping the loopback of 10.250.1.1 on the "internet", and to do this we need to do some NATing on the ASA, because when we clear the interface on the ASA it removes our NAT statement:
ASA1/act(config)# nat (INSIDE) 1 10.40.1.0 255.255.255.0
Once that is done we can ping the loopback again:
SW1#ping 10.250.1.1

Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to 10.250.1.1, timeout is 2 seconds:
!!!!!
Success rate is 100 percent (5/5), round-trip min/avg/max = 1/2/9 ms

SW2#ping 10.250.1.1

Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to 10.250.1.1, timeout is 2 seconds:
!!!!!
Success rate is 100 percent (5/5), round-trip min/avg/max = 1/2/8 ms
SW2#

Testing the redundant interface with failover

Now if we failover the ASAs (ASA1 being the primary, so testing from SW1) do we lose any pings?
SW1#ping 10.250.1.1 repeat 4000

Type escape sequence to abort.
Sending 4000, 100-byte ICMP Echos to 10.250.1.1, timeout is 2 seconds:
!!!!!
<omitted>
!!!!!
Success rate is 100 percent (4000/4000), round-trip min/avg/max = 1/2/9 ms
No we dont.

If we remove SW1 (simulating a power outage) does SW2 lose any pings?
SW2#ping 10.250.1.1 repeat 4000

Type escape sequence to abort.
Sending 4000, 100-byte ICMP Echos to 10.250.1.1, timeout is 2 seconds:
<omitted>
*Mar  5 02:12:17.681: %LINEPROTO-5-UPDOWN: Line protocol on Interface FastEthernet0/2, changed state to down
<omitted>
*Mar  5 02:12:18.746: %LINK-3-UPDOWN: Interface FastEthernet0/2, changed state to down
Success rate is 100 percent (4000/4000), round-trip min/avg/max = 1/3/34 ms
Again no ping loss. And if we turn off ASA2 (now the primary):
SW2#ping 10.250.1.1 repeat 4000

Type escape sequence to abort.
Sending 4000, 100-byte ICMP Echos to 10.250.1.1, timeout is 2 seconds:
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!.
*Mar  5 02:13:49.133: %LINEPROTO-5-UPDOWN: Line protocol on Interface FastEthernet0/1, changed state to down
*Mar  5 02:13:50.140: %LINK-3-UPDOWN: Interface FastEthernet0/1, changed state to down.......!!!!!!!!!!!!!!!!!!!!!!!
<omitted>
Success rate is 99 percent (3992/4000), round-trip min/avg/max = 1/3/17 ms
Then yes we do lose a few pings, but only in the time it takes ASA 1 to become the active again. But losing a couple of pings is better than losing all communication! Now we can lose one switch and one ASA on different sides of the topology and still have a working (redundant) network.