MPLS: The big blackhole question

Whilst I don't get many emails from readers of my books saying "I don't understand such-and-such", I have had two referring to the same thing. So I feel that this warrants a deeper explanation.

In section 9.3 of the MPLS for Cisco Networks book we cover Internet access, supplied as a dedicated VRF.

This caused some issues with the network though, namely that R1 and R5 could reach each other, due to having default routes pointing to the IGW router.

The fix itself was a simple two-line affair, but the reasoning behind the fix is not as simple.

So, let's go through a little of the configurations, starting with the IGW router:

IGW#sh run
hostname IGW
!
interface Loopback0
 ip address 4.4.4.4 255.255.255.255
!
interface Loopback1
 ip address 5.5.5.5 255.255.255.255
!
interface Ethernet0/0
 ip address 10.20.1.1 255.255.255.0
!
router bgp 200
 bgp log-neighbor-changes
 network 0.0.0.0
 neighbor 10.20.1.2 remote-as 100
!
ip route 0.0.0.0 0.0.0.0 Null0
!
end

IGW#

Now R1:

R1#sh run
hostname R1
!
interface Loopback0
 ip address 192.168.1.1 255.255.255.0
 ip ospf network point-to-point
!
interface Ethernet0/0
 ip address 10.1.1.2 255.255.255.0
!
router bgp 65001
 bgp log-neighbor-changes
 network 10.1.1.0 mask 255.255.255.0
 network 192.168.1.0
 neighbor 10.1.1.1 remote-as 100
!
ip forward-protocol nd
!
end

R1#

R5:

R5#sh run
hostname R5
!
interface Loopback0
 ip address 192.168.5.1 255.255.255.0
 ip ospf network point-to-point
 ip ospf 3 area 0
!         
interface Ethernet0/0
 ip address 10.1.5.2 255.255.255.0
 ip ospf 3 area 0
!
router ospf 3
!
end
          
R5#

And PE1, where all the magic happens:

PE1(config-router)#do sh run
hostname PE1
!
ip vrf BLUE
 rd 100:3
 export map VRF-BLUE-SUBNETS
 route-target export 100:3
 route-target import 100:3
 route-target import 100:200
!
ip vrf INTERNET
 rd 100:200
 route-target export 100:200
 route-target import 100:201
 route-target import 100:205
!         
ip vrf RED
 rd 100:1
 export map VRF-RED-INTERNET
 route-target export 100:1
 route-target import 100:1
 route-target import 100:200
!
interface Loopback0
 ip address 10.250.3.1 255.255.255.255
!
interface Ethernet0/2
 ip vrf forwarding INTERNET
 ip address 10.20.1.2 255.255.255.0
!
interface Ethernet1/0
 ip vrf forwarding RED
 ip address 10.1.1.1 255.255.255.0
 ip ospf 1 area 0
!
interface Ethernet1/2
 ip vrf forwarding BLUE
 ip address 10.1.5.1 255.255.255.0
 ip ospf 3 area 0
!
router ospf 3 vrf BLUE
 router-id 10.250.3.5
 redistribute bgp 100 subnets
 default-information originate
!
router bgp 100
 bgp log-neighbor-changes
 !
 address-family ipv4 vrf BLUE
  redistribute ospf 3
 exit-address-family
 !
 address-family ipv4 vrf INTERNET
  neighbor 10.20.1.1 remote-as 200
  neighbor 10.20.1.1 activate
 exit-address-family
 !
 address-family ipv4 vrf RED
  neighbor 10.1.1.2 remote-as 65001
  neighbor 10.1.1.2 activate
 exit-address-family
!
ip access-list standard VRF_BLUE_SUBNETS
 permit 10.1.5.0 0.0.0.255
 permit 192.168.5.0 0.0.0.255
ip access-list standard VRF_RED_SUBNETS
 permit 10.1.1.0 0.0.0.255
 permit 192.168.1.0 0.0.0.255
!
route-map VRF-RED-INTERNET permit 10
 match ip address VRF_RED_SUBNETS
 set extcommunity rt 100:201 additive
!
route-map VRF-BLUE-SUBNETS permit 10
 match ip address VRF_BLUE_SUBNETS
 set extcommunity rt 100:205 additive
!
end

PE1(config-router)#

With this configuration, R1 and R5 can reach the Internet address (4.4.4.4), which is desired, but also each other, which is not desired:

R1#ping 4.4.4.4
Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to 4.4.4.4, timeout is 2 seconds:
!!!!!
Success rate is 100 percent (5/5), round-trip min/avg/max = 1/1/1 ms
R1#
R1#ping 192.168.5.1
Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to 192.168.5.1, timeout is 2 seconds:
!!!!!
Success rate is 100 percent (5/5), round-trip min/avg/max = 1/1/1 ms
R1#

R5#ping 192.168.1.1
Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to 192.168.1.1, timeout is 2 seconds:
!!!!!
Success rate is 100 percent (5/5), round-trip min/avg/max = 4/4/5 ms
R5#ping 4.4.4.4
Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to 4.4.4.4, timeout is 2 seconds:
!!!!!
Success rate is 100 percent (5/5), round-trip min/avg/max = 4/4/5 ms
R5#

In the book we go through a couple of attempts at blocking the behaviour, until landing on one that works:

IGW(config)#access-list 1 perm 0.0.0.0
IGW(config)#int e0/0
IGW(config-if)#ip access-group 1 out
IGW(config-if)#end
IGW#sh access-list
Standard IP access list 1
    10 permit 0.0.0.0
    20 deny   any log (20 matches)
IGW#

This works. But why, as let's face it it does not look like it should.

Before we get into the explanation, if we tack on some logging to the ACLs Default Deny rule, we can see it in action:

IGW(config)#access-list 1 deny any log
IGW(config)#
%SEC-6-IPACCESSLOGNP: list 1 denied 0 10.1.1.2 -> 192.168.5.1, 1 packet  
IGW(config)#

So by turning the implicit default deny into an explicit default deny, we can use logging, and once we try the ping again, we can see that it is this part of the rule that gets the hits.

The access-list itself permits the address 0.0.0.0, which is any host. So really, it should let it pass. Let's start by clearing up one thing, and by doing so, making life a little easier. This is not an MPLS thing. It's purely an access-list thing. We can test this out with three routers - the configs are below:

vIOS1(config)#int lo0
vIOS1(config-if)#ip add 1.1.1.1 255.255.255.255
vIOS1(config-if)#int gi0/0
vIOS1(config-if)#ip add 10.1.1.1 255.255.255.0
vIOS1(config-if)#no shut
vIOS1(config-if)#ip route 0.0.0.0 0.0.0.0 10.1.1.2
vIOS1(config)#

vIOS2(config)#int gi0/0
vIOS2(config-if)#ip add 10.1.1.2 255.255.255.0
vIOS2(config-if)#no sh
vIOS2(config-if)#ip route 1.1.1.1 255.255.255.255 10.1.1.1
vIOS2(config)#int gi 0/1
vIOS2(config-if)#ip add 20.2.2.2 255.255.255.0
vIOS2(config-if)#no shu
vIOS2(config-if)#ip route 3.3.3.3 255.255.255.255 20.2.2.3 
vIOS2(config)#

vIOS3(config)#int lo3
vIOS3(config-if)#ip add 3.3.3.3 255.255.255.255
vIOS3(config-if)#int gi0/0
vIOS3(config-if)#ip add 20.2.2.3 255.255.255.0
vIOS3(config-if)#no shut
vIOS3(config-if)#ip route  0.0.0.0 0.0.0.0 20.2.2.2  
vIOS3(config)#

We have end to end connectivity, because of the static routing:

vIOS3#ping 1.1.1.1
Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to 1.1.1.1, timeout is 2 seconds:
!!!!!
Success rate is 100 percent (5/5), round-trip min/avg/max = 3/4/6 ms
vIOS3#ping 1.1.1.1 source lo3
Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to 1.1.1.1, timeout is 2 seconds:
Packet sent with a source address of 3.3.3.3 
!!!!!
Success rate is 100 percent (5/5), round-trip min/avg/max = 4/4/6 ms
vIOS3#

Now let's assume that we don't want vIOS2 to be a transit router. We can use the same access-list as above to do this:

vIOS2(config)#int lo0                 
vIOS2(config-if)#ip add 2.2.2.2 255.255.255.255
vIOS2(config-if)#
vIOS2(config)#ip access-list standard 1 
vIOS2(config-std-nacl)#perm 0.0.0.0
vIOS2(config-std-nacl)#deny any log 
vIOS2(config-std-nacl)#int gi0/0
vIOS2(config-if)#ip access-group 1 out
vIOS2(config-if)#

vIOS1(config)#do ping 3.3.3.3
Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to 3.3.3.3, timeout is 2 seconds:
.....
Success rate is 0 percent (0/5)
vIOS1(config)#

vIOS2(config-if)#
%SEC-6-IPACCESSLOGNP: list 1 denied 0 3.3.3.3 -> 10.1.1.1, 1 packet  
vIOS2(config-if)#

vIOS1(config)#do ping 2.2.2.2
Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to 2.2.2.2, timeout is 2 seconds:
!!!!!
Success rate is 100 percent (5/5), round-trip min/avg/max = 3/3/4 ms
vIOS1(config)#

So, whilst we can still get to the IP addresses configured on vIOS2 (and IGW, returning to the book), we are no longer letting the router be a transit between the other two routers.

Weird, right?

Why permit 0.0.0.0 works

Without a mask on the access-list we are specifying a host. We could have used "ip access-list 1 permit 3.3.3.3", and this would allow us to get to the 3.3.3.3 IP address. However, we do not have an 0.0.0.0 IP address, therefore the line is not matched, and we jump down to the next rule, which is an implicit deny rule. Once we make this explicit, and enable logging, we can see that this is the one getting the hits.

Because of the direction of the rule (out) we are still able to reach the addresses local to the router (IGW from the book, or vIOS2 here), but we prevent the router being used as a transit.