DAMN YOU ACS!!!!

My lab exam is on Friday, so I have a few hours today, a few tomorrow and then Monday to Thursday, it's a mere five days to get stuff right and it isn't.

ACS is kicking my arse and I can't figure out why. I have had this working before, but I have spent two days trying to get this to work, and it's two days that I cannot afford to spend fucking about.

I am consoling myself by listening to some angry hip-hop:

My musical taste has not really moved on since the 90's. But I digress. Hopefully, by blogging about it, it might give me a chance to figure out why it is not working - or someone else might point out my problem (with ACS, not any of my other problems)...

We have a location:

We have a device type:

We have two devices. Originally I was just using SW6 (a router image with a switch module), thinking it might be a version issue I added R6:

We have two identity groups (Admins and Users):

We have two users (Admin1 and User1). These are in the groups above:

We have two Shell profiles, one for Admins, one for Users.

The User one gets a privilege of 8.

We have some allowed commands:

We have a policy called Telnet:

Telnet looks for a device type of Switch, and a location of Inside. It matches the Tacacs protocol, and internal users:

The Authorization part of Telnet set up the mapping Users should get the User-Shell and the User-Commands.

But when I try, I do not get the desired result. We do not match a permit rule, when logging in as User1:

As the next two grabs show, we should get the User-Commands command set, we are found in the Users Identity group, the service selection rule matches Telnet, as does the Identity policy.

We have a hit on the Telnet-Users - so we *should* get the shell and command set matching this.

But we don't. We pass authentication, but not authorization:

From the device:

R6#sh priv
Command authorization failed.

R6#sh ver
Command authorization failed.

R6#conf t
      ^
% Invalid input detected at '^' marker.

R6#en
% Error in authentication.

R6#

Admin works fine, though:

So, clearly, something is different between the User and Admin configurations.

The Admin-Commands has the tick next to "Permit any command that is not listed in the table below:

If this is unticked, commands that are permitted actually fail:

R6#sh ver
Command authorization failed.

R6#sh ver
Cisco IOS Software, IOSv Software (VIOS-ADVENTERPRISEK9-M), Version 15.5(2)T, RELEASE SOFTWARE (fc1)
Technical Support: http://www.cisco.com/techsupport
Copyright (c) 1986-2015 by Cisco Systems, Inc.
Compiled Wed 25-Mar-15 13:34 by prod_rel_team


ROM: Bootstrap program is IOSv

R6 uptime is 2 hours, 7 minutes
System returned to ROM by reload
System image file is "flash0:/vios-adventerprisek9-m"
Last reload reason: Unknown reason



This product contains cryptographic features and is subject to United
States and local country laws governing import, export, transfer and
use. Delivery of Cisco cryptographic products does not imply
third-party authority to import, export, distribute or use encryption.
Importers, exporters, distributors and users are responsible for
compliance with U.S. and local country laws. By using this product you
agree to comply with applicable laws and regulations. If you are unable
          
R6#

You can see that when unticked the command fails and works when ticked. So, actually, there is no difference in working/not working when comparing the User setup to the Admin setup. Something is just wrong, somewhere.

Here is the R6 config:

R6#sh run | i aaa
aaa new-model
aaa authentication login Admin group tacacs+ local
aaa authorization config-commands
aaa authorization exec Admin group tacacs+ local 
aaa authorization commands 1 Admin group tacacs+ 
aaa authorization commands 8 Admin group tacacs+ 
aaa authorization commands 15 Admin group tacacs+ 
aaa session-id common
R6#sh run | s line vty
line vty 0 4
 password cisco
 authorization commands 1 Admin
 authorization commands 8 Admin
 authorization commands 15 Admin
 authorization exec Admin
 login authentication Admin
 transport input telnet
R6#sh run | i tacacs
aaa authentication login Admin group tacacs+ local
aaa authorization exec Admin group tacacs+ local 
aaa authorization commands 1 Admin group tacacs+ 
aaa authorization commands 8 Admin group tacacs+ 
aaa authorization commands 15 Admin group tacacs+ 
tacacs-server host 192.168.20.102 key cisco
tacacs-server directed-request
R6#

I added a few changes:

R6(config)#aaa authentication login CONSOLE none
R6(config)#aaa authorization console 
R6(config)#line con 0
R6(config-line)#login authentication CONSOLE
R6(config-line)#exi
R6(config)#aaa accounting exec Admin start-stop group tacacs+ 
R6(config)#aaa accounting commands 8 Admin start-stop group tacacs+ 
R6(config)#aaa accounting commands 15 Admin start-stop group tacacs+ 
R6(config)#aaa accounting commands 1 Admin start-stop group tacacs+  
R6(config)#line vty 0 4
R6(config-line)#accounting commands 1 Admin
R6(config-line)#accounting commands 8 Admin
R6(config-line)#accounting commands 15 Admin
R6(config-line)#

But the problem remains the same.

I now have four days left before the lab. not feeling too good about it, to be honest!

But I need to move on, give myself a break from ACS and come back to it another day, not that I have many days remaining.

EDIT: 28/09/2016

I now have it working. I created a new access policy (Telnet-2), but on the Identity I set the conditions to be:

NDG:Location = in All Locations:Inside
NDG:Device Type = in All Device Types:Inside
Device IP Address = 150.1.7.66

The authorization part remained the same (albeit with the Command Sets result being first).

This gave me the desired result.

I think the issue with the first try was that the protocol condition would not be correct, the connection would be Telnet.

Got there in the end.

One day 19 hours to go before D-Day...

802101 is moving to Wordpress

Instead of using the last few days before my CCIE Security lab exam wisely, I have decided to move the site to the Wordpress platform.

Blogger has been great, I just feel it's time to take the next step up.

This will take some time, but expect some changes over the next week or so.

The drivers for this are that I will be able to do much more with the site, such as easier hosting of downloadable PDFs and cool things like that.

Anyway... I really should do some studying, but I thought I'd let you know in case the site goes offline for any amount of time.

CCIE by the numbers

As the CCIE Hall of Fame has recently been updated, I thought it would be fun to get a bit of a breakdown of the current state of play and see what the trends are.

It's not a perfect list, the numbers are only as good as the data, and not everyone is listed on the website, but the percentages will probably be about right. I would also like to thank Marc La Porte for maintaining the HOF and I am sure that it is no easy task!

According to Brad Reese, around 23% are not active or suspended. this includes very notable and revered engineers such as Ivan Pepelnjak. So while the numbers below are not perfect, they do give a decent view of the CCIE as it breaks down.

There are 17434 CCIEs listed on the website, and the current numbers are in the 53000's (or above), so clearly a big gap between those on the HOF, and how many there are, but again, if we look at the percentages, rather than then number, it'll probably be pretty close (unless the missing 40,000 all have the Data Center certification or something like that).

1868 were listed as "not verifiable", this is around 10% and accounts for the discrepancies.

Double and triple trouble...

So, of the 17434 CCIEs on the HOF, how many have more than one CCIE?

# of certifications	Number	Percentage
1	13373	76.71
2	1636	9.38
3	411	2.36
4	84	0.48
5	33	0.19
6	13	0.07
7	13	0.07
8	2	0.01

Just under one-quarter of CCIEs hold more than one certification.

Popularity

What are the most popular CCIE tracks?

Track	Number	Percentage
Routing and Switching	11722	67.24
Service Provider	1872	10.74
Security	1864	10.69
Voice	1112	6.38
Data Center	893	5.12
Collaboration	590	3.38
Wireless	188	1.08
Storage	121	0.69

Then we have older tracks, such as WAN Switching (286 / 1.64%) and ISP Dial (108 / 0.62%).

It is hardly surprising that R&S is the more popular. Service Provider and Security are about the same at just under 11% each. It is quite surprising that the others are so low, though, but then this is usually the case with specialist areas.

Anyway, hope this was mildly interesting... I am back off to my CCIE Security notes.

30 days to go.

It's now just 30 days till the CCIE Security lab exam. Nerves are starting to kick in, and I did not get much time to study whilst sunning myself in the glorious South of France sun, but a break was most definitely needed.

I did do some studying, though, watched a few of the INE videos on VPNs and would highly recommend INE to anyone following the same path.

What to do now, in order to get lab-ready over the next month?

I will start by listing the areas from strongest to weakest:

• ASAs (basic configs, NAT)
• IPS (it's pretty intuitive)
• DMVPN
• Routing
• Services (NTP/DHCP)
• Other VPNs
• WSA
• ISE/ACS

ISE and ACS come bottom of the list, and so this is where I need to focus my efforts for the majority of the time remaining.

The joy of it is that I do find both of these topics interesting, so that makes it easier to concentrate on them. Brian McGahan's videos will certainly help, and I have also started to read the CCNP Security SISAS 300-208 Official Cert Guide, which although it's for the CCNP Security, should certainly help things.

It is truly amazing how quickly the time goes, I am not sure (at this stage) whether I will be fully prepared when the day comes, but I can go in and give it a damn good try (and hopefully squeeze in a re-sit in December if required and seats are available). 

I probably won't blog much over the next month, so I wish you all the best and will catch up soon.

A CCIE Security engineer walks into a bar, things get weird

This might get a bit weird, and you can blame me for watching too much of The Might Boosh, but bear with it.

Picture the scene.

You walk into a bar, with a friend who you refer to as "silly knickers".

At the first table, you point to your friend, and can see a pissed-up architect, trying to draw whilst wearing cashmere mittens. What an idiot.

You look at the second table, and point to your friend again, as there are two pissed-up architects, both trying to draw whilst wearing cashmere mittens. Idiots,

Getting a drink is Easy. You politely point at the bar and grab an alcoholic apple juice. Then you head to the bathroom and pee in private. You fancy a vanilla tequila.

Back at the bar, the barman is also a fitness instructor. He's flexing happily. Two authors walk in, they politely propose a policy of only drinking alcoholic apple juice, in order to keep bar profits set high. You prefer a vanilla tequila.

One of the authors is actually a client of the barman, so he does not point and laugh. Instead, he says he'd like the same, but instead of vanilla tequila, he'd like a cup of tea in two cups.

The authors get served.

The next to get served is a policeman with a key. He's carrying Optimus Prime under one arm, and Bumblebee under the other. He has another key, which is huge, it's a really strong key. He orders a cucumber and a lemonade and nods to the group.

As is a bar policy, anyone with a key gets to make the rules. Before you know it, everyone is wearing cashmere mittens. Idiots!

Ok, story time over. All good stories have meanings, so what's this one all about (if you haven't worked it out from the clues)? I am sure lots of readers are thinking.. WTF?

Well, this is to try and remember VPN setups.

I made the VPNs cheatsheet a week or two ago, and this is good for showing where things fit in with each other, but I was still forgetting the steps.

I tried mnemonics, but they just came out as unrelated words, so decided to turn it into a story, with enough information to remember all the steps.

Let's break it down.

The first table is IKEv1.

I = ISAMKP
Point = Policy
To = Transform
Silly = Set
Knickers = Keyring
If = ISAMKP
Pissed = Profile
Architect = ACL
Cashmere = Crypto
Mittens = MAP
Idiot = Interface

Table 2 is IKEv2

I = ISAMKP
Point = Policy
To = Transform
Silly = Set
Knickers = Keyring
If = ISAMKP
2
Pissed = Profile
Architect = ACL
Cashmere = Crypto
Mittens = MAP
Idiots = Interface

A little harder, but the 2 signifies IKEv2 commands, and we need four of them (proposal, policy, keyring and profile). Each starts with "crypto ikev2", so we can use the context-sensitive help.

Easy VPN is next (its EASY to get served...)

I = ISAKMP
Politely = Policy
Point = Pool
Grab = Group
An Alcoholic Apple (juice) = AAA 
I = ISAKMP
Pee = Profile
In = IPSec 
Private = Profile
Vanilla = Virtual
Tequila = Template

I have left the client-side out, as that's pretty easy (create the "crypto ipsec client ez group", and assign the outside and inside interfaces)

Then comes FlexVPN Server (our flexing barman)

Point = Pool
And = Access
Laugh = List
2 Authors = IKEv2 Authorization
Politely = Policy
Propose = Proposal
Policy = Policy
Alcoholic Apple = AAA
2 Keep = Ikev2 Keyring
Profiles = IKEv2 Profile
Set = Transform Set
I = IPSec 
Prefer = Profile
Vanilla = Virtual
Tequila = Template

A server is no good without a client And this is much the same. Here the author wants the same as the server, without the pointing and laughing, but he does not want the vanilla tequila (virtual template), and orders:

Tea = Tunnel
In = Interface
2 = IKEv2
Cups = Client

The policeman is getting served next, which brings us onto GETVPN.

Is A = ISAMKP
Policeman = Policy
Key = ISAKMP key
Transformers = Transform Set
Really Strong Key = RSA key
A Cucumber and Lemonade = ACL
(nods to the) Group = GDOI Group

I have left out the IPSec profile from the server. I could not think of anything to fit, and the IOS will actually complain (I think) if this is missing, so it should be easy to figure out the missing bit(s).

Finally, we have the GETVPN client:

(As) Is A = ISAKMP
(bar) policy = Policy
Key = ISAKMP Key
Group = GDOI group
Cashmere = Crypto
Mittens = Map
Idiots = Interface

Trying to keep things like crypto map (cashmere mittens), interface (idiot/s) and virtual template (vanilla tequila) the same across the story, as it makes it (slightly) easier to remember. It's a weird story, but with enough repetition, and picturing yourself in the bar, it should aid memory.

CCIE:Sec: Day 12 - Flex VPN.

I did not manage to get any studying done over the weekend, so need to make up for it, that said, it was a good weekend. The annual Thai/Prague barbecue, where some of my wife's friends come over, and we eat some good food, have some good conversation and drink.

And I certainly did drink. Being the suave and cool-cat kind of guy I decided that (at about 10pm) bed would be the best option for me. My wife found me asleep on the bathroom floor at about midnight, whereupon I muttered something about having to "do the chicken", and then she put me to bed.

Yup. Stay classy Stuart.

Anyway, I am picking up from last week and finishing off FlexVPN. To be honest I am having a hard time remembering all the required components of all the different VPNs, so created a VPNs Cheatsheet for each of the IOS VPNs. You can download it, I'll be doing more later on as well. It definitely came in handy today. But before we get to today, let's go back to last week - cue flashback wobbly screen and weird music...

Sometime last week...

Time to set up FlexVPN between Telnet-2 (Flex VPN client) and Telnet-1:

Flex VPN Client configuration

Telnet-2(config)#crypto ikev2 client flexvpn FLEX-VPN 
Telnet-2(config-ikev2-flexvpn)#peer 1 1.1.1.1 
Telnet-2(config-ikev2-flexvpn)#client connect tunnel 1
Telnet-2(config-ikev2-flexvpn)#exit          
Telnet-2(config)#int tunnel 1
Telnet-2(config-if)#tunnel destinatio 1.1.1.1
Telnet-2(config-if)#tun source loop 0
Telnet-2(config-if)#tunnel mode ipsec ipv4
Telnet-2(config-if)#tunnel protectio ipsec profile Flex-Protect
Profile Flex-Protect is not defined.
Telnet-2(config-if)#
Telnet-2(config-if)#exit
Telnet-2(config)#cryp ipsec profile Flex-Profile
Telnet-2(ipsec-profile)#set transform-set 3des
%ERROR: transform set with tag "3des" does not exist.

Telnet-2(ipsec-profile)#exit
Telnet-2(config)#cry ipsec transform-set 3des esp-3des esp-sha-hmac
Telnet-2(cfg-crypto-trans)#exit
Telnet-2(config)#cryp ipsec profile Flex-Profile                   
Telnet-2(ipsec-profile)#set transform-set 3des         
Telnet-2(ipsec-profile)#int tunnel 1                   
Telnet-2(config-if)#tunnel protectio ipsec profile Flex-Profile
Telnet-2(config-if)#
%CRYPTO-6-ISAKMP_ON_OFF: ISAKMP is ON
Telnet-2(config-if)#ip add nego  
Telnet-2(config-if)#do sh run | s crypto
crypto ikev2 client flexvpn FLEX-VPN
  peer 1 1.1.1.1
crypto ipsec transform-set 3des esp-3des esp-sha-hmac 
 mode tunnel
crypto ipsec profile Flex-Profile
 set transform-set 3des 
Telnet-2(config-if)#exi                 
Telnet-2(config)#aa new-model 
Telnet-2(config)#aaa authorization network AuthZ-list local
Telnet-2(config)#crypto ikev2 proposal Flex-IKEv2-Prop
IKEv2 proposal MUST either have a set of an encryption algorithm other than aes-gcm, an integrity algorithm and a DH group configured or 
 encryption algorithm aes-gcm, a prf algorithm and a DH group configured
Telnet-2(config-ikev2-proposal)#encryption 3des 
Telnet-2(config-ikev2-proposal)#integrity sha1
Telnet-2(config-ikev2-proposal)#group 5
Telnet-2(config-ikev2-proposal)#exit
Telnet-2(config)#crypto ikev2 policy Flex-Policy
IKEv2 policy MUST have atleast one complete proposal attached 
Telnet-2(config-ikev2-policy)#proposal Flex-IKEv2-Prop
Telnet-2(config-ikev2-policy)#exit
Telnet-2(config)#crypto ikev2 profile Flex-IKEv2-Prof 
IKEv2 profile MUST have:
   1. A local and a remote authentication method.
   2. A match identity or a match certificate or match any statement.
Telnet-2(config-ikev2-profile)#authentication local pre-share 
Telnet-2(config-ikev2-profile)#auth remote pre-share 
Telnet-2(config-ikev2-profile)#$ity remote fqdn Telnet-1.ccielab.local       
Telnet-2(config-ikev2-profile)#identity local fqdn Telnet-2.ccielab.local
Telnet-2(config-ikev2-profile)#
Telnet-2(config-ikev2-profile)#keyring local Telnet1-Keyring
% Invalid keyring Telnet1-Keyring

Telnet-2(config-ikev2-profile)#exit
Telnet-2(config)#cry ikev2 keyring Telnet1-Keyring
Telnet-2(config-ikev2-keyring)#peer Telnet-1
Telnet-2(config-ikev2-keyring-peer)#address 1.1.1.1
Telnet-2(config-ikev2-keyring-peer)#pre-shared-key CCIE
Telnet-2(config-ikev2-keyring-peer)#exit
Telnet-2(config-ikev2-keyring)#exit
Telnet-2(config)#crypto ikev2 profile Flex-IKEv2-Prof
Telnet-2(config-ikev2-profile)#keyring local Telnet1-Keyring
Telnet-2(config-ikev2-profile)#
Telnet-2(config-ikev2-profile)#exit
Telnet-2(config)#crypto ikev2 client flexvpn FLEX-VPN
Telnet-2(config-ikev2-flexvpn)#client connect tunnel 1
Telnet-2(config-ikev2-flexvpn)#
Telnet-2(config-ikev2-flexvpn)#exit
Telnet-2(config)#
Telnet-2(config)#do sh run | i profile
crypto ikev2 profile Flex-IKEv2-Prof
crypto ipsec profile Flex-Profile
 tunnel protection ipsec profile Flex-Profile
Telnet-2(config)#crypto ipsec profile Flex-Profile
Telnet-2(ipsec-profile)#set transform-set 3des
Telnet-2(ipsec-profile)#set ikev2-profile Flex-IKEv2-Prof
Telnet-2(ipsec-profile)#

Hardly the smoothest setup in the work! Let's move on to the server:

Flex VPN Server configuration

Telnet-1(config)#ip local pool Flex-Pool 1.1.2.10 1.1.2.20
Telnet-1(config)#
Telnet-1(config)#aaa new-model 
Telnet-1(config)#
Telnet-1(config)#crypto ikev2 authorization policy default
%Warning: This will Modify Default IKEv2 Authorization Policy. Exit if you don't want
Telnet-1(config-ikev2-author-policy)#pool Flex-Pool
Telnet-1(config-ikev2-author-policy)#exit
Telnet-1(config)#
Telnet-1(config)#cry ikev2 profile Flex-IKEv2-Policy
IKEv2 profile MUST have:
   1. A local and a remote authentication method.
   2. A match identity or a match certificate or match any statement.
Telnet-1(config-ikev2-profile)#authentication local pre-share 
Telnet-1(config-ikev2-profile)#auth remote pre
Telnet-1(config-ikev2-profile)#$ity remote fqdn Telnet2.ccielab.local        
Telnet-1(config-ikev2-profile)#keyring local Flex-Keyring
% Invalid keyring Flex-Keyring

Telnet-1(config-ikev2-profile)#exi
Telnet-1(config)#cry ikev2 keyring Flex-Keyring
Telnet-1(config-ikev2-keyring)#peer Telnet1
Telnet-1(config-ikev2-keyring-peer)#address 2.2.2.2
Telnet-1(config-ikev2-keyring-peer)#pre CCIE
Telnet-1(config-ikev2-keyring-peer)#exit
Telnet-1(config-ikev2-keyring)#exit
Telnet-1(config)#cry ikev2 profile Flex-IKEv2-Policy             
Telnet-1(config-ikev2-profile)#keyring local Flex-Keyring         
Telnet-1(config-ikev2-profile)#exit
Telnet-1(config)#
Telnet-1(config)#cry ikev2 proposal Flex-Prop
IKEv2 proposal MUST either have a set of an encryption algorithm other than aes-gcm, an integrity algorithm and a DH group configured or 
 encryption algorithm aes-gcm, a prf algorithm and a DH group configured
Telnet-1(config-ikev2-proposal)#en 3des
Telnet-1(config-ikev2-proposal)#int sha1
Telnet-1(config-ikev2-proposal)#gr 5
Telnet-1(config-ikev2-proposal)#exit
Telnet-1(config)#
Telnet-1(config)#cry ikev2 poli
Telnet-1(config)#do sh run | i policy 
crypto ikev2 authorization policy default
Telnet-1(config)#cry ikev2 policy Flex-Policy
IKEv2 policy MUST have atleast one complete proposal attached 
Telnet-1(config-ikev2-policy)#proposal Flex-Prop
Telnet-1(config-ikev2-policy)#exit
Telnet-1(config)#cry ipsec profile Flex-IPSec-Prof
Telnet-1(ipsec-profile)#set transform-set 3des
%ERROR: transform set with tag "3des" does not exist.

Telnet-1(ipsec-profile)#do sh run | i transf
Telnet-1(ipsec-profile)#exit
Telnet-1(config)#cry ips transform-set 3des esp-3des esp-sha-hmac
Telnet-1(cfg-crypto-trans)#cry ipsec profile Flex-IPSec-Prof               
Telnet-1(ipsec-profile)#set transform-set 3des
Telnet-1(ipsec-profile)#set ikev2-profile Flex-IKEv2-Policy
Telnet-1(ipsec-profile)#exit
Telnet-1(config)#
Telnet-1(config)#int tun 1
Telnet-1(config-if)#ip unn loop0
Telnet-1(config-if)#
Telnet-1(config-if)#tun so lo0
Telnet-1(config-if)#tun mo ipsec ipv4
Telnet-1(config-if)#tun prot ipsec prof Flex-IPSec-Prof
Telnet-1(config-if)#
Telnet-1(config-if)#cry ikev2 profile Flex-IKEv2-Policy
Telnet-1(config-ikev2-profile)#virtual-template 1
Telnet-1(config-ikev2-profile)#exi
Telnet-1(config)#int virtual-tem 1 t t
Telnet-1(config-if)#ip unnu lo0
Telnet-1(config-if)#tun so lo0
Telnet-1(config-if)#tun mo ipse ipv4
Telnet-1(config-if)#tun prot ipsec profile Flex-IPSec-Prof
Telnet-1(config-if)#
%CRYPTO-6-ISAKMP_ON_OFF: ISAKMP is ON
Telnet-1(config-if)#
Telnet-1(config-if)#

Present day:

The configuration looks OK, and as the Flex VPN traffic will be carried within the IKE traffic, we do not need to worry about opening up the firewalls for the traffic between Telnet-1 and Telnet-2. However, the Flex VPN is not working. I can see the traffic, but nothing is getting established:

We'll need some debugging:

Telnet-1#debug cry isak
Crypto ISAKMP debugging is on
Telnet-1#debug cry ikev2 clie
Telnet-1#debug cry ikev2 client flex
ISAKMP: (0):peer matches *none* of the profiles
Telnet-1#debug cry ikev2 client flexvpn 
FlexVPN debugging is on
Telnet-1# 
ISAKMP: (0):peer matches *none* of the profiles
ISAKMP: (0):peer matches *none* of the profiles
Telnet-1#

We have the profiles set up, each should be identifying itself by the FQDN. Let's check:

Telnet-1#sh run | s crypto
crypto ikev2 authorization policy default
 pool Flex-Pool
 no route set interface
 route set access-list Flex-Routes
crypto ikev2 proposal Flex-Prop 
 encryption 3des
 integrity sha1
 group 5
crypto ikev2 policy Flex-Policy 
 proposal Flex-Prop
crypto ikev2 keyring Flex-Keyring
 peer Telnet1
  address 2.2.2.2
  pre-shared-key CCIE
 !
crypto ikev2 profile Flex-IKEv2-Policy
 match identity remote fqdn Telnet2.ccielab.local
 authentication remote pre-share
 authentication local pre-share
 keyring local Flex-Keyring
 aaa authorization group psk list AuthC default
 virtual-template 1
crypto ipsec transform-set 3des esp-3des esp-sha-hmac 
 mode tunnel
crypto ipsec profile Flex-IPSec-Prof
 set transform-set 3des 
 set ikev2-profile Flex-IKEv2-Policy
Telnet-1#

We are expecting an FQDN of Telnet2.ccielab.local, but are actually being sent Telnet-2.ccielab.local. Let's fix it:

Telnet-2(config)#crypto ikev2 profile Flex-IKEv2-Prof
Telnet-2(config-ikev2-profile)#no identity local fqdn Telnet-2.ccielab.local
Telnet-2(config-ikev2-profile)#identity local fqdn Telnet2.ccielab.local 
Telnet-2(config-ikev2-profile)#

Telnet-1(config)#
ISAKMP: (0):peer matches Flex-IKEv2-Policy profile
%LINEPROTO-5-UPDOWN: Line protocol on Interface Virtual-Access1, changed state to down
%LINK-3-UPDOWN: Interface Virtual-Access1, changed state to down
%LINEPROTO-5-UPDOWN: Line protocol on Interface Virtual-Access2, changed state to up
Telnet-1(config)#

Now we can start to see some action, but what is true for one is true for the other, and the line flaps. I actually missed the local identity command from Telnet-1, I'd better add it:

Telnet-1(config)#crypto ikev2 profile Flex-IKEv2-Policy
Telnet-1(config-ikev2-profile)#identity local fqdn Telnet-1.ccielab.local
Telnet-1(config-ikev2-profile)#
ISAKMP: (0):peer matches Flex-IKEv2-Policy profile
%LINEPROTO-5-UPDOWN: Line protocol on Interface Virtual-Access2, changed state to down
%LINK-3-UPDOWN: Interface Virtual-Access2, changed state to down
%LINEPROTO-5-UPDOWN: Line protocol on Interface Virtual-Access1, changed state to down
%LINEPROTO-5-UPDOWN: Line protocol on Interface Virtual-Access1, changed state to up
Telnet-1(config-ikev2-profile)#do un all
All possible debugging has been turned off
Telnet-1(config-ikev2-profile)#

Telnet-2(config-ikev2-profile)#
%LINEPROTO-5-UPDOWN: Line protocol on Interface Tunnel1, changed state to up
Telnet-2(config-ikev2-profile)#do sh ip int bri      
Interface                  IP-Address      OK? Method Status                Protocol
GigabitEthernet0/0         10.1.13.1       YES NVRAM  up                    up      
GigabitEthernet0/1         unassigned      YES NVRAM  administratively down down    
GigabitEthernet0/2         unassigned      YES NVRAM  administratively down down    
GigabitEthernet0/3         unassigned      YES NVRAM  administratively down down    
Loopback0                  2.2.2.2         YES NVRAM  up                    up      
Tunnel1                    1.1.2.12        YES manual up                    up      
Telnet-2(config-ikev2-profile)#

We now have a working tunnel, and the mistake I made was a very simple one. Still, hopefully the cheatsheet should help. I think its better to have the visual of how the different parts make up the VPNs, then just trying to blindly remember it!

That's the hope anyway.

With not just over two months to go, things are starting to fall into place. I still need to memorize the VPN components, and build up the speed. But there is still plenty of time.

Cisco IOS VPNs Cheatsheet

I have put together a little VPNs cheatsheet, it has got IKEv1, IKEv2, EasyVPN, DMVPN, FlexVPN and GETVPN.

Each VPN can be created with just two routers, and the steps are shown, like this:

You can download the file by heading over to the forum post.

I will be doing another one with ASA-based VPNs (L2L, Remote access etc), and one for troubleshooting.

CCIE:Sec Practice lab day 7 & 8: IKEdentify yourself!

I think I need to pick up the pace a bit today. I lost too much time trying to shoehorn things into GETVPN (and failing, but seeing as no-one else has replied to my challenge as yet, I am starting to feel less bad about it).

GETVPN is not finished yet. We still need to get the servers access to the 10.1.3.0/24 and 10.1.4.0/24 networks. TO do this we will probably need to run DMVPN over the top of GETVPN. But not today.

Today we'll be working on the right-hand side of the network, and aim to get IKEv1 running (see this post).

IKEv1 VPNs

We'll set up EIGRP named mode to get us the routes:

GETVPN-S1(config)#router eigrp LowerLeft 
GETVPN-S1(config-router)#address-family ipv4 auto 103
GETVPN-S1(config-router-af)#eigrp router-id 103.1.1.1
GETVPN-S1(config-router-af)#network 10.1.15.0 0.0.0.255
GETVPN-S1(config-router-af)#

Chicago(config)#router eigrp LowerLeft
Chicago(config-router)#address-family ipv4 auto 103
Chicago(config-router-af)#eigrp router-id 10.1.15.2
Chicago(config-router-af)#network 10.1.15.0 0.0.0.255
Chicago(config-router-af)#network 10.1.15.0 0.0.0.255
%DUAL-5-NBRCHANGE: EIGRP-IPv4 103: Neighbor 10.1.15.1 (GigabitEthern                           
Chicago(config-router-af)#
Chicago(config-router-af)#network 10.1.16.0 0.0.0.255
Chicago(config-router-af)#network 10.1.17.0 0.0.0.255
Chicago(config-router-af)#

IKEv1(config)#router eigrp LowerLeft
IKEv1(config-router)#add ipv4 auto 103
IKEv1(config-router-af)#eigrp router-id 10.1.17.1
IKEv1(config-router-af)#
IKEv1(config-router-af)#network 10.1.17.0 0.0.0.255
IKEv1(config-router-af)#
%DUAL-5-NBRCHANGE: EIGRP-IPv4 103: Neighbor 10.1.17.2 (GigabitEthernet0/1) is up: new adjacency
IKEv1(config-router-af)#
IKEv1(config-router-af)#network 10.1.18.0 0.0.0.255
IKEv1(config-router-af)#

NYC(config)#router eigrp EIGRP
NYC(config-router)#add ipv4 auto 103
NYC(config-router-af)#
NYC(config-router-af)#eigrp router-id 10.1.12.254
NYC(config-router-af)#network 10.1.14.0 0.0.0.255
NYC(config-router-af)#

GETVPN-S1(config-router-af)#network 10.1.14.0 0.0.0.255
GETVPN-S1(config-router-af)#
GETVPN-S1(config-router-af)#
%DUAL-5-NBRCHANGE: EIGRP-IPv4 103: Neighbor 10.1.14.254 (GigabitEthernet0/0) is up: new adjacency
GETVPN-S1(config-router-af)#

NYC now has routers up to IKEv1:

NYC(config-router-af)#do sh ip route | b Gate
Gateway of last resort is not set

      10.0.0.0/8 is variably subnetted, 10 subnets, 2 masks
C        10.1.12.0/24 is directly connected, GigabitEthernet0/1
L        10.1.12.254/32 is directly connected, GigabitEthernet0/1
C        10.1.13.0/24 is directly connected, GigabitEthernet0/2
L        10.1.13.254/32 is directly connected, GigabitEthernet0/2
C        10.1.14.0/24 is directly connected, GigabitEthernet0/0
L        10.1.14.254/32 is directly connected, GigabitEthernet0/0
D        10.1.15.0/24 [90/15360] via 10.1.14.1, 00:00:29, GigabitEthernet0/0
D        10.1.16.0/24 [90/20480] via 10.1.14.1, 00:00:29, GigabitEthernet0/0
D        10.1.17.0/24 [90/20480] via 10.1.14.1, 00:00:29, GigabitEthernet0/0
D        10.1.18.0/24 [90/25600] via 10.1.14.1, 00:00:29, GigabitEthernet0/0
NYC(config-router-af)#
NYC(config-router-af)#do ping 10.1.17.1
Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to 10.1.17.1, timeout is 2 seconds:
!!!!!
Success rate is 100 percent (5/5), round-trip min/avg/max = 6/8/11 ms
NYC(config-router-af)#

As the goal is for IKEv1 to have access over the VPN to 2.2.2.2, we need to give NYC access to that network, and likewise, Telnet-1 needs a route back:

NYC(config-router-af)#ip route 2.2.2.2 255.255.255.255 10.1.13.1
NYC(config)#do ping 2.2.2.2
Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to 2.2.2.2, timeout is 2 seconds:
!!!!!
Success rate is 100 percent (5/5), round-trip min/avg/max = 3/4/6 ms
NYC(config)#

Telnet-2(config)#ip route 0.0.0.0 0.0.0.0 10.1.13.254
Telnet-2(config)#

Start with the policy, create it on one, paste into the other:

IKEv1(config)#cry isakmp policy 10 
IKEv1(config-isakmp)#encryption 3des
IKEv1(config-isakmp)#hash sha
IKEv1(config-isakmp)#group 2
IKEv1(config-isakmp)#auth pre
IKEv1(config-isakmp)#life 3600
IKEv1(config-isakmp)#exit
IKEv1(config)#do sh run | s crypto
crypto isakmp policy 10
 encr 3des
 authentication pre-share
 group 2
 lifetime 3600
IKEv1(config)#

NYC(config)#crypto isakmp policy 10
NYC(config-isakmp)# encr 3des
NYC(config-isakmp)# authentication pre-share
NYC(config-isakmp)# group 2
NYC(config-isakmp)# lifetime 3600
NYC(config-isakmp)#

Now we need a transport set:

IKEv1(config)#crypto ipsec transform-set 3des-sha esp-3des esp-sha-hmac 
IKEv1(cfg-crypto-trans)#exit
IKEv1(config)#
IKEv1(config)#do sh run | s transform
crypto ipsec transform-set 3des-sha esp-3des esp-sha-hmac 
 mode tunnel
IKEv1(config)#

NYC(config-isakmp)#exit
NYC(config)#crypto ipsec transform-set 3des-sha esp-3des esp-sha-hmac 
NYC(cfg-crypto-trans)# mode tunnel
NYC(cfg-crypto-trans)#exit
NYC(config)#

Define the interesting traffic:

IKEv1(config)#ip access-list standard IKEv1-Tunnel-Traffic 
IKEv1(config-std-nacl)#permit 2.2.2.2
IKEv1(config-std-nacl)#exit
IKEv1(config)#

NYC(config)#ip access-list extended IKEv1-Tunnel-Traffic
NYC(config-ext-nacl)#permit ip host 2.2.2.2 host 10.1.17.1
NYC(config-ext-nacl)#permit ip host 10.1.17.1 host 2.2.2.2
NYC(config-ext-nacl)#

We set the keyring:

IKEv1(config)#crypto keyring NYC 
IKEv1(conf-keyring)#pre-shared-key address 10.1.14.254 key CCIE
IKEv1(conf-keyring)#exit
IKEv1(config)#

NYC(config)#crypto keyring IKEv1 
NYC(conf-keyring)#pre-shared-key address 10.1.17.1 key CCIE
NYC(conf-keyring)#exit
NYC(config)#

Now we can create the crypto map, starting with NYC:

NYC(config)#crypto map CRY-MAP 1 ipsec-isakmp
% NOTE: This new crypto map will remain disabled until a peer
        and a valid access list have been configured.
NYC(config-crypto-map)#set peer 10.1.17.1
NYC(config-crypto-map)#match address IKEv1-Tunnel-Traffic
NYC(config-crypto-map)#set transform-set 3des-sha
NYC(config-crypto-map)#
NYC(config-crypto-map)#set isakmp-profile IKEv1-ISAK-Profile
% IKEv1 Profile IKEv1-ISAK-Profile not found
NYC(config-crypto-map)#
NYC(config-crypto-map)#exit
NYC(config)#cry isakmp profile IKEv1-ISAK-Profile
% A profile is deemed incomplete until it has match identity statements
NYC(conf-isa-prof)#
NYC(conf-isa-prof)#match identity address 10.1.17.1
NYC(conf-isa-prof)#keyring IKEv1
NYC(conf-isa-prof)#exit
NYC(config)#crypto map CRY-MAP 1 ipsec-isakmp
NYC(config-crypto-map)#set isakmp-profile IKEv1-ISAK-Profile
NYC(config-crypto-map)#
NYC(config-crypto-map)#do sh run | s crypto
crypto keyring IKEv1  
  pre-shared-key address 10.1.17.1 key CCIE
crypto isakmp policy 10
 encr 3des
 authentication pre-share
 group 2
 lifetime 3600
crypto isakmp profile IKEv1-ISAK-Profile
   keyring IKEv1
   match identity address 10.1.17.1 255.255.255.255 
crypto ipsec transform-set 3des-sha esp-3des esp-sha-hmac 
 mode tunnel
crypto map CRY-MAP 1 ipsec-isakmp 
 set peer 10.1.17.1
 set transform-set 3des-sha 
 set isakmp-profile IKEv1-ISAK-Profile
 match address IKEv1-Tunnel-Traffic
NYC(config-crypto-map)#

It looks like my attempts at NOT throwing every bit of configuration I can find, meant that I forgot to create the ISAKMP profile. Let's do IKEv1:

IKEv1(config)#crypto keyring NYC
IKEv1(conf-keyring)#pre-shared-key address 10.1.14.254 key CCIE
IKEv1(conf-keyring)#
IKEv1(config)#crypto isakmp profile NYC-ISAK-Profile
% A profile is deemed incomplete until it has match identity statements
IKEv1(conf-isa-prof)#match identity address 10.1.14.254
IKEv1(conf-isa-prof)#keyring NYC
IKEv1(conf-isa-prof)#exit
IKEv1(config)#crypto map CRY-MAP 1 ipsec-isakmp 
% NOTE: This new crypto map will remain disabled until a peer
        and a valid access list have been configured.
IKEv1(config-crypto-map)#set peer 10.1.14.254
IKEv1(config-crypto-map)#match add IKEv1-Tunnel-Traffic
Access-list type conflicts with prior definitionERROR: "IKEv1-Tunnel-Traffic" is either an invalid name or the
        list already exists but is the wrong type.

IKEv1(config-crypto-map)#exit
IKEv1(config)#do sh run | s access-list
ip access-list standard IKEv1-Tunnel-Traffic
 permit 2.2.2.2
IKEv1(config)#no ip access-list standard IKEv1-Tunnel-Traffic                
IKEv1(config)#ip access-list extended IKEv1-Tunnel-Traffic
IKEv1(config-ext-nacl)#permit ip host 2.2.2.2 host 10.1.17.1   
IKEv1(config-ext-nacl)#permit ip host 10.1.17.1 host 2.2.2.2
IKEv1(config-ext-nacl)#
IKEv1(config-ext-nacl)#crypto map CRY-MAP 1 ipsec-isakmp                     
% NOTE: This new crypto map will remain disabled until a peer
        and a valid access list have been configured.
IKEv1(config-crypto-map)#match add IKEv1-Tunnel-Traffic                 
IKEv1(config-crypto-map)#set transform-set 3des-sha
IKEv1(config-crypto-map)#set isakmp-profile NYC-ISAK-Profile                
IKEv1(config-crypto-map)#do sh run | s crypto
crypto keyring NYC  
  pre-shared-key address 10.1.14.254 key CCIE
crypto isakmp policy 10
 encr 3des
 authentication pre-share
 group 2
 lifetime 3600
crypto isakmp profile NYC-ISAK-Profile
   keyring NYC
   match identity address 10.1.14.254 255.255.255.255 
crypto ipsec transform-set 3des-sha esp-3des esp-sha-hmac 
 mode tunnel
crypto map CRY-MAP 1 ipsec-isakmp 
 set peer 10.1.14.254
 set transform-set 3des-sha 
 set isakmp-profile NYC-ISAK-Profile
 match address IKEv1-Tunnel-Traffic
IKEv1(config-crypto-map)#

It needs to be an extended list, not a standard one. Now we attach this to the relevant interfaces:

IKEv1(config-crypto-map)#int gi0/1
IKEv1(config-if)#crypto map CRY-MAP
IKEv1(config-if)#
%CRYPTO-6-ISAKMP_ON_OFF: ISAKMP is ON
IKEv1(config-if)#

NYC(config-crypto-map)#int gi0/0
NYC(config-if)#crypto map CRY-MAP
NYC(config-if)#
%CRYPTO-6-ISAKMP_ON_OFF: ISAKMP is ON
NYC(config-if)#

Is everything there? Let's find out:

Telnet-2#ping 10.1.17.1 so lo0
Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to 10.1.17.1, timeout is 2 seconds:
Packet sent with a source address of 2.2.2.2 
.....
Success rate is 0 percent (0/5)
Telnet-2#

NYC(config-if)#do sh cry isa sa                  
IPv4 Crypto ISAKMP SA
dst             src             state          conn-id status
10.1.17.1       10.1.14.254     QM_IDLE           1001 ACTIVE

IPv6 Crypto ISAKMP SA

NYC(config-if)#do sh cry ips sa | i local|rem|enc|dec   
    Crypto map tag: CRY-MAP, local addr 10.1.14.254
   local  ident (addr/mask/prot/port): (10.1.17.1/255.255.255.255/0/0)
   remote ident (addr/mask/prot/port): (2.2.2.2/255.255.255.255/0/0)
    #pkts encaps: 0, #pkts encrypt: 0, #pkts digest: 0
    #pkts decaps: 0, #pkts decrypt: 0, #pkts verify: 0
    #pkts compressed: 0, #pkts decompressed: 0
    #pkts not decompressed: 0, #pkts decompress failed: 0
     local crypto endpt.: 10.1.14.254, remote crypto endpt.: 10.1.17.1
   local  ident (addr/mask/prot/port): (2.2.2.2/255.255.255.255/0/0)
   remote ident (addr/mask/prot/port): (10.1.17.1/255.255.255.255/0/0)
    #pkts encaps: 4, #pkts encrypt: 4, #pkts digest: 4
    #pkts decaps: 0, #pkts decrypt: 0, #pkts verify: 0
    #pkts compressed: 0, #pkts decompressed: 0
    #pkts not decompressed: 0, #pkts decompress failed: 0
     local crypto endpt.: 10.1.14.254, remote crypto endpt.: 10.1.17.1
        sa timing: remaining key lifetime (k/sec): (4257472/3450)
        sa timing: remaining key lifetime (k/sec): (4257472/3450)
NYC(config-if)#

IKEv1(config-if)#do sh cry isak sa
IPv4 Crypto ISAKMP SA
dst             src             state          conn-id status
10.1.17.1       10.1.14.254     QM_IDLE           1001 ACTIVE

IPv6 Crypto ISAKMP SA

IKEv1(config-if)#do sh cry ips | i local|rem|enc|dec
% Incomplete command.

IKEv1(config-if)#do sh cry ips sa | i local|rem|enc|dec
    Crypto map tag: CRY-MAP, local addr 10.1.17.1
   local  ident (addr/mask/prot/port): (10.1.17.1/255.255.255.255/0/0)
   remote ident (addr/mask/prot/port): (2.2.2.2/255.255.255.255/0/0)
    #pkts encaps: 0, #pkts encrypt: 0, #pkts digest: 0
    #pkts decaps: 4, #pkts decrypt: 4, #pkts verify: 4
    #pkts compressed: 0, #pkts decompressed: 0
    #pkts not decompressed: 0, #pkts decompress failed: 0
     local crypto endpt.: 10.1.17.1, remote crypto endpt.: 10.1.14.254
        sa timing: remaining key lifetime (k/sec): (4186964/3396)
        sa timing: remaining key lifetime (k/sec): (4186965/3396)
   local  ident (addr/mask/prot/port): (2.2.2.2/255.255.255.255/0/0)
   remote ident (addr/mask/prot/port): (10.1.17.1/255.255.255.255/0/0)
    #pkts encaps: 0, #pkts encrypt: 0, #pkts digest: 0
    #pkts decaps: 0, #pkts decrypt: 0, #pkts verify: 0
    #pkts compressed: 0, #pkts decompressed: 0
    #pkts not decompressed: 0, #pkts decompress failed: 0
     local crypto endpt.: 10.1.17.1, remote crypto endpt.: 10.1.14.254
IKEv1(config-if)#

The tunnel is up, the problem is that IKEv1 doesn't know where to send traffic for 2.2.2.2, this is an easy fix:

IKEv1(config-if)#ip route 2.2.2.2 255.255.255.255 10.1.14.254
IKEv1(config)#do ping 2.2.2.2
Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to 2.2.2.2, timeout is 2 seconds:
!!!!!
Success rate is 100 percent (5/5), round-trip min/avg/max = 16/17/20 ms
IKEv1(config)#

Telnet-2#ping 10.1.17.1 so lo0
Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to 10.1.17.1, timeout is 2 seconds:
Packet sent with a source address of 2.2.2.2 
!!!!!
Success rate is 100 percent (5/5), round-trip min/avg/max = 11/17/33 ms
Telnet-2#

NYC#sh cry ips sa identity 

interface: GigabitEthernet0/0
    Crypto map tag: CRY-MAP, local addr 10.1.14.254

   protected vrf: (none)
   local  ident (addr/mask/prot/port): (0.0.0.0/0.0.0.0/0/0)
   remote ident (addr/mask/prot/port): (0.0.0.0/0.0.0.0/0/0)
   current_peer (none) port 500
     DENY, flags={ident_is_root,}
    #pkts encaps: 0, #pkts encrypt: 0, #pkts digest: 0
    #pkts decaps: 0, #pkts decrypt: 0, #pkts verify: 0
    #pkts compressed: 0, #pkts decompressed: 0
    #pkts not compressed: 0, #pkts compr. failed: 0
    #pkts not decompressed: 0, #pkts decompress failed: 0
    #send errors 0, #recv errors 0

   protected vrf: (none)
   local  ident (addr/mask/prot/port): (10.1.17.1/255.255.255.255/0/0)
   remote ident (addr/mask/prot/port): (2.2.2.2/255.255.255.255/0/0)
   current_peer 10.1.17.1 port 500
     PERMIT, flags={origin_is_acl,}
    #pkts encaps: 0, #pkts encrypt: 0, #pkts digest: 0
    #pkts decaps: 0, #pkts decrypt: 0, #pkts verify: 0
    #pkts compressed: 0, #pkts decompressed: 0
    #pkts not compressed: 0, #pkts compr. failed: 0
    #pkts not decompressed: 0, #pkts decompress failed: 0
    #send errors 0, #recv errors 0

   protected vrf: (none)
   local  ident (addr/mask/prot/port): (2.2.2.2/255.255.255.255/0/0)
   remote ident (addr/mask/prot/port): (10.1.17.1/255.255.255.255/0/0)
   current_peer 10.1.17.1 port 500
     PERMIT, flags={origin_is_acl,}
    #pkts encaps: 14, #pkts encrypt: 14, #pkts digest: 14
    #pkts decaps: 10, #pkts decrypt: 10, #pkts verify: 10
    #pkts compressed: 0, #pkts decompressed: 0
    #pkts not compressed: 0, #pkts compr. failed: 0
    #pkts not decompressed: 0, #pkts decompress failed: 0
    #send errors 0, #recv errors 0
NYC#

All is working. However, there is a little bit of tidying up to do. The access-lists are too permissive, and we only need one entry:

NYC#sh access-list
Extended IP access list IKEv1-Tunnel-Traffic
    10 permit ip host 2.2.2.2 host 10.1.17.1 (44 matches)
    20 permit ip host 10.1.17.1 host 2.2.2.2
NYC#
NYC#conf t
NYC(config)#ip access-list extended IKEv1-Tunnel-Traffic
NYC(config-ext-nacl)#no 20
NYC(config-ext-nacl)#exit
NYC(config)#exit
NYC#sh access-list
Extended IP access list IKEv1-Tunnel-Traffic
    10 permit ip host 2.2.2.2 host 10.1.17.1 (44 matches)
NYC#

IKEv1(config-ext-nacl)#do sh access-list
Extended IP access list IKEv1-Tunnel-Traffic
    10 permit ip host 2.2.2.2 host 10.1.17.1 (1 match)
    20 permit ip host 10.1.17.1 host 2.2.2.2 (25 matches)
IKEv1(config-ext-nacl)#no 10
IKEv1(config-ext-nacl)#do ping 2.2.2.2  
Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to 2.2.2.2, timeout is 2 seconds:
.!!!!
Success rate is 80 percent (4/5), round-trip min/avg/max = 12/13/15 ms
IKEv1(config-ext-nacl)#

The pings still work, and we can for the output from "sh cry ips sa" into one window. More importantly, the access-lists are cleaner:

NYC#sh cry ips sa

interface: GigabitEthernet0/0
    Crypto map tag: CRY-MAP, local addr 10.1.14.254

   protected vrf: (none)
   local  ident (addr/mask/prot/port): (2.2.2.2/255.255.255.255/0/0)
   remote ident (addr/mask/prot/port): (10.1.17.1/255.255.255.255/0/0)
   current_peer 10.1.17.1 port 500
     PERMIT, flags={origin_is_acl,}
    #pkts encaps: 23, #pkts encrypt: 23, #pkts digest: 23
    #pkts decaps: 23, #pkts decrypt: 23, #pkts verify: 23
    #pkts compressed: 0, #pkts decompressed: 0
    #pkts not compressed: 0, #pkts compr. failed: 0
    #pkts not decompressed: 0, #pkts decompress failed: 0
    #send errors 0, #recv errors 0

     local crypto endpt.: 10.1.14.254, remote crypto endpt.: 10.1.17.1
     plaintext mtu 1446, path mtu 1500, ip mtu 1500, ip mtu idb GigabitEthernet0/0
     current outbound spi: 0x6E4A5A46(1850366534)
     PFS (Y/N): N, DH group: none

     inbound esp sas:
      spi: 0x38DA9074(953847924)
        transform: esp-3des esp-sha-hmac ,
        in use settings ={Tunnel, }
        conn id: 11, flow_id: SW:11, sibling_flags 80000040, crypto map: CRY-MAP
        sa timing: remaining key lifetime (k/sec): (4257669/3536)
        IV size: 8 bytes
        replay detection support: Y
        Status: ACTIVE(ACTIVE)

     inbound ah sas:

     inbound pcp sas:

     outbound esp sas:
      spi: 0x6E4A5A46(1850366534)
        transform: esp-3des esp-sha-hmac ,
        in use settings ={Tunnel, }
        conn id: 12, flow_id: SW:12, sibling_flags 80000040, crypto map: CRY-MAP
        sa timing: remaining key lifetime (k/sec): (4257669/3536)
        IV size: 8 bytes
        replay detection support: Y
        Status: ACTIVE(ACTIVE)

     outbound ah sas:

     outbound pcp sas:
NYC#

For the last part of this we need to be able to telnet through, so let's set that up:

Telnet-2(config)#line vty 0 4
Telnet-2(config-line)#password cisco
Telnet-2(config-line)#login
Telnet-2(config-line)#transport input telnet
Telnet-2(config-line)#

IKEv1#telnet 2.2.2.2
Trying 2.2.2.2 ... Open

****************************************************************
* Banner.                                                      *
****************************************************************

User Access Verification

Password: 
****************************************************************
* Banner.                                                      *
****************************************************************
Telnet-2>en
% No password set
Telnet-2>exit

[Connection to 2.2.2.2 closed by foreign host]
IKEv1#

Not bad. One of the tasks can be crossed off, and we can move on and do IKEv2 between NYC and the Easy-Server.

IKEv2 VPNs

NYC(config)#crypto keyring EASY-Server  
NYC(conf-keyring)#pre-shared-key address 10.1.6.254 key CCIE
NYC(conf-keyring)#exit
NYC(config)#crypto isakmp profile Easy-ISAK-Profile
% A profile is deemed incomplete until it has match identity statements
NYC(conf-isa-prof)#keyring EASY-Server
NYC(conf-isa-prof)#match identity address 10.1.6.254
NYC(conf-isa-prof)#
NYC(config)#crypto ikev2 keyring EASY-Server-Keyring
NYC(config-ikev2-keyring)#peer EASY-Server
NYC(config-ikev2-keyring-peer)#address 10.1.6.254
NYC(config-ikev2-keyring-peer)#exit
NYC(config-ikev2-keyring)#exit
NYC(config)#crypto ikev2 profile IKEv2-Profile
IKEv2 profile MUST have:
   1. A local and a remote authentication method.
   2. A match identity or a match certificate or match any statement.
NYC(config-ikev2-profile)#match identity remote add 10.1.6.254
NYC(config-ikev2-profile)#authen local pre-share 
NYC(config-ikev2-profile)#authen remote pre-share 
NYC(config-ikev2-profile)#keyring local EASY-Server-Keyring
NYC(config-ikev2-profile)#exit
NYC(config)#
NYC(config)#crypto map CRY-MAP 2 ipsec-isakmp
% NOTE: This new crypto map will remain disabled until a peer
        and a valid access list have been configured.
NYC(config-crypto-map)#set peer 10.1.6.254
NYC(config-crypto-map)#set transform-set 3des-sha
NYC(config-crypto-map)#set ikev2-profile IKEv2-Profile
NYC(config-crypto-map)#match address IKEv2-Tunnel-Traffic
NYC(config-crypto-map)#
NYC(config-crypto-map)#exit
NYC(config)#ip access-list extended IKEv2-Tunnel-Traffic
NYC(config-ext-nacl)#permit ip host 2.2.2.2 host 10.1.6.254
NYC(config-ext-nacl)#permit ip host 2.2.2.2 host 1.1.1.1
NYC(config-ext-nacl)#exit
NYC(config)#

Easy-Server(config)#crypto isakmp policy 10
Easy-Server(config-isakmp)# encr 3des
Easy-Server(config-isakmp)# authentication pre-share
Easy-Server(config-isakmp)# group 2
Easy-Server(config-isakmp)# lifetime 3600
Easy-Server(config-isakmp)#exi
Easy-Server(config)#crypto keyring NYC  
Easy-Server(conf-keyring)#  pre-shared-key address 10.1.14.254 key CCIE
Easy-Server(conf-keyring)#exit
Easy-Server(config)#cry ikev2 keyring NYC
Easy-Server(config-ikev2-keyring)#peer NYC
Easy-Server(config-ikev2-keyring-peer)#address 10.1.14.254
Easy-Server(config-ikev2-keyring-peer)#exit
Easy-Server(config-ikev2-keyring)#exit
Easy-Server(config)#crypto ikev2 profile IKEv2-Profile
IKEv2 profile MUST have:
   1. A local and a remote authentication method.
   2. A match identity or a match certificate or match any statement.
Easy-Server(config-ikev2-profile)# authentication remote pre-share
Easy-Server(config-ikev2-profile)# authentication local pre-share
Easy-Server(config-ikev2-profile)#keyring local NYC
Easy-Server(config-ikev2-profile)#match identity remote address 10.1.14.254
Easy-Server(config-ikev2-profile)#exit
Easy-Server(config)#crypto isakmp profile NYC
% A profile is deemed incomplete until it has match identity statements
Easy-Server(conf-isa-prof)#keyring NYC
Easy-Server(conf-isa-prof)#match identity address 10.1.14.254
Easy-Server(conf-isa-prof)#exit
Easy-Server(config)#$c transform-set 3des-sha esp-3des esp-sha-hmac          
Easy-Server(cfg-crypto-trans)# mode tunnel
Easy-Server(cfg-crypto-trans)#exit
Easy-Server(config)#crypto ipsec profile NYC-IPSec-Profile  
Easy-Server(ipsec-profile)#set transform-set 3des-sha
Easy-Server(ipsec-profile)#exit
Easy-Server(config)#ip access-list extended IKEv2-Tunnel-Traffic
Easy-Server(config-ext-nacl)#permit ip host 1.1.1.1 host 2.2.2.2
Easy-Server(config-ext-nacl)#permit ip host 1.1.1.1 host 10.1.14.254
Easy-Server(config-ext-nacl)#
Easy-Server(config-ext-nacl)#crypto map CRY-MAP 2 ipsec-isakmp
% NOTE: This new crypto map will remain disabled until a peer
        and a valid access list have been configured.
Easy-Server(config-crypto-map)#set peer 10.1.14.254
Easy-Server(config-crypto-map)#set transform-set 3des-sha 
Easy-Server(config-crypto-map)# set ikev2-profile IKEv2-Profile
Easy-Server(config-crypto-map)# match address IKEv2-Tunnel-Traffic
Easy-Server(config-crypto-map)#
Easy-Server(config-crypto-map)#int gi0/0
Easy-Server(config-if)#cry map CRY-MAP
Easy-Server(config-if)#
%CRYPTO-6-ISAKMP_ON_OFF: ISAKMP is ON
Easy-Server(config-if)#

We need a couple of routes:

Easy-Server(config)#ip route 1.1.1.1 255.255.255.255 10.1.7.1
Easy-Server(config)#ip route 0.0.0.0 0.0.0.0 10.1.6.1
Easy-Server(config)#

Telnet-1(config)#ip route 0.0.0.0 0.0.0.0 10.1.7.254
Telnet-1(config)#

NYC(config)#ip route 1.1.1.1 255.255.255.255 10.1.6.254

NYC(config)#

I forgot that the NYC router is (from EASY-Server) behind the ASA firewalls! Doh! Pretty dumb of me. We also need routes! Let's sort that out:

ASA9(config)# object network obj-NYC
ASA9(config-network-object)# host 10.1.14.254
ASA9(config-network-object)# 
ASA9(config-network-object)# exi
ASA9(config)# object network obj-NYC-External
ASA9(config-network-object)# host 10.1.9.100
ASA9(config-network-object)# nat (Inside,Outside) source static obj-NYC obj-NYC-External
ASA9(config)# 
ASA9(config)# access-list Inside->Outside extended permit esp host 10.1.14.254 host 10.1.6.254
ASA9(config)# access-list Inside->Outside extended permit udp host 10.1.14.254 host 10.1.6.254 eq isakmp
ASA9(config)# access-list Inside->Outside extended permit udp host 10.1.14.254 host 10.1.6.254 eq 4500  
ASA9(config)# access-list Inside->Outside extended permit icmp host 10.1.14.254 host 10.1.6.254 
ASA9(config)# access-list Outside->Inside extended permit udp host 10.1.6.254 host 10.1.14.254 eq isakmp
ASA9(config)# access-list Outside->Inside extended permit udp host 10.1.6.254 host 10.1.14.254 eq 4500  
ASA9(config)# access-list Outside->Inside extended permit icmp host 10.1.6.254 host 10.1.14.254       
ASA9(config)# access-list Outside->Inside extended permit esp host 10.1.6.254 host 10.1.14.254       
ASA9(config)# 

GETVPN-S1(config)#router eigrp LowerLeft
GETVPN-S1(config-router)#address-family ipv4 unicast autonomous-system 103
GETVPN-S1(config-router-af)#topology base
GETVPN-S1(config-router-af-topology)#redistribute static 
GETVPN-S1(config-router-af-topology)#exit
GETVPN-S1(config-router-af)#network 10.1.26.0 0.0.0.255                      
GETVPN-S1(config-router-af)#

Switch(config)#router eigrp LowerLeft
Switch(config-router)# address-family ipv4 unicast autonomous-system 103
Switch(config-router-af)#network 10.1.26.0 0.0.0.255
Switch(config-router-af)#
%DUAL-5-NBRCHANGE: EIGRP-IPv4 103: Neighbor 10.1.26.1 (Vlan26) is up: new adjacency
Switch(config-router-af)#topology base
Switch(config-router-af-topology)#red connected 
Switch(config-router-af-topology)#

ISP-2(config)#ip route 10.1.6.0 255.255.255.0 10.1.8.1

Easy-Server#ping 10.1.9.100
Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to 10.1.9.100, timeout is 2 seconds:
!!!!!
Success rate is 100 percent (5/5), round-trip min/avg/max = 11/16/24 ms
Easy-Server#

Now we need to fix EASY-Server:

Easy-Server(config)#crypto keyring NYC
Easy-Server(conf-keyring)#no pre-shared-key address 10.1.14.254 key CCIE
Easy-Server(conf-keyring)#pre-shared-key address 10.1.9.100 key CCIE
Easy-Server(conf-keyring)#crypto ikev2 keyring NYC
Easy-Server(config-ikev2-keyring)#peer NYC
Easy-Server(config-ikev2-keyring-peer)#no address 10.1.14.254
Easy-Server(config-ikev2-keyring-peer)#address 10.1.9.100
Easy-Server(config-ikev2-keyring-peer)#exit                                  
Easy-Server(config-ikev2-keyring)#exit
Easy-Server(config)#crypto ikev2 profile IKEv2-Profile
Easy-Server(config-ikev2-profile)#no match identity remote address 10.1.14.254 255.255.255.255   
Easy-Server(config-ikev2-profile)#match identity remote address 10.1.9.100
Easy-Server(config-ikev2-profile)#exit
Easy-Server(config)#crypto isakmp profile NYC
Easy-Server(conf-isa-prof)#no match identity address 10.1.14.254 255.255.255.255       
Easy-Server(conf-isa-prof)#match identity address 10.1.9.100
Easy-Server(conf-isa-prof)#exit
Easy-Server(config)#crypto map CRY-MAP 2 ipsec-isakmp
Easy-Server(config-crypto-map)#no set peer 10.1.14.254
Easy-Server(config-crypto-map)#set peer 10.1.9.100
Easy-Server(config-crypto-map)#

Missed something:

Easy-Server(ipsec-profile)#crypto ikev2 keyring NYC
Easy-Server(config-ikev2-keyring)#peer NYC
Easy-Server(config-ikev2-keyring-peer)#pre-shared-key CCIE
Easy-Server(config-ikev2-keyring-peer)#

NYC(ipsec-profile)#crypto ikev2 keyring EASY-Server-Keyring
NYC(config-ikev2-keyring)#peer EASY-Server
NYC(config-ikev2-keyring-peer)#pre-shared-key CCIE
NYC(config-ikev2-keyring-peer)#

NYC(config)#crypto ikev2 policy 10
NYC(config-ikev2-policy)#match add local 10.1.14.254
NYC(config-ikev2-policy)#crypto ikev2 profile IKEv2-Profile
NYC(config-ikev2-profile)#ident loc add 10.1.14.254
NYC(config-ikev2-profile)#match add loca int gi0/0
NYC(config-ikev2-profile)#exi
NYC(config)#cry isakmp invalid-spi-recovery 
NYC(config)#

Easy-Server(config-crypto-map)#crypto ikev2 policy 10
Easy-Server(config-ikev2-policy)#match add local 10.1.6.254
Easy-Server(config-ikev2-policy)#crypto ikev2 profile IKEv2-Profile
Easy-Server(config-ikev2-profile)#match add local int gi0/0
Easy-Server(config-ikev2-profile)#identity local add 10.1.6.254
Easy-Server(config-ikev2-profile)#
%CRYPTO-4-RECVD_PKT_INV_SPI: decaps: rec'd IPSEC packet has invalid spi for destaddr=10.1.6.254, prot=50, spi=0x2BAAE5CE(732620238), srcaddr=10.1.9.100, input interface=GigabitEthernet0/0
Easy-Server(config-ikev2-profile)#exi
Easy-Server(config)#cry isakmp invalid-spi-recovery 
Easy-Server(config)#

It's now day 8, and I am back at it, trying to get the tunnels up. I am a little tired today after going to see the truly incredible Belly play at the O2 Forum in Kentish town. The wife and I got back to the hotel room about 1 am, and I barely slept. Note to self, when they ask what room you want, always opt for the quiet one!

Tanya

Gail

Brilliant gig. I got a little squashed up at the front row and was very tempted to go further back, but I stayed, and scored (half of) a setlist. All work and no play make Stuart a dull boy.

But it's back to sorting out the IKEv2 VPN now.

The access-lists are getting the hits, but the traffic is not passing. I enabled debugging on the EASY-Server router (debug crypto ikev2), and here is a little bit of that output (with notes below the lines):

IKEv2:(SESSION ID = 1,SA ID = 1):[IKEv2 -> Crypto Engine] Computing DH public key, DH Group 5
IKEv2:(SA ID = 1):[Crypto Engine -> IKEv2] DH key Computation PASSED
^^ DH passes ^^
IKEv2:(SESSION ID = 1,SA ID = 1):Checking NAT discovery
IKEv2:(SESSION ID = 1,SA ID = 1):NAT OUTSIDE found
IKEv2:(SESSION ID = 1,SA ID = 1):NAT detected float to init port 4500, resp port 4500
^^ We work out that we are behind a NAT device, and switch to nat traversal ^^
IKEv2:(SESSION ID = 1,SA ID = 1):Completed SA init exchange
^^ SA completes ^^
IKEv2:(SESSION ID = 1,SA ID = 1):Check for EAP exchange
IKEv2:(SESSION ID = 1,SA ID = 1):Generate my authentication data
IKEv2:(SESSION ID = 1,SA ID = 1):Use preshared key for id 10.1.6.254, key len 4
^^ We know to use a PSK ^^
IKEv2:[IKEv2 -> Crypto Engine] Generate IKEv2 authentication data
IKEv2:[Crypto Engine -> IKEv2] IKEv2 authentication data generation PASSED
^^ IKEv2 authentication passes ^^
IKEv2:(SESSION ID = 1,SA ID = 1):Sending Packet [To 10.1.9.100:4500/From 10.1.6.254:4500/VRF i0:f0] 
Initiator SPI : 7C4CB6E87F26A509 - Responder SPI : 6BDD0554932AC653 Message id: 1
IKEv2 IKE_AUTH Exchange REQUEST 
Payload contents: 
 ENCR 
 
IKEv2:(SESSION ID = 1,SA ID = 1):Received Packet [From 10.1.9.100:4500/To 10.1.6.254:4500/VRF i0:f0] 
Initiator SPI : 7C4CB6E87F26A509 - Responder SPI : 6BDD0554932AC653 Message id: 1
IKEv2 IKE_AUTH Exchange RESPONSE 
Payload contents: 
 VID IDr AUTH SA TSi TSr NOTIFY(SET_WINDOW_SIZE) NOTIFY(ESP_TFC_NO_SUPPORT) NOTIFY(NON_FIRST_FRAGS) 
IKEv2:(SESSION ID = 1,SA ID = 1):Process auth response notify
^^ We start to send the IKEv2 packets ^^
IKEv2:(SESSION ID = 1,SA ID = 1):Searching policy based on peer's identity '10.1.14.254' of type 'IPv4 address'
^^ JFK sends us it's identity, which is the IP address ^^
IKEv2-ERROR:(SESSION ID = 1,SA ID = 1):: Failed to locate an item in the database
IKEv2:(SESSION ID = 1,SA ID = 1):Verification of peer's authentication data FAILED
IKEv2:(SESSION ID = 1,SA ID = 1):Auth exchange failed
IKEv2-ERROR:(SESSION ID = 1,SA ID = 1):: Auth exchange failed
IKEv2:(SESSION ID = 1,SA ID = 1):Abort exchange
IKEv2:(SESSION ID = 1,SA ID = 1):Deleting SA
^^ This is where and why it fails ^^

So, the resolution seems easy enough, we change the identity. But what do we change it to? We could change it to the internal IP address, so that:

crypto ikev2 profile IKEv2-Profile
 match address local interface GigabitEthernet0/0
 match identity remote address 10.1.9.100 255.255.255.255 

Becomes

crypto ikev2 profile IKEv2-Profile
 match address local interface GigabitEthernet0/0
 match identity remote address 10.1.14.254 255.255.255.255 

But this method means we are not secure, we have exposed our internal network IP addresses to the outside world, and this is bad security design.

Instead, the better option would be for the NYC router to identify itself by its hostname:

NYC(config)#crypto ikev2 profile IKEv2-Profile
NYC(config-ikev2-profile)#identity ?
  local  Specify the local IKE identity to use for the negotiation

NYC(config-ikev2-profile)#identity local ?
  address  address
  dn       Distinguished Name
  email    Fully qualified email string
  fqdn     Fully qualified domain name string
  key-id   key-id opaque string - proprietary types of identification

NYC(config-ikev2-profile)#identity local fqdn
% Incomplete command.

NYC(config-ikev2-profile)#identity local fqdn ?
  WORD  FQDN 

NYC(config-ikev2-profile)#identity local fqdn NYC.ccielab.local
NYC(config-ikev2-profile)#

Easy-Server(config)#crypto ikev2 profile IKEv2-Profile
Easy-Server(config-ikev2-profile)#no match identity remote address 10.1.9.100 255.255.255.255    
Easy-Server(config-ikev2-profile)#
Easy-Server(config-ikev2-profile)#match identity remote ?
  address  IP Address(es)
  any      match any peer identity
  email    Fully qualified email string [Max. 255 char(s)]
  fqdn     Fully qualified domain name string [Max. 255 char(s)]
  key-id   key-id opaque string

Easy-Server(config-ikev2-profile)#match identity remote fqdn NYC.ccielab.local         
Easy-Server(config-ikev2-profile)#

Now the tunnels come up:

Telnet-1#ping 2.2.2.2 so lo0
Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to 2.2.2.2, timeout is 2 seconds:
Packet sent with a source address of 1.1.1.1 
.!!!!
Success rate is 80 percent (4/5), round-trip min/avg/max = 21/26/31 ms
Telnet-1#

Telnet-2#ping 1.1.1.1 so lo0
Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to 1.1.1.1, timeout is 2 seconds:
Packet sent with a source address of 2.2.2.2 
!!!!!
Success rate is 100 percent (5/5), round-trip min/avg/max = 17/23/27 ms
Telnet-2#

Easy-Server#sh cry ikev2 sa
 IPv4 Crypto IKEv2  SA 

Tunnel-id Local                 Remote                fvrf/ivrf            Status 
1         10.1.6.254/4500       10.1.9.100/4500       none/none            READY  
      Encr: 3DES, PRF: SHA1, Hash: SHA96, DH Grp:5, Auth sign: PSK, Auth verify: PSK
      Life/Active Time: 86400/231 sec

 IPv6 Crypto IKEv2  SA 

Easy-Server#

Whilst NAT traversal gets around the issue of having a firewall in the way of a VPN, it cannot change the packets we are sending, such as the identity.

I still need to get quicker on the VPNs, so will be creating a little "cheat sheet" for the commands, differences, and similarities between the different ones. I'll put it on the site somewhere for everyone else.

Now that we have connectivity between Telnet-1 and Telnet-2 it would make sense to connect them together. So, next up is FlexVPN

CCIE:Sec practice lab - Day 2-5 - VRF-Aware GETVPN

Day 2, 3, 4 and now 5. and I am getting my arse kicked! It's all about GETVPN. Bearing in mind this will only be the second/third time I have tried it, and this time, it's VRF based. One VRF will go to one GETVPN server, the other VRF to the other, and these will share the routes they get down to the NYC server, which should then pass them where needed.

This part of the topology is actually a little more complex than I had originally envisaged, and if you don't mind me switching from present to past tense, I spent a good few hours trying to get this to work but have made some errors in either design or understanding.

Here is the issue. GETVPN-Client needs to have two different VRFs. This is the easy part. The hard part is that the routes to the tunnel destination are outside of any VRF. Options here are:

1. Set up the main interface (Gi0/0) in a third VRF and import/export the RT's into and out of this.
2. Set up a tunnel VRF as well as the VRF forwarding, using different VRFs.
3. Set up VLANs - this will mean a change to the IP addressing scheme

Number one has issues in that we cannot change the topology to suit our needs. Certainly in a lab exam, I cannot change the topology to support and fix a wrong solution. This goes for number three as well. So, I tried out number 2, and although I thought I had cracked it, I have ended up reaching out for help.

The nuts and bolts are in place in terms of connectivity, and it's now day 5 and I need to move on (one way or another), so we'll crack on and set up the switch (to give access to the WWW and ACS boxes):

Switch(config)#vlan 3,4
Switch(config-vlan)#exit
Switch(config)#int gi0/1
Switch(config-if)#swi mo acc
Switch(config-if)#swi acc vl 3
Switch(config-if)#int gi0/2
Switch(config-if)#swi mo acc
Switch(config-if)#swi acc vl 4
Switch(config-if)#exi
Switch(config)#int gi0/0
Switch(config-if)#swi tru enc dot
Switch(config-if)#swi mo tru
Switch(config-if)#

For GDOI to work, we need to permit a bunch of stuff through the firewalls:

ASA9(config)# sh run | i access-list
access-list Inside->Outside extended permit icmp host 10.1.26.1 host 10.1.2.254 
access-list Inside->Outside extended permit udp host 10.1.26.1 host 10.1.2.254 eq 848 
access-list Inside->Outside extended permit udp host 10.1.26.1 host 10.1.2.254 eq isakmp 
access-list Inside->Outside extended permit udp host 10.1.26.1 host 10.1.2.254 eq 4500 
access-list Inside->Outside extended permit esp host 10.1.26.1 host 10.1.2.254 
access-list Outside->Inside extended permit icmp host 10.1.2.254 host 10.1.26.1 
access-list Outside->Inside extended permit udp host 10.1.2.254 host 10.1.26.1 eq 848 
access-list Outside->Inside extended permit udp host 10.1.2.254 host 10.1.26.1 eq isakmp 
access-list Outside->Inside extended permit udp host 10.1.2.254 host 10.1.26.1 eq 4500 
access-list Outside->Inside extended permit esp host 10.1.2.254 host 10.1.26.1 
threat-detection statistics access-list
ASA9(config)#

ASAv6(config)# sh run | i access-list
access-list Outside->Inside extended permit icmp host 10.1.2.1 host 10.1.2.254 
access-list Outside->Inside extended permit udp host 10.1.2.254 eq 848 host 10.1.9.103 eq 848 
access-list Outside->Inside extended permit udp host 10.1.9.103 eq 848 host 10.1.2.254 eq 848 
access-list Outside->Inside extended permit udp host 10.1.9.103 host 10.1.2.254 eq 848 
access-list Outside->Inside extended permit icmp host 10.1.9.103 host 10.1.2.254 
access-list Outside->Inside extended permit esp host 10.1.9.103 host 10.1.2.254 
access-list Outside->Inside extended permit udp host 10.1.9.103 host 10.1.2.254 eq 4500 
access-list Outside->Inside extended permit udp host 10.1.9.103 host 10.1.2.254 eq isakmp 
threat-detection statistics access-list
ASAv6(config)# 

I do have more rules than needed, but the plan is to then prune back at the end of the lab (keep it neat and so on).

Originally I rushed in trying to get things working with virtual templates, VRFs and so on, but have now reverted back to a single layer, with no VRFs. It's working and the client has registered. Here are the (relevant) configs, the VRFs are still present, but not being used:

GETVPN-Client#sh run
!
ip vrf FVRF
 rd 100:100
!
ip vrf RED
 rd 103:103
!
ip vrf WHITE
 rd 104:104
!
crypto keyring RED-G1  
  pre-shared-key address 10.1.9.103 key CCIE
!
crypto isakmp policy 10
 encr 3des
 authentication pre-share
 group 2
 lifetime 3600
!
crypto isakmp policy 20
 encr aes 256
 authentication pre-share
 group 2
crypto isakmp key CCIE address 10.1.9.103     
crypto isakmp invalid-spi-recovery
crypto isakmp keepalive 10
crypto isakmp profile GET-ISAK-Profile-RED
   keyring RED-G1
   match identity address 10.1.9.103 255.255.255.255 
!
!
crypto ipsec transform-set GET-TS esp-3des esp-sha-hmac 
 mode tunnel
!
crypto ipsec profile GET-IPS-Profile-RED
 set group G1-RED
 set transform-set GET-TS 
 set isakmp-profile GET-ISAK-Profile-RED
!
!
crypto gdoi group G1-RED
 identity number 103
 server address ipv4 10.1.9.103
 client registration interface GigabitEthernet0/0
!
!
crypto map G1-RED isakmp-profile GET-ISAK-Profile-RED
crypto map G1-RED 103 gdoi 
 set group G1-RED
!
interface GigabitEthernet0/0
 ip address 10.1.2.254 255.255.255.0
 crypto map G1-RED
!
ip route 0.0.0.0 0.0.0.0 10.1.2.1
end

GETVPN-S1#sh run
!
crypto keyring RED-Keyring  
  pre-shared-key address 10.1.2.254 key CCIE
!
crypto ikev2 profile GET-IKEv2-Profile
 match address local interface GigabitEthernet0/1
 match identity remote address 10.1.2.254 255.255.255.255 
 authentication remote pre-share
 authentication local pre-share
!
!
!
crypto ipsec transform-set GET-TS esp-3des esp-sha-hmac 
 mode tunnel
!
crypto ipsec profile GET-Profile
!
crypto gdoi group G1-RED
 identity number 103
 server local
  rekey algorithm aes 128
  rekey retransmit 10 number 3
  rekey authentication mypubkey rsa GET-RSA-Key
  rekey transport unicast
  registration interface GigabitEthernet0/1
  sa ipsec 103
   profile GET-Profile
   match address ipv4 103
   replay counter window-size 64
   no tag
  address ipv4 10.1.26.1
!
ip route 0.0.0.0 0.0.0.0 10.1.26.200
!
access-list 103 permit tcp any any

The NAT rules are set up on the ASA:

ASA9(config)# object network obj-GETVPN-S1
ASA9(config-network-object)# host 10.1.26.1
ASA9(config-network-object)# nat (inside,outside) 1 source static obj-GETVPN-S1 obj-GETVPN-S1-external 
ERROR: obj-GETVPN-S1-external doesn't match an existing object or object-group
ASA9(config)# object network obj-GETVPN-S1-external                                     
ASA9(config-network-object)# host 10.1.9.103
ASA9(config-network-object)# exit
ASA9(config)# object network obj-GETVPN-S1                                              
ASA9(config-network-object)# nat (inside,outside) 1 source static obj-GETVPN-S1 obj-GETVPN-S1-external 
ASA9(config)# 

The client happily registers with this configuration:

GETVPN-Client#sh cry gdoi
GROUP INFORMATION

    Group Name               : G1-RED
    Group Identity           : 103
    Group Type               : GDOI (ISAKMP)
    Crypto Path              : ipv4
    Key Management Path      : ipv4
    Rekeys received          : 0
    IPSec SA Direction       : Both

     Group Server list       : 10.1.9.103
                               
Group Member Information For Group G1-RED:
    IPSec SA Direction       : Both
    ACL Received From KS     : gdoi_group_G1-RED_temp_acl

    Group member             : 10.1.2.254      vrf: None
       Local addr/port       : 10.1.2.254/848
       Remote addr/port      : 10.1.9.103/848
       fvrf/ivrf             : None/None
       Version               : 1.0.17
       Registration status   : Registered
       Registered with       : 10.1.9.103
       Re-registers in       : 2222 sec
       Succeeded registration: 1
       Attempted registration: 5
       Last rekey from       : 0.0.0.0
       Last rekey seq num    : 0
       Unicast rekey received: 0
       Rekey ACKs sent       : 0
       Rekey Received        : never
       DP Error Monitoring   : OFF
       IPSEC init reg executed    : 0
       IPSEC init reg postponed   : 0
       Active TEK Number     : 1
       SA Track (OID/status) : disabled

       allowable rekey cipher: any
       allowable rekey hash  : any
       allowable transformtag: any ESP

    Rekeys cumulative
       Total received        : 0
       After latest register : 0
       Rekey Acks sents      : 0

 ACL Downloaded From KS 10.1.9.103:
   access-list   permit tcp any any

KEK POLICY:
    Rekey Transport Type     : Unicast
    Lifetime (secs)          : 85231
    Encrypt Algorithm        : AES
    Key Size                 : 128     
    Sig Hash Algorithm       : HMAC_AUTH_SHA
    Sig Key Length (bits)    : 1296    

TEK POLICY for the current KS-Policy ACEs Downloaded:
  GigabitEthernet0/0:
    IPsec SA:
        spi: 0x80E62B18(2162567960)
        KGS: Disabled
        transform: esp-aes esp-sha-hmac 
        sa timing:remaining key lifetime (sec): (2433)
        Anti-Replay(Counter Based) : 64
        tag method : disabled
        alg key size: 16 (bytes)
        sig key size: 20 (bytes)
        encaps: ENCAPS_TUNNEL
          
GETVPN-Client#

Now this is where it gets frustrating fun. I have proved that GDOI *should* work. Now, GETVPN-Client needs to be switched about, so that the Gi0/0 interface is in VRF FVRF (Front-VRF) and the G1-RED (and then G2-WHITE) are connecting from different VRFs.

I'll start by moving everything to FVRF and confirm that it still works:

GETVPN-Client(config)#interface GigabitEthernet0/0
GETVPN-Client(config-if)# ip vrf for FVRF
% Interface GigabitEthernet0/0 IPv4 disabled and address(es) removed due to enabling VRF FVRF
GETVPN-Client(config-if)# ip address 10.1.2.254 255.255.255.0
GETVPN-Client(config-if)# crypto map G1-RED
GETVPN-Client(config-if)#exit
GETVPN-Client(config)#ip route vrf FVRF 0.0.0.0 0.0.0.0 10.1.2.1
GETVPN-Client(config)#crypto isakmp profile GET-ISAK-Profile-RED
GETVPN-Client(conf-isa-prof)#   no keyring RED-G1
GETVPN-Client(conf-isa-prof)#no match identity address 10.1.9.103 255.255.255.255   
GETVPN-Client(conf-isa-prof)#no crypto keyring RED-G1 
GETVPN-Client(config)#crypto keyring RED-G1 vrf FVRF
GETVPN-Client(conf-keyring)#  pre-shared-key address 10.1.9.103 key CCIE
GETVPN-Client(conf-keyring)# crypto isakmp profile GET-ISAK-Profile-RED
% A profile is deemed incomplete until it has match identity statements
GETVPN-Client(conf-isa-prof)#   keyring RED-G1
GETVPN-Client(conf-isa-prof)#match identity address 10.1.9.103 255.255.255.255 FVRF 
*Jul 19 11:23:22.674: %CRYPTO-5-GM_REGSTER: Start registration to KS 10.1.9.103 for group G1-RED using address 10.1.2.254 fvrf FVRF ivrf FVRF
GETVPN-Client(conf-isa-prof)#

This gets registered:

GETVPN-Client(conf-keyring)#do sh cry gdo | i fvrf|status|access-list
       fvrf/ivrf             : FVRF/FVRF
       Registration status   : Registered
       SA Track (OID/status) : disabled
   access-list   permit tcp any any
GETVPN-Client(conf-keyring)#

So far so good. Now, can we change the G1-RED GDOI group to use VRF RED? We'll need to use the ISAKMP policy to split the G1-RED into using a different interface, and the only option here is to use a virtual-template:

GETVPN-Client(config-if)#crypto isakmp profile GET-ISAK-Profile-RED
GETVPN-Client(conf-isa-prof)#?     
Crypto ISAKMP Profile Commands are:

  accounting        Enable AAA Accounting for IPSec Sessions
  ca                Specify certificate authorities to trust
  client            Specify client configuration settings
  default           Set a command to its defaults
  description       Specify a description of this profile
  exit              Exit from crypto isakmp profile sub mode
  initiate          Initiator property
  isakmp            ISAKMP Authorization command
  keepalive         Set a keepalive interval for use with IOS peers
  keyring           Specify keyring to use
  local-address     Interface to use for local address for this isakmp profile
  match             Match values of peer
  no                Negate a command or set its defaults
  qos-group         Apply a Qos policy class map for this profile
  self-identity     Specify Identity to use
  virtual-template  Specify the virtual-template for dynamic interface creation.
  vrf               Specify the VRF it is related to

GETVPN-Client(conf-isa-prof)#

Here is the virtual-template:

GETVPN-Client(config-if)#do sh run int virtual-templ 3 | b interface
interface Virtual-Template3 type tunnel
 ip vrf forwarding RED
 ip unnumbered GigabitEthernet0/0
 tunnel source GigabitEthernet0/0
 tunnel mode ipsec ipv4
 tunnel destination 10.1.9.103
 tunnel vrf FVRF
 tunnel protection ipsec profile GET-IPS-Profile-RED
end

GETVPN-Client(config-if)#

We specify that we are forwarding for VRF RED, but the tunnel VRF should use FVRF, these are the "ivrf" and "fvrf" shown in the "sh cry gdoi" output above. So, should I be using the IP address from Gi0/0 (as it is in a different VRF)? Well, the template seems happy about this:

GETVPN-Client(conf-isa-prof)#do sh ip int bri | i Virtual
Virtual-Template3          10.1.2.254      YES unset  up                    down    
GETVPN-Client(conf-isa-prof)#

Let's change the ISAKMP profile and find out if this works:

GETVPN-Client(config-if)#crypto isakmp profile GET-ISAK-Profile-RED
GETVPN-Client(conf-isa-prof)#virtual-template 3
GETVPN-Client(conf-isa-prof)#
GETVPN-Client(conf-isa-prof)#int gi 0/0
GETVPN-Client(config-if)#shut
GETVPN-Client(config-if)#
%LINK-5-CHANGED: Interface GigabitEthernet0/0, changed state to administratively down
GETVPN-Client(config-if)#
%LINEPROTO-5-UPDOWN: Line protocol on Interface GigabitEthernet0/0, changed state to down
GETVPN-Client(config-if)#
GETVPN-Client(config-if)#no shut
GETVPN-Client(config-if)#
%LINK-3-UPDOWN: Interface GigabitEthernet0/0, changed state to up
%LINEPROTO-5-UPDOWN: Line protocol on Interface GigabitEthernet0/0, changed state to up
GETVPN-Client(config-if)#

The GDOI tunnel is still up, but I don't think it's picked up the change yet:

ETVPN-Client(config-if)#
%GDOI-5-SA_TEK_UPDATED: SA TEK was updated
%GDOI-5-GM_RECV_REKEY: Received Rekey for group G1-RED from 10.1.26.1 to 10.1.2.254 with seq # 1, spi 0xCB99183CA67992C99B4503A3EDCF75C2
%GDOI-5-GM_INSTALL_POLICIES_SUCCESS: SUCCESS: Installation of Reg/Rekey policies from KS 10.1.9.103 for group G1-RED & gm identity 10.1.2.254 fvrf FVRF ivrf FVRF
GETVPN-Client(config-if)#end
GETVPN-Client#clear cry  gdoi 
% The Key Server and Group Member will destroy created and downloaded policies.
% All Group Members are required to re-register.

Are you sure you want to proceed ? [yes/no]: yes
GETVPN-Client#
%GDOI-4-GM_RE_REGISTER: The IPSec SA created for group G1-RED may have expired/been cleared, or didn't go through. Re-register to KS.
%CRYPTO-5-GM_REGSTER: Start registration to KS 10.1.9.103 for group G1-RED using address 10.1.2.254 fvrf FVRF ivrf FVRF
%GDOI-5-SA_TEK_UPDATED: SA TEK was updated
%GDOI-5-SA_KEK_UPDATED: SA KEK was updated 0xCB99183CA67992C99B4503A3EDCF75C2
%GDOI-5-GM_REGS_COMPL: Registration to KS 10.1.9.103 complete for group G1-RED using address 10.1.2.254 fvrf FVRF ivrf FVRF
%GDOI-5-GM_INSTALL_POLICIES_SUCCESS: SUCCESS: Installation of Reg/Rekey policies from KS 10.1.9.103 for group G1-RED & gm identity 10.1.2.254 fvrf FVRF ivrf FVRF
GETVPN-Client#
GETVPN-Client#sh cry gdoi | i status
       Registration status   : Registered
       SA Track (OID/status) : disabled
GETVPN-Client#

It's not using the ivrf of RED, like I hoped it would. I need to make one more change to the ISAKMP profile:

GETVPN-Client(config)#crypto isakmp profile GET-ISAK-Profile-RED
GETVPN-Client(conf-isa-prof)#vrf RED
% VRF configured in VirtualTemplate will get precedence over IKE Profile VRF
GETVPN-Client(conf-isa-prof)#end
GETVPN-Client#
GETVPN-Client#clear cry  gdoi       
% The Key Server and Group Member will destroy created and downloaded policies.
% All Group Members are required to re-register.

Are you sure you want to proceed ? [yes/no]: yes
GETVPN-Client#
%GDOI-4-GM_RE_REGISTER: The IPSec SA created for group G1-RED may have expired/been cleared, or didn't go through. Re-register to KS.
%CRYPTO-5-GM_REGSTER: Start registration to KS 10.1.9.103 for group G1-RED using address 10.1.2.254 fvrf FVRF ivrf FVRF
%GDOI-5-SA_TEK_UPDATED: SA TEK was updated
%GDOI-5-SA_KEK_UPDATED: SA KEK was updated 0xCB99183CA67992C99B4503A3EDCF75C2
%GDOI-5-GM_REGS_COMPL: Registration to KS 10.1.9.103 complete for group G1-RED using address 10.1.2.254 fvrf FVRF ivrf FVRF
%GDOI-5-GM_INSTALL_POLICIES_SUCCESS: SUCCESS: Installation of Reg/Rekey policies from KS 10.1.9.103 for group G1-RED & gm identity 10.1.2.254 fvrf FVRF ivrf FVRF
GETVPN-Client#

The tunnel is still using FRVF for both the ivrf and the fvrf. I'll make some changes, one at a time, changing them back if there is no success. Starting with the crypto keyring:

GETVPN-Client(config)#crypto keyring RED-G1-RED vrf RED
GETVPN-Client(conf-keyring)#pre-shared-key address 10.1.9.103 key CCIE
GETVPN-Client(conf-keyring)#exit
GETVPN-Client(config)#crypto isakmp profile GET-ISAK-Profile-RED
GETVPN-Client(conf-isa-prof)#keyring RED-G1-RED
GETVPN-Client(conf-isa-prof)#
GETVPN-Client(conf-isa-prof)#do clear cry gd
%GDOI-5-GM_REGS_COMPL: Registration to KS 10.1.9.103 complete for group G1-RED using address 10.1.2.254 fvrf FVRF ivrf FVRF
%GDOI-5-GM_INSTALL_POLICIES_SUCCESS: SUCCESS: Installation of Reg/Rekey policies from KS 10.1.9.103 for group G1-RED & gm identity 10.1.2.254 fvrf FVRF ivrf FVRF
GETVPN-Client(conf-isa-prof)#
GETVPN-Client(conf-isa-prof)#!! NOPE !!
GETVPN-Client(conf-isa-prof)#
GETVPN-Client(conf-isa-prof)#no keyring RED-G1
GETVPN-Client(conf-isa-prof)#
GETVPN-Client(conf-isa-prof)#do clear cry gd  
%GDOI-5-GM_REGS_COMPL: Registration to KS 10.1.9.103 complete for group G1-RED using address 10.1.2.254 fvrf FVRF ivrf FVRF
%GDOI-5-GM_INSTALL_POLICIES_SUCCESS: SUCCESS: Installation of Reg/Rekey policies from KS 10.1.9.103 for group G1-RED & gm identity 10.1.2.254 fvrf FVRF ivrf FVRF
GETVPN-Client(conf-isa-prof)#
GETVPN-Client(conf-isa-prof)#!! NOPE !!       
GETVPN-Client(conf-isa-prof)#
GETVPN-Client(conf-isa-prof)#no keyring RED-G1-RED
GETVPN-Client(conf-isa-prof)#keyring RED-G1       
GETVPN-Client(conf-isa-prof)#
GETVPN-Client(conf-isa-prof)#do clear cry gd      
%GDOI-5-GM_REGS_COMPL: Registration to KS 10.1.9.103 complete for group G1-RED using address 10.1.2.254 fvrf FVRF ivrf FVRF
%GDOI-5-GM_INSTALL_POLICIES_SUCCESS: SUCCESS: Installation of Reg/Rekey policies from KS 10.1.9.103 for group G1-RED & gm identity 10.1.2.254 fvrf FVRF ivrf FVRF
GETVPN-Client(conf-isa-prof)#

It's not the keyring. Maybe we need to try connecting from a different interface, such as a loopback:

GETVPN-Client(conf-isa-prof)#do sh run int loop 103 | b interface
interface Loopback103
 ip vrf forwarding RED
 ip address 10.1.103.1 255.255.255.0
end

GETVPN-Client(conf-isa-prof)#int loop 103
GETVPN-Client(config-if)#crypto map G1-RED
GETVPN-Client(config)#int gi0/0
GETVPN-Client(config-if)#no crypto map G1-RED
GETVPN-Client(config-if)#
GETVPN-Client(config-if)#int virtual-template 3
GETVPN-Client(config-if)#tun source loop 103
GETVPN-Client(config-if)#
GETVPN-Client(config-if)#do clear cry gdoi
%GDOI-5-GM_REGS_COMPL: Registration to KS 10.1.9.103 complete for group G1-RED using address 10.1.2.254 fvrf FVRF ivrf FVRF
%GDOI-5-GM_INSTALL_POLICIES_SUCCESS: SUCCESS: Installation of Reg/Rekey policies from KS 10.1.9.103 for group G1-RED & gm identity 10.1.2.254 fvrf FVRF ivrf FVRF
GETVPN-Client(config-if)#

Nope.

GETVPN-Client(config-if)#ip unnum loop 103
GETVPN-Client(config-if)#
GETVPN-Client(config-if)#do clear cry gdo
%GDOI-5-GM_REGS_COMPL: Registration to KS 10.1.9.103 complete for group G1-RED using address 10.1.2.254 fvrf FVRF ivrf FVRF
%GDOI-5-GM_INSTALL_POLICIES_SUCCESS: SUCCESS: Installation of Reg/Rekey policies from KS 10.1.9.103 for group G1-RED & gm identity 10.1.2.254 fvrf FVRF ivrf FVRF
GETVPN-Client(config-if)#

Still nope.

GETVPN-Client(config-if)#crypto gdoi group G1-RED                        
GETVPN-Client(config-gkm-group)#client registration interface loop 103
GETVPN-Client(config-gkm-group)#
%CRYPTO-5-GM_REGSTER: Start registration to KS 10.1.9.103 for group G1-RED using address 10.1.103.1 fvrf RED ivrf RED
GETVPN-Client(config-gkm-group)#
GETVPN-Client(config-gkm-group)#

The first change, but now everything is going through VRF RED, and we are not registering with the GDOI server.

GETVPN-Client(config-gkm-group)#interface Virtual-Template3 type tunnel
GETVPN-Client(config-if)#tunnel source gi0/0
GETVPN-Client(config-if)#
GETVPN-Client(config-if)#do clear cry gdoi
GETVPN-Client(config-if)#
%CRYPTO-5-GM_REGSTER: Start registration to KS 10.1.9.103 for group G1-RED using address 10.1.103.1 fvrf RED ivrf RED
GETVPN-Client(config-if)#
GETVPN-Client(config-if)#

OK, I can spot an issue with the configuration as it currently is:

GETVPN-Client(config-if)#do sh run int virtual-tem 3
interface Virtual-Template3 type tunnel
 ip vrf forwarding RED
 ip unnumbered Loopback103
 tunnel source GigabitEthernet0/0
 tunnel mode ipsec ipv4
 tunnel destination 10.1.9.103
 tunnel vrf FVRF
 tunnel protection ipsec profile GET-IPS-Profile-RED
end

GETVPN-Client(config-if)#do sh run int loop103
interface Loopback103
 ip vrf forwarding RED
 ip address 10.1.103.1 255.255.255.0
 crypto map G1-RED
end

GETVPN-Client(config-if)#do sh run int gi0/0
interface GigabitEthernet0/0
 ip vrf forwarding FVRF
 ip address 10.1.2.254 255.255.255.0
 duplex auto
 speed auto
 media-type rj45
end

GETVPN-Client(config-if)#

The crypto map is applied to an interface that will not be accepting the traffic (loop103). We are just using this for the IP address, not for taking any actual traffic. Let's set up the crypto map on the virtual template:

GETVPN-Client(config-if)#interface Virtual-Template3 type tunnel
GETVPN-Client(config-if)#crypto map G1-RED
% NOTE: crypto map is configured on tunnel interface.
        Currently only GDOI crypto map is supported on tunnel interface.

GETVPN-Client(config-if)#do clear cry gdoi
%CRYPTO-5-GM_REGSTER: Start registration to KS 10.1.9.103 for group G1-RED using address 10.1.103.1 fvrf RED ivrf RED
GETVPN-Client(config-if)#

Still not right.

GETVPN-Client(config-if)#crypto gdoi group G1-RED
GETVPN-Client(config-gkm-group)#client registration interface gi0/0
GETVPN-Client(config-gkm-group)#
%GDOI-5-GM_REGS_COMPL: Registration to KS 10.1.9.103 complete for group G1-RED using address 10.1.2.254 fvrf FVRF ivrf FVRF
%GDOI-5-GM_INSTALL_POLICIES_SUCCESS: SUCCESS: Installation of Reg/Rekey policies from KS 10.1.9.103 for group G1-RED & gm identity 10.1.2.254 fvrf FVRF ivrf FVRF
GETVPN-Client(config-gkm-group)#

After much tooing-and-frowing (including setting up the G2-WHITE GDOI group, I have come to the conclusion that what I am trying to achieve is not possible. Therefore need to reconfigure the network.

Screw it. We'll go down the same route as all the docs I have found us, which is to use VLANs and sub-interfaces. The clock is ticking and I need to get on

LON-1(config)#int gi0/1
LON-1(config-if)#no ip add
LON-1(config-if)#int gi0/1.1
LON-1(config-subif)#encap dot 10
LON-1(config-subif)#ip address 10.1.2.1 255.255.255.0
LON-1(config-subif)#exi
LON-1(config)#int gi0/1.103
LON-1(config-subif)#encap dot 103
LON-1(config-subif)#ip add 10.1.103.1 255.255.255.0
LON-1(config-subif)#exi
LON-1(config)#int gi0/1.104
LON-1(config-subif)#encap dot 104
LON-1(config-subif)#ip add 10.1.104.1 255.255.255.0
LON-1(config-subif)#do sh ip int bri
Interface                  IP-Address      OK? Method Status                Protocol
GigabitEthernet0/0         10.1.1.1        YES NVRAM  up                    up      
GigabitEthernet0/1         unassigned      YES manual up                    up      
GigabitEthernet0/1.1       10.1.2.1        YES manual up                    up      
GigabitEthernet0/1.103     10.1.103.1      YES manual up                    up      
GigabitEthernet0/1.104     10.1.104.1      YES manual up                    up      
Tunnel0                    192.168.1.11    YES manual up                    up      
LON-1(config-subif)#

ASAv6(config)# int gi0/0
ASAv6(config-if)# no nameif outside
ASAv6(config-if)# no bridge-gro 1
ASAv6(config-if)# 
ASAv6(config-if)# int gi0/0.1
ASAv6(config-subif)# vlan 10
ASAv6(config-subif)# bridge-group 1
ASAv6(config-subif)# exi
ASAv6(config)# int gi0/0.103
ASAv6(config-subif)# vlan 301
ASAv6(config-subif)# bridge-group 103
ASAv6(config-subif)# exi
ASAv6(config)# int gi0/0.104
ASAv6(config-subif)# vlan 104
ASAv6(config-subif)# bridge 104
ASAv6(config-subif)# exit
ASAv6(config)# sh int ip bri
Interface                  IP-Address      OK? Method Status                Protocol
GigabitEthernet0/0         unassigned      YES unset  up                    up  
GigabitEthernet0/0.1       10.1.2.200      YES unset  up                    up  
GigabitEthernet0/0.103     unassigned      YES unset  up                    up  
GigabitEthernet0/0.104     unassigned      YES unset  up                    up  
GigabitEthernet0/1         10.1.2.200      YES unset  up                    up  
BVI1                       10.1.2.200      YES manual up                    up  
ASAv6(config)# int bvi 103
ASAv6(config-if)# ip ad 10.1.103.200 255.255.255.0
ASAv6(config-if)# int bvi 104
ASAv6(config-if)# ip ad 10.1.104.200 255.255.255.0
ASAv6(config-if)# int gi0/1
ASAv6(config-if)# no nameif inside
ASAv6(config-if)# no bridg 1
ASAv6(config-if)# int gi0/1.1
ASAv6(config-subif)# vlan 11
ASAv6(config-subif)# bridge 1
ASAv6(config-subif)# int gi0/1.103
ASAv6(config-subif)# vlan 103
ASAv6(config-subif)# bridg 103
ASAv6(config-subif)# int gi0/1.104
ASAv6(config-subif)# vlan 401
ASAv6(config-subif)# bridge 104
ASAv6(config-subif)#  
ASAv6(config-subif)# int GigabitEthernet0/0.1
ASAv6(config-subif)# nameif Outside
INFO: Security level for "Outside" set to 0 by default.
ASAv6(config-subif)# int GigabitEthernet0/1.1
ASAv6(config-subif)# nameif Inside
INFO: Security level for "Inside" set to 100 by default.
ASAv6(config-subif)# int GigabitEthernet0/0.103
ASAv6(config-subif)# nameif Outside-RED
INFO: Security level for "Outside-RED" set to 0 by default.
ASAv6(config-subif)# int GigabitEthernet0/0.104
ASAv6(config-subif)# nameif Outside-WHITE
INFO: Security level for "Outside-WHITE" set to 0 by default.
ASAv6(config-subif)# int GigabitEthernet0/1.103                      
ASAv6(config-subif)# nameif Inside-RED
INFO: Security level for "Inside-RED" set to 0 by default.
ASAv6(config-subif)# sec 100
ASAv6(config-subif)# int GigabitEthernet0/1.104
ASAv6(config-subif)# nameif Inside-WHITE       
INFO: Security level for "Inside-WHITE" set to 0 by default.
ASAv6(config-subif)# sec 100
ASAv6(config-subif)#    

GETVPN-Client(config)#interface GigabitEthernet0/0
GETVPN-Client(config-if)#no ip vrf forwarding FVRF
GETVPN-Client(config-if)#
GETVPN-Client(config-if)#int gi0/0.1
GETVPN-Client(config-subif)#encap dot 11
GETVPN-Client(config-subif)#ip vrf for FVRF
GETVPN-Client(config-subif)#ip address 10.1.2.254 255.255.255.0
GETVPN-Client(config-subif)#
GETVPN-Client(config-subif)#int gi0/0.103
GETVPN-Client(config-subif)#encap dot 103
GETVPN-Client(config-subif)#ip add 10.1.103.254 255.255.255.0
GETVPN-Client(config-subif)#ip vrf for RED
GETVPN-Client(config-subif)#ip add 10.1.103.254 255.255.255.0
% 10.1.103.0 overlaps with Loopback103
GETVPN-Client(config-subif)#
GETVPN-Client(config-subif)#no int lo 103  
GETVPN-Client(config)#int gi0/1.103                    
GETVPN-Client(config-subif)#ip add 10.1.103.254 255.255.255.0
GETVPN-Client(config-subif)#
GETVPN-Client(config-subif)#no int loo 104
GETVPN-Client(config-subif)#int gi 0/0.104
GETVPN-Client(config-subif)#ip vrf for WHITE
GETVPN-Client(config-subif)#encap dot 401
GETVPN-Client(config-subif)#ip add 10.1.104.254 255.255.255.0
GETVPN-Client(config-subif)#

LON-1(config)#router eigrp 1
LON-1(config-router)#network 10.1.103.0 0.0.0.255
LON-1(config-router)#network 10.1.104.0 0.0.0.255
LON-1(config-router)#no redistr static metric 100 10 255 1 1500 route-map vrf-routes        
LON-1(config-router)#

Now, let's get the GDOI's working!

GETVPN-Client(config-gkm-group)#crypto gdoi group G1-RED
GETVPN-Client(config-gkm-group)#no client registration interface GigabitEthernet0/0.1
GETVPN-Client(config-gkm-group)#crypto gdoi group G2-WHITE
GETVPN-Client(config-gkm-group)#no client registration interface GigabitEthernet0/0.1
GETVPN-Client(config-gkm-group)#
GETVPN-Client(config-gkm-group)#interface Virtual-Template3 type tunnel
GETVPN-Client(config-if)#int gi0/1.103
GETVPN-Client(config-subif)#crypto map G1-RED
GETVPN-Client(config-subif)#   
%CRYPTO-5-GM_REGSTER: Start registration to KS 10.1.9.103 for group G1-RED using address 10.1.103.254 fvrf RED ivrf RED
GETVPN-Client(config-subif)#int gi0/1.104
GETVPN-Client(config-subif)#crypto map G2-WHITE                   
GETVPN-Client(config-subif)#
%CRYPTO-5-GM_REGSTER: Start registration to KS 10.1.9.103 for group G1-RED using address 10.1.103.254 fvrf RED ivrf RED
GETVPN-Client(config-subif)#interface Virtual-Template3 type tunnel
GETVPN-Client(config-if)#no tunnel vrf FVRF
GETVPN-Client(config-if)#interface Virtual-Template4 type tunnel                                       
GETVPN-Client(config-if)#no tunnel vrf FVRF                     
GETVPN-Client(config-if)#exit
GETVPN-Client(config)#ip route vrf RED 0.0.0.0 0.0.0.0 10.1.103.1
GETVPN-Client(config)#ip route vrf WHITE 0.0.0.0 0.0.0.0 10.1.104.1 
GETVPN-Client(config)#

ASAv6(config)# access-list FVRF extended permit icmp host 10.1.2.1 host 10.1.2.254
ASAv6(config)# access-group FVRF in int Outside
ASAv6(config)# access-list RED extended permit icmp host 10.1.103.1 host 10.1.103.254
ASAv6(config)# access-group RED in interface Outside-RED
ASAv6(config)# access-list WHITE extended permit icmp host 10.1.104.1 host 10.1.104.254
ASAv6(config)# access-group WHITE in interface Outside-WHITE  
ASAv6(config)# access-list RED extended permit udp host 10.1.9.103 eq 848 host 10.1.103.254 eq 848
ASAv6(config)# access-list RED extended permit udp host 10.1.9.103 host 10.1.103.254 eq 4500
ASAv6(config)# access-list RED extended permit udp host 10.1.9.103 host 10.1.103.254 eq isakmp
ASAv6(config)# access-list RED extended permit icmp host 10.1.9.103 host 10.1.103.254
ASAv6(config)# access-list WHITE extended permit udp host 10.1.9.104 eq 848 host 10.1.104.254 eq 848
ASAv6(config)# access-list WHITE extended permit udp host 10.1.9.104 host 10.1.104.254 eq 4500
ASAv6(config)# access-list WHITE extended permit udp host 10.1.9.104 host 10.1.104.254 eq isakmp
ASAv6(config)# access-list WHITE extended permit icmp host 10.1.9.104 host 10.1.104.254

ASA9(config)# access-list Inside->Outside extended permit udp host 10.1.26.1 host 10.1.103.254 eq 848 
ASA9(config)# access-list Inside->Outside extended permit udp host 10.1.26.1 host 10.1.103.254 eq 4500 
ASA9(config)# access-list Inside->Outside extended permit udp host 10.1.26.1 host 10.1.103.254 eq isakmp 
ASA9(config)# access-list Inside->Outside extended permit icmp host 10.1.26.1 host 10.1.103.254 
ASA9(config)# access-list Inside->Outside extended permit udp host 10.1.11.1 host 10.1.104.254 eq 848 
ASA9(config)# access-list Inside->Outside extended permit udp host 10.1.11.1 host 10.1.104.254 eq 4500 
ASA9(config)# access-list Inside->Outside extended permit udp host 10.1.11.1 host 10.1.104.254 eq isakmp
ASA9(config)# access-list Inside->Outside extended permit icmp host 10.1.11.1 host 10.1.104.254  
ASA9(config)# 

GETVPN-S1#sh run | s crypto
crypto keyring RED-Keyring  
  pre-shared-key address 10.1.2.254 key CCIE
crypto ikev2 profile GET-IKEv2-Profile
 match address local interface GigabitEthernet0/1
 match identity remote address 10.1.2.254 255.255.255.255 
 authentication remote pre-share
 authentication local pre-share
crypto ipsec transform-set GET-TS esp-3des esp-sha-hmac 
 mode tunnel
crypto ipsec profile GET-Profile
crypto gdoi group G1-RED
 identity number 103
 server local
  rekey algorithm aes 128
  rekey retransmit 10 number 3
  rekey authentication mypubkey rsa GET-RSA-Key
  rekey transport unicast
  registration interface GigabitEthernet0/1
  sa ipsec 103
   profile GET-Profile
   match address ipv4 103
   replay counter window-size 64
   no tag
  address ipv4 10.1.26.1
GETVPN-S1#conf t
Enter configuration commands, one per line.  End with CNTL/Z.
GETVPN-S1(config)#crypto keyring RED-Keyring
GETVPN-S1(conf-keyring)#no pre-shared-key address 10.1.2.254 key CCIE
GETVPN-S1(conf-keyring)#pre-shared-key address 10.1.103.254 key CCIE
GETVPN-S1(conf-keyring)#crypto ikev2 profile GET-IKEv2-Profile
GETVPN-S1(config-ikev2-profile)#no match identity remote address 10.1.2.254 255.255.255.255      
GETVPN-S1(config-ikev2-profile)#match identity remote address 10.1.103.254 255.255.255.255       
GETVPN-S1(config-ikev2-profile)#

GETVPN-S2#sh run | s crypto
crypto keyring WHITE-Keyring  
  pre-shared-key address 10.1.2.254 key CCIE
crypto ikev2 profile GET-IKEv2-Profile
 match address local interface GigabitEthernet0/0
 match identity remote address 10.1.2.254 255.255.255.255 
 authentication remote pre-share
 authentication local pre-share
crypto ipsec transform-set GET-TS esp-3des esp-sha-hmac 
 mode tunnel
crypto ipsec profile GET-Profile
crypto gdoi group G2-WHITE
 identity number 104
 server local
  rekey algorithm aes 128
  rekey retransmit 10 number 3
  rekey authentication mypubkey rsa GET-RSA-Key
  rekey transport unicast
  registration interface GigabitEthernet0/0
  sa ipsec 104
   profile GET-Profile
   match address ipv4 104
   replay counter window-size 64
   no tag
  address ipv4 10.1.11.1
GETVPN-S2#conf t
Enter configuration commands, one per line.  End with CNTL/Z.
GETVPN-S2(config)#line con 0
GETVPN-S2(config-line)#width 255
GETVPN-S2(config-line)#exit
GETVPN-S2(config)#crypto keyring WHITE-Keyring
GETVPN-S2(conf-keyring)#no pre-shared-key address 10.1.2.254 key CCIE
GETVPN-S2(conf-keyring)#pre-shared-key address 10.1.104.254 key CCIE
GETVPN-S2(conf-keyring)#crypto ikev2 profile GET-IKEv2-Profile
GETVPN-S2(config-ikev2-profile)#no match identity remote address 10.1.2.254 255.255.255.255
GETVPN-S2(config-ikev2-profile)#match identity remote address 10.1.104.254 255.255.255.255
GETVPN-S2(config-ikev2-profile)#

GETVPN-Client(conf-isa-prof)#do sh run | s crypto
crypto keyring RED-G1 vrf FVRF 
  pre-shared-key address 10.1.9.103 key CCIE
crypto keyring RED-G1-RED vrf RED 
  pre-shared-key address 10.1.9.103 key CCIE
crypto keyring WHITE-G2-WHITE vrf WHITE 
  pre-shared-key address 10.1.9.104 key CCIE
crypto isakmp policy 10
 encr 3des
 authentication pre-share
 group 2
 lifetime 3600
crypto isakmp policy 20
 encr aes 256
 authentication pre-share
 group 2
crypto isakmp key CCIE address 10.1.9.103     
crypto isakmp key CCIE address 10.1.9.104     
crypto isakmp invalid-spi-recovery
crypto isakmp keepalive 10
crypto isakmp profile GET-ISAK-Profile-RED
   vrf RED
   keyring RED-G1-RED
   match identity address 10.1.9.103 255.255.255.255 
crypto isakmp profile GET-ISAK-Profile-WHITE
   vrf WHITE
   keyring WHITE-G2-WHITE
   match identity address 10.1.9.104 255.255.255.255 
crypto ipsec transform-set GET-TS esp-3des esp-sha-hmac 
 mode tunnel
crypto ipsec profile GET-IPS-Profile-RED
 set group G1-RED
 set transform-set GET-TS 
 set isakmp-profile GET-ISAK-Profile-RED
crypto ipsec profile GET-IPS-Profile-WHITE
 set group G2-WHITE
 set transform-set GET-TS 
 set isakmp-profile GET-ISAK-Profile-WHITE
crypto gdoi group G1-RED
 identity number 103
 server address ipv4 10.1.9.103
crypto gdoi group G2-WHITE
 identity number 104
 server address ipv4 10.1.9.104
crypto map G1-RED local-address GigabitEthernet0/0.103
crypto map G1-RED isakmp-profile GET-ISAK-Profile-RED
crypto map G1-RED 103 gdoi 
 set group G1-RED
crypto map G2-WHITE local-address GigabitEthernet0/0.104
crypto map G2-WHITE isakmp-profile GET-ISAK-Profile-WHITE
crypto map G2-WHITE 104 gdoi 
 set group G2-WHITE
 crypto map G1-RED
 crypto map G2-WHITE
GETVPN-Client(conf-isa-prof)#

GETVPN-Client(conf-isa-prof)#do sh cry gdo | i Group Name|status
    Group Name               : G1-RED
       Registration status   : Registered
       SA Track (OID/status) : disabled
    Group Name               : G2-WHITE
       Registration status   : Registered
       SA Track (OID/status) : disabled
GETVPN-Client(conf-isa-prof)#

It's working but I am actually pretty annoyed. I have spent way too long battling against this, probably five days (not constant, I still went to work, cooked food, had family time, took the boys to a party and did family stuff), but it's five days worth of study that I could have spent studying something else.

Trying to put a positive spin on things, this is actually a good lesson for the lab. If something is taking too long then move on. Either you'll figure it out later, something else will give you a clue, you'll fix a prerequisite, or you'll leave that exercise and get points on something else that you'd otherwise have missed because of taking too long to fix one thing instead of moving on.

So, the things I still don't get are why it would not work, and whether I am trying to fix something that will never work, or whether it's an issue with vIOS and this is the reason it's not working, Maybe the topology and constraints were wrong from the start. Maybe someone else can solve this one, but I need to be strict and move on. With only 72 days to go, I need to spend the time wisely, and not get like a dog with a bone when it comes to an issue.

In the next post, we'll be extending the routes coming into the GETVPN servers, so that we have access to the WWW and ACS servers, and setting up IKEv1 between NYC and the IKEv1 router.