Cisco Nexus and AAA authentication using Radius on Microsoft 2008 NPS

In a previous post I wrote about how to integrate Cisco IPS modules with Microsoft 2008 NPS server, for Radius authentication.

Now we are going to cover how to integrate Cisco Nexus with radius. The format is very similar to the IPS setup, so it may be worth having a read of the first post to get an idea.

We start with some basic assumptions, and one caveat:

Your basic Nexus switch configuration is already in place and can ping your NPS server (via the management vrf)
You already have an NPS server in place serving clients.

I am using the Cisco Titanium Nexus 7000 emulator (but the same process should apply to the NX5000 series, I need to do this on real Nexus 5000's so if there are any differences I will update this post).

Nexus client and profile settings on Microsoft 2008 NPS

We start by adding a client onto the NPS, we give it a friendly name, specify the IP address and set the radius secret (here I am using nxnps123). I have also set the vendor name to Cisco.


Cisco NPS as a Radius client

Cisco NPS as a Radius client

And now we have a client set up:


Cisco NPS as a Radius client

Now we create a policy to map access to the client. I have called this "TestNexus Admin", and the plan will be to have a read only policy added later on.


Nexus Radius policy in NPS

In the next window I start to specify the conditions, and will use the security group "sec-FW-admin", so click on Add to add a condition and select "Windows Groups"


Specifying windows groups for NPS

Now you can add your specific groups.
Next I add the Client Friendly Name, and use the same name I called the client:


Specifying client friendly name in NPS

We keep the default of Access granted and move on till we see the "Configure Authentication Methods", here we select just PAP and SPAP:

Cisco Radius authentication PAP

We can skip the "Configure Restraints" window and move on to "Configure Settings". Here we remove the two options under "Standard"

AAA radius properties

And now we can add a Vendor Specific entry:


Cisco AV pair Nexus radius

And for this entry we will use "shell-roles=*admin" (before anyone says this is wrong, please read the rest of the post to see why I havn't corrected this yet...)


Cisco av-pair NPS Nexus

And that's all the configuration on the Microsoft side (for the moment at least).

Nexus Radius setup and AAA Authentication

Just in case you havn't set up the basics on the Nexus the screenshots below show how to set the management vrf IP, and default routing, as well as confirming reachability to the NPS server:

Nexus basic setup for management

Notice here that we have to specify vrf management in the ping command for it to work

Specifying routing in Cisco Nexus

Now we know that we can "talk" to the NPS box we can start setting up the radius parts.

We start by setting the radius key, it should match the key used to set up the client under NPS (again here we are using "nxnps123"). The 0 next to "key" means that its unencrypted.

Then we set the host (which we should have at least two of for redundancy), and create an aaa group and add the server to this. the last command tells the Nexus to use the management vrf to communicate with the server.


Cisco Nexus radius setup

Now we can tell the Nexus to use radius for authentication, and we also tell it to keep track of errors:


Cisco Nexus AAA authentication setup

Finally, just in case our Radius server is down for any reason, the Nexus should use its local database for login:


Cisco Nexus AAA authentication local fallback
Now we can test login!


Using radius for authentication on Cisco Nexus
It works!

But we can't stop here. Like I pointed out earlier there was an issue with the shell:roles command within the NPS setup.

With the existing configuration we try saving the config:


Cisco Nexus permission denied

So lets look at the privilege levels:


Cisco Nexus privilege levels

Well, -1 was never a good thing in my book. So I changed the AV-pair to "shell:roles=*"network-admin vdc-admin"", logged out, and back in again:


Cisco Nexus copy run start

Now although the displayed privilege level is still showing -1, we can save the config.

Lastly I copied the profile in NPS, changed the Windows Group to one that has people we want with just read only access in it, and changed the role to network operator:


Cisco Nexus radius read-only network-operator

And again we test, this time we are expecting the copy run start to fail


Cisco Nexus Network Operator privilege

Which it does, but they can still issue show commands, so the achieves exactly what is required.

Fallback on Nexus

Lastly we need to make sure that if the radius server is down, we can still get in. I stopped the NPS service and tried logging in. Login failed. I reconnected and tried logging in with the admin username and password, and got in:


Fallback to local authentication if Radius server is down

TFTPDNLD in ROMMON mode and licensing flash blocks

As you will see from the INE topology the Cisco 1841's (R4, R5, and R6) require 256MB memory and 64MB flash. Generally the 1841's come with the base 128MB memory and 32MB flash (or no flash at all if you unlucky). So I purchased a three 128MB SODIMMs and three 32MB Compact Flash cards to put in them.

Installing memory in an 1841 router

Installing the memory is easy enough, just take out the one screw and open the case up and install. If you have every installed any memory sticks before then this won't prove complicated. The flash install is just a case of popping out the old one and installing the new one.

You can find Cisco's page on installing the memory here.

The dreaded ROMMON mode

When you first power on the router with the new flash card in it, it will (obviously) be empty, and you will be dropped into the ROMMON mode.

There is the option in ROMMON to download a file from a tftp server, and to do this is relatively straight forward, but the commands are kind of hidden.

If you do a ? at the rommon mode you will see an option for tftpdnld, but first we have to set the system up for the network. The commands to use won't show up if you just do a ? at the rommon prompt, but if you type tftpdnld then you will see what you need to enter.

First we start with setting the IP address, at this stage make sure you have plugged a network cable into the first network port (0/0):

rommon 10 > IP_ADDRESS=192.168.5.250

Then set the subnet mask:

rommon 11 > IP_SUBNET_MASK=255.255.240.0

Then set the default gateway:

rommon 12 > DEFAULT_GATEWAY=192.168.2.1

Now we move on to specifying the tftp details

rommon 13 > TFTP_SERVER=192.168.3.241

And the file:

rommon 14 > TFTP_FILE=c1841-adventerprisek9-mz.124-24.T1.bin

And lastly we start the process by entering the tftpdnld command again:

rommon 15 > tftpdnld

Your screen should look something like this:

tftp setup for rommon tftpdnld

You should see some nice output summarizing your settings, and prompting you to continue:

tftpdnld rommon confirm

And once you confirm that you are happy with the settings rommon should start to download the file you have specified, validate the checksum, and copy it to flash:

loading image from tftp in rommon

Once its done you can type "reset" to restart the router, and, if all is good you will be able to see the router boot up properly, using the new image:

tftp rommon reset

Licensing flash block error

I did get the following errors though "licensing flash block 0 needs to be initialized" and "licensing flash block 1 needs to be initialized":

licensing flash block 0 needs to be initialized

But this is just on the first boot, and once I reloaded the router, the message did not reappear.

CCIE, CCNA and CCNP Salaries and demand

It's fair to say that many companies have felt the pinch in this last world wide recession. Certainly Information Technology (IT) in all it's forms has not been excluded from this, jobs have been outsourced, or shared amongst other team members as another is let go, or cheaper workers have taken their place. As a personal comment, I went from a very well paying job in the city to a job in the suburbs paying half of what I was on after being made redundant, thankfully I have bounced back, but times were tough, and i know of many others in a similar situation.

Most information technology salaries have, over the last few years, dropped considerably, when compared to the pre-recession era. So how has the number of CCIE jobs, the Cisco salary scale, and especially the CCIE salary scale faired?

CCIE Salary compared to CCNA and CCNP

Looking at the data listed on itjobswatch.co.uk we can see that in 2011 the average CCIE salary offered was in the region of £57,500, in 2012 this dropped to £57,000 and has risen back to the 2011 figure in 2013, so thankfully the last couple of years have been kind to the CCIE certified!  

Over the last 9 years the CCIE salary scale has actually changed very little, which is good news for anyone trying to justify the cost of the CCIE exams vs the potential salary increase.

CCIE job Salary scale
CCIE Salary scale
Now if we compare this against the CCNA and CCNP job salary scales, again we see very little change over the same period:

CCNA job Salary scale
CCNA Salary scale
CCNP job Salary scale
CCNP Salary scale
We can see that there has been very little movement within the CCNA and CCNP job salaries on offer, in fact the CCNA is showing a very steady climb.

Number of CCIE, CCNA and CCNP roles

Although the CCIE salary has stayed around the same, the number of roles available has fluctuated dramatically:

CCIE role availability
Number of CCIE roles as a percentage of total roles
Granted, we are only talking about a couple of percentage of overall IT jobs, but there is still a considerable change. Now let's look at the CCNA and CCNP for comparison.

CCNA role availability
CCNA roles as percentage of total
CCNP role availability
CCNP roles as percentage of total
So over all we can see that the recession as it was in the period 2009 to 2011 is now slowly recovering and demand is, once again, climbing back to pre-recession levels.

Summary

The number of roles requiring any form of Cisco certification has risen and fallen in-line with the recession. The salaries for the CCNA, CCNP and CCIE have only changed by a few percent over the years and are, in general, on the rise again, which is good news for all Cisco certified people!

We can see that the average CCNA salary is around £37,000, this rises to around £46-47,000 with the CCNP qualification and then to £57,000 with the CCIE certification. So this does show that all the study, hard work and dedication will pay off! 

Fun with QinQ tunnels - Part 2 (Routing different subnets)

From part one, we know that the purpose of a QinQ tunnel is to extend a VLAN across the WAN or network. But what if site 1 is connected to site 2 and they use different IP schemes (such as 10.1.250.0/24 and 10.100.250.0/24)? Well, making the two talk is actually very simple.

The way I have configured this is to set up a loopback interface on each side to emulate the different network, and set up routes between them. If you recall from part 1 I am using a 3550 on one side and a 3750 on the other.

We start by enabling IP routing on both sides:
3550(config)#ip routing
3750(config)#ip routing

Then we assign a virtual IP address to the VLAN:

         3550(config)#int vlan 501
         3550(config-if)#ip address 10.250.1.21 255.255.255.0



         3750(config)#int vlan 501
         3750(config-if)#ip address 10.250.1.20 255.255.255.0



And then we create our loopback interfaces to emulate the different networks:

        3550(config)#interface Loopback2
        3550(config-if)#ip address 10.1.250.1 255.255.255.0


        3750(config)interface Loopback2
        3750(config-if)#ip address 10.100.250.1 255.255.255.0



Lastly we set a routing statement with the destination network and the destination IP address, which is the virtual IP address of the vlan at the other side
3550(config)#ip route 10.100.250.0 255.255.255.0 10.250.1.20
3750(config)#ip route 10.1.250.0 255.255.255.0 10.250.1.21
Now we should be able to ping the loopback in each site:


QinQ routing different subnets
QinQ routing different subnets

 Pretty neat!

Fun with QinQ tunnels - Part 1


A QinQ tunnel extends a VLAN across the network, or the internet. The usual way this is done is by having a standard VLAN in your network connecting to a QinQ tunnel in the service provider network at both ends.

This allows mutiple VLANs in your network to be encapsulated within another VLAN across the demarc boundaries, and back into your network at another site.

There are some prerequisites with setting up a QinQ, in that the MTU size must be increased to accommodate the larger packet size. You also need a switch that supports them, for this I used a 3560 and a 3750, both running Advanced IP Services, the Inside switches (in the first diagram are 3550s, and in the final diagram we used the same 3750).

The basic diagram looks like this:
QinQ tunnels basic setup



A standard Trunk port from the Inside Switch (e0/1) connects to the QinQ trunk on the provider switch (e0/1), which then connects to the other provider switch via another standard trunk (VLAN 10, e0/10 - e0/10), and finally a QinQ tunnel port on the other Provider switch (e0/1) connects to a standard trunk port (e0/1) on the other Inside switch. The dotted line shows how the switches, and the end user, see the link.

The configuration of the switches would be as follows:

Inside switch (left hand side)

vlan 4
  name Client_VLAN
vlan 5
  name Server_VLAN
vlan 6
  name Other_VLAN

int e0/1
  description **** Link to QinQ ****
  switchport trunk encapsulation dot1q
  switchport trunk allowed vlan 4,5,6
  switchport mode trunk

int e0/4
  switchport access vlan 4 

int e0/5
  switchport access vlan 5

int e0/6
  switchport access vlan 6

Provider Switch (left hand side)

system mtu 1998
system mtu jumbo 9000
vlan 10
  name QinQ_VLAN

int e0/1
  description **** QinQ VLAN ****
  switchport access vlan 10
  switchport trunk encapsulation dot1q
  switchport mode dot1q-tunnel
  no keepalive
  l2protocol-tunnel cdp
  l2protocol-tunnel stp

int e0/10
  description **** Provider to Provider link ****
  switchport trunk encapsulation dot1q
  switchport trunk allowed vlan 10
  switchport mode trunk

Provider Switch (right hand side)

system mtu 1998
system mtu jumbo 9000
vlan 10
  name QinQ_VLAN

int e0/1
  description **** QinQ VLAN ****
  switchport access vlan 10
  switchport trunk encapsulation dot1q
  switchport mode dot1q-tunnel
  no keepalive
  l2protocol-tunnel cdp
  l2protocol-tunnel stp

int e0/10
  description **** Provider to Provider link ****
  switchport trunk encapsulation dot1q
  switchport trunk allowed vlan 10
  switchport mode trunk

Inside switch (right hand side)

vlan 4
  name Client_VLAN
vlan 5
  name Server_VLAN
vlan 6
  name Other_VLAN

int e0/1
  description **** Link to QinQ ****
  switchport trunk encapsulation dot1q
  switchport trunk allowed vlan 4,5,6
  switchport mode trunk

int e0/4
  switchport access vlan 4

int e0/5
  switchport access vlan 5

int e0/6
  switchport access vlan 6



Now if you attach a laptop to the same ports on both sides and assign an IP address to both laptops (say 10.1.1.10/24 and 10.1.1.11/24) they should be able to ping each other.

The above is an in-an-ideal-world scenario. Really you just want to be able to configure standard trunk links on your equipment and have the service provider take care of all the QinQ configuration. But sometimes what you get is slightly different. And what we got was this:
QinQ tunnels advanced setup


Now our options were to either (A) purchase a new switch so we can replicate the layout in the first picture, or (B) try and find a way of having the QinQ settings and the trunk settings on the same switch. Option A would cost quite a bit of money, but is option B possible? Can a QinQ tunnel exist on the same switch as the trunk? The dangers are that it won't work due to loopguard and bpduguard. But it's worth a shot, right?

Turns out that it is, and all it takes is one little ethernet cable, now connected from port e0/1 to e0/2. Port e0/10 is then used to link up to the provider switch at the other site.

The settings for the switches on the left hand side remain the same, but what we have done is loop a cable from one port back into another port. 

QinQ tunnels using a loopback and one switch

So now all of the config for both the right hand side switches goes into the one switch (we used a 3750):

system mtu 1998
system mtu jumbo 9000

vlan 10
  name QinQ_VLAN
vlan 4
  name Client_VLAN
vlan 5
  name Server_VLAN
vlan 6
  name Other_VLAN

int e0/1
  description **** Link to QinQ ****
  switchport trunk encapsulation dot1q
  switchport trunk allowed vlan 4,5,6
  switchport mode trunk

int e0/2
  description **** QinQ VLAN ****
  switchport access vlan 10
  switchport trunk encapsulation dot1q
  switchport mode dot1q-tunnel
  no keepalive
  l2protocol-tunnel cdp
  l2protocol-tunnel stp

int e0/4
  switchport access vlan 4 

int e0/5
  switchport access vlan 5

int e0/6
  switchport access vlan 6

int e0/10
  description **** Provider to Provider link ****
  switchport trunk encapsulation dot1q
  switchport trunk allowed vlan 10
  switchport mode trunk

Traffic will (effectively) come in on e0/1, get encapsulated in e0/2 and traverse to the other side via e0/10, again we tested via ping and all was good.

So it turns out that you can have your QinQ trunk and the VLAN trunk living on the same switch.

Part two answers the question "Can we route different subnets across a QinQ link?"
Subnetting - Part 4: Route Summarization (2.10)

Subnetting - Part 4: Route Summarization (2.10)

In the final part of this series we are going to cover Route Summarization. If you havn't read the other parts of the series then we covered Standard Length Subnet Masks (SLSM), how to find all the subnets in a given network and prefix, and then we moved on to Variable Length Subnet Masks (VLSM). They are always worth a bit of a recap at any stage.

So route summarization is very similar to finding all the subnets within a network and a prefix. In fact much of the logic behind it is exactly the same.

Again we can do this through the binary method (which I actually think is easier for this) or the decimal method.

In the binary method we follow this process:

1. Write down the binary version of each subnet
2. Look to see where the numbers start to differ and draw a line down that portion.
3. Copy the bits that are the same into a new line and write 0s for the remaining bits. 
4. Convert the result from step 3 back into decimal.

We could use the same subnets as we used in part 2 of this series, but that would be a little too easy, so lets use (as we have done so far) the example from the Odom book and use the subnets of 172.31.20.0, 172.31.21.0, 172.31.22.0, and 172.31.23.0 with a /24 prefix.



Subnet Octet 1 Octet 2 Octet 3 Octet 4
172.31.20.0/24 10101100 00011111 00010100 00000000
172.31.21.0/24 10101100 00011111 00010101 00000000
172.31.22.0/24 10101100 00011111 00010110 00000000
172.31.23.0/24 10101100 00011111 00010111 00000000
Summary 10101100 00011111 00010100 00000000

So now we convert the summary address back into decimal and we have a prefix length of /22 and an address of 172.31.20.0

In the decimal method we do the following:

1. Count the number of subnets, then find the smallest value of y where 2y equals the number of subnets. So for four subnets 22 = 4, therefore y = 2
2. Subtract Y from the longest prefix length of the component subnets (with /24 subnets this would be 24 - 2 = 22)
3. Taking the lowest numeric subnet number in the component subnets, and then using it as an IP address calculate the subnet that this would live in (i.e. 172.31.20.0/22).
4. Repeat step 3 this time using the largest component subnet. If the resulting subnet matches step 3 then this is the best summarized route. (172.31.23.0 with a /22 gets us to 172.31.20.0/22)
5. If step 4 does not get the best result repeat steps 3 and 4 again, this time subtracting 1 from the previously used prefix length.

i do think that the binary method is clearly easier than the decimal method here, but then everyone is different and you may find that the decimal method works better for you.

And that wraps up this series on subnetting. Again I will be posting more examples in the near future.


Subnetting - Part 3: VLSM (2.10)

Subnetting - Part 3: VLSM (2.10)

In the third part of this series we will look at how we can take on subnet block and divide it into a range of subnets of differing size based on our requirements. If you need a refresher on the basics of SLSM then check out the first part of this series here.

Variable Length Subnet Masks (VLSM) is one of the more complicated aspects of subnetting to understand. With VLSM we take a network block and subdivide into a number of smaller networks.

We start off with the standard concepts for SLSM, and once we know the shortest prefix length and from there we can start to subdivide based on our requirements.

The process is as follows:

1: Find the shortest prefix length that will cover all of our required subnets.

2: Divide the available address block into prefixes of equal size based on step 1.

3: Knowing the subnets we need we allocate these to the beginning of the address block, leaving equal-sized address blocks at the end of the original block.

4: The first unallocated address block will then be subdivided by repeating steps 1 - 3, using the shortest requires prefix length for the remaining subnets.

5: If we need to allocate very small address blocks (say router to router i.e. /30) this should be done at the very end of the address range, which gives us some scope for future usage if requirements change later.

Confused yet? Imagine it as a funnel, with our assigned address block at the top, and as we assign subnets we start to reach the end of the funnel.

Again I will use the same IP's used in the Odom book, and if you havn't already got the book then you should go and buy it. 

So we have an assigned address range of 172.31.28.0/23. This will cover 172.31.28.0 to 172.31.29.255. Here we have one clearly defined summarized route, but our requirements are to split this as follows:

3 /25's
2 /27's
3 /30's

We can divide this into four blocks, which gives us four /25 subnets, and here we are using the same math as we used in part one with SLSM:

172.31.28.0/25 (172.31.28.1 - 172.31.28.128)
172.31.28.128/25 (172.31.28.129 - 172.31.28.254)
172.31.29.0/25 (172.31.29.1 - 172.31.29.128)
172.31.29.128/25 (172.31.29.129 - 172.31.29.254)

This takes care of the first requirement (3 lots of /25's). Now using the last subnet (172.31.29.128/25) we can start allocating the /27's.

These will be 

172.31.29.128/27
172.31.29.160/27
172.31.29.192/27
172.31.29.224/27

We stop at 172.31.29.224 because as we are incrementing by 32 the next subnet in the range would be 172.31.29.256 which is an invalid subnet. The first two subnets will take care of the second requirement (2 lots of /27's), and we have two blocks left over. As per step 5 we know that the the final requirement is for 3 lots of /30's so we leave the third subnet unallocated, which we can use at a later stage, instead we will focus on the 172.31.29.224/27 subnet instead.

The available address range for this subnet is 172.31.29.225 - 172.31.29.254. This can be divided into our /30's with a large number of blocks remaining, in fact we end up with eight blocks. A /30 subnet has four IPs in it, with two usable, so we just increment by 4 to find the underlying subnets.

We end up with the following subnets:

172.31.29.224/30
172.31.29.228/30
172.31.29.232/30
172.31.29.236/30
172.31.29.240/30
172.31.29.244/30
172.31.29.248/30
172.31.29.252/30

This gives us plenty of scope to have the final requirement (3 /30's) which will be the final three subnets in the list (remembering step 5 to use the end of the block, and leave the beginning of the block unallocated for future usage).

And there we have VLSM. It's really not that scary.

Our final post in this series will cover route summarization, and I will post some more examples of SLSM, VLSM, Route summarization and how to find all the subnets within a given network subnet in the resources section later.
Subnetting - Part 2: Finding all the subnets within a network (2.10)

Subnetting - Part 2: Finding all the subnets within a network (2.10)

Following on from part one of this series where we went through the basics of subnetting, and we are going to expand on that by finding all of the available subnets within an class and prefix.

Again this can be found either using the binary or decimal ways, and we are going to go through both here.

Finding all the subnets - Binary

1. Start by writing down the binary version of the classful network.
2. Separate the network and subnet parts of the number with one line, and the subnet and host parts with another line (I have actually bolded the number where the lines would be).
3. Calculate the number of subnets (including the zero-subnet and the broadcast subnet) using the method 2y where y is the number of subnet bits.
4. Write down y-1 copies of the binary network below the first one, leaving the subnet field blank.
5. Using the subnet field increment the values (001, 010, 011 etc)
6. Convert the binary numbers back to decimal

Again we are using the Odom book as the basis of this, so we are using the class B network 172.31.0.0, and a subnet mask of 255.255.224.0. we are using 3 subnet bits so there will be 23 subnets (if your algebra is as good as mine then this is the same as 2*2 = 4, 4*2 = 8).


Subnet Octet 1 Octet 2 Octet 3 Octet 4
Network number/Subnet 0 10101100 00011111 00000000 00000000
2nd Subnet 10101100 00011111 00100000 00000000
3rd Subnet 10101100 00011111 01000000 00000000
4th Subnet 10101100 00011111 01100000 00000000
5th Subnet 10101100 00011111 10000000 00000000
6th Subnet 10101100 00011111 10100000 00000000
7th Subnet 10101100 00011111 11000000 00000000
8th Subnet (Broadcast) 10101100 00011111 11100000 00000000

Converting these back into decimal we would see that the networks within this subnet are: 172.31.0.0, 172.31.32.0, 172.31.64.0, 172.31.96.0, 172.31.128.0, 172.31.160.0, 172.31.192.0 and 172.31.224.0.

Finding all the subnets - Decimal

The decimal way is very similar. As we saw with SLSM we look at the "interesting" octet. We know that 256-224 = 32 so we increment based on that.

So let's look at this in a bit more detail.

1. Start with the classful network number
2. For the first subnet number, copy the entire network number - this is subnet zero
3. Find the octet that contains the entire subnet field (the interesting octet)
4. find the magic number by subtracting this octet from 256
5. Copy down the non-interesting octets onto the next line as the next subnet number. There should only be one octet that's missing a value at this stage.
6. Start adding the magic number to the previous octet value, adding again and again as you go down the list
7. Repeat 5 and 6 until you get to 256. This subnet is not valid, so the one before this one will be the last valid one, and also the broadcast subnet.



Octet Comments
1 2 3 4
Network Number 172 31 0 0 Step 1
Mask 255 255 224 0 256-224=32
Subnet Zero 172 31 0 0 Step 2
1st Subnet 172 31 32 0 0+32 (Step 5 and 6)
2nd Subnet 172 31 64 0 32+32
3rd Subnet 172 31 96 0 64+32
4th Subnet 172 31 128 0 96+32
5th Subnet 172 31 160 0 128+32
6th Subnet 172 31 192 0 160+32
7th Subnet 172 31 224 0 192+32 (Broadcast subnet)
Invalid Subnet 172 31 256 0 Invalid

I'll post some more examples on the subnetting PDF that will follow the last part of this series shortly. But before that we have VLSM and Route Summarisation to complete.