Unetlab - NX-OS (Titanium)

I know I said in the last post that I was saving NX-OS (Titanium) for another day, well, today is that day!

Finding a suitable vmdk file to use was the hardest part.. but in the end I did. I copied it to the tmp directory and tried the usual:

root@iou:/tmp# ls
N7K.vmdk
root@iou:/tmp# qemu-img convert -f vmdk -O qcow2 N7K.vmdk hda.qcow2
root@iou:/tmp# mkdir /p /opt/unetlab/addons/qemu/titanium-7
root@iou:/tmp# mv hda.qcow2 /opt/unetlab/addons/qemu/titanium-7/
root@iou:/tmp# /opt/unetlab/wrappers/unl_wrapper -a fixpermissions
root@iou:/tmp# ls -l /opt/unetlab/addons/qemu/
total 16
drwxr-xr-x 2 root root 4096 Feb 19 14:26 asa-8.42
drwxr-xr-x 2 root root 4096 Feb 19 16:16 titanium-7
drwxr-xr-x 2 root root 4096 Feb 19 13:53 vios-adventerprisek9-m15.4-1.2.0-173
drwxr-xr-x 2 root root 4096 Feb 17 11:02 xrv-k9-5.2.2
root@iou:/tmp#

We have the files loaded and they are visible in Unetlab:

Topology:

Do they run?

Nope. They refuse to start.

From the documentation for CSR1000v it says to run:

# /usr/bin/qemu-system-x86_64 --enable-kvm -serial mon:stdio -nographic -boot order=c,once=d -smp 1 -m 3072 -usb -hda hda.qcow2 -cdrom csr1000v-universalk9.03.13.00.S.154-3.S-ext.iso

However, this is not working for me:

root@iou:/opt/unetlab/addons/qemu/titanium-7# /usr/bin/qemu-system-x86_64 --enable-kvm -serial mon:stdio -nographic -boot order=c,once=d -smp 1 -m 3072 -usb -hda hda.qcow2 -cdrom nexus.iso
Could not access KVM kernel module: No such file or directory
failed to initialize KVM: No such file or directory
root@iou:/opt/unetlab/addons/qemu/titanium-7#

This link would indicate that the module is not installed. I get the same when trying in VirtualBox. I did some googling and tried everything I can, turns out that my bargain ESXi box is missing one vital piece:

So I imported it into VMWare Fusion.
Now the message that the necessary bits arn't working has gone.

If you are having a similar issue then this is the screen that you don't want:

This is what you do want:

I can check that Qemu will be happy by doing this:

More importantly, the NX-OS routers fire up:

We have connectivity...

NXOS1# sh ver
Cisco Nexus Operating System (NX-OS) Software
TAC support: http://www.cisco.com/tac
Documents: http://www.cisco.com/en/US/products/ps9372/tsd_products_support_serie
s_home.html
Copyright (c) 2002-2014, Cisco Systems, Inc. All rights reserved.
The copyrights to certain works contained herein are owned by
other third parties and are used and distributed under license.
Some parts of this software are covered under the GNU Public
License. A copy of the license is available at
http://www.gnu.org/licenses/gpl.html.

Titanium is a demo version of the Nexus Operating System

Software
  loader:    version N/A
  kickstart: version 7.0(1) [build 7.0(1)ZD(0.216)]
  system:    version 7.0(1) [build 7.0(1)ZD(0.216)]
  kickstart image file is: bootflash:///titanium-d1-kickstart.7.0.1.ZD.0.216.bin
  kickstart compile time:  6/13/2014 20:00:00 [06/14/2014 05:45:18]
  system image file is:    bootflash:///titanium-d1.7.0.1.ZD.0.216.bin
  system compile time:     6/13/2014 20:00:00 [06/14/2014 09:32:30]


Hardware
  cisco Nexus7000 C7018 (18 Slot) Chassis ("Unknown Module")
  Unknown CPU with 2042092 kB of memory.
  Processor Board ID TM00010000B

  Device name: NXOS1
  bootflash:          0 kB

Kernel uptime is 0 day(s), 1 hour(s), 7 minute(s), 29 second(s)


plugin
  Core Plugin, Ethernet Plugin
NXOS1#
NXOS1# sh cdp neigh
Capability Codes: R - Router, T - Trans-Bridge, B - Source-Route-Bridge
                  S - Switch, H - Host, I - IGMP, r - Repeater,
                  V - VoIP-Phone, D - Remotely-Managed-Device,
                  s - Supports-STP-Dispute

Device-ID          Local Intrfce  Hldtme Capability  Platform      Port ID
NXOS2(TB00020000B)
                    Eth2/1         141    R S s     N7K-C7018     Eth2/1        
NXOS1# 
NXOS2# sh cdp neigh
Capability Codes: R - Router, T - Trans-Bridge, B - Source-Route-Bridge
                  S - Switch, H - Host, I - IGMP, r - Repeater,
                  V - VoIP-Phone, D - Remotely-Managed-Device,
                  s - Supports-STP-Dispute

Device-ID          Local Intrfce  Hldtme Capability  Platform      Port ID
NXOS1(TB00010000B)
                    Eth2/1         175    R S s     N7K-C7018     Eth2/1        
NXOS2# 

There we have it... NX-OS (Titanium) running inside of Unetlab. So if your virtualization platform supports a KVM guest, and you don't see "Warning: neither Intel VT-x or AMD-V found", then you should be OK.

Automated ASA, IOS and NX-OS backups using SSH, EXPECT and SCP

In the previous post I covered how to perform automated backups of Cisco IOS using EEM (Embedded Event Manager). It's a good script, but not all Cisco devices support EEM. Today we will look at how to backup an ASA, a Catalyst switch and a Nexus switch using an EXPECT script and SCP the backups to a linux server.

You will need a linux host that has SSH installed and EXPECT.

For an Ubuntu box the commands are:

sudo apt-get install ssh
sudo apt-get install expect

On Centos the commands are:

yum install expect
yum install openssh-server

With those installed and a suitable location in mind to store your backups (I am using "/data/site/") we'll crack on.

An expect script performs like a question and answer scenario, so given a few parameters (the answers) we can run a script that calls another process and when prompted, provide the answers it needs.

We will have four scripts in all. One will be the master script, which in turn will call three different scripts.

Assuming you have one ASA, two Nexus switches and two Catalyst switches you'll have one master script that calls the ASA script once, and then makes one call to each of the Nexus, and one to each of the Catalysts. This script will also work well if you have multiple sites.

In this script the naming format of the devices, and the linux machine that they are running from follow the the same format of 802-<SITE-ID>-<DEVICE>.

Naming formats:

Device	UK Name	UK IP	US Name	US IP
Backup Server	802-UK-BCKP01	1.1.1.100	802-UK-BCKP01	2.2.2.100
ASA	802-UK-FW01	1.1.1.254	802-US-FW01	2.2.2.254
Switch 1	802-UK-SW01	1.1.1.2	802-US-SW01	2.2.2.2
Switch 2	802-UK-SW02	1.1.1.3	802-US-SW02	2.2.2.3
Nexus 1	802-UK-NX01	1.1.1.4	802-US-NX01	2.2.2.4
Nexus 2	802-UK-NX02	1.1.1.5	802-US-NX02	2.2.2.5

We have a number of parameters to send with our script, such as the login name and password to connect to the devices, which will be the same across all of the sites, and on the hardware:

admin / Passw0rd

The account used to connect to the Linux box via SCP:

root / Passw0rd

The master script

The general "master" script starts with the standard "#!/bin/bash".

#!/bin/bash

We then have a number of functions, one is called getPassword and here we will specify the account details to log into the devices (loginn and loginp) and the details to perform the SCP copy (SCPUser and SCPPass).

getPassword ()
{
        loginn=admin
        loginp=Passw0rd
        SCPUser=root
        SCPPass=Passw0rd
}

The next function in the script will pass the parameters we set to an expect script to perform the Nexus backup. We then have similar functions to pass the parameters to the scripts that perform the ASA and Catalyst backups.

doNexusBackup ()
{
        echo "Doing Nexus Backup"
        ./NX-expect.sh $loginn $loginp $NX01 $SCPUser $SCPPass $SCPDir $Site $NX01Output
        ./NX-expect.sh $loginn $loginp $NX02 $SCPUser $SCPPass $SCPDir $Site $NX02Output
}

doFirewallBackup ()
{
        echo "Starting Firewall Backup"
        ./FW-expect.sh $loginn $loginp $FW $SCPUser $SCPPass $SCPDir $Site $FWOutput
}

doCatalystBackup ()
{
        echo "Starting Catalyst Backup"
        ./SW-expect.sh $loginn $loginp $CORE01 $SCPUser $SCPPass $SCPDir $Site $Core01Output
        ./SW-expect.sh $loginn $loginp $CORE02 $SCPUser $SCPPass $SCPDir $Site $Core02Output
}

To each script we are passing the login name (admin) and it's password (Passw0rd), then the IP address of the device we are connecting to, as well as the scp details (root, Passw0rd and the IP address of the SCP server). The last three variables we are passing are the directory we want to save our file to, the site (UK or US) and the filename we want to store our backup as.

We then get to the main part of our program. Firstly it's always nice to give our backed up file a date in the filename, this puts it in the format ddmmyy:

DATE=`date +%d%m%y`

Next we get the UK or US out of the hostname on the linux server, which will also form part of the filename for the backup:

Site=`hostname | cut -d- -f 2 | awk '{print tolower($0)}'`

Next we set the file names we want to use in our backup, so the Nexus output would be 802-UK-NX01.110614.bak for the first Nexus in the UK:

NX01Output=802-$Site-NX01.$DATE.bak
NX02Output=802-$Site-NX02.$DATE.bak
Core01Output=802-$Site-Core01.$DATE.bak
Core02Output=802-$Site-Core02.$DATE.bak
FWOutput=802-$Site-FW01.$DATE.bak

Next we call our getPassword function so we have the passwords ready to use, and then we look to see what site we are in, and set our IP addresses accordingly. We then go and call each of the different functions above in turn:

case $Site in
 uk|UK)
        SCPLocation=1.1.1.100
        NX01=1.1.1.4
        NX02=1.1.1.5
        FW=1.1.1.254
        CORE01=1.1.1.2
        CORE02=1.1.1.3
        doNexusBackup
        doFirewallBackup
        doCatalystBackup

 ;;
 us|US)
        SCPLocation=2.2.2.100
        NX01=2.2.2.4
        NX02=2.2.2.5
        FW=2.2.2.254
        CORE01=2.2.2.2
        CORE02=2.2.2.3
        doNexusBackup
        doFirewallBackup
        doCatalystBackup

 ;;
esac

The finished script looks like this:

#!/bin/bash
getPassword ()
{
        loginn=admin
        loginp=Passw0rd
        SCPUser=root
        SCPPass=Passw0rd
}

doNexusBackup ()
{
        echo "Doing Nexus Backup"
        ./NX-expect.sh $loginn $loginp $NX01 $SCPUser $SCPPass $SCPDir $Site $NX01Output
        ./NX-expect.sh $loginn $loginp $NX02 $SCPUser $SCPPass $SCPDir $Site $NX02Output
}

doFirewallBackup ()
{
        echo "Starting Firewall Backup"
        ./FW-expect.sh $loginn $loginp $FW $SCPUser $SCPPass $SCPDir $Site $FWOutput
}

doCatalystBackup ()
{
        echo "Starting Catalyst Backup"
        ./SW-expect.sh $loginn $loginp $CORE01 $SCPUser $SCPPass $SCPDir $Site $Core01Output
        ./SW-expect.sh $loginn $loginp $CORE02 $SCPUser $SCPPass $SCPDir $Site $Core02Output
}


#### Starts here ####
DATE=`date +%d%m%y`
Site=`hostname | cut -d- -f 2 | awk '{print tolower($0)}'`
NX01Output=802-$Site-NX01.$DATE.bak
NX02Output=802-$Site-NX02.$DATE.bak
Core01Output=802-$Site-Core01.$DATE.bak
Core02Output=802-$Site-Core02.$DATE.bak
FWOutput=802-$Site-FW01.$DATE.bak

getPassword
case $Site in
 uk|UK)
        SCPDir=1.1.1.100
        NX01=1.1.1.4
        NX02=1.1.1.5
        FW=1.1.1.254
        CORE01=1.1.1.2
        CORE02=1.1.1.3
        doNexusBackup
        doFirewallBackup
        doCatalystBackup

 ;;
 us|US)
        SCPDir=2.2.2.100
        NX01=2.2.2.4
        NX02=2.2.2.5
        FW=2.2.2.254
        CORE01=2.2.2.2
        CORE02=2.2.2.3
        doNexusBackup
        doFirewallBackup
        doCatalystBackup

 ;;
esac

Our first expect script is for the ASA and it is called "FW-expect.sh". We tell the script that we are going to use expect and to use the variables we are passing to it, and these start at 0:

#!/usr/bin/expect -f

set loginname [lindex $argv 0]
set loginpassword [lindex $argv 1]
set FWIP [lindex $argv 2]
set SCPLogin [lindex $argv 3]
set SCPPassword [lindex $argv 4]
set SCPLocation [lindex $argv 5]
set Site [lindex $argv 6]
set FWOutput [lindex $argv 7]

We then set a time out and spawn an ssh connection using the login name and firewall IP, and get it to the enable prompt. We use \n to signify that we are pressing enter:

set timeout 30
spawn ssh $loginname@$FWIP
expect "password:"
send "$loginpassword\n"
expect ">"
send "en\n"
expect "Password:"
send "$loginpassword\n"
expect "#"

The last part of the script performs the backup, using the command "copy run scp:" and passes the login details. For the most part we can send a carriage return to accept the default values. Where we have "expect "?"" we are being prompted for the source filename and the default is the running-config, so we just want to hit enter. Similarly where the script says "expect "Address"" we have already passed this and the firewall will be prompting us to accept the default value of the SCP servers IP address that was passed in the variable SCPLocation. We then pass the login name and password for the root account on the SCP server, and then pass the directory and filename that we want to save the file to, which would be translated to /backup/UK/802-UK-FW01.110614.bak. Lastly we exit from our SSH session.

send "copy run scp://$SCPLogin@$SCPLocation\n"
expect "?"
send "\n"
expect "Address"
send "\r"
expect "username"
send "$SCPLogin:$SCPPassword\n"
expect "filename"
send "/backup/$Site/$FWOutput\n"
expect "#"
send "exit\n"

The finished script looks like this:

#!/usr/bin/expect -f

set loginname [lindex $argv 0]
set loginpassword [lindex $argv 1]
set FWIP [lindex $argv 2]
set SCPLogin [lindex $argv 3]
set SCPPassword [lindex $argv 4]
set SCPLocation [lindex $argv 5]
set Site [lindex $argv 6]
set FWOutput [lindex $argv 7]

set timeout 30
spawn ssh $loginname@$FWIP
expect "password:"
send "$loginpassword\n"
expect ">"
send "en\n"
expect "Password:"
send "$loginpassword\n"
expect "#"
send "copy run scp://$SCPLogin@$SCPLocation\n"
expect "?"
send "\n"
expect "Address"
send "\r"
expect "username"
send "$SCPLogin:$SCPPassword\n"
expect "filename"
send "/backup/$Site/$FWOutput\n"
expect "#"
send "exit\n"

The expect scripts for the Catalyst (or any IOS) which is called "SW-expect.sh" looks very similar. We are passing the same variables, and the challenge/responses are very similar. So I have just included the whole script:

#!/usr/bin/expect -f

set loginname [lindex $argv 0]
set loginpassword [lindex $argv 1]
set Switchnum [lindex $argv 2]
set SCPLogin [lindex $argv 3]
set SCPPassword [lindex $argv 4]
set SCPLocation [lindex $argv 5]
set Site [lindex $argv 6]
set CoreOutput [lindex $argv 7]

set timeout 30
spawn ssh $loginname@$Switchnum
expect "Password:"
send "$loginpassword\n"
expect "#"
send "copy run scp:\n"
expect "Address"
send "$SCPLocation\n"
expect "username"
send "$SCPLogin\n"
expect "filename"
send "/backup/$Site/$CoreOutput\n"
expect "Password:"
send "$SCPPassword\n"
expect "#"
send "exit\n"

The Nexus backup script (NX-expect.sh) is also very similar, the main difference being that we need to specify the vrf we are using for the backup:

#!/usr/bin/expect -f

set loginname [lindex $argv 0]
set loginpassword [lindex $argv 1]
set NXnum [lindex $argv 2]
set SCPLogin [lindex $argv 3]
set SCPPassword [lindex $argv 4]
set SCPLocation [lindex $argv 5]
set Site [lindex $argv 6]
set NXOutput [lindex $argv 7]

set timeout 30
spawn ssh $loginname@$NXnum
match_max 100000
expect "Password:"
send "$loginpassword\n"
expect "#"
send "copy run scp://$SCPLogin@$SCPLocation/backup/$Site/$NXOutput vrf management\n"
expect "password:"
send "$SCPPassword\n"
expect "#"
send "exit\n"
interact

Because this script uses SSH we need to have the SSH keys stored so that when the script is called it won't sit there expecting you to accept the SSH certificate. Before running the script you should manually SSH to the device as the user that the script will run under - so from the root account if the script is going to run as root SSH to the device, or do SSH using sudo. If you don't do this part the script won't accept the SSH key from the device and the script will not run.

Once you have all the scripts they need to be executable (so do "chmod a+x" on each of them, or "sudo chmod a+x").

You can then add an entry to the crontab to run the scripts at a time of your choosing.

Hope you find this useful, you might want to include some error checking and confirmation through email into the script. Because of the inclusion of the site in the backup file location it's then quite easy to perform an rsync to keep offsite copies on the other data center.

Cisco Nexus and AAA authentication using Radius on Microsoft 2008 NPS

In a previous post I wrote about how to integrate Cisco IPS modules with Microsoft 2008 NPS server, for Radius authentication.

Now we are going to cover how to integrate Cisco Nexus with radius. The format is very similar to the IPS setup, so it may be worth having a read of the first post to get an idea.

We start with some basic assumptions, and one caveat:

Your basic Nexus switch configuration is already in place and can ping your NPS server (via the management vrf)
You already have an NPS server in place serving clients.

I am using the Cisco Titanium Nexus 7000 emulator (but the same process should apply to the NX5000 series, I need to do this on real Nexus 5000's so if there are any differences I will update this post).

Nexus client and profile settings on Microsoft 2008 NPS

We start by adding a client onto the NPS, we give it a friendly name, specify the IP address and set the radius secret (here I am using nxnps123). I have also set the vendor name to Cisco.

And now we have a client set up:

Now we create a policy to map access to the client. I have called this "TestNexus Admin", and the plan will be to have a read only policy added later on.

In the next window I start to specify the conditions, and will use the security group "sec-FW-admin", so click on Add to add a condition and select "Windows Groups"

Now you can add your specific groups.
Next I add the Client Friendly Name, and use the same name I called the client:

We keep the default of Access granted and move on till we see the "Configure Authentication Methods", here we select just PAP and SPAP:

We can skip the "Configure Restraints" window and move on to "Configure Settings". Here we remove the two options under "Standard"

And now we can add a Vendor Specific entry:

And for this entry we will use "shell-roles=*admin" (before anyone says this is wrong, please read the rest of the post to see why I havn't corrected this yet...)

And that's all the configuration on the Microsoft side (for the moment at least).

Nexus Radius setup and AAA Authentication

Just in case you havn't set up the basics on the Nexus the screenshots below show how to set the management vrf IP, and default routing, as well as confirming reachability to the NPS server:

Notice here that we have to specify vrf management in the ping command for it to work

Now we know that we can "talk" to the NPS box we can start setting up the radius parts.

We start by setting the radius key, it should match the key used to set up the client under NPS (again here we are using "nxnps123"). The 0 next to "key" means that its unencrypted.

Then we set the host (which we should have at least two of for redundancy), and create an aaa group and add the server to this. the last command tells the Nexus to use the management vrf to communicate with the server.

Now we can tell the Nexus to use radius for authentication, and we also tell it to keep track of errors:

Finally, just in case our Radius server is down for any reason, the Nexus should use its local database for login:

Now we can test login!

It works!

But we can't stop here. Like I pointed out earlier there was an issue with the shell:roles command within the NPS setup.

With the existing configuration we try saving the config:

So lets look at the privilege levels:

Well, -1 was never a good thing in my book. So I changed the AV-pair to "shell:roles=*"network-admin vdc-admin"", logged out, and back in again:

Now although the displayed privilege level is still showing -1, we can save the config.

Lastly I copied the profile in NPS, changed the Windows Group to one that has people we want with just read only access in it, and changed the role to network operator:

And again we test, this time we are expecting the copy run start to fail

Which it does, but they can still issue show commands, so the achieves exactly what is required.

Fallback on Nexus

Lastly we need to make sure that if the radius server is down, we can still get in. I stopped the NPS service and tried logging in. Login failed. I reconnected and tried logging in with the admin username and password, and got in: