Automated ASA, IOS and NX-OS backups using SSH, EXPECT and SCP

In the previous post I covered how to perform automated backups of Cisco IOS using EEM (Embedded Event Manager). It's a good script, but not all Cisco devices support EEM. Today we will look at how to backup an ASA, a Catalyst switch and a Nexus switch using an EXPECT script and SCP the backups to a linux server.

You will need a linux host that has SSH installed and EXPECT.

For an Ubuntu box the commands are:

sudo apt-get install ssh
sudo apt-get install expect

On Centos the commands are:

yum install expect
yum install openssh-server

With those installed and a suitable location in mind to store your backups (I am using "/data/site/") we'll crack on.

An expect script performs like a question and answer scenario, so given a few parameters (the answers) we can run a script that calls another process and when prompted, provide the answers it needs.

We will have four scripts in all. One will be the master script, which in turn will call three different scripts.

Assuming you have one ASA, two Nexus switches and two Catalyst switches you'll have one master script that calls the ASA script once, and then makes one call to each of the Nexus, and one to each of the Catalysts. This script will also work well if you have multiple sites.

In this script the naming format of the devices, and the linux machine that they are running from follow the the same format of 802-<SITE-ID>-<DEVICE>.

Naming formats:

Device	UK Name	UK IP	US Name	US IP
Backup Server	802-UK-BCKP01	1.1.1.100	802-UK-BCKP01	2.2.2.100
ASA	802-UK-FW01	1.1.1.254	802-US-FW01	2.2.2.254
Switch 1	802-UK-SW01	1.1.1.2	802-US-SW01	2.2.2.2
Switch 2	802-UK-SW02	1.1.1.3	802-US-SW02	2.2.2.3
Nexus 1	802-UK-NX01	1.1.1.4	802-US-NX01	2.2.2.4
Nexus 2	802-UK-NX02	1.1.1.5	802-US-NX02	2.2.2.5

We have a number of parameters to send with our script, such as the login name and password to connect to the devices, which will be the same across all of the sites, and on the hardware:

admin / Passw0rd

The account used to connect to the Linux box via SCP:

root / Passw0rd

The master script

The general "master" script starts with the standard "#!/bin/bash".

#!/bin/bash

We then have a number of functions, one is called getPassword and here we will specify the account details to log into the devices (loginn and loginp) and the details to perform the SCP copy (SCPUser and SCPPass).

getPassword ()
{
        loginn=admin
        loginp=Passw0rd
        SCPUser=root
        SCPPass=Passw0rd
}

The next function in the script will pass the parameters we set to an expect script to perform the Nexus backup. We then have similar functions to pass the parameters to the scripts that perform the ASA and Catalyst backups.

doNexusBackup ()
{
        echo "Doing Nexus Backup"
        ./NX-expect.sh $loginn $loginp $NX01 $SCPUser $SCPPass $SCPDir $Site $NX01Output
        ./NX-expect.sh $loginn $loginp $NX02 $SCPUser $SCPPass $SCPDir $Site $NX02Output
}

doFirewallBackup ()
{
        echo "Starting Firewall Backup"
        ./FW-expect.sh $loginn $loginp $FW $SCPUser $SCPPass $SCPDir $Site $FWOutput
}

doCatalystBackup ()
{
        echo "Starting Catalyst Backup"
        ./SW-expect.sh $loginn $loginp $CORE01 $SCPUser $SCPPass $SCPDir $Site $Core01Output
        ./SW-expect.sh $loginn $loginp $CORE02 $SCPUser $SCPPass $SCPDir $Site $Core02Output
}

To each script we are passing the login name (admin) and it's password (Passw0rd), then the IP address of the device we are connecting to, as well as the scp details (root, Passw0rd and the IP address of the SCP server). The last three variables we are passing are the directory we want to save our file to, the site (UK or US) and the filename we want to store our backup as.

We then get to the main part of our program. Firstly it's always nice to give our backed up file a date in the filename, this puts it in the format ddmmyy:

DATE=`date +%d%m%y`

Next we get the UK or US out of the hostname on the linux server, which will also form part of the filename for the backup:

Site=`hostname | cut -d- -f 2 | awk '{print tolower($0)}'`

Next we set the file names we want to use in our backup, so the Nexus output would be 802-UK-NX01.110614.bak for the first Nexus in the UK:

NX01Output=802-$Site-NX01.$DATE.bak
NX02Output=802-$Site-NX02.$DATE.bak
Core01Output=802-$Site-Core01.$DATE.bak
Core02Output=802-$Site-Core02.$DATE.bak
FWOutput=802-$Site-FW01.$DATE.bak

Next we call our getPassword function so we have the passwords ready to use, and then we look to see what site we are in, and set our IP addresses accordingly. We then go and call each of the different functions above in turn:

case $Site in
 uk|UK)
        SCPLocation=1.1.1.100
        NX01=1.1.1.4
        NX02=1.1.1.5
        FW=1.1.1.254
        CORE01=1.1.1.2
        CORE02=1.1.1.3
        doNexusBackup
        doFirewallBackup
        doCatalystBackup

 ;;
 us|US)
        SCPLocation=2.2.2.100
        NX01=2.2.2.4
        NX02=2.2.2.5
        FW=2.2.2.254
        CORE01=2.2.2.2
        CORE02=2.2.2.3
        doNexusBackup
        doFirewallBackup
        doCatalystBackup

 ;;
esac

The finished script looks like this:

#!/bin/bash
getPassword ()
{
        loginn=admin
        loginp=Passw0rd
        SCPUser=root
        SCPPass=Passw0rd
}

doNexusBackup ()
{
        echo "Doing Nexus Backup"
        ./NX-expect.sh $loginn $loginp $NX01 $SCPUser $SCPPass $SCPDir $Site $NX01Output
        ./NX-expect.sh $loginn $loginp $NX02 $SCPUser $SCPPass $SCPDir $Site $NX02Output
}

doFirewallBackup ()
{
        echo "Starting Firewall Backup"
        ./FW-expect.sh $loginn $loginp $FW $SCPUser $SCPPass $SCPDir $Site $FWOutput
}

doCatalystBackup ()
{
        echo "Starting Catalyst Backup"
        ./SW-expect.sh $loginn $loginp $CORE01 $SCPUser $SCPPass $SCPDir $Site $Core01Output
        ./SW-expect.sh $loginn $loginp $CORE02 $SCPUser $SCPPass $SCPDir $Site $Core02Output
}


#### Starts here ####
DATE=`date +%d%m%y`
Site=`hostname | cut -d- -f 2 | awk '{print tolower($0)}'`
NX01Output=802-$Site-NX01.$DATE.bak
NX02Output=802-$Site-NX02.$DATE.bak
Core01Output=802-$Site-Core01.$DATE.bak
Core02Output=802-$Site-Core02.$DATE.bak
FWOutput=802-$Site-FW01.$DATE.bak

getPassword
case $Site in
 uk|UK)
        SCPDir=1.1.1.100
        NX01=1.1.1.4
        NX02=1.1.1.5
        FW=1.1.1.254
        CORE01=1.1.1.2
        CORE02=1.1.1.3
        doNexusBackup
        doFirewallBackup
        doCatalystBackup

 ;;
 us|US)
        SCPDir=2.2.2.100
        NX01=2.2.2.4
        NX02=2.2.2.5
        FW=2.2.2.254
        CORE01=2.2.2.2
        CORE02=2.2.2.3
        doNexusBackup
        doFirewallBackup
        doCatalystBackup

 ;;
esac

Our first expect script is for the ASA and it is called "FW-expect.sh". We tell the script that we are going to use expect and to use the variables we are passing to it, and these start at 0:

#!/usr/bin/expect -f

set loginname [lindex $argv 0]
set loginpassword [lindex $argv 1]
set FWIP [lindex $argv 2]
set SCPLogin [lindex $argv 3]
set SCPPassword [lindex $argv 4]
set SCPLocation [lindex $argv 5]
set Site [lindex $argv 6]
set FWOutput [lindex $argv 7]

We then set a time out and spawn an ssh connection using the login name and firewall IP, and get it to the enable prompt. We use \n to signify that we are pressing enter:

set timeout 30
spawn ssh $loginname@$FWIP
expect "password:"
send "$loginpassword\n"
expect ">"
send "en\n"
expect "Password:"
send "$loginpassword\n"
expect "#"

The last part of the script performs the backup, using the command "copy run scp:" and passes the login details. For the most part we can send a carriage return to accept the default values. Where we have "expect "?"" we are being prompted for the source filename and the default is the running-config, so we just want to hit enter. Similarly where the script says "expect "Address"" we have already passed this and the firewall will be prompting us to accept the default value of the SCP servers IP address that was passed in the variable SCPLocation. We then pass the login name and password for the root account on the SCP server, and then pass the directory and filename that we want to save the file to, which would be translated to /backup/UK/802-UK-FW01.110614.bak. Lastly we exit from our SSH session.

send "copy run scp://$SCPLogin@$SCPLocation\n"
expect "?"
send "\n"
expect "Address"
send "\r"
expect "username"
send "$SCPLogin:$SCPPassword\n"
expect "filename"
send "/backup/$Site/$FWOutput\n"
expect "#"
send "exit\n"

The finished script looks like this:

#!/usr/bin/expect -f

set loginname [lindex $argv 0]
set loginpassword [lindex $argv 1]
set FWIP [lindex $argv 2]
set SCPLogin [lindex $argv 3]
set SCPPassword [lindex $argv 4]
set SCPLocation [lindex $argv 5]
set Site [lindex $argv 6]
set FWOutput [lindex $argv 7]

set timeout 30
spawn ssh $loginname@$FWIP
expect "password:"
send "$loginpassword\n"
expect ">"
send "en\n"
expect "Password:"
send "$loginpassword\n"
expect "#"
send "copy run scp://$SCPLogin@$SCPLocation\n"
expect "?"
send "\n"
expect "Address"
send "\r"
expect "username"
send "$SCPLogin:$SCPPassword\n"
expect "filename"
send "/backup/$Site/$FWOutput\n"
expect "#"
send "exit\n"

The expect scripts for the Catalyst (or any IOS) which is called "SW-expect.sh" looks very similar. We are passing the same variables, and the challenge/responses are very similar. So I have just included the whole script:

#!/usr/bin/expect -f

set loginname [lindex $argv 0]
set loginpassword [lindex $argv 1]
set Switchnum [lindex $argv 2]
set SCPLogin [lindex $argv 3]
set SCPPassword [lindex $argv 4]
set SCPLocation [lindex $argv 5]
set Site [lindex $argv 6]
set CoreOutput [lindex $argv 7]

set timeout 30
spawn ssh $loginname@$Switchnum
expect "Password:"
send "$loginpassword\n"
expect "#"
send "copy run scp:\n"
expect "Address"
send "$SCPLocation\n"
expect "username"
send "$SCPLogin\n"
expect "filename"
send "/backup/$Site/$CoreOutput\n"
expect "Password:"
send "$SCPPassword\n"
expect "#"
send "exit\n"

The Nexus backup script (NX-expect.sh) is also very similar, the main difference being that we need to specify the vrf we are using for the backup:

#!/usr/bin/expect -f

set loginname [lindex $argv 0]
set loginpassword [lindex $argv 1]
set NXnum [lindex $argv 2]
set SCPLogin [lindex $argv 3]
set SCPPassword [lindex $argv 4]
set SCPLocation [lindex $argv 5]
set Site [lindex $argv 6]
set NXOutput [lindex $argv 7]

set timeout 30
spawn ssh $loginname@$NXnum
match_max 100000
expect "Password:"
send "$loginpassword\n"
expect "#"
send "copy run scp://$SCPLogin@$SCPLocation/backup/$Site/$NXOutput vrf management\n"
expect "password:"
send "$SCPPassword\n"
expect "#"
send "exit\n"
interact

Because this script uses SSH we need to have the SSH keys stored so that when the script is called it won't sit there expecting you to accept the SSH certificate. Before running the script you should manually SSH to the device as the user that the script will run under - so from the root account if the script is going to run as root SSH to the device, or do SSH using sudo. If you don't do this part the script won't accept the SSH key from the device and the script will not run.

Once you have all the scripts they need to be executable (so do "chmod a+x" on each of them, or "sudo chmod a+x").

You can then add an entry to the crontab to run the scripts at a time of your choosing.

Hope you find this useful, you might want to include some error checking and confirmation through email into the script. Because of the inclusion of the site in the backup file location it's then quite easy to perform an rsync to keep offsite copies on the other data center.

EEM Script to backup Cisco IOS configuration using SCP

EEM (Embedded Event Manager) is pretty powerful for scripting changes to routers and switches. It's supported in IOS, but not on ASAs or Nexus switches (NX-OS). It allows you to script IOS commands which you can run manually, or through using kron, at a time of your choosing.

Here we will go through two scripts, they are very similar, but one will run at a preconfigured time, the other will run when there is a change to the system.

The scripts will perform a backup of the running configuration to an SCP server, so it's nice and secure, and it will send an email when the script has completed.

We start by defining a couple of variables to set the email server and the recipient:

R1(config)#event manager environment _email_server 1.1.1.1
R1(config)#event manager environment _email_to user@domain.com

We then start defining our script for the scheduled backup. We create an event manager "applet" and give it a name:

event manager applet SCHEDULED-BACKUP-CFG

We then set the event type to none - we'll cover the options for this on the next script.

event none

Next we start defining the commands we will use. These are all numbered actions. The first action we will use is to get the router name (this works the same for switches) so we can use it in our script:

action 0.1 info type routername

Next we can enter the commands to perform the backup

action 1.0 cli command "enable"
action 1.1 cli command "copy run scp:" pattern "Address"
action 1.2 cli command "1.1.1.10" pattern "username"
action 1.3 cli command "root" pattern "filename"
action 1.4 cli command "/backup/routers/$_info_routername-config" pattern "Password"
action 1.5 cli command "Passw0rd"

The first line sends us into enable mode.

The second line sends the command "copy run scp:" and we define a pattern, which will match the next line (1.2), so when the router returns a line with the pattern "Address" in it, it will send the IP address of the SCP server (1.1.1.10).

Again the script will then specify a pattern for the next line (1.3) which it expects to be "username", and we will send the username of "root".

The router will respond with a line asking for the destination filename, which we match with the pattern "filename", and will send the filename in line 1.4.

Because we defined a variable in line 0.1 which captures the routername we can supply this as part of the filename, along with the path on the SCP server, so if our router is called "R!" the filename we send would be "/backup/routers/R1-config".

Lastly as we are expecting the router to return a prompt for the root user's password we send that in line 1.5.

We will finish with a couple of actions, firstly to send an email to us to confirm that the backup has worked, secondly to log the actions in the event log:

action 2.0 mail server "$_email_server" to "$_email_to" from "$_info_routername@domain.com" subject "Backup completed" body "Backup completed"
action 3.0 syslog priority informational msg "Config change detected. Copy to SCP succesfull"

We must then exit from EEM in order to save our applet:

exit

The full command sequence would look like this:

R1(config)#event manager environment _email_server 1.1.1.1
R1(config)#event manager environment _email_to user@domain.com
R1(config)#event manager applet SCHEDULED-BACKUP-CFG
R1(config-applet)#event none
R1(config-applet)#action 0.1 info type routername
R1(config-applet)#action 1.0 cli command "enable"
R1(config-applet)#action 1.1 cli command "copy run scp:" pattern "Address"
R1(config-applet)#action 1.2 cli command "1.1.1.10" pattern "username"
R1(config-applet)#action 1.3 cli command "root" pattern "filename"
R1(config-applet)#action 1.4 cli command "/backup/routers/$_info_routername-config" pattern "Password"
R1(config-applet)#action 1.5 cli command "Passw0rd"
R1(config-applet)#action 2.0 mail server "$_email_server" to "$_email_to" from "$_info_routername@domain.com" subject "Backup completed" body "Backup completed"
R1(config-applet)#action 3.0 syslog priority informational msg "Config change detected. Copy to SCP succesfull"
R1(config-applet)#exit
R1(config)#

We can run this command whenever we like using the command:

R1#event manager run SCHEDULED-BACKUP-CFG

All being well you should receive an email stating that the backup has been performed.
We can set this applet to run at a schedule of our choosing by using the kron scheduler.
For this we create a new kron job called "weekly-backup" which will occur at 1am every sunday. The kron job will call a policy, called "SCHEDULED-BACKUP-CFG", which in turn runs the cli command that calls our EEM script:

R1(config)# kron occurrence weekly-backup at 01:00 sun recurring
R1(config-kron-occurrence)# policy-list SCHEDULED-BACKUP-CFG
R1(config-kron-occurrence)# exit
R1(config)# kron policy-list SCHEDULED-BACKUP-CFG
R1(config-kron-policy)# cli event manager run SCHEDULED-BACKUP-CFG
R1(config-kron-policy)# exit

The next script will perform ad-hoc backups whenever the configuration is saved. In order to do this we will capture a syslog event, and that event is matched using a pattern of "%SYS-5-CONFIG_I: Configured from", so that when a change is made we get an email telling us so.
Instead of using "event none" we are now using "event syslog pattern "%SYS-5-CONFIG_I: Configured from"" to capture the event.

event manager applet ADHOC-BACKUP-CFG
event syslog pattern "%SYS-5-CONFIG_I: Configured from"
action 0.1 info type routername
action 1.0 cli command "enable"
action 1.1 cli command "copy run scp:" pattern "Address"
action 1.2 cli command "1.1.1.10" pattern "username"
action 1.3 cli command "root" pattern "filename"
action 1.4 cli command "/backup/routers/$_info_routername-config" pattern "Password"
action 1.5 cli command "Passw0rd"
action 2.0 mail server "$_email_server" to "$_email_to" from "$_info_routername@domain.com" subject "Backup completed" body "Backup completed"
action 3.0 syslog priority informational msg "Config change detected. Copy to SCP succesfull"
exit

EEM is not hard to get into, probably the hardest part is working out the pattern matching. When you do start getting into it you will realise that you can use it for a wide range of tasks.