Recently I decided that I would backup the network and device configurations for the gear at my work, because we didn’t have any backups more recent than 2018. I started off trying to find some automated solution in the devices themselves but unfortunately only one of them was able to be backed up in this way. I took some manual backups and tried to be happy with that but the problem gnawed at my brain for a bit till I finally looked for a proper automated solution that would work with the majority of our random assortment of devices.

I initially looked at using RANCID however it didn’t support CheckPoint Gaia devices out of the box and I didn’t much feel equipped to create my own check. It was around this time that I had Oxidized suggested to me and I started looking into and installing it.

All went smoothly with the installation by following the official Oxidized instructions. My configurations would download with no issues when I was running directly from the command line during testing. I ran into real issues when I started looking into how to daemonize Oxidized. It seemed like there wasn’t a very good set of instructions for CentOS specifically. I did find a blog post that covers it off really well for Ubuntu and I did actually follow this one for the Oxidized user tasks. In the end, I did a lot of searching and a lot of trying different things (and removing things I tried to see which ones I could keep), before finding the solution outlined here.

With that said let’s step through the config I did.

Installation

This config is simply copied from the GitHub page and is pretty standard stuff.

1
2
3
4
5
6

yum install centos-release-scl
yum install rh-ruby23 rh-ruby23-ruby-devel
scl enable rh-ruby23 bash  
yum install make cmake which sqlite-devel openssl-devel libssh2-devel ruby gcc ruby-devel libicu-devel gcc-c++  
gem install oxidized  
gem install oxidized-script oxidized-web

User Configuration

The following is taken from the Ubuntu guide above.

1
2
3
4

useradd oxidized
chsh -s /usr/sbin/nologin oxidized
mkdir -p /opt/oxidized/{output,.config/oxidized/}
usermod -m -d /opt/oxidized oxidized

The first line here is self-explanatory.

The second line changes the default shell of the Oxidized user to disallow logins, this is a security measure so that you cannot su to the user. It may be useful to leave this line out for initial testing.

The third line creates (if it does not already exist), the oxidized folder in /opt/ as well as subfolders output, config, and oxidized. Basically just creating the basic folder structure that Oxidized will put in your home directory if allowed to default.

The fourth line changes the home directory of the oxidized user to the folders we just created. This is a good way to ensure that if Oxidized decides to default to the home directory, it defaults somewhere sensible. 

Symlinks, Service File and Shared Libraries

Oxidized requires Ruby 2.3 and the version that ships with CentOS 7.6 is Ruby 2.0. This obviously poses a bit of a problem when you install the version from the SCL repository. To fix this, symlink the newer version to /usr/bin/ruby to use this as the default version like the below.

1

ln -s /opt/rh/rh-ruby23/root/bin/ruby /usr/bin/ruby

This will allow the service to act as though the scl enable rh-ruby23 bash command was run by virtue of it using the newer binary for Ruby. Obviously do not do this if you need to maintain backwards compatibility with Ruby 2.0.0, in this case I would suggest using the an environment file called /etc/sysconfig/oxidized and including this in your systemd config under [Service] EnvironmentFile=/etc/sysconfig/oxidized. In the file, you would add the following line of configuration.

1

LD_LIBRARY_PATH=/opt/rh/rh-ruby23/root/usr/local/lib64:/opt/rh/rh-ruby23/root/usr/lib64

At this point I also symlinked /usr/local/bin/oxidized to /opt/rh/rh-ruby23/root/usr/local/bin/oxidized to make my service file a little cleaner.

From here, I created the service file under /lib/systemd/system/oxidized.service and added the below:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12

[Unit]
Description=Oxidized - Network Device Configuration Backup Tool
After=network-online.target multi-user.target
Wants=network-online.target

[Service]
ExecStart=/usr/local/bin/oxidized
KillSignal=SIGKILL
User=oxidized

[Install]
WantedBy=multi-user.target

Simply, when the machine boots, Oxidized will wait for the network module to be completely loaded before executing the program located in /usr/local/bin under user oxidized, utilising the environment file to set the required environment variables.

Finally, I added /opt/rh/rh-ruby23/root/usr/lib64 as a line in /etc/ld.so.conf. After this, I ran the ldconfig command to ensure that the path was updated and added.

Oxidized Config

At this point, all the bones are in place and Oxidized will start correctly and generate a sample configuration in the Oxidized user’s home directory (/opt/oxidized). The next logical step is to create a config file so that you can start importing your configs. This is a lot more arbitrary, but I went with a CSV source to import my devices as I only had a relatively small number of them. If you go for the CSV route, you will need to create a router.db file which will be mapped in the main config file.

The below is a sample of what my config looks like. The config is in YAML format and the # symbol is a comment. I will add comments to each line in order to try to explain what each line/section does. The config is located in the /opt/oxidized/.config/oxidized folder and is simply called config.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46

username: admin #The (optional) default username on your devices. This can be overridden at group/individual level.
password: ###### #As above, optional default password and can be overridden at group/individual level.
resolve_dns: true #If you provide a hostname, this will DNS lookup the hostname.
interval: 3600 #How often to check for new configs.
use_syslog: false #This option relates to whether you want to use syslog messages to trigger a config fetch.
debug: true #Debug mode! Great for setup.
threads: 30 
timeout: 20 #Timeout after 20 seconds.
retries: 3 #Try 3 times.
remove_secret: true #Remove secret/enable passwords from the resulting config file - does not work on all models.
prompt: !ruby/regexp /^([\w.@-]+[#>]\s?)$/ #Detection of the user prompt on the device.
rest: 0.0.0.0:8888 #Listening address for API/web UI endpoint.
next_adds_job: false
pid: "/opt/oxidized/.config/oxidized/pid" 
crash:
  directory: "/opt/oxidized/.config/oxidized/crashes" #Where to store the log when the application crashes
  hostnames: false
stats:
  history_size: 10 #How many past configs to keep.
input:
  default: ssh 
  debug: false
  ssh:
    secure: false  #I believe this has something to do with using keys or certificates to connect to the devices.
  utf8_encoded: true
output:
  default: git #Output to a Git repository. Oxidized will create the repository based on the settings below.
  git:
    user: Oxidized #Does not need to be an actual user, this is just the name that will appear on the commits
    email: abc@company.com #Again, does not need to be an actual email address.
    repo: /opt/oxidized/.git/default.git #Where to store the repository
  file:
    directory: "/opt/oxidized/output/configs" #By default, Oxidized will create a repository for each device.
source:
  default: csv
  csv:
    file: "/opt/oxidized/.config/oxidized/router.db" #File containing all the devices to pull configs from.
    delimiter: !ruby/regexp /:/ #The delimiter for the file. This is the default.
    map:
      name: 0 #Hostname is the first item in the router.db configuration.
      ip: 1 #IP is the second item and so on.
      model: 2 #In this case, the router.db would look something like hostname:10.0.0.1:procurve:switches
      group: 3 #An arbitrary grouping to keep like devices together.
groups:
  switches:
    username: manager #Set a different username for any item in the switches group.  

Once you’ve got all your configuration together, you should be able to start the service and see the magic happen. To confirm that everything is working as expect, you can go to the website that is presented on port 8888 and you should see all your devices starting to pull configs.