All administrators already face the following nightmare: It’s 01:00am and you changed a parameter in an application. A few days later, due to instability, you need to rollback. What the hell did you change?”. Of course, changes “on the go” must be avoided like the plague but sometimes, they’re mandatory.
Change management is defined as a set of strict rules (procedure) used to keep track of modifications on any existing systems configuration. In which cases could change management help you? A lot! To restore system parameters in case of crash or rebuild, to track who did what and when, to simplify team working, to speed up deployment of new servers or to perform audits.
They are different levels of change management: from simple rules like writing down everything in a logbook to complex set of rules organized in standardized process like ITIL. Whatever the procedure which apply to your business, you need tools to keep track of changes.
Which tools are available? To maintain a log book, a Wiki will perfectly do the job. Administrators can describe the change they made on systems. But sometimes, more technical details must be logged (ex: to quickly revert to a previous state). In this case, why not use developer’s tool like “Concurrent Versions System (CVS in short)?
CVS allows teams of developers to work on projects transparently across networks and client platforms. With CVS, users may perform the following operations:
- Commit changes when a step of reached
- Start new “branches” to work on parallel versions of the same project
- Synchronize data from the server
- Lock data
- and much more!
If developers use it, why not extend to system or network administrators? I will not describe how to install CVS. I assume here that a server is already available for you. Even if running CVS locally is possible, I would recommend to centralize the repositories on a central securized server (with limited access and safe storage – RAID and regular backups). For details how to install a CVS server, check this tutorial.
As example, let’s manage an Apache configuration via CVS. First, we need to import the existing configuration files into the repository:
# cd /var/www # export CVSROOT=:pserver:cvs@server.domain.com:/home/cvs # cvs login Logging in to: :pserver:cvs@server.domain.com:/home/cvs CVS password: xxxxxxxxxx # cvs import -m "Initial import" conf webserver start N conf/httpd.conf N conf/httpd.conf-dist N conf/bgplg.css N conf/bgplg.foot N conf/bgplg.head N conf/bgplg.conf N conf/magic N conf/mime.types No conflicts created by this import
Now, we can safely remote (or rename) the conf directory and perform a checkout. A checkout operation is necessary to create a local (working) copy of the files.
# cd /var/www # mv conf conf.orig # cvs checkout conf cvs server: Updating conf U conf/bgplg.css U conf/bgplg.foot U conf/bgplg.head U conf/bgplg.conf U conf/httpd.conf U conf/httpd.conf-dist U conf/magic U conf/mime.types #
Now, let’s make some changes to the Apache configuration. Edit the file httpd.conf and add a new virtual host:
# cat <<_END_ >>httpd.conf
<VirtualHost 10.0.0.1>
DocumentRoot /var/www/htdocs/vhost1
ErrorLog logs/vhost1.error
CustomLog logs/vhost1.access combined
</VirtualHost>
_END_
We can check the changes (“diff”) with the file saved in the repository:
# cvs diff httpd.conf Index: httpd.conf ======================================== RCS file: /home/cvs/conf/httpd.conf,v retrieving revision 1.1.1.1 diff -r1.1.1.1 httpd.conf 11116a1117,1121 > <VirtualHost 10.0.0.1> > DocumentRoot /var/www/htdocs/vhost1 > ErrorLog logs/vhost1.error > CustomLog logs/vhost1.access combined > </VirtualHost>
Now, commit the changes to the repository. The “-m” option allow to specify useful comments about the changes. Note that it’s important to give here relevant information and not just obscure data like “fixed a problem“. If no “-m” switch is given on the command line, an editor will be started to allow you to write some comments.
# cvs commit -m "Added a new virtual host for customer xxx" httpd.conf Checking in httpd.conf; /home/cvs/conf/httpd.conf,v <-- httpd.conf new revision: 1.2; previous revision: 1.1 done
Finally, maybe the most important command: “log”. It allows you to display an history of all (or partial) changes made on a file.
# cvs log httpd.conf
RCS file: /home/cvs/conf/httpd.conf,v
Working file: httpd.conf
head: 1.2
branch:
locks: strict
access list:
symbolic names:
start: 1.1.1.1
www-server: 1.1.1
keywork substitution: kv
total revisions: 3; selected revisions: 3
descriptions:
--------------------
revision 1.2
date: 2009/05/26 18:06:40; author: cvs; state: Exp; lines: +2 -0
Added a new virtual host
--------------------
revision 1.1
date: 2009/05/26 17:43:52; author: cvs; state: Exp;
branches: 1.1.1;
Initial revision;
--------------------
revision: 1.1.1.1
date: 2009/05/26 17:43:52; author: cvs; state: Exp; lines: +0 -0
Initial import
====================
#
As you can see, CVS can be very helpful to maintain operating systems and or applications configuration files. This is a short introduction of CVS, lot of options and features were not covered here. The demo above was performed on an OpenBSD system but CVS is available for all UNIX flavors, even, Windows.
Also, note that CVS (a quite old application) is not the only tool to perform revision control system: subversion is much more used today.
Combined with a good monitoring, a good backup procedure for your data and centralization of logs, you will be ready to face a lot of incidents without big problems!