We recently had an enquiry from a client who has around 150 Linux servers in about 20 data centres scattered around the UK. They have a perfectly effective backup solution but were interested in improving their Disaster Recovery, in particular how to speed up recovery of a server in the event of hardware failure. In particular they wanted a solution that would allow them to take complete system images that when required could be quickly restored to a replacement server. Not only that but it had to be doable remotely and by a non-Linux technician following a simple sheet of instructions.

At help4IT we mainly use two products for system imaging; Symantec Backup Exec System Recovery and Acronis True Image Echo Enterprise Server. Both have their pros and cons which I won't discuss here, except for a major Symantec one - it doesn't support Linux. Trouble was the client didnt like the Acronis option as it requires recompiling the kernel to install the snapapi module, seems their Linux servers have been heavily customised and any mention of fundamental changes scares them.

Back to the drawing board and we came up with a couple of options:

 The Linux Way: Having spoken to our Linux guru Andy it
appears the usual way of doing Linux system images is to boot the
server off a "Live CD" into a temporary OS and then create a backup of
the system. Subsequently the normal backup procedure is used to backup
all non-system files and data. Should a full restore be required the
process is then reversed, the new server is booted off the Live CD and
the system backup restored, then the rest of the data is restored from
a recent backup.
Advantages: No changes to the system are required.
Disadvantages: complicated, although to a certain degree it could be
automated with a menu based frontend. Server has to be taken offline to
create system image. Main problem is when restoring to different
hardware, it may be necessary to install new drivers which would
require Linux skills, and if the drives aren't the same size then the
partition tables may have to be rebuilt.

The Virtualisation Way: This method is similar a system that help4IT use for
our "vHSDR" (virtual hot site disaster recovery - https://www.help4it.co.uk/consultancy-mainmenu-28/disaster-recovery.html ). The server is booted
off the Acronis CD and a system image is backed up to network storage.
The server is then wiped and VMware ESXi (free edition) is installed,
then a Virtual Machine is created and the Acronis system image restored
to that, the Acronis software takes care of replacing the necessary
drivers and the server will then be online as before. After that
creating full system images just require creating a snapshot in ESX,
this then freezes the main VMDK (virtual machine image file) so it can
simply be copied off. Its simple enough to automate this as well so it can be included in the existing backup strategy, see https://wiki.contribs.org/Backup_of_ESXi_Virtual_Machines_using_Affa for an example of this. Should a bare metal restore be required the VMDK backup just has to be copied onto any ESX server and the VM started, then it is back online.
Advantages: Only changes to the system are drivers, and ESX virtual
hardware is industry standard so there will be no
compatibility/reliability issues. Once the server has been converted to
a virtual machine it is then effectively hardware independent. Disadvantages: Initial setup is more involved than simply rolling out
a software agent to each server.

Two ways to go, which way to choose?

Virtualisation would appear to be the ideal choice here, not only as a solution to the current problem but also for its future potential benefits such as server consolidation and high availability. Apparently another supplier is recommending Syncsort's Backup Express, whose salesman claims does not need a kernel module which seems hard to believe. Apart from that licensing for 150 servers would be a significant extra cost compared to the free ESX3i so we will see which way they go.