Recovering Linux systems with mkcdrec

Originally published in SysAdmin Magazine, September 2002

Contents

      Introduction
      Overview
      Installation and Configuration
      Making a recovery CD
      Performing a recovery
      Some Caveats
      Conclusion
      References

Introduction

Sooner or later you will need to perform a system recovery. You may be in the position where you are administering a system which has suffered a hard drive failure, or maybe someone has hacked into it, or maybe you just made a small mistake and wiped the kernel image or boot track. For Linux systems one of my favourite tools is a package called mkcdrec - for Make CD Recovery.

Originally written in November of 2000 by Gratien D'haese, mkcdrec has evolved to become a compendium of Linux recovery tools.

While there's planty of utilities around to backup your system to a writeable CD, and Linux release CDs can usually be booted into some sort of recovery mode, mkcdrec goes a long way further. Much more that the usual minimal "Swiss Army knife" of recovery that is typical on Linux distribution disks, mkcdrec builds a recovery CD which is a fully equipped workshop containing almost every tool you could every want to rebuild a failed system.

And if that wasn't enough, mkcdrec can also be used to clone systems - building identical copies of a system on a new platform. This is simply indispensible if you have a number of boxes to build with identical configurations - just the thing for web server farms, DNS servers, Mail servers, and so on.

Mkcdrec is also the ideal tool to use to recover systems from intrusions. Because the media is read-only it is impervious to alteration by attackers. If you know, or even suspect, that your Internet web server has been invaded then you can simply shut down the system and quickly rebuild it from scratch from a media which you know to be clean. Alternatively, you can take a spare box, build it as a clone of the one attacked and do a simple replacement.

Overview

mkcdrec can be used in a number of ways. The first, and most important, is the rescue CD-ROM. This builds a CD image ( an ISO file which can be burned onto a CD ) which is simply a minimal bootable copy of your Linux system. It contains the Linux kernel, kernel modules, necessary utilities such as a shell, and a number of useful system recovery tools like fdisk, mkfs, debugfs, and many others. It also includes network support and network client programs such as telnet, ftp, rsh, and so on. The essential configuration details of your system are also included - obvious configuration files like /etc/hosts, /etc/services; as well as less-obvious things such as your filesystem layout and disk partitioning information.

In addition to the rescue CD and its utilities, mkcdrec can also save all of your files in a compressed tar-archive and add these to your recovery CD.

You don't strictly have to write all of the backup information onto a CD. You also have the option of writing the backup information into another directory - such as a NFS shared filesystem - or onto a local or even remotely served magnetic tape drive.

Installation and Configuration

The first thing to do is to download the mkcdrec package ( see URL at the end of this article ) the current version at the time of writing this is v0.6.1. Simply unzip this into the directory of your choice - I use /opt for my tools : 
    # cd /opt
    # wget http://mkcdrec.ota.be/project/mkCDrec_v0.6.1.tar.gz
    # gzip -cd
The files will be extracted into a directory called 'mkcdrec'. Because I've always got multiple versions of these tools lying around, I like to have each version separate, so I'll rename it: 
    # mv mkcdrec mkcdrec-0.6.1
Next, check that you've got everything in place by running 'make test', this should produce a display similar to the following : 
    # cd mkcdrec-0.6.1
    # make test
    /opt/cd_utils/mkcdrec-0.6.1/scripts/test.sh
    make test output of mkCDrec v0.6.1
    Test 1:  Are we root?					Passed
    Test 2:  missing executables needed by mkCDrec
    dd:								Found
    mount:							Found
    bc:								Found
    umount:							Found
    gcc:							Found
    ldd:							Found
    mformat:							Found
    mkisofs:							Found
    cdrecord:							Found
    nasm:							Found
    mt:								Not found
    mt: needed with tape back-up!
    ash:							Found
    bzip2:							Found
    gzip:							Found
    rsh:							Found
    ssh:							Found
    genromfs:							Found
    file:							Found
    openssl:							Found
    Test 3:  Filesystem for Initial ramdisk allowed?		Passed
    Test 4:  loopback device works?				Passed
    Test 5:  ram device available				Passed
    Test 6:  romfs supported  by the kernel?			N/A
    Test 7:  cramfs supported  by the kernel?			N/A
    Test 8:  strip (from binutils) available?			Passed
    Test 9:  BOOT_FLOPPY_DENSITY=ED ok?				Passed
    Test 10:  cdrecord -scanbus					N/A
    Test 11: Header files present?				Passed
    Test 12: DEVFS supported by kernel?				N/A
    Test 13: filesystem tools present?
    ext2:							Passed
    vfat:							Passed
    Test 14: initrd must be compiled in kernel!			Passed
    Test 15: Amount of memory available				129 Mb
    Test 16: scripts/Config.sh a link?				Passed

    #
Note that the 'make test' didn't find the mt command on my system - I don't have any magnetic tape devices so this is nothing to worry about.

After running 'make test', the next thing to do is set a number of configuration parameters in the file Config.sh. I'll go over the most important ones here. 

    RAMDISK_SIZE=32
Set this to be the size of the ramdisk which will hold the running linux recovery system. The default is 24 Mb, set it higher if you have plenty of memory. 
    ISOFS_DIR=/var/tmp/backup
This is where the target linux system will be assembled before being converted into an ISO filesystem. This should be an empty directory as mkcdrec will deleet any existing files in this directory. The directory should have at least 32 Mb of free space. 
    CDREC_ISO_DIR=/var/tmp/iso
This is where the ISO CD image will be created. If you are creating a full backup type of recovery CD then you'll need a good 700 Mb of space here, otherwise for a recovery-only CD 20 Mb should be plently. 
    LINUX_KERNEL="/boot/vmlinuz"
The name of your current running kernel which will be the kernel booted by the recover CD. If this is left blank, mkcdrec will try to find the current kernel by looking in /etc/lilo.conf, if you don't use LILO then you should specify this parateter explicitly or you will see strange error messages. 
    BURNCDR=n
An option which defines if mkcdrec will automatically burn the ISO image to CD ( when BURNCDR=y ) or just leave the image in the directory on disk for you do do it manually. If your CD writer is on another system then you will want to set this second option so you can copy it over to where your CD writer is. Most CD-writing software supports the ISO CD image format. 
    CDRECORD=cdrecord
    SCSIDEVICE="0,1,0"
    WRITERSPEED="2"
Parameters for the CD writer device on your local Linux system. Setting up a CD writer on your Linux system is somewhat beyond the scope of this article - but the only way to get an IDE CD writer working seems to be through a SCSI emulation kernel module, hence the SCSIDEVICE parameter used to specify the CD writer. My CD writer is a little old, and is only capable of 2x write speed, hopefully yours will be quicker. 
    EXCLUDE_LIST=""
A list of directories to be excluded from the full backup. Its advisable to put temporary directories into this list, also those directories which you sepcified as the mkcdrec "backup" and "iso" directories.

Most of the other parameters in Config.sh are secondary - they include things such as lists of kernel modules, utility programs and manual pages which get written onto the CD image. If you have extra utilities that you want to add to the recovery image then you can easily add them into the Config.sh file.

After you have modified Config.sh, run 'make test' again to make sure that everything is still OK. With everything configured and ready to go, you're ready to make a recovery CD of your system.

Making a recovery CD

There is 4 types of recovery image that you can make:
  • A basic recovery-only CD which will boot to give you a mini-linux system with enough tools and features to correct most system problems.
  • A recovery CD which includes the above, but also contains compressed backups of your filesystems. While this is very useful to have it does take some time to create this if you have a lot of disk space in use.
  • A backup of your filesystems to another directory - such as an NFS directory on another system.
  • A backup of your filesystems to a magnetic tape drive.
To begin the process of making a recovery CD, just run 'make' while in the mkcdrec installation directory. mkcdrec will present you with a nice friendly menu requesting your choice of recovery CD : 

    Make CD-ROM recovery (mkCDrec v0.6.1) by Gratien D'haese

    mkCDrec v0.6.1 - Backing up your partitions Enter your selection:

    1) Rescue CD-ROM only (no backups)
    2) Into /home/tmp/backup (to burn on CDROM)
    3) Enter another path (spare disk or NFS)
    4) Enter (remote) tape device
    5) Quit

    Please choose from the above list [1-5]:

Performing a Recovery

To perform a recovery of your Linux system, simply insert the CD into the drive and boot from it. Most modern PCs will do this, although in some cases you will need to change the boot order in the BIOS if you want it to try booting from the CD before booting from the hard disk.

Mkcdrec boots quite quickly, loading a mini-kernel into an initial RAMDISK, then it presents you with a rather colorful splash screen and a "boot:" prompt.

Just hit at the boot prompt and the Linux system will begin booting. This usually takes about a minute to complete. When done, you should see the root shell prompt, looking something like : 

    I have no name!@host:/ #
The reason you see "I have no name!" is because the /etc/password file is missing or corrupted ( possibly a bug ), its nothing to worry about.

At this stage, using 'df' or 'mount' will show you that only the root ramdisk and CD filesystem are mounted - the recovery boot does not mount or touch the hard drive at all. You can now use the common utilities such as fdisk, fsck, debugfs, etc. to work on recovering your failed system - note that the manual pages for these utilities are also installed so you don't have to go hunting around for them.

If you look into the directory /etc/recovery/ you will see a bunch of configuration files and scripts. Here is a complete record of how your system was setup and there are two scripts in particular which will help you recover, rebuild, or clone your system :

  • start-restore.sh will completely restore your system from the information on the CD. It will partition the disks, make filesystems, and read back the compressed archive of your files.
  • clone-disk.sh will also do a complete restore, but this is aimed at restoring onto a system which is not quite the same as the original - such as a sytem with a different number of disk drives or they have a different size, and so on. clone-disk.sh is more flexible and gives you more options about what to restore.
Using these tools is quite straightforward, and if you're confident with using tools like fdisk, mkfs, and mount then you will have no problems in performing a recovery or building a new system based on the contents of the mkcdrec recovery CD.

Some Caveats

While mkcdrec is a wonderful tool for the rapid recovery and cloning of a Linux system, there is some things to watch out for :
  • you may not be able to boot the recovery CD on a system which has a different architecture from the one which was used to build it. I found this particularly when building a recovery CD on a Celeron system and finding it would not boot on an old Pentium 200/MMX - I believe the Linux kernel on the Celeron box was compiled for that CPU and would not run on an older architecture.
  • mkcdrec runs pretty slow when making a full backup of a large system - this is due to the way that it makes a gzipped-tar of all of you files which then gets placed into the ISO image to be burned onto your CD. Most CD-writing software has quicker ways of doing this.
  • some modules - particularly the ext2 filesystem support and the initrd module - must be compiled into your kernel and not be implemented as loadable modules.

Conclusion

mkcdrec could save your life. Well, almost. It will take a snapshot of your Linux system and write it onto a CD to be ready to recover from the inevitable.

References

http://mkcdrec.ota.be - mkcdrec home page
http://sourceforge.net/projects/mkcdrec - mkcdrec page at SourceForge