The rsync-system-backup program uses rsync to create full system backups of Linux systems. Supported backup destinations include local disks (possibly encrypted using LUKS) and remote systems that are running an SSH server or rsync daemon. Each backup produces a timestamped snapshot and these snapshots are rotated according to a rotation scheme that you can configure. The package is currently tested on cPython 2.7, 3.4, 3.5, 3.6, 3.7 and PyPy (2.7).
While this project brings together more than ten years of experience in creating (system) backups using rsync, all of the actual Python code was written in the first few months of 2017 and has seen limited real world use. The project does however have an automated test suite with more than 90% test coverage and my intention is to extend the test coverage further.
In May 2018 I changed the status from alpha to beta as part of release 1.0. The bump in major version number was triggered by a backwards incompatible code change, however at that point I had been using rsync-system-backup to make local backups of several of my Linux systems for the majority of a year. Also several colleagues of mine have used the how-to on setting up unattended backups to an encrypted USB disk.
Warning
Please use the --dry-run option when you're getting familiar with how rsync-system-backup works and don't remove the option until you're confident that you have the right command line, because using rsync-system-backup in the wrong way can cause data loss (for example by accidentally swapping the SOURCE and DESTINATION arguments).
The rsync-system-backup package is available on PyPI which means installation should be as simple as:
$ pip install rsync-system-backupThere's actually a multitude of ways to install Python packages (e.g. the per user site-packages directory, virtual environments or just installing system wide) and I have no intention of getting into that discussion here, so if this intimidates you then read up on your options before returning to these instructions ;-).
There are two ways to use the rsync-system-backup package: As the command line program rsync-system-backup and as a Python API. For details about the Python API please refer to the API documentation available on Read the Docs. The command line interface is described below.
Usage: rsync-system-backup [OPTIONS] [SOURCE] DESTINATION
Use rsync to create full system backups.
The required DESTINATION argument specifies the (possibly remote) location where the backup is stored, in the syntax of rsync's command line interface. The optional SOURCE argument defaults to '/' which means the complete root filesystem will be included in the backup (other filesystems are excluded).
Please use the --dry-run option when getting familiar with this program and don't remove it until you're confident that you have the right command line, because using this program in the wrong way can cause data loss (for example by accidentally swapping the SOURCE and DESTINATION arguments).
Supported locations include:
- Local disks (possibly encrypted using LUKS).
- Remote systems that allow SSH connections.
- Remote systems that are running an rsync daemon.
- Connections to rsync daemons tunneled over SSH.
The backup process consists of several steps:
- First rsync is used to transfer all (relevant) files to a destination directory (whether on the local system or a remote system). Every time a backup is made, this same destination directory is updated.
- After the files have been transferred a 'snapshot' of the destination directory is taken and stored in a directory with a timestamp in its name. These snapshots are created using 'cp
--archive--link'. - Finally the existing snapshots are rotated to purge old backups according to a rotation scheme that you can customize.
Supported options:
I've been finetuning my approach to Linux system backups for years now and during that time rsync has become my swiss army knife of choice :-). I also believe that comprehensive documentation can be half the value of an open source project. The following sections attempt to provide a high level overview of my system backup strategy:
Each backup is a full copy of the filesystem tree, stored in the form of individual files and directories on the destination. This "backup format" makes it really easy to navigate through and recover from backups because you can use whatever method you are comfortable with, whether that is a file browser, terminal, Python script or even chroot :-).
Note
You may want to configure updatedb to exclude the directory containing your system backups, otherwise the locate database will grow enormously.
Every time a backup is made the same destination directory is updated with additions, updates and deletions since the last backup. After the backup is done a snapshot of the destination directory is created using the command cp --archive --link with the current date and time encoded in the name.
Due to the use of hard links each "version" of a file is only stored once. Because rsync by default doesn't modify files inplace it breaks hard links and thereby avoids modifying existing inodes. This ensures that the contents of snapshots don't change when a new backup updates existing files. The combination of hard links and the avoidance of inplace modifications effectively provides a limited form of deduplication. Each snapshot requires a couple of megabytes to store the directory names and hard links but the contents of files aren't duplicated.
The article Easy Automated Snapshot-Style Backups with Linux and Rsync contains more details about this technique.
Snapshots can be rotated according to a flexible rotation scheme, for example I've configured my laptop backup rotation to preserve the most recent 24 hourly backups, 30 daily backups and endless monthly backups.
While developing, maintaining and evolving backup scripts for various Linux laptops and servers I've learned that backups for different systems require different backup destinations and connection methods:
There's a LUKS encrypted USB disk on my desk at work that I use to keep hourly, daily and monthly backups of my work laptop. The disk is connected through the same USB hub that also connects my keyboard and mouse so I can't really forget about it :-).
Before the backup starts, the encrypted disk is automatically unlocked and mounted. The use of a key file enables this process to run unattended in the background. Once the backup is done the disk will be unmounted and locked again, so that it can be unplugged at any time (as long as a backup isn't running of course).
My personal laptop transfers hourly backups to the rsync daemon running on the server in my home network using a direct TCP connection without SSH. Most of the time the laptop has an USB Ethernet adapter connected but the backup runs fine over a wireless connection as well.
My VPS (virtual private server) transfers nightly backups to the rsync daemon running on the server in my home network over an SSH tunnel in order to encrypt the traffic and restrict access. The SSH account is configured to allow tunneling but disallow command execution. This setup enables the rsync client and server to run with root privileges without allowing the client to run arbitrary commands on the server.
Backing up to a local disk limits the effectiveness of backups but using SSH access between systems gives you more than you bargained for, because you're allowing arbitrary command execution. The rsync daemon provides an alternative that does not allow arbitrary command execution. The following sections discuss this option in more detail.
To be able to write files as root and preserve all filesystem metadata, rsync must be running with root privileges. However most of my backups are stored on remote systems and opening up remote root access over SSH just to transfer backups feels like a very blunt way to solve the problem :-).
Fortunately another solution is available: Configure an rsync daemon on the destination and instruct your rsync client to connect to the rsync daemon instead of connecting to the remote system over SSH. The rsync daemon configuration can restrict the access of the rsync client so that it can only write to the directory that contains the backup tree.
In this setup no SSH connections are used and the traffic between the rsync client and server is not encrypted. If this is a problem for you then continue reading the next section.
On Debian and derivatives like Ubuntu you can enable and configure an rsync daemon quite easily:
Make sure that rsync is installed:
$ sudo apt-get install rsync
Enable the rsync daemon by editing
/etc/default/rsyncand changing the lineRSYNC_ENABLE=falsetoRSYNC_ENABLE=true. Here's a one liner that accomplishes the task:$ sudo sed -i 's/RSYNC_ENABLE=false/RSYNC_ENABLE=true/' /etc/default/rsyncCreate the configuration file
/etc/rsyncd.confand define at least one module. Here's an example based on my rsync daemon configuration:# Global settings. max connections = 4 log file = /var/log/rsyncd.log # Defaults for modules. read only = no uid = 0 gid = 0 # Daily backups of my VPS. [vps_backups] path = /mnt/backups/vps/latest post-xfer exec = /usr/sbin/process-vps-backups # Hourly backups of my personal laptop. [laptop_backups] path = /mnt/backups/laptop/latest post-xfer exec = /usr/sbin/process-laptop-backups
The
post-xfer execdirectives configure the rsync daemon to create a snapshot once the backup is done and rotate old snapshots afterwards.Once you've created
/etc/rsyncd.confyou can start the rsync daemon:$ sudo service rsync start
If you're using a firewall you should make sure that the rsync daemon port is whitelisted to allow incoming connections. The rsync daemon port number defaults to 873. Here's an iptables command to accomplish this:
$ sudo iptables -A INPUT -p tcp -m tcp --dport 873 -m comment --comment "rsync daemon" -j ACCEPT
When your backups are transferred over the public internet you should definitely use SSH to encrypt the traffic, but if you're at all security conscious then you probably won't like having to open up remote root access over SSH just to transfer backups :-).
The alternative is to use a non privileged SSH account to set up an SSH tunnel that redirects network traffic to the rsync daemon. The login shell of the SSH account can be set to /usr/sbin/nologin (or something similar like /bin/false) to disable command execution, in this case you need to pass -N to the SSH client.
The latest version of rsync-system-backup is available on PyPI and GitHub. The documentation is hosted on Read the Docs and includes a changelog. For bug reports please create an issue on GitHub. If you have questions, suggestions, etc. feel free to send me an e-mail at peter@peterodding.com.
This software is licensed under the MIT license.
© 2019 Peter Odding.