Restic | Alt255 Blog

February 20, 2019

Restic

Intro

I’ve been using the restic backup system for a bit over a year now. It seems to be working well, was easy to configure, and is speedy.

It’s got the cryptos (client-side!), deduplication, snapshotting, and supports bacically all the cloud backends. I haven’t had any disasters to recover from yet, so the recovery system isn’t battle-tested from my perspective, but for one-off file recoveries, it’s worked well. Just like other common utilities, like rsync, it’s easy to configure a backup destination, called a “repository” in restic-speak, to help facilitate learning and debugging usage before committing uplink bandwidth and storage costs.

The restic documentation is thorough and nicely written, I recommend purusing it.

To install restic, there are a bunch of options depending on your OS and preferences. I don’t like waiting for official, packages versions of software to trickle down to my platform’s repos (there’s even a note stating that the Debian version is ancient). But even more importantly when it comes to backup software, I don’t want my backup software to change out from under me without my knowledge and I absolutely must be able to retreive the last-known-good version of the software. My preference is to git clone and build locally; then copy the binary to /usr/local/bin/restic-vX.X and symlink /usr/local/bin/restic.

The initial configuration of restic is covered in Preparing a new repository. It basically boils down to restic -r ${LOCATION OF REPO} init; the specifics on how to specify the repo location depends on the type of repo you’re using. Restic supports a handful of popular storage backends, notably local (ie. another location on the filesystem), S3 API, and Backblaze B2.

Restic also supports a backend called rclone, which is a reference to the rclone project. rclone bills itself as “rsync for cloud storage” and supports over 30 different storage backends(!). rclone, like rsync is good for syncing/copying files between two locations, but, like rsync, doesn’t make for a robust backup solution in itself. Because rclone has such wide and capable backend support, the two projects collaborated together to allow restic to use rclone for backends that restic didn’t already support. Setting up a restic repository to backup to an rclone backend is slightly more complicated because first rclone needs to be configured and then restic is configured separately. Also, while rclone is a very capable utility and I’m thankful that it exists, its UI choices are... unique.

Example usage

In this example, I have an S3 bucket already created where I’ll store my restic backups. IIRC, initial configuration was:

$ restic -r 's3:https://s3.us-west-1.amazonaws.com/example.com/restic/system' init
$ restic -r 's3:https://s3.us-west-1.amazonaws.com/example.com/restic/home' init

I have two restic repositories in this example, more on that later. Here’s the script I invoke via crontab or manually:

#!/bin/bash

set -e   # Exit on error

# Add `restic` binary to PATH:
export PATH=$PATH:/usr/local/bin

# Customize where restic stores it's temp files:
export TMPDIR='/archive/restic/tmp/'

# i18n, some files have "exotic" characters
export LC_ALL=en_US.UTF-8

# Load S3 credentials:
source /root/.restic/s3-example.com-credentials.sh

QUIET="--quiet"
if [ -n "${NOQUIET}" -o -n "${VERBOSE}" ]; then
  QUIET=""
fi

[[ -z "${QUIET}" ]] && echo "Starting backup set: system"
export RESTIC_REPOSITORY='s3:https://s3.us-west-1.amazonaws.com/example.com/restic/system'
restic backup ${QUIET} \
  --cache-dir="$HOME/.cache/restic-system" \
  --exclude-caches \
  --exclude-if-present '.nobackup' \
  --exclude=**/.cache/restic* \
  /etc /root /var/spool/cron/crontabs

[[ -z "${QUIET}" ]] && echo "Starting backup set: home"
export RESTIC_REPOSITORY='s3:https://s3.us-west-1.amazonaws.com/example.com/restic/home'
restic backup ${QUIET} \
  --exclude-caches \
  --exclude-if-present '.nobackup' \
  --cache-dir="$HOME/.cache/restic-home" \
  --exclude=**/.cache \
  --exclude=**/.local \
  --exclude="/home/jburke/.dropbox" \
  --exclude="/home/jburke/Dropbox/.dropbox.cache" \
  /home 

(stoneja, look, a bash script! Your favorite! ;-))

I run this script as root so that it can backup everything regardless of file permissions. I suppose I could split it out so that only the system files were backed up as root and everything else under my home directory is backed up as my user account, but I don’t see a strong reason to do so.

I keep logically separate things in different backup repositories. Above, there’s a “system” repo for system files and a “home” repo for home directories. This separation is mostly a matter of preference, but would allow me to administer each repo with different policies (eg. put them on different cleanup & deletion intervals).

(Pro tip: if you maintain multiple repositories, make sure to maintain a different cache directory for each repo. In the above examples, there’s a different --cache-dir option for each of the repos. As far as I’ve experienced, nothing bad happens if the same cache directories are used (by accident or otherwise), I think restic is able to detect the error and will blow away a bogus cache. Keeping the caches separate means that a cache for each repos is available and probably speeds things up.)

I left some examples above showing how to exclude files from the backups. One technique is to use the --exclude-if-present '.nobackup' option; if I drop a file called .nobackup into a directory, then that entire directory subtree will be skipped by restic. Another option is to exclude by path name (wildcards are possible, **). The --exclude-caches option will skip any subtree containing a file named CACHEDIR.TAG; so that’s similar to my preferred .nobackup technique, but CACHEDIR.TAG is what restic uses for the directories it maintains.

About the credentials file: First, what’s in it?

# Example credentials file for an S3 backend:
export RESTIC_PASSWORD='some complex passphrase that I do not need to remember'
export AWS_ACCESS_KEY_ID='self explanatory'
export AWS_SECRET_ACCESS_KEY='ditto'

The RESTIC_PASSWORD variable is the password to the repository. Without that password, the repo is (hopefully) unreadable. The other two variables are S3 specific. Since I don’t need to remember the repo passphrase, I use something like this to generate it: tr -dc "[:graph:]" < /dev/urandom | tr -d "'" | head -c 32.

Second point about the credentials file: Why not just incorporate those variables into the backup script? I do this for a few reasons: (A) I occasionally edit the backup script and it feels icky to see credentials, even if I know that they’re sitting in nearby plaintext file. (B) it gives me some flexibility: for example, it allows me to keep the credentials encrypted on disk, but decrypt them into memory before enabling restic.

Usage on a laptop

When running from a laptop, which connects to different networks and will acquire different network configurations from time to time, it’s helpful to tell restic explictly use a single specific hostname via the --hostname option. IIRC, the reason this is helpful is that restic tags backup sets with the hostname. And if the hostname changes, you might panic one day when you go to search for your backups and don’t see them at first. The previously backed up files are still there (phew!), but restic won’t show them to you by default because of the hostname mismatch. Speaking from experience, setting this variable might save yourself a minor heart attack.

AWS Configuration

Suppose that I want to backup host.example.com. After creating the S3 bucket via the AWS Console, I would create a new AWS user account specifically for the restic client and would set policies on the bucket.

AWS IAM

One of the things that AWS has really nailed is RBAC and user policy. It’s fine-grained, the syntax is reasonable, and it’s easy to update and apply policies. For each of my restic clients, I will create a new user account. Each restic-client user account has, what I believe are, minimal access to an S3 bucket dedicated to the client; in other words, each machine that I backup has it’s own S3 bucket as well as it’s own AWS user account with permissions only to that bucket.

My restic-client accounts have two policies, the first policy allows the client to list buckets and the second policy grants permissions to a subtree in the bucket.

Listing buckets:

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Sid": "VisualEditor0",
            "Effect": "Allow",
            "Action": "s3:ListBucket",
            "Resource": "arn:aws:s3:::host.example.com"
        }
    ]
}

Granting permission to a subtree in the bucket:

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Sid": "VisualEditor0",
            "Effect": "Allow",
            "Action": [
                "s3:PutObject",
                "s3:GetObject",
                "s3:DeleteObject"
            ],
            "Resource": "arn:aws:s3:::host.example.com/restic/*"
        }
    ]
}

The intention of the above permissions is to allow the restic client to manipulate blobs only under the restic/ subtree in the host.example.com bucket.

S3 permission are listed here. What I specifically omit from the above client permissions is s3:DeleteObjectVersion. With s3:DeleteObject, only the most recent version of a file may be deleted. What is not allowed with s3:DeleteObject is the ability to delete older versions of a file. Old versions of files will automatically be deleted using bucket lifecycle rules (described below).

Bucket properties

After creating a dedicated bucket for a machine that I want to backup, I’ll configure bucket policies in AWS.

The first policy I enable is Default encryption. I set this to AES-256, which is a free option where AWS generates and keeps the encryption key. I figure this option is useful since it protects blobs stored on S3 while at rest on physical media. Restic has its own encryption which is maintained client-side, so the AWS encryption is a cost-free and performance-free defense-in-depth measure.

I also enable Versioning on the backup bucket. My understanding is that without some additional care (see Bucket Lifecycle rules section below), this option could result in wasted hidden costs from old versions of files accumulating. What this option does enable is that each version of a file uploaded to S3 will get its own version number. If a file is accidentally deleted from S3, for example, this Versioning functionally will allow me to recover an older, non-deleted version of the file.

Lifecycle Management

I thought that I had enabled automatic transition to Glacier for my backed up files via restic, but after reviewing my S3 configuration, that’s not the case. For some non-restic managed backups, I do have Glacier enabled, those backups are primarily for very large files.

Note to self: investigate enabling Glacier storage for restic backups. A quick Google search reveals that this topic has previously been brought up and dismissed as infeasible for restic.

Bucket Lifecycle rules

AWS storage costs, especially on Glacier, are pretty reasonable, but it’s not free. So I’ve configured my S3 buckets to permanently delete previous versions of files after 365 days from becoming a previous version. With the way that restic works, if I delete a file on my filesystem that restic has been backing up, a copy of that file will remain in the backups forever, unless I prune the backup archive. When I do prune the backup archive, the previously delete file will have been represented as one or more data chunks (stored under restic’s data directory). A file pruned from restic’s archive will be marked as deleted using the S3 API. With my S3 settings, that deleted file will be marked as deleted, but won’t be purged from S3’s storage system until 365 days after the mark for deletion. This is cheap insurance that if I do something terribly wrong with restic, or restic goes berserk and deletes all my backed up files, that I won’t immediately lose the backup.

Clean up expired object delete markers and incomplete multipart uploads: yes, clean that crap up, I don’t want to pay for it. And make sure to do so 7 days after the start of the upload.

Wrap up

This blog entry doesn’t (yet?) cover actually using the backup system. As in, restoring from backups. There are several ways to explore the backups, they are covered in the official docs.