Backup Block

Table of Contents

Features
Usage
Demo
Monitoring
Maintenance
Options Reference

Defined in /modules/blocks/backup.nix.

This block sets up backup jobs for Self Host Blocks.

Features

Two implementations for this block are provided:

No integration tests are provided yet.

Usage

One folder backed up to mounted hard drives

The following snippet shows how to configure backup of 1 folder using the Restic implementation to 1 repository.

Assumptions:

  • 1 hard drive pool is used for backup and is mounted on /srv/pool1.

shb.backup.instances.myfolder = {
  enable = true;

  backend = "restic";

  keySopsFile = ./secrets.yaml;

  repositories = [{
    path = "/srv/pool1/backups/myfolder";
    timerConfig = {
      OnCalendar = "00:00:00";
      RandomizedDelaySec = "3h";
    };
  }];

  sourceDirectories = [
    "/var/lib/myfolder"
  ];

  retention = {
    keep_within = "1d";
    keep_hourly = 24;
    keep_daily = 7;
    keep_weekly = 4;
    keep_monthly = 6;
  };

  consistency = {
    repository = "2 weeks";
    archives = "1 month";
  };
};

The referenced Sops file must follow this structure:

restic:
    passphrases:
        myfolder: <secret>

To generate a secret, use: nix run nixpkgs#openssl -- rand -hex 64.

With the borgmatic implementation, the structure should be:

borgmatic:
    keys:
        myfolder: |
            BORG_KEY <key>
    passphrases:
        myfolder: <secret>

You can have both borgmatic and restic implementations working at the same time.

One folder backed up to S3

This is only supported by the Restic implementation.

Here we will only highlight the differences with the previous configuration.

This assumes you have access to such a remote S3 store, for example by using Backblaze.

  shb.backup.instances.myfolder = {

    repositories = [{
-     path = "/srv/pool1/backups/myfolder";
+     path = "s3:s3.us-west-000.backblazeb2.com/backups/myfolder";
      timerConfig = {
        OnCalendar = "00:00:00";
        RandomizedDelaySec = "3h";
      };
    }];


+   environmentFile = true; # Needed for s3
  }

The Sops file has a new required field:


  restic:
      passphrases:
          myfolder: <secret>
+     environmentfiles:
+         myfolder: |-
+             AWS_ACCESS_KEY_ID=<aws_key_id>
+             AWS_SECRET_ACCESS_KEY=<aws_secret_key>

Multiple folder to multiple destinations

The following snippet shows how to configure backup of any number of folders using the Restic implementation to 3 repositories, each happening at different times to avoid contending for I/O time.

We will also make sure to be able to re-use as much as the configuration as possible.

A few assumptions:

  • 2 hard drive pools used for backup are mounted respectively on /srv/pool1 and /srv/pool2.

  • You have a backblaze account.

First, let’s define a variable to hold all our repositories you want to back up to:

repos = [
  {
    path = "/srv/pool1/backups";
    timerConfig = {
      OnCalendar = "00:00:00";
      RandomizedDelaySec = "3h";
    };
  }
  {
    path = "/srv/pool2/backups";
    timerConfig = {
      OnCalendar = "08:00:00";
      RandomizedDelaySec = "3h";
    };
  }
  {
    path = "s3:s3.us-west-000.backblazeb2.com/backups";
    timerConfig = {
      OnCalendar = "16:00:00";
      RandomizedDelaySec = "3h";
    };
  }
];

Compared to the previous examples, we do not include the name of what we will back up in the repository paths.

Now, let’s define a function to create a backup configuration. It will take a list of repositories, a name identifying the backup and a list of folders to back up.

backupcfg = repositories: name: sourceDirectories {
  enable = true;

  backend = "restic";

  keySopsFile = ../secrets/backup.yaml;

  repositories = builtins.map (r: {
    path = "${r.path}/${name}";
    inherit (r) timerConfig;
  }) repositories;

  inherit sourceDirectories;

  retention = {
    keep_within = "1d";
    keep_hourly = 24;
    keep_daily = 7;
    keep_weekly = 4;
    keep_monthly = 6;
  };

  consistency = {
    repository = "2 weeks";
    archives = "1 month";
  };

  environmentFile = true;
};

Now, we can define multiple backup jobs to backup different folders:

shb.backup.instances.myfolder1 = backupcfg repos ["/var/lib/myfolder1"];
shb.backup.instances.myfolder2 = backupcfg repos ["/var/lib/myfolder2"];

The difference between the above snippet and putting all the folders into one configuration (shown below) is the former splits the backups into sub-folders on the repositories.

shb.backup.instances.all = backupcfg repos ["/var/lib/myfolder1" "/var/lib/myfolder2"];

Demo

[WIP]

Monitoring

[WIP]

Maintenance

[WIP]

Options Reference

shb.backup.borgServer

Add borgbackup package so external backups can use this server as a remote.

Type: boolean

Default: false

Example: true

Declared by:

<selfhostblocks/modules/blocks/backup.nix>
shb.backup.group

Unix group doing the backups.

Type: string

Default: "backup"

Declared by:

<selfhostblocks/modules/blocks/backup.nix>
shb.backup.instances

Each instance is a backup setting

Type: attribute set of (submodule)

Default: { }

Declared by:

<selfhostblocks/modules/blocks/backup.nix>
shb.backup.instances.<name>.enable

Whether to enable shb backup instance.

Type: boolean

Default: false

Example: true

Declared by:

<selfhostblocks/modules/blocks/backup.nix>
shb.backup.instances.<name>.backend

What program to use to make the backups.

Type: one of “borgmatic”, “restic”

Example: "borgmatic"

Declared by:

<selfhostblocks/modules/blocks/backup.nix>
shb.backup.instances.<name>.consistency

Consistency frequency options. Only applicable for borgmatic

Type: attribute set of non-empty string

Default: { }

Example:

{
  archives = "1 month";
  repository = "2 weeks";
}

Declared by:

<selfhostblocks/modules/blocks/backup.nix>
shb.backup.instances.<name>.environmentFile

Add environment file to be read by the systemd service.

Type: boolean

Default: false

Example: true

Declared by:

<selfhostblocks/modules/blocks/backup.nix>
shb.backup.instances.<name>.excludePatterns

Borgmatic exclude patterns.

Type: list of string

Default: [ ]

Declared by:

<selfhostblocks/modules/blocks/backup.nix>
shb.backup.instances.<name>.hooks

Borgmatic hooks.

Type: submodule

Default: { }

Declared by:

<selfhostblocks/modules/blocks/backup.nix>
shb.backup.instances.<name>.hooks.after_backup

Hooks to run after backup

Type: list of string

Default: [ ]

Declared by:

<selfhostblocks/modules/blocks/backup.nix>
shb.backup.instances.<name>.hooks.before_backup

Hooks to run before backup

Type: list of string

Default: [ ]

Declared by:

<selfhostblocks/modules/blocks/backup.nix>
shb.backup.instances.<name>.keySopsFile

Sops file that holds this instance’s Borgmatic repository key and passphrase.

Type: path

Example: "secrets/backup.yaml"

Declared by:

<selfhostblocks/modules/blocks/backup.nix>
shb.backup.instances.<name>.repositories

Repositories to back this instance to.

Type: non-empty (list of (submodule))

Declared by:

<selfhostblocks/modules/blocks/backup.nix>
shb.backup.instances.<name>.repositories.*.path

Repository location

Type: string

Declared by:

<selfhostblocks/modules/blocks/backup.nix>
shb.backup.instances.<name>.repositories.*.timerConfig

When to run the backup. See systemd.timer(5) for details.

Type: attribute set of (systemd option)

Default:

{
  OnCalendar = "daily";
  Persistent = true;
}

Example:

{
  OnCalendar = "00:05";
  Persistent = true;
  RandomizedDelaySec = "5h";
}

Declared by:

<selfhostblocks/modules/blocks/backup.nix>
shb.backup.instances.<name>.retention

Retention options.

Type: attribute set of (signed integer or non-empty string)

Default:

{
  keep_daily = 7;
  keep_hourly = 24;
  keep_monthly = 6;
  keep_weekly = 4;
  keep_within = "1d";
}

Declared by:

<selfhostblocks/modules/blocks/backup.nix>
shb.backup.instances.<name>.secretName

Secret name, if null use the name of the backup instance.

Type: null or string

Default: null

Declared by:

<selfhostblocks/modules/blocks/backup.nix>
shb.backup.instances.<name>.sourceDirectories

Borgmatic source directories.

Type: non-empty (list of string)

Declared by:

<selfhostblocks/modules/blocks/backup.nix>
shb.backup.onlyOnAC

Run backups only if AC power is plugged in.

Type: boolean

Default: true

Example: false

Declared by:

<selfhostblocks/modules/blocks/backup.nix>
shb.backup.performance

Reduce performance impact of backup jobs.

Type: submodule

Default: { }

Declared by:

<selfhostblocks/modules/blocks/backup.nix>
shb.backup.performance.ioPriority

ionice priority, defaults to 7 for lowest priority IO. Only used for restic backup, restic forget and restic check commands.

Type: null or integer between 0 and 7 (both inclusive)

Default: 7

Declared by:

<selfhostblocks/modules/blocks/backup.nix>
shb.backup.performance.ioSchedulingClass

ionice scheduling class, defaults to best-effort IO. Only used for restic backup, restic forget and restic check commands.

Type: one of “idle”, “best-effort”, “realtime”

Default: "best-effort"

Declared by:

<selfhostblocks/modules/blocks/backup.nix>
shb.backup.performance.niceness

nice priority adjustment, defaults to 15 for ~20% CPU time of normal-priority process

Type: integer between -20 and 19 (both inclusive)

Default: 15

Declared by:

<selfhostblocks/modules/blocks/backup.nix>
shb.backup.user

Unix user doing the backups.

Type: string

Default: "backup"

Declared by:

<selfhostblocks/modules/blocks/backup.nix>