Commit f4aa3b4e authored by remy's avatar remy
Browse files

borgbackup recipe public first commit

parent f55a008e
# borgbackup
# BorgBackup SaltStack formula
> Note: this formula is provided "as is" without any warranty of any kind. It has been tested successfully on my debian-like minions.
## Requirements
First, you need a working [`borgbackup`](https://www.borgbackup.org/) server (in fact, it just needs ssh and a huge amount of free space).
Then, you need to setup a SaltStack grain's list named `pool` to determine current network subnets. Take a look at [this formula](https://gitlab.mbb.univ-montp2.fr/saltstack-formulas/set_grains) if needed.
For example, here I am using `isem212` for our network `192.168.212.0/24`.
Then, check the `pillar.example` file to manage your borg server(s).
## Usage
The `borbackup.server` formula will then add the borg user on the server.
```bash
# $minion_borg_server is your borg server's SaltStack minion id
salt '$minion_borg_server' -v state.sls borgbackup.server
```
In order to add the client rsa key, you may need to create the key.
In that case, you will just have to apply that formula:
```bash
salt '$minion' -v state.sls borgbackup.generate_rsa_key
```
Then, refresh mines and apply the add_clients recipe, to sync ssh keys:
```bash
salt '*' mine.update
salt '$minion_borg_server' -v state.sls borgbackup.add_clients
```
To configure backup directories, please take a look at `hosts212.pillar.example`.
`borgbackup.client` will then look at `RsyncShareName` values and `BackupFilesExclude` (I kept the BackupPC naming format for formula compatibility).
The borgclient recipe installs `borgbackup` and [`borgmatic`](https://torsion.org/borgmatic/) using pip3.
`borgmatic` allows us to use an easier way to manage the borg backups.
```bash
salt '$minion' -v state.sls borgbackup.client
```
References:
- https://torsion.org/borgmatic/docs/how-to/set-up-backups/
- https://doc.ubuntu-fr.org/borgbackup
- https://sebsauvage.net/wiki/doku.php?id=borgbackup
- https://www.youtube.com/watch?v=QIMJdnghEmU
- https://borgbackup.readthedocs.io/en/stable/deployment/central-backup-server.html
- https://wiki.fiat-tux.fr/books/administration-syst%C3%A8mes/page/borgmatic
- https://torsion.org/borgmatic/docs/how-to/set-up-backups/
### Comparing to BackupPC
With BorgBackup, there is no real Borg server, contrary to BackupPC. Indeed, by default, there is no AdminUI/WebUI.
Moreover, connections are done by SSH from the clients; they connect to the server to backup their data.
With BackupPC, it is the opposite : connections are initialized by the server to retrieve data on the clients.
However, with deduplication and encryption, BorgBackup is really more powerful.
This recipe is helpful to add ssh authorized_keys on the borg server. SSH RSA Keys are taken on each minion using an updated mine.
If the key does not exist on the server side, it is added by this formula.
This formula also create an empty directory for the client host, to allow it to create a borg backup repository on the borg server.
Usage:
```bash
salt '$minion_borg_server' state.sls borgbackup.add_clients
```
{% set allkeys = salt['mine.get']('*', 'get_ssh_rsa_key') %}
{% set BORGDATA = salt['pillar.get']('borg:servers') %}
{% for server, data in BORGDATA.items() %}
{% if server == grains['fqdn_ip4'][0] %}
{% set mainrepo = data['repo'] %}
{% for host, rsakey in allkeys.items() %}
creating empty directory repo for {{ host }}:
cmd.run:
- name: "mkdir -p {{ mainrepo }}/{{ host }}"
- shell: /bin/bash
- runas: {{ data['borguser'] }}
# CAUTION!
# If you change the ssh command= option below, it won't necessarily get pushed to the backup
# server correctly unless you delete the ~/.ssh/authorized_keys file and re-create it!
adding_rsa_key_{{ host }}:
ssh_auth.present:
- user: {{ data['borguser'] }}
- enc: {{ data['host_key_enc'] }}
- names:
- {{ rsakey }}
- options:
- command="cd {{ mainrepo }}/{{ host }}; borg --umask=077 serve --append-only --restrict-to-path {{ mainrepo }}/{{ host }}"
- restrict
{% endfor %}
{% endif %}
{% endfor %}
\ No newline at end of file
include:
- borgbackup.add_clients.add_keys
\ No newline at end of file
1. Be sure that the key has been added on the borg server (see `borgbackup.add_clients`) and the remote directory exists.
If not, refresh mines `salt '*' mine.update`, then re-run `borgbackup.add_clients` on borg server (check pillar `pillar.example`),
2. If the machine does not have enough memory, borgbackup build may be killed by the OS (2GB is enough),
3. By default, this recipe set the repository in append-only mode, avoiding issue with data corruptions,
4. SSH key used to access to the repository is the root default ssh key on client (without passphrase), and the server host ssh fingerprint is `ssh-rsa`.
Usage:
```bash
salt '$minion' state.sls borgbackup.client
```
#!/bin/bash
{% for borgserver, borgdata in salt['pillar.get']('borg:servers').items() %}{% if borgdata['pool'] in grains['pool'] %}
PATH=$PATH:/usr/bin:/usr/local/bin:/root/.local/bin BORG_PASSPHRASE={{ borgdata['passphrase'] }} borg key export {{ borgdata['borguser'] }}@{{ borgserver }}:{{ borgdata['repo'] }}/{{ grains['id'] }} /etc/ssl/private/borg-{{ borgserver }}.key
PATH=$PATH:/usr/bin:/usr/local/bin:/root/.local/bin BORG_PASSPHRASE={{ borgdata['passphrase'] }} borgmatic --verbosity 1 --files 2>&1 > /var/log/borginit.log
{% endif %}{% endfor %}
# Where to look for files to backup, and where to store those backups.
# See https://borgbackup.readthedocs.io/en/stable/quickstart.html and
# https://borgbackup.readthedocs.io/en/stable/usage/create.html
# for details.
location:
# List of source directories to backup (required). Globs and
# tildes are expanded.
source_directories:
{% for savedir in HOSTINFO['RsyncShareName'] %}- {{ savedir }}
{% endfor %}
# Paths to local or remote repositories (required). Tildes are
# expanded. Multiple repositories are backed up to in
# sequence. Borg placeholders can be used. See the output of
# "borg help placeholders" for details. See ssh_command for
# SSH options like identity file or port. If systemd service
# is used, then add local repository paths in the systemd
# service file to the ReadWritePaths list.
repositories:
{% for borgserver, borgdata in salt['pillar.get']('borg:servers').items() %}{% if borgdata['pool'] in grains['pool'] %}- {{ borgdata['borguser'] }}@{{ borgserver }}:{{ borgdata['repo'] }}/{{ grains['id'] }}
{% endif %}{% endfor %}
# Stay in same file system (do not cross mount points).
# Defaults to false. But when a database hook is used, the
# setting here is ignored and one_file_system is considered
# true.
# one_file_system: true
one_file_system: true
# Only store/extract numeric user and group identifiers.
# Defaults to false.
# numeric_owner: true
# Store atime into archive. Defaults to true.
# atime: false
# Store ctime into archive. Defaults to true.
# ctime: false
# Store birthtime (creation date) into archive. Defaults to
# true.
# birthtime: false
# Use Borg's --read-special flag to allow backup of block and
# other special devices. Use with caution, as it will lead to
# problems if used when backing up special devices such as
# /dev/zero. Defaults to false. But when a database hook is
# used, the setting here is ignored and read_special is
# considered true.
# read_special: false
# Record bsdflags (e.g. NODUMP, IMMUTABLE) in archive.
# Defaults to true.
# bsd_flags: true
# Mode in which to operate the files cache. See
# http://borgbackup.readthedocs.io/en/stable/usage/create.html
# for details. Defaults to "ctime,size,inode".
# files_cache: ctime,size,inode
# Alternate Borg local executable. Defaults to "borg".
# local_path: borg1
# Alternate Borg remote executable. Defaults to "borg".
# remote_path: borg1
# Any paths matching these patterns are included/excluded from
# backups. Globs are expanded. (Tildes are not.) Note that
# Borg considers this option experimental. See the output of
# "borg help patterns" for more details. Quote any value if it
# contains leading punctuation, so it parses correctly.
# patterns:
# - R /
# - '- /home/*/.cache'
# - + /home/susan
# - '- /home/*'
# Read include/exclude patterns from one or more separate
# named files, one pattern per line. Note that Borg considers
# this option experimental. See the output of "borg help
# patterns" for more details.
# patterns_from:
# - /etc/borgmatic/patterns
# Any paths matching these patterns are excluded from backups.
# Globs and tildes are expanded. See the output of "borg help
# patterns" for more details.
exclude_patterns:
- '*.pyc'
- '*/.ssh'
- '*/.git'
- '*.tmp'
- '*.swp'
- '*/.cache'
- /etc/ssl
- /etc/ssh
- '*/deprecated*'
{% if 'BackupFilesExclude' in HOSTINFO %}{% for excldir in HOSTINFO['BackupFilesExclude'] %}- {{ excldir }}
{% endfor %}{% endif %}
# Read exclude patterns from one or more separate named files,
# one pattern per line. See the output of "borg help patterns"
# for more details.
# exclude_from:
# - /etc/borgmatic/excludes
# Exclude directories that contain a CACHEDIR.TAG file. See
# http://www.brynosaurus.com/cachedir/spec.html for details.
# Defaults to false.
# exclude_caches: true
# Exclude directories that contain a file with the given
# filenames. Defaults to not set.
# exclude_if_present:
# - .nobackup
# If true, the exclude_if_present filename is included in
# backups. Defaults to false, meaning that the
# exclude_if_present filename is omitted from backups.
# keep_exclude_tags: true
# Exclude files with the NODUMP flag. Defaults to false.
# exclude_nodump: true
# Path for additional source files used for temporary internal
# state like borgmatic database dumps. Note that changing this
# path prevents "borgmatic restore" from finding any database
# dumps created before the change. Defaults to ~/.borgmatic
# borgmatic_source_directory: /tmp/borgmatic
# Repository storage options. See
# https://borgbackup.readthedocs.io/en/stable/usage/create.html and
# https://borgbackup.readthedocs.io/en/stable/usage/general.html for
# details.
storage:
# The standard output of this command is used to unlock the
# encryption key. Only use on repositories that were
# initialized with passcommand/repokey/keyfile encryption.
# Note that if both encryption_passcommand and
# encryption_passphrase are set, then encryption_passphrase
# takes precedence. Defaults to not set.
# encryption_passcommand: secret-tool lookup borg-repository repo-name
# Passphrase to unlock the encryption key with. Only use on
# repositories that were initialized with
# passphrase/repokey/keyfile encryption. Quote the value if it
# contains punctuation, so it parses correctly. And backslash
# any quote or backslash literals as well. Defaults to not
# set.
# encryption_passphrase: "!\"#$%&'()*+,-./:;<=>?@[\\]^_`{|}~"
# Number of seconds between each checkpoint during a
# long-running backup. See
# https://borgbackup.readthedocs.io/en/stable/faq.html
# for details. Defaults to checkpoints every 1800 seconds (30
# minutes).
# checkpoint_interval: 1800
# Specify the parameters passed to then chunker
# (CHUNK_MIN_EXP, CHUNK_MAX_EXP, HASH_MASK_BITS,
# HASH_WINDOW_SIZE). See
# https://borgbackup.readthedocs.io/en/stable/internals.html
# for details. Defaults to "19,23,21,4095".
# chunker_params: 19,23,21,4095
# Type of compression to use when creating archives. See
# http://borgbackup.readthedocs.io/en/stable/usage/create.html
# for details. Defaults to "lz4".
# compression: lz4
compression: zstd
# Remote network upload rate limit in kiBytes/second. Defaults
# to unlimited.
# remote_rate_limit: 100
# Directory where temporary files are stored. Defaults to
# $TMPDIR
# temporary_directory: /path/to/tmpdir
# Command to use instead of "ssh". This can be used to specify
# ssh options. Defaults to not set.
# ssh_command: ssh -i /path/to/private/key
# Base path used for various Borg directories. Defaults to
# $HOME, ~$USER, or ~.
# borg_base_directory: /path/to/base
# Path for Borg configuration files. Defaults to
# $borg_base_directory/.config/borg
# borg_config_directory: /path/to/base/config
# Path for Borg cache files. Defaults to
# $borg_base_directory/.cache/borg
# borg_cache_directory: /path/to/base/cache
# Path for Borg security and encryption nonce files. Defaults
# to $borg_base_directory/.config/borg/security
# borg_security_directory: /path/to/base/config/security
# Path for Borg encryption key files. Defaults to
# $borg_base_directory/.config/borg/keys
# borg_keys_directory: /path/to/base/config/keys
# Umask to be used for borg create. Defaults to 0077.
# umask: 0077
# Maximum seconds to wait for acquiring a repository/cache
# lock. Defaults to 1.
# lock_wait: 5
# Name of the archive. Borg placeholders can be used. See the
# output of "borg help placeholders" for details. Defaults to
# "{hostname}-{now:%Y-%m-%dT%H:%M:%S.%f}". If you specify this
# option, you must also specify a prefix in the retention
# section to avoid accidental pruning of archives with a
# different archive name format. And you should also specify a
# prefix in the consistency section as well.
# archive_name_format: '{hostname}-documents-{now}'
# Bypass Borg error about a repository that has been moved.
# Defaults to false.
# relocated_repo_access_is_ok: true
# Bypass Borg error about a previously unknown unencrypted
# repository. Defaults to false.
# unknown_unencrypted_repo_access_is_ok: true
# Additional options to pass directly to particular Borg
# commands, handy for Borg options that borgmatic does not yet
# support natively. Note that borgmatic does not perform any
# validation on these options. Running borgmatic with
# "--verbosity 2" shows the exact Borg command-line
# invocation.
# extra_borg_options:
# Extra command-line options to pass to "borg init".
# init: --make-parent-dirs
# Extra command-line options to pass to "borg prune".
# prune: --save-space
# Extra command-line options to pass to "borg create".
# create: --no-files-cache
# Extra command-line options to pass to "borg check".
# check: --save-space
# Retention policy for how many backups to keep in each category. See
# https://borgbackup.readthedocs.io/en/stable/usage/prune.html for
# details. At least one of the "keep" options is required for pruning
# to work. To skip pruning entirely, run "borgmatic create" or "check"
# without the "prune" action. See borgmatic documentation for details.
retention:
# Keep all archives within this time interval.
# keep_within: 3H
# Number of secondly archives to keep.
# keep_secondly: 60
# Number of minutely archives to keep.
# keep_minutely: 60
# Number of hourly archives to keep.
# keep_hourly: 24
# Number of daily archives to keep.
keep_daily: 7
# Number of weekly archives to keep.
keep_weekly: 3
# Number of monthly archives to keep.
keep_monthly: 2
# Number of yearly archives to keep.
# keep_yearly: 1
# When pruning, only consider archive names starting with this
# prefix. Borg placeholders can be used. See the output of
# "borg help placeholders" for details. Defaults to
# "{hostname}-". Use an empty value to disable the default.
# prefix: sourcehostname
# Consistency checks to run after backups. See
# https://borgbackup.readthedocs.io/en/stable/usage/check.html and
# https://borgbackup.readthedocs.io/en/stable/usage/extract.html for
# details.
# consistency:
# List of one or more consistency checks to run: "repository",
# "archives", "data", and/or "extract". Defaults to
# "repository" and "archives". Set to "disabled" to disable
# all consistency checks. "repository" checks the consistency
# of the repository, "archives" checks all of the archives,
# "data" verifies the integrity of the data within the
# archives, and "extract" does an extraction dry-run of the
# most recent archive. Note that "data" implies "archives".
# checks:
# - repository
# - archives
# Paths to a subset of the repositories in the location
# section on which to run consistency checks. Handy in case
# some of your repositories are very large, and so running
# consistency checks on them would take too long. Defaults to
# running consistency checks on all repositories configured in
# the location section.
# check_repositories:
# - user@backupserver:sourcehostname.borg
# Restrict the number of checked archives to the last n.
# Applies only to the "archives" check. Defaults to checking
# all archives.
# check_last: 3
# When performing the "archives" check, only consider archive
# names starting with this prefix. Borg placeholders can be
# used. See the output of "borg help placeholders" for
# details. Defaults to "{hostname}-". Use an empty value to
# disable the default.
# prefix: sourcehostname
# Options for customizing borgmatic's own output and logging.
# output:
# Apply color to console output. Can be overridden with
# --no-color command-line flag. Defaults to true.
# color: false
# Shell commands, scripts, or integrations to execute at various
# points during a borgmatic run. IMPORTANT: All provided commands and
# scripts are executed with user permissions of borgmatic. Do not
# forget to set secure permissions on this configuration file (chmod
# 0600) as well as on any script called from a hook (chmod 0700) to
# prevent potential shell injection or privilege escalation.
# hooks:
# List of one or more shell commands or scripts to execute
# before creating a backup, run once per configuration file.
# before_backup:
# - echo "Starting a backup."
# List of one or more shell commands or scripts to execute
# before pruning, run once per configuration file.
# before_prune:
# - echo "Starting pruning."
# List of one or more shell commands or scripts to execute
# before consistency checks, run once per configuration file.
# before_check:
# - echo "Starting checks."
# List of one or more shell commands or scripts to execute
# before extracting a backup, run once per configuration file.
# before_extract:
# - echo "Starting extracting."
# List of one or more shell commands or scripts to execute
# after creating a backup, run once per configuration file.
# after_backup:
# - echo "Finished a backup."
# List of one or more shell commands or scripts to execute
# after pruning, run once per configuration file.
# after_prune:
# - echo "Finished pruning."
# List of one or more shell commands or scripts to execute
# after consistency checks, run once per configuration file.
# after_check:
# - echo "Finished checks."
# List of one or more shell commands or scripts to execute
# after extracting a backup, run once per configuration file.
# after_extract:
# - echo "Finished extracting."
# List of one or more shell commands or scripts to execute
# when an exception occurs during a "prune", "create", or
# "check" action or an associated before/after hook.
# on_error:
# - echo "Error during prune/create/check."
# List of one or more shell commands or scripts to execute
# before running all actions (if one of them is "create").
# These are collected from all configuration files and then
# run once before all of them (prior to all actions).
# before_everything:
# - echo "Starting actions."
# List of one or more shell commands or scripts to execute
# after running all actions (if one of them is "create").
# These are collected from all configuration files and then
# run once after all of them (after any action).
# after_everything:
# - echo "Completed actions."
# List of one or more PostgreSQL databases to dump before
# creating a backup, run once per configuration file. The
# database dumps are added to your source directories at
# runtime, backed up, and removed afterwards. Requires
# pg_dump/pg_dumpall/pg_restore commands. See
# https://www.postgresql.org/docs/current/app-pgdump.html and
# https://www.postgresql.org/docs/current/libpq-ssl.html for
# details.
# postgresql_databases:
# Database name (required if using this hook). Or
# "all" to dump all databases on the host. Note
# that using this database hook implicitly enables
# both read_special and one_file_system (see
# above) to support dump and restore streaming.
# - name: users
# Database hostname to connect to. Defaults to
# connecting via local Unix socket.
# hostname: database.example.org
# Port to connect to. Defaults to 5432.
# port: 5433
# Username with which to connect to the database.
# Defaults to the username of the current user.
# You probably want to specify the "postgres"
# superuser here when the database name is "all".
# username: dbuser
# Password with which to connect to the database.
# Omitting a password will only work if PostgreSQL
# is configured to trust the configured username
# without a password, or you create a ~/.pgpass
# file.
# password: trustsome1
# Database dump output format. One of "plain",
# "custom", "directory", or "tar". Defaults to
# "custom" (unlike raw pg_dump). See pg_dump
# documentation for details. Note that format is
# ignored when the database name is "all".
# format: directory
# SSL mode to use to connect to the database
# server. One of "disable", "allow", "prefer",
# "require", "verify-ca" or "verify-full".
# Defaults to "disable".
# ssl_mode: require
# Path to a client certificate.
# ssl_cert: /root/.postgresql/postgresql.crt
# Path to a private client key.
# ssl_key: /root/.postgresql/postgresql.key
# Path to a root certificate containing a list of
# trusted certificate authorities.
# ssl_root_cert: /root/.postgresql/root.crt
# Path to a certificate revocation list.
# ssl_crl: /root/.postgresql/root.crl
# Additional pg_dump/pg_dumpall options to pass
# directly to the dump command, without performing
# any validation on them. See pg_dump
# documentation for details.
# options: --role=someone
# List of one or more MySQL/MariaDB databases to dump before
# creating a backup, run once per configuration file. The
# database dumps are added to your source directories at
# runtime, backed up, and removed afterwards. Requires
# mysqldump/mysql commands (from either MySQL or MariaDB). See
# https://dev.mysql.com/doc/refman/8.0/en/mysqldump.html or
# https://mariadb.com/kb/en/library/mysqldump/ for details.
# mysql_databases:
# Database name (required if using this hook). Or
# "all" to dump all databases on the host. Note
# that using this database hook implicitly enables
# both read_special and one_file_system (see
# above) to support dump and restore streaming.
# - name: users
# Database hostname to connect to. Defaults to
# connecting via local Unix socket.
# hostname: database.example.org
# Port to connect to. Defaults to 3306.
# port: 3307
# Username with which to connect to the database.
# Defaults to the username of the current user.
# username: dbuser
# Password with which to connect to the database.
# Omitting a password will only work if MySQL is
# configured to trust the configured username
# without a password.
# password: trustsome1
# Additional mysqldump options to pass directly to
# the dump command, without performing any
# validation on them. See mysqldump documentation
# for details.
# options: --skip-comments
# Healthchecks ping URL or UUID to notify when a backup
# begins, ends, or errors. Create an account at
# https://healthchecks.io if you'd like to use this service.
# See borgmatic monitoring documentation for details. <