borgbackup recipe public first commit

README.md

-# borgbackup
+# BorgBackup SaltStack formula
 
+> Note: this formula is provided "as is" without any warranty of any kind. It has been tested successfully on my debian-like minions.
+
+## Requirements
+
+First, you need a working [`borgbackup`](https://www.borgbackup.org/) server (in fact, it just needs ssh and a huge amount of free space).
+
+Then, you need to setup a SaltStack grain's list named `pool` to determine current network subnets. Take a look at [this formula](https://gitlab.mbb.univ-montp2.fr/saltstack-formulas/set_grains) if needed.
+For example, here I am using `isem212` for our network `192.168.212.0/24`.
+
+Then, check the `pillar.example` file to manage your borg server(s).
+
+## Usage
+
+The `borbackup.server` formula will then add the borg user on the server.
+
+```bash
+# $minion_borg_server is your borg server's SaltStack minion id
+salt '$minion_borg_server' -v state.sls borgbackup.server
+```
+
+In order to add the client rsa key, you may need to create the key.
+In that case, you will just have to apply that formula:
+
+```bash
+salt '$minion' -v state.sls borgbackup.generate_rsa_key
+```
+
+Then, refresh mines and apply the add_clients recipe, to sync ssh keys:
+
+```bash
+salt '*' mine.update
+salt '$minion_borg_server' -v state.sls borgbackup.add_clients
+```
+
+To configure backup directories, please take a look at `hosts212.pillar.example`.
+
+`borgbackup.client` will then look at `RsyncShareName` values and `BackupFilesExclude` (I kept the BackupPC naming format for formula compatibility).
+
+The borgclient recipe installs `borgbackup` and [`borgmatic`](https://torsion.org/borgmatic/) using pip3.
+
+`borgmatic` allows us to use an easier way to manage the borg backups.
+
+```bash
+salt '$minion' -v state.sls borgbackup.client
+```
+
+References:
+
+  - https://torsion.org/borgmatic/docs/how-to/set-up-backups/
+  - https://doc.ubuntu-fr.org/borgbackup
+  - https://sebsauvage.net/wiki/doku.php?id=borgbackup
+  - https://www.youtube.com/watch?v=QIMJdnghEmU
+  - https://borgbackup.readthedocs.io/en/stable/deployment/central-backup-server.html
+  - https://wiki.fiat-tux.fr/books/administration-syst%C3%A8mes/page/borgmatic
+  - https://torsion.org/borgmatic/docs/how-to/set-up-backups/
+
+### Comparing to BackupPC
+
+With BorgBackup, there is no real Borg server, contrary to BackupPC. Indeed, by default, there is no AdminUI/WebUI.
+Moreover, connections are done by SSH from the clients; they connect to the server to backup their data.
+With BackupPC, it is the opposite : connections are initialized by the server to retrieve data on the clients.
+
+However, with deduplication and encryption, BorgBackup is really more powerful.

add_clients/README.md

+This recipe is helpful to add ssh authorized_keys on the borg server. SSH RSA Keys are taken on each minion using an updated mine.
+If the key does not exist on the server side, it is added by this formula.
+
+This formula also create an empty directory for the client host, to allow it to create a borg backup repository on the borg server.
+
+Usage:
+
+```bash
+salt '$minion_borg_server' state.sls borgbackup.add_clients
+```

add_clients/add_keys.sls

+{% set allkeys = salt['mine.get']('*', 'get_ssh_rsa_key') %}
+{% set BORGDATA = salt['pillar.get']('borg:servers') %}
+{% for server, data in BORGDATA.items() %}
+ {% if server == grains['fqdn_ip4'][0] %}
+  {% set mainrepo = data['repo'] %}
+  {% for host, rsakey in allkeys.items() %}
+
+creating empty directory repo for {{ host }}:
+  cmd.run:
+    - name: "mkdir -p {{ mainrepo }}/{{ host }}"
+    - shell: /bin/bash
+    - runas: {{ data['borguser'] }}
+
+# CAUTION!
+# If you change the ssh command= option below, it won't necessarily get pushed to the backup
+# server correctly unless you delete the ~/.ssh/authorized_keys file and re-create it!
+adding_rsa_key_{{ host }}:
+  ssh_auth.present:
+    - user: {{ data['borguser'] }}
+    - enc: {{ data['host_key_enc'] }}
+    - names:
+      - {{ rsakey }}
+    - options:
+      - command="cd {{ mainrepo }}/{{ host }}; borg --umask=077 serve --append-only --restrict-to-path {{ mainrepo }}/{{ host }}"
+      - restrict
+
+  {% endfor %}
+ {% endif %}
+{% endfor %}
\ No newline at end of file

add_clients/init.sls

+include:
+  - borgbackup.add_clients.add_keys
\ No newline at end of file

client/README.md

+1. Be sure that the key has been added on the borg server (see `borgbackup.add_clients`) and the remote directory exists.
+  If not, refresh mines `salt '*' mine.update`, then re-run `borgbackup.add_clients` on borg server (check pillar `pillar.example`),
+2. If the machine does not have enough memory, borgbackup build may be killed by the OS (2GB is enough),
+3. By default, this recipe set the repository in append-only mode, avoiding issue with data corruptions,
+4. SSH key used to access to the repository is the root default ssh key on client (without passphrase), and the server host ssh fingerprint is `ssh-rsa`.
+
+Usage:
+
+```bash
+salt '$minion' state.sls borgbackup.client
+```

client/backupinit.sh.jinja

+#!/bin/bash
+
+{% for borgserver, borgdata in salt['pillar.get']('borg:servers').items() %}{% if borgdata['pool'] in grains['pool'] %}
+PATH=$PATH:/usr/bin:/usr/local/bin:/root/.local/bin BORG_PASSPHRASE={{ borgdata['passphrase'] }} borg key export {{ borgdata['borguser'] }}@{{ borgserver }}:{{ borgdata['repo'] }}/{{ grains['id'] }} /etc/ssl/private/borg-{{ borgserver }}.key
+PATH=$PATH:/usr/bin:/usr/local/bin:/root/.local/bin BORG_PASSPHRASE={{ borgdata['passphrase'] }} borgmatic --verbosity 1 --files 2>&1 > /var/log/borginit.log
+{% endif %}{% endfor %}

client/borgmatic.config.yaml

+# Where to look for files to backup, and where to store those backups.
+# See https://borgbackup.readthedocs.io/en/stable/quickstart.html and
+# https://borgbackup.readthedocs.io/en/stable/usage/create.html
+# for details.
+location:
+    # List of source directories to backup (required). Globs and
+    # tildes are expanded.
+    source_directories:
+        {% for savedir in HOSTINFO['RsyncShareName'] %}- {{ savedir }}
+        {% endfor %}
+
+    # Paths to local or remote repositories (required). Tildes are
+    # expanded. Multiple repositories are backed up to in
+    # sequence. Borg placeholders can be used. See the output of
+    # "borg help placeholders" for details. See ssh_command for
+    # SSH options like identity file or port. If systemd service
+    # is used, then add local repository paths in the systemd
+    # service file to the ReadWritePaths list.
+    repositories:
+      {% for borgserver, borgdata in salt['pillar.get']('borg:servers').items() %}{% if borgdata['pool'] in grains['pool'] %}- {{ borgdata['borguser'] }}@{{ borgserver }}:{{ borgdata['repo'] }}/{{ grains['id'] }}
+      {% endif %}{% endfor %}
+    # Stay in same file system (do not cross mount points).
+    # Defaults to false. But when a database hook is used, the
+    # setting here is ignored and one_file_system is considered
+    # true.
+    # one_file_system: true
+    one_file_system: true
+
+    # Only store/extract numeric user and group identifiers.
+    # Defaults to false.
+    # numeric_owner: true
+
+    # Store atime into archive. Defaults to true.
+    # atime: false
+
+    # Store ctime into archive. Defaults to true.
+    # ctime: false
+
+    # Store birthtime (creation date) into archive. Defaults to
+    # true.
+    # birthtime: false
+
+    # Use Borg's --read-special flag to allow backup of block and
+    # other special devices. Use with caution, as it will lead to
+    # problems if used when backing up special devices such as
+    # /dev/zero. Defaults to false. But when a database hook is
+    # used, the setting here is ignored and read_special is
+    # considered true.
+    # read_special: false
+
+    # Record bsdflags (e.g. NODUMP, IMMUTABLE) in archive.
+    # Defaults to true.
+    # bsd_flags: true
+
+    # Mode in which to operate the files cache. See
+    # http://borgbackup.readthedocs.io/en/stable/usage/create.html
+    # for details. Defaults to "ctime,size,inode".
+    # files_cache: ctime,size,inode
+
+    # Alternate Borg local executable. Defaults to "borg".
+    # local_path: borg1
+
+    # Alternate Borg remote executable. Defaults to "borg".
+    # remote_path: borg1
+
+    # Any paths matching these patterns are included/excluded from
+    # backups. Globs are expanded. (Tildes are not.) Note that
+    # Borg considers this option experimental. See the output of
+    # "borg help patterns" for more details. Quote any value if it
+    # contains leading punctuation, so it parses correctly.
+    # patterns:
+        # - R /
+        # - '- /home/*/.cache'
+        # - + /home/susan
+        # - '- /home/*'
+
+    # Read include/exclude patterns from one or more separate
+    # named files, one pattern per line. Note that Borg considers
+    # this option experimental. See the output of "borg help
+    # patterns" for more details.
+    # patterns_from:
+        # - /etc/borgmatic/patterns
+
+    # Any paths matching these patterns are excluded from backups.
+    # Globs and tildes are expanded. See the output of "borg help
+    # patterns" for more details.
+    exclude_patterns:
+        - '*.pyc'
+        - '*/.ssh'
+        - '*/.git'
+        - '*.tmp'
+        - '*.swp'
+        - '*/.cache'
+        - /etc/ssl
+        - /etc/ssh
+        - '*/deprecated*'
+        {% if 'BackupFilesExclude' in HOSTINFO %}{% for excldir in HOSTINFO['BackupFilesExclude'] %}- {{ excldir }}
+        {% endfor %}{% endif %}
+
+
+    # Read exclude patterns from one or more separate named files,
+    # one pattern per line. See the output of "borg help patterns"
+    # for more details.
+    # exclude_from:
+        # - /etc/borgmatic/excludes
+
+    # Exclude directories that contain a CACHEDIR.TAG file. See
+    # http://www.brynosaurus.com/cachedir/spec.html for details.
+    # Defaults to false.
+    # exclude_caches: true
+
+    # Exclude directories that contain a file with the given
+    # filenames. Defaults to not set.
+    # exclude_if_present:
+        # - .nobackup
+
+    # If true, the exclude_if_present filename is included in
+    # backups. Defaults to false, meaning that the
+    # exclude_if_present filename is omitted from backups.
+    # keep_exclude_tags: true
+
+    # Exclude files with the NODUMP flag. Defaults to false.
+    # exclude_nodump: true
+
+    # Path for additional source files used for temporary internal
+    # state like borgmatic database dumps. Note that changing this
+    # path prevents "borgmatic restore" from finding any database
+    # dumps created before the change. Defaults to ~/.borgmatic
+    # borgmatic_source_directory: /tmp/borgmatic
+
+# Repository storage options. See
+# https://borgbackup.readthedocs.io/en/stable/usage/create.html and
+# https://borgbackup.readthedocs.io/en/stable/usage/general.html for
+# details.
+storage:
+    # The standard output of this command is used to unlock the
+    # encryption key. Only use on repositories that were
+    # initialized with passcommand/repokey/keyfile encryption.
+    # Note that if both encryption_passcommand and
+    # encryption_passphrase are set, then encryption_passphrase
+    # takes precedence. Defaults to not set.
+    # encryption_passcommand: secret-tool lookup borg-repository repo-name
+
+    # Passphrase to unlock the encryption key with. Only use on
+    # repositories that were initialized with
+    # passphrase/repokey/keyfile encryption. Quote the value if it
+    # contains punctuation, so it parses correctly. And backslash
+    # any quote or backslash literals as well. Defaults to not
+    # set.
+    # encryption_passphrase: "!\"#$%&'()*+,-./:;<=>?@[\\]^_`{|}~"
+
+    # Number of seconds between each checkpoint during a
+    # long-running backup. See
+    # https://borgbackup.readthedocs.io/en/stable/faq.html
+    # for details. Defaults to checkpoints every 1800 seconds (30
+    # minutes).
+    # checkpoint_interval: 1800
+
+    # Specify the parameters passed to then chunker
+    # (CHUNK_MIN_EXP, CHUNK_MAX_EXP, HASH_MASK_BITS,
+    # HASH_WINDOW_SIZE). See
+    # https://borgbackup.readthedocs.io/en/stable/internals.html
+    # for details. Defaults to "19,23,21,4095".
+    # chunker_params: 19,23,21,4095
+
+    # Type of compression to use when creating archives. See
+    # http://borgbackup.readthedocs.io/en/stable/usage/create.html
+    # for details. Defaults to "lz4".
+    # compression: lz4
+    compression: zstd
+
+    # Remote network upload rate limit in kiBytes/second. Defaults
+    # to unlimited.
+    # remote_rate_limit: 100
+
+    # Directory where temporary files are stored. Defaults to
+    # $TMPDIR
+    # temporary_directory: /path/to/tmpdir
+
+    # Command to use instead of "ssh". This can be used to specify
+    # ssh options.  Defaults to not set.
+    # ssh_command: ssh -i /path/to/private/key
+
+    # Base path used for various Borg directories. Defaults to
+    # $HOME, ~$USER, or ~.
+    # borg_base_directory: /path/to/base
+
+    # Path for Borg configuration files. Defaults to
+    # $borg_base_directory/.config/borg
+    # borg_config_directory: /path/to/base/config
+
+    # Path for Borg cache files. Defaults to
+    # $borg_base_directory/.cache/borg
+    # borg_cache_directory: /path/to/base/cache
+
+    # Path for Borg security and encryption nonce files. Defaults
+    # to $borg_base_directory/.config/borg/security
+    # borg_security_directory: /path/to/base/config/security
+
+    # Path for Borg encryption key files. Defaults to
+    # $borg_base_directory/.config/borg/keys
+    # borg_keys_directory: /path/to/base/config/keys
+
+    # Umask to be used for borg create. Defaults to 0077.
+    # umask: 0077
+
+    # Maximum seconds to wait for acquiring a repository/cache
+    # lock. Defaults to 1.
+    # lock_wait: 5
+
+    # Name of the archive. Borg placeholders can be used. See the
+    # output of "borg help placeholders" for details. Defaults to
+    # "{hostname}-{now:%Y-%m-%dT%H:%M:%S.%f}". If you specify this
+    # option, you must also specify a prefix in the retention
+    # section to avoid accidental pruning of archives with a
+    # different archive name format. And you should also specify a
+    # prefix in the consistency section as well.
+    # archive_name_format: '{hostname}-documents-{now}'
+
+    # Bypass Borg error about a repository that has been moved.
+    # Defaults to false.
+    # relocated_repo_access_is_ok: true
+
+    # Bypass Borg error about a previously unknown unencrypted
+    # repository. Defaults to false.
+    # unknown_unencrypted_repo_access_is_ok: true
+
+    # Additional options to pass directly to particular Borg
+    # commands, handy for Borg options that borgmatic does not yet
+    # support natively. Note that borgmatic does not perform any
+    # validation on these options. Running borgmatic with
+    # "--verbosity 2" shows the exact Borg command-line
+    # invocation.
+    # extra_borg_options:
+        # Extra command-line options to pass to "borg init".
+        # init: --make-parent-dirs
+
+        # Extra command-line options to pass to "borg prune".
+        # prune: --save-space
+
+        # Extra command-line options to pass to "borg create".
+        # create: --no-files-cache
+
+        # Extra command-line options to pass to "borg check".
+        # check: --save-space
+
+# Retention policy for how many backups to keep in each category. See
+# https://borgbackup.readthedocs.io/en/stable/usage/prune.html for
+# details. At least one of the "keep" options is required for pruning
+# to work. To skip pruning entirely, run "borgmatic create" or "check"
+# without the "prune" action. See borgmatic documentation for details.
+retention:
+    # Keep all archives within this time interval.
+    # keep_within: 3H
+
+    # Number of secondly archives to keep.
+    # keep_secondly: 60
+
+    # Number of minutely archives to keep.
+    # keep_minutely: 60
+
+    # Number of hourly archives to keep.
+    # keep_hourly: 24
+
+    # Number of daily archives to keep.
+    keep_daily: 7
+
+    # Number of weekly archives to keep.
+    keep_weekly: 3
+
+    # Number of monthly archives to keep.
+    keep_monthly: 2
+
+    # Number of yearly archives to keep.
+    # keep_yearly: 1
+
+    # When pruning, only consider archive names starting with this
+    # prefix.  Borg placeholders can be used. See the output of
+    # "borg help placeholders" for details. Defaults to
+    # "{hostname}-". Use an empty value to disable the default.
+    # prefix: sourcehostname
+
+# Consistency checks to run after backups. See
+# https://borgbackup.readthedocs.io/en/stable/usage/check.html and
+# https://borgbackup.readthedocs.io/en/stable/usage/extract.html for
+# details.
+# consistency:
+    # List of one or more consistency checks to run: "repository",
+    # "archives", "data", and/or "extract". Defaults to
+    # "repository" and "archives". Set to "disabled" to disable
+    # all consistency checks. "repository" checks the consistency
+    # of the repository, "archives" checks all of the archives,
+    # "data" verifies the integrity of the data within the
+    # archives, and "extract" does an extraction dry-run of the
+    # most recent archive. Note that "data" implies "archives".
+    # checks:
+        # - repository
+        # - archives
+
+    # Paths to a subset of the repositories in the location
+    # section on which to run consistency checks. Handy in case
+    # some of your repositories are very large, and so running
+    # consistency checks on them would take too long. Defaults to
+    # running consistency checks on all repositories configured in
+    # the location section.
+    # check_repositories:
+        # - user@backupserver:sourcehostname.borg
+
+    # Restrict the number of checked archives to the last n.
+    # Applies only to the "archives" check. Defaults to checking
+    # all archives.
+    # check_last: 3
+
+    # When performing the "archives" check, only consider archive
+    # names starting with this prefix. Borg placeholders can be
+    # used. See the output of "borg help placeholders" for
+    # details. Defaults to "{hostname}-". Use an empty value to
+    # disable the default.
+    # prefix: sourcehostname
+
+# Options for customizing borgmatic's own output and logging.
+# output:
+    # Apply color to console output. Can be overridden with
+    # --no-color command-line flag. Defaults to true.
+    # color: false
+
+# Shell commands, scripts, or integrations to execute at various
+# points during a borgmatic run. IMPORTANT: All provided commands and
+# scripts are executed with user permissions of borgmatic. Do not
+# forget to set secure permissions on this configuration file (chmod
+# 0600) as well as on any script called from a hook (chmod 0700) to
+# prevent potential shell injection or privilege escalation.
+# hooks:
+    # List of one or more shell commands or scripts to execute
+    # before creating a backup, run once per configuration file.
+    # before_backup:
+        # - echo "Starting a backup."
+
+    # List of one or more shell commands or scripts to execute
+    # before pruning, run once per configuration file.
+    # before_prune:
+        # - echo "Starting pruning."
+
+    # List of one or more shell commands or scripts to execute
+    # before consistency checks, run once per configuration file.
+    # before_check:
+        # - echo "Starting checks."
+
+    # List of one or more shell commands or scripts to execute
+    # before extracting a backup, run once per configuration file.
+    # before_extract:
+        # - echo "Starting extracting."
+
+    # List of one or more shell commands or scripts to execute
+    # after creating a backup, run once per configuration file.
+    # after_backup:
+        # - echo "Finished a backup."
+
+    # List of one or more shell commands or scripts to execute
+    # after pruning, run once per configuration file.
+    # after_prune:
+        # - echo "Finished pruning."
+
+    # List of one or more shell commands or scripts to execute
+    # after consistency checks, run once per configuration file.
+    # after_check:
+        # - echo "Finished checks."
+
+    # List of one or more shell commands or scripts to execute
+    # after extracting a backup, run once per configuration file.
+    # after_extract:
+        # - echo "Finished extracting."
+
+    # List of one or more shell commands or scripts to execute
+    # when an exception occurs during a "prune", "create", or
+    # "check" action or an associated before/after hook.
+    # on_error:
+        # - echo "Error during prune/create/check."
+
+    # List of one or more shell commands or scripts to execute
+    # before running all actions (if one of them is "create").
+    # These are collected from all configuration files and then
+    # run once before all of them (prior to all actions).
+    # before_everything:
+        # - echo "Starting actions."
+
+    # List of one or more shell commands or scripts to execute
+    # after running all actions (if one of them is "create").
+    # These are collected from all configuration files and then
+    # run once after all of them (after any action).
+    # after_everything:
+        # - echo "Completed actions."
+
+    # List of one or more PostgreSQL databases to dump before
+    # creating a backup, run once per configuration file. The
+    # database dumps are added to your source directories at
+    # runtime, backed up, and removed afterwards. Requires
+    # pg_dump/pg_dumpall/pg_restore commands. See
+    # https://www.postgresql.org/docs/current/app-pgdump.html and
+    # https://www.postgresql.org/docs/current/libpq-ssl.html for
+    # details.
+    # postgresql_databases:
+        # Database name (required if using this hook). Or
+        # "all" to dump all databases on the host. Note
+        # that using this database hook implicitly enables
+        # both read_special and one_file_system (see
+        # above) to support dump and restore streaming.
+        # - name: users
+
+          # Database hostname to connect to. Defaults to
+          # connecting via local Unix socket.
+          # hostname: database.example.org
+
+          # Port to connect to. Defaults to 5432.
+          # port: 5433
+
+          # Username with which to connect to the database.
+          # Defaults to the username of the current user.
+          # You probably want to specify the "postgres"
+          # superuser here when the database name is "all".
+          # username: dbuser
+
+          # Password with which to connect to the database.
+          # Omitting a password will only work if PostgreSQL
+          # is configured to trust the configured username
+          # without a password, or you create a ~/.pgpass
+          # file.
+          # password: trustsome1
+
+          # Database dump output format. One of "plain",
+          # "custom", "directory", or "tar". Defaults to
+          # "custom" (unlike raw pg_dump). See pg_dump
+          # documentation for details. Note that format is
+          # ignored when the database name is "all".
+          # format: directory
+
+          # SSL mode to use to connect to the database
+          # server. One of "disable", "allow", "prefer",
+          # "require", "verify-ca" or "verify-full".
+          # Defaults to "disable".
+          # ssl_mode: require
+
+          # Path to a client certificate.
+          # ssl_cert: /root/.postgresql/postgresql.crt
+
+          # Path to a private client key.
+          # ssl_key: /root/.postgresql/postgresql.key
+
+          # Path to a root certificate containing a list of
+          # trusted certificate authorities.
+          # ssl_root_cert: /root/.postgresql/root.crt
+
+          # Path to a certificate revocation list.
+          # ssl_crl: /root/.postgresql/root.crl
+
+          # Additional pg_dump/pg_dumpall options to pass
+          # directly to the dump command, without performing
+          # any validation on them. See pg_dump
+          # documentation for details.
+          # options: --role=someone
+
+    # List of one or more MySQL/MariaDB databases to dump before
+    # creating a backup, run once per configuration file. The
+    # database dumps are added to your source directories at
+    # runtime, backed up, and removed afterwards. Requires
+    # mysqldump/mysql commands (from either MySQL or MariaDB). See
+    # https://dev.mysql.com/doc/refman/8.0/en/mysqldump.html or
+    # https://mariadb.com/kb/en/library/mysqldump/ for details.
+    # mysql_databases:
+        # Database name (required if using this hook). Or
+        # "all" to dump all databases on the host. Note
+        # that using this database hook implicitly enables
+        # both read_special and one_file_system (see
+        # above) to support dump and restore streaming.
+        # - name: users
+
+          # Database hostname to connect to. Defaults to
+          # connecting via local Unix socket.
+          # hostname: database.example.org
+
+          # Port to connect to. Defaults to 3306.
+          # port: 3307
+
+          # Username with which to connect to the database.
+          # Defaults to the username of the current user.
+          # username: dbuser
+
+          # Password with which to connect to the database.
+          # Omitting a password will only work if MySQL is
+          # configured to trust the configured username
+          # without a password.
+          # password: trustsome1
+
+          # Additional mysqldump options to pass directly to
+          # the dump command, without performing any
+          # validation on them. See mysqldump documentation
+          # for details.
+          # options: --skip-comments
+
+    # Healthchecks ping URL or UUID to notify when a backup
+    # begins, ends, or errors. Create an account at
+    # https://healthchecks.io if you'd like to use this service.
+    # See borgmatic monitoring documentation for details.