borgbackup recipe public first commit

README.md

-# borgbackup
+# BorgBackup SaltStack formula
 
+> Note: this formula is provided "as is" without any warranty of any kind. It has been tested successfully on my debian-like minions.
+
+## Requirements
+
+First, you need a working [`borgbackup`](https://www.borgbackup.org/) server (in fact, it just needs ssh and a huge amount of free space).
+
+Then, you need to setup a SaltStack grain's list named `pool` to determine current network subnets. Take a look at [this formula](https://gitlab.mbb.univ-montp2.fr/saltstack-formulas/set_grains) if needed.
+For example, here I am using `isem212` for our network `192.168.212.0/24`.
+
+Then, check the `pillar.example` file to manage your borg server(s).
+
+## Usage
+
+The `borbackup.server` formula will then add the borg user on the server.
+
+```bash
+# $minion_borg_server is your borg server's SaltStack minion id
+salt '$minion_borg_server' -v state.sls borgbackup.server
+```
+
+In order to add the client rsa key, you may need to create the key.
+In that case, you will just have to apply that formula:
+
+```bash
+salt '$minion' -v state.sls borgbackup.generate_rsa_key
+```
+
+Then, refresh mines and apply the add_clients recipe, to sync ssh keys:
+
+```bash
+salt '*' mine.update
+salt '$minion_borg_server' -v state.sls borgbackup.add_clients
+```
+
+To configure backup directories, please take a look at `hosts212.pillar.example`.
+
+`borgbackup.client` will then look at `RsyncShareName` values and `BackupFilesExclude` (I kept the BackupPC naming format for formula compatibility).
+
+The borgclient recipe installs `borgbackup` and [`borgmatic`](https://torsion.org/borgmatic/) using pip3.
+
+`borgmatic` allows us to use an easier way to manage the borg backups.
+
+```bash
+salt '$minion' -v state.sls borgbackup.client
+```
+
+References:
+
+  - https://torsion.org/borgmatic/docs/how-to/set-up-backups/
+  - https://doc.ubuntu-fr.org/borgbackup
+  - https://sebsauvage.net/wiki/doku.php?id=borgbackup
+  - https://www.youtube.com/watch?v=QIMJdnghEmU
+  - https://borgbackup.readthedocs.io/en/stable/deployment/central-backup-server.html
+  - https://wiki.fiat-tux.fr/books/administration-syst%C3%A8mes/page/borgmatic
+  - https://torsion.org/borgmatic/docs/how-to/set-up-backups/
+
+### Comparing to BackupPC
+
+With BorgBackup, there is no real Borg server, contrary to BackupPC. Indeed, by default, there is no AdminUI/WebUI.
+Moreover, connections are done by SSH from the clients; they connect to the server to backup their data.
+With BackupPC, it is the opposite : connections are initialized by the server to retrieve data on the clients.
+
+However, with deduplication and encryption, BorgBackup is really more powerful.

add_clients/README.md

+This recipe is helpful to add ssh authorized_keys on the borg server. SSH RSA Keys are taken on each minion using an updated mine.
+If the key does not exist on the server side, it is added by this formula.
+
+This formula also create an empty directory for the client host, to allow it to create a borg backup repository on the borg server.
+
+Usage:
+
+```bash
+salt '$minion_borg_server' state.sls borgbackup.add_clients
+```

add_clients/add_keys.sls

+{% set allkeys = salt['mine.get']('*', 'get_ssh_rsa_key') %}
+{% set BORGDATA = salt['pillar.get']('borg:servers') %}
+{% for server, data in BORGDATA.items() %}
+ {% if server == grains['fqdn_ip4'][0] %}
+  {% set mainrepo = data['repo'] %}
+  {% for host, rsakey in allkeys.items() %}
+
+creating empty directory repo for {{ host }}:
+  cmd.run:
+    - name: "mkdir -p {{ mainrepo }}/{{ host }}"
+    - shell: /bin/bash
+    - runas: {{ data['borguser'] }}
+
+# CAUTION!
+# If you change the ssh command= option below, it won't necessarily get pushed to the backup
+# server correctly unless you delete the ~/.ssh/authorized_keys file and re-create it!
+adding_rsa_key_{{ host }}:
+  ssh_auth.present:
+    - user: {{ data['borguser'] }}
+    - enc: {{ data['host_key_enc'] }}
+    - names:
+      - {{ rsakey }}
+    - options:
+      - command="cd {{ mainrepo }}/{{ host }}; borg --umask=077 serve --append-only --restrict-to-path {{ mainrepo }}/{{ host }}"
+      - restrict
+
+  {% endfor %}
+ {% endif %}
+{% endfor %}
\ No newline at end of file

add_clients/init.sls

+include:
+  - borgbackup.add_clients.add_keys
\ No newline at end of file

client/README.md

+1. Be sure that the key has been added on the borg server (see `borgbackup.add_clients`) and the remote directory exists.
+  If not, refresh mines `salt '*' mine.update`, then re-run `borgbackup.add_clients` on borg server (check pillar `pillar.example`),
+2. If the machine does not have enough memory, borgbackup build may be killed by the OS (2GB is enough),
+3. By default, this recipe set the repository in append-only mode, avoiding issue with data corruptions,
+4. SSH key used to access to the repository is the root default ssh key on client (without passphrase), and the server host ssh fingerprint is `ssh-rsa`.
+
+Usage:
+
+```bash
+salt '$minion' state.sls borgbackup.client
+```

client/backupinit.sh.jinja

+#!/bin/bash
+
+{% for borgserver, borgdata in salt['pillar.get']('borg:servers').items() %}{% if borgdata['pool'] in grains['pool'] %}
+PATH=$PATH:/usr/bin:/usr/local/bin:/root/.local/bin BORG_PASSPHRASE={{ borgdata['passphrase'] }} borg key export {{ borgdata['borguser'] }}@{{ borgserver }}:{{ borgdata['repo'] }}/{{ grains['id'] }} /etc/ssl/private/borg-{{ borgserver }}.key
+PATH=$PATH:/usr/bin:/usr/local/bin:/root/.local/bin BORG_PASSPHRASE={{ borgdata['passphrase'] }} borgmatic --verbosity 1 --files 2>&1 > /var/log/borginit.log
+{% endif %}{% endfor %}

client/crons/borgmatic.cron

+# You can drop this file into /etc/cron.d/ to run borgmatic nightly. {% set randomhour = range(0,6) |random %}
+# random hour = {{ randomhour }}
+SHELL=/bin/bash
+{% for borgserver, borgdata in salt['pillar.get']('borg:servers').items() %}{% if borgdata['pool'] in grains['pool'] %}
+10 {{ randomhour + loop.index }} * * * root PATH=$PATH:/usr/bin:/usr/local/bin:/root/.local/bin BORG_PASSPHRASE="{{ borgdata['passphrase']|replace('%','\%') }}" borgmatic --syslog-verbosity 1
+{% endif %}{% endfor -%}

client/crons/files.sls

+
+{%- set fulldict = {} %}
+#{#%- set hosts211 = salt['pillar.get']('machines211', {}) %#}
+{%- set hosts212 = salt['pillar.get']('machines212', {}) %}
+#{#%- do fulldict.update(hosts211) %#}
+{%- do fulldict.update(hosts212) %}
+
+{%- for host, hostinfo in fulldict.items() %}
+ {% if 'RsyncShareName' in hostinfo %}
+  {% if 'SaltHostname' in hostinfo %}
+   {% set host = grains['id'] %}
+   {% if hostinfo['SaltHostname'] == host %}
+
+/etc/cron.d/borgmatic:
+  file.managed:
+    - user: root
+    - group: root
+    - mode: '0600'
+    - source: salt://borgbackup/client/crons/borgmatic/borgmatic.cron
+    - template: jinja
+    - onlyif:
+        - command -v borg
+
+/var/log/borg:
+  file.directory
+
+/etc/cron.d/jsoncheck:
+  file.managed:
+    - user: root
+    - group: root
+    - mode: '0600'
+    - source: salt://borgbackup/client/crons/borgmatic/json_check.cron
+    - template: jinja
+    - onlyif:
+        - command -v borg
+
+   {% endif %}
+  {% endif %}
+ {% endif %}
+{% endfor %}
\ No newline at end of file

client/crons/init.sls

+include:
+  - .files
+  - .services
\ No newline at end of file

client/crons/json_check.cron

+# You can drop this file into /etc/cron.d/ to run borg list every day at 5 am + a random number of minutes. {% set randommin = range(20,50) |random %}
+# random minute = {{ randommin }}
+SHELL=/bin/bash
+{% for borgserver, borgdata in salt['pillar.get']('borg:servers').items() %}{% if borgdata['pool'] in grains['pool'] %}
+{{ randommin + loop.index }} 5 * * * root PATH=$PATH:/usr/bin:/usr/local/bin:/root/.local/bin BORG_PASSPHRASE="{{ borgdata['passphrase']|replace('%','\%') }}" borg list --json {{ borgdata['borguser'] }}@{{ borgserver }}:{{ borgdata['repo'] }}/{{ grains['id'] }} > /var/log/borg/`date +"%Y%m%d"`.json
+{% endif %}{% endfor -%}

client/crons/services.sls

+{% if grains["os_family"] == "RedHat" %}
+crond:
+  service.running:
+    - enable: True
+    - reload: True
+{% else %}
+cron:
+  service.running:
+    - enable: True
+    - reload: True
+{% endif %}
+

client/exec.sls

+{% for borgserver, borgdata in salt['pillar.get']('borg:servers').items() %}
+ {% set PASSPHRASE = borgdata['passphrase'] %}
+ {% set current_path = salt['environ.get']('PATH', '/bin:/usr/bin') %}
+
+ {% if borgdata['pool'] in grains['pool'] %}
+
+initialize host repo {{ loop.index }}:
+  cmd.run:
+    - name: "borgmatic init --append-only --encryption repokey"
+    - shell: /bin/bash
+    - ignore_timeout: true
+    - env:
+        - PATH: {{ [current_path, '/root/.local/bin']|join(':') }}
+        - BORG_PASSPHRASE: {{ PASSPHRASE }}
+
+first archive {{ loop.index }}:
+  cmd.run:
+    - name: at 22:00 -f /usr/local/sbin/backupinit.sh
+    - shell: /bin/bash
+
+ {% endif %}
+{% endfor %}
\ No newline at end of file

client/files.sls

+{% set fulldict = {} %}
+#{#% set hosts211 = salt['pillar.get']('machines211', {}) %#}
+{% set hosts212 = salt['pillar.get']('machines212', {}) %}
+#{#% do fulldict.update(hosts211) %#}
+{% do fulldict.update(hosts212) %}
+
+{% for host, hostinfo in fulldict.items() %}
+ {% if 'RsyncShareName' in hostinfo %}
+  {% if 'SaltHostname' in hostinfo %}
+   {% set host = grains['id'] %}
+   {% if hostinfo['SaltHostname'] == host %}
+
+/etc/borgmatic:
+  file.directory:
+    - user: root
+    - group: root
+    - dir_mode: '0700'
+/etc/borgmatic/config.yaml:
+  file.managed:
+    - source: salt://borgbackup/client/borgmatic.config.yaml
+    - template: jinja
+    - context:
+        HOSTINFO: {{ hostinfo }}
+/usr/local/sbin/backupinit.sh:
+  file.managed:
+    - source: salt://borgbackup/client/backupinit.sh.jinja
+    - user: root
+    - group: root
+    - mode: '0700'
+    - template: jinja
+    {% for borgserver, borgdata in salt['pillar.get']('borg:servers').items() %}
+     {% if borgdata['pool'] in grains['pool'] %}
+adding Host fingerprint in known_hosts {{ loop.index }}:
+  ssh_known_hosts.present:
+    - name: {{ borgserver }}
+    - user: root
+    - key: {{ borgdata['host_key'] }}
+    - enc: {{ borgdata['host_key_enc'] }}
+     {% endif %}
+    {% endfor %}
+  
+   {% endif %}
+  {% endif %}
+ {% endif %}
+{% endfor %}
\ No newline at end of file

client/init.sls

+{% set fulldict = {} %}
+#{#% set hosts211 = salt['pillar.get']('machines211', {}) %#}
+{% set hosts212 = salt['pillar.get']('machines212', {}) %}
+#{#% do fulldict.update(hosts211) %#}
+{% do fulldict.update(hosts212) %}
+
+{% for host, hostinfo in fulldict.items() %}
+ {% if 'RsyncShareName' in hostinfo %}
+  {% if 'SaltHostname' in hostinfo %}
+   {% set host = grains['id'] %}
+   {% if hostinfo['SaltHostname'] == host %}
+
+include:
+  - .client.packages
+  - .client.files
+  - .client.exec
+  - .client.crons
+
+   {% endif %}
+  {% endif %}
+ {% endif %}
+{% endfor %}
+  
\ No newline at end of file

client/packages.sls

+install apt packages:
+  pkg.installed:
+    - pkgs:
+      - python3
+      - python3-dev
+      - libssl-dev
+      - openssl
+      - build-essential
+      - libfuse-dev
+      - fuse
+      - pkg-config
+      - libacl1-dev
+      - libacl1
+      - libzstd1
+      - python3-pip
+      - at
+
+install pip cython:
+  pip.installed:
+    - name: cython
+    - require:
+      - pkg: install apt packages
+install pip borg:
+  pip.installed:
+    - name: borgbackup
+    - bin_env: /usr/bin/pip3
+    - require:
+      - pip: install pip cython
+install pip borgmatic:
+  pip.installed:
+    - name: borgmatic
+    - bin_env: /usr/bin/pip3
+    - require:
+      - pip: install pip borg
\ No newline at end of file

generate_rsa_key/exec.sls

+generate missing key:
+  cmd.run:
+    - name: ssh-keygen -a 100 -t rsa -f /root/.ssh/id_rsa -N '' -q
+    - shell: /bin/bash
+    - unless: ls /root/.ssh/id_rsa.pub

generate_rsa_key/init.sls

+include:
+  - common_tools.scripts.generate_rsa_key.packages
+  - common_tools.scripts.generate_rsa_key.exec
\ No newline at end of file

generate_rsa_key/packages.sls

+
+needed packages for ssh client:
+  pkg.installed:
+    - pkgs:
+{% if grains['os_family']  == 'Debian' %}
+      - openssh-client
+      - libssl1.1
+{% else %}
+      - openssh
+      - openssl-libs
+{% endif %}
\ No newline at end of file

hosts212.pillar.example

+machines212:
+  ns1:
+    ip: 192.168.212.87
+    user: isi
+    ldap_client: false
+    machine_type: container
+    url_mgmt: https://192.168.212.208:8006
+    SaltHostname: ns1.domain.tld
+    RsyncShareName:
+      - /root
+      - /etc
+      - /var/spool/cron
+      - /var/lib/knot
+      - /var/backups
+    BackupFilesExclude:
+      - /etc/ssh
+      - /root/.ssh
+  salt:
+    ip: 192.168.212.150
+    user: isi
+    ldap_client: false
+    machine_type: container
+    url_mgmt: https://192.168.212.203:8006
+    SaltHostname: salt.domain.tld
+    RsyncShareName:
+      - /root
+      - /etc
+      - /var/spool/cron
+      - /var/backups
+      - /var/www
+    BackupFilesExclude:
+      - /etc/ssh
+      - /root/.ssh
+  nas-isi:
+    ip: 192.168.212.85
+    user: isi
+    ldap_client: false
+    machine_type: host
+    url_mgmt: https://192.168.212.228
+    SaltHostname: nas-isi.domain.tld
+    RsyncShareName:
+      - /root
+      - /etc
+      - /var/spool/cron
+    BackupFilesExclude:
+      - /etc/ssh
+      - /root/.ssh