Commit f4aa3b4e authored by remy's avatar remy
Browse files

borgbackup recipe public first commit

parent f55a008e
# borgbackup
# BorgBackup SaltStack formula
> Note: this formula is provided "as is" without any warranty of any kind. It has been tested successfully on my debian-like minions.
## Requirements
First, you need a working [`borgbackup`](https://www.borgbackup.org/) server (in fact, it just needs ssh and a huge amount of free space).
Then, you need to setup a SaltStack grain's list named `pool` to determine current network subnets. Take a look at [this formula](https://gitlab.mbb.univ-montp2.fr/saltstack-formulas/set_grains) if needed.
For example, here I am using `isem212` for our network `192.168.212.0/24`.
Then, check the `pillar.example` file to manage your borg server(s).
## Usage
The `borbackup.server` formula will then add the borg user on the server.
```bash
# $minion_borg_server is your borg server's SaltStack minion id
salt '$minion_borg_server' -v state.sls borgbackup.server
```
In order to add the client rsa key, you may need to create the key.
In that case, you will just have to apply that formula:
```bash
salt '$minion' -v state.sls borgbackup.generate_rsa_key
```
Then, refresh mines and apply the add_clients recipe, to sync ssh keys:
```bash
salt '*' mine.update
salt '$minion_borg_server' -v state.sls borgbackup.add_clients
```
To configure backup directories, please take a look at `hosts212.pillar.example`.
`borgbackup.client` will then look at `RsyncShareName` values and `BackupFilesExclude` (I kept the BackupPC naming format for formula compatibility).
The borgclient recipe installs `borgbackup` and [`borgmatic`](https://torsion.org/borgmatic/) using pip3.
`borgmatic` allows us to use an easier way to manage the borg backups.
```bash
salt '$minion' -v state.sls borgbackup.client
```
References:
- https://torsion.org/borgmatic/docs/how-to/set-up-backups/
- https://doc.ubuntu-fr.org/borgbackup
- https://sebsauvage.net/wiki/doku.php?id=borgbackup
- https://www.youtube.com/watch?v=QIMJdnghEmU
- https://borgbackup.readthedocs.io/en/stable/deployment/central-backup-server.html
- https://wiki.fiat-tux.fr/books/administration-syst%C3%A8mes/page/borgmatic
- https://torsion.org/borgmatic/docs/how-to/set-up-backups/
### Comparing to BackupPC
With BorgBackup, there is no real Borg server, contrary to BackupPC. Indeed, by default, there is no AdminUI/WebUI.
Moreover, connections are done by SSH from the clients; they connect to the server to backup their data.
With BackupPC, it is the opposite : connections are initialized by the server to retrieve data on the clients.
However, with deduplication and encryption, BorgBackup is really more powerful.
This recipe is helpful to add ssh authorized_keys on the borg server. SSH RSA Keys are taken on each minion using an updated mine.
If the key does not exist on the server side, it is added by this formula.
This formula also create an empty directory for the client host, to allow it to create a borg backup repository on the borg server.
Usage:
```bash
salt '$minion_borg_server' state.sls borgbackup.add_clients
```
{% set allkeys = salt['mine.get']('*', 'get_ssh_rsa_key') %}
{% set BORGDATA = salt['pillar.get']('borg:servers') %}
{% for server, data in BORGDATA.items() %}
{% if server == grains['fqdn_ip4'][0] %}
{% set mainrepo = data['repo'] %}
{% for host, rsakey in allkeys.items() %}
creating empty directory repo for {{ host }}:
cmd.run:
- name: "mkdir -p {{ mainrepo }}/{{ host }}"
- shell: /bin/bash
- runas: {{ data['borguser'] }}
# CAUTION!
# If you change the ssh command= option below, it won't necessarily get pushed to the backup
# server correctly unless you delete the ~/.ssh/authorized_keys file and re-create it!
adding_rsa_key_{{ host }}:
ssh_auth.present:
- user: {{ data['borguser'] }}
- enc: {{ data['host_key_enc'] }}
- names:
- {{ rsakey }}
- options:
- command="cd {{ mainrepo }}/{{ host }}; borg --umask=077 serve --append-only --restrict-to-path {{ mainrepo }}/{{ host }}"
- restrict
{% endfor %}
{% endif %}
{% endfor %}
\ No newline at end of file
include:
- borgbackup.add_clients.add_keys
\ No newline at end of file
1. Be sure that the key has been added on the borg server (see `borgbackup.add_clients`) and the remote directory exists.
If not, refresh mines `salt '*' mine.update`, then re-run `borgbackup.add_clients` on borg server (check pillar `pillar.example`),
2. If the machine does not have enough memory, borgbackup build may be killed by the OS (2GB is enough),
3. By default, this recipe set the repository in append-only mode, avoiding issue with data corruptions,
4. SSH key used to access to the repository is the root default ssh key on client (without passphrase), and the server host ssh fingerprint is `ssh-rsa`.
Usage:
```bash
salt '$minion' state.sls borgbackup.client
```
#!/bin/bash
{% for borgserver, borgdata in salt['pillar.get']('borg:servers').items() %}{% if borgdata['pool'] in grains['pool'] %}
PATH=$PATH:/usr/bin:/usr/local/bin:/root/.local/bin BORG_PASSPHRASE={{ borgdata['passphrase'] }} borg key export {{ borgdata['borguser'] }}@{{ borgserver }}:{{ borgdata['repo'] }}/{{ grains['id'] }} /etc/ssl/private/borg-{{ borgserver }}.key
PATH=$PATH:/usr/bin:/usr/local/bin:/root/.local/bin BORG_PASSPHRASE={{ borgdata['passphrase'] }} borgmatic --verbosity 1 --files 2>&1 > /var/log/borginit.log
{% endif %}{% endfor %}
This diff is collapsed.
# You can drop this file into /etc/cron.d/ to run borgmatic nightly. {% set randomhour = range(0,6) |random %}
# random hour = {{ randomhour }}
SHELL=/bin/bash
{% for borgserver, borgdata in salt['pillar.get']('borg:servers').items() %}{% if borgdata['pool'] in grains['pool'] %}
10 {{ randomhour + loop.index }} * * * root PATH=$PATH:/usr/bin:/usr/local/bin:/root/.local/bin BORG_PASSPHRASE="{{ borgdata['passphrase']|replace('%','\%') }}" borgmatic --syslog-verbosity 1
{% endif %}{% endfor -%}
{%- set fulldict = {} %}
#{#%- set hosts211 = salt['pillar.get']('machines211', {}) %#}
{%- set hosts212 = salt['pillar.get']('machines212', {}) %}
#{#%- do fulldict.update(hosts211) %#}
{%- do fulldict.update(hosts212) %}
{%- for host, hostinfo in fulldict.items() %}
{% if 'RsyncShareName' in hostinfo %}
{% if 'SaltHostname' in hostinfo %}
{% set host = grains['id'] %}
{% if hostinfo['SaltHostname'] == host %}
/etc/cron.d/borgmatic:
file.managed:
- user: root
- group: root
- mode: '0600'
- source: salt://borgbackup/client/crons/borgmatic/borgmatic.cron
- template: jinja
- onlyif:
- command -v borg
/var/log/borg:
file.directory
/etc/cron.d/jsoncheck:
file.managed:
- user: root
- group: root
- mode: '0600'
- source: salt://borgbackup/client/crons/borgmatic/json_check.cron
- template: jinja
- onlyif:
- command -v borg
{% endif %}
{% endif %}
{% endif %}
{% endfor %}
\ No newline at end of file
include:
- .files
- .services
\ No newline at end of file
# You can drop this file into /etc/cron.d/ to run borg list every day at 5 am + a random number of minutes. {% set randommin = range(20,50) |random %}
# random minute = {{ randommin }}
SHELL=/bin/bash
{% for borgserver, borgdata in salt['pillar.get']('borg:servers').items() %}{% if borgdata['pool'] in grains['pool'] %}
{{ randommin + loop.index }} 5 * * * root PATH=$PATH:/usr/bin:/usr/local/bin:/root/.local/bin BORG_PASSPHRASE="{{ borgdata['passphrase']|replace('%','\%') }}" borg list --json {{ borgdata['borguser'] }}@{{ borgserver }}:{{ borgdata['repo'] }}/{{ grains['id'] }} > /var/log/borg/`date +"%Y%m%d"`.json
{% endif %}{% endfor -%}
{% if grains["os_family"] == "RedHat" %}
crond:
service.running:
- enable: True
- reload: True
{% else %}
cron:
service.running:
- enable: True
- reload: True
{% endif %}
{% for borgserver, borgdata in salt['pillar.get']('borg:servers').items() %}
{% set PASSPHRASE = borgdata['passphrase'] %}
{% set current_path = salt['environ.get']('PATH', '/bin:/usr/bin') %}
{% if borgdata['pool'] in grains['pool'] %}
initialize host repo {{ loop.index }}:
cmd.run:
- name: "borgmatic init --append-only --encryption repokey"
- shell: /bin/bash
- ignore_timeout: true
- env:
- PATH: {{ [current_path, '/root/.local/bin']|join(':') }}
- BORG_PASSPHRASE: {{ PASSPHRASE }}
first archive {{ loop.index }}:
cmd.run:
- name: at 22:00 -f /usr/local/sbin/backupinit.sh
- shell: /bin/bash
{% endif %}
{% endfor %}
\ No newline at end of file
{% set fulldict = {} %}
#{#% set hosts211 = salt['pillar.get']('machines211', {}) %#}
{% set hosts212 = salt['pillar.get']('machines212', {}) %}
#{#% do fulldict.update(hosts211) %#}
{% do fulldict.update(hosts212) %}
{% for host, hostinfo in fulldict.items() %}
{% if 'RsyncShareName' in hostinfo %}
{% if 'SaltHostname' in hostinfo %}
{% set host = grains['id'] %}
{% if hostinfo['SaltHostname'] == host %}
/etc/borgmatic:
file.directory:
- user: root
- group: root
- dir_mode: '0700'
/etc/borgmatic/config.yaml:
file.managed:
- source: salt://borgbackup/client/borgmatic.config.yaml
- template: jinja
- context:
HOSTINFO: {{ hostinfo }}
/usr/local/sbin/backupinit.sh:
file.managed:
- source: salt://borgbackup/client/backupinit.sh.jinja
- user: root
- group: root
- mode: '0700'
- template: jinja
{% for borgserver, borgdata in salt['pillar.get']('borg:servers').items() %}
{% if borgdata['pool'] in grains['pool'] %}
adding Host fingerprint in known_hosts {{ loop.index }}:
ssh_known_hosts.present:
- name: {{ borgserver }}
- user: root
- key: {{ borgdata['host_key'] }}
- enc: {{ borgdata['host_key_enc'] }}
{% endif %}
{% endfor %}
{% endif %}
{% endif %}
{% endif %}
{% endfor %}
\ No newline at end of file
{% set fulldict = {} %}
#{#% set hosts211 = salt['pillar.get']('machines211', {}) %#}
{% set hosts212 = salt['pillar.get']('machines212', {}) %}
#{#% do fulldict.update(hosts211) %#}
{% do fulldict.update(hosts212) %}
{% for host, hostinfo in fulldict.items() %}
{% if 'RsyncShareName' in hostinfo %}
{% if 'SaltHostname' in hostinfo %}
{% set host = grains['id'] %}
{% if hostinfo['SaltHostname'] == host %}
include:
- .client.packages
- .client.files
- .client.exec
- .client.crons
{% endif %}
{% endif %}
{% endif %}
{% endfor %}
\ No newline at end of file
install apt packages:
pkg.installed:
- pkgs:
- python3
- python3-dev
- libssl-dev
- openssl
- build-essential
- libfuse-dev
- fuse
- pkg-config
- libacl1-dev
- libacl1
- libzstd1
- python3-pip
- at
install pip cython:
pip.installed:
- name: cython
- require:
- pkg: install apt packages
install pip borg:
pip.installed:
- name: borgbackup
- bin_env: /usr/bin/pip3
- require:
- pip: install pip cython
install pip borgmatic:
pip.installed:
- name: borgmatic
- bin_env: /usr/bin/pip3
- require:
- pip: install pip borg
\ No newline at end of file
generate missing key:
cmd.run:
- name: ssh-keygen -a 100 -t rsa -f /root/.ssh/id_rsa -N '' -q
- shell: /bin/bash
- unless: ls /root/.ssh/id_rsa.pub
include:
- common_tools.scripts.generate_rsa_key.packages
- common_tools.scripts.generate_rsa_key.exec
\ No newline at end of file
needed packages for ssh client:
pkg.installed:
- pkgs:
{% if grains['os_family'] == 'Debian' %}
- openssh-client
- libssl1.1
{% else %}
- openssh
- openssl-libs
{% endif %}
\ No newline at end of file
machines212:
ns1:
ip: 192.168.212.87
user: isi
ldap_client: false
machine_type: container
url_mgmt: https://192.168.212.208:8006
SaltHostname: ns1.domain.tld
RsyncShareName:
- /root
- /etc
- /var/spool/cron
- /var/lib/knot
- /var/backups
BackupFilesExclude:
- /etc/ssh
- /root/.ssh
salt:
ip: 192.168.212.150
user: isi
ldap_client: false
machine_type: container
url_mgmt: https://192.168.212.203:8006
SaltHostname: salt.domain.tld
RsyncShareName:
- /root
- /etc
- /var/spool/cron
- /var/backups
- /var/www
BackupFilesExclude:
- /etc/ssh
- /root/.ssh
nas-isi:
ip: 192.168.212.85
user: isi
ldap_client: false
machine_type: host
url_mgmt: https://192.168.212.228
SaltHostname: nas-isi.domain.tld
RsyncShareName:
- /root
- /etc
- /var/spool/cron
BackupFilesExclude:
- /etc/ssh
- /root/.ssh
Supports Markdown
0% or .
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment