Bareos
Contents
About
Backup Archiving Recovery Open Sourced
Backed by a German corporation Bareos GmbH & Co. KG that offers support. Bareos Pricing
Source Code:
1 git clone https://github.com/bareos/bareos.git
What is Bareos?
Bareos (Backup Archiving Recovery Open Sourced) is a reliable, cross-network open source software for backup, archiving and recovery of data for all well-established operating systems. Emerged from the Bacula Project in 2010, Bareos was and is actively developed as a fork and enriched with lots of new features.
Thus Bareos today offers among other features LTO hardware encryption, bandwidth limitation and new practical console commands. The source code of Bareos is available on https://github.com/bareos/ under the AGPL v3 license. Additionally Bareos offers binary package repositories for the most important Linux distributions as well as for Windows.
Source: bareos.com - What is Bareos
Preface
Make sure you have some disk-resources when experimenting. In my little lab I gathered 1TiB in just 5 days of active usage.
Installation
Debian packages are quite outdated even in Sid, just use the bareos repo.
/etc/apt/sources.list.d/bareos.list
Import the public key to proof the signature.
Package "bareos" pulls in all dependencies. As a base OS i suggest Debian 10 Codename "Buster". Latest version in Debian is currently 17.2.7, but on Github 18.5 is in the works.
1 aptitude install bareos
VM Setup
I'm running bareos in a VM running on Libvirt/KVM (2Cores, 8GiB Ram, 20GiB Storage, 1Nic).
9p
Nothing special besides a filesystem residing on btrfs raid1 on the host mapped to the guest with gp9-virtio in mode pass-through. This is where backups are written to.
Example domain xml
/etc/fstab
mount -t 9p
1 bareos_storage on /var/lib/bareos/storage type 9p (rw,relatime,sync,dirsync,access=client,trans=virtio)
The driver will be exchanged with virtio-fs when Linux 5.4 becomes stable and decent support is available in the tooling.
virtio-fs
You'll need
- Linux 5.4+
- Libvirt 6.2.0+ (Debian has only 6.0)
qemu 5.0.0
Configure
Bareos Director-Daemon
Ressource Director (self)
/etc/bareos/bareos-dir.d/director/bareos-dir.conf
1 Director { # define myself
2 Name = bareos-dir
3 QueryFile = "/usr/lib/bareos/scripts/query.sql"
4 Maximum Concurrent Jobs = 10
5 Password = "very_log_director_password" # Console password
6 Messages = Daemon
7 Auditing = yes
8
9 # Enable the Heartbeat if you experience connection losses
10 # (eg. because of your router or firewall configuration).
11 # Additionally the Heartbeat can be enabled in bareos-sd and bareos-fd.
12 #
13 # Heartbeat Interval = 1 min
14
15 # remove comment in next line to load dynamic backends from specified directory
16 # Backend Directory = /usr/lib/bareos/backends
17
18 # remove comment from "Plugin Directory" to load plugins from specified directory.
19 # if "Plugin Names" is defined, only the specified plugins will be loaded,
20 # otherwise all director plugins (*-dir.so) from the "Plugin Directory".
21 #
22 # Plugin Directory = /usr/lib/bareos/plugins
23 # Plugin Names = ""
24
25 ### Collect and keep statistics
26 ### This enabled by default in the director resource,
27 ### but must be enabled manually in the storage resource.
28 Statistics Collect Interval = 150 ### DEFAULT: 150
29 Statistics Retention = 160704000 ### Defaults: 160704000 (5y)
30 }
Filesets
LinuxBase
/etc/bareos/bareos-dir.d/fileset/LinuxBase.conf
1 FileSet {
2 Name = "LinuxBase"
3 Description = "Backup a minimal set of a Linux Server."
4 Enable VSS = false
5
6 Include {
7 Options {
8 Compression = GZIP
9 Signature = SHA256
10 One FS = yes
11 noatime = yes
12 Sparse = yes
13 IgnoreCase = no
14 Wildfile = /boot/config-*
15 }
16 Options {
17 exclude = yes
18 Regexdir = "/home/[^/]+/.steam"
19 Regexdir = "/home/[^/]+/.local/share/Trash/files"
20 Regexdir = "/home/[^/]+/Downloads"
21 }
22 Exclude Dir Containing = .nobackup
23 File = /etc
24 File = /home
25 File = /opt
26 File = /root
27 File = /var/lib
28 }
29
30 Include {
31 Options {
32 Wildfile = /boot/config-*
33 }
34 Options {
35 Exclude = yes
36 RegexFile = ".*"
37 }
38 File = /boot
39 }
40
41 Exclude {
42 }
43 }
Windows10AllMedia
Gather all media files of every user.
/etc/bareos/bareos-dir.d/fileset/Windows10AllMedia.conf
1 FileSet {
2 Name = "Windows10AllMedia"
3
4 Include {
5
6 File = "C:/Users"
7
8 Options {
9 signature = SHA256
10 verify = s1
11 IgnoreCase = yes
12
13 # Include all users’ directories so we reach the inner ones. Unlike a
14 # WildDir pattern ending in *, this RegExDir only matches the top-level
15 # directories and not any inner ones.
16 RegExDir = "^C:/Users/[^/]+$"
17
18 # Ditto all users’ Media directories.
19 WildDir = "C:/Users/*/Bibliothek"
20 WildDir = "C:/Users/*/Documents"
21 WildDir = "C:/Users/*/Music"
22 WildDir = "C:/Users/*/Pictures"
23 WildDir = "C:/Users/*/Videos"
24
25 # Include the contents of the Media directories and any subdirectories.
26 Wild = "C:/Users/*/Bibliothek/*"
27 Wild = "C:/Users/*/Documents/*"
28 Wild = "C:/Users/*/Music/*"
29 Wild = "C:/Users/*/Pictures/*"
30 Wild = "C:/Users/*/Videos/*"
31 }
32
33 Options {
34 Exclude = yes
35 IgnoreCase = yes
36
37 # Exclude everything else, in particular any files at the top level and
38 # any other directories or files in the users’ directories.
39 Wild = "C:/Users/*"
40 }
41 }
42 }
Job Defaults (JobDefs)
Tox simplify configuration of a job, sets of defaults for jobs can be created, which than may be referenced during the creation of a job with jobdef="DefaultJob-Name"
/etc/bareos/bareos-dir.d/jobdefs/DefaultJob-Linux.conf
1 JobDefs {
2 Name = "DefaultJob-Linux"
3 Type = Backup
4 Level = Incremental
5 Messages = "Standard"
6 Storage = "File"
7 Pool = "Incremental"
8 FullBackupPool = "Full"
9 IncrementalBackupPool = "Incremental"
10 DifferentialBackupPool = "Differential"
11 Client = "bareos-fd"
12 FileSet = "LinuxBase"
13 Schedule = "WeeklyCycle"
14 WriteBootstrap = "/var/lib/bareos/%c.bsr"
15 runscript {
16 command = "/usr/local/lib/bareos/scripts/bareos-run_script.sh before"
17 runswhen = before
18 failonerror = No
19 }
20 runscript {
21 command = "/usr/local/lib/bareos/scripts/bareos-run_script.sh after"
22 runswhen = after
23 failonerror = No
24 }
25 }
/etc/bareos/bareos-dir.d/jobdefs/DefaultJob-Windows10.conf
1 JobDefs {
2 Name = "DefaultJob-Windows10"
3 Type = Backup
4 Level = Incremental
5 Messages = "Standard"
6 Storage = "File"
7 Pool = "Incremental"
8 FullBackupPool = "Full"
9 IncrementalBackupPool = "Incremental"
10 DifferentialBackupPool = "Differential"
11 Client = "muckie-pc-fd"
12 FileSet = "Windows10AllMedia"
13 Schedule = "WeeklyCycle"
14 WriteBootstrap = "/var/lib/bareos/%c.bsr"
15 }
Pre/Post Backup Scripts
Create a directory for custom scripts on the client
1 install -o root -g root -d /usr/local/lib/bareos/scripts
A little script to be used to run some tasks before and after the backup with some sensitivity of the distribution.
/usr/local/lib/bareos/scripts/bareos-run_script.sh
1 #!/bin/bash
2
3 APPEND_DATE=false
4
5 usage () {
6 cat <<-EOL
7 $(basename $0) [-d||--date] [--] [before||after]
8 EOL
9 }
10
11 # Note that we use "$@" to let each command-line parameter expand to a
12 # separate word. The quotes around "$@" are essential!
13 # We need TEMP as the 'eval set --' would nuke the return value of getopt.
14 TEMP=$(getopt -o 'd' --long 'date' -n "$(basename $0)" -- "$@")
15
16 if [ $? -ne 0 ]; then
17 echo 'Terminating...' >&2
18 exit 1
19 fi
20
21 # Note the quotes around "$TEMP": they are essential!
22 eval set -- "$TEMP"
23 unset TEMP
24
25 while true; do
26 case "$1" in
27 '-d'|'--date')
28 APPEND_DATE=true
29 shift
30 continue
31 ;;
32 #'-b'|'--b-long')
33 # echo "Option b, argument '$2'"
34 # shift 2
35 # continue
36 #;;
37 #'-c'|'--c-long')
38 # # c has an optional argument. As we are in quoted mode,
39 # # an empty parameter will be generated if its optional
40 # # argument is not found.
41 # case "$2" in
42 # '')
43 # echo 'Option c, no argument'
44 # ;;
45 # *)
46 # echo "Option c, argument '$2'"
47 # ;;
48 # esac
49 # shift 2
50 # continue
51 #;;
52 '--')
53 shift
54 break
55 ;;
56 *)
57 echo 'Internal error!' >&2
58 exit 1
59 ;;
60 esac
61 done
62
63 ### SANITIZE
64 if [ ${#@} -eq 0 ]; then
65 WHEN="before"
66 elif [ ${#@} -gt 1 ]; then
67 cat<<-EOL
68 Please decide between "before" and "after".
69 Default is before.
70 EOL
71 usage
72 exit 2
73 elif [ $1 == "before" ] || [ $1 == "after" ]; then
74 WHEN="$1"
75 else
76 echo "Unknown execution timepoint '$1'."
77 echo "Exiting…"
78 usage
79 exit 2
80 fi
81
82 FILE_SUFFIX="txt"
83 if $APPEND_DATE; then
84 DATE="$(date +%F)"
85 FILE_SUFFIX="_$DATE.$FILE_SUFFIX"
86 else
87 FILE_SUFFIX=".$FILE_SUFFIX"
88 fi
89
90 ISSUE="$(cat /etc/issue)"
91 if egrep -q '(Debian|Ubuntu)' <<< "$ISSUE"; then
92 OS="Debian"
93 fi
94
95 ### MAIN
96
97
98 if [ "$OS" == "Debian" ]; then
99 BACKUP_DIR="/var/backups/bareos"
100 FILE_SELECTIONS="dpkg-selections$FILE_SUFFIX"
101 FILE_LIST="dpkg-list$FILE_SUFFIX"
102 else
103 echo "Unsupported OS. Exiting…"
104 exit 1
105 fi
106
107 ### COMMAND LIST: AFTER
108 if [ "$WHEN" == "before" ]; then
109 [ -d "$BACKUP_DIR" ] || mkdir "$BACKUP_DIR"
110 dpkg --get-selections > "$BACKUP_DIR/$FILE_SELECTIONS"
111 dpkg -l > "$BACKUP_DIR/$FILE_LIST"
112
113 ### COMMAND LIST: AFTER
114 elif [ "$WHEN" == "after" ]; then
115 rm "$BACKUP_DIR/$FILE_SELECTIONS"
116 rm "$BACKUP_DIR/$FILE_LIST"
117
118 fi
119
120 echo "Script executed successfully $WHEN backup."
121 exit 0
Storage
There are some statistics that should come in handy if you want to estimate how long you backup will take or if e.g. you want to account and bill the backup storage.
/etc/bareos/bareos-dir.d/storage/File.conf
Pools
- In Maximum Volume Bytes it makes no difference if you write a capital "G" or a lowercase "g".
/etc/bareos/bareos-dir.d/pool/Differential.conf
1 Pool {
2 Name = Differential
3 Pool Type = Backup
4 Recycle = yes # Bareos can automatically recycle Volumes
5 AutoPrune = yes # Prune expired volumes
6 Volume Retention = 90 days # How long should the Differential Backups be kept? (#09)
7 Maximum Volume Bytes = 10G # Limit Volume size to something reasonable
8 Maximum Volumes = 100 # Limit number of Volumes in Pool
9 Label Format = "Differential-" # Volumes will be labeled "Differential-<volume-id>"
10 }
/etc/bareos/bareos-dir.d/pool/Full.conf
1 Pool {
2 Name = Full
3 Pool Type = Backup
4 Recycle = yes # Bareos can automatically recycle Volumes
5 AutoPrune = yes # Prune expired volumes
6 Volume Retention = 365 days # How long should the Full Backups be kept? (#06)
7 Maximum Volume Bytes = 50G # Limit Volume size to something reasonable
8 Maximum Volumes = 100 # Limit number of Volumes in Pool
9 Label Format = "Full-" # Volumes will be labeled "Full-<volume-id>"
10 }
/etc/bareos/bareos-dir.d/pool/Incremental.conf
1 Pool {
2 Name = Incremental
3 Pool Type = Backup
4 Recycle = yes # Bareos can automatically recycle Volumes
5 AutoPrune = yes # Prune expired volumes
6 Volume Retention = 30 days # How long should the Incremental Backups be kept? (#12)
7 Maximum Volume Bytes = 5g # Limit Volume size to something reasonable
8 Maximum Volumes = 200 # Limit number of Volumes in Pool
9 Label Format = "Incremental-" # Volumes will be labeled "Incremental-<volume-id>"
10 }
/etc/bareos/bareos-dir.d/pool/Scratch.conf
Trouble Shooting
Wrong Unix Permissions in Config Directories
bareos-director does not start if it can't read a config-file. So make sure not to create files that bareos can't read. This can easily happen if you copy a file with unix permission o-rwx as root, which changes owner.
It will definetely complain in the logs.
Okt 16 09:12:46 backup1 systemd[1]: Starting Bareos Director Daemon service... Okt 16 09:12:46 backup1 bareos-dir[1573]: 0bareos-dir: ERROR TERMINATION at parse_conf.c:198 Okt 16 09:12:46 backup1 bareos-dir[1573]: Config error: Cannot open config file "/etc/bareos/bareos-dir.d/*/*.conf": Keine Berechtigung Okt 16 09:12:46 backup1 bareos-dir[1574]: [55B blob data] Okt 16 09:12:46 backup1 bareos-dir[1574]: Config error: Cannot open config file "/etc/bareos/bareos-dir.d/*/*.conf": Keine Berechtigung Okt 16 09:12:46 backup1 systemd[1]: bareos-director.service: Can't open PID file /var/lib/bareos/bareos-dir.9101.pid (yet?) after start: No such file or directory Okt 16 09:12:46 backup1 systemd[1]: bareos-director.service: Failed with result 'protocol'.
To identify the file you can also use the following command:
1 find /etc/bareos -type f \! -user bareos \! -group bareos -exec ls -l {} \;
bconsole
Reload Configuration
Atfer changes to the configuration files, bareos config can be reloaded without restarting the service.
1 echo reload |bconsole
Add Client
Create a new password and hash it to md5:
On bareos server configure director:
Add Job
Add a job with bandwidth limit that does not exceed 54MBit/s WiFi (max. 6.75MB/s -> 4096KiB/s)
Label Mediums
Edit pool to hold more and eventually larger mediums
/etc/bareos/bareos-dir.d/pool/Incremental.conf
1 Pool {
2 Name = Incremental
3 Pool Type = Backup
4 Recycle = yes # Bareos can automatically recycle Volumes
5 AutoPrune = yes # Prune expired volumes
6 Volume Retention = 30 days # How long should the Incremental Backups be kept? (#12)
7 Maximum Volume Bytes = 4G # Limit Volume size to something reasonable
8 Maximum Volumes = 150 # Limit number of Volumes in Pool
9 Label Format = "Incremental-" # Volumes will be labeled "Incremental-<volume-id>"
10 }
Label
1 # bconsole
2 Connecting to Director localhost:9101
3 1000 OK: bareos-dir Version: 17.2.7 (16 Jul 2018)
4 Enter a period to cancel a command.
5 *reload
6 reloaded
7 *
8 *list volumes pool=Incremental
9 *
10 *label pool=Incremental storage=File volume=Incremental-0139
11 Connecting to Storage daemon File at backup1:9103 ...
12 Sending label command for Volume "Incremental-0139" Slot 0 ...
13 3000 OK label. VolBytes=204 Volume="Incremental-0139" Device="FileStorage" (/var/lib/bareos/storage)
14 Catalog record for Volume "Incremental-0139", Slot 0 successfully created.
15 Requesting to mount FileStorage ...
16 3001 OK mount requested. Device="FileStorage" (/var/lib/bareos/storage)
Job rescued.
Limit bandwidth
Limit is in KiB/s
Bareos Storage-Daemon
Bareos Filesystem-Daemon
Caveats
bareos-fd.service has dependencies on
nss-lookup.target network.target remote-fs.target time-sync.target
It won't start successfully if any of these targets can't be reached. So make sure that e.g. any remote-mount (like nfs, smb, glusterfs, …) in /etc/fstab works under any condition. In doubt use autofs.
Bareos-Webui
http://backup1/bareos-webui/dashboard/
1 aptitude install bareos-webui
Http2 support
Apache2 mod_http2 does not support mod_mpm_prefork. So we switch to php-fpm, mod_proxy_fcgi and mod_mpm_event.
Trouble Shooting
Uninitialized variable "extras"
mod_proxy_fcgi throws an error with php7.3+, regarding a undefined variable "extras".
[Mon Oct 14 12:54:19.526596 2019] [proxy_fcgi:error] [pid 842:tid 139696600672000] [client 192.168.182.13:40176] AH01071: Got error 'PHP message: PHP Notice: compact(): Undefined variable: extras in /usr/share/bareos-webui/vendor/zendframework/zend-view/src/Helper/HeadLink.php on line 403PHP message: PHP Notice: compact(): Undefined variable: extras in /usr/share/bareos-webui/vendor/zendframework/zend-view/src/Helper/HeadLink.php on line 403'
So i applied a little patch to /usr/share/bareos-webui/vendor/zendframework/zend-view/src/Helper/HeadLink.php which initializes extras to empty string "" if (0 < count($args) && is_array($args[0])
HeadLink.php.patch
375a376
> $extras = "";
400a402,403
> } else {
> $extras = (string) $extras;patch /usr/share/bareos-webui/vendor/zendframework/zend-view/src/Helper/HeadLink.php < HeadLink.php.patch
1 /**
2 * Create item for stylesheet link item
3 *
4 * @param array $args
5 * @return stdClass|false Returns false if stylesheet is a duplicate
6 */
7 public function createDataStylesheet(array $args)
8 {
9 $rel = 'stylesheet';
10 $type = 'text/css';
11 $media = 'screen';
12 $conditionalStylesheet = false;
13 $href = array_shift($args);
14 $extras = "";
15
16 if ($this->isDuplicateStylesheet($href)) {
17 return false;
18 }
19
20 if (0 < count($args)) {
21 $media = array_shift($args);
22 if (is_array($media)) {
23 $media = implode(',', $media);
24 } else {
25 $media = (string) $media;
26 }
27 }
28 if (0 < count($args)) {
29 $conditionalStylesheet = array_shift($args);
30 if (!empty($conditionalStylesheet) && is_string($conditionalStylesheet)) {
31 $conditionalStylesheet = (string) $conditionalStylesheet;
32 } else {
33 $conditionalStylesheet = null;
34 }
35 }
36
37 if (0 < count($args) && is_array($args[0])) {
38 $extras = array_shift($args);
39 $extras = (array) $extras;
40 } else {
41 $extras = (array) $extras;
42 }
43
44 $attributes = compact('rel', 'type', 'href', 'media', 'conditionalStylesheet', 'extras');
45
46 return $this->createData($attributes);
47 }
I'm not a programmer per definition - but works, so far.
"continue" targeting switch is equivalent to "break"
https://www.php.net/manual/de/control-structures.continue.php
Error is in:
/usr/share/bareos-webui/vendor/zendframework/zend-stdlib/src/ArrayObject.php
Create a patch file ArrayObject.php.patch
426c426 < continue; --- > continue 2;
{{{#highlight bash patch /usr/share/bareos-webui/vendor/zendframework/zend-stdlib/src/ArrayObject.php < ArrayObject.php.patch }}}
fwrite(): send of 26 bytes failed with errno=104
Notice: fwrite(): send of 26 bytes failed with errno=104 Connection reset by peer in /usr/share/bareos-webui/vendor/Bareos/library/Bareos/BSock/BareosBSock.php on line 219
Apply both solutions:
BareosBSock.php.patch
538c538,548
< $crypto_method = STREAM_CRYPTO_METHOD_TLSv1_2_CLIENT;
---
> /*
> * STREAM_CRYPTO_METHOD_TLSv1_2_CLIENT was introduced with PHP version
> * 5.6.0. We need to care that calling stream_socket_enable_crypto method
> * works with versions < 5.6.0 as well.
> */
> $crypto_method = STREAM_CRYPTO_METHOD_TLS_CLIENT;
>
> if (defined('STREAM_CRYPTO_METHOD_TLSv1_2_CLIENT')) {
> $crypto_method |= STREAM_CRYPTO_METHOD_TLSv1_2_CLIENT;
> }
> patch /usr/share/bareos-webui/vendor/Bareos/library/Bareos/BSock/BareosBSock.php < BareosBSock.php.patch
Disable TLS in
/etc/bareos/bareos-dir.d/console/admin.conf
#
# Restricted console used by bareos-webui
#
Console {
Name = admin
Password = "jekoo2OdaiQuaeng"
Profile = "webui-admin"
TLS Enable = No
}
BConsole webview to small
Bconsole width and height are limited to 1080x650 but can be adjusted.
Adjust bconsole size /usr/share/bareos-webui/module/Director/view/director/director/console.phtml
---- /!\ '''Edit conflict - other version:''' ----
<pre class="bconsole" id="bconsole" style="font-size: 9pt; padding-left: 5px; overflow: scroll; width: 98vw; height: 80vh; background: #000; color: #fff;" onclick="focusScreen();">
---- /!\ '''Edit conflict - your version:''' ----
<pre class="bconsole" id="bconsole" style="font-size: 9pt; padding-left: 5px; overflow: scroll; width: 98vw; height: 80vh; background: #000; color: #fff;" onclick="focusScreen();">
---- /!\ '''End of edit conflict''' ----
<p><?php echo $this->translate("bconsole (batch-mode), please handle with care."); ?><br /><?php echo $this->translate("All commands have to be a one liner, dialogs are not working."); ?><br /><?php echo $this->translate("Type help for a list of commands."); ?></p>
<div class="prompt"><label> * </label><input type="text" class="cli" id="cli" value ="" size=120 style="display: inline; background: #000; color: #fff; border-color: #000; border-style: hidden; outline: none;"/></div>
</pre>console.phtml.patch
41c41 < <pre class="bconsole" id="bconsole" style="font-size: 9pt; padding-left: 5px; overflow: scroll; width: 1080px; height: 650px; background: #000; color: #fff;" onclick="focusScreen();"> --- > <pre class="bconsole" id="bconsole" style="font-size: 9pt; padding-left: 5px; overflow: scroll; width: 98vw; height: 80vh; background: #000; color: #fff;" onclick="focusScreen();">
patch /usr/share/bareos-webui/module/Director/view/director/director/console.phtml < console.phtml.patch
Bareos Tray Monitor
Does not start correctly, when it is called from application menu, becasue it is called with a configuration file, that does not exist.
A strace told me, that bareos tray monitor tries to read this files by default.
So we can just leave the option -c and its argument away. Just make a copy of the entry in the application launcher and change the exec call to /usr/bin/bareos-tray-monitor. You may now start this with your GUI-Session.