Bareos

Contents

Bareos

About

Backup Archiving Recovery Open Sourced

Backed by a German corporation Bareos GmbH & Co. KG that offers support. Bareos Pricing

Source Code:

   1 git clone https://github.com/bareos/bareos.git

What is Bareos?

Bareos (Backup Archiving Recovery Open Sourced) is a reliable, cross-network open source software for backup, archiving and recovery of data for all well-established operating systems. Emerged from the Bacula Project in 2010, Bareos was and is actively developed as a fork and enriched with lots of new features.

Thus Bareos today offers among other features LTO hardware encryption, bandwidth limitation and new practical console commands. The source code of Bareos is available on https://github.com/bareos/ under the AGPL v3 license. Additionally Bareos offers binary package repositories for the most important Linux distributions as well as for Windows.

Source: bareos.com - What is Bareos

Preface

Make sure you have some disk-resources when experimenting. In my little lab I gathered 1TiB in just 5 days of active usage.

Installation

Debian packages are quite outdated even in Sid, just use the bareos repo.

/etc/apt/sources.list.d/bareos.list

   1 deb http://download.bareos.org/bareos/release/17.2//Debian_9.0 /
   2 deb http://download.bareos.org/bareos/release/18.2//Debian_9.0 /
   3 deb http://download.bareos.org/bareos/release/latest/Debian_9.0/ /

Import the public key to proof the signature.

   1 apt-key adv --keyserver keyserver.ubuntu.com --recv-keys 118283D9A7862CEE

Package "bareos" pulls in all dependencies. As a base OS i suggest Debian 10 Codename "Buster". Latest version in Debian is currently 17.2.7, but on Github 18.5 is in the works.

   1 aptitude install bareos

VM Setup

I'm running bareos in a VM running on Libvirt/KVM (2Cores, 8GiB Ram, 20GiB Storage, 1Nic). Nothing special besides a filesystem residing on btrfs raid1 on the host mapped to the guest with gp9-virtio in mode pass-through mode. This is where backups are written to. The driver will be exchanged with virtio-fs when Linux 5.4 becomes stable and decent support is available in the tooling.

   1 mount -t 9p
   2 bareos_storage on /var/lib/bareos/storage type 9p (rw,relatime,sync,dirsync,access=client,trans=virtio)

Configure

Bareos Director-Daemon

Ressource Director (self)

/etc/bareos/bareos-dir.d/director/bareos-dir.conf

   1 Director {                            # define myself
   2   Name = bareos-dir
   3   QueryFile = "/usr/lib/bareos/scripts/query.sql"
   4   Maximum Concurrent Jobs = 10
   5   Password = "very_log_director_password"         # Console password
   6   Messages = Daemon
   7   Auditing = yes
   8 
   9   # Enable the Heartbeat if you experience connection losses
  10   # (eg. because of your router or firewall configuration).
  11   # Additionally the Heartbeat can be enabled in bareos-sd and bareos-fd.
  12   #
  13   # Heartbeat Interval = 1 min
  14 
  15   # remove comment in next line to load dynamic backends from specified directory
  16   # Backend Directory = /usr/lib/bareos/backends
  17 
  18   # remove comment from "Plugin Directory" to load plugins from specified directory.
  19   # if "Plugin Names" is defined, only the specified plugins will be loaded,
  20   # otherwise all director plugins (*-dir.so) from the "Plugin Directory".
  21   #
  22   # Plugin Directory = /usr/lib/bareos/plugins
  23   # Plugin Names = ""
  24 
  25   ### Collect and keep statistics
  26   ### This enabled by default in the director resource,
  27   ### but must be enabled manually in the storage resource.
  28   Statistics Collect Interval = 150             ### DEFAULT: 150
  29   Statistics Retention = 160704000              ### Defaults: 160704000 (5y)
  30 }

Filesets

LinuxBase

/etc/bareos/bareos-dir.d/fileset/LinuxBase.conf

   1 FileSet {
   2   Name = "LinuxBase"
   3   Description = "Backup a minimal set of a Linux Server."
   4   Enable VSS = false
   5 
   6   Include {
   7     Options {
   8       Compression       = GZIP
   9       Signature         = SHA256
  10       One FS            = yes
  11       noatime           = yes
  12       Sparse            = yes
  13       IgnoreCase        = no
  14       Wildfile = /boot/config-*
  15     }
  16     Options {
  17       exclude = yes
  18       Regexdir = "/home/[^/]+/.steam"
  19       Regexdir = "/home/[^/]+/.local/share/Trash/files"
  20       Regexdir = "/home/[^/]+/Downloads"
  21     }
  22     Exclude Dir Containing = .nobackup
  23     File = /etc
  24     File = /home
  25     File = /opt
  26     File = /root
  27     File = /var/lib
  28   }
  29 
  30   Include {
  31      Options {
  32         Wildfile = /boot/config-*
  33      }
  34      Options {
  35         Exclude = yes
  36         RegexFile = ".*"
  37      }
  38      File = /boot
  39   }
  40 
  41   Exclude {
  42   }
  43 }

Windows10AllMedia

Gather all media files of every user.

/etc/bareos/bareos-dir.d/fileset/Windows10AllMedia.conf

   1 FileSet {
   2   Name = "Windows10AllMedia"
   3 
   4   Include {
   5 
   6     File  = "C:/Users"
   7 
   8     Options {
   9       signature = SHA256
  10       verify = s1
  11       IgnoreCase = yes
  12 
  13       # Include all users’ directories so we reach the inner ones.  Unlike a
  14       # WildDir pattern ending in *, this RegExDir only matches the top-level
  15       # directories and not any inner ones.
  16       RegExDir = "^C:/Users/[^/]+$"
  17 
  18       # Ditto all users’ Media directories.
  19       WildDir = "C:/Users/*/Bibliothek"
  20       WildDir = "C:/Users/*/Documents"
  21       WildDir = "C:/Users/*/Music"
  22       WildDir = "C:/Users/*/Pictures"
  23       WildDir = "C:/Users/*/Videos"
  24 
  25       # Include the contents of the Media directories and any subdirectories.
  26       Wild = "C:/Users/*/Bibliothek/*"
  27       Wild = "C:/Users/*/Documents/*"
  28       Wild = "C:/Users/*/Music/*"
  29       Wild = "C:/Users/*/Pictures/*"
  30       Wild = "C:/Users/*/Videos/*"
  31     }
  32 
  33     Options {
  34       Exclude = yes
  35       IgnoreCase = yes
  36 
  37       # Exclude everything else, in particular any files at the top level and
  38       # any other directories or files in the users’ directories.
  39       Wild = "C:/Users/*"
  40     }
  41   }
  42 }

Job Defaults (JobDefs)

Tox simplify configuration of a job, sets of defaults for jobs can be created, which than may be referenced during the creation of a job with jobdef="DefaultJob-Name"

/etc/bareos/bareos-dir.d/jobdefs/DefaultJob-Linux.conf

   1 JobDefs {
   2   Name = "DefaultJob-Linux"
   3   Type = Backup
   4   Level = Incremental
   5   Messages = "Standard"
   6   Storage = "File"
   7   Pool = "Incremental"
   8   FullBackupPool = "Full"
   9   IncrementalBackupPool = "Incremental"
  10   DifferentialBackupPool = "Differential"
  11   Client = "bareos-fd"
  12   FileSet = "LinuxBase"
  13   Schedule = "WeeklyCycle"
  14   WriteBootstrap = "/var/lib/bareos/%c.bsr"
  15   runscript {
  16     command = "/usr/local/lib/bareos/scripts/bareos-run_script.sh before"
  17     runswhen = before
  18     failonerror = No
  19   }
  20   runscript {
  21     command = "/usr/local/lib/bareos/scripts/bareos-run_script.sh after"
  22     runswhen = after
  23     failonerror = No
  24   }
  25 }

/etc/bareos/bareos-dir.d/jobdefs/DefaultJob-Windows10.conf

   1 JobDefs {
   2   Name = "DefaultJob-Windows10"
   3   Type = Backup
   4   Level = Incremental
   5   Messages = "Standard"
   6   Storage = "File"
   7   Pool = "Incremental"
   8   FullBackupPool = "Full"
   9   IncrementalBackupPool = "Incremental"
  10   DifferentialBackupPool = "Differential"
  11   Client = "muckie-pc-fd"
  12   FileSet = "Windows10AllMedia"
  13   Schedule = "WeeklyCycle"
  14   WriteBootstrap = "/var/lib/bareos/%c.bsr"
  15 }

Pre/Post Backup Scripts

Create a directory for custom scripts on the client

   1 install -o root -g root -d /usr/local/lib/bareos/scripts

A little script to be used to run some tasks before and after the backup with some sensitivity of the distribution.

/usr/local/lib/bareos/scripts/bareos-run_script.sh

   1 #!/bin/bash
   2 
   3 APPEND_DATE=false
   4 
   5 usage () {
   6 cat <<-EOL
   7 $(basename $0) [-d||--date] [--] [before||after]
   8 EOL
   9 }
  10 
  11 # Note that we use "$@" to let each command-line parameter expand to a
  12 # separate word. The quotes around "$@" are essential!
  13 # We need TEMP as the 'eval set --' would nuke the return value of getopt.
  14 TEMP=$(getopt -o 'd' --long 'date' -n "$(basename $0)" -- "$@")
  15 
  16 if [ $? -ne 0 ]; then
  17         echo 'Terminating...' >&2
  18         exit 1
  19 fi
  20 
  21 # Note the quotes around "$TEMP": they are essential!
  22 eval set -- "$TEMP"
  23 unset TEMP
  24 
  25 while true; do
  26         case "$1" in
  27                 '-d'|'--date')
  28                         APPEND_DATE=true
  29                         shift
  30                         continue
  31                 ;;
  32                 #'-b'|'--b-long')
  33                 #       echo "Option b, argument '$2'"
  34                 #       shift 2
  35                 #       continue
  36                 #;;
  37                 #'-c'|'--c-long')
  38                 #       # c has an optional argument. As we are in quoted mode,
  39                 #       # an empty parameter will be generated if its optional
  40                 #       # argument is not found.
  41                 #       case "$2" in
  42                 #               '')
  43                 #                       echo 'Option c, no argument'
  44                 #               ;;
  45                 #               *)
  46                 #                       echo "Option c, argument '$2'"
  47                 #               ;;
  48                 #       esac
  49                 #       shift 2
  50                 #       continue
  51                 #;;
  52                 '--')
  53                         shift
  54                         break
  55                 ;;
  56                 *)
  57                         echo 'Internal error!' >&2
  58                         exit 1
  59                 ;;
  60         esac
  61 done
  62 
  63 ### SANITIZE
  64 if   [ ${#@} -eq 0 ]; then
  65         WHEN="before"
  66 elif [ ${#@} -gt 1 ]; then
  67         cat<<-EOL
  68                 Please decide between "before" and "after".
  69                 Default is before.
  70         EOL
  71         usage
  72         exit 2
  73 elif [ $1 == "before" ] || [ $1 == "after" ]; then
  74         WHEN="$1"
  75 else
  76         echo "Unknown execution timepoint '$1'."
  77         echo "Exiting…"
  78         usage
  79         exit 2
  80 fi
  81 
  82 FILE_SUFFIX="txt"
  83 if $APPEND_DATE; then
  84         DATE="$(date +%F)"
  85         FILE_SUFFIX="_$DATE.$FILE_SUFFIX"
  86 else
  87         FILE_SUFFIX=".$FILE_SUFFIX"
  88 fi
  89 
  90 ISSUE="$(cat /etc/issue)"
  91 if egrep -q '(Debian|Ubuntu)' <<< "$ISSUE"; then
  92         OS="Debian"
  93 fi
  94 
  95 ### MAIN
  96 
  97 
  98 if [ "$OS" == "Debian" ]; then
  99         BACKUP_DIR="/var/backups/bareos"
 100         FILE_SELECTIONS="dpkg-selections$FILE_SUFFIX"
 101         FILE_LIST="dpkg-list$FILE_SUFFIX"
 102 else
 103         echo "Unsupported OS. Exiting…"
 104         exit 1
 105 fi
 106 
 107 ### COMMAND LIST: AFTER
 108 if [ "$WHEN" == "before" ]; then
 109         [ -d "$BACKUP_DIR" ] || mkdir "$BACKUP_DIR"
 110         dpkg --get-selections > "$BACKUP_DIR/$FILE_SELECTIONS"
 111         dpkg -l > "$BACKUP_DIR/$FILE_LIST"
 112 
 113 ### COMMAND LIST: AFTER
 114 elif [ "$WHEN" == "after" ]; then
 115         rm "$BACKUP_DIR/$FILE_SELECTIONS"
 116         rm "$BACKUP_DIR/$FILE_LIST"
 117 
 118 fi
 119 
 120 echo "Script executed successfully $WHEN backup."
 121 exit 0

Storage

There are some statistics that should come in handy if you want to estimate how long you backup will take or if e.g. you want to account and bill the backup storage.

/etc/bareos/bareos-dir.d/storage/File.conf

   1 Storage {
   2   Name = File
   3   Address = backup1                # N.B. Use a fully qualified name here (do not use "localhost" here).
   4   Password = "very_long_storage_daemon_password"
   5   Device = FileStorage
   6   Media Type = File
   7   Collect Statistics = yes      ### DEFAULT: no
   8 }

Pools

In Maximum Volume Bytes it makes no difference if you write a capital "G" or a lowercase "g".

/etc/bareos/bareos-dir.d/pool/Differential.conf

   1 Pool {
   2   Name = Differential
   3   Pool Type = Backup
   4   Recycle = yes                       # Bareos can automatically recycle Volumes
   5   AutoPrune = yes                     # Prune expired volumes
   6   Volume Retention = 90 days          # How long should the Differential Backups be kept? (#09)
   7   Maximum Volume Bytes = 10G          # Limit Volume size to something reasonable
   8   Maximum Volumes = 100               # Limit number of Volumes in Pool
   9   Label Format = "Differential-"      # Volumes will be labeled "Differential-<volume-id>"
  10 }

/etc/bareos/bareos-dir.d/pool/Full.conf

   1 Pool {
   2   Name = Full
   3   Pool Type = Backup
   4   Recycle = yes                       # Bareos can automatically recycle Volumes
   5   AutoPrune = yes                     # Prune expired volumes
   6   Volume Retention = 365 days         # How long should the Full Backups be kept? (#06)
   7   Maximum Volume Bytes = 50G          # Limit Volume size to something reasonable
   8   Maximum Volumes = 100               # Limit number of Volumes in Pool
   9   Label Format = "Full-"              # Volumes will be labeled "Full-<volume-id>"
  10 }

/etc/bareos/bareos-dir.d/pool/Incremental.conf

   1 Pool {
   2   Name = Incremental
   3   Pool Type = Backup
   4   Recycle = yes                       # Bareos can automatically recycle Volumes
   5   AutoPrune = yes                     # Prune expired volumes
   6   Volume Retention = 30 days          # How long should the Incremental Backups be kept?  (#12)
   7   Maximum Volume Bytes = 5g           # Limit Volume size to something reasonable
   8   Maximum Volumes = 200               # Limit number of Volumes in Pool
   9   Label Format = "Incremental-"       # Volumes will be labeled "Incremental-<volume-id>"
  10 }

/etc/bareos/bareos-dir.d/pool/Scratch.conf

   1 Pool {
   2   Name = Scratch
   3   Pool Type = Scratch
   4 }

Trouble Shooting

Wrong Unix Permissions in Config Directories

bareos-director does not start if it can't read a config-file. So make sure not to create files that bareos can't read. This can easily happen if you copy a file with unix permission o-rwx as root, which changes owner.

It will definetely complain in the logs.

Okt 16 09:12:46 backup1 systemd[1]: Starting Bareos Director Daemon service...
Okt 16 09:12:46 backup1 bareos-dir[1573]: 0bareos-dir: ERROR TERMINATION at parse_conf.c:198
Okt 16 09:12:46 backup1 bareos-dir[1573]: Config error: Cannot open config file "/etc/bareos/bareos-dir.d/*/*.conf": Keine Berechtigung
Okt 16 09:12:46 backup1 bareos-dir[1574]: [55B blob data]
Okt 16 09:12:46 backup1 bareos-dir[1574]: Config error: Cannot open config file "/etc/bareos/bareos-dir.d/*/*.conf": Keine Berechtigung
Okt 16 09:12:46 backup1 systemd[1]: bareos-director.service: Can't open PID file /var/lib/bareos/bareos-dir.9101.pid (yet?) after start: No such file or directory
Okt 16 09:12:46 backup1 systemd[1]: bareos-director.service: Failed with result 'protocol'.

To identify the file you can also use the following command:

   1 find /etc/bareos -type f \! -user bareos \! -group bareos -exec ls -l {} \;

bconsole

Reload Configuration

Atfer changes to the configuration files, bareos config can be reloaded without restarting the service.

   1 echo reload |bconsole

Add Client

Create a new password and hash it to md5:

   1 PASSWORD="$(pwgen 32 1)"
   2 echo -en "$PASSWORD\t";echo -n "$PASSWORD"|md5sum| sed -r 's/\s+.*$//;s/^/[md5]/'
   3 very_long_P455W0rD      [md5]1e83f0211757b383fd4715ba90e229a9

On bareos server configure director:

   1 bconsole
   2 # PLAIN PASSWORD DISCOURAGED
   3 configure add client name="fuxtop" address="fuxtop.dungeon.rockstable.org" description="Fux's laptop" catalog="MyCatalog" password="very_long_P455W0rD"

Add Job

   1 bconsole
   2 configure add job name=muckie-pc-job jobdefs=DefaultJob-Windows10 client=muckie-pc-fd

Label Mediums

Edit pool to hold more and eventually larger mediums

/etc/bareos/bareos-dir.d/pool/Incremental.conf

   1 Pool {
   2   Name = Incremental
   3   Pool Type = Backup
   4   Recycle = yes                       # Bareos can automatically recycle Volumes
   5   AutoPrune = yes                     # Prune expired volumes
   6   Volume Retention = 30 days          # How long should the Incremental Backups be kept?  (#12)
   7   Maximum Volume Bytes = 4G           # Limit Volume size to something reasonable
   8   Maximum Volumes = 150               # Limit number of Volumes in Pool
   9   Label Format = "Incremental-"       # Volumes will be labeled "Incremental-<volume-id>"
  10 }

Label

   1 # bconsole
   2 Connecting to Director localhost:9101
   3 1000 OK: bareos-dir Version: 17.2.7 (16 Jul 2018)
   4 Enter a period to cancel a command.
   5 *reload
   6 reloaded
   7 *
   8 *label pool=Incremental storage=File volume=Incremental-0139
   9 Connecting to Storage daemon File at backup1:9103 ...
  10 Sending label command for Volume "Incremental-0139" Slot 0 ...
  11 3000 OK label. VolBytes=204 Volume="Incremental-0139" Device="FileStorage" (/var/lib/bareos/storage)
  12 Catalog record for Volume "Incremental-0139", Slot 0 successfully created.
  13 Requesting to mount FileStorage ...
  14 3001 OK mount requested. Device="FileStorage" (/var/lib/bareos/storage)

Job rescued.

Bareos Storage-Daemon

Bareos Filesystem-Daemon

Caveats

bareos-fd.service has dependencies on

nss-lookup.target network.target remote-fs.target time-sync.target

It won't start successfully if any of these targets can't be reached. So make sure that e.g. any remote-mount (like nfs, smb, glusterfs, …) in /etc/fstab works under any condition. In doubt use autofs.

Bareos-Webui

http://backup1/bareos-webui/dashboard/

   1 aptitude install bareos-webui

Http2 support

Apache2 mod_http2 does not support mod_mpm_prefork. So we switch to php-fpm, mod_proxy_fcgi and mod_mpm_event.

   1 aptitude install php-fpm
   2 a2dismod php7.0
   3 a2enconf php7.3-fpm
   4 a2enmod ssl
   5 a2enmod proxy_fcgi
   6 a2dismod mpm_prefork
   7 a2enmod mpm_event
   8 systemctl restart apache2

Trouble Shooting

Bareos Bug-Report #1120

mod_proxy_fcgi throws an error with php7.3+, regarding a undefined variable "extras".

[Mon Oct 14 12:54:19.526596 2019] [proxy_fcgi:error] [pid 842:tid 139696600672000] [client 192.168.182.13:40176] AH01071: Got error 'PHP message: PHP Notice:  compact(): Undefined variable: extras in /usr/share/bareos-webui/vendor/zendframework/zend-view/src/Helper/HeadLink.php on line 403PHP message: PHP Notice:  compact(): Undefined variable:
extras in /usr/share/bareos-webui/vendor/zendframework/zend-view/src/Helper/HeadLink.php on line 403'

So i applied a little patch to /usr/share/bareos-webui/vendor/zendframework/zend-view/src/Helper/HeadLink.php which initializes extras to empty string "" if (0 < count($args) && is_array($args[0])

375a376
>         $extras                = "";
400a402,403
>       } else {
>             $extras = (string) $extras;

   1     /**
   2      * Create item for stylesheet link item
   3      *
   4      * @param  array $args
   5      * @return stdClass|false Returns false if stylesheet is a duplicate
   6      */
   7     public function createDataStylesheet(array $args)
   8     {
   9         $rel                   = 'stylesheet';
  10         $type                  = 'text/css';
  11         $media                 = 'screen';
  12         $conditionalStylesheet = false;
  13         $href                  = array_shift($args);
  14         $extras                = "";
  15 
  16         if ($this->isDuplicateStylesheet($href)) {
  17             return false;
  18         }
  19 
  20         if (0 < count($args)) {
  21             $media = array_shift($args);
  22             if (is_array($media)) {
  23                 $media = implode(',', $media);
  24             } else {
  25                 $media = (string) $media;
  26             }
  27         }
  28         if (0 < count($args)) {
  29             $conditionalStylesheet = array_shift($args);
  30             if (!empty($conditionalStylesheet) && is_string($conditionalStylesheet)) {
  31                 $conditionalStylesheet = (string) $conditionalStylesheet;
  32             } else {
  33                 $conditionalStylesheet = null;
  34             }
  35         }
  36 
  37         if (0 < count($args) && is_array($args[0])) {
  38             $extras = array_shift($args);
  39             $extras = (array) $extras;
  40         } else {
  41             $extras = (string) $extras;
  42         }
  43 
  44         $attributes = compact('rel', 'type', 'href', 'media', 'conditionalStylesheet', 'extras');
  45 
  46         return $this->createData($attributes);
  47     }

Yes it's ugly - but works, so far.

BConsole webview

Adjust bconsole size in /usr/share/bareos-webui/module/Director/view/director/director/console.phtml

<pre class="bconsole" id="bconsole" style="font-size: 9pt; padding-left: 5px; overflow: scroll; width: 99vw; height: 80vh; background: #000; color: #fff;" onclick="focusScreen();">
   <p><?php echo $this->translate("bconsole (batch-mode), please handle with care."); ?><br /><?php echo $this->translate("All commands have to be a one liner, dialogs are not working."); ?><br /><?php echo $this->translate("Type help for a list of commands."); ?></p>
   <div class="prompt"><label> * </label><input type="text" class="cli" id="cli" value ="" size=120 style="display: inline; background: #000; color: #fff; border-color: #000; border-style: hidden; outline: none;"/></div>
</pre>

Configurations

Services

Sports

Navigation

Bareos

About

What is Bareos?

Preface

Installation

VM Setup

Configure

Bareos Director-Daemon

Ressource Director (self)

Filesets

LinuxBase

Windows10AllMedia

Job Defaults (JobDefs)

Pre/Post Backup Scripts

Storage

Pools

Trouble Shooting

Wrong Unix Permissions in Config Directories

bconsole

Reload Configuration

Add Client

Add Job

Label Mediums

Bareos Storage-Daemon

Bareos Filesystem-Daemon

Caveats

Bareos-Webui

Http2 support

Trouble Shooting

BConsole webview