Bareos
Contents
About
Backup Archiving Recovery Open Sourced
- Repositories
Official Bareos Subscription Repository
- Requires subscription
Bareos Community Repository Nightly Builds
- Be careful 0,o
Backed by a German corporation Bareos GmbH & Co. KG that offers support.
Please take a look at Bareos Pricing
Source code:
1 git clone https://github.com/bareos/bareos.git
What is Bareos?
Bareos (Backup Archiving Recovery Open Sourced) is a reliable, cross-network open source software for backup, archiving and recovery of data for all well-established operating systems. Emerged from the Bacula Project in 2010, Bareos was and is actively developed as a fork and enriched with lots of new features.
Thus Bareos today offers among other features LTO hardware encryption, bandwidth limitation and new practical console commands. The source code of Bareos is available on https://github.com/bareos/ under the AGPL v3 license. Additionally Bareos offers binary package repositories for the most important Linux distributions as well as for Windows.
Source: bareos.com - What is Bareos
Ports
tcp/9101 bareos-dir
tcp/9102 bareos-fd
- must be open on client for a backup to be initiated by the director
tcp/9103 bareos-sd
There are predefined services in firewalld "bacula" and "bacula-client".
Preface
Make sure you have some disk-resources when experimenting. In my little lab I gathered 1TiB in just 5 days of active usage.
VM Setup
I'm running bareos in a VM running on Libvirt/KVM (2Cores, 4GiB Ram, 20GiB Storage, 1Nic).
Nothing special besides a filesystem residing on a btrfs raid1 made of 2 8TiB CMR HDDs on the host, which is mapped to the guest. This is where backups are written to.
9p
This worked quite well for a some time, but with the appearance of #virtio-fs and the general availablity of support in the tools, i definetly recommend to go with the new approach.
- 9p is slower
- 9p had problems during boot,
when mounted via /etc/fstab and the vm stuck in on the password prompt for the single user target.
Example domain xml
/etc/fstab
mount -t 9p
1 bareos_storage on /var/lib/bareos/storage type 9p (rw,relatime,sync,dirsync,access=client,trans=virtio)
The driver will be exchanged with #virtio-fs when Linux 5.4 becomes stable and decent support is available in the tooling.
Fio test
Just a naive test.
1 mkdir /var/lib/bareos/storage_9p/fio
2 cd /var/lib/bareos/storage_9p/fio
3 fio /usr/share/doc/fio/examples/fio-seq-RW.fio
4 file1: (g=0): rw=rw, bs=(R) 256KiB-256KiB, (W) 256KiB-256KiB, (T) 256KiB-256KiB, ioengine=libaio, iodepth=16
5 ...
6 fio-3.12
7 Starting 4 processes
8 file1: Laying out IO file (1 file / 10240MiB)
9 Jobs: 4 (f=4): [M(4)][100.0%][r=59.3MiB/s,w=39.8MiB/s][r=237,w=159 IOPS][eta 00m:00s]
10 file1: (groupid=0, jobs=1): err= 0: pid=17725: Thu Jan 14 12:21:36 2021
11 read: IOPS=24, BW=6324KiB/s (6476kB/s)(5559MiB/900030msec)
12 slat (usec): min=2851, max=14972, avg=5514.29, stdev=1014.60
13 clat (msec): min=64, max=3525, avg=360.71, stdev=304.31
14 lat (msec): min=68, max=3529, avg=366.23, stdev=304.39
15 clat percentiles (msec):
16 | 1.00th=[ 77], 5.00th=[ 85], 10.00th=[ 97], 20.00th=[ 144],
17 | 30.00th=[ 186], 40.00th=[ 228], 50.00th=[ 271], 60.00th=[ 326],
18 | 70.00th=[ 409], 80.00th=[ 518], 90.00th=[ 709], 95.00th=[ 911],
19 | 99.00th=[ 1552], 99.50th=[ 1821], 99.90th=[ 2702], 99.95th=[ 2937],
20 | 99.99th=[ 3473]
21 bw ( KiB/s): min= 510, max=34304, per=25.99%, avg=6531.75, stdev=4966.58, samples=1742
22 iops : min= 1, max= 134, avg=25.47, stdev=19.41, samples=1742
23 write: IOPS=16, BW=4203KiB/s (4303kB/s)(3694MiB/900030msec); 0 zone resets
24 slat (msec): min=4, max=2373, avg=52.60, stdev=76.79
25 clat (usec): min=5, max=3472.5k, avg=370781.79, stdev=315624.33
26 lat (msec): min=72, max=3577, avg=423.38, stdev=348.86
27 clat percentiles (msec):
28 | 1.00th=[ 75], 5.00th=[ 84], 10.00th=[ 96], 20.00th=[ 146],
29 | 30.00th=[ 190], 40.00th=[ 234], 50.00th=[ 279], 60.00th=[ 338],
30 | 70.00th=[ 422], 80.00th=[ 535], 90.00th=[ 735], 95.00th=[ 944],
31 | 99.00th=[ 1620], 99.50th=[ 1921], 99.90th=[ 2903], 99.95th=[ 3138],
32 | 99.99th=[ 3339]
33 bw ( KiB/s): min= 510, max=24064, per=26.19%, avg=4406.25, stdev=3253.58, samples=1716
34 iops : min= 1, max= 94, avg=17.17, stdev=12.72, samples=1716
35 lat (usec) : 10=0.01%
36 lat (msec) : 100=10.91%, 250=34.10%, 500=32.98%, 750=13.00%, 1000=5.04%
37 cpu : usr=0.08%, sys=2.91%, ctx=2374272, majf=0, minf=12
38 IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=100.0%, 32=0.0%, >=64=0.0%
39 submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
40 complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
41 issued rwts: total=22235,14775,0,0 short=0,0,0,0 dropped=0,0,0,0
42 latency : target=0, window=0, percentile=100.00%, depth=16
43 file1: (groupid=0, jobs=1): err= 0: pid=17726: Thu Jan 14 12:21:36 2021
44 read: IOPS=24, BW=6339KiB/s (6491kB/s)(5571MiB/900030msec)
45 slat (usec): min=2584, max=11855, avg=5517.96, stdev=1015.75
46 clat (msec): min=62, max=3945, avg=359.68, stdev=306.16
47 lat (msec): min=67, max=3950, avg=365.20, stdev=306.24
48 clat percentiles (msec):
49 | 1.00th=[ 75], 5.00th=[ 83], 10.00th=[ 93], 20.00th=[ 142],
50 | 30.00th=[ 184], 40.00th=[ 228], 50.00th=[ 275], 60.00th=[ 330],
51 | 70.00th=[ 405], 80.00th=[ 518], 90.00th=[ 709], 95.00th=[ 902],
52 | 99.00th=[ 1603], 99.50th=[ 1821], 99.90th=[ 2635], 99.95th=[ 3004],
53 | 99.99th=[ 3876]
54 bw ( KiB/s): min= 510, max=33280, per=26.02%, avg=6540.27, stdev=5022.02, samples=1743
55 iops : min= 1, max= 130, avg=25.43, stdev=19.63, samples=1743
56 write: IOPS=16, BW=4211KiB/s (4312kB/s)(3701MiB/900030msec); 0 zone resets
57 slat (msec): min=4, max=2370, avg=52.47, stdev=76.13
58 clat (usec): min=13, max=3766.0k, avg=370292.61, stdev=320940.52
59 lat (msec): min=70, max=4049, avg=422.76, stdev=354.82
60 clat percentiles (msec):
61 | 1.00th=[ 75], 5.00th=[ 84], 10.00th=[ 94], 20.00th=[ 146],
62 | 30.00th=[ 190], 40.00th=[ 234], 50.00th=[ 279], 60.00th=[ 338],
63 | 70.00th=[ 418], 80.00th=[ 535], 90.00th=[ 726], 95.00th=[ 919],
64 | 99.00th=[ 1653], 99.50th=[ 2022], 99.90th=[ 2903], 99.95th=[ 3272],
65 | 99.99th=[ 3675]
66 bw ( KiB/s): min= 510, max=23504, per=26.16%, avg=4400.15, stdev=3193.90, samples=1722
67 iops : min= 1, max= 91, avg=17.07, stdev=12.48, samples=1722
68 lat (usec) : 20=0.01%
69 lat (msec) : 100=11.49%, 250=32.92%, 500=33.83%, 750=13.13%, 1000=4.83%
70 cpu : usr=0.08%, sys=2.90%, ctx=2379462, majf=0, minf=15
71 IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=100.0%, 32=0.0%, >=64=0.0%
72 submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
73 complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
74 issued rwts: total=22285,14805,0,0 short=0,0,0,0 dropped=0,0,0,0
75 latency : target=0, window=0, percentile=100.00%, depth=16
76 file1: (groupid=0, jobs=1): err= 0: pid=17727: Thu Jan 14 12:21:36 2021
77 read: IOPS=24, BW=6207KiB/s (6356kB/s)(5455MiB/900030msec)
78 slat (usec): min=2971, max=18847, avg=5537.71, stdev=1016.32
79 clat (msec): min=61, max=3428, avg=366.17, stdev=307.00
80 lat (msec): min=66, max=3433, avg=371.71, stdev=307.07
81 clat percentiles (msec):
82 | 1.00th=[ 78], 5.00th=[ 86], 10.00th=[ 100], 20.00th=[ 148],
83 | 30.00th=[ 190], 40.00th=[ 234], 50.00th=[ 279], 60.00th=[ 338],
84 | 70.00th=[ 418], 80.00th=[ 531], 90.00th=[ 709], 95.00th=[ 911],
85 | 99.00th=[ 1569], 99.50th=[ 1871], 99.90th=[ 2836], 99.95th=[ 3205],
86 | 99.99th=[ 3339]
87 bw ( KiB/s): min= 510, max=33792, per=25.53%, avg=6416.51, stdev=4803.03, samples=1740
88 iops : min= 1, max= 132, avg=24.94, stdev=18.77, samples=1740
89 write: IOPS=16, BW=4204KiB/s (4305kB/s)(3695MiB/900030msec); 0 zone resets
90 slat (msec): min=4, max=2442, avg=52.70, stdev=77.42
91 clat (usec): min=9, max=3406.3k, avg=372761.24, stdev=322041.57
92 lat (msec): min=71, max=3548, avg=425.46, stdev=355.36
93 clat percentiles (msec):
94 | 1.00th=[ 78], 5.00th=[ 86], 10.00th=[ 97], 20.00th=[ 148],
95 | 30.00th=[ 194], 40.00th=[ 239], 50.00th=[ 284], 60.00th=[ 342],
96 | 70.00th=[ 422], 80.00th=[ 535], 90.00th=[ 718], 95.00th=[ 927],
97 | 99.00th=[ 1653], 99.50th=[ 2039], 99.90th=[ 3104], 99.95th=[ 3306],
98 | 99.99th=[ 3373]
99 bw ( KiB/s): min= 510, max=22528, per=26.13%, avg=4396.33, stdev=3237.55, samples=1720
100 iops : min= 1, max= 88, avg=17.05, stdev=12.66, samples=1720
101 lat (usec) : 10=0.01%
102 lat (msec) : 100=10.55%, 250=32.81%, 500=34.00%, 750=13.96%, 1000=4.75%
103 cpu : usr=0.08%, sys=2.95%, ctx=2347348, majf=0, minf=14
104 IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=100.0%, 32=0.0%, >=64=0.0%
105 submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
106 complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
107 issued rwts: total=21821,14780,0,0 short=0,0,0,0 dropped=0,0,0,0
108 latency : target=0, window=0, percentile=100.00%, depth=16
109 file1: (groupid=0, jobs=1): err= 0: pid=17728: Thu Jan 14 12:21:36 2021
110 read: IOPS=24, BW=6263KiB/s (6413kB/s)(5505MiB/900029msec)
111 slat (usec): min=2843, max=13630, avg=5508.24, stdev=1017.93
112 clat (msec): min=64, max=3615, avg=363.82, stdev=310.11
113 lat (msec): min=67, max=3619, avg=369.33, stdev=310.19
114 clat percentiles (msec):
115 | 1.00th=[ 75], 5.00th=[ 85], 10.00th=[ 95], 20.00th=[ 144],
116 | 30.00th=[ 186], 40.00th=[ 230], 50.00th=[ 275], 60.00th=[ 334],
117 | 70.00th=[ 418], 80.00th=[ 523], 90.00th=[ 709], 95.00th=[ 911],
118 | 99.00th=[ 1603], 99.50th=[ 1905], 99.90th=[ 2769], 99.95th=[ 3138],
119 | 99.99th=[ 3540]
120 bw ( KiB/s): min= 510, max=34304, per=25.70%, avg=6457.68, stdev=4939.59, samples=1745
121 iops : min= 1, max= 134, avg=25.16, stdev=19.30, samples=1745
122 write: IOPS=16, BW=4206KiB/s (4307kB/s)(3697MiB/900029msec); 0 zone resets
123 slat (msec): min=4, max=2436, avg=52.64, stdev=77.04
124 clat (usec): min=5, max=3420.2k, avg=371105.32, stdev=316602.38
125 lat (msec): min=69, max=3621, avg=423.75, stdev=350.07
126 clat percentiles (msec):
127 | 1.00th=[ 75], 5.00th=[ 85], 10.00th=[ 96], 20.00th=[ 146],
128 | 30.00th=[ 190], 40.00th=[ 236], 50.00th=[ 288], 60.00th=[ 347],
129 | 70.00th=[ 422], 80.00th=[ 531], 90.00th=[ 718], 95.00th=[ 927],
130 | 99.00th=[ 1670], 99.50th=[ 1989], 99.90th=[ 2836], 99.95th=[ 3071],
131 | 99.99th=[ 3406]
132 bw ( KiB/s): min= 510, max=24576, per=26.31%, avg=4426.71, stdev=3227.28, samples=1709
133 iops : min= 1, max= 96, avg=17.23, stdev=12.61, samples=1709
134 lat (usec) : 10=0.01%
135 lat (msec) : 100=11.49%, 250=32.57%, 500=33.66%, 750=13.50%, 1000=4.83%
136 cpu : usr=0.08%, sys=2.90%, ctx=2360901, majf=0, minf=13
137 IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=100.0%, 32=0.0%, >=64=0.0%
138 submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
139 complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
140 issued rwts: total=22018,14788,0,0 short=0,0,0,0 dropped=0,0,0,0
141 latency : target=0, window=0, percentile=100.00%, depth=16
142
143 Run status group 0 (all jobs):
144 READ: bw=24.5MiB/s (25.7MB/s), 6207KiB/s-6339KiB/s (6356kB/s-6491kB/s), io=21.6GiB (23.2GB), run=900029-900030msec
145 WRITE: bw=16.4MiB/s (17.2MB/s), 4203KiB/s-4211KiB/s (4303kB/s-4312kB/s), io=14.4GiB (15.5GB), run=900029-900030msec
146 fio /usr/share/doc/fio/examples/fio-seq-RW.fio 7,62s user 125,55s system 10% cpu 21:30,52 total
Cleanup
virtio-fs
You'll need
- virtiofs support in the guest kernel (Linux v5.4 or later)
- Libvirt 6.2.0+, better 6.9+
qemu 5.0.0
With shared memory (tmpfs)
I personally fapour the tmpfs approach, because it does not set upper boundaries to the amount of hugepages. However by default tmpfs is also limited to half of the RAM.
/etc/libvirt/qemu.conf
Restart libvirtd
1 systemctl restart libvirtd.service
Example domain xml
1 <domain type='kvm'>
2 …
3 <memory unit='>MiB'>4096</memory>
4 <currentMemory unit='KiB'>2097152</currentMemory>
5 <memoryBacking>
6 <access mode='shared'/>
7 </memoryBacking>
8 …
9 <cpu mode='host-model' check='partial'>
10 <numa>
11 <cell id='0' cpus='0-3' memory='4096' unit='MiB' memAccess='shared'/>
12 </numa>
13 </cpu>
14 …
15 <devices>
16 …
17 <filesystem type='mount' accessmode='passthrough'>
18 <driver type='virtiofs'/>
19 <source dir='/media/space/backup/backup1'/>
20 <target dir='bareos_storage_virtiofs'/>
21 <address type='pci' domain='0x0000' bus='0x08' slot='0x00' function='0x0'/>
22 </filesystem>
23 …
24 </devices>
25 </domain>
With explicit hugepages
The VM won't start, unless you allocated the full amount of RAM in hugepages. Preallocate hugepages to back the memory of the VM.
1 virsh allocpages 2M 2048
Please see Linux#Hugepages for a persistent configuration.
Example domain xml (necessary)
1 <domain type='kvm'>
2 …
3 <memory unit='MiB'>4096</memory>
4 <currentMemory unit='MiB'>2048</currentMemory>
5 <memoryBacking>
6 <hugepages>
7 <page size='2048' unit='KiB'/>
8 </hugepages>
9 <access mode='shared'/>
10 </memoryBacking>
11 …
12 <cpu mode='host-model' check='partial'>
13 <numa>
14 <cell id='0' cpus='0-3' memory='4096' unit='MiB' memAccess='shared'/>
15 </numa>
16 </cpu>
17 …
18 <devices>
19 …
20 <filesystem type='mount' accessmode='passthrough'>
21 <driver type='virtiofs'/>
22 <source dir='/media/space/backup/backup1'/>
23 <target dir='bareos_storage_virtiofs'/>
24 <address type='pci' domain='0x0000' bus='0x08' slot='0x00' function='0x0'/>
25 </filesystem>
26 …
27 </devices>
28 </domain>
Fio test
Just a naive test
1 mkdir /var/lib/bareos/storage/fio
2 cd /var/lib/bareos/storage/fio
3 fio /usr/share/doc/fio/examples/fio-seq-RW.fio
4
5 file1: (g=0): rw=rw, bs=(R) 256KiB-256KiB, (W) 256KiB-256KiB, (T) 256KiB-256KiB, ioengine=libaio, iodepth=16
6 ...
7 fio-3.12
8 Starting 4 processes
9 file1: Laying out IO file (1 file / 10240MiB)
10 Jobs: 4 (f=4): [M(4)][100.0%][r=1069MiB/s,w=727MiB/s][r=4277,w=2906 IOPS][eta 00m:00s]
11 file1: (groupid=0, jobs=1): err= 0: pid=13103: Thu Jan 14 11:43:20 2021
12 read: IOPS=612, BW=153MiB/s (161MB/s)(135GiB/900001msec)
13 slat (usec): min=43, max=853169, avg=393.69, stdev=1263.28
14 clat (usec): min=1602, max=37792k, avg=14890.06, stdev=436567.44
15 lat (usec): min=1717, max=37793k, avg=15284.17, stdev=436569.72
16 clat percentiles (msec):
17 | 1.00th=[ 5], 5.00th=[ 6], 10.00th=[ 6], 20.00th=[ 7],
18 | 30.00th=[ 8], 40.00th=[ 8], 50.00th=[ 9], 60.00th=[ 9],
19 | 70.00th=[ 10], 80.00th=[ 11], 90.00th=[ 12], 95.00th=[ 13],
20 | 99.00th=[ 16], 99.50th=[ 17], 99.90th=[ 25], 99.95th=[ 30],
21 | 99.99th=[17113]
22 bw ( KiB/s): min= 512, max=353792, per=41.39%, avg=259549.94, stdev=43036.90, samples=1086
23 iops : min= 2, max= 1382, avg=1013.84, stdev=168.11, samples=1086
24 write: IOPS=408, BW=102MiB/s (107MB/s)(89.8GiB/900001msec); 0 zone resets
25 slat (usec): min=143, max=37780k, avg=1848.00, stdev=173203.68
26 clat (usec): min=2, max=37793k, avg=14390.07, stdev=406456.91
27 lat (usec): min=1080, max=37793k, avg=16238.60, stdev=441873.69
28 clat percentiles (msec):
29 | 1.00th=[ 5], 5.00th=[ 6], 10.00th=[ 7], 20.00th=[ 7],
30 | 30.00th=[ 8], 40.00th=[ 9], 50.00th=[ 9], 60.00th=[ 10],
31 | 70.00th=[ 10], 80.00th=[ 11], 90.00th=[ 12], 95.00th=[ 13],
32 | 99.00th=[ 16], 99.50th=[ 17], 99.90th=[ 24], 99.95th=[ 30],
33 | 99.99th=[17113]
34 bw ( KiB/s): min= 512, max=225280, per=41.37%, avg=173290.20, stdev=27300.81, samples=1086
35 iops : min= 2, max= 880, avg=676.88, stdev=106.65, samples=1086
36 lat (usec) : 4=0.01%
37 lat (msec) : 2=0.01%, 4=0.55%, 10=74.15%, 20=25.09%, 50=0.16%
38 lat (msec) : 250=0.01%, 750=0.01%, 1000=0.01%
39 cpu : usr=0.80%, sys=27.88%, ctx=1953111, majf=0, minf=12
40 IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=100.0%, 32=0.0%, >=64=0.0%
41 submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
42 complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
43 issued rwts: total=551075,368000,0,0 short=0,0,0,0 dropped=0,0,0,0
44 latency : target=0, window=0, percentile=100.00%, depth=16
45 file1: (groupid=0, jobs=1): err= 0: pid=13104: Thu Jan 14 11:43:20 2021
46 read: IOPS=611, BW=153MiB/s (160MB/s)(134GiB/900001msec)
47 slat (usec): min=42, max=35260k, avg=455.56, stdev=47539.00
48 clat (usec): min=2, max=37788k, avg=15216.77, stdev=447333.20
49 lat (usec): min=243, max=37788k, avg=15672.74, stdev=449851.87
50 clat percentiles (msec):
51 | 1.00th=[ 5], 5.00th=[ 6], 10.00th=[ 6], 20.00th=[ 7],
52 | 30.00th=[ 8], 40.00th=[ 8], 50.00th=[ 9], 60.00th=[ 9],
53 | 70.00th=[ 10], 80.00th=[ 11], 90.00th=[ 12], 95.00th=[ 13],
54 | 99.00th=[ 16], 99.50th=[ 17], 99.90th=[ 24], 99.95th=[ 31],
55 | 99.99th=[17113]
56 bw ( KiB/s): min= 1532, max=344910, per=41.32%, avg=259077.80, stdev=42335.00, samples=1086
57 iops : min= 5, max= 1347, avg=1011.96, stdev=165.36, samples=1086
58 write: IOPS=409, BW=102MiB/s (107MB/s)(89.9GiB/900001msec); 0 zone resets
59 slat (usec): min=148, max=37779k, avg=1755.64, stdev=163159.15
60 clat (usec): min=245, max=37785k, avg=13931.32, stdev=389051.86
61 lat (usec): min=503, max=37789k, avg=15687.48, stdev=421922.18
62 clat percentiles (msec):
63 | 1.00th=[ 5], 5.00th=[ 6], 10.00th=[ 7], 20.00th=[ 7],
64 | 30.00th=[ 8], 40.00th=[ 9], 50.00th=[ 9], 60.00th=[ 10],
65 | 70.00th=[ 10], 80.00th=[ 11], 90.00th=[ 12], 95.00th=[ 13],
66 | 99.00th=[ 16], 99.50th=[ 17], 99.90th=[ 24], 99.95th=[ 29],
67 | 99.99th=[17113]
68 bw ( KiB/s): min= 510, max=220672, per=41.39%, avg=173362.11, stdev=27355.21, samples=1086
69 iops : min= 1, max= 862, avg=677.13, stdev=106.86, samples=1086
70 lat (usec) : 4=0.01%, 250=0.01%, 750=0.01%
71 lat (msec) : 2=0.01%, 4=0.55%, 10=74.03%, 20=25.21%, 50=0.17%
72 lat (msec) : 250=0.01%, 750=0.01%, 1000=0.01%
73 cpu : usr=0.84%, sys=27.71%, ctx=1963887, majf=0, minf=15
74 IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=100.0%, 32=0.0%, >=64=0.0%
75 submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
76 complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
77 issued rwts: total=550190,368159,0,0 short=0,0,0,0 dropped=0,0,0,0
78 latency : target=0, window=0, percentile=100.00%, depth=16
79 file1: (groupid=0, jobs=1): err= 0: pid=13105: Thu Jan 14 11:43:20 2021
80 read: IOPS=612, BW=153MiB/s (161MB/s)(135GiB/900001msec)
81 slat (usec): min=44, max=7437.3k, avg=412.49, stdev=11059.38
82 clat (usec): min=742, max=37792k, avg=14685.10, stdev=427440.19
83 lat (usec): min=1247, max=37793k, avg=15098.00, stdev=427598.94
84 clat percentiles (msec):
85 | 1.00th=[ 5], 5.00th=[ 6], 10.00th=[ 6], 20.00th=[ 7],
86 | 30.00th=[ 8], 40.00th=[ 8], 50.00th=[ 9], 60.00th=[ 9],
87 | 70.00th=[ 10], 80.00th=[ 11], 90.00th=[ 12], 95.00th=[ 13],
88 | 99.00th=[ 16], 99.50th=[ 17], 99.90th=[ 24], 99.95th=[ 30],
89 | 99.99th=[17113]
90 bw ( KiB/s): min= 510, max=362496, per=41.35%, avg=259291.17, stdev=43366.78, samples=1088
91 iops : min= 1, max= 1416, avg=1012.81, stdev=169.40, samples=1088
92 write: IOPS=409, BW=102MiB/s (107MB/s)(89.9GiB/900001msec); 0 zone resets
93 slat (usec): min=112, max=37779k, avg=1818.40, stdev=172556.40
94 clat (usec): min=4, max=37794k, avg=14684.07, stdev=420477.17
95 lat (usec): min=740, max=37795k, avg=16503.00, stdev=454556.31
96 clat percentiles (msec):
97 | 1.00th=[ 5], 5.00th=[ 6], 10.00th=[ 7], 20.00th=[ 7],
98 | 30.00th=[ 8], 40.00th=[ 9], 50.00th=[ 9], 60.00th=[ 10],
99 | 70.00th=[ 10], 80.00th=[ 11], 90.00th=[ 12], 95.00th=[ 13],
100 | 99.00th=[ 16], 99.50th=[ 17], 99.90th=[ 23], 99.95th=[ 28],
101 | 99.99th=[17113]
102 bw ( KiB/s): min= 510, max=236544, per=41.37%, avg=173290.47, stdev=27898.89, samples=1088
103 iops : min= 1, max= 924, avg=676.87, stdev=108.99, samples=1088
104 lat (usec) : 10=0.01%, 750=0.01%
105 lat (msec) : 2=0.01%, 4=0.50%, 10=74.25%, 20=25.05%, 50=0.15%
106 lat (msec) : 250=0.01%, 750=0.01%, 1000=0.01%
107 cpu : usr=0.80%, sys=27.80%, ctx=1954852, majf=0, minf=14
108 IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=100.0%, 32=0.0%, >=64=0.0%
109 submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
110 complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
111 issued rwts: total=551089,368308,0,0 short=0,0,0,0 dropped=0,0,0,0
112 latency : target=0, window=0, percentile=100.00%, depth=16
113 file1: (groupid=0, jobs=1): err= 0: pid=13106: Thu Jan 14 11:43:20 2021
114 read: IOPS=613, BW=153MiB/s (161MB/s)(135GiB/900001msec)
115 slat (usec): min=41, max=3805.1k, avg=397.25, stdev=5198.37
116 clat (usec): min=1265, max=37792k, avg=14655.43, stdev=422874.96
117 lat (usec): min=1535, max=37792k, avg=15053.09, stdev=422908.42
118 clat percentiles (msec):
119 | 1.00th=[ 5], 5.00th=[ 6], 10.00th=[ 6], 20.00th=[ 7],
120 | 30.00th=[ 8], 40.00th=[ 8], 50.00th=[ 9], 60.00th=[ 9],
121 | 70.00th=[ 10], 80.00th=[ 11], 90.00th=[ 12], 95.00th=[ 13],
122 | 99.00th=[ 16], 99.50th=[ 17], 99.90th=[ 24], 99.95th=[ 31],
123 | 99.99th=[17113]
124 bw ( KiB/s): min= 512, max=351232, per=41.43%, avg=259803.88, stdev=43436.51, samples=1088
125 iops : min= 2, max= 1372, avg=1014.84, stdev=169.67, samples=1088
126 write: IOPS=409, BW=102MiB/s (107MB/s)(89.9GiB/900001msec); 0 zone resets
127 slat (usec): min=143, max=37779k, avg=1840.99, stdev=173080.88
128 clat (usec): min=3, max=37791k, avg=14694.66, stdev=426993.13
129 lat (usec): min=776, max=37795k, avg=16536.17, stdev=460779.24
130 clat percentiles (msec):
131 | 1.00th=[ 5], 5.00th=[ 6], 10.00th=[ 7], 20.00th=[ 7],
132 | 30.00th=[ 8], 40.00th=[ 9], 50.00th=[ 9], 60.00th=[ 10],
133 | 70.00th=[ 10], 80.00th=[ 11], 90.00th=[ 12], 95.00th=[ 13],
134 | 99.00th=[ 16], 99.50th=[ 17], 99.90th=[ 24], 99.95th=[ 30],
135 | 99.99th=[17113]
136 bw ( KiB/s): min= 512, max=225280, per=41.39%, avg=173373.41, stdev=27446.78, samples=1087
137 iops : min= 2, max= 880, avg=677.22, stdev=107.21, samples=1087
138 lat (usec) : 4=0.01%, 1000=0.01%
139 lat (msec) : 2=0.01%, 4=0.55%, 10=74.24%, 20=25.02%, 50=0.15%
140 lat (msec) : 250=0.01%, 750=0.01%, 1000=0.01%
141 cpu : usr=0.80%, sys=27.74%, ctx=1881579, majf=0, minf=14
142 IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=100.0%, 32=0.0%, >=64=0.0%
143 submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
144 complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
145 issued rwts: total=552137,368114,0,0 short=0,0,0,0 dropped=0,0,0,0
146 latency : target=0, window=0, percentile=100.00%, depth=16
147
148 Run status group 0 (all jobs):
149 READ: bw=612MiB/s (642MB/s), 153MiB/s-153MiB/s (160MB/s-161MB/s), io=538GiB (578GB), run=900001-900001msec
150 WRITE: bw=409MiB/s (429MB/s), 102MiB/s-102MiB/s (107MB/s-107MB/s), io=360GiB (386GB), run=900001-900001msec
151 fio /usr/share/doc/fio/examples/fio-seq-RW.fio 33,09s user 1006,59s system 106% cpu 16:14,10 total
For easier comparision
1 Run status group 0 (all jobs):
2 READ: bw=24.5MiB/s (25.7MB/s), 6207KiB/s-6339KiB/s (6356kB/s-6491kB/s), io=21.6GiB (23.2GB), run=900029-900030msec
3 WRITE: bw=16.4MiB/s (17.2MB/s), 4203KiB/s-4211KiB/s (4303kB/s-4312kB/s), io=14.4GiB (15.5GB), run=900029-900030msec
4 fio /usr/share/doc/fio/examples/fio-seq-RW.fio 7,62s user 125,55s system 10% cpu 21:30,52 total
The difference is simply tremendous. Therefor my recommendation use virtiofs instead of 9p.
Cleanup
Virtiofs additional options
There are some additional options that may be passed via xml please see
man 1 virtiofsd
/usr/share/libvirt/schemas/domain.rng
Here is a example configuration
1 <filesystem type='mount' accessmode='passthrough'>
2 <driver type='virtiofs' queue='1024'/>
3 <!--<binary path='/usr/libexec/virtiofsd' xattr='on'>-->
4 <binary path='/usr/lib/qemu/virtiofsd' xattr='on'>
5 <cache mode='always'/>
6 <lock posix='on' flock='on'/>
7 </binary>
8 <source dir='/media/space/backup/backup1'/>
9 <target dir='bareos_storage_virtiofs'/>
10 <address type='pci' domain='0x0000' bus='0x08' slot='0x00' function='0x0'/>
11 </filesystem>
Installation
3rd party repo
Debian packages are quite outdated even in Sid, just use the bareos repo.
/etc/apt/sources.list.d/bareos.list
Import the public key to proof the signature.
The public key for Bareos 20 has not yet been uploaded to the pgp infrastructure. so i recommend running some checks on the key.
1 LANG=C apt update 2>&1 | grep -E '^(E|W): '
2 W: GPG error: http://download.bareos.org/bareos/release/20/Debian_10 InRelease: The following signatures couldn't be verified because the public key is not available: NO_PUBKEY 0CCBA537DBE083A6
3 E: The repository 'http://download.bareos.org/bareos/release/20/Debian_10 InRelease' is not signed.
4
5 wget -qO /tmp/bareos_release.key \
6 'https://download.bareos.org/bareos/release/20/Debian_10/Release.key'
7 gpg --keyid-format LONG --show-keys < /tmp/bareos_release.key
8 pub rsa4096 2020-12-11 [SC]
9 C68B001F74D2F20243D0B7A20CCBA537DBE083A6
10 uid Bareos 20 Signing Key <signing@bareos.com
11 apt-key add /tmp/bareos_release.key
12 apt update
Install bareos
Package "bareos" pulls in all dependencies. As a base OS i suggest Debian 10 Codename "Buster". Latest version in Debian is currently 17.2.7, but on Github 18.5 is in the works.
1 aptitude install bareos
Configure
Bareos Director-Daemon
Ressource Director (self)
/etc/bareos/bareos-dir.d/director/bareos-dir.conf
1 Director { # define myself
2 Name = bareos-dir
3 QueryFile = "/usr/lib/bareos/scripts/query.sql"
4 Maximum Concurrent Jobs = 10
5 Password = "very_log_director_password" # Console password
6 Messages = Daemon
7 Auditing = yes
8
9 # Enable the Heartbeat if you experience connection losses
10 # (eg. because of your router or firewall configuration).
11 # Additionally the Heartbeat can be enabled in bareos-sd and bareos-fd.
12 #
13 # Heartbeat Interval = 1 min
14
15 # remove comment in next line to load dynamic backends from specified directory
16 # Backend Directory = /usr/lib/bareos/backends
17
18 # remove comment from "Plugin Directory" to load plugins from specified directory.
19 # if "Plugin Names" is defined, only the specified plugins will be loaded,
20 # otherwise all director plugins (*-dir.so) from the "Plugin Directory".
21 #
22 # Plugin Directory = /usr/lib/bareos/plugins
23 # Plugin Names = ""
24
25 ### Collect and keep statistics
26 ### This enabled by default in the director resource,
27 ### but must be enabled manually in the storage resource.
28 Statistics Collect Interval = 150 ### DEFAULT: 150
29 Statistics Retention = 160704000 ### Defaults: 160704000 (5y)
30 }
Filesets
LinuxBase
/etc/bareos/bareos-dir.d/fileset/LinuxBase.conf
1 FileSet {
2 Name = "LinuxBase"
3 Description = "Backup a minimal set of a Linux Server."
4 Enable VSS = false
5
6 Include {
7 Options {
8 Compression = GZIP
9 Signature = SHA256
10 One FS = yes
11 noatime = yes
12 Sparse = yes
13 IgnoreCase = no
14 Wildfile = /boot/config-*
15 }
16 Options {
17 exclude = yes
18 Regexdir = "/home/[^/]+/.steam"
19 Regexdir = "/home/[^/]+/.local/share/Trash/files"
20 Regexdir = "/home/[^/]+/Downloads"
21 }
22 Exclude Dir Containing = .nobackup
23 File = /etc
24 File = /home
25 File = /opt
26 File = /root
27 File = /var/lib
28 }
29
30 Include {
31 Options {
32 Wildfile = /boot/config-*
33 }
34 Options {
35 Exclude = yes
36 RegexFile = ".*"
37 }
38 File = /boot
39 }
40
41 Exclude {
42 }
43 }
Windows10AllMedia
Gather all media files of every user.
/etc/bareos/bareos-dir.d/fileset/Windows10AllMedia.conf
1 FileSet {
2 Name = "Windows10AllMedia"
3
4 Include {
5
6 File = "C:/Users"
7
8 Options {
9 signature = SHA256
10 verify = s1
11 IgnoreCase = yes
12
13 # Include all users’ directories so we reach the inner ones. Unlike a
14 # WildDir pattern ending in *, this RegExDir only matches the top-level
15 # directories and not any inner ones.
16 RegExDir = "^C:/Users/[^/]+$"
17
18 # Ditto all users’ Media directories.
19 WildDir = "C:/Users/*/Bibliothek"
20 WildDir = "C:/Users/*/Documents"
21 WildDir = "C:/Users/*/Music"
22 WildDir = "C:/Users/*/Pictures"
23 WildDir = "C:/Users/*/Videos"
24
25 # Include the contents of the Media directories and any subdirectories.
26 Wild = "C:/Users/*/Bibliothek/*"
27 Wild = "C:/Users/*/Documents/*"
28 Wild = "C:/Users/*/Music/*"
29 Wild = "C:/Users/*/Pictures/*"
30 Wild = "C:/Users/*/Videos/*"
31 }
32
33 Options {
34 Exclude = yes
35 IgnoreCase = yes
36
37 # Exclude everything else, in particular any files at the top level and
38 # any other directories or files in the users’ directories.
39 Wild = "C:/Users/*"
40 }
41 }
42 }
Job Defaults (JobDefs)
Tox simplify configuration of a job, sets of defaults for jobs can be created, which than may be referenced during the creation of a job with jobdef="DefaultJob-Name"
/etc/bareos/bareos-dir.d/jobdefs/DefaultJob-Linux.conf
1 JobDefs {
2 Name = "DefaultJob-Linux"
3 Type = Backup
4 Level = Incremental
5 Messages = "Standard"
6 Storage = "File"
7 Pool = "Incremental"
8 FullBackupPool = "Full"
9 IncrementalBackupPool = "Incremental"
10 DifferentialBackupPool = "Differential"
11 Client = "bareos-fd"
12 FileSet = "LinuxBase"
13 Schedule = "WeeklyCycle"
14 WriteBootstrap = "/var/lib/bareos/%c.bsr"
15 runscript {
16 command = "/usr/local/lib/bareos/scripts/bareos-run_script.sh before"
17 runswhen = before
18 failonerror = No
19 }
20 runscript {
21 command = "/usr/local/lib/bareos/scripts/bareos-run_script.sh after"
22 runswhen = after
23 failonerror = No
24 }
25 }
/etc/bareos/bareos-dir.d/jobdefs/DefaultJob-Windows10.conf
1 JobDefs {
2 Name = "DefaultJob-Windows10"
3 Type = Backup
4 Level = Incremental
5 Messages = "Standard"
6 Storage = "File"
7 Pool = "Incremental"
8 FullBackupPool = "Full"
9 IncrementalBackupPool = "Incremental"
10 DifferentialBackupPool = "Differential"
11 Client = "muckie-pc-fd"
12 FileSet = "Windows10AllMedia"
13 Schedule = "WeeklyCycle"
14 WriteBootstrap = "/var/lib/bareos/%c.bsr"
15 }
Pre/Post Backup Scripts
Create a directory for custom scripts on the client
1 install -o root -g root -d /usr/local/lib/bareos/scripts
A little script to be used to run some tasks before and after the backup with some sensitivity of the distribution.
/usr/local/lib/bareos/scripts/bareos-run_script.sh
1 #!/bin/bash
2
3 APPEND_DATE=false
4
5 usage () {
6 cat <<-EOL
7 $(basename $0) [-d||--date] [--] [before||after]
8 EOL
9 }
10
11 # Note that we use "$@" to let each command-line parameter expand to a
12 # separate word. The quotes around "$@" are essential!
13 # We need TEMP as the 'eval set --' would nuke the return value of getopt.
14 TEMP=$(getopt -o 'd' --long 'date' -n "$(basename $0)" -- "$@")
15
16 if [ $? -ne 0 ]; then
17 echo 'Terminating...' >&2
18 exit 1
19 fi
20
21 # Note the quotes around "$TEMP": they are essential!
22 eval set -- "$TEMP"
23 unset TEMP
24
25 while true; do
26 case "$1" in
27 '-d'|'--date')
28 APPEND_DATE=true
29 shift
30 continue
31 ;;
32 #'-b'|'--b-long')
33 # echo "Option b, argument '$2'"
34 # shift 2
35 # continue
36 #;;
37 #'-c'|'--c-long')
38 # # c has an optional argument. As we are in quoted mode,
39 # # an empty parameter will be generated if its optional
40 # # argument is not found.
41 # case "$2" in
42 # '')
43 # echo 'Option c, no argument'
44 # ;;
45 # *)
46 # echo "Option c, argument '$2'"
47 # ;;
48 # esac
49 # shift 2
50 # continue
51 #;;
52 '--')
53 shift
54 break
55 ;;
56 *)
57 echo 'Internal error!' >&2
58 exit 1
59 ;;
60 esac
61 done
62
63 ### SANITIZE
64 if [ ${#@} -eq 0 ]; then
65 WHEN="before"
66 elif [ ${#@} -gt 1 ]; then
67 cat<<-EOL
68 Please decide between "before" and "after".
69 Default is before.
70 EOL
71 usage
72 exit 2
73 elif [ $1 == "before" ] || [ $1 == "after" ]; then
74 WHEN="$1"
75 else
76 echo "Unknown execution timepoint '$1'."
77 echo "Exiting…"
78 usage
79 exit 2
80 fi
81
82 FILE_SUFFIX="txt"
83 if $APPEND_DATE; then
84 DATE="$(date +%F)"
85 FILE_SUFFIX="_$DATE.$FILE_SUFFIX"
86 else
87 FILE_SUFFIX=".$FILE_SUFFIX"
88 fi
89
90 ISSUE="$(cat /etc/issue)"
91 if egrep -q '(Debian|Ubuntu)' <<< "$ISSUE"; then
92 OS="Debian"
93 fi
94
95 ### MAIN
96
97
98 if [ "$OS" == "Debian" ]; then
99 BACKUP_DIR="/var/backups/bareos"
100 FILE_SELECTIONS="dpkg-selections$FILE_SUFFIX"
101 FILE_LIST="dpkg-list$FILE_SUFFIX"
102 else
103 echo "Unsupported OS. Exiting…"
104 exit 1
105 fi
106
107 ### COMMAND LIST: AFTER
108 if [ "$WHEN" == "before" ]; then
109 [ -d "$BACKUP_DIR" ] || mkdir "$BACKUP_DIR"
110 dpkg --get-selections > "$BACKUP_DIR/$FILE_SELECTIONS"
111 dpkg -l > "$BACKUP_DIR/$FILE_LIST"
112
113 ### COMMAND LIST: AFTER
114 elif [ "$WHEN" == "after" ]; then
115 rm "$BACKUP_DIR/$FILE_SELECTIONS"
116 rm "$BACKUP_DIR/$FILE_LIST"
117
118 fi
119
120 echo "Script executed successfully $WHEN backup."
121 exit 0
Storage
There are some statistics that should come in handy if you want to estimate how long you backup will take or if e.g. you want to account and bill the backup storage.
/etc/bareos/bareos-dir.d/storage/File.conf
Pools
- In Maximum Volume Bytes it makes no difference if you write a capital "G" or a lowercase "g".
/etc/bareos/bareos-dir.d/pool/Differential.conf
1 Pool {
2 Name = Differential
3 Pool Type = Backup
4 Recycle = yes # Bareos can automatically recycle Volumes
5 AutoPrune = yes # Prune expired volumes
6 Volume Retention = 90 days # How long should the Differential Backups be kept? (#09)
7 Maximum Volume Bytes = 10G # Limit Volume size to something reasonable
8 Maximum Volumes = 100 # Limit number of Volumes in Pool
9 Label Format = "Differential-" # Volumes will be labeled "Differential-<volume-id>"
10 }
/etc/bareos/bareos-dir.d/pool/Full.conf
1 Pool {
2 Name = Full
3 Pool Type = Backup
4 Recycle = yes # Bareos can automatically recycle Volumes
5 AutoPrune = yes # Prune expired volumes
6 Volume Retention = 365 days # How long should the Full Backups be kept? (#06)
7 Maximum Volume Bytes = 50G # Limit Volume size to something reasonable
8 Maximum Volumes = 100 # Limit number of Volumes in Pool
9 Label Format = "Full-" # Volumes will be labeled "Full-<volume-id>"
10 }
/etc/bareos/bareos-dir.d/pool/Incremental.conf
1 Pool {
2 Name = Incremental
3 Pool Type = Backup
4 Recycle = yes # Bareos can automatically recycle Volumes
5 AutoPrune = yes # Prune expired volumes
6 Volume Retention = 30 days # How long should the Incremental Backups be kept? (#12)
7 Maximum Volume Bytes = 5g # Limit Volume size to something reasonable
8 Maximum Volumes = 200 # Limit number of Volumes in Pool
9 Label Format = "Incremental-" # Volumes will be labeled "Incremental-<volume-id>"
10 }
/etc/bareos/bareos-dir.d/pool/Scratch.conf
Trouble Shooting
Wrong Unix Permissions in Config Directories
bareos-director does not start if it can't read a config-file. So make sure not to create files that bareos can't read. This can easily happen if you copy a file with unix permission o-rwx as root, which changes owner.
It will definetely complain in the logs.
Okt 16 09:12:46 backup1 systemd[1]: Starting Bareos Director Daemon service... Okt 16 09:12:46 backup1 bareos-dir[1573]: 0bareos-dir: ERROR TERMINATION at parse_conf.c:198 Okt 16 09:12:46 backup1 bareos-dir[1573]: Config error: Cannot open config file "/etc/bareos/bareos-dir.d/*/*.conf": Keine Berechtigung Okt 16 09:12:46 backup1 bareos-dir[1574]: [55B blob data] Okt 16 09:12:46 backup1 bareos-dir[1574]: Config error: Cannot open config file "/etc/bareos/bareos-dir.d/*/*.conf": Keine Berechtigung Okt 16 09:12:46 backup1 systemd[1]: bareos-director.service: Can't open PID file /var/lib/bareos/bareos-dir.9101.pid (yet?) after start: No such file or directory Okt 16 09:12:46 backup1 systemd[1]: bareos-director.service: Failed with result 'protocol'.
To identify the file you can also use the following command:
1 find /etc/bareos -type f \! -user bareos \! -group bareos -exec ls -l {} \;
bconsole
Reload Configuration
Atfer changes to the configuration files, bareos config can be reloaded without restarting the service.
1 echo reload |bconsole
Add Client
Create a new password and hash it to md5:
On bareos server configure director:
Add Job
Add a job with bandwidth limit that does not exceed 54MBit/s WiFi (max. 6.75MB/s -> 4096KiB/s)
Label Mediums
Edit pool to hold more and eventually larger mediums
/etc/bareos/bareos-dir.d/pool/Incremental.conf
1 Pool {
2 Name = Incremental
3 Pool Type = Backup
4 Recycle = yes # Bareos can automatically recycle Volumes
5 AutoPrune = yes # Prune expired volumes
6 Volume Retention = 30 days # How long should the Incremental Backups be kept? (#12)
7 Maximum Volume Bytes = 4G # Limit Volume size to something reasonable
8 Maximum Volumes = 150 # Limit number of Volumes in Pool
9 Label Format = "Incremental-" # Volumes will be labeled "Incremental-<volume-id>"
10 }
Label
1 # bconsole
2 Connecting to Director localhost:9101
3 1000 OK: bareos-dir Version: 17.2.7 (16 Jul 2018)
4 Enter a period to cancel a command.
5 *reload
6 reloaded
7 *
8 *list volumes pool=Incremental
9 *
10 *label pool=Incremental storage=File volume=Incremental-0139
11 Connecting to Storage daemon File at backup1:9103 ...
12 Sending label command for Volume "Incremental-0139" Slot 0 ...
13 3000 OK label. VolBytes=204 Volume="Incremental-0139" Device="FileStorage" (/var/lib/bareos/storage)
14 Catalog record for Volume "Incremental-0139", Slot 0 successfully created.
15 Requesting to mount FileStorage ...
16 3001 OK mount requested. Device="FileStorage" (/var/lib/bareos/storage)
Job rescued.
Limit bandwidth
The limit is in KiB/s.
For example a 54Mbit/s WiFi can only transport 6912KiB/s and the networking gets close unusable. In this case it's a good idea to limit the backup to 4096KiB/s.
Reduce bandwidth to 4MiB/s
The maximum-bandwidth may be specified when adding the job.
A persistent configuration looks like this
/etc/bareos/bareos-dir.d/job/muckie-pc-job.conf
Database queries
Querieng the database you may achive some flexibility.
List all media that is
- in pool 'Incremental'
- smaller than 4GiB and
- not in state append or purged
- not used by a job
1 SELECT
2 p.name "poolname",
3 m.volumename,
4 m.volstatus,
5 m.maxvolbytes / POWER(2, 30) "Size [GiB]"
6 FROM media AS m
7 INNER JOIN pool AS p ON m.poolid = p.poolid
8 WHERE
9 p.name='Incremental'
10 AND m.maxvolbytes < 4 * POWER(2,30)
11 AND m.volstatus NOT IN ('Append','Purged')
12 AND NOT EXISTS(
13 SELECT 1 FROM jobmedia jm
14 WHERE jm.mediaid = m.mediaid
15 )
16 ORDER BY m.volumename ASC;
The above query in a slightly modified way, can be used as input for scripting. I determined some small files to be deleted.
1 sudo -u bareos -- /usr/bin/psql -qt <<EOF \
2 |sed '/^$/d' > /tmp/small_volumes.txt
3 SELECT
4 m.volumename
5 FROM media AS m
6 INNER JOIN pool AS p ON m.poolid = p.poolid
7 WHERE
8 p.name='Incremental'
9 AND m.maxvolbytes < 4 * POWER(2,30)
10 AND m.volstatus NOT IN ('Append','Purged')
11 ORDER BY m.volumename ASC;
12 EOF
Now use it as a input for a shell loop to perform various actions. 
You might also take only 5
I don't want this files to be recycled, so i can delete them later from the catalog and reclaim their space.
Bareos Storage-Daemon
Bareos Filesystem-Daemon
/etc/bareos/bareos-fd.d/client/myself.conf
1 Client {
2 Name = fuxtop-fd
3 Maximum Concurrent Jobs = 20
4
5 # remove comment from "Plugin Directory" to load plugins from specified directory.
6 # if "Plugin Names" is defined, only the specified plugins will be loaded,
7 # otherwise all storage plugins (*-fd.so) from the "Plugin Directory".
8 #
9 # Plugin Directory = /usr/lib/bareos/plugins
10 # Plugin Names = ""
11
12 # if compatible is set to yes, we are compatible with bacula
13 # if set to no, new bareos features are enabled which is the default
14 # compatible = yes
15 }
Connection From Client To Director
It's very useful for a mobile device to try a backup when you are actually @home. 
# TODO - Inititiate not only the connection, do the backup …
Some inspiration
On the client this the Director configuration needs to be changed to be able to initiate the connection by the client.
/etc/bareos/bareos-fd.d/director/bareos-dir.conf
1 Director {
2 Name = bareos-dir
3 #Password = "ezoh9EeGhiquahping5eihaayadohr5Ei"
4 Password = "[md5615b12ba593fe1a31d4e75c0b86e9aaa"
5 Description = "Allow the configured Director to access this file daemon."
6
7 # Allow connection from client to director
8 Connection From Client To Director = yes
9 # Director Network Address
10 Address = backup1.dungeon.rockstable.org
11 }
On the server this the Client configuration needs to be changed to be able to initiate the connection by the client.
/etc/bareos/bareos-fd.d/client/fuxtop-fd.conf
Tail the log-buffer and restart the daemon
Check the connection on the client
Caveats
bareos-fd.service has dependencies on
nss-lookup.target network.target remote-fs.target time-sync.target
It won't start successfully if any of these targets can't be reached. So make sure that e.g. any remote-mount (like nfs, smb, glusterfs, …) in /etc/fstab works under any condition. In doubt use autofs.
Bareos-Webui
http://backup1/bareos-webui/dashboard/
1 aptitude install bareos-webui
Redirect to https
/etc/apache2/sites-available/redirect_80.conf
1 <VirtualHost *:80>
2 # The ServerName directive sets the request scheme, hostname and port that
3 # the server uses to identify itself. This is used when creating
4 # redirection URLs. In the context of virtual hosts, the ServerName
5 # specifies what hostname must appear in the request's Host: header to
6 # match this virtual host. For the default virtual host (this file) this
7 # value is not decisive as it is used as a last resort host regardless.
8 # However, you must set it for any further virtual host explicitly.
9 Define SERVER_NAME backup1.dungeon.rockstable.org
10 ServerName ${SERVER_NAME}
11 ServerAlias backup1
12
13 DocumentRoot /var/www/html
14
15 # Available loglevels: trace8, ..., trace1, debug, info, notice, warn,
16 # error, crit, alert, emerg.
17 # It is also possible to configure the loglevel for particular
18 # modules, e.g.
19 #LogLevel info ssl:warn
20
21 ErrorLog ${APACHE_LOG_DIR}/${SERVER_NAME}_error.log
22 CustomLog ${APACHE_LOG_DIR}/${SERVER_NAME}_access.log combined
23
24 # For most configuration files from conf-available/, which are
25 # enabled or disabled at a global level, it is possible to
26 # include a line for only one particular virtual host. For example the
27 # following line enables the CGI configuration for this host only
28 # after it has been globally disabled with "a2disconf".
29 #Include conf-available/serve-cgi-bin.conf
30
31 ### Redirect anything up to the encrypted vHost
32 ### (breaks /server-status if certificate is invalid)
33 #RedirectMatch ^/(.*)$ https://${SERVER_NAME}/$1
34
35 ### Finally nothing leads around mod_rewrite
36 RewriteEngine on
37 RewriteCond %{REQUEST_URI} !=/server-status
38 RewriteRule ^/(.*)$ https://%{HTTP_HOST}/$1 [R=301,L]
39 </VirtualHost>
40
41 # vim: syntax=apache ts=4 sw=4 sts=4 sr noet
Bareos vHost
/etc/apache2/sites-available/bareos_443.conf
1 <IfModule mod_ssl.c>
2 <VirtualHost _default_:443>
3 ServerAdmin webmaster@rockstable.it
4
5 DocumentRoot /var/www/html
6
7 # Available loglevels: trace8, ..., trace1, debug, info, notice, warn,
8 # error, crit, alert, emerg.
9 # It is also possible to configure the loglevel for particular
10 # modules, e.g.
11 #LogLevel info ssl:warn
12
13 ErrorLog ${APACHE_LOG_DIR}/error.log
14 CustomLog ${APACHE_LOG_DIR}/access.log combined
15
16 # For most configuration files from conf-available/, which are
17 # enabled or disabled at a global level, it is possible to
18 # include a line for only one particular virtual host. For example the
19 # following line enables the CGI configuration for this host only
20 # after it has been globally disabled with "a2disconf".
21 #Include conf-available/serve-cgi-bin.conf
22
23 # SSL Engine Switch:
24 # Enable/Disable SSL for this virtual host.
25 SSLEngine on
26
27 # A self-signed (snakeoil) certificate can be created by installing
28 # the ssl-cert package. See
29 # /usr/share/doc/apache2/README.Debian.gz for more info.
30 # If both key and certificate are stored in the same file, only the
31 # SSLCertificateFile directive is needed.
32 SSLCertificateFile /etc/ssl/certs/ssl-cert-snakeoil.pem
33 SSLCertificateKeyFile /etc/ssl/private/ssl-cert-snakeoil.key
34
35 # Server Certificate Chain:
36 # Point SSLCertificateChainFile at a file containing the
37 # concatenation of PEM encoded CA certificates which form the
38 # certificate chain for the server certificate. Alternatively
39 # the referenced file can be the same as SSLCertificateFile
40 # when the CA certificates are directly appended to the server
41 # certificate for convinience.
42 #SSLCertificateChainFile /etc/apache2/ssl.crt/server-ca.crt
43
44 # Certificate Authority (CA):
45 # Set the CA certificate verification path where to find CA
46 # certificates for client authentication or alternatively one
47 # huge file containing all of them (file must be PEM encoded)
48 # Note: Inside SSLCACertificatePath you need hash symlinks
49 # to point to the certificate files. Use the provided
50 # Makefile to update the hash symlinks after changes.
51 #SSLCACertificatePath /etc/ssl/certs/
52 #SSLCACertificateFile /etc/apache2/ssl.crt/ca-bundle.crt
53
54 # Certificate Revocation Lists (CRL):
55 # Set the CA revocation path where to find CA CRLs for client
56 # authentication or alternatively one huge file containing all
57 # of them (file must be PEM encoded)
58 # Note: Inside SSLCARevocationPath you need hash symlinks
59 # to point to the certificate files. Use the provided
60 # Makefile to update the hash symlinks after changes.
61 #SSLCARevocationPath /etc/apache2/ssl.crl/
62 #SSLCARevocationFile /etc/apache2/ssl.crl/ca-bundle.crl
63
64 # Client Authentication (Type):
65 # Client certificate verification type and depth. Types are
66 # none, optional, require and optional_no_ca. Depth is a
67 # number which specifies how deeply to verify the certificate
68 # issuer chain before deciding the certificate is not valid.
69 #SSLVerifyClient require
70 #SSLVerifyDepth 10
71
72 # SSL Engine Options:
73 # Set various options for the SSL engine.
74 # o FakeBasicAuth:
75 # Translate the client X.509 into a Basic Authorisation. This means that
76 # the standard Auth/DBMAuth methods can be used for access control. The
77 # user name is the `one line' version of the client's X.509 certificate.
78 # Note that no password is obtained from the user. Every entry in the user
79 # file needs this password: `xxj31ZMTZzkVA'.
80 # o ExportCertData:
81 # This exports two additional environment variables: SSL_CLIENT_CERT and
82 # SSL_SERVER_CERT. These contain the PEM-encoded certificates of the
83 # server (always existing) and the client (only existing when client
84 # authentication is used). This can be used to import the certificates
85 # into CGI scripts.
86 # o StdEnvVars:
87 # This exports the standard SSL/TLS related `SSL_*' environment variables.
88 # Per default this exportation is switched off for performance reasons,
89 # because the extraction step is an expensive operation and is usually
90 # useless for serving static content. So one usually enables the
91 # exportation for CGI and SSI requests only.
92 # o OptRenegotiate:
93 # This enables optimized SSL connection renegotiation handling when SSL
94 # directives are used in per-directory context.
95 #SSLOptions +FakeBasicAuth +ExportCertData +StrictRequire
96 <FilesMatch "\.(cgi|shtml|phtml|php)$">
97 SSLOptions +StdEnvVars
98 </FilesMatch>
99 <Directory /usr/lib/cgi-bin>
100 SSLOptions +StdEnvVars
101 </Directory>
102
103 # SSL Protocol Adjustments:
104 # The safe and default but still SSL/TLS standard compliant shutdown
105 # approach is that mod_ssl sends the close notify alert but doesn't wait for
106 # the close notify alert from client. When you need a different shutdown
107 # approach you can use one of the following variables:
108 # o ssl-unclean-shutdown:
109 # This forces an unclean shutdown when the connection is closed, i.e. no
110 # SSL close notify alert is send or allowed to received. This violates
111 # the SSL/TLS standard but is needed for some brain-dead browsers. Use
112 # this when you receive I/O errors because of the standard approach where
113 # mod_ssl sends the close notify alert.
114 # o ssl-accurate-shutdown:
115 # This forces an accurate shutdown when the connection is closed, i.e. a
116 # SSL close notify alert is send and mod_ssl waits for the close notify
117 # alert of the client. This is 100% SSL/TLS standard compliant, but in
118 # practice often causes hanging connections with brain-dead browsers. Use
119 # this only for browsers where you know that their SSL implementation
120 # works correctly.
121 # Notice: Most problems of broken clients are also related to the HTTP
122 # keep-alive facility, so you usually additionally want to disable
123 # keep-alive for those clients, too. Use variable "nokeepalive" for this.
124 # Similarly, one has to force some clients to use HTTP/1.0 to workaround
125 # their broken HTTP/1.1 implementation. Use variables "downgrade-1.0" and
126 # "force-response-1.0" for this.
127 # BrowserMatch "MSIE [2-6]" \
128 # nokeepalive ssl-unclean-shutdown \
129 # downgrade-1.0 force-response-1.0
130
131 Include conf-available/bareos-webui.conf
132 RedirectMatch "^/$" "/bareos-webui"
133 </VirtualHost>
134 </IfModule>
135
136 # vim: syntax=apache ts=4 sw=4 sts=4 sr noet
Http2 support
Apache2 mod_http2 does not support mod_mpm_prefork. So we switch to php-fpm, mod_proxy_fcgi and mod_mpm_event.
Trouble Shooting
Uninitialized variable "extras"
mod_proxy_fcgi throws an error with php7.3+, regarding a undefined variable "extras".
[Mon Oct 14 12:54:19.526596 2019] [proxy_fcgi:error] [pid 842:tid 139696600672000] [client 192.168.182.13:40176] AH01071: Got error 'PHP message: PHP Notice: compact(): Undefined variable: extras in /usr/share/bareos-webui/vendor/zendframework/zend-view/src/Helper/HeadLink.php on line 403PHP message: PHP Notice: compact(): Undefined variable: extras in /usr/share/bareos-webui/vendor/zendframework/zend-view/src/Helper/HeadLink.php on line 403'
So i applied a little patch to /usr/share/bareos-webui/vendor/zendframework/zend-view/src/Helper/HeadLink.php which initializes extras to empty string "" if (0 < count($args) && is_array($args[0])
HeadLink.php.patch
1 /**
2 * Create item for stylesheet link item
3 *
4 * @param array $args
5 * @return stdClass|false Returns false if stylesheet is a duplicate
6 */
7 public function createDataStylesheet(array $args)
8 {
9 $rel = 'stylesheet';
10 $type = 'text/css';
11 $media = 'screen';
12 $conditionalStylesheet = false;
13 $href = array_shift($args);
14 $extras = "";
15
16 if ($this->isDuplicateStylesheet($href)) {
17 return false;
18 }
19
20 if (0 < count($args)) {
21 $media = array_shift($args);
22 if (is_array($media)) {
23 $media = implode(',', $media);
24 } else {
25 $media = (string) $media;
26 }
27 }
28 if (0 < count($args)) {
29 $conditionalStylesheet = array_shift($args);
30 if (!empty($conditionalStylesheet) && is_string($conditionalStylesheet)) {
31 $conditionalStylesheet = (string) $conditionalStylesheet;
32 } else {
33 $conditionalStylesheet = null;
34 }
35 }
36
37 if (0 < count($args) && is_array($args[0])) {
38 $extras = array_shift($args);
39 $extras = (array) $extras;
40 } else {
41 $extras = (array) $extras;
42 }
43
44 $attributes = compact('rel', 'type', 'href', 'media', 'conditionalStylesheet', 'extras');
45
46 return $this->createData($attributes);
47 }
I'm not a programmer per definition - but works, so far.
"continue" targeting switch is equivalent to "break"
https://www.php.net/manual/de/control-structures.continue.php
Error is in:
/usr/share/bareos-webui/vendor/zendframework/zend-stdlib/src/ArrayObject.php
Create a patch file
ArrayObject.php.patch
fwrite(): send of 26 bytes failed with errno=104
Notice: fwrite(): send of 26 bytes failed with errno=104 Connection reset by peer in /usr/share/bareos-webui/vendor/Bareos/library/Bareos/BSock/BareosBSock.php on line 219
Apply both solutions:
BareosBSock.php.patch
1 538c538,548
2 < $crypto_method = STREAM_CRYPTO_METHOD_TLSv1_2_CLIENT;
3 ---
4 > /*
5 > * STREAM_CRYPTO_METHOD_TLSv1_2_CLIENT was introduced with PHP version
6 > * 5.6.0. We need to care that calling stream_socket_enable_crypto method
7 > * works with versions < 5.6.0 as well.
8 > */
9 > $crypto_method = STREAM_CRYPTO_METHOD_TLS_CLIENT;
10 >
11 > if (defined('STREAM_CRYPTO_METHOD_TLSv1_2_CLIENT')) {
12 > $crypto_method |= STREAM_CRYPTO_METHOD_TLSv1_2_CLIENT;
13 > }
14 >
Disable TLS in
/etc/bareos/bareos-dir.d/console/admin.conf
BConsole webview to small
Bconsole width and height are limited to 1080x650 but can be adjusted.
Adjust bconsole size /usr/share/bareos-webui/module/Director/view/director/director/console.phtml
1 <pre class="bconsole" id="bconsole" style="font-size: 9pt; padding-left: 5px; overflow: scroll; width: 98vw; height: 80vh; background: #000; color: #fff;" onclick="focusScreen();">
2 <p><?php echo $this->translate("bconsole (batch-mode), please handle with care."); ?><br /><?php echo $this->translate("All commands have to be a one liner, dialogs are not working."); ?><br /><?php echo $this->translate("Type help for a list of commands."); ?></p>
3 <div class="prompt"><label> * </label><input type="text" class="cli" id="cli" value ="" size=120 style="display: inline; background: #000; color: #fff; border-color: #000; border-style: hidden; outline: none;"/></div>
4 </pre>
console.phtml.patch
1 41c41
2 < <pre class="bconsole" id="bconsole" style="font-size: 9pt; padding-left: 5px; overflow: scroll; width: 1080px; height: 650px; background: #000; color: #fff;" onclick="focusScreen();">
3 ---
4 > <pre class="bconsole" id="bconsole" style="font-size: 9pt; padding-left: 5px; overflow: scroll; width: 98vw; height: 80vh; background: #000; color: #fff;" onclick="focusScreen();">
Bareos Tray Monitor
Does not start correctly, when it is called from application menu, becasue it is called with a configuration file, that does not exist.
A strace told me, that bareos tray monitor tries to read this files by default.
So we can just leave the option -c and its argument away. Just make a copy of the entry in the application launcher and change the exec call to /usr/bin/bareos-tray-monitor. You may now start this with your GUI-Session.