libvirt

Contents

libvirt

About

Libvirt

KVM

The libvirt project

is a toolkit to manage virtualization platforms
is accessible from C, Python, Perl, Java and more
is licensed under open source licenses
supports KVM, QEMU, Xen, Virtuozzo, VMWare ESX, LXC, BHyve and more
targets Linux, FreeBSD, Windows and OS-X
is used by many applications

Clients

virt-manager
virsh
cockpit

Installation

Install hypervisor utilities

   1 aptitude install \
   2         libvirt-daemon-system ksmtuned \
   3         virt-manager virt-top virt-what python3-guestfs guestfs-tools

Configure

Networking

Prepare the definition files

Configuring libvirt networks in virt-manager is very limited, so we define the networks it in virsh.

Use xmllint to check the files, it'll save your nerves.

   1 xmllint
   2 while read -r FILE; do
   3         echo -e '\n'"$FILE"
   4         xmllint "$FILE" || break
   5 done < <(find /root/libvirt/networks -maxdepth 1 -type f)

This is the definition file of a bare isolated bridge
/root/libvirt/networks/ovs-iso.xml

   1 <network>
   2   <name>ovs-iso</name>
   3   <forward mode='bridge'/>
   4   <bridge name='ovs-iso'/>
   5   <virtualport type='openvswitch'/>
   6 </network>

Fake bridges are handled the same way.
/root/libvirt/networks/net_pub1.xml

   1 <network>
   2   <name>ovs-iso</name>
   3   <forward mode='bridge'/>
   4   <bridge name='net_pub1'/>
   5   <virtualport type='openvswitch'/>
   6 </network>

Or a somewhat more complex definition, with portgroups and tagged vlans.
/root/libvirt/networks/net_ovs-virt.xml

   1 <network>
   2   <name>net_ovs-virt</name>
   3   <forward mode='bridge'/>
   4   <bridge name='ovs-virt'/>
   5   <virtualport type='openvswitch'/>
   6   <portgroup name="trunk_internal">
   7     <vlan trunk="yes">
   8       <tag id='1000'/>
   9       <tag id='1500'/>
  10       <tag id='2000'/>
  11       <tag id='2500'/>
  12       <tag id='3000'/>
  13     </vlan>
  14   </portgroup>
  15   <portgroup name="trf1">
  16     <vlan>
  17       <tag id='100'/>
  18     </vlan>
  19   </portgroup>
  20   <portgroup name="pub1">
  21     <vlan>
  22       <tag id='500'/>
  23     </vlan>
  24   </portgroup>
  25   <portgroup name="pub2">
  26     <vlan>
  27       <tag id='501'/>
  28     </vlan>
  29   </portgroup>
  30   <portgroup name="lan_1a">
  31     <vlan>
  32       <tag id='1000'/>
  33     </vlan>
  34   </portgroup>
  35   <portgroup name="lan_1n">
  36     <vlan>
  37       <tag id='1500'/>
  38     </vlan>
  39   </portgroup>
  40   <portgroup name="lan_2a">
  41     <vlan>
  42       <tag id='2000'/>
  43     </vlan>
  44   </portgroup>
  45   <portgroup name="lan_2n">
  46     <vlan>
  47       <tag id='2500'/>
  48     </vlan>
  49   </portgroup>
  50   <portgroup name="lan_mon">
  51     <vlan>
  52       <tag id='3000'/>
  53     </vlan>
  54   </portgroup>
  55 </network>

If the keyword trunk=yes is not given in the top-level vlan tag and only a single vlan is given, the port is handled as an access port.
For using port groups in virt-manager, virt-manager 4.0+ needs to be installed. ;-D

Please see

Volatile creation

   1 root@infinitas ~ # virsh net-create net_ovs-iso.xml
   2 Network ovs-iso created from net_ovs-iso.xml
   3 root@infinitas ~ # virsh 
   4 Welcome to virsh, the virtualization interactive terminal.
   5 
   6 Type:  'help' for help with commands
   7        'quit' to quit
   8 
   9 virsh # net-list --all
  10  Name                 State      Autostart     Persistent
  11 ----------------------------------------------------------
  12  default              inactive   no            yes
  13  ovs-iso              active     no            no

Persistent definition

   1 root@infinitas ~ # virsh
   2 Welcome to virsh, the virtualization interactive terminal.
   3 
   4 Type:  'help' for help with commands
   5        'quit' to quit
   6 
   7 virsh # net-list --all
   8  Name                 State      Autostart     Persistent
   9 ----------------------------------------------------------
  10  default              inactive   no            yes
  11 
  12 virsh # net-define /root/net_ovs-iso.xml 
  13 Network ovs-iso defined from /root/net_ovs-iso.xml
  14 
  15 virsh # net-list --all
  16  Name                 State      Autostart     Persistent
  17 ----------------------------------------------------------
  18  default              inactive   no            yes
  19  ovs-iso              inactive   no            yes
  20 
  21 virsh # net-autostart ovs-iso
  22 Network ovs-iso marked as autostarted
  23 
  24 virsh # net-list --all
  25  Name                 State      Autostart     Persistent
  26 ----------------------------------------------------------
  27  default              inactive   no            yes
  28  ovs-iso              inactive   yes           yes
  29 
  30 virsh # net-start ovs-iso
  31 Network ovs-iso started
  32 
  33 virsh # net-list --all
  34  Name                 State      Autostart     Persistent
  35 ----------------------------------------------------------
  36  default              inactive   no            yes
  37  ovs-iso              active     yes           yes
  38 
  39 virsh # 
  40

Deleting a network

If the network is defined persistently first undefine, then destroy the network.

   1 virsh # net-list --all
   2  Name                 State      Autostart     Persistent
   3 ----------------------------------------------------------
   4  default              inactive   no            yes
   5  ovs-iso              inactive   yes           yes
   6 
   7 virsh # net-start ovs-iso
   8 Network ovs-iso started
   9 
  10 virsh # net-undefine ovs-iso
  11 Network ovs-iso has been undefined
  12 
  13 virsh # net-list --all
  14  Name                 State      Autostart     Persistent
  15 ----------------------------------------------------------
  16  default              inactive   no            yes
  17  ovs-iso              active     no            no
  18 
  19 virsh # net-destroy ovs-iso
  20 Network ovs-iso destroyed
  21 
  22 virsh # net-list --all
  23  Name                 State      Autostart     Persistent
  24 ----------------------------------------------------------
  25  default              inactive   no            yes
  26 
  27 virsh # 
  28

Configure a guest interface of openvSwitch

The guest is configured as a member of the virtual network ovs-iso. Here is the excerpt from the virtual machine definition.

   1 ...
   2     <interface type='network'>
   3       <mac address='52:54:00:49:35:80'/>
   4       <source network='ovs-iso'/>
   5       <model type='virtio'/>
   6       <address type='pci' domain='0x0000' bus='0x00' slot='0x03' function='0x0'/>
   7     </interface>
   8 ...

Static IP reservations via DHCP

Just edit your network and add add the <host … /> tags. {{{virsh net-edit --network default

   1 <network connections="1">
   2   <name>default</name>
   3   <uuid>55888abe-fde2-4ff5-812a-edb7968b915e</uuid>
   4   <forward mode="nat">
   5     <nat>
   6       <port start="1024" end="65535"/>
   7     </nat>
   8   </forward>
   9   <bridge name="virbr0" stp="on" delay="0"/>
  10   <mac address="52:54:00:53:18:d1"/>
  11   <ip address="192.168.122.1" netmask="255.255.255.0">
  12     <dhcp>
  13       <range start="192.168.122.2" end="192.168.122.254"/>
  14       <host mac="52:54:00:e7:12:f0" name="fmc1" ip="192.168.122.11"/>
  15       <host mac="52:54:00:e7:12:f1" name="fmc2" ip="192.168.122.12"/>
  16     </dhcp>
  17   </ip>
  18 </network>

Spice guest agents

The spice guest agent for Linux consists of 2 parts, a system wide daemon spice-vdagentd and a X11 session agent spice-vdagent of which there is one per X11 session.

spice-vdagentd gets started through a Sys-V initscript or a systemd unit.
spice-vdagent gets automatically started in desktop environments,
which honor /etc/xdg/autostart, and under gdm.

The spice guest agent adds the following features to spice Linux guests:

Client mouse mode (no need to grab mouse by client, no mouse lag)
Automatic adjustment of the X11 session's number of virtual monitors, and their resolution, to the number of client windows and their resolution
Support of copy and paste (text and images) between the active X11 session and the client, this supports both the primary selection and the clipboard
Support for transferring files from the client to the agent

spice-vdagentd

spice-vdagentd - Spice guest agent daemon

man 1 spice-vdagentd

Install spice-vdagentd in guest

   1 sudo apt install spice-vdagentd

The service
/usr/lib/systemd/system/spice-vdagentd.service

   1 [Unit]
   2 Description=Agent daemon for Spice guests
   3 Requires=spice-vdagentd.socket
   4 
   5 [Service]
   6 Type=forking
   7 EnvironmentFile=-/etc/default/spice-vdagentd
   8 ExecStart=/usr/sbin/spice-vdagentd $SPICE_VDAGENTD_EXTRA_ARGS
   9 PIDFile=/run/spice-vdagentd/spice-vdagentd.pid
  10 PrivateTmp=true
  11 Restart=on-failure
  12 
  13 [Install]
  14 Also=spice-vdagentd.socket

Is activated via a unix socket
/usr/lib/systemd/system/spice-vdagentd.socket

   1 [Unit]
   2 Description=Activation socket for spice guest agent daemon
   3 
   4 [Socket]
   5 ListenStream=/run/spice-vdagentd/spice-vdagent-sock

You may specify extra args in
/etc/default/spice-vdagentd

   1 #SPICE_VDAGENTD_EXTRA_ARGS=-d
   2

spice-vdagentd by default listens to VirtIO serial channel com.redhat.spice.0 port 2

So a VirtIO serial controller and the channel must be attached to the VM.

   1 <!-- VirtIO serial controller -->
   2 <controller type="virtio-serial" index="0">
   3   <alias name="virtio-serial0"/>
   4   <address type="pci" domain="0x0000" bus="0x03" slot="0x00" function="0x0"/>
   5 </controller>
   6 <!-- VirtIO serial channel -->
   7 <channel type="spicevmc">
   8   <target type="virtio" name="com.redhat.spice.0" state="disconnected"/>
   9   <alias name="channel1"/>
  10   <address type="virtio-serial" controller="0" bus="0" port="2"/>
  11 </channel>

You can find them in the VM
ll /dev/virtio-ports/*

   1 lrwxrwxrwx 1 root root 11  8. Mär 14:31 /dev/virtio-ports/com.redhat.spice.0 -> ../vport1p2
   2 lrwxrwxrwx 1 root root 11  8. Mär 14:31 /dev/virtio-ports/org.qemu.guest_agent.0 -> ../vport1p1

spice-vdagent

spice-vdagent - Spice guest agent X11 session agent
- man 1 spice-vdagent

spice-vdagent gets automatically started in desktop environments which honor /etc/xdg/autostart, and under gdm.

/etc/xdg/autostart/spice-vdagent.desktop

   1 [Desktop Entry]
   2 Name=Spice vdagent
   3 Comment=Agent for Spice guests
   4 Exec=/usr/bin/spice-vdagent
   5 Terminal=false
   6 Type=Application
   7 Categories=
   8 X-GNOME-Autostart-Phase=WindowManager
   9 NoDisplay=true

KDE Plasma does not honor /etc/xdg/autostart

/lib/systemd/user/spice-vdagent.service

   1 [Unit]
   2 Description=Spice guest session agent
   3 
   4 [Service]
   5 ExecStart=/usr/bin/spice-vdagent -x

Start spice guest agents on plasma

Both processes will be dead after login

   1 # systemctl status spice-vdagent.service 
   2 ○ spice-vdagentd.service - Agent daemon for Spice guests
   3      Loaded: loaded (/lib/systemd/system/spice-vdagentd.service; indirect; preset: enabled)
   4      Active: inactive (dead)
   5 TriggeredBy: ● spice-vdagentd.socket
   6 # systemctl --user status spice-vdagent.service
   7 ○ spice-vdagent.service - Spice guest session agent
   8      Loaded: loaded (/usr/lib/systemd/user/spice-vdagent.service; static)
   9      Active: inactive (dead)
  10 # ps faux| grep spice-vdagent
  11 tobias      4897  0.0  0.0   6700  2252 pts/4    S+   17:11   0:00              \_ grep --color=auto spice-vdagent

Create an override
/etc/systemd/user/plasma-core.target.d/override.conf

   1 [Unit]
   2 Wants=spice-vdagent.service

Reload systemd

   1 systemctl daemon-reload

As user take a look at the new target

   1 % systemctl --user cat plasma-core.target
   2 # /usr/lib/systemd/user/plasma-core.target
   3 [Unit]
   4 Description=KDE Plasma Workspace Core
   5 Wants=plasma-plasmashell.service plasma-kcminit.service plasma-kded.service plasma-kcminit-phase1.service graphical-session-pre.target
   6 Requires=plasma-ksmserver.service
   7 After=graphical-session-pre.target plasma-kwin_wayland.service
   8 RefuseManualStart=yes
   9 StopWhenUnneeded=true
  10 
  11 # /etc/systemd/user/plasma-core.target.d/override.conf
  12 [Unit]
  13 Wants=spice-vdagent.service

Log off from and back into a X11 session and spice-vdagent should be started.

   1 % ps faux| grep spice-vdagent
   2 tobias      7437  0.0  0.0   6700  2440 pts/4    S+   17:43   0:00              \_ grep --color=auto spice-vdagent
   3 tobias      6842  0.0  1.0 479016 83092 ?        Ssl  17:40   0:00  \_ /usr/bin/spice-vdagent -x
   4 root        6869  0.0  0.0  86420  4688 ?        Ssl  17:40   0:00 /usr/sbin/spice-vdagentd

Autoresize resolution

Only with VirtIO Video
- not with QXL
- Also with virgl enabled
  - but then in the Spice-Server listentyp must be "None". So there is not socket:port to connect on for virt-viewer. This effectively disables the usage of virt-viewer.
```
   1 <video>
   2   <model type="virtio" heads="1" primary="yes">
   3     <acceleration accel3d="yes"/>
   4   </model>
   5   <alias name="video0"/>
   6   <address type="pci" domain="0x0000" bus="0x00" slot="0x01" function="0x0"/>
   7 </video>
```
In virt-viewer the option virt-viewer --connect qemu:///system system-nameXY
- Burger menu in the upper right > Auto resize (Enable)
in virt-manager
- Enable Open VM > Burger Menu > View > Scale Display > Autoresize VM with windows

Spice USB redirection of a Yubikey

You may encounter an error during USB redirection of a Yubikey with spice: Device is in use by another application (0)

   1 % lsusb | grep 1050:0407
   2 Bus 001 Device 005: ID 1050:0407 Yubico.com Yubikey 4/5 OTP+U2F+CCID
   3 % ll /dev/bus/usb/001/004 
   4 crw-rw-r--+ 1 root root 189, 3  9. Mär 16:19 /dev/bus/usb/001/004
   5 % sudo fuser /dev/bus/usb/001/004
   6 /dev/bus/usb/001/004:  7320
   7 % ps faux | grep 7320                            
   8 tobias      7379  0.0  0.0  14216  7320 pts/9    Ss+  16:07   0:00  |   \_ /bin/zsh --login
   9 tobias     18449  0.0  0.0   6900  2240 pts/10   S+   16:20   0:00  |       \_ grep --color=auto 7320
  10 root        7320  0.0  0.0 617964  8064 ?        Ssl  16:07   0:00 /usr/sbin/pcscd --foreground --auto-exit
  11 % sudo systemctl stop pcscd.service 
  12 Stopping 'pcscd.service', but its triggering units are still active:
  13 pcscd.socket
  14 % sudo systemctl stop pcscd.socket

USB redirection should now work as expected.

Filesystem passthrough

9p

This worked quite well for some time, but with the appearance of #virtio-fs and the general availability of support in the tools, I definetly recommend to go with the new approach.

9p is slower
9p had problems during boot,
when mounted via /etc/fstab, the vm stuck in on the password prompt for the single user target.

Example domain xml

   1 <filesystem type="mount" accessmode="mapped">
   2   <driver type="path" wrpolicy="immediate"/>
   3   <source dir="/media/space/backup/backup1"/>
   4   <target dir="bareos_storage"/>
   5   <alias name="fs0"/>
   6   <address type="pci" domain="0x0000" bus="0x07" slot="0x00" function="0x0"/>
   7 </filesystem>

Mounting

/etc/fstab

   1 #<file_system>  <mount_point>            <type>  <options>                         <dump>  <pass>
   2 bareos_storage  /var/lib/bareos/storage  9p      trans=virtio,version=9p2000.L,rw  0       0

mount -t 9p

   1 bareos_storage on /var/lib/bareos/storage type 9p (rw,relatime,sync,dirsync,access=client,trans=virtio)

The driver will be exchanged with #virtio-fs when Linux 5.4 becomes stable and decent support is available in the tooling.

Fio test

To see the longish results, enable the comments.

Just a naive test.

   1 mkdir /var/lib/bareos/storage_9p/fio
   2 cd /var/lib/bareos/storage_9p/fio
   3 fio /usr/share/doc/fio/examples/fio-seq-RW.fio
   4 file1: (g=0): rw=rw, bs=(R) 256KiB-256KiB, (W) 256KiB-256KiB, (T) 256KiB-256KiB, ioengine=libaio, iodepth=16
   5 ...
   6 fio-3.12
   7 Starting 4 processes
   8 file1: Laying out IO file (1 file / 10240MiB)
   9 Jobs: 4 (f=4): [M(4)][100.0%][r=59.3MiB/s,w=39.8MiB/s][r=237,w=159 IOPS][eta 00m:00s]
  10 file1: (groupid=0, jobs=1): err= 0: pid=17725: Thu Jan 14 12:21:36 2021
  11   read: IOPS=24, BW=6324KiB/s (6476kB/s)(5559MiB/900030msec)
  12     slat (usec): min=2851, max=14972, avg=5514.29, stdev=1014.60
  13     clat (msec): min=64, max=3525, avg=360.71, stdev=304.31
  14      lat (msec): min=68, max=3529, avg=366.23, stdev=304.39
  15     clat percentiles (msec):
  16      |  1.00th=[   77],  5.00th=[   85], 10.00th=[   97], 20.00th=[  144],
  17      | 30.00th=[  186], 40.00th=[  228], 50.00th=[  271], 60.00th=[  326],
  18      | 70.00th=[  409], 80.00th=[  518], 90.00th=[  709], 95.00th=[  911],
  19      | 99.00th=[ 1552], 99.50th=[ 1821], 99.90th=[ 2702], 99.95th=[ 2937],
  20      | 99.99th=[ 3473]
  21    bw (  KiB/s): min=  510, max=34304, per=25.99%, avg=6531.75, stdev=4966.58, samples=1742
  22    iops        : min=    1, max=  134, avg=25.47, stdev=19.41, samples=1742
  23   write: IOPS=16, BW=4203KiB/s (4303kB/s)(3694MiB/900030msec); 0 zone resets
  24     slat (msec): min=4, max=2373, avg=52.60, stdev=76.79
  25     clat (usec): min=5, max=3472.5k, avg=370781.79, stdev=315624.33
  26      lat (msec): min=72, max=3577, avg=423.38, stdev=348.86
  27     clat percentiles (msec):
  28      |  1.00th=[   75],  5.00th=[   84], 10.00th=[   96], 20.00th=[  146],
  29      | 30.00th=[  190], 40.00th=[  234], 50.00th=[  279], 60.00th=[  338],
  30      | 70.00th=[  422], 80.00th=[  535], 90.00th=[  735], 95.00th=[  944],
  31      | 99.00th=[ 1620], 99.50th=[ 1921], 99.90th=[ 2903], 99.95th=[ 3138],
  32      | 99.99th=[ 3339]
  33    bw (  KiB/s): min=  510, max=24064, per=26.19%, avg=4406.25, stdev=3253.58, samples=1716
  34    iops        : min=    1, max=   94, avg=17.17, stdev=12.72, samples=1716
  35   lat (usec)   : 10=0.01%
  36   lat (msec)   : 100=10.91%, 250=34.10%, 500=32.98%, 750=13.00%, 1000=5.04%
  37   cpu          : usr=0.08%, sys=2.91%, ctx=2374272, majf=0, minf=12
  38   IO depths    : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=100.0%, 32=0.0%, >=64=0.0%
  39      submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
  40      complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
  41      issued rwts: total=22235,14775,0,0 short=0,0,0,0 dropped=0,0,0,0
  42      latency   : target=0, window=0, percentile=100.00%, depth=16
  43 file1: (groupid=0, jobs=1): err= 0: pid=17726: Thu Jan 14 12:21:36 2021
  44   read: IOPS=24, BW=6339KiB/s (6491kB/s)(5571MiB/900030msec)
  45     slat (usec): min=2584, max=11855, avg=5517.96, stdev=1015.75
  46     clat (msec): min=62, max=3945, avg=359.68, stdev=306.16
  47      lat (msec): min=67, max=3950, avg=365.20, stdev=306.24
  48     clat percentiles (msec):
  49      |  1.00th=[   75],  5.00th=[   83], 10.00th=[   93], 20.00th=[  142],
  50      | 30.00th=[  184], 40.00th=[  228], 50.00th=[  275], 60.00th=[  330],
  51      | 70.00th=[  405], 80.00th=[  518], 90.00th=[  709], 95.00th=[  902],
  52      | 99.00th=[ 1603], 99.50th=[ 1821], 99.90th=[ 2635], 99.95th=[ 3004],
  53      | 99.99th=[ 3876]
  54    bw (  KiB/s): min=  510, max=33280, per=26.02%, avg=6540.27, stdev=5022.02, samples=1743
  55    iops        : min=    1, max=  130, avg=25.43, stdev=19.63, samples=1743
  56   write: IOPS=16, BW=4211KiB/s (4312kB/s)(3701MiB/900030msec); 0 zone resets
  57     slat (msec): min=4, max=2370, avg=52.47, stdev=76.13
  58     clat (usec): min=13, max=3766.0k, avg=370292.61, stdev=320940.52
  59      lat (msec): min=70, max=4049, avg=422.76, stdev=354.82
  60     clat percentiles (msec):
  61      |  1.00th=[   75],  5.00th=[   84], 10.00th=[   94], 20.00th=[  146],
  62      | 30.00th=[  190], 40.00th=[  234], 50.00th=[  279], 60.00th=[  338],
  63      | 70.00th=[  418], 80.00th=[  535], 90.00th=[  726], 95.00th=[  919],
  64      | 99.00th=[ 1653], 99.50th=[ 2022], 99.90th=[ 2903], 99.95th=[ 3272],
  65      | 99.99th=[ 3675]
  66    bw (  KiB/s): min=  510, max=23504, per=26.16%, avg=4400.15, stdev=3193.90, samples=1722
  67    iops        : min=    1, max=   91, avg=17.07, stdev=12.48, samples=1722
  68   lat (usec)   : 20=0.01%
  69   lat (msec)   : 100=11.49%, 250=32.92%, 500=33.83%, 750=13.13%, 1000=4.83%
  70   cpu          : usr=0.08%, sys=2.90%, ctx=2379462, majf=0, minf=15                                                                                                                                                                                                                                 
  71   IO depths    : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=100.0%, 32=0.0%, >=64=0.0%
  72      submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
  73      complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
  74      issued rwts: total=22285,14805,0,0 short=0,0,0,0 dropped=0,0,0,0
  75      latency   : target=0, window=0, percentile=100.00%, depth=16
  76 file1: (groupid=0, jobs=1): err= 0: pid=17727: Thu Jan 14 12:21:36 2021
  77   read: IOPS=24, BW=6207KiB/s (6356kB/s)(5455MiB/900030msec)
  78     slat (usec): min=2971, max=18847, avg=5537.71, stdev=1016.32
  79     clat (msec): min=61, max=3428, avg=366.17, stdev=307.00
  80      lat (msec): min=66, max=3433, avg=371.71, stdev=307.07
  81     clat percentiles (msec):
  82      |  1.00th=[   78],  5.00th=[   86], 10.00th=[  100], 20.00th=[  148],
  83      | 30.00th=[  190], 40.00th=[  234], 50.00th=[  279], 60.00th=[  338],
  84      | 70.00th=[  418], 80.00th=[  531], 90.00th=[  709], 95.00th=[  911],
  85      | 99.00th=[ 1569], 99.50th=[ 1871], 99.90th=[ 2836], 99.95th=[ 3205],
  86      | 99.99th=[ 3339]
  87    bw (  KiB/s): min=  510, max=33792, per=25.53%, avg=6416.51, stdev=4803.03, samples=1740
  88    iops        : min=    1, max=  132, avg=24.94, stdev=18.77, samples=1740
  89   write: IOPS=16, BW=4204KiB/s (4305kB/s)(3695MiB/900030msec); 0 zone resets
  90     slat (msec): min=4, max=2442, avg=52.70, stdev=77.42
  91     clat (usec): min=9, max=3406.3k, avg=372761.24, stdev=322041.57
  92      lat (msec): min=71, max=3548, avg=425.46, stdev=355.36
  93     clat percentiles (msec):
  94      |  1.00th=[   78],  5.00th=[   86], 10.00th=[   97], 20.00th=[  148],
  95      | 30.00th=[  194], 40.00th=[  239], 50.00th=[  284], 60.00th=[  342],
  96      | 70.00th=[  422], 80.00th=[  535], 90.00th=[  718], 95.00th=[  927],
  97      | 99.00th=[ 1653], 99.50th=[ 2039], 99.90th=[ 3104], 99.95th=[ 3306],
  98      | 99.99th=[ 3373]
  99    bw (  KiB/s): min=  510, max=22528, per=26.13%, avg=4396.33, stdev=3237.55, samples=1720
 100    iops        : min=    1, max=   88, avg=17.05, stdev=12.66, samples=1720
 101   lat (usec)   : 10=0.01%
 102   lat (msec)   : 100=10.55%, 250=32.81%, 500=34.00%, 750=13.96%, 1000=4.75%
 103   cpu          : usr=0.08%, sys=2.95%, ctx=2347348, majf=0, minf=14
 104   IO depths    : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=100.0%, 32=0.0%, >=64=0.0%
 105      submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
 106      complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
 107      issued rwts: total=21821,14780,0,0 short=0,0,0,0 dropped=0,0,0,0
 108      latency   : target=0, window=0, percentile=100.00%, depth=16
 109 file1: (groupid=0, jobs=1): err= 0: pid=17728: Thu Jan 14 12:21:36 2021
 110   read: IOPS=24, BW=6263KiB/s (6413kB/s)(5505MiB/900029msec)
 111     slat (usec): min=2843, max=13630, avg=5508.24, stdev=1017.93
 112     clat (msec): min=64, max=3615, avg=363.82, stdev=310.11
 113      lat (msec): min=67, max=3619, avg=369.33, stdev=310.19
 114     clat percentiles (msec):
 115      |  1.00th=[   75],  5.00th=[   85], 10.00th=[   95], 20.00th=[  144],
 116      | 30.00th=[  186], 40.00th=[  230], 50.00th=[  275], 60.00th=[  334],
 117      | 70.00th=[  418], 80.00th=[  523], 90.00th=[  709], 95.00th=[  911],
 118      | 99.00th=[ 1603], 99.50th=[ 1905], 99.90th=[ 2769], 99.95th=[ 3138],
 119      | 99.99th=[ 3540]
 120    bw (  KiB/s): min=  510, max=34304, per=25.70%, avg=6457.68, stdev=4939.59, samples=1745
 121    iops        : min=    1, max=  134, avg=25.16, stdev=19.30, samples=1745
 122   write: IOPS=16, BW=4206KiB/s (4307kB/s)(3697MiB/900029msec); 0 zone resets
 123     slat (msec): min=4, max=2436, avg=52.64, stdev=77.04
 124     clat (usec): min=5, max=3420.2k, avg=371105.32, stdev=316602.38
 125      lat (msec): min=69, max=3621, avg=423.75, stdev=350.07
 126     clat percentiles (msec):
 127      |  1.00th=[   75],  5.00th=[   85], 10.00th=[   96], 20.00th=[  146],
 128      | 30.00th=[  190], 40.00th=[  236], 50.00th=[  288], 60.00th=[  347],
 129      | 70.00th=[  422], 80.00th=[  531], 90.00th=[  718], 95.00th=[  927],
 130      | 99.00th=[ 1670], 99.50th=[ 1989], 99.90th=[ 2836], 99.95th=[ 3071],
 131      | 99.99th=[ 3406]
 132    bw (  KiB/s): min=  510, max=24576, per=26.31%, avg=4426.71, stdev=3227.28, samples=1709
 133    iops        : min=    1, max=   96, avg=17.23, stdev=12.61, samples=1709
 134   lat (usec)   : 10=0.01%
 135   lat (msec)   : 100=11.49%, 250=32.57%, 500=33.66%, 750=13.50%, 1000=4.83%
 136   cpu          : usr=0.08%, sys=2.90%, ctx=2360901, majf=0, minf=13
 137   IO depths    : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=100.0%, 32=0.0%, >=64=0.0%
 138      submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
 139      complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
 140      issued rwts: total=22018,14788,0,0 short=0,0,0,0 dropped=0,0,0,0
 141      latency   : target=0, window=0, percentile=100.00%, depth=16
 142 
 143 Run status group 0 (all jobs):
 144    READ: bw=24.5MiB/s (25.7MB/s), 6207KiB/s-6339KiB/s (6356kB/s-6491kB/s), io=21.6GiB (23.2GB), run=900029-900030msec
 145   WRITE: bw=16.4MiB/s (17.2MB/s), 4203KiB/s-4211KiB/s (4303kB/s-4312kB/s), io=14.4GiB (15.5GB), run=900029-900030msec
 146 fio /usr/share/doc/fio/examples/fio-seq-RW.fio  7,62s user 125,55s system 10% cpu 21:30,52 total

Cleanup

   1 cd ..
   2 rm -r fio

virtio-fs

Requirements:

virtiofs support in the guest kernel (Linux v5.4 or later)
Libvirt 6.2.0+, better 6.9+
qemu 5.0.0
'virtiofs' requires shared memory

With shared memory (tmpfs)

I personally faour the tmpfs approach, because it does not set upper boundaries to the amount of hugepages. However by default tmpfs is also limited to half of the RAM.

/etc/libvirt/qemu.conf

   1 # This directory is used for memoryBacking source if configured as file.
   2 # NOTE: big files will be stored here
   3 #memory_backing_dir = "/var/lib/libvirt/qemu/ram"
   4 memory_backing_dir = "/dev/shm"

Restart libvirtd

   1 systemctl restart libvirtd.service

Example domain xml

   1 <domain type='kvm'>
   2   …
   3   <memory unit='>MiB'>4096</memory>
   4   <currentMemory unit='KiB'>2097152</currentMemory>
   5   <memoryBacking>
   6     <access mode='shared'/>
   7   </memoryBacking>
   8   …
   9   <cpu mode='host-model' check='partial'>
  10     <numa>
  11       <cell id='0' cpus='0-3' memory='4096' unit='MiB' memAccess='shared'/>
  12     </numa>
  13   </cpu>
  14   …
  15   <devices>
  16     …
  17     <filesystem type='mount' accessmode='passthrough'>
  18       <driver type='virtiofs'/>
  19       <source dir='/media/space/backup/backup1'/>
  20       <target dir='bareos_storage_virtiofs'/>
  21       <address type='pci' domain='0x0000' bus='0x08' slot='0x00' function='0x0'/>
  22     </filesystem>
  23     …
  24   </devices>
  25 </domain>

With explicit hugepages

The VM won't start, unless you allocated the full amount of RAM in hugepages. Preallocate hugepages to back the memory of the VM.

   1 virsh allocpages 2M 2048

Please see Linux#Hugepages for a persistent configuration.

Example domain xml (necessary)

   1 <domain type='kvm'>
   2   …
   3   <memory unit='MiB'>4096</memory>
   4   <currentMemory unit='MiB'>2048</currentMemory>
   5   <memoryBacking>
   6     <hugepages>
   7       <page size='2048' unit='KiB'/>
   8     </hugepages>
   9     <access mode='shared'/>
  10   </memoryBacking>
  11   …
  12   <cpu mode='host-model' check='partial'>
  13     <numa>
  14       <cell id='0' cpus='0-3' memory='4096' unit='MiB' memAccess='shared'/>
  15     </numa>
  16   </cpu>
  17   …
  18   <devices>
  19     …
  20     <filesystem type='mount' accessmode='passthrough'>
  21       <driver type='virtiofs'/>
  22       <source dir='/media/space/backup/backup1'/>
  23       <target dir='bareos_storage_virtiofs'/>
  24       <address type='pci' domain='0x0000' bus='0x08' slot='0x00' function='0x0'/>
  25     </filesystem>
  26     …
  27   </devices>
  28 </domain>

Mounting

Create the mountpoint in the VM

   1 install -o tobias:tobias /home/tobias/guest_host

/etc/fstab

   1 #<file_system>           <mount_point>            <type>    <options>  <dump>  <pass>
   2 bareos_storage_virtiofs  /var/lib/bareos/storage  virtiofs  rw         0       0
   3 #virtiofs_host_guest     /home/tobias/guest_host  virtiofs  rw         0       0
   4

mount -t virtiofs

   1 bareos_storage on /var/lib/bareos/storage type 9p (rw,relatime,sync,dirsync,access=client,trans=virtio)
   2 virtiofs_host_guest on /home/tobias/guest_host type virtiofs (rw,relatime)

Fio test

To see the longish results, enable the comments.

Just a naive test

   1 mkdir /var/lib/bareos/storage/fio
   2 cd /var/lib/bareos/storage/fio
   3 fio /usr/share/doc/fio/examples/fio-seq-RW.fio
   4 
   5 file1: (g=0): rw=rw, bs=(R) 256KiB-256KiB, (W) 256KiB-256KiB, (T) 256KiB-256KiB, ioengine=libaio, iodepth=16
   6 ...
   7 fio-3.12
   8 Starting 4 processes
   9 file1: Laying out IO file (1 file / 10240MiB)
  10 Jobs: 4 (f=4): [M(4)][100.0%][r=1069MiB/s,w=727MiB/s][r=4277,w=2906 IOPS][eta 00m:00s]
  11 file1: (groupid=0, jobs=1): err= 0: pid=13103: Thu Jan 14 11:43:20 2021
  12   read: IOPS=612, BW=153MiB/s (161MB/s)(135GiB/900001msec)
  13     slat (usec): min=43, max=853169, avg=393.69, stdev=1263.28
  14     clat (usec): min=1602, max=37792k, avg=14890.06, stdev=436567.44
  15      lat (usec): min=1717, max=37793k, avg=15284.17, stdev=436569.72
  16     clat percentiles (msec):
  17      |  1.00th=[    5],  5.00th=[    6], 10.00th=[    6], 20.00th=[    7],
  18      | 30.00th=[    8], 40.00th=[    8], 50.00th=[    9], 60.00th=[    9],
  19      | 70.00th=[   10], 80.00th=[   11], 90.00th=[   12], 95.00th=[   13],
  20      | 99.00th=[   16], 99.50th=[   17], 99.90th=[   25], 99.95th=[   30],
  21      | 99.99th=[17113]
  22    bw (  KiB/s): min=  512, max=353792, per=41.39%, avg=259549.94, stdev=43036.90, samples=1086
  23    iops        : min=    2, max= 1382, avg=1013.84, stdev=168.11, samples=1086
  24   write: IOPS=408, BW=102MiB/s (107MB/s)(89.8GiB/900001msec); 0 zone resets
  25     slat (usec): min=143, max=37780k, avg=1848.00, stdev=173203.68
  26     clat (usec): min=2, max=37793k, avg=14390.07, stdev=406456.91
  27      lat (usec): min=1080, max=37793k, avg=16238.60, stdev=441873.69
  28     clat percentiles (msec):
  29      |  1.00th=[    5],  5.00th=[    6], 10.00th=[    7], 20.00th=[    7],
  30      | 30.00th=[    8], 40.00th=[    9], 50.00th=[    9], 60.00th=[   10],
  31      | 70.00th=[   10], 80.00th=[   11], 90.00th=[   12], 95.00th=[   13],
  32      | 99.00th=[   16], 99.50th=[   17], 99.90th=[   24], 99.95th=[   30],
  33      | 99.99th=[17113]
  34    bw (  KiB/s): min=  512, max=225280, per=41.37%, avg=173290.20, stdev=27300.81, samples=1086
  35    iops        : min=    2, max=  880, avg=676.88, stdev=106.65, samples=1086
  36   lat (usec)   : 4=0.01%
  37   lat (msec)   : 2=0.01%, 4=0.55%, 10=74.15%, 20=25.09%, 50=0.16%
  38   lat (msec)   : 250=0.01%, 750=0.01%, 1000=0.01%
  39   cpu          : usr=0.80%, sys=27.88%, ctx=1953111, majf=0, minf=12
  40   IO depths    : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=100.0%, 32=0.0%, >=64=0.0%
  41      submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
  42      complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
  43      issued rwts: total=551075,368000,0,0 short=0,0,0,0 dropped=0,0,0,0
  44      latency   : target=0, window=0, percentile=100.00%, depth=16
  45 file1: (groupid=0, jobs=1): err= 0: pid=13104: Thu Jan 14 11:43:20 2021
  46   read: IOPS=611, BW=153MiB/s (160MB/s)(134GiB/900001msec)
  47     slat (usec): min=42, max=35260k, avg=455.56, stdev=47539.00
  48     clat (usec): min=2, max=37788k, avg=15216.77, stdev=447333.20
  49      lat (usec): min=243, max=37788k, avg=15672.74, stdev=449851.87
  50     clat percentiles (msec):
  51      |  1.00th=[    5],  5.00th=[    6], 10.00th=[    6], 20.00th=[    7],
  52      | 30.00th=[    8], 40.00th=[    8], 50.00th=[    9], 60.00th=[    9],
  53      | 70.00th=[   10], 80.00th=[   11], 90.00th=[   12], 95.00th=[   13],
  54      | 99.00th=[   16], 99.50th=[   17], 99.90th=[   24], 99.95th=[   31],
  55      | 99.99th=[17113]
  56    bw (  KiB/s): min= 1532, max=344910, per=41.32%, avg=259077.80, stdev=42335.00, samples=1086
  57    iops        : min=    5, max= 1347, avg=1011.96, stdev=165.36, samples=1086
  58   write: IOPS=409, BW=102MiB/s (107MB/s)(89.9GiB/900001msec); 0 zone resets
  59     slat (usec): min=148, max=37779k, avg=1755.64, stdev=163159.15
  60     clat (usec): min=245, max=37785k, avg=13931.32, stdev=389051.86
  61      lat (usec): min=503, max=37789k, avg=15687.48, stdev=421922.18
  62     clat percentiles (msec):
  63      |  1.00th=[    5],  5.00th=[    6], 10.00th=[    7], 20.00th=[    7],
  64      | 30.00th=[    8], 40.00th=[    9], 50.00th=[    9], 60.00th=[   10],
  65      | 70.00th=[   10], 80.00th=[   11], 90.00th=[   12], 95.00th=[   13],
  66      | 99.00th=[   16], 99.50th=[   17], 99.90th=[   24], 99.95th=[   29],
  67      | 99.99th=[17113]
  68    bw (  KiB/s): min=  510, max=220672, per=41.39%, avg=173362.11, stdev=27355.21, samples=1086
  69    iops        : min=    1, max=  862, avg=677.13, stdev=106.86, samples=1086
  70   lat (usec)   : 4=0.01%, 250=0.01%, 750=0.01%
  71   lat (msec)   : 2=0.01%, 4=0.55%, 10=74.03%, 20=25.21%, 50=0.17%
  72   lat (msec)   : 250=0.01%, 750=0.01%, 1000=0.01%
  73   cpu          : usr=0.84%, sys=27.71%, ctx=1963887, majf=0, minf=15
  74   IO depths    : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=100.0%, 32=0.0%, >=64=0.0%
  75      submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%                                                                                                                                                                                                                     
  76      complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
  77      issued rwts: total=550190,368159,0,0 short=0,0,0,0 dropped=0,0,0,0
  78      latency   : target=0, window=0, percentile=100.00%, depth=16
  79 file1: (groupid=0, jobs=1): err= 0: pid=13105: Thu Jan 14 11:43:20 2021
  80   read: IOPS=612, BW=153MiB/s (161MB/s)(135GiB/900001msec)
  81     slat (usec): min=44, max=7437.3k, avg=412.49, stdev=11059.38
  82     clat (usec): min=742, max=37792k, avg=14685.10, stdev=427440.19
  83      lat (usec): min=1247, max=37793k, avg=15098.00, stdev=427598.94
  84     clat percentiles (msec):
  85      |  1.00th=[    5],  5.00th=[    6], 10.00th=[    6], 20.00th=[    7],
  86      | 30.00th=[    8], 40.00th=[    8], 50.00th=[    9], 60.00th=[    9],
  87      | 70.00th=[   10], 80.00th=[   11], 90.00th=[   12], 95.00th=[   13],
  88      | 99.00th=[   16], 99.50th=[   17], 99.90th=[   24], 99.95th=[   30],
  89      | 99.99th=[17113]
  90    bw (  KiB/s): min=  510, max=362496, per=41.35%, avg=259291.17, stdev=43366.78, samples=1088
  91    iops        : min=    1, max= 1416, avg=1012.81, stdev=169.40, samples=1088
  92   write: IOPS=409, BW=102MiB/s (107MB/s)(89.9GiB/900001msec); 0 zone resets
  93     slat (usec): min=112, max=37779k, avg=1818.40, stdev=172556.40
  94     clat (usec): min=4, max=37794k, avg=14684.07, stdev=420477.17
  95      lat (usec): min=740, max=37795k, avg=16503.00, stdev=454556.31
  96     clat percentiles (msec):
  97      |  1.00th=[    5],  5.00th=[    6], 10.00th=[    7], 20.00th=[    7],
  98      | 30.00th=[    8], 40.00th=[    9], 50.00th=[    9], 60.00th=[   10],
  99      | 70.00th=[   10], 80.00th=[   11], 90.00th=[   12], 95.00th=[   13],
 100      | 99.00th=[   16], 99.50th=[   17], 99.90th=[   23], 99.95th=[   28],
 101      | 99.99th=[17113]
 102    bw (  KiB/s): min=  510, max=236544, per=41.37%, avg=173290.47, stdev=27898.89, samples=1088
 103    iops        : min=    1, max=  924, avg=676.87, stdev=108.99, samples=1088
 104   lat (usec)   : 10=0.01%, 750=0.01%
 105   lat (msec)   : 2=0.01%, 4=0.50%, 10=74.25%, 20=25.05%, 50=0.15%
 106   lat (msec)   : 250=0.01%, 750=0.01%, 1000=0.01%
 107   cpu          : usr=0.80%, sys=27.80%, ctx=1954852, majf=0, minf=14
 108   IO depths    : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=100.0%, 32=0.0%, >=64=0.0%
 109      submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
 110      complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
 111      issued rwts: total=551089,368308,0,0 short=0,0,0,0 dropped=0,0,0,0
 112      latency   : target=0, window=0, percentile=100.00%, depth=16
 113 file1: (groupid=0, jobs=1): err= 0: pid=13106: Thu Jan 14 11:43:20 2021
 114   read: IOPS=613, BW=153MiB/s (161MB/s)(135GiB/900001msec)
 115     slat (usec): min=41, max=3805.1k, avg=397.25, stdev=5198.37
 116     clat (usec): min=1265, max=37792k, avg=14655.43, stdev=422874.96
 117      lat (usec): min=1535, max=37792k, avg=15053.09, stdev=422908.42
 118     clat percentiles (msec):
 119      |  1.00th=[    5],  5.00th=[    6], 10.00th=[    6], 20.00th=[    7],
 120      | 30.00th=[    8], 40.00th=[    8], 50.00th=[    9], 60.00th=[    9],
 121      | 70.00th=[   10], 80.00th=[   11], 90.00th=[   12], 95.00th=[   13],
 122      | 99.00th=[   16], 99.50th=[   17], 99.90th=[   24], 99.95th=[   31],
 123      | 99.99th=[17113]
 124    bw (  KiB/s): min=  512, max=351232, per=41.43%, avg=259803.88, stdev=43436.51, samples=1088
 125    iops        : min=    2, max= 1372, avg=1014.84, stdev=169.67, samples=1088
 126   write: IOPS=409, BW=102MiB/s (107MB/s)(89.9GiB/900001msec); 0 zone resets
 127     slat (usec): min=143, max=37779k, avg=1840.99, stdev=173080.88
 128     clat (usec): min=3, max=37791k, avg=14694.66, stdev=426993.13
 129      lat (usec): min=776, max=37795k, avg=16536.17, stdev=460779.24
 130     clat percentiles (msec):
 131      |  1.00th=[    5],  5.00th=[    6], 10.00th=[    7], 20.00th=[    7],
 132      | 30.00th=[    8], 40.00th=[    9], 50.00th=[    9], 60.00th=[   10],
 133      | 70.00th=[   10], 80.00th=[   11], 90.00th=[   12], 95.00th=[   13],
 134      | 99.00th=[   16], 99.50th=[   17], 99.90th=[   24], 99.95th=[   30],
 135      | 99.99th=[17113]
 136    bw (  KiB/s): min=  512, max=225280, per=41.39%, avg=173373.41, stdev=27446.78, samples=1087
 137    iops        : min=    2, max=  880, avg=677.22, stdev=107.21, samples=1087
 138   lat (usec)   : 4=0.01%, 1000=0.01%
 139   lat (msec)   : 2=0.01%, 4=0.55%, 10=74.24%, 20=25.02%, 50=0.15%
 140   lat (msec)   : 250=0.01%, 750=0.01%, 1000=0.01%
 141   cpu          : usr=0.80%, sys=27.74%, ctx=1881579, majf=0, minf=14
 142   IO depths    : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=100.0%, 32=0.0%, >=64=0.0%
 143      submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
 144      complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
 145      issued rwts: total=552137,368114,0,0 short=0,0,0,0 dropped=0,0,0,0
 146      latency   : target=0, window=0, percentile=100.00%, depth=16
 147 
 148 Run status group 0 (all jobs):
 149    READ: bw=612MiB/s (642MB/s), 153MiB/s-153MiB/s (160MB/s-161MB/s), io=538GiB (578GB), run=900001-900001msec
 150   WRITE: bw=409MiB/s (429MB/s), 102MiB/s-102MiB/s (107MB/s-107MB/s), io=360GiB (386GB), run=900001-900001msec
 151 fio /usr/share/doc/fio/examples/fio-seq-RW.fio  33,09s user 1006,59s system 106% cpu 16:14,10 total

For easier comparision

   1 Run status group 0 (all jobs):
   2    READ: bw=24.5MiB/s (25.7MB/s), 6207KiB/s-6339KiB/s (6356kB/s-6491kB/s), io=21.6GiB (23.2GB), run=900029-900030msec
   3   WRITE: bw=16.4MiB/s (17.2MB/s), 4203KiB/s-4211KiB/s (4303kB/s-4312kB/s), io=14.4GiB (15.5GB), run=900029-900030msec
   4 fio /usr/share/doc/fio/examples/fio-seq-RW.fio  7,62s user 125,55s system 10% cpu 21:30,52 total

The difference is simply tremendous. Therefor my recommendation use virtiofs instead of 9p.

Cleanup

   1 cd ..
   2 rm -r fio

Virtiofs additional options

There are some additional options that may be passed via xml please see

libvirt.org - Domain XML format Filesystems
man 1 virtiofsd
/usr/share/libvirt/schemas/domain.rng

Here is a example configuration

   1     <filesystem type='mount' accessmode='passthrough'>
   2       <driver type='virtiofs' queue='1024'/>
   3       <!--<binary path='/usr/libexec/virtiofsd' xattr='on'>-->
   4       <binary path='/usr/lib/qemu/virtiofsd' xattr='on'>
   5         <cache mode='always'/>
   6         <lock posix='on' flock='on'/>
   7       </binary>
   8       <source dir='/media/space/backup/backup1'/>
   9       <target dir='bareos_storage_virtiofs'/>
  10       <address type='pci' domain='0x0000' bus='0x08' slot='0x00' function='0x0'/>
  11     </filesystem>

Backup

Create backups of VMs

Mount a storage in e.g. in
/mnt/qcow2

Encrypt it with an overlay filesystem like gocryptfs and mount it on
/mnt/mnt/qcow2_crypt

Create the following backup script /usr/local/sbin/kvmbackup

   1 #!/usr/bin/env bash
   2 #
   3 # Save as: /usr/local/sbin/kvmbackup
   4 #
   5 # @implements Non-interactive backup operations cycle for active kvm machines
   6 #
   7 # Utilizes virtnbdbackup tool under the hood
   8 #
   9 # The first backup of every vm on every new month is forced to be full size,
  10 # the rest backups on the month are incremental
  11 # Every backup is processed with lz4 compression 'cause this feature really
  12 # saves the host drive space a lot
  13 #
  14 # Implemented and tested under Ubuntu 20.04.2 LTS
  15 #
  16 # @see https://github.com/abbbi/virtnbdbackup for requirements and options of virtnbdbackup tool
  17 
  18 if [ "${UID}" -ne 0 ]
  19 then
  20     exec sudo bash "$(realpath "${0}")"
  21 fi
  22 
  23 ### INIT
  24 APP="virtnbdbackup"
  25 SOCKET="/var/tmp/${APP}.sock"
  26 VMBUROOT="/mnt/qcow2_crypt"
  27 TYPE="stream"
  28 LEVEL="inc"
  29 MOYE="$(date +'%m.%Y')"
  30 DIR_TARGET="${VMBUROOT}/${VM}/${MOYE}"
  31 VERBOSE=false
  32 
  33 ### MAIN
  34 readarray -t VMLIST < <(virsh list | tail -n +3 | awk '{print $2}')
  35 for VM in "${VMLIST[@]}"; do
  36         if [ ! -d "$DIR_TARGET" ]; then
  37                 MKDIR_ARGS=( -p -m 0755 )
  38                 $VERBOSE && MKDIR_ARGS+=( -v )
  39                 mkdir "${MKDIR_ARGS[@]}" "$DIR_TARGET"
  40                 LEVEL="full"
  41         fi
  42 
  43         ${APP} --domain "$VM" --socketfile "$SOCKET" \
  44                 --level "$LEVEL" --type "$TYPE" --compress \
  45                 --output "$DIR_TARGET"
  46 done

Set it executable

   1 chmod a+x /usr/local/sbin/kvmbackup

Create a systemd-service for that script /lib/systemd/system/kvmbackup.service

   1 [Unit]
   2 Description=Create backups of VMs in remote storage
   3 Requires=network.target
   4 After=network.target
   5 
   6 [Service]
   7 Type=oneshot
   8 ExecStart=/usr/local/sbin/kvmbackup
   9 
  10 [Install]
  11 WantedBy=multi-user.target

/lib/systemd/system/kvmbackup.timer

   1 [Unit]
   2 Description=Nightly kvm backup
   3 Requires=network.target
   4 
   5 [Timer]
   6 OnCalendar=*-*-* 03:00:00
   7 RandomizedDelaySec=0
   8 Persistent=true
   9 
  10 [Install]
  11 WantedBy=timers.target

Reload systemd and enable the service and timers

   1 systemctl daemon-reload
   2 systemctl enable kvmbackup.service
   3 systemctl enable kvmbackup.timer

Cleanup Backups

Create a script
/usr/local/sbin/kvmbackup_clean

   1 #!/bin/bash
   2 BACKUP_DIR="/mnt/qcow2_crypt"
   3 AGE_DAYS=60
   4 
   5 if [ -n "$BACKUP_DIR" ] && [ -d "$BACKUP_DIR" ]; then
   6   find "$BACKUP_DIR" -mindepth 2 -type d \
   7           -mtime +$AGE_DAYS -exec rm -rfv {} \;
   8 fi

Set it executeable

   1 chmod a+x /usr/local/sbin/kvmbackup_clean

Create a systemd service
/usr/lib/systemd/system/kvmbackup_clean.service

   1 [Unit]
   2 Description=Cleanup backups of VMs in remote storage
   3 Requires=network.target
   4 After=network.target
   5 
   6 [Service]
   7 Type=oneshot
   8 ExecStart=/usr/local/sbin/kvmbackup_clean
   9 
  10 [Install]
  11 WantedBy=multi-user.target

Create a systemd timer
/usr/lib/systemd/system/kvmbackup_clean.timer

   1 [Unit]
   2 Description=Nightly cleanup of backups
   3 Requires=network.target
   4 
   5 [Timer]
   6 OnCalendar=*-*-* 03:00:00
   7 RandomizedDelaySec=0
   8 Persistent=true
   9 
  10 [Install]
  11 WantedBy=timers.target

Reload systemd and enable the service and timers

   1 systemctl daemon-reload
   2 systemctl enable kvmbackup_clean.service
   3 systemctl enable kvmbackup_clean.timer

libvirt and ansible

Dynamic inventory

inventory/libvirt-localhost.yml

   1 ---
   2 ### CONNECT TO LIBVIRT QEMU ON LOCALHOST
   3 plugin: community.libvirt.libvirt
   4 uri: 'qemu:///system'

inventory/libvirt-remote.yml

   1 ---
   2 ### CONNECT TO LIBVIRT QEMU ON REMOTE HOST
   3 plugin: community.libvirt.libvirt
   4 uri: 'qemu+ssh://remote-host.example.com/system'

First steps

List hosts

   1 ansible -i inventory/libvirt-localhost.yml --list-hosts all

Tipps and Tricks

Stop all running VMs

   1 for DOMAIN in $(virsh list |grep running |awk '{print $2}'); do
   2         virsh shutdown "$DOMAIN"
   3         sleep 10
   4 done

Mount qcow2 image

Qemu images may be mounted as "Network Block Devices" using the Kernel module nbd

modinfo nbd

   1 filename:       /lib/modules/5.3.0-rc5-amd64/kernel/drivers/block/nbd.ko
   2 license:        GPL
   3 description:    Network Block Device
   4 depends:        
   5 retpoline:      Y
   6 intree:         Y
   7 name:           nbd
   8 vermagic:       5.3.0-rc5-amd64 SMP mod_unload modversions 
   9 sig_id:         PKCS#7
  10 signer:         Debian Secure Boot CA
  11 sig_key:        A7:46:8D:EF
  12 sig_hashalgo:   sha256
  13 signature:      5B:51:56:EF:15:5D:8C:65:A2:F1:B4:60:9D:92:DA:D3:C1:0B:36:07:
  14                 7E:2C:9F:EF:A9:6A:FF:18:93:39:B5:65:F9:45:3E:BB:C2:77:7F:A8:
  15                 35:8E:CA:E0:2A:57:C9:15:4B:0E:A1:1C:DB:B0:45:1E:43:01:95:15:
  16                 6F:F5:E7:3E:35:0A:8E:D2:62:90:35:1B:AB:73:66:88:E3:55:15:88:
  17                 26:37:B7:7B:B2:67:52:35:15:39:D1:AA:57:8A:C8:1F:2C:AA:DB:A6:
  18                 EF:00:0E:D4:39:43:D7:DD:EE:C2:85:AC:9C:CE:73:69:2D:F4:6E:D2:
  19                 48:F3:09:CB:E6:01:C2:B5:78:3D:A2:91:2D:F8:BB:C6:E7:40:56:0D:
  20                 63:31:5B:11:17:DF:19:A3:89:E3:14:48:B5:FB:3A:F4:3F:52:80:63:
  21                 1B:2A:C0:AB:98:8A:50:D3:37:1B:70:7D:A6:BF:4F:5A:38:7E:92:E7:
  22                 B6:53:58:8E:A2:1E:80:DB:7F:00:F2:77:43:7E:ED:20:1C:EB:58:39:
  23                 6E:9F:E4:DC:2A:D0:C1:A2:D5:74:F3:90:C0:E5:1A:67:F2:86:A2:4A:
  24                 4D:1E:18:FA:1D:59:CA:C3:45:6F:0F:1B:1F:FD:D1:E6:A5:ED:E1:A5:
  25                 13:D7:B7:A2:A5:2A:A8:24:39:FB:68:DF:E7:37:A4:A9
  26 parm:           nbds_max:number of network block devices to initialize (default: 16) (int)
  27 parm:           max_part:number of partitions per device (default: 16) (int)

According to modinfo nbd has two parameters, whose defaults are fine.

Load module

   1 modprobe nbd

Connect immage as nbd-device

   1 qemu-nbd --connect=/dev/nbd0 /var/lib/libvirt/images/disk_image.qcow2

Now the image and all its partitions are available as devices in the device-tree. ll /dev/nbd*

   1 brw-rw---- 1 root disk 43,   0 Oct 20 12:44 /dev/nbd0
   2 brw-rw---- 1 root disk 43,   1 Oct 20 12:44 /dev/nbd0p1
   3 brw-rw---- 1 root disk 43,   2 Oct 20 12:44 /dev/nbd0p2
   4 brw-rw---- 1 root disk 43,   3 Oct 20 12:44 /dev/nbd0p3
   5 brw-rw---- 1 root disk 43,   4 Oct 20 12:44 /dev/nbd0p4
   6 brw-rw---- 1 root disk 43,  32 Oct 20 12:40 /dev/nbd1
   7 brw-rw---- 1 root disk 43, 320 Oct 20 12:40 /dev/nbd10
   8 brw-rw---- 1 root disk 43, 352 Oct 20 12:40 /dev/nbd11
   9 brw-rw---- 1 root disk 43, 384 Oct 20 12:40 /dev/nbd12
  10 brw-rw---- 1 root disk 43, 416 Oct 20 12:40 /dev/nbd13
  11 brw-rw---- 1 root disk 43, 448 Oct 20 12:40 /dev/nbd14
  12 brw-rw---- 1 root disk 43, 480 Oct 20 12:40 /dev/nbd15
  13 brw-rw---- 1 root disk 43,  64 Oct 20 12:40 /dev/nbd2
  14 brw-rw---- 1 root disk 43,  96 Oct 20 12:40 /dev/nbd3
  15 brw-rw---- 1 root disk 43, 128 Oct 20 12:40 /dev/nbd4
  16 brw-rw---- 1 root disk 43, 160 Oct 20 12:40 /dev/nbd5
  17 brw-rw---- 1 root disk 43, 192 Oct 20 12:40 /dev/nbd6
  18 brw-rw---- 1 root disk 43, 224 Oct 20 12:40 /dev/nbd7
  19 brw-rw---- 1 root disk 43, 256 Oct 20 12:40 /dev/nbd8
  20 brw-rw---- 1 root disk 43, 288 Oct 20 12:40 /dev/nbd9

Simply mount the device where you like

   1 DEVICE_BASE="/dev/nbd0"
   2 for DEVICE in "$DEVICE_BASE"*; do
   3         MOUNT_PATH="/media/nbd/${DEVICE#/dev/}"
   4         mkdir -vp "$MOUNT_PATH"
   5         mount "$DEVICE" "$MOUNT_PATH"
   6 done

Unmount it, when done

   1 cd /media/nbd
   2 umount *

Disconnect the nbd-device

   1 qemu-nbd --disconnect /dev/nbd0
   2 /dev/nbd0 disconnected

Optionally unload the module nbd from the kernel.

   1 modprobe -r nbd

Libvirt with EFI

Little quote from this OVMF whitepaper

The Unified Extensible Firmware Interface (UEFI) is a specification that defines a software interface between an operating system and platform firmware. UEFI is designed to replace the Basic Input/Output System (BIOS) firmware interface.

Hardware platform vendors have been increasingly adopting the UEFI Specification to govern their boot firmware developments. OVMF (Open Virtual Machine Firmware), a sub-project of Intel's EFI Development Kit II (edk2), enables UEFI support for Ia32 and X64 Virtual Machines.

Install UEFI-firmware for qemu

   1 aptitude install ovmf qemu-efi

Code-Snippet for UEFI-boot in KVM

   1   <os>
   2     <type arch='x86_64' machine='pc-q35-4.2'>hvm</type>
   3     <loader readonly='yes' secure='no' type='pflash'>/usr/share/OVMF/OVMF_CODE.fd</loader>
   4     <nvram>/var/lib/libvirt/qemu/nvram/template_stretch_VARS.fd</nvram>
   5     <bootmenu enable='yes' timeout='3000'/>
   6   </os>

VM boots fine with UEFI.

But nnapshots are no longer supported.

   1 Error creating snapshot: Operation not supported:
   2 internal snapshots of a VM with pflash based firmware are not supported

To workaround this limitation, we have to change type='pflash' to type='rom'. Okay the vm does not boot any more and we are dropped in a UEFI shell.

Command reference

Obviously the nvram is missing. To get the system working again i had to point it to the bootloader

   1 fs0:
   2 cd EFI
   3 cd debian
   4 grubx64.efi

You may also exit the UEFI shell to enter a BIOS like UEFI menu Select

Boot Maintenance Manager
Boot from file
Select directory "EFI"
Select directory "debian"
Select file "grubx64.efi"

Well, i heard that tianocore is the best UEFI implementation. Unfortunately i cannot give up the ability to take snapshots for UEFI. So i'll stick with BIOS for the moment. Everything is in place - when kvm/libvirt is ready - i'm the first one to switch to the new interface.

   1 # apt install grub-pc grub-efi-amd64-
   2

Remove all the UEFI stuff
virsh edit $VM

   1   <os>
   2     <type arch='x86_64' machine='pc-q35-4.2'>hvm</type>
   3     <boot dev="hd"/>
   4   </os>

So far …

btrfs nocow

To improve the IO-performance of your virtual machines on btrfs, please see filesystems/btrfs#No Copy on Write (NOCOW)

Convert raw disk to qcow2 image

   1 qemu-img convert \
   2         -m 16 \
   3         -p -c -O qcow2 \
   4         disk.raw disk.qcow2

Sparsify qcow2 image

Install guestfs-tools

   1 apt install guestfs-tools

Resize qcow2 image

This procedure allows resizing the disk image without entering the VM e.g. with a live image.

Install utilities
```
   1 apt install guestfs-tools
```
Shutdown VM
```
   1 DOMAIN="nextcloud1"
   2 virsh shutdown "$DOMAIN"
```
Create a backup copy of the original image
```
   1 cd /var/lib/libvirt/images
   2 cp --reflink=always "$DOMAIN".qcow2 "$DOMAIN".qcow2_bak
```
Aquire some info about the qcow2 image
```
   1 virt-filesystems --long -h --all -a "$DOMAIN".qcow2
```
Resize the VM image to the new size
```
   1 qemu-img resize "$DOMAIN".qcow2 256G
```
Create a copy of the resized VM image
```
   1 cp --reflink=always "$DOMAIN".qcow2 "$DOMAIN".qcow2_bak2
```
Expand partitions and filesystems to the new disk size
```
   1 virt-resize -v -expand /dev/sda4 "$DOMAIN".qcow2_bak2 "$DOMAIN".qcow2
```

Aquire some info about the qcow2 image

   1 virt-filesystems --long -h --all -a "$DOMAIN".qcow2
   2 Name                                                                                                                     Type       VFS     Label  MBR Size Parent                                 
   3 /dev/sda1                                                                                                                filesystem unknown -      -   1,0M -                                      
   4 /dev/sda2                                                                                                                filesystem unknown -      -   244M -                                      
   5 /dev/sda3                                                                                                                filesystem swap    -      -   3,7G -                                      
   6 /dev/sda4                                                                                                                filesystem btrfs   rootfs -   252G -                                      
   7 btrfsvol:/dev/sda4/var/lib/docker/btrfs/subvolumes/a7dbcdb5896eaf8f841705c9805f37609abcda0771601875eb32eb9b58fcdeda      filesystem btrfs   rootfs -   -    -                                      
   8 btrfsvol:/dev/sda4/var/lib/docker/btrfs/subvolumes/8253cfad42dae565527e927d28b79468e0e4b348add029d3a6f48f0187ee0ea7      filesystem btrfs   rootfs -   -    -                                      
   9 btrfsvol:/dev/sda4/var/lib/docker/btrfs/subvolumes/e79011c2a2dcf3c4ae59ceb9c0c60fc364326467e066782442dbcc4305af2bac      filesystem btrfs   rootfs -   -    -                                      
  10 btrfsvol:/dev/sda4/var/lib/docker/btrfs/subvolumes/aeaba1c43b3a78776ee8426b944c2cda31976e5651ca1f3bb9106665d5d33297      filesystem btrfs   rootfs -   -    -                                      
  11 btrfsvol:/dev/sda4/var/lib/docker/btrfs/subvolumes/0345843bacc85bb63378440906a8e5baa53fc05bacdcd01f304a1ddef2a5c6f1      filesystem btrfs   rootfs -   -    -                                      
  12 btrfsvol:/dev/sda4/var/lib/docker/btrfs/subvolumes/e43dc3eb04fa5f4e2c86051ec87575d6687b221f2a1623944c2cfec3efe8bd55      filesystem btrfs   rootfs -   -    -                                      
  13 btrfsvol:/dev/sda4/var/lib/docker/btrfs/subvolumes/8bf622e74770efed48316800b08d1eb43f34caf1bd13132cedaae8c01f6a0da1      filesystem btrfs   rootfs -   -    -                                      
  14 btrfsvol:/dev/sda4/var/lib/docker/btrfs/subvolumes/3104a94e82483e588c1623d6c075a9d7023368dab3e38d1ed0ec097dd6f7f390-init filesystem btrfs   rootfs -   -    -                                      
  15 btrfsvol:/dev/sda4/var/lib/docker/btrfs/subvolumes/3104a94e82483e588c1623d6c075a9d7023368dab3e38d1ed0ec097dd6f7f390      filesystem btrfs   rootfs -   -    -                                      
  16 /dev/sda1                                                                                                                partition  -       -      -   1,0M /dev/sda                               
  17 /dev/sda2                                                                                                                partition  -       -      -   244M /dev/sda                               
  18 /dev/sda3                                                                                                                partition  -       -      -   3,7G /dev/sda                               
  19 /dev/sda4                                                                                                                partition  -       -      -   252G /dev/sda                               
  20 /dev/sda                                                                                                                 device     -       -      -   256G -                                      
  21 virt-filesystems --long -h --all -a "$DOMAIN".qcow2  4,43s user 3,09s system 8% cpu 1:24,77 total

Start and test the VM
```
   1 virsh start "$DOMAIN"
```
(Some days later) cleanup the backup images
```
   1 cd /var/lib/libvirt/images
   2 rm "$DOMAIN".qcow2_bak*
```

Remote viewer

Create a LocalForward
with the integrated ssh shell

   1 ### HIT ENTER
   2 ~C
   3 -L 5900:localhost:5900

or with CLI arguments

   1 ssh -L 5900:localhost:5900 remote-machine

Then in a new shell start remote viewer with the connection URL as argument

   1 remote-viewer 'spice://127.0.0.1:5900'

Or create a file
vm-name.ini

   1 [virt-viewer]
   2 type=spice
   3 host=127.0.0.1
   4 port=5900
   5 delete-this-file=0
   6 fullscreen=0

Start the remote viewer with this configuration file as argument

   1 remote-viewer vm-name.ini

Daily snapshot report

Create a little script that queries libvirt for domains with attached snapshots, gathers some snapshot info and sends an email to some recipients.

/usr/local/sbin/libvirt_snapshot_report.sh

   1 #!/bin/bash
   2 
   3 DOMAINS_WITH_SNAPSHOTS="$(virsh list --with-snapshot)"
   4 FQDN="$(hostname -f)"
   5 DATE="$(date '+%F %T %Z')"
   6 SUBJECT="Libvirt snapshot report of host: '$FQDN'"
   7 RECIPIENTS=("root")
   8 declare -A DOMAIN_SNAPSHOT_INFO
   9 declare -a DOMAINS
  10 
  11 ### GATHER SNAPSHOT INFO
  12 readarray -t DOMAINS \
  13         < <(virsh list --with-snapshot --name \
  14                 |sed '/^\s*$/d')
  15 
  16 for DOMAIN in "${DOMAINS[@]}"; do
  17         DOMAIN_SNAPSHOT_INFO["$DOMAIN"]="$(virsh snapshot-list "$DOMAIN")"
  18 done
  19 
  20 SNAPSHOT_DETAILS_PRETTY="$(
  21         for DOMAIN in "${DOMAINS[@]}"; do
  22                 cat <<-EOI
  23                         #### Domain: '$DOMAIN'
  24                         ${DOMAIN_SNAPSHOT_INFO["$DOMAIN"]}
  25 
  26                 EOI
  27         done
  28 )"
  29 
  30 mail -s "$SUBJECT" "${RECIPIENTS[@]}" <<-EOM
  31         ## Libvirt snapshot report
  32 
  33         ### Metainformation
  34 
  35         fqdn: "$FQDN"
  36         date: "$DATE"
  37 
  38         ### Domains with snapshots:
  39 
  40         $DOMAINS_WITH_SNAPSHOTS
  41 
  42         ### Per domain snapshot information:
  43 
  44         $SNAPSHOT_DETAILS_PRETTY
  45 
  46         Please take care of dangling snaphots.
  47 EOM

Note: The heredoc statements need TAB characters or won't work. Make sure to replace it correctly when copying.

Make script executable

   1 chmod a+x /usr/local/sbin/libvirt_snapshot_report.sh

Create a systemd-service
/lib/systemd/system/libvirt-snapshot-report.service

   1 [Unit]
   2 Description=Generate report of snapshots attached to libvirt domains.
   3 
   4 [Service]
   5 Type=simple
   6 ExecStart=/usr/local/sbin/libvirt_snapshot_report.sh
   7 IOSchedulingClass=idle
   8 CPUSchedulingPolicy=idle

Pleas also see
Systemd#systemd.timers

Create a systemd-timer
/lib/systemd/system/libvirt-snapshot-report.timer

   1 [Unit]
   2 Description=Generate report of snapshots attached to libvirt domains.
   3 
   4 [Timer]
   5 OnCalendar=*-*-* 08:00:00
   6 AccuracySec=1h
   7 Persistent=true
   8 
   9 [Install]
  10 WantedBy=timers.target

Enable the systemd-timer

   1 systemctl enable libvirt-snapshot-report.timer

Trouble Shooting

Error starting NAT networks

   1 Fehler beim Starten des Netzwerks »uplink«: Interner Fehler: Failed to run firewall command iptables -w --table nat --insert LIBVIRT_PRT --source 192.168.20.0/24 '!' --destination 192.168.20.0/24 --jump MASQUERADE: Warning: Extension MASQUERADE revision 0 not supported, missing kernel module?
   2 iptables v1.8.11 (nf_tables):  RULE_INSERT failed (No such file or directory): rule in chain LIBVIRT_PRT
   3 
   4 
   5 Traceback (most recent call last):
   6   File "/usr/share/virt-manager/virtManager/asyncjob.py", line 67, in cb_wrapper
   7     callback(asyncjob, *args, **kwargs)
   8     ~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^
   9   File "/usr/share/virt-manager/virtManager/asyncjob.py", line 101, in tmpcb
  10     callback(*args, **kwargs)
  11     ~~~~~~~~^^^^^^^^^^^^^^^^^
  12   File "/usr/share/virt-manager/virtManager/object/libvirtobject.py", line 57, in newfn
  13     ret = fn(self, *args, **kwargs)
  14   File "/usr/share/virt-manager/virtManager/object/network.py", line 61, in start
  15     self._backend.create()
  16     ~~~~~~~~~~~~~~~~~~~~^^
  17   File "/usr/lib/python3/dist-packages/libvirt.py", line 3623, in create
  18     raise libvirtError('virNetworkCreate() failed')
  19 libvirt.libvirtError: Interner Fehler: Failed to run firewall command iptables -w --table nat --insert LIBVIRT_PRT --source 192.168.20.0/24 '!' --destination 192.168.20.0/24 --jump MASQUERADE: Warning: Extension MASQUERADE revision 0 not supported, missing kernel module?
  20 iptables v1.8.11 (nf_tables):  RULE_INSERT failed (No such file or directory): rule in chain LIBVIRT_PRT

Switch to nftables in
/etc/libvirt/network.conf

   1 # Master configuration file for the network driver.
   2 # All settings described here are optional - if omitted, sensible
   3 # defaults are used.
   4 
   5 # firewall_backend:
   6 #
   7 #   determines which subsystem to use to setup firewall packet
   8 #   filtering rules for virtual networks.
   9 #
  10 #   Supported settings:
  11 #
  12 #     iptables - use iptables commands to construct the firewall
  13 #     nftables - use nft commands to construct the firewall
  14 #
  15 #   If firewall_backend isn't configured, libvirt will choose the
  16 #   first available backend from the following list:
  17 #
  18 #     [iptables, nftables]
  19 #
  20 #   If no backend is available on the host, then the network driver
  21 #   will fail to start, and an error will be logged.
  22 #
  23 #   (NB: switching from one backend to another while there are active
  24 #   virtual networks *is* supported. The change will take place the
  25 #   next time that libvirtd/virtnetworkd is restarted - all existing
  26 #   virtual networks will have their old firewalls removed, and then
  27 #   reloaded using the new backend.)
  28 #
  29 #firewall_backend = "iptables"
  30 firewall_backend = "nftables"

And restart libvirtd

   1 systemctl restart libvirtd.service

Might be a good idea anyway.