Debian

Contents

Debian

About

Releases

Base system from scratch

Remote system is a dedicated root server in the Hetzner datacenter. Only a live linux amd64 is running and it's reachable by ssh with pubkey auth. We got the option to restart/reset the live linux or boot into the new system. So the logs of the new system is all information we can get (besides the status: running/stopped).

Partitioning

Debian partman units

Offtopic (debian-installer) but nice to know.

Hint: Debian partman units

Debian partman does not support binary units like KiB, MiB or GiB. All units are SI units based on 10. Default unit (no unit) is "1m" aligned to the next cylinder (1Mib).

The sizes of the partitions will not be what you expect.

To achieve correct sizes to may:

partition manually with parted in a separate shell and enter partman when done.
enter the sizes in byte with unit "b" or "B" (don't leave it away) Some examples:
```
   1 ~ # echo $((4*2**30))b ### 4GiB
   2 4294967296b
   3 ~ # echo $((256*2**20))b ### 256MiB
   4 268435456b
```

Create the partitions on the first disk

Aline it to the full 1 MiB/Cylinder (MegaByte binary), because

this allows aligned read/writes. Otherwise it will degrade performance significantly.
this is a power of 2 (1, 2, 4(MEMORY PAGESIZE), 8, ...2^n)KiB. This means if you want to stripe or change some cluster sizes, it will always match (if less).

The boot-partition should be at least 256MiB, initial ram-disks can be large.

(vmlinuz 6MiB + initrd 50MiB + system-map 4MiB) = 60MiB/Kernel So max 3 Kernels …

   1 parted
   2 GNU Parted 3.2
   3 Using /dev/sda
   4 Welcome to GNU Parted! Type 'help' to view a list of commands.
   5 (parted) unit MiB
   6 (parted) print free
   7 Model: ATA HGST HUS726060AL (scsi)
   8 Disk /dev/sda: 5723167MiB
   9 Sector size (logical/physical): 512B/4096B
  10 Partition Table: gpt
  11 Disk Flags: 
  12 
  13 Number  Start       End         Size        File system  Name           Flags
  14         0.02MiB     1.00MiB     0.98MiB     Free Space
  15  1      1.00MiB     2.00MiB     1.00MiB                  bios_grub_sda  bios_grub
  16  2      2.00MiB     256MiB      254MiB      fat16        EFI_sda        boot, esp
  17  3      256MiB      131072MiB   130816MiB                swap1_sda      raid
  18  4      131072MiB   5723166MiB  5592094MiB  btrfs        root_sda
  19         5723166MiB  5723167MiB  0.98MiB     Free Space

Clone GPT to other disk

Clone gpt and create unique GUIDs

   1 sgdisk -G /dev/sda -R /dev/sdb

Create a multidisk RAID1 as swap

   1 mdadm -v --create md_swap1 --level=1 --symlinks yes --raid-devices=2 /dev/sda3 /dev/sdb3
   2 mdadm --detail --scan
   3 mkswap /dev/md/md_swap1 --label md_swap1
   4 swapon /dev/md/md_swap1

Filesystems

   1 ### CREATE FILESYSTEM
   2 mkfs.btrfs -L rootfs --data raid1 --metadata raid1 /dev/sda4 /dev/sdb4
   3 
   4 ### CREATE SUBVOLUMES
   5 mount -o noatime,compress=lzo,space_cache,autodefrag /dev/sdb4 /mnt
   6 btrfs subvolume create /mnt/root
   7 btrfs subvolume create /mnt/home
   8 
   9 btrfs subvolume list /mnt
  10 ID 258 gen 8 top level 5 path root
  11 
  12 btrfs subvolume get-default /mnt/
  13 ID 5 (FS_TREE)
  14 btrfs subvolume set-default 258 /mnt
  15 
  16 umount /mnt
  17 mount -o noatime,compress=lzo,space_cache,autodefrag /dev/sdb4 /mnt
  18 mount -t btrfs
  19 /dev/sda4 on /mnt type btrfs (rw,noatime,compress=lzo,space_cache,autodefrag,subvolid=258,subvol=/root)
  20 
  21 debootstrap buster /mnt http://ftp.de.debian.org/debian

Migrate data to subvolumes

   1 umount /mnt
   2 mount -o noatime,compress=lzo,space_cache,autodefrag,subvol=/ /dev/sdb4 /mnt
   3 
   4 btrfs subvolume create /mnt/home
   5 
   6 btrfs subvolume snapshot /mnt/root/ /mnt/var_log
   7 find /mnt/var_log -mindepth 1 -maxdepth 1 \! -name var | xargs rm -r --
   8 mv /mnt/var_log/var/log/* /mnt/var_log/; rm -r /mnt/var_log/var/
   9 find /mnt/root/var/log -mindepth 1 -maxdepth 1 | xargs rm -r --
  10 
  11 mount -o noatime,compress=lzo,space_cache,autodefrag /dev/sdb4 /mnt
  12 mount -o noatime,compress=lzo,space_cache,autodefrag,subvol=/home /dev/sdb4 /mnt/home
  13 mount -o noatime,compress=lzo,space_cache,autodefrag,subvol=/var_log /dev/sdb4 /mnt/var/log/
  14 
  15 mount -t btrfs
  16 /dev/sda4 on /mnt type btrfs (rw,noatime,compress=lzo,space_cache,autodefrag,subvolid=258,subvol=/root)
  17 /dev/sda4 on /mnt/home type btrfs (rw,noatime,compress=lzo,space_cache,autodefrag,subvolid=261,subvol=/home)
  18 /dev/sda4 on /mnt/var/log type btrfs (rw,noatime,compress=lzo,space_cache,autodefrag,subvolid=262,subvol=/var_log)

Prepare EFI filesystems

Unnecessary - just skip EFI.

   1 ### USE UPPERCASE FS-LABELS
   2 mkfs.vfat -n EFI_SDA /dev/sda2
   3 mkfs.vfat -n EFI_SDB /dev/sdb2
   4 mkdir /mnt/boot/EFI /mnt/boot/EFI_SDB
   5 mount /mnt/boot/EFI
   6 mount /mnt/boot/EFI_SDB
   7 rsync -a EFI ../EFI_SDB/

fstab

Make sure to use FS-UUIDS and no devices in fstab

   1 blkid
   2 /dev/sdb4: LABEL="rootfs" UUID="0cc274fd-4cb8-4cc7-9f60-f59de41f6891" UUID_SUB="d65eac71-86ca-45e0-b59a-7c872de54e59" TYPE="btrfs" PARTLABEL="root_sdb" PARTUUID="20199b69-b0a7-4151-9dd1-83c741d580a0"
   3 /dev/loop0: UUID="40c4ea95-0ecc-4c51-9f3e-e49d8f62f160" TYPE="ext2"
   4 /dev/sda1: PARTLABEL="bios_grub_sda" PARTUUID="1db5c109-b489-4466-8d51-23ef189253b7"
   5 /dev/sda2: SEC_TYPE="msdos" LABEL_FATBOOT="EFI" LABEL="EFI" UUID="8AC4-4574" TYPE="vfat" PARTLABEL="EFI_sda" PARTUUID="14827a48-5a73-4012-bd28-176d365bd903"
   6 /dev/sda3: UUID="4b81ee7b-00bb-3ce6-e83d-c0c00c449861" UUID_SUB="9e9e8cf9-9b01-6128-e2b6-acf8e04e7e9e" LABEL="rescue:md_swap1" TYPE="linux_raid_member" PARTLABEL="swap1_sda" PARTUUID="95586f2b-45ac-478a-8481-fd176f16cbb5"
   7 /dev/sda4: LABEL="rootfs" UUID="0cc274fd-4cb8-4cc7-9f60-f59de41f6891" UUID_SUB="6c72fb67-6c5d-40bc-a4b4-dd34199a1d2b" TYPE="btrfs" PARTLABEL="root_sda" PARTUUID="20199b69-b0a7-4151-9dd1-83c741d580a0"
   8 /dev/sdb1: PARTLABEL="bios_grub_sdb" PARTUUID="1db5c109-b489-4466-8d51-23ef189253b7"
   9 /dev/sdb2: PARTLABEL="EFI_sdb" PARTUUID="14827a48-5a73-4012-bd28-176d365bd903"
  10 /dev/sdb3: UUID="4b81ee7b-00bb-3ce6-e83d-c0c00c449861" UUID_SUB="981dfa0d-6584-2576-a51a-02102935a87b" LABEL="rescue:md_swap1" TYPE="linux_raid_member" PARTLABEL="swap1_sdb" PARTUUID="95586f2b-45ac-478a-8481-fd176f16cbb5"
  11 /dev/md127: LABEL="md_swap1" UUID="87294740-52c4-4557-b838-ddc44ba8aa4b" TYPE="swap"

Edit fstab to reflect new structure of filesystem

   1 cat /etc/fstab
   2 # /etc/fstab: static file system information.
   3 #
   4 # Use 'blkid' to print the universally unique identifier for a
   5 # device; this may be used with UUID= as a more robust way to name devices
   6 # that works even if disks are added and removed. See fstab(5).
   7 
   8 #<file_system>                             <mount_point>  <type>  <options>                                                                    <dump>  <pass>
   9 UUID=87294740-52c4-4557-b838-ddc44ba8aa4b  none           swap    sw                                                                           0       0
  10 UUID=0cc274fd-4cb8-4cc7-9f60-f59de41f6891  /              btrfs   rw,noatime,compress=lzo,space_cache,autodefrag,subvolid=258,subvol=/root     0       1
  11 UUID=8AC4-4574                             /boot/EFI      vfat    utf8                                                                         0       0
  12 UUID=B3B5-67FA                             /boot/EFI_SDB  vfat    utf8                                                                         0       0
  13 UUID=0cc274fd-4cb8-4cc7-9f60-f59de41f6891  /home          btrfs   rw,noatime,compress=lzo,space_cache,autodefrag,subvolid=261,subvol=/home     0       0
  14 UUID=0cc274fd-4cb8-4cc7-9f60-f59de41f6891  /var/log       btrfs   rw,noatime,compress=lzo,space_cache,autodefrag,subvolid=262,subvol=/var_log  0       2

Validate fstab

   1 findmnt --verify
   2 Success, no errors or warnings detected

Chroot into new system

   1 mount -t proc none /mnt/proc
   2 mount -t tmpfs none /mnt/tmp
   3 
   4 for MNT in /dev /dev/pts /run /sys;do
   5         mount --bind $MNT /mnt/${MNT%/}
   6 done
   7 chroot /mnt /bin/bash

Basic Configuration

Adjust hostname and mailname

Set the hostname

   1 echo 'kvm2' > /etc/hostname
   2 ### DEFINED $myorigin
   3 echo 'kvm2.rockstable.org' > /etc/mailname

Please see also Names#hostnames on this topic.

Load the hostname and exec a new shell

   1 hostname -F /etc/hostname
   2 exec "$SHELL"

Optionally renew your ssh host-keys (e.g. when cloned)

   1 rm /etc/ssh/ssh_host_*
   2 dpkg-reconfigure openssh-server
   3 shutdown -r now

Adjust hosts

/etc/hosts

   1 127.0.0.1       localhost
   2 ::1             localhost ip6-localhost ip6-loopback
   3 ff02::1         ip6-allnodes
   4 ff02::2         ip6-allrouters
   5 
   6 195.201.246.253      kvm2.rockstable.org. kvm2
   7 2a01:4f8:231:702::2  kvm2.rockstable.org. kvm2

Please make sure the canonical name/FQDN is listed in front of any additional aliases or the server may not be able to determine its own domain. Further a canonical name/FQDN should not point to the loopback (lo) interface of the server.

Adjust timezone

   1 dpkg-reconfigure tzdata                                                                                                                                     
   2 
   3 Current default time zone: 'Europe/Berlin'
   4 Local time is now:      Wed May 22 14:22:32 CEST 2019.
   5 Universal Time is now:  Wed May 22 12:22:32 UTC 2019.

Configure apt-sources

   1 cat /etc/apt/sources.list
   2 deb       http://ftp.de.debian.org/debian              buster            main  contrib  non-free
   3 deb-src   http://ftp.de.debian.org/debian/             buster            main  contrib  non-free
   4 deb       http://ftp.de.debian.org/debian/             buster-updates    main  contrib  non-free
   5 deb-src   http://ftp.de.debian.org/debian/             buster-updates    main  contrib  non-free
   6 deb       http://security.debian.org/debian-security/  buster/updates    main  contrib  non-free
   7 deb-src   http://security.debian.org/debian-security/  buster/updates    main  contrib  non-free
   8 #deb      http://ftp.de.debian.org/debian/             buster-backports  main  contrib  non-free
   9 #deb-src  http://ftp.de.debian.org/debian/             buster-backports  main  contrib  non-free
  10

Pinning: apt_preferences

If you create a preferences file like /etc/apt/preferences{,.d/filename{,.pref}} make sure the Pinning blocks are separated by a line, which must not contain any whitespace characters or apt will not respect your preference.

This example has cause (invisible) problems with the ^I Tab-characters between the blocks!

   1 # Debian preferences^I^I$
   2 Package:^I*$    
   3 Pin:^I^Irelease n=jessie$
   4 Pin-Priority:^I500$
   5 ^I^I$
   6 Package:^I*$    
   7 Pin:^I^Irelease n=jessie-updates$
   8 Pin-Priority:^I500$

Check preferences:

   1 apt-cache policy|grep -C1 release

Some notes regarding preferences

If you are using multiple codenames on one system at once (e.g. Buster: 500, Bullseye: 400) you should change the priority of the respective backports to be slightly less than their corresponding codename (e.g. Buster: 490, Bullseye: 390).

Background: If you left them at 100 (default) and you install a package from backports explicitly like an updated kernel, it will be upgraded to the codename with the higher priority when a package of the same name but a higher version is available there. This may not be what you intended.

Debian codenames with suffixes or from debian security should have the same priority as the main codename. This ensures you have the most recent and secure version installed.

If you are switching between packages from different codenames, make sure to mark dependencies as "Installed Automatically" A ("M" in aptitude). Or manually installed packages that are not required by any other manually installed package will end up as garbage. Garbage can be identified using:

   1 aptitude search '~g'

Add some essential packages

   1 apt install \
   2         apt-file aptitude bash-completion byobu btrfs-progs ca-certificates curl \
   3         dmidecode dosfstools git gpm htop iftop info iotop jq libcrack2 locales \
   4         lsb-release lsof man-db mc mlocate openssl parted pigz psmisc pv \
   5         pwgen python3-apt rsync screen sqlite3 ssl-cert strace sudo \
   6         sysstat tmux vim wget zsh

Configure locales

   1 dpkg-reconfigure locales

Configure vim

Adjust it to your needs like in vim

zsh grml-flavoured

I strongly recommend this config! It's simply awesome. Thanks for this!

   1 git clone git://git.grml.org/grml-etc-core.git /opt/grml-etc-core
   2 mv /etc/zsh{,_dist}
   3 ln -s /opt/grml-etc-core/etc/zsh /etc/zsh
   4 chsh -s /bin/zsh
   5 exec zsh

Networking

ifupdown vs. ifupdown2

Cons

ifupdown2 currently does not support the interfaces-keyword metric provided by ifmetric, which is still required as a package. If you get multiple default routes e.g. via dhcp-client and you don't have access to ifupdown >= 1.2.7-1, you should stick with ifupdown.
- 1.2.5-1 supports metric but has Bug 930839
- 1.2.7-1 Fixed, working.
Changing is service interrupting

Pros

New command ifreload which can change

status of interfaces without taking them down.

Install ifupdown2

You better do it in a tmux session, because ssh-session will break and won't return.

   1 tmux
   2 PRIMARY_INTERFACE="enp1s0"
   3 aptitude install \
   4         ifupdown- \
   5         bridge-utils dnsutils ethtool fail2ban ifupdown2 iputils-tracepath \
   6         isc-dhcp-client net-tools pciutils python-gvgen python-mako \
   7         python-pkg-resources traceroute; \
   8         ifup "$PRIMARY_INTERFACE"

Use Enter ~ . to terminate frozen ssh-clients. Help about ssh escape commands: Enter ~ ?

Predictable device names

www.freedesktop.org: PredictableNetworkInterfaceNames

Let's predict some names.

   1 lspci| grep -i eth
   2 00:1f.6 Ethernet controller: Intel Corporation Ethernet Connection (2) I219-LM

Okay, so PCI-Address is 00:1f.6

Bus:Device.Function (BDF)

BUS: 0
Device/Slot: 31 = 1*16¹ + 15*16⁰
Function: 6

access.redhat.com: Understanding the predictable network interface device names

So we predict the interface name to be: enp0s31f6 = "en" + "p" + "0" + s + "31" + "f" + "6"

On Upgrade to buster

If the freshly upgraded system still has old interface naming scheme, you may wish to upgrade. To achieve this, you have to remove /etc/udev/rules.d/80-net-setup-link.rules as well as /etc/systemd/network/50-virtio-kernel-names.link and rebuild your initial-ramdisks.

Please also read: /usr/share/doc/udev/README.Debian.gz

   1 rm /etc/udev/rules.d/80-net-setup-link.rules
   2 rm /etc/systemd/network/50-virtio-kernel-names.link
   3 update-initramfs -u

Configure interfaces

   1 cat /etc/network/interfaces
   2 # interfaces(5) file used by ifup(8) and ifdown(8)
   3 
   4 auto lo
   5 iface lo inet loopback
   6 
   7 auto enp0s31f6
   8 iface enp0s31f6 inet static
   9         address         195.201.246.253/26
  10         address         2a01:4f8:231:702::2/64
  11         gateway         195.201.246.193
  12 
  13 # Include files from /etc/network/interfaces.d:
  14 source-directory /etc/network/interfaces.d

Check the configuration of the DNS-resolver

   1 ### CONTROL CONFIGURATION DNS-RESOLVER
   2 cat /etc/resolv.conf

Install openssh-server

   1 aptitude install openssh-server ssh-askpass openssh-client

If you skipped the step of setting up the hostname your should renew the hostkeys or edit the wrong comment.

   1 rm /etc/ssh/ssh_host_*
   2 dpkg-reconfigure openssh-server

Make sure ssh-server starts on boot

   1 systemctl enable ssh
   2 Synchronizing state of ssh.service with SysV service script with /lib/systemd/systemd-sysv-install.
   3 Executing: /lib/systemd/systemd-sysv-install enable ssh

Embed pubkey for authentication

   1 ls /mnt/root/.ssh/authorized_keys
   2 /mnt/root/.ssh/authorized_keys

Make sure root can login via pubkey-auth (default is fine)

   1 #PermitRootLogin prohibit-password
   2

Prepare boot

   1 aptitude install \
   2         acl apparmor-profiles-extra apparmor-utils firmware-linux \
   3         firmware-linux-free grub-pc initramfs-tools \
   4         linux-headers-amd64 linux-image-amd64 mdadm nfs-common

Just forget about EFI-boot in situations where you can't control that machines UEFI/BIOS and stick with grub-pc.

Last safetys infront of reboot

   1 grub-install /dev/sda
   2 i386-pc wird für Ihre Plattform installiert.
   3 installation beendet. Keine Fehler aufgetreten.
   4 grub-install /dev/sdb
   5 i386-pc wird für Ihre Plattform installiert.
   6 installation beendet. Keine Fehler aufgetreten.

   1 update-grub2
   2 Generating grub configuration file ...
   3 Found linux image: /boot/vmlinuz-4.19.0-5-amd64
   4 Found initrd image: /boot/initrd.img-4.19.0-5-amd64
   5 done

   1 update-initramfs -k all -u 
   2 update-initramfs: Generating /boot/initrd.img-4.19.0-5-amd64

Make sure MD-RAID reassebles on next boot

If you assembled the RAID array earlier in the live system, you will have to change the name configuration from the live systems hostname to your new one.

   1 cat /etc/mdadm/mdadm.conf                                                                                                                                                                                                                                                                                                                
   2 # mdadm.conf
   3 #
   4 # !NB! Run update-initramfs -u after updating this file.
   5 # !NB! This will ensure that initramfs has an uptodate copy.
   6 #
   7 # Please refer to mdadm.conf(5) for information about this file.
   8 #
   9 
  10 # by default (built-in), scan all partitions (/proc/partitions) and all
  11 # containers for MD superblocks. alternatively, specify devices to scan, using
  12 # wildcards if desired.
  13 #DEVICE partitions containers
  14 
  15 # automatically tag new arrays as belonging to the local system
  16 HOMEHOST <system>
  17 
  18 # instruct the monitoring daemon where to send mail alerts
  19 MAILADDR root
  20 
  21 # definitions of existing MD arrays
  22 ARRAY /dev/md/md_swap1  metadata=1.2 UUID=4b81ee7b:00bb3ce6:e83dc0c0:0c449861 name=kvm2:md_swap1
  23 
  24 # This configuration was auto-generated on Wed, 22 May 2019 09:56:16 +0000 by mkconf
  25

Create new mdadm.conf

If there is no file your can create a new one which at least assemles your md-arrays on boot.

   1 mdadm --detail --scan >> /etc/mdadm/mdadm.conf

WELL, GOOD LUCK

   1 shutdown -r now

Extending to a hypervisor

Please see libvirt

Some more filesystem

Moving libvirt to own subvolume

   1 ### CREATE A MOUNT POINT
   2 mkdir /media/btrfs5
   3 ### MOUNT ROOT SUBVOLUME TO THIS MOUNT POINT
   4 mount -o rw,noatime,compress=lzo,space_cache,autodefrag,subvolid=5,subvol=/ \
   5         /dev/sda4 /media/btrfs5
   6 ### STOP LIBVIRTD
   7 systemctl stop libvirtd.service
   8 ### CHECK FOR OPEN INODES
   9 lsof /var/lib/libvirt
  10 ### CREATE A SNAPSHOT OF THE FILESYSTEM "root"
  11 ### AND NAME IT "var_lib_libvirt"
  12 btrfs subvolume snapshot \
  13         /media/btrfs5/root \
  14         /media/btrfs5/var_lib_libvirt
  15 ### LIST SUBVOLUMES
  16 btrfs subvol list /media/btrfs5/
  17 ID 258 gen 4933 top level 5 path root
  18 ID 261 gen 1520 top level 5 path home
  19 ID 262 gen 4933 top level 5 path var_log
  20 ID 265 gen 4933 top level 5 path var_lib_libvirt
  21 ### MOUNT FRESHLY CREATED SUBVOLUME TO /var/lib/libvirt
  22 mount -o rw,noatime,compress=lzo,space_cache,autodefrag,subvol=/var_lib_libvirt \
  23         /dev/sda4 /var/lib/libvirt
  24 ### DELETE EVERYTHING BELOW THE NEW MOUNT POINT
  25 ### WHOSE NAME IS NOT "var" RECURSIVELY
  26 find /media/btrfs5/var_lib_libvirt \
  27         -mindepth 1 -maxdepth 1 \! -name var \
  28         |xargs rm -r --
  29 ### MOVE CONTENTS OF THE SUBDIR libvirt
  30 ### TO TOP-LEVEL OF THE SUBVOLUME
  31 mv /media/btrfs5/var_lib_libvirt/var/lib/libvirt/* \
  32         /media/btrfs5/var_lib_libvirt
  33 ### DELETE var IN SUBVOLUME RECURSIVELY
  34 rm -r /media/btrfs5/var_lib_libvirt/var
  35 ### DELETE CONTENT OF libvirt IN THE ROOT-SUBVOLUME RECURSIVELY
  36 rm -r /media/btrfs5/root/var/lib/libvirt/*
  37 ### CHECK OLD DIRECTORY
  38 ll /media/btrfs5/root/var/lib/libvirt/
  39 insgesamt 0
  40 ### CHECK NEW DIRECTORY
  41 ll /media/btrfs5/var_lib_libvirt/
  42 insgesamt 0
  43 drwx--x--x 1 root         root          0 Apr  7 12:36 boot
  44 drwx--x--x 1 root         root          0 Apr  7 12:36 images
  45 drwxr-x--- 1 libvirt-qemu libvirt-qemu 62 Mai 23 10:16 qemu
  46 drwx------ 1 root         root          0 Apr  7 12:36 sanlock
  47 ### MOUNT THE NEW SUBVOLUME TO THE DESTINATION
  48 mount -o rw,noatime,compress=lzo,space_cache,autodefrag,subvol=/var_lib_libvirt \
  49         /dev/sda4 /var/lib/libvirt
  50 ### START LIBVIRTD
  51 systemctl start libvirtd.service

Adjust fstab

   1 # /etc/fstab: static file system information.
   2 #
   3 # Use 'blkid' to print the universally unique identifier for a
   4 # device; this may be used with UUID= as a more robust way to name devices
   5 # that works even if disks are added and removed. See fstab(5).
   6 
   7 #<file_system>                             <mount_point>     <type>  <options>                                                                            <dump>  <pass>
   8 UUID=87294740-52c4-4557-b838-ddc44ba8aa4b  none              swap    sw                                                                                   0       0
   9 UUID=0cc274fd-4cb8-4cc7-9f60-f59de41f6891  /                 btrfs   rw,noatime,compress=lzo,space_cache,autodefrag,subvolid=258,subvol=/root             0       1
  10 #UUID=8AC4-4574                            /boot/EFI         vfat    utf8                                                                                 0       0
  11 #UUID=B3B5-67FA                            /boot/EFI_SDB     vfat    utf8                                                                                 0       0
  12 UUID=0cc274fd-4cb8-4cc7-9f60-f59de41f6891  /var/log          btrfs   rw,noatime,compress=lzo,space_cache,autodefrag,subvolid=262,subvol=/var_log          0       2
  13 UUID=0cc274fd-4cb8-4cc7-9f60-f59de41f6891  /var/lib/libvirt  btrfs   rw,noatime,compress=lzo,space_cache,autodefrag,subvolid=266,subvol=/var_lib_libvirt  0       2
  14 UUID=0cc274fd-4cb8-4cc7-9f60-f59de41f6891  /home             btrfs   rw,noatime,compress=lzo,space_cache,autodefrag,subvolid=261,subvol=/home             0       0
  15 UUID=0cc274fd-4cb8-4cc7-9f60-f59de41f6891  /media/btrfs5     btrfs   rw,noatime,compress=lzo,space_cache,autodefrag,subvolid=5,subvol=/                   0       0

Filesystem maintenance

Please see filesystems/btrfs#Maintenance

ksmtuned

All machines idle? Trade in some CPU and SAVE A WHOLE BUNCH OF MEM!

Redhat: KSM

   1 aptitude install ksmtuned

Information can be aquired under /sys/kernel/mm/ksm

   1 grep -rH "" /sys/kernel/mm/ksm*
   2 echo "Saved mem: $(($(cat /sys/kernel/mm/ksm/pages_sharing)*4/2^1024)) MiB"
   3 
   4 ### Or as watch
   5 watch -n 1 -- '\
   6         grep -rH "" /sys/kernel/mm/ksm*; \
   7         echo "Saved mem: $(($(cat /sys/kernel/mm/ksm/pages_sharing)*4/(2^1024))) MiB"'

Run ksm earlier by lowering KSM_THRES_COEF in
/etc/ksmtuned.conf

   1 # Configuration file for ksmtuned.
   2 
   3 # How long ksmtuned should sleep between tuning adjustments
   4 # KSM_MONITOR_INTERVAL=60
   5 
   6 # Millisecond sleep between ksm scans for 16Gb server.
   7 # Smaller servers sleep more, bigger sleep less.
   8 # KSM_SLEEP_MSEC=10
   9 
  10 # KSM_NPAGES_BOOST=300
  11 # KSM_NPAGES_DECAY=-50
  12 # KSM_NPAGES_MIN=64
  13 # KSM_NPAGES_MAX=1250
  14 
  15 #KSM_THRES_COEF=20
  16 KSM_THRES_COEF=50
  17 # KSM_THRES_CONST=2048
  18 
  19 # uncomment the following if you want ksmtuned debug info
  20 
  21 # LOGFILE=/var/log/ksmtuned
  22 # DEBUG=1

Example - Debian Buster monoculture in production:

   1 Every 1,0s: grep -rH "" /sys/kernel/mm/ksm*;echo "Saved mem: $(($(cat /sys/kernel/mm/ksm/pages_sharing)*4/(2^1024))) MiB"                                                                                                                                              kvm2: Fri Jan 31 14:03:57 2020
   2 
   3 /sys/kernel/mm/ksm/stable_node_dups:5938
   4 /sys/kernel/mm/ksm/max_page_sharing:256
   5 /sys/kernel/mm/ksm/pages_volatile:271474
   6 /sys/kernel/mm/ksm/stable_node_chains_prune_millisecs:2000
   7 /sys/kernel/mm/ksm/merge_across_nodes:1
   8 /sys/kernel/mm/ksm/pages_unshared:5277423
   9 /sys/kernel/mm/ksm/stable_node_chains:497
  10 /sys/kernel/mm/ksm/pages_shared:1049263
  11 /sys/kernel/mm/ksm/use_zero_pages:0
  12 /sys/kernel/mm/ksm/pages_to_scan:1250
  13 /sys/kernel/mm/ksm/sleep_millisecs:10
  14 /sys/kernel/mm/ksm/run:1
  15 /sys/kernel/mm/ksm/full_scans:153
  16 /sys/kernel/mm/ksm/pages_sharing:5635769
  17 Saved mem: 21971 MiB

Well, ~22GiB of 64GiB saved! WTF?!1!! I dare to claim, this is a must have!

I assume KSM doesn't search already swapped pages to save expensive disk io. Maybe it's a good idea to gain some memory in a first run and to move content of the swap space back to the main memory in a second step by umounting and mounting swap-space.

   1 swapoff -a; swapon -a

In addition it may be a good idea to reduce vm.swappiness to once more profit from KSM.

OpenvSwitch

To attach out VMs to the network we can use serveral approaches:

Shared hostbridge
1. classical linux bridge
2. openvswitch
network device passthrough
1. physical device
2. virtual device funtion
macvtap direct connection
1. with vepa
2. without vepa
libvirt networks
1. NAT based network
2. Routed network config
3. Isolated network config
4. Isolated IPv6 network config
5. Network config with no gateway addresses

Since i have no contraints or requirements, i like to use openvswitch as a bridge (as "OpenStack would probably probably use it).

Install

   1 aptitude install openvswitch-switch

Add vSwitch

Create bridge

   1 ovs-vsctl add-br ovs-virt

Add fakebridges

Fake bridges are just virtual sub bridges of a parent bridge, that assign an attached port to a specific vlan in access mode.

   1 ovs-vsctl add-br ovs-1a ovs-virt 1000   #public dmz
   2 ovs-vsctl add-br ovs-1n ovs-virt 1500   #extranet
   3 ovs-vsctl add-br ovs-2a ovs-virt 2000   #intranet
   4 ovs-vsctl add-br ovs-2n ovs-virt 2500   #secure zone
   5 ovs-vsctl add-br ovs-mon1 ovs-virt 3000 #monitoring
   6

Configure new interfaces

Conventional networks

One public interface used:

for general conectivity
in routing the public network to the host and
as a NAT-address for the internal private networks.

/etc/network/interfaces

   1 auto lo
   2 iface lo inet loopback
   3 
   4 ### OUTSIDE
   5 auto enp0s31f6
   6 iface enp0s31f6 inet static
   7         address         195.201.246.253/26
   8         address         2a01:4f8:231:702::2/64
   9         gateway         195.201.246.193
  10         alias           "OUTSIDE"
  11 
  12 ### FAKE BRIDGES
  13 ### VLANS 500-999
  14 auto ovs-pub1
  15 iface ovs-pub1 inet static
  16   address       178.63.149.225/28
  17   alias         "fake-bridge: public dmz - public network"
  18 
  19 ### VLANS 1000-1499
  20 auto ovs-1a
  21 iface ovs-1a inet static
  22   address       172.18.0.1/24
  23   alias         "fake-bridge: public dmz - private network"
  24 
  25 ### VLANS 1500-1999
  26 auto ovs-1n
  27 iface ovs-1n inet static
  28   address       172.18.64.1/24
  29   alias         "fake-bridge: extranet dmz"
  30 
  31 ### VLANS 2000-2499
  32 auto ovs-2a
  33 iface ovs-2a inet static
  34   address       172.18.128.1/24
  35   alias         "fake-bridge: intranet dmz"
  36 
  37 ### VLANS 2500-2999
  38 auto ovs-2n
  39 iface ovs-2n inet static
  40   address       172.18.192.1/24
  41   alias         "fake-bridge: secure zone"
  42 
  43 ### VLANS 3000-3499
  44 auto ovs-mon1
  45 iface ovs-mon1 inet static
  46   address       172.19.255.1/24
  47   alias         "fake-bridge: monitoring"

PPP Guest-Host

With point-to-point connection from guest to host. This is useful:

for single IPs or
to reduce overhead in small networks (/30) originating from network and broadcast addresses.

Please don't get irritated by the last octet of the IP-addresses.

Create a fake-brigde on the host and assign a private IP-address, set a route (for the public address) to the guest and optionally set a static arp entry.

   1 ### external1
   2 auto ovs-ext1
   3 iface ovs-ext1 inet static
   4   address       172.31.255.252
   5   netmask       255.255.255.255
   6   up            ip r add 176.9.99.252/32 dev ovs-ext1
   7   #up           arp -v -H ether -i ovs-ext1 -s 176.9.99.252 52:54:00:aa:bb:cc
   8

In the guest assign the public IP-address and create a point-to-point connection to the (private address of the) host.

   1 auto ens9
   2 iface ens9 inet static
   3         address         176.9.99.252
   4         netmask         255.255.255.255
   5         #broadcast      172.31.255.252
   6         #Note the spelling of pointopoint
   7         pointopoint     172.31.255.252
   8         gateway         172.31.255.252
   9         dns-nameserver  172.31.255.252

Firewalling

   1 aptitude install iptables-persistent ipset iptables fwbuilder

Configure fwbuilder

   1 addgroup firewall
   2 adduser --ingroup firewall --home /etc/fwbuilder --gecos 'remote firewall admin,,,' --system firefighter

   1 cat /etc/sudoers.d/firewall
   2 firefighter     ALL= (root:root) NOPASSWD: /etc/init.d/firewall, /usr/sbin/ipset, /sbin/modprobe, /sbin/iptables

Performance

Laptop

To save power on your laptop and therefore increase time that may be spend on battery, install the laptop-mode-tools. They will e.g. set the cpu-frequency scaling_governor to ondemand and other more conservative options. In my specific case battery usage was reduced by 1/3 and time on battery increased by factor 3/2, which is significant.

   1 aptitude install laptop-mode-tools

They can be configured in /etc/laptop-mode/.

With powertop settings consuming too much power may be configured.

CPU govenour

Another way to adjust cpu-frequency scaling_governor is via cpufrequtils.

/etc/default/cpufrequtils

   1 ENABLE="true"
   2 GOVERNOR="ondemand"
   3 MAX_SPEED="0"
   4 MIN_SPEED="0"

IO-Scheduler

Linux#IO-Scheduler

Security

CPU microcode

Microcode updates are ephemeral: they will be lost after a processor hard reset or after the processor is powered off. They must be reapplied at every boot and after the system wakes up from suspend to RAM or disk.

Depending on your CPU vendor install either of the following:

   1 aptitude install amd64-microcode
   2 aptitude install intel-microcode

To force a microcode update at runtime (on your own risk) run as root.

   1 echo 1 > /sys/devices/system/cpu/microcode/reload

To omit loading of the microcode at boot time add dis_ucode_ldr to your kernel command line in grub menu editor.

   1 linux   /boot/vmlinuz-5.4.0-4-amd64 root=UUID=75258d3e-37f9-42f7-9187-444be692f85d ro quiet dis_ucode_ldr

You may configure tthe microcode packages in

/etc/default/amd64-microcode
/etc/default/intel-microcode

Early loading microcode maybe blacklisted by

/etc/modprobe.d/amd64-microcode
/etc/modprobe.d/intel-microcode

You can get the running microcode revision from /proc/cpuinfo

   1 grep -E 'stepping|model|microcode' /proc/cpuinfo

Compare the latest manufacturer microcode update guidance document.

unattended-upgrades

By default upgrades are installed only from

the current release
security release of the current release

Because versions are frozen in a Debian release only revision upgrades (x.y.z -> x.y.z') are installed. This means you can rely on its operation. There are

no additional or obsolete features
no changes in the API

But it supports security.

Install automatic upgrades

   1 aptitude install unattended-upgrades

You should probably combine unattended-upgrades with apt-listchanges, to receive an email to "root", what and when something has changed. … Just in case.

/etc/apt/apt.conf.d/50unattended-upgrades

   1 // Unattended-Upgrade::Origins-Pattern controls which packages are
   2 // upgraded.
   3 //
   4 // Lines below have the format "keyword=value,...".  A
   5 // package will be upgraded only if the values in its metadata match
   6 // all the supplied keywords in a line.  (In other words, omitted
   7 // keywords are wild cards.) The keywords originate from the Release
   8 // file, but several aliases are accepted.  The accepted keywords are:
   9 //   a,archive,suite (eg, "stable")
  10 //   c,component     (eg, "main", "contrib", "non-free")
  11 //   l,label         (eg, "Debian", "Debian-Security")
  12 //   o,origin        (eg, "Debian", "Unofficial Multimedia Packages")
  13 //   n,codename      (eg, "jessie", "jessie-updates")
  14 //     site          (eg, "http.debian.net")
  15 // The available values on the system are printed by the command
  16 // "apt-cache policy", and can be debugged by running
  17 // "unattended-upgrades -d" and looking at the log file.
  18 //
  19 // Within lines unattended-upgrades allows 2 macros whose values are
  20 // derived from /etc/debian_version:
  21 //   ${distro_id}            Installed origin.
  22 //   ${distro_codename}      Installed codename (eg, "buster")
  23 Unattended-Upgrade::Origins-Pattern {
  24         // Codename based matching:
  25         // This will follow the migration of a release through different
  26         // archives (e.g. from testing to stable and later oldstable).
  27         // Software will be the latest available for the named release,
  28         // but the Debian release itself will not be automatically upgraded.
  29 //      "origin=Debian,codename=${distro_codename}-updates";
  30 //      "origin=Debian,codename=${distro_codename}-proposed-updates";
  31         "origin=Debian,codename=${distro_codename},label=Debian";
  32         "origin=Debian,codename=${distro_codename},label=Debian-Security";
  33         "origin=Debian,codename=${distro_codename}-security,label=Debian-Security";
  34 
  35         // Archive or Suite based matching:
  36         // Note that this will silently match a different release after
  37         // migration to the specified archive (e.g. testing becomes the
  38         // new stable).
  39 //      "o=Debian,a=stable";
  40 //      "o=Debian,a=stable-updates";
  41 //      "o=Debian,a=proposed-updates";
  42 //      "o=Debian Backports,a=${distro_codename}-backports,l=Debian Backports";
  43 };
  44 
  45 // Python regular expressions, matching packages to exclude from upgrading
  46 Unattended-Upgrade::Package-Blacklist {
  47     // The following matches all packages starting with linux-
  48 //  "linux-";
  49 
  50     // Use $ to explicitely define the end of a package name. Without
  51     // the $, "libc6" would match all of them.
  52 //  "libc6$";
  53 //  "libc6-dev$";
  54 //  "libc6-i686$";
  55 
  56     // Special characters need escaping
  57 //  "libstdc\+\+6$";
  58 
  59     // The following matches packages like xen-system-amd64, xen-utils-4.1,
  60     // xenstore-utils and libxenstore3.0
  61 //  "(lib)?xen(store)?";
  62 
  63     // For more information about Python regular expressions, see
  64     // https://docs.python.org/3/howto/regex.html
  65 };
  66 
  67 // This option allows you to control if on a unclean dpkg exit
  68 // unattended-upgrades will automatically run 
  69 //   dpkg --force-confold --configure -a
  70 // The default is true, to ensure updates keep getting installed
  71 //Unattended-Upgrade::AutoFixInterruptedDpkg "true";
  72 
  73 // Split the upgrade into the smallest possible chunks so that
  74 // they can be interrupted with SIGTERM. This makes the upgrade
  75 // a bit slower but it has the benefit that shutdown while a upgrade
  76 // is running is possible (with a small delay)
  77 //Unattended-Upgrade::MinimalSteps "true";
  78 
  79 // Install all updates when the machine is shutting down
  80 // instead of doing it in the background while the machine is running.
  81 // This will (obviously) make shutdown slower.
  82 // Unattended-upgrades increases logind's InhibitDelayMaxSec to 30s.
  83 // This allows more time for unattended-upgrades to shut down gracefully
  84 // or even install a few packages in InstallOnShutdown mode, but is still a
  85 // big step back from the 30 minutes allowed for InstallOnShutdown previously.
  86 // Users enabling InstallOnShutdown mode are advised to increase
  87 // InhibitDelayMaxSec even further, possibly to 30 minutes.
  88 //Unattended-Upgrade::InstallOnShutdown "false";
  89 
  90 // Send email to this address for problems or packages upgrades
  91 // If empty or unset then no email is sent, make sure that you
  92 // have a working mail setup on your system. A package that provides
  93 // 'mailx' must be installed. E.g. "user@example.com"
  94 //Unattended-Upgrade::Mail "";
  95 
  96 // Set this value to one of:
  97 //    "always", "only-on-error" or "on-change"
  98 // If this is not set, then any legacy MailOnlyOnError (boolean) value
  99 // is used to chose between "only-on-error" and "on-change"
 100 //Unattended-Upgrade::MailReport "on-change";
 101 
 102 // Remove unused automatically installed kernel-related packages
 103 // (kernel images, kernel headers and kernel version locked tools).
 104 //Unattended-Upgrade::Remove-Unused-Kernel-Packages "true";
 105 
 106 // Do automatic removal of newly unused dependencies after the upgrade
 107 //Unattended-Upgrade::Remove-New-Unused-Dependencies "true";
 108 
 109 // Do automatic removal of unused packages after the upgrade
 110 // (equivalent to apt-get autoremove)
 111 //Unattended-Upgrade::Remove-Unused-Dependencies "false";
 112 
 113 // Automatically reboot *WITHOUT CONFIRMATION* if
 114 //  the file /var/run/reboot-required is found after the upgrade
 115 //Unattended-Upgrade::Automatic-Reboot "false";
 116 
 117 // Automatically reboot even if there are users currently logged in
 118 // when Unattended-Upgrade::Automatic-Reboot is set to true
 119 //Unattended-Upgrade::Automatic-Reboot-WithUsers "true";
 120 
 121 // If automatic reboot is enabled and needed, reboot at the specific
 122 // time instead of immediately
 123 //  Default: "now"
 124 //Unattended-Upgrade::Automatic-Reboot-Time "02:00";
 125 
 126 // Use apt bandwidth limit feature, this example limits the download
 127 // speed to 70kb/sec
 128 //Acquire::http::Dl-Limit "70";
 129 
 130 // Enable logging to syslog. Default is False
 131 // Unattended-Upgrade::SyslogEnable "false";
 132 
 133 // Specify syslog facility. Default is daemon
 134 // Unattended-Upgrade::SyslogFacility "daemon";
 135 
 136 // Download and install upgrades only on AC power
 137 // (i.e. skip or gracefully stop updates on battery)
 138 // Unattended-Upgrade::OnlyOnACPower "true";
 139 
 140 // Download and install upgrades only on non-metered connection
 141 // (i.e. skip or gracefully stop updates on a metered connection)
 142 // Unattended-Upgrade::Skip-Updates-On-Metered-Connections "true";
 143 
 144 // Verbose logging
 145 // Unattended-Upgrade::Verbose "false";
 146 
 147 // Print debugging information both in unattended-upgrades and
 148 // in unattended-upgrade-shutdown
 149 // Unattended-Upgrade::Debug "false";
 150 
 151 // Allow package downgrade if Pin-Priority exceeds 1000
 152 // Unattended-Upgrade::Allow-downgrade "false";
 153 
 154 // When APT fails to mark a package to be upgraded or installed try adjusting
 155 // candidates of related packages to help APT's resolver in finding a solution
 156 // where the package can be upgraded or installed.
 157 // This is a workaround until APT's resolver is fixed to always find a
 158 // solution if it exists. (See Debian bug #711128.)
 159 // The fallback is enabled by default, except on Debian's sid release because
 160 // uninstallable packages are frequent there.
 161 // Disabling the fallback speeds up unattended-upgrades when there are
 162 // uninstallable packages at the expense of rarely keeping back packages which
 163 // could be upgraded or installed.
 164 // Unattended-Upgrade::Allow-APT-Mark-Fallback "true";

Additional releases

There is only a limited set of Origins-patterns in
/etc/apt/apt.conf.d/50unattended-upgrades

You may also allow unattended-upgrades for additional Releases than defined by default. Take a look at the Release-files in /var/lib/apt/lists/ to extract the relevant Release metadata.

   1 for FILE in /var/lib/apt/lists/*Release; do
   2         echo -e '\n'"$FILE"
   3         grep -h ": " "$FILE"
   4 done |less
   5 
   6 /var/lib/apt/lists/ftp2.de.debian.org_debian_dists_bullseye-backports_InRelease
   7 Hash: SHA256
   8 Origin: Debian Backports
   9 Label: Debian Backports
  10 Suite: bullseye-backports
  11 Codename: bullseye-backports
  12 Changelogs: https://metadata.ftp-master.debian.org/changelogs/@CHANGEPATH@_changelog
  13 Date: Fri, 08 Jan 2021 08:05:41 UTC
  14 Valid-Until: Fri, 15 Jan 2021 08:05:41 UTC
  15 NotAutomatic: yes
  16 ButAutomaticUpgrades: yes
  17 Acquire-By-Hash: yes
  18 No-Support-for-Architecture-all: Packages
  19 Architectures: all amd64 arm64 armel armhf i386 mips64el mipsel ppc64el s390x
  20 Components: main contrib non-free
  21 Description: Debian X.Y - Backports
  22 
  23 …

Available filters:

"o", "origin"
"l", "label"
"a", "suite", "archive"
"c", "component"
"site"
"n", "codename"

Available variables:

${distro_codename} <- lsb_release -c -s
${distro_id} <- lsb_release -i -s

Allow upgrade by creating
/etc/apt/apt.conf.d/51unattended-upgrades

   1 // SOME EXAMPLES
   2 Unattended-Upgrade::Origins-Pattern:: "o=Debian Backports,a=${distro_codename}-backports,l=Debian Backports";
   3 Unattended-Upgrade::Origins-Pattern:: "origin=Debian,codename=${distro_codename}-updates,l=Debian";
   4 // SOME EXAMPLES 3RD-PARTY
   5 Unattended-Upgrade::Origins-Pattern:: "origin=PowerDNS,label=PowerDNS";
   6 Unattended-Upgrade::Origins-Pattern:: "o=http://www.dovecot.org,a=${distro_codename}-auto,l=Automatic Dovecot 2.3 Debian Repository";
   7 Unattended-Upgrade::Origins-Pattern:: "o=debian icinga-${distro_codename},a=icinga-${distro_codename},l=debian icinga-${distro_codename}";
   8 Unattended-Upgrade::Origins-Pattern:: "o=Bareos,n=Debian_10,l=Bareos";
   9 Unattended-Upgrade::Origins-Pattern:: "o=apt.postgresql.org,a=${distro_codename}-pgdg,l=PostgreSQL for Debian/Ubuntu repository";
  10 Unattended-Upgrade::Origins-Pattern:: "o=matrix.org,n=${distro_codename}";
  11 Unattended-Upgrade::Origins-Pattern:: "o=Docker,a=${distro_codename},l=Docker CE";
  12 // CHROME STABLE
  13 Unattended-Upgrade::Origins-Pattern:: "o=Google LLC,a=stable,l=Google";

For Raspbian
/etc/apt/apt.conf.d/51unattended-upgrades

   1 Unattended-Upgrade::Origins-Pattern:: "o=Raspberry Pi Foundation,n=${distro_codename},l=Raspberry Pi Foundation";
   2 Unattended-Upgrade::Origins-Pattern:: "origin=Raspbian,codename=${distro_codename},l=Raspbian";

Check the patterns
apt-config dump Unattended-Upgrade::Origins-Pattern

   1 Unattended-Upgrade::Origins-Pattern "";
   2 Unattended-Upgrade::Origins-Pattern:: "origin=Debian,codename=${distro_codename},label=Debian";
   3 Unattended-Upgrade::Origins-Pattern:: "origin=Debian,codename=${distro_codename},label=Debian-Security";
   4 Unattended-Upgrade::Origins-Pattern:: "origin=PowerDNS,label=PowerDNS";
   5 Unattended-Upgrade::Origins-Pattern:: "o=Debian Backports,a=${distro_codename}-backports,l=Debian Backports";

Additional tunables

/etc/apt/apt.conf.d/51unattended-upgrades

   1 // ALLOW SIGTERM
   2 Unattended-Upgrade::MinimalSteps "True";
   3 // ALLOW UPGRADES WHEN ON BATTERY 
   4 Unattended-Upgrade::OnlyOnACPower "False";
   5 // SKIP UPDATES ON METERED CONNECTIONS
   6 Unattended-Upgrade::Skip-Updates-On-Metered-Connections "True";

There is a hint in verbose mode for checking battery status.

   1 Checking if system is running on battery is skipped.
   2 Please install powermgmt-base package to check power status and
   3 skip installing updates when the system is running on battery.

For battery checking to work install powermgmt-base

   1 apt install powermgmt-base

Debugging

Check ephemeral pinning with

   1 ### FOR VERY VERBOS OUTPUT ADD "--debug"
   2 unattended-upgrades --dry-run -v
   3 unattended-upgrades --dry-run --verbose

You may run unattended-upgrades with debugging output in foreground, to check what it is doing.

   1 unattended-upgrades -d

If it is slow, check if you have

packages on hold apt-mark showhold or
broken packages

To solve the problem use aptitude, it tends to be more powerful in untangling dependency problems.

Check OS Integrity

   1 dpkg -V

debsecan

   1 aptitude install debsecan

Is run automatically via a cronjob, but you should add the parameter --suite

   1 # cron entry for debsecan
   2 MAILTO=root
   3 
   4 4 * * * * daemon test -x /usr/bin/debsecan \
   5         && /usr/bin/debsecan --cron --suite "$(lsb_release -cs)"
   6 # (Note: debsecan delays actual processing past 2:00 AM, and runs only
   7 # once per day.)
   8

Mobile security

Goal is to protect as much as possible as early and strong as possible.

Requirements:

basics:
- bios-password is set
- hdd-password is set
mandatory:
- be maintainable
- perform well
- flexible sizing like in LVM2/btrfs
- include swap, root, …
- protect kernel/initrd
optionally:
- include /boot, means to enter the password a 3rd time in grub2 to unlock /boot
- use trusted platform module (tpm)

The inspiration:

This ultimatively leads to multi-layer full disk encryption (OPAL+LUKS).

ATA hard drive password

ATA Security Feature Set or ATA Security (since ATA-3 (1996–2002, ANSI X3.298-1997))

Tools

hdparm

About

Available on disks with AT Attachment (ATA) storage interface, this includes SATA, SCSI/SAS?, NVMe

"first comer” ownership model
- first that sets the password owns the device
access control mechanism only
32byte master and user keys (NULL-Byte padded)
high or maximum security mode

Mode high

In High security mode, the device can be unlocked with either the User or Master password, using the "SECURITY UNLOCK DEVICE" ATA command. There is an attempt limit, normally set to 5, after which the disk must be power cycled or hard-reset before unlocking can be attempted again. Also in High security mode, the SECURITY ERASE UNIT command can be used with either the User or Master password.

Mode maximum

In Maximum security mode, the device can be unlocked only with the User password. If the User password is not available, the only remaining way to get at least the bare hardware back to a usable state is to issue the SECURITY ERASE PREPARE command, immediately followed by SECURITY ERASE UNIT. In Maximum security mode, the SECURITY ERASE UNIT command requires the Master password and will completely erase all data on the disk. Word 89 in the IDENTIFY response indicates how long the operation will take.

Opal

Before building check if your drive supports Opal.

   1 lspci -vvv

Tools

msed (manage self-encrypting devices) and OpalTool, the two known Open Source code bases available for self-encrypting drives support on Linux, have both been retired, and their development efforts officially merged to form sedutil, under the umbrella of The Drive Trust Alliance (DTA). sedutil is "an Open Source (GPLv3) effort to make Self Encrypting Drive technology freely available to everyone."

https://github.com/Drive-Trust-Alliance/sedutil

Build sedutil

   1 mkdir ~/workspace/
   2 cd ~/workspace
   3 git clone https://github.com/Drive-Trust-Alliance/sedutil.git
   4 cd sedutil
   5 sudo aptitude install build-essential autoconf
   6 autoreconf -i
   7 autoconf
   8 ./configure --prefix=/usr/local
   9 make
  10 #make clean
  11

Scan for drives

   1 % sudo ./sedutil-cli --scan 
   2 [sudo] Passwort für tobias: 
   3 Scanning for Opal compliant disks
   4 /dev/nvme0 No  HFM512GDHTNG-8710B                       80010C00
   5 Unable to verify Kernel flag libata.allow_tpm 
   6 /dev/sda   No  SanDisk pSSD                            6EB 1030
   7 No more disks present ending scan

Yeah, it works and
as the no in column states my current NVMe does not support Opal. :-/ I'll come back once i got a Opal drive.

About

Opal “Family” of specifications:

Opal
Opalite
Pyrite

Found on Arch-Wiki - Self-Encrypting_Drives: Self-encrypting drives adhering to the TCG OPAL 2.0 standard specification (almost all modern self-encrypting drives) implement key management via an authentication key, and a 2nd-level data encryption key, both stored in the disk controller. The data encryption key is the key against which the data is actually encrypted/decrypeted. The authentication key is the user-facing 1st-level password/passphrase which decrypts the data encryption key (which in turn decrypts the data). Data writen to the disk is always encrypted. This approach has specific advantages:

Allows the user to change the passphrase without losing the existing encrypted data on the disk.
- This improves security, as it is fast and easy to respond to security threats and revoke a compromised passphrase
Facilitates near-instant and cryptographically secure full disk erasure.

For those who are familiar; this concept is similar to the LUKS key management layer often used in a dm-crypt deployment. Using LUKS, the user can effectively have up to 8 different key-files / passphrases to decrypt the encryption key, which in turn decrypts the underlying data. This approach allows the user to revoke / change these key-files / passphrases as required without needing to re-encrypt the data, as the 2nd-level encryption key is unchanged (itself being re-encrypted by the new passphrase).

In fact, in drives featuring full-disk encryption, data is always encrypted with the data encryption key when stored to disk, even if there is no password set (e.g. a new drive). Manufacturers do this to make it easier for users who are not able to, or do not wish to enable the security features of the self-encrypting drive. This can be thought of as all drives by default having a zero-length password that transparently encrypts/decrypts the data always (similar to how passwordless SSH keys provide (somewhat) secure access without user intervention).

The key point to note is that if at a later stage the user wishes to "enable" encryption, they can configure the passphrase (authentication key), which will then be used to encrypt the existing data encryption key (thus prompting for passphrase before decrypting the data encryption key in future). However, as the existing data encryption key will not be changed (regenerated), this in effect locks the drive, while preserving the existing encrypted data on the disk.

Advantages:

Easier to setup (compared to software-based encryption)
Notably transparent to the user, except for initial bootup authentication
Data-at-Rest protection
Increased performance (CPU is freed up from encryption/decryption calculations)
The main CPU and RAM are eliminated as possible attack targets
Optimally fast and #Secure disk erasure (sanitation) (regardless of disk size)
Protection from alternative boot methods due to the possibility to encrypt the MBR, rendering the drive inaccessible before pre-boot authentication

Disadvantages:

Constant-power exploits:
Typical self-encrypting drives, once unlocked, will remain unlocked as long as power is provided. This vulnerability can be exploited by means of altering the environment external to the drive, without cutting power, in effect keeping the drive in an unlocked state. For example, it has been shown (by researchers at University of Erlangen-Nuremberg) that it is possible to reboot the computer into an attacker-controlled operating system without cutting power to the drive. The researchers have also demonstrated moving the drive to another computer without cutting power.[1]
Key-in-memory exploits:
When the system is powered down into S3 ("sleep") mode, the drive is powered down, but the drive keeps access to the encryption key in its internal memory (NVRAM) to allow for a resume ("wake"). This is necessary because for system booted with an arbitrary operating system there is no standard mechanism to prompt the user to re-enter the pre-boot decryption passphrase again. An attacker (with physical access to the drive) can leverage this to access the drive. Taking together known exploits the researchers summarize "we were able to break hardware-based full-disk encryption on eleven [of twelve] of those systems provided they were running or in standby mode".[2] Note, however, S3 ("sleep") is not currently supported by sedutil (the current available toolset for managing a TCG OPAL 2.0 self-encrypting drives via Linux)
Compromised firmware:
The firmware of the drive may be compromised (backdoor) and data sent to it thus potentially compromised (decryptable by the malicious third party in question, provided access to physical drive is achievable). A study demonstrated methods for compromising device firmware, as well as applying invalid passwords to access data on OPAL devices.[3] If data is encrypted by the operating system (e.g. dm-crypt), the encryption key is unknown to the compromised drive, thus circumventing this attack vector entirely.

Found in White Paper - Storage Opal and NVMe

requires AES-128 or AES-256
hardware-based encrpytion that may be scaled to meet the bandwidth of the storage device.
credentials
- 1-4 admin for provisioning, configuration or erasure
- 2-8 user to perform various actions
subdivision of the storage device into multiple locking ranges of contiguous LBAs
each locking range
- has different media encryption key (MEK)
- is unlocked independently
- is erased independently (by destruction of media encryption key and generation of a new one)
n>=0 users may
- unlock locking ranges
- erase locking ranges
fast and reliable erasure of locking ranges
supports MBR-shadowding, through which a host-application can store and execute a “Pre-Boot Authentication Environment”. Such a mechanism is necessary to allow unlock of the range in which the OS is stored, in order to allow the OS to boot.

Storage glossary

advanced encryption standard (AES)
authentication key (AK)
advanced technology attachment (ATA) -> P-ATA
AT attachment packet interface (ATAPI)
data encryption key (DEK)
full disk encryption (FDE)
full encryption disks (FED), self-encrypting HDD
hard drive disk (HDD)
integrated drive electronics (IDE)
logical block addressing (LBA), number of blocks starting with zero (size with 512byte blocks)
- 28bit LBA (128GiB)
- 48bit LBA (128PiB)
- SCSI Command Descriptor Block (CDB)
  - 10Byte CDB with 4byte (32bit) LBA (2TiB)
  - 16/32Byte CDB with 8byte (64bit) Long-LBA (16EiB)
non-volatile memory express (NVMe)
opal security subsystem class (SSC)
Self-Monitoring, Analysis and Reporting Technology (S.M.A.R.T.)
storage device (SD)
self-encrypting drive (SED)
small computer system interface (SCSI)
trusted computing group (TCG)

LUKS

Tools

   1 apt install gdisk parted cryptsetup cryptsetup-initramfs dosfstools xfsprogs

About

The Linux Unified Key Setup (LUKS) is a disk encryption specification created by Clemens Fruhwirth in 2004 and was originally intended for Linux.

Why LUKS?

compatibility via standardization,
secure against low entropy attacks,
support for multiple keys,
effective passphrase revocation,
free.

Transform

Target configuration

   1 NAME                    FSTYPE      LABEL  FSAVAIL FSUSE% MOUNTPOINT
   2 nvme0n1
   3 ├─nvme0n1p1             vfat               
   4 ├─nvme0n1p2             vfat        EFI    213.8M    16% /mnt/boot/efi
   5 ├─nvme0n1p3
   6 └─nvme0n1p4             crypto_LUKS
   7   └─crypt1              LVM2_member
   8     ├─vg_crypt-lv_swap1 swap        swap1  
   9     └─vg_crypt-lv_root  xfs         rootfs 355.7G    14% /mnt

FDE walkthrough

Perform a backup Preparation

   1 ### MOUNT A REMOTE STORAGE (e.g. NFS4)
   2 mount libertas:/media/space/tmp /media/external1
   3 
   4 ### AND BACKUP PARTITIONING TABLE (JUST INCASE)
   5 mkdir /media/external1/backup
   6 sgdisk --backup=/media/external1/backup/nvme0n1.sgdisk /dev/nvme0n1
   7 The operation has completed successfully.
   8 ### TO RESTORE SIMPLY
   9 # sgdisk -l /media/external1/backup/nvme0n1 /dev/nvme0n1
  10

Partitioning

   1 ### watch -n1 -- 'lsblk -f; echo; blkid'
   2 
   3 parted /dev/nvme0n1
   4 unit MiB
   5 print free
   6 mktable gpt
   7 mkpart bios_grub 1 2
   8 mkpart EFI 2 256
   9 mkpart boot1 256 1024
  10 mkpart crypt1 4 -1
  11 set 1 bios_grub on
  12 set 2 esp on
  13 set 2 boot on
  14 print free
  15 quit

Grub2 disk: Implement support for LUKS2:

With cryptsetup 2.0, a new version of LUKS was introduced that breaks compatibility with the previous version due to various reasons. GRUB currently lacks any support for LUKS2, making it impossible to decrypt disks encrypted with that version. This commit implements support for this new format.

This commit has not landed in Debian, yet. So we need to downgrade header to LUKS version 1 and the password based key derivation functions (PBKDF) to PBKDF2 (from Argon2i or Argon2id).

See also the documentation of the cryptsetup-team of Debian.net

Fresh cryptsetup

   1 cryptsetup luksFormat --type luks1 /dev/nvme0n1p4
   2 cryptsetup open /dev/nvme0n1p4 crypt1

To downgrade manually from LUKS2 to version 1

   1 cryptsetup luksDump /dev/nvme0n1p4 \
   2         | grep -i -e version -e PBKDF *
   3 cryptsetup luksConvertKey --pbkdf pbkdf2 /dev/nvme0n1p4
   4 Enter passphrase for keyslot to be converted: 
   5 cryptsetup convert --type luks1 /dev/nvme0n1p4

LVM2

   1 pvcreate /dev/mapper/crypt1
   2 vgcreate vg_crypt /dev/mapper/crypt1
   3 lvcreate -L 64g -n lv_swap1 vg_crypt
   4 lvcreate -l 100%FREE -n lv_root vg_crypt
   5 lvrename vg_crypt lv_root1 lv_root

Formating filesystems and swap

   1 mkfs.vfat -n EFI /dev/nvme0n1p2
   2 mkfs.xfs -L rootfs /dev/vg_crypt/lv_root
   3 mkswap -L swap1 /dev/vg_crypt/lv_swap1
   4 mount /dev/vg_crypt/lv_root /mnt
   5 mkdir /mnt/boot/efi
   6 mount /dev/nvme0n1p2 /mnt/boot/efi

Restore the data from backup.
I created the tar-archive without changing the directory, so i have to strip "mnt/" away at extraction-time using --strip-components=1.

   1 ### RESTORE ROOTFS
   2 MOUNT_POINT="/mnt"
   3 ssh user@target.host \
   4         "cat path/to/archive.tar.gz" \
   5         |pigz -dc \
   6         |tar -xf - --strip-components=1 -C "$MOUNT_POINT"
   7 ### RESTORE EFIFS
   8 MOUNT_POINT="/mnt/boot/efi"
   9 ssh user@target.host \
  10         "cat path/to/archive_efi.tar.gz" \
  11         |pigz -dc \
  12         |tar -xf - --strip-components=1 -C "$MOUNT_POINT"

   1 grml-chroot /mnt
   2 blkid |sed 's/^/#/' >> /etc/fstab
   3 ### EDIT /etc/fstab
   4 mount -a

   1 aptitude install cryptsetup cryptsetup-initramfs

Change swap uuid to new value

   1 blkid /dev/nvme0n1p2 \
   2         |sed 's/^/#/' >> /etc/initramfs-tools/conf.d/resume

/etc/initramfs-tools/conf.d/resume

   1 RESUME=UUID=3c8d7d58-c524-4a74-94f6-ec66a3bb07af

Adjust crypttab

   1 blkid /dev/nvme0n1p4 |sed 's/^/#/' >> /etc/crypttab

/etc/crypttab

   1 #<target_name>  <source_device>                            <key_file>  <options>
   2 crypt1          UUID=cb58fb88-fc1e-4102-bfdb-9d943b1f01db  none        luks,discard

/etc/default/grub

   1 # If you change this file, run 'update-grub' afterwards to update
   2 # /boot/grub/grub.cfg.
   3 # For full documentation of the options in this file, see:
   4 #   info -f grub -n 'Simple configuration'
   5 
   6 GRUB_DEFAULT=0
   7 GRUB_TIMEOUT=5
   8 GRUB_DISTRIBUTOR=`lsb_release -i -s 2> /dev/null || echo Debian`
   9 GRUB_CMDLINE_LINUX_DEFAULT="quiet"
  10 GRUB_CMDLINE_LINUX=""
  11 
  12 # Uncomment to enable BadRAM filtering, modify to suit your needs
  13 # This works with Linux (no patch required) and with any kernel that obtains
  14 # the memory map information from GRUB (GNU Mach, kernel of FreeBSD ...)
  15 #GRUB_BADRAM="0x01234567,0xfefefefe,0x89abcdef,0xefefefef"
  16 
  17 # Uncomment to disable graphical terminal (grub-pc only)
  18 #GRUB_TERMINAL=console
  19 
  20 # The resolution used on graphical terminal
  21 # note that you can use only modes which your graphic card supports via VBE
  22 # you can see them in real GRUB with the command `vbeinfo'
  23 #GRUB_GFXMODE=640x480
  24 
  25 # Uncomment if you don't want GRUB to pass "root=UUID=xxx" parameter to Linux
  26 #GRUB_DISABLE_LINUX_UUID=true
  27 
  28 # Uncomment to disable generation of recovery mode menu entries
  29 #GRUB_DISABLE_RECOVERY="true"
  30 
  31 # Uncomment to get a beep at grub start
  32 #GRUB_INIT_TUNE="480 440 1"
  33 
  34 ### GRUB-INSTALL DEMANDS IT
  35 GRUB_ENABLE_CRYPTODISK=y
  36 #GRUB_PRELOAD_MODULES="lvm cryptodisk mdraid1x luks"
  37

   1 update-initramfs -k all -u
   2 grub-install /dev/nvme0n1
   3 update-grub2

Unmount remote storage
Reboot

Logitech devices

Install solaar

   1 aptitude install solaar solaar-gnome3

Add users to the group plugdev

   1 adduser "$USERNAME" plugdev

Grub-customizer

Themes are available at:

Install grub-customizer

   1 aptitude install grub-customizer

Clone the theme

   1 git clone "$GITURL" /opt

Start grub-customizer (enter your password) for the privileges. Go to tab appreance:

set resolution
choose the theme and save

Plymouth

Install bootsplash "plymouth"

   1 aptitude install plymouth plymouth-themes plymouth-x11

Add splash to GRUB_CMDLINE_LINUX_DEFAULT
/etc/default/grub

   1 GRUB_CMDLINE_LINUX_DEFAULT="quiet splash"

   1 update-initramfs -k all -u
   2 update-grub2

List and set available themes

   1 plymouth-set-default-theme --list

Set other theme and update grub and initramfs

   1 sudo plymouth-set-default-theme -R lines

Get new themes from store.kde.org.
Just unpack them to /usr/share/plymouth/themes

Preview plymouth themes in X11

Inspired by this blogpost

plymouth_preview.sh

   1 #!/bin/bash
   2 
   3 ## Preview Plymouth Splash ##
   4 ##      by _khAttAm_       ##
   5 ##    www.khattam.info     ##
   6 ##    License: GPL v3      ##
   7 
   8 chk_root () {
   9   if [ ! $( id -u ) -eq 0 ]; then
  10     echo "Must be run as root"
  11     exit
  12   fi
  13 }
  14 
  15 chk_root
  16 
  17 DURATION=$1
  18 if [ $# -ne 1 ]; then
  19   DURATION=5
  20 fi
  21 
  22 #CURRENT_THEME=
  23 THEMES="$(plymouth-set-default-theme --list \
  24         |sort |uniq)"
  25 
  26 while read THEME; do
  27         plymouth-set-default-theme "$THEME"
  28         echo "$THEME"
  29         sleep 1.5
  30         plymouthd
  31         plymouth --show-splash
  32         for ((I=0; I<$DURATION; I++)); do
  33                 plymouth --update=test$I;
  34                 sleep 1;
  35         done
  36         plymouth quit
  37 done <<< "$THEMES"

Run the preview

   1 chmod a+x ./plymouth_preview.sh
   2 sudo ./plymouth_preview.sh

Convertible with Gnome3

On-screen keyboard

Install the some extensions for an on-screen display

Disable keyboard by script

/etc/sudoers.d/keyboard

   1 Cmnd_Alias KEYBOARD = /usr/local/bin/keyboard_toggle_wayland.sh
   2 %tablet ALL = NOPASSWD: KEYBOARD

/usr/local/bin/keyboard_toggle_wayland.sh

   1 #!/bin/bash
   2 
   3 DEVICE='AT Translated Set 2 keyboard'
   4 DEVICE_PATH="$(libinput list-devices \
   5         |sed -e "1,/$DEVICE/d" \
   6         |grep '^Kernel:'\
   7         |awk '{print $2}')"
   8 
   9 COMMAND="evtest --grab $DEVICE_PATH"
  10 
  11 declare -a PIDS
  12 readarray -t PIDS < <(ps -eo pid,args \
  13         |grep -v grep\
  14         |grep " $COMMAND$" \
  15         |awk '{print $1}')
  16 
  17 if [ "${#PIDS[@]}" -gt "0" ]; then
  18         echo "LAPTOP MODE: enabling keyboard input."
  19         kill "${PIDS[@]}"
  20 else
  21         echo "TABLET MODE: disabling keyboard input."
  22         nohup $COMMAND >/dev/null 2>&1 &
  23 fi

/usr/share/applications/keyboard.desktop

   1 [Desktop Entry]
   2 Type=Application
   3 Version=1.0
   4 Name=Laptop/Tablet
   5 GenericName=Toggle Laptop/Tablet Mode
   6 GenericName[de]=Laptop/Tablet Mode umschalten
   7 Comment=Show System Processes
   8 Icon=input-keyboard
   9 Exec=sudo keyboard_toggle_wayland.sh
  10 Terminal=true
  11 Categories=System;ConsoleOnly;Settings
  12 Keywords=keyboard;input;laptop;tablet

Now the script shows up in the menu and can be be dragged into the favorites menu.

Automatic screen rotation

If you have iio-sensor-proxy installed automatic screen rotation should work automagically with xorg.
gitlab.freedesktop.org hadess/iio-sensor-proxy

However, wayland is not there yet …

Keys in gsettings

If the gsettings-backend is in dconf you may take alook at the values by installing dconf-editor

   1 apt install dconf-editor

Or just use the cli gsettings

   1 gsettings get org.gnome.settings-daemon.peripherals.touchscreen orientation-lock
   2 gsettings get org.gnome.settings-daemon.plugins.orientation active

Debian CD-image with jigdo

Jigsaw Download, or short jigdo, is a tool designed to ease the distribution of very large files over the internet, for example CD or DVD images. Its aim is to make downloading the images as easy for users as a click on a direct download link in a browser, while avoiding all the problems that server administrators have with hosting such large files.

   1 aptitude install jigdo-file jigit

Example: Create a CD-image of Debian Jessie

Create a configuration for jigdo-lite to reliefe primary Debian mirrors. ~/.jigdo-lite

   1 debianMirror='http://debian.inf.tu-dresden.de/debian/'
   2 nonusMirror='http://debian.inf.tu-dresden.de/debian/'

   1 mkdir jessie
   2 cd jessie
   3 jigdo-lite --noask 'https://cdimage.debian.org/cdimage/archive/8.0.0/amd64/jigdo-cd/debian-8.0.0-amd64-netinst.jigdo'

Done:

   1 jigdo-lite --noask 'https://cdimage.debian.org/cdimage/archive/8.0.0/amd64/jigdo-cd/debian-8.0.0-amd64-netinst.jigdo'
   2 
   3 Jigsaw Download "lite"
   4 Copyright (C) 2001-2005  |  jigdo@
   5 Richard Atterer          |  atterer.org
   6 Loading settings from `/home/tobias/.jigdo-lite'
   7 
   8 Downloading .jigdo file
   9 --2019-11-05 11:47:20--  https://cdimage.debian.org/cdimage/archive/8.0.0/amd64/jigdo-cd/debian-8.0.0-amd64-netinst.jigdo
  10 Auflösen des Hostnamens cdimage.debian.org (cdimage.debian.org)… 194.71.11.173, 194.71.11.165, 2001:6b0:19::165, ...
  11 Verbindungsaufbau zu cdimage.debian.org (cdimage.debian.org)|194.71.11.173|:443 … verbunden.
  12 HTTP-Anforderung gesendet, auf Antwort wird gewartet … 200 OK
  13 Länge: 24660 (24K)
  14 Wird in »debian-8.0.0-amd64-netinst.jigdo« gespeichert.
  15 
  16 debian-8.0.0-amd64-netinst.jigdo                                          100%[==================================================================================================================================================================================>]  24.08K  --.-KB/s    in 0.06s   
  17 
  18 2019-11-05 11:47:20 (428 KB/s) - »debian-8.0.0-amd64-netinst.jigdo« gespeichert [24660/24660]
  19 
  20 
  21 -----------------------------------------------------------------
  22 Images offered by `https://cdimage.debian.org/cdimage/archive/8.0.0/amd64/jigdo-cd/debian-8.0.0-amd64-netinst.jigdo':
  23   1: 'Debian GNU/Linux 8.0.0 "Jessie" - Official amd64 NETINST Binary-1 20150425-12:50 (20150425)' (debian-8.0.0-amd64-netinst.iso)
  24 
  25 Further information about `debian-8.0.0-amd64-netinst.iso':
  26 Generated on Sat, 25 Apr 2015 12:53:05 +0000
  27 
  28 -----------------------------------------------------------------
  29 If you already have a previous version of the CD you are
  30 downloading, jigdo can re-use files on the old CD that are also
  31 present in the new image, and you do not need to download them
  32 again. Mount the old CD ROM and enter the path it is mounted under
  33 (e.g. `/mnt/cdrom').
  34 Alternatively, just press enter if you want to start downloading
  35 the remaining files.
  36 Files to scan: 
  37 
  38 -----------------------------------------------------------------
  39 The jigdo file refers to files stored on Debian mirrors. Please
  40 choose a Debian mirror as follows: Either enter a complete URL
  41 pointing to a mirror (in the form
  42 `ftp://ftp.debian.org/debian/'), or enter any regular expression
  43 for searching through the list of mirrors: Try a two-letter
  44 country code such as `de', or a country name like `United
  45 States', or a server name like `sunsite'.
  46 Debian mirror [http://debian.inf.tu-dresden.de/debian/]: 
  47 
  48 Downloading .template file
  49 --2019-11-05 11:47:20--  https://cdimage.debian.org/cdimage/archive/8.0.0/amd64/jigdo-cd/debian-8.0.0-amd64-netinst.template
  50 Auflösen des Hostnamens cdimage.debian.org (cdimage.debian.org)… 194.71.11.165, 194.71.11.173, 2001:6b0:19::173, ...
  51 Verbindungsaufbau zu cdimage.debian.org (cdimage.debian.org)|194.71.11.165|:443 … verbunden.
  52 HTTP-Anforderung gesendet, auf Antwort wird gewartet … 302 Found
  53 Platz: https://saimei.ftp.acc.umu.se/cdimage/archive/8.0.0/amd64/jigdo-cd/debian-8.0.0-amd64-netinst.template [folgend]
  54 --2019-11-05 11:47:20--  https://saimei.ftp.acc.umu.se/cdimage/archive/8.0.0/amd64/jigdo-cd/debian-8.0.0-amd64-netinst.template
  55 Auflösen des Hostnamens saimei.ftp.acc.umu.se (saimei.ftp.acc.umu.se)… 194.71.11.138, 2001:6b0:19::138
  56 Verbindungsaufbau zu saimei.ftp.acc.umu.se (saimei.ftp.acc.umu.se)|194.71.11.138|:443 … verbunden.
  57 HTTP-Anforderung gesendet, auf Antwort wird gewartet … 200 OK
  58 Länge: 8641689 (8.2M)
  59 Wird in »debian-8.0.0-amd64-netinst.template« gespeichert.
  60 
  61 debian-8.0.0-amd64-netinst.template                                       100%[=========================================================>]   8.24M  1.11MB/s    in 7.5s    
  62 
  63 2019-11-05 11:47:28 (1.10 MB/s) - »debian-8.0.0-amd64-netinst.template« gespeichert [8641689/8641689]
  64 
  65 
  66 -----------------------------------------------------------------
  67 Merging parts from `file:' URIs, if any...
  68 0 der 813 vom Template benötigten Dateien gefunden
  69 Es wird keine Image-Datei oder temporäre Datei erzeugt - versuchen Sie es mit anderen Eingabedateien
  70 --2019-11-05 11:47:28--  http://debian.inf.tu-dresden.de/debian/pool/main/s/systemd/libpam-systemd_215-17_amd64.deb
  71 Auflösen des Hostnamens debian.inf.tu-dresden.de (debian.inf.tu-dresden.de)… 141.76.2.4
  72 Verbindungsaufbau zu debian.inf.tu-dresden.de (debian.inf.tu-dresden.de)|141.76.2.4|:80 … verbunden.
  73 HTTP-Anforderung gesendet, auf Antwort wird gewartet … 404 Not Found
  74 2019-11-05 11:47:28 FEHLER 404: Not Found.
  75 
  76 --2019-11-05 11:47:28--  http://debian.inf.tu-dresden.de/debian/pool/main/j/jfsutils/jfsutils_1.1.15-2.1_amd64.deb
  77 Wiederverwendung der bestehenden Verbindung zu debian.inf.tu-dresden.de:80.
  78 HTTP-Anforderung gesendet, auf Antwort wird gewartet … 200 OK
  79 Länge: 285760 (279K) [application/x-debian-package]
  80 Wird in »./debian-8.0.0-amd64-netinst.iso.tmpdir/debian.inf.tu-dresden.de/debian/pool/main/j/jfsutils/jfsutils_1.1.15-2.1_amd64.deb« gespeichert.
  81 
  82 debian.inf.tu-dresden.de/debian/pool/main/j/jfsutils/jfsutils_1.1.15-2.1_ 100%[=========================================================>] 279.06K   894KB/s    in 0.3s    
  83 
  84 2019-11-05 11:47:29 (894 KB/s) - »./debian-8.0.0-amd64-netinst.iso.tmpdir/debian.inf.tu-dresden.de/debian/pool/main/j/jfsutils/jfsutils_1.1.15-2.1_amd64.deb« gespeichert [285760/285760]
  85 
  86 --2019-11-05 11:47:29--  http://debian.inf.tu-dresden.de/debian/pool/main/l/less/less_458-3_amd64.deb
  87 Wiederverwendung der bestehenden Verbindung zu debian.inf.tu-dresden.de:80.
  88 HTTP-Anforderung gesendet, auf Antwort wird gewartet … 200 OK
  89 Länge: 124466 (122K) [application/x-debian-package]
  90 Wird in »./debian-8.0.0-amd64-netinst.iso.tmpdir/debian.inf.tu-dresden.de/debian/pool/main/l/less/less_458-3_amd64.deb« gespeichert.
  91 
  92 debian.inf.tu-dresden.de/debian/pool/main/l/less/less_458-3_amd64.deb     100%[=========================================================>] 121.55K  --.-KB/s    in 0.1s    
  93 
  94 2019-11-05 11:47:29 (1.13 MB/s) - »./debian-8.0.0-amd64-netinst.iso.tmpdir/debian.inf.tu-dresden.de/debian/pool/main/l/less/less_458-3_amd64.deb« gespeichert [124466/124466]
  95 
  96 
  97 <… OUTPUT OMITED …>
  98 
  99 
 100 --2019-11-05 11:52:06--  http://snapshot.debian.org/archive/debian/20150425T124850Z/pool/main/l/linux/loop-modules-3.16.0-4-amd64-di_3.16.7-ckt9-2_amd64.udeb
 101 Verbindungsaufbau zu snapshot.debian.org (snapshot.debian.org)|193.62.202.27|:80 … verbunden.
 102 HTTP-Anforderung gesendet, auf Antwort wird gewartet … 200 OK
 103 Länge: 13294 (13K)
 104 Wird in »./debian-8.0.0-amd64-netinst.iso.tmpdir/snapshot.debian.org/archive/debian/20150425T124850Z/pool/main/l/linux/loop-modules-3.16.0-4-amd64-di_3.16.7-ckt9-2_amd64.udeb« gespeichert.
 105 
 106 snapshot.debian.org/archive/debian/20150425T124850Z/pool/main/l/linux/loo 100%[=========================================================>]  12.98K  --.-KB/s    in 0.01s   
 107 
 108 2019-11-05 11:52:06 (1.23 MB/s) - »./debian-8.0.0-amd64-netinst.iso.tmpdir/snapshot.debian.org/archive/debian/20150425T124850Z/pool/main/l/linux/loop-modules-3.16.0-4-amd64-di_3.16.7-ckt9-2_amd64.udeb« gespeichert [13294/13294]
 109 
 110 BEENDET --2019-11-05 11:52:06--
 111 Verstrichene Zeit: 6.3s
 112 Geholt: 10 Dateien, 5.0M in 4.7s (1.07 MB/s)
 113 10 der 11 vom Template benötigten Dateien gefunden                                                                                                                      '                                    
 114 Eingabedateien wurden in temporäre Datei »debian-8.0.0-amd64-netinst.iso.tmp« geschrieben - wiederholen Sie das Kommando mit weiteren Dateien
 115 --2019-11-05 11:52:06--  http://snapshot.debian.org/archive/debian/20150425T124850Z/pool/main/l/linux/linux-headers-3.16.0-4-common_3.16.7-ckt9-2_amd64.deb
 116 Auflösen des Hostnamens snapshot.debian.org (snapshot.debian.org)… 185.17.185.185, 193.62.202.27, 2001:630:206:4000:1a1a:0:c13e:ca1b, ...
 117 Verbindungsaufbau zu snapshot.debian.org (snapshot.debian.org)|185.17.185.185|:80 … verbunden.
 118 HTTP-Anforderung gesendet, auf Antwort wird gewartet … 200 OK
 119 Länge: 4480432 (4.3M)
 120 Wird in »./debian-8.0.0-amd64-netinst.iso.tmpdir/snapshot.debian.org/archive/debian/20150425T124850Z/pool/main/l/linux/linux-headers-3.16.0-4-common_3.16.7-ckt9-2_amd64.deb« gespeichert.
 121 
 122 snapshot.debian.org/archive/debian/20150425T124850Z/pool/main/l/linux/lin 100%[=========================================================>]   4.27M  1.11MB/s    in 3.9s    
 123 
 124 2019-11-05 11:52:10 (1.11 MB/s) - »./debian-8.0.0-amd64-netinst.iso.tmpdir/snapshot.debian.org/archive/debian/20150425T124850Z/pool/main/l/linux/linux-headers-3.16.0-4-common_3.16.7-ckt9-2_amd64.deb« gespeichert [4480432/4480432]
 125 
 126 1 der 1 vom Template benötigten Dateien gefunden                                                                                                                                                  '
 127 »debian-8.0.0-amd64-netinst.iso« wurde erfolgreich erzeugt
 128 
 129 -----------------------------------------------------------------
 130 Finished!
 131 The fact that you got this far is a strong indication that `debian-8.0.0-amd64-netinst.iso'
 132 was generated correctly. I will perform an additional, final check,
 133 which you can interrupt safely with Ctrl-C if you do not want to wait.
 134 
 135 OK: Prüfsummen stimmen überein, Image-Datei ist in Ordnung!
 136 jigdo-lite --noask   3.90s user 7.87s system 4% cpu 4:50.47 total

This worked out fine. Now i can test upgrading "oldoldstable" to "stable". Installation succeded.

Aptitude

This cli and gui interface adds in my option some essential features to the apt ecosystem.

It offers search patterns that are very useful.

https://www.debian.org/doc/manuals/aptitude/ch02s04s05.en.html#tableSearchTermQuickGuide

Example: identify installed packages from "Debian Backports". Using the shorthands for:

?installed, ~i

    Matches package versions which are currently installed.

    Since all versions are tested by default, this normally
    matches packages which are currently installed.

?narrow(filter, pattern), ~S filter pattern

    This term “narrows” the search to package versions matching filter.
    In particular, it matches any package version which matches both filter and pattern.
    The string value of the match is the string value of pattern.

?origin(origin), ~Oorigin

    Matches package versions whose origin matches the regular expression origin.
    For instance, “!?origin(debian)” will find any unofficial packages on your system
    (packages not from the Debian archive).

   1 aptitude search '~S ~i ~O"Debian Backports"'
   2 aptitude search '~S ~i ~O"Debian Backports"' \
   3         |sed 's/^i [A ]\+//;s/ - .*//'

netcfg.cfg

To force a downgrade, set the priority of the stable codename above 990 (to force install). Example preference for downgrade:

   1 Package:        *
   2 Pin:            release n=buster
   3 Pin-Priority:   1000

Open aptitude and search for ~Vbpo to identify once again all packages from backports and select the packages to be downgraded by pressing +. They will be marked as to be downgraded i W in organge.

Don't forget to remove the preferences afterwards.

Preseeding

It's not a bad idea to have some knowledge about preseeding Debian via the installer, even if you are using a new smart and fancy technology of a higher level. In the end it probably boils down to pxe and preseeding/kickstart.

You may want to end up in having

the configuration management client installed and ready
and a user that is able to connect to the server.

Extract preseed information

Install a Debian VM manually (as kind of a template).

Extract preseed information from this VM. This alleviates some decisions.

   1 apt install debconf-utils
   2 
   3 PRESEED="preseed-decisions.cfg"
   4 echo "#_preseed_V1" > "$PRESEED"
   5 echo '### INSTALLER DEBCONF DATABASE' >> "$PRESEED"
   6 debconf-get-selections --installer >> "$PRESEED"
   7 echo '### CURRENT DEBCONF DATABASE' >> "$PRESEED"
   8 debconf-get-selections >> "$PRESEED"

Formatting

Filter your export and format it a bit more readable

   1 ### FORMAT DEBCONF TO TABLE
   2 grep -v '\s*#' preseed-decisions.cfg \
   3         |sort |uniq \
   4         |column -t -s'INSERT_RAW_TAB_WITH CTRL+V TAB' \
   5         --table-columns PACKAGE,KEY,TYPE,VALUE --table-noheadings \
   6         > preseed_table.cfg

Revert the intermediate format to the original export format with tabs, if you like …

   1 ### FORMAT TABLE TO DEBCONF
   2 sed -r 's/ +/\t/;s/ +/\t/;s/ +/\t/' \
   3         < preseed_table.cfg \
   4         > preseed.cfg

Serve the preseed via webserver

Serve preseed.cfg

   1 apt install apache2
   2 install -d /var/www/preseed
   3 cat ~tobias/.ssh/id_rsa.pub >> /var/www/preseed/authorized_keys

/etc/apache2/sites-available/preseed.conf

   1 <VirtualHost *:80>
   2     ServerAdmin webmaster@rockstable.it
   3 
   4     DocumentRoot /var/www/html
   5 
   6     # Available loglevels: trace8, ..., trace1, debug, info, notice, warn,
   7     # error, crit, alert, emerg.
   8     # It is also possible to configure the loglevel for particular
   9     # modules, e.g.
  10     #LogLevel info ssl:warn
  11 
  12     ErrorLog ${APACHE_LOG_DIR}/preseed_error.log
  13     CustomLog ${APACHE_LOG_DIR}/preseed_access.log combined
  14 
  15     Alias /ps /var/www/preseed
  16     Alias /preseed /var/www/preseed
  17     <Directory /var/www/preseed>
  18         Require all granted
  19         Options +Indexes
  20     </Directory>
  21 </VirtualHost>
  22 
  23 # vim: syntax=apache ts=4 sw=4 sts=4 sr noet
  24

Enable preseed.conf

   1 a2ensite preseed
   2 apache2ctl configtest
   3 systemctl reload apache2.service

For conveniance i created a symlink during testing.

   1 ln ../preseed/bullseye-fs1.cfg fs1

Custom configuration

Start with the example preconfiguration file (for bullseye)

Prepare a directory structure

   1 mkdir -p /var/www/preseed/example
   2 cd /var/www/preseed
   3 wget -p example 'https://www.debian.org/releases/testing/example-preseed.txt'
   4 wget -p example 'https://preseed.debian.net/debian-preseed/bullseye/amd64-main-full.txt'
   5 cp example/example-preseed.txt base-preseed.cfg
   6 touch {netcfg,partman,passwd}.cfg

After a change concatenate all files to the final config.

   1 ### A VERBOSE VERSION (26K)
   2 cat base-preseed.cfg netcfg.cfg passwd.cfg partman.cfg > bullseye.cfg
   3 ### OR FOR A MORE CLEAN VERSION (~3.7K)
   4 grep -h '^[^#]' base-preseed.cfg netcfg.cfg passwd.cfg partman.cfg > bullseye.cfg

Base preseed

/var/www/preseed/base-preseed.cfg

   1 #_preseed_V1
   2 #### Contents of the preconfiguration file (for bullseye)
   3 d-i      anna/choose_modules     multiselect     choose-mirror, network-console, parted-udeb
   4 
   5 ### Localization
   6 # Preseeding only locale sets language, country and locale.
   7 #d-i debian-installer/locale string en_US
   8 
   9 
  10 # The values can also be preseeded individually for greater flexibility.
  11 d-i debian-installer/language string de
  12 d-i debian-installer/country string DE
  13 d-i debian-installer/locale string en_US.UTF-8
  14 # Optionally specify additional locales to be generated.
  15 #d-i localechooser/supported-locales multiselect en_US.UTF-8, nl_NL.UTF-8
  16 
  17 # Keyboard selection.
  18 d-i keyboard-configuration/xkb-keymap select de
  19 # d-i keyboard-configuration/toggle select No toggling
  20 
  21 ### Network configuration
  22 # Disable network configuration entirely. This is useful for cdrom
  23 # installations on non-networked devices where the network questions,
  24 # warning and long timeouts are a nuisance.
  25 #d-i netcfg/enable boolean false
  26 
  27 # netcfg will choose an interface that has link if possible. This makes it
  28 # skip displaying a list if there is more than one interface.
  29 d-i netcfg/choose_interface select auto
  30 
  31 # To pick a particular interface instead:
  32 #d-i netcfg/choose_interface select eth1
  33 
  34 # To set a different link detection timeout (default is 3 seconds).
  35 # Values are interpreted as seconds.
  36 #d-i netcfg/link_wait_timeout string 10
  37 
  38 # If you have a slow dhcp server and the installer times out waiting for
  39 # it, this might be useful.
  40 #d-i netcfg/dhcp_timeout string 60
  41 #d-i netcfg/dhcpv6_timeout string 60
  42 
  43 # If you prefer to configure the network manually, uncomment this line and
  44 # the static network configuration below.
  45 #d-i netcfg/disable_autoconfig boolean true
  46 
  47 # If you want the preconfiguration file to work on systems both with and
  48 # without a dhcp server, uncomment these lines and the static network
  49 # configuration below.
  50 #d-i netcfg/dhcp_failed note
  51 #d-i netcfg/dhcp_options select Configure network manually
  52 
  53 # Static network configuration.
  54 #
  55 # IPv4 example
  56 #d-i netcfg/get_ipaddress string 192.168.1.42
  57 #d-i netcfg/get_netmask string 255.255.255.0
  58 #d-i netcfg/get_gateway string 192.168.1.1
  59 #d-i netcfg/get_nameservers string 192.168.1.1
  60 #d-i netcfg/confirm_static boolean true
  61 #
  62 # IPv6 example
  63 #d-i netcfg/get_ipaddress string fc00::2
  64 #d-i netcfg/get_netmask string ffff:ffff:ffff:ffff::
  65 #d-i netcfg/get_gateway string fc00::1
  66 #d-i netcfg/get_nameservers string fc00::1
  67 #d-i netcfg/confirm_static boolean true
  68 
  69 # Any hostname and domain names assigned from dhcp take precedence over
  70 # values set here. However, setting the values still prevents the questions
  71 # from being shown, even if values come from dhcp.
  72 #d-i netcfg/get_hostname string unassigned-hostname
  73 #d-i netcfg/get_domain string unassigned-domain
  74 
  75 # If you want to force a hostname, regardless of what either the DHCP
  76 # server returns or what the reverse DNS entry for the IP is, uncomment
  77 # and adjust the following line.
  78 #d-i netcfg/hostname string somehost
  79 
  80 # Disable that annoying WEP key dialog.
  81 d-i netcfg/wireless_wep string
  82 # The wacky dhcp hostname that some ISPs use as a password of sorts.
  83 #d-i netcfg/dhcp_hostname string radish
  84 
  85 # If non-free firmware is needed for the network or other hardware, you can
  86 # configure the installer to always try to load it, without prompting. Or
  87 # change to false to disable asking.
  88 #d-i hw-detect/load_firmware boolean true
  89 
  90 ### Network console
  91 # Use the following settings if you wish to make use of the network-console
  92 # component for remote installation over SSH. This only makes sense if you
  93 # intend to perform the remainder of the installation manually.
  94 #d-i anna/choose_modules string network-console
  95 #d-i network-console/authorized_keys_url string http://10.0.0.1/openssh-key
  96 #d-i network-console/password password r00tme
  97 #d-i network-console/password-again password r00tme
  98 
  99 ### Mirror settings
 100 # If you select ftp, the mirror/country string does not need to be set.
 101 #d-i mirror/protocol string ftp
 102 d-i mirror/country string manual
 103 d-i mirror/http/hostname string deb.debian.org
 104 d-i mirror/http/directory string /debian
 105 d-i mirror/http/proxy   string  http://192.168.110.1:3142/
 106 
 107 # Suite to install.
 108 #d-i mirror/suite string testing
 109 # Suite to use for loading installer components (optional).
 110 #d-i mirror/udeb/suite string testing
 111 
 112 ### Account setup
 113 # Skip creation of a root account (normal user account will be able to
 114 # use sudo).
 115 #d-i passwd/root-login boolean false
 116 # Alternatively, to skip creation of a normal user account.
 117 #d-i passwd/make-user boolean false
 118 
 119 # Root password, either in clear text
 120 #d-i passwd/root-password password r00tme
 121 #d-i passwd/root-password-again password r00tme
 122 # or encrypted using a crypt(3)  hash.
 123 #d-i passwd/root-password-crypted password [crypt(3) hash]
 124 
 125 # To create a normal user account.
 126 #d-i passwd/user-fullname string Debian User
 127 #d-i passwd/username string debian
 128 # Normal user's password, either in clear text
 129 #d-i passwd/user-password password insecure
 130 #d-i passwd/user-password-again password insecure
 131 # or encrypted using a crypt(3) hash.
 132 #d-i passwd/user-password-crypted password [crypt(3) hash]
 133 # Create the first user with the specified UID instead of the default.
 134 #d-i passwd/user-uid string 1010
 135 
 136 # The user account will be added to some standard initial groups. To
 137 # override that, use this.
 138 #d-i passwd/user-default-groups string audio cdrom video
 139 
 140 ### Clock and time zone setup
 141 # Controls whether or not the hardware clock is set to UTC.
 142 d-i clock-setup/utc boolean true
 143 
 144 # You may set this to any valid setting for $TZ; see the contents of
 145 # /usr/share/zoneinfo/ for valid values.
 146 d-i time/zone string Europe/Berlin
 147 
 148 # Controls whether to use NTP to set the clock during the install
 149 d-i clock-setup/ntp boolean true
 150 # NTP server to use. The default is almost always fine here.
 151 #d-i clock-setup/ntp-server string ntp.example.com
 152 
 153 ### Partitioning
 154 ## Partitioning example
 155 # If the system has free space you can choose to only partition that space.
 156 # This is only honoured if partman-auto/method (below) is not set.
 157 #d-i partman-auto/init_automatically_partition select biggest_free
 158 
 159 # Alternatively, you may specify a disk to partition. If the system has only
 160 # one disk the installer will default to using that, but otherwise the device
 161 # name must be given in traditional, non-devfs format (so e.g. /dev/sda
 162 # and not e.g. /dev/discs/disc0/disc).
 163 # For example, to use the first SCSI/SATA hard disk:
 164 #d-i partman-auto/disk string /dev/sda
 165 # In addition, you'll need to specify the method to use.
 166 # The presently available methods are:
 167 # - regular: use the usual partition types for your architecture
 168 # - lvm:     use LVM to partition the disk
 169 # - crypto:  use LVM within an encrypted partition
 170 #d-i partman-auto/method string lvm
 171 
 172 # You can define the amount of space that will be used for the LVM volume
 173 # group. It can either be a size with its unit (eg. 20 GB), a percentage of
 174 # free space or the 'max' keyword.
 175 #d-i partman-auto-lvm/guided_size string max
 176 
 177 # If one of the disks that are going to be automatically partitioned
 178 # contains an old LVM configuration, the user will normally receive a
 179 # warning. This can be preseeded away...
 180 #d-i partman-lvm/device_remove_lvm boolean true
 181 # The same applies to pre-existing software RAID array:
 182 #d-i partman-md/device_remove_md boolean true
 183 # And the same goes for the confirmation to write the lvm partitions.
 184 #d-i partman-lvm/confirm boolean true
 185 #d-i partman-lvm/confirm_nooverwrite boolean true
 186 
 187 # You can choose one of the three predefined partitioning recipes:
 188 # - atomic: all files in one partition
 189 # - home:   separate /home partition
 190 # - multi:  separate /home, /var, and /tmp partitions
 191 #d-i partman-auto/choose_recipe select atomic
 192 
 193 # Or provide a recipe of your own...
 194 # If you have a way to get a recipe file into the d-i environment, you can
 195 # just point at it.
 196 #d-i partman-auto/expert_recipe_file string /hd-media/recipe
 197 
 198 # If not, you can put an entire recipe into the preconfiguration file in one
 199 # (logical) line. This example creates a small /boot partition, suitable
 200 # swap, and uses the rest of the space for the root partition:
 201 #d-i partman-auto/expert_recipe string                         \
 202 #      boot-root ::                                            \
 203 #              40 50 100 ext3                                  \
 204 #                      $primary{ } $bootable{ }                \
 205 #                      method{ format } format{ }              \
 206 #                      use_filesystem{ } filesystem{ ext3 }    \
 207 #                      mountpoint{ /boot }                     \
 208 #              .                                               \
 209 #              500 10000 1000000000 ext3                       \
 210 #                      method{ format } format{ }              \
 211 #                      use_filesystem{ } filesystem{ ext3 }    \
 212 #                      mountpoint{ / }                         \
 213 #              .                                               \
 214 #              64 512 300% linux-swap                          \
 215 #                      method{ swap } format{ }                \
 216 #              .
 217 
 218 # The full recipe format is documented in the file partman-auto-recipe.txt
 219 # included in the 'debian-installer' package or available from D-I source
 220 # repository. This also documents how to specify settings such as file
 221 # system labels, volume group names and which physical devices to include
 222 # in a volume group.
 223 
 224 ## Partitioning for EFI
 225 # If your system needs an EFI partition you could add something like
 226 # this to the recipe above, as the first element in the recipe:
 227 #               538 538 1075 free                              \
 228 #                      $iflabel{ gpt }                         \
 229 #                      $reusemethod{ }                         \
 230 #                      method{ efi }                           \
 231 #                      format{ }                               \
 232 #               .                                              \
 233 #
 234 # The fragment above is for the amd64 architecture; the details may be
 235 # different on other architectures. The 'partman-auto' package in the
 236 # D-I source repository may have an example you can follow.
 237 
 238 # This makes partman automatically partition without confirmation, provided
 239 # that you told it what to do using one of the methods above.
 240 #d-i partman-partitioning/confirm_write_new_label boolean true
 241 #d-i partman/choose_partition select finish
 242 #d-i partman/confirm boolean true
 243 #d-i partman/confirm_nooverwrite boolean true
 244 
 245 # Force UEFI booting ('BIOS compatibility' will be lost). Default: false.
 246 #d-i partman-efi/non_efi_system boolean true
 247 # Ensure the partition table is GPT - this is required for EFI
 248 #d-i partman-partitioning/choose_label string gpt
 249 #d-i partman-partitioning/default_label string gpt
 250 
 251 # When disk encryption is enabled, skip wiping the partitions beforehand.
 252 #d-i partman-auto-crypto/erase_disks boolean false
 253 
 254 ## Partitioning using RAID
 255 # The method should be set to "raid".
 256 #d-i partman-auto/method string raid
 257 # Specify the disks to be partitioned. They will all get the same layout,
 258 # so this will only work if the disks are the same size.
 259 #d-i partman-auto/disk string /dev/sda /dev/sdb
 260 
 261 # Next you need to specify the physical partitions that will be used. 
 262 #d-i partman-auto/expert_recipe string \
 263 #      multiraid ::                                         \
 264 #              1000 5000 4000 raid                          \
 265 #                      $primary{ } method{ raid }           \
 266 #              .                                            \
 267 #              64 512 300% raid                             \
 268 #                      method{ raid }                       \
 269 #              .                                            \
 270 #              500 10000 1000000000 raid                    \
 271 #                      method{ raid }                       \
 272 #              .
 273 
 274 # Last you need to specify how the previously defined partitions will be
 275 # used in the RAID setup. Remember to use the correct partition numbers
 276 # for logical partitions. RAID levels 0, 1, 5, 6 and 10 are supported;
 277 # devices are separated using "#".
 278 # Parameters are:
 279 # <raidtype> <devcount> <sparecount> <fstype> <mountpoint> \
 280 #          <devices> <sparedevices>
 281 
 282 #d-i partman-auto-raid/recipe string \
 283 #    1 2 0 ext3 /                    \
 284 #          /dev/sda1#/dev/sdb1       \
 285 #    .                               \
 286 #    1 2 0 swap -                    \
 287 #          /dev/sda5#/dev/sdb5       \
 288 #    .                               \
 289 #    0 2 0 ext3 /home                \
 290 #          /dev/sda6#/dev/sdb6       \
 291 #    .
 292 
 293 # For additional information see the file partman-auto-raid-recipe.txt
 294 # included in the 'debian-installer' package or available from D-I source
 295 # repository.
 296 
 297 # This makes partman automatically partition without confirmation.
 298 #d-i partman-md/confirm boolean true
 299 #d-i partman-partitioning/confirm_write_new_label boolean true
 300 #d-i partman/choose_partition select finish
 301 #d-i partman/confirm boolean true
 302 #d-i partman/confirm_nooverwrite boolean true
 303 
 304 ## Controlling how partitions are mounted
 305 # The default is to mount by UUID, but you can also choose "traditional" to
 306 # use traditional device names, or "label" to try filesystem labels before
 307 # falling back to UUIDs.
 308 #d-i partman/mount_style select uuid
 309 
 310 ### Base system installation
 311 # Configure APT to not install recommended packages by default. Use of this
 312 # option can result in an incomplete system and should only be used by very
 313 # experienced users.
 314 #d-i base-installer/install-recommends boolean false
 315 
 316 # The kernel image (meta) package to be installed; "none" can be used if no
 317 # kernel is to be installed.
 318 d-i base-installer/kernel/image string linux-image-amd64
 319 
 320 ### Apt setup
 321 # You can choose to install non-free and contrib software.
 322 #d-i apt-setup/non-free boolean true
 323 #d-i apt-setup/contrib boolean true
 324 # Uncomment this if you don't want to use a network mirror.
 325 #d-i apt-setup/use_mirror boolean false
 326 # Select which update services to use; define the mirrors to be used.
 327 # Values shown below are the normal defaults.
 328 #d-i apt-setup/services-select multiselect security, updates
 329 #d-i apt-setup/security_host string security.debian.org
 330 
 331 # Additional repositories, local[0-9] available
 332 #d-i apt-setup/local0/repository string \
 333 #       http://local.server/debian stable main
 334 #d-i apt-setup/local0/comment string local server
 335 # Enable deb-src lines
 336 #d-i apt-setup/local0/source boolean true
 337 # URL to the public key of the local repository; you must provide a key or
 338 # apt will complain about the unauthenticated repository and so the
 339 # sources.list line will be left commented out.
 340 #d-i apt-setup/local0/key string http://local.server/key
 341 # If the provided key file ends in ".asc" the key file needs to be an
 342 # ASCII-armoured PGP key, if it ends in ".gpg" it needs to use the
 343 # "GPG key public keyring" format, the "keybox database" format is
 344 # currently not supported.
 345 
 346 # By default the installer requires that repositories be authenticated
 347 # using a known gpg key. This setting can be used to disable that
 348 # authentication. Warning: Insecure, not recommended.
 349 #d-i debian-installer/allow_unauthenticated boolean true
 350 
 351 # Uncomment this to add multiarch configuration for i386
 352 #d-i apt-setup/multiarch string i386
 353 
 354 
 355 ### Package selection
 356 #tasksel tasksel/first multiselect standard, web-server, kde-desktop
 357 #d-i tasksel/first      multiselect     SSH server, standard system utilities
 358 #d-i tasksel/first   multiselect     ssh-server, standard
 359 taskel tasksel/first   multiselect     ssh-server, standard
 360 
 361 # Individual additional packages to install
 362 d-i pkgsel/include string \
 363         apt-file aptitude bash-completion byobu btrfs-progs ca-certificates curl \
 364         dmidecode dosfstools git gpm htop iftop info iotop jq libcrack2 locales \
 365         lsb-release lsof man-db mc mlocate openssl parted pigz psmisc pv \
 366         pwgen python3-apt rsync screen sqlite3 ssl-cert strace sudo \
 367         sysstat tmux vim wget zsh
 368 # Whether to upgrade packages after debootstrap.
 369 # Allowed values: none, safe-upgrade, full-upgrade
 370 d-i pkgsel/upgrade select full-upgrade
 371 
 372 # Some versions of the installer can report back on what software you have
 373 # installed, and what software you use. The default is not to report back,
 374 # but sending reports helps the project determine what software is most
 375 # popular and should be included on the first CD/DVD.
 376 popularity-contest popularity-contest/participate boolean true
 377 
 378 ### Boot loader installation
 379 # Grub is the boot loader (for x86).
 380 
 381 # This is fairly safe to set, it makes grub install automatically to the UEFI
 382 # partition/boot record if no other operating system is detected on the machine.
 383 d-i grub-installer/only_debian boolean true
 384 
 385 # This one makes grub-installer install to the UEFI partition/boot record, if
 386 # it also finds some other OS, which is less safe as it might not be able to
 387 # boot that other OS.
 388 d-i grub-installer/with_other_os boolean true
 389 
 390 # Due notably to potential USB sticks, the location of the primary drive can
 391 # not be determined safely in general, so this needs to be specified:
 392 #d-i grub-installer/bootdev  string /dev/sda
 393 # To install to the primary device (assuming it is not a USB stick):
 394 d-i grub-installer/bootdev  string default
 395 #d-i  grub-installer/choose_bootdev   select  /dev/vda
 396 
 397 # Alternatively, if you want to install to a location other than the UEFI
 398 # parition/boot record, uncomment and edit these lines:
 399 #d-i grub-installer/only_debian boolean false
 400 #d-i grub-installer/with_other_os boolean false
 401 #d-i grub-installer/bootdev  string (hd0,1)
 402 # To install grub to multiple disks:
 403 #d-i grub-installer/bootdev  string (hd0,1) (hd1,1) (hd2,1)
 404 
 405 # Optional password for grub, either in clear text
 406 #d-i grub-installer/password password r00tme
 407 #d-i grub-installer/password-again password r00tme
 408 # or encrypted using an MD5 hash, see grub-md5-crypt(8).
 409 #d-i grub-installer/password-crypted password [MD5 hash]
 410 
 411 # Use the following option to add additional boot parameters for the
 412 # installed system (if supported by the bootloader installer).
 413 # Note: options passed to the installer will be added automatically.
 414 d-i debian-installer/add-kernel-opts string zswap.enabled=1
 415 
 416 
 417 ### Finishing up the installation
 418 # During installations from serial console, the regular virtual consoles
 419 # (VT1-VT6) are normally disabled in /etc/inittab. Uncomment the next
 420 # line to prevent this.
 421 #d-i finish-install/keep-consoles boolean true
 422 
 423 # Avoid that last message about the install being complete.
 424 d-i finish-install/reboot_in_progress note
 425 
 426 # This will prevent the installer from ejecting the CD during the reboot,
 427 # which is useful in some situations.
 428 #d-i cdrom-detect/eject boolean false
 429 
 430 # This is how to make the installer shutdown when finished, but not
 431 # reboot into the installed system.
 432 #d-i debian-installer/exit/halt boolean true
 433 # This will power off the machine instead of just halting it.
 434 #d-i debian-installer/exit/poweroff boolean true
 435 
 436 ### Preseeding other packages
 437 # Depending on what software you choose to install, or if things go wrong
 438 # during the installation process, it's possible that other questions may
 439 # be asked. You can preseed those too, of course. To get a list of every
 440 # possible question that could be asked during an install, do an
 441 # installation, and then run these commands:
 442 #   debconf-get-selections --installer > file
 443 #   debconf-get-selections >> file
 444 
 445 
 446 #### Advanced options
 447 ### Running custom commands during the installation
 448 # d-i preseeding is inherently not secure. Nothing in the installer checks
 449 # for attempts at buffer overflows or other exploits of the values of a
 450 # preconfiguration file like this one. Only use preconfiguration files from
 451 # trusted locations! To drive that home, and because it's generally useful,
 452 # here's a way to run any shell command you'd like inside the installer,
 453 # automatically.
 454 
 455 # This first command is run as early as possible, just after
 456 # preseeding is read.
 457 #d-i preseed/early_command string anna-install some-udeb
 458 # This command is run immediately before the partitioner starts. It may be
 459 # useful to apply dynamic partitioner preseeding that depends on the state
 460 # of the disks (which may not be visible when preseed/early_command runs).
 461 #d-i partman/early_command \
 462 #       string debconf-set partman-auto/disk "$(list-devices disk | head -n1)"
 463 # This command is run just before the install finishes, but when there is
 464 # still a usable /target directory. You can chroot to /target and use it
 465 # directly, or use the apt-install and in-target commands to easily install
 466 # packages and run commands in the target system.
 467 #d-i preseed/late_command string apt-install zsh; in-target chsh -s /bin/zsh
 468 
 469 
 470 ### CUSTOM
 471 # Restart services during package upgrades without asking?
 472 d-i libraries/restart-without-asking    boolean true
 473 d-i libpam0g/restart-services           string  cron
 474 #d-i glibc/restart-services             string
 475 #d-i libssl1.1/restart-services         string
 476 libc6           libraries/restart-without-asking        boolean false
 477 #libc6:amd64    libraries/restart-without-asking        boolean false
 478 #libpam0g:amd64 libraries/restart-without-asking        boolean false
 479

Netcfg

/var/www/preseed/netcfg.cfg

   1 #d-i     anna/choose_modules                  string    network-console
   2 d-i      network-console/authorized_keys_url  string    http://192.168.110.1/preseed/authorized_keys
   3 #d-i     network-console/password password    r00tme
   4 #d-i     network-console/password password    r00tme
   5 #d-i     network-console/password-again       password  r00tme
   6 #d-i     netcfg/choose_interface              select
   7 d-i      netcfg/confirm_static                boolean   true
   8 #d-i     netcfg/dhcp_failed                   note
   9 #d-i     netcfg/dhcp_hostname                 string
  10 d-i      netcfg/dhcp_options                  select    Configure network manually
  11 d-i      netcfg/dhcp_timeout                  string    25
  12 d-i      netcfg/dhcpv6_timeout                string    15
  13 d-i      netcfg/disable_autoconfig            boolean   false
  14 d-i      netcfg/disable_dhcp                  boolean   false
  15 d-i      netcfg/enable                        boolean   true
  16 #d-i     netcfg/error                         error
  17 #d-i     netcfg/gateway_unreachable           error
  18 #d-i     netcfg/get_gateway                   string
  19 #d-i     netcfg/get_ipaddress                 string
  20 #d-i     netcfg/get_nameservers               string
  21 #d-i     netcfg/get_netmask                   string
  22 #d-i     netcfg/get_pointopoint               string
  23 #d-i     netcfg/hostname                      string
  24 #d-i     netcfg/invalid_essid                 error
  25 #d-i     netcfg/invalid_hostname              error
  26 #d-i     netcfg/invalid_pass                  error
  27 #d-i     netcfg/invalid_wep                   error
  28 #d-i     netcfg/kill_switch_enabled           note
  29 d-i      netcfg/link_wait_timeout             string    3
  30 #d-i     netcfg/no_default_route              boolean
  31 #d-i     netcfg/no_dhcp_client                error
  32 #d-i     netcfg/no_interfaces                 error
  33 #d-i     netcfg/no_ipv6_pointopoint           error
  34 d-i      netcfg/target_network_config         select    ifupdown
  35 d-i      netcfg/use_autoconfig                boolean   true
  36 d-i      netcfg/wireless_adhoc_managed        select    Infrastructure (Managed) network
  37 #d-i     netcfg/wireless_essid_again          string
  38 #d-i     netcfg/wireless_essid                string
  39 d-i      netcfg/wireless_security_type        select    wpa
  40 #d-i     netcfg/wireless_show_essids          select
  41 #d-i     netcfg/wireless_wep                  string
  42 #d-i     netcfg/wireless_wpa                  string
  43 #d-i     netcfg/wpa_supplicant_failed         note
  44 
  45 ### MAKE NETCFG RUN AGAIN WHEN PRESEED-FILE WAS LOADED OVER NETWORK
  46 #d-i    preseed/run           string http://192.168.110.1/ps/restart-netcfg.sh
  47 d-i     preseed/early_command string kill-all-dhcp; netcfg

Network config over the network

When you are loading the preseeding from the network, netcfg has already done its magic. But it's not lost yet, you can force network configuration to run again from within the preseed file and this time use the preseeded values.

Command solution

If you are using preseeding over the network you can modify this a bit and use a preseed/early_command, which can be

   1 d-i     preseed/early_command string kill-all-dhcp; netcfg

If it is really necessary to load a shell script use a more complex command to retrieve and execute the script.

   1 wget --no-proxy -p /tmp "$URL" ; \
   2 sh /tmp/schript.sh

Debian solution

My recommendation is to use the #Command solution

There is a trick from described in
Debian GNU/Linux Installation Guide - Network configuration.

Create a shell-script to be called by the preseed configuration.

/var/www/preseed/restart-netcfg.sh

   1 kill-all-dhcp; netcfg

And add this line to
/var/www/preseed/netcfg.cfg

   1 ### MAKE NETCFG RUN AGAIN WHEN PRESEED-FILE WAS LOADED OVER NETWORK
   2 #d-i preseed/run string http://192.168.110.1/ps/restart-netcfg.sh
   3 d-i preseed/early_command string kill-all-dhcp; netcfg

But a shell script of mime-type text/x-sh cannot be loaded<<br>> when d-i mirror/http/proxy is set to an apt-cacher-ng server. Because the shell environment variable http_proxy is set to this url and apt-cacher-ng refuses to server this file, which is right in IMHO.

There are several ancient bugs out there:

Hostname and Domainname

Names are assigned to the machine via DHCP or reverse DNS. So make sure you have registered a static lease for the MAC-address.

Alternativly create
/var/www/preseed/dc2.cfg

And generate a host-specific configuration

   1 grep -h '^[^#]' bullseye.cfg dc2.cfg > bullseye-dc2.cfg

Passwd

The hashes have been generated using a binary from the package whois

   1 mkpasswd -m sha-512
   2 Passwort: 
   3 $6$.6YAwaP96KHXMzqg$0v5CIxUulkM1sbKcbqKVaSlunSB3G6M7E6/X8ExLcb7jK754jbmFMY2gDPo22oSYb2KgBjGZi20J1lP2pWBFQ.

Providing secrets this way is in evitably insecure. Please change it after preseeding.

/var/www/preseed/passwd.cfg

   1 ### Account setup
   2 # Skip creation of a root account (normal user account will be able to
   3 # use sudo).
   4 #d-i passwd/root-login boolean false
   5 # Alternatively, to skip creation of a normal user account.
   6 #d-i passwd/make-user boolean false
   7 
   8 # Root password, either in clear text
   9 #d-i passwd/root-password password r00tme
  10 #d-i passwd/root-password-again password r00tme
  11 # or encrypted using a crypt(3)  hash.
  12 #d-i passwd/root-password-crypted password [crypt(3) hash]
  13 #d-i passwd/root-password-crypted  password $(mkpasswd -m sha512crypt)
  14 d-i passwd/root-password-crypted  password  $6$tAw5x69a6TedGlSA$4V7oY/zC6Q4PylHk0kP0zRvL9KRnTgz4n379elsUAcEftxnHf2JW3qlUXKDIIcMjNuS31vglQ72bUqT3EUEsx1
  15 
  16 # To create a normal user account.
  17 d-i passwd/user-fullname string Tobias Stein
  18 d-i passwd/username string tobias
  19 # Normal user's password, either in clear text
  20 #d-i passwd/user-password password insecure
  21 #d-i passwd/user-password-again password insecure
  22 # or encrypted using a crypt(3) hash.
  23 #d-i passwd/user-password-crypted password [crypt(3) hash]
  24 d-i passwd/user-password-crypted  password  $6$.6YAwaP96KHXMzqg$0v5CIxUulkM1sbKcbqKVaSlunSB3G6M7E6/X8ExLcb7jK754jbmFMY2gDPo22oSYb2KgBjGZi20J1lP2pWBFQ.
  25 # Create the first user with the specified UID instead of the default.
  26 #d-i passwd/user-uid string 1010
  27 
  28 # The user account will be added to some standard initial groups. To
  29 # override that, use this.
  30 #d-i passwd/user-default-groups string audio cdrom video
  31

Partman receipes

partman cannot preseed a partition label. Filesystem labels are possible.

The documentation is sparse, but it's open source and if you know where to search …

There are some recipes for partman in the source package of the debian-installer. They contain the EBNF of a partman recipe.

   1 mkdir -p workspace/debian-installer
   2 cd workspace/debian-installer
   3 apt-get source debian-installer
   4 find . -name 'partman-auto-*'
   5 ./debian-installer-20201202/doc/devel/partman-auto-raid-recipe.txt
   6 ./debian-installer-20201202/doc/devel/partman-auto-recipe.txt

There is a build.sh for a dokbook html documentation in
./debian-installer-20201202/doc/devel/partman/build.sh

Finally: the source is documentation enough.

   1 mkdir partman
   2 cd partman
   3 for PKG in partman-{base,auto,basicfilesystems,basicmethods,btrfs,partitioning}; do
   4         mkdir "$PKG"
   5         cd "$PKG"
   6         apt-get source "$PKG"
   7         cd ..
   8 done

In partman-base/lib/base.sh a variable DEVICES is defined, which points to a directory /var/lib/partman/devices. This leads to a filesystem-interface-tree, which is used to interchange information between the partman modules and is helpful in debugging.

/var/www/preseed/partman.cfg

   1 d-i     partman-auto/method             string  regular
   2 #d-i    partman-auto/choose_recipe      select
   3 
   4 
   5 # Force UEFI booting ('BIOS compatibility' will be lost). Default: false.
   6 #d-i partman-efi/non_efi_system boolean true
   7 # Ensure the partition table is GPT - this is required for EFI
   8 d-i     partman-partitioning/choose_label       string  gpt
   9 d-i     partman-partitioning/default_label      string  gpt
  10 
  11 
  12 d-i     partman/default_filesystem      string          btrfs
  13 d-i     partman-auto/expert_recipe      string          \
  14         partman-auto/text/grub_efi_swap_default_scheme :: \
  15                 1 1 1 free                              \
  16                         $iflabel{ gpt }                 \
  17                         $reusemethod{ }                 \
  18                         method{ biosgrub }              \
  19                 .                                       \
  20                 64 96 128 fat32                         \
  21                         $iflabel{ gpt }                 \
  22                         use_filesystem{ }               \
  23                         method{ efi }                   \
  24                         format{ }                       \
  25                 .                                       \
  26                 100% 2048 200% linux-swap               \
  27                         $lvmok{ }                       \
  28                         $reusemethod{ }                 \
  29                         method{ swap }                  \
  30                         format{ }                       \
  31                 .                                       \
  32                 2048 4096 -1 $default_filesystem        \
  33                         $lvmok{ }                       \
  34                         method{ format }                \
  35                         format{ }                       \
  36                         use_filesystem{ }               \
  37                         $default_filesystem{ }          \
  38                         options/noatime{ noatime }      \
  39                         label{ rootfs }                 \
  40                         mountpoint{ / }                 \
  41                 .
  42 # This makes partman automatically partition without confirmation,
  43 # provided that you told it what to do using one of the methods above.
  44 d-i partman-partitioning/confirm_write_new_label boolean true
  45 d-i partman/choose_partition select finish
  46 d-i partman/confirm boolean true
  47 d-i partman/confirm_nooverwrite boolean true

Use the preseeding

When the config is joined and is available at
http://localhost/ps/bullseye.cfg

In the Debian installer add some custom bootloader options (press ESC)

   1 auto url=http://webserver.domain.tld/ps/bullseye.cfg

There are several mechanisms like Foreman that may automate the task. But this is a story for another day.

LVM

https://www.tldp.org/LDP/sag/html/partitions.html

Debian

About

Releases

Base system from scratch

Partitioning

Debian partman units

Create the partitions on the first disk

Clone GPT to other disk

Create a multidisk RAID1 as swap

Filesystems

Migrate data to subvolumes

Prepare EFI filesystems

fstab

Make sure to use FS-UUIDS and no devices in fstab

Edit fstab to reflect new structure of filesystem

Validate fstab

Chroot into new system

Basic Configuration

Adjust hostname and mailname

Adjust hosts

Adjust timezone

Configure apt-sources

Pinning: apt_preferences

Some notes regarding preferences

Add some essential packages

Configure locales

Configure vim

zsh grml-flavoured

Networking

ifupdown vs. ifupdown2

Cons

Pros

Install ifupdown2

Predictable device names

On Upgrade to buster

Configure interfaces

Check the configuration of the DNS-resolver

Install openssh-server

Make sure ssh-server starts on boot

Embed pubkey for authentication

Make sure root can login via pubkey-auth (default is fine)

Prepare boot

Last safetys infront of reboot

Make sure MD-RAID reassebles on next boot

Create new mdadm.conf

WELL, GOOD LUCK

Extending to a hypervisor

Some more filesystem

Moving libvirt to own subvolume

Adjust fstab

Filesystem maintenance

ksmtuned

OpenvSwitch

Add vSwitch

Add fakebridges

Configure new interfaces

Conventional networks

PPP Guest-Host

Firewalling

Configure fwbuilder

Performance

Laptop

CPU govenour

IO-Scheduler

Security

CPU microcode

unattended-upgrades

Additional releases

Additional tunables

Debugging

Check OS Integrity

debsecan

Mobile security

ATA hard drive password

Tools

Links

About

Mode high

Mode maximum

Opal