• Linux

Linux

About

• Latest Kernel Docs

• https://www.kernel.org/doc/Documentation/

Talks

• An Introduction to the Linux Kernel Block I/O Stack - Benjamin Block (Linux on IBM Z Kernel Development, IBM Deutschland Research & Development GmbH) on Chemitzer-Linux-Tage 2021

  • Amazing talk - must watch!

Kernel Boot-Parameter

Different distributions ship different boot-parameters. Look them up via:

   1 man 7 bootparam

Here are some important kernel command line parameters that should not be forgotten.

   1 GRUB_CMDLINE_LINUX_DEFAULT="quiet zswap.enabled=1 cgroup.enable=memory swapaccount=1 scsi_mod.use_blk_mq=1 nomodeset"

Networking

IPv6

Source of the hint: FreeIPA Deployment Recommendations

DO NOT use ipv6.disable=1 on the kernel commandline: It disables the whole IPv6 stack and breaks Samba.

If necessary, adding ipv6.disable_ipv6=1 will keep the IPv6 stack functional but will not assign IPv6 addresses to any of your network devices. This is recommended approach for cases when you don't use IPv6 networking.

You may also disable "all" or very specific interfaces.
/etc/sysctl.d/ipv6.conf

   1 net.ipv6.conf.all.disable_ipv6 = 1
   2 # Disabling "all" does not apply to interfaces
   3 # that are already "up" when sysctl settings are applied. 
   4 net.ipv6.conf.<interface0>.disable_ipv6 = 1

Enable IP forwarding

In Linux there are only a fwe switches for enabling global forwarding and they are not interface specific.

   1 net.ipv4.conf.all.forwarding = 1
   2 net.ipv6.conf.all.forwarding = 1
   3 #net.ipv4.ip_forwarding = 1
   4 

net.ipv4.ip_forward

Please compare to

• https://www.kernel.org/doc/html/latest/networking/ip-sysctl.html

• kernel.org doc/html/latest /proc/sys/net/ipv4/* Variables and search for ip_forward

ip_forward - BOOLEAN

        0 - disabled (default)
        not 0 - enabled

    Forward Packets between interfaces.

    This variable is special,
    its change resets all configuration parameters to their default state
    (RFC1122 for hosts, RFC1812 for routers)

• path: /proc/sys/net/ipv4/ip_forward

• sysctl key: net.ipv4.ip_forward

• default: 0
• configuration:

  1. at boottime via sysctl

  2. at runtime via procfs

/proc/sys/net/ip_forward

   1 # cat /proc/sys/net/ipv4/ip_forward
   2 0
   3 # echo 1 > proc/sys/net/ipv4/ip_forward
   4 # cat  proc/sys/net/ipv4/ip_forward
   5 1

/etc/sysctl.d/net.conf

   1 net.ipv4.ip_forward = 1

net.ipv4.conf.all.forwarding

kernel.org doc/html/latest IP Variables
and search for
forwarding - BOOLEAN

Enable IP forwarding on this interface. This controls whether packets received _on_ this interface can be forwarded.

net.ipv6.conf.all.forwarding

Please compare to kernel.org doc/html/latest /proc/sys/net/ipv6/* Variables
And search for
conf/all/forwarding - BOOLEAN

Enable global IPv6 forwarding between all interfaces.

IPv4 and IPv6 work differently here; e.g. netfilter must be used to control, which interfaces may forward packets and which not.

This also sets all interfaces’ Host/Router setting ‘forwarding’ to the specified value. See below for details.

This referred to as global forwarding.

https://www.kernel.org/doc/html/latest/networking/ip-sysctl.html#proc-sys-net-ipv6-variables
And search for
forwarding - INTEGER

Configure interface-specific Host/Router behavior.

Note:
It is recommended to have the same setting on all interfaces; mixed router/host scenarios are rather uncommon.

Possible values are:

• 0 Forwarding disabled
• 1 Forwarding enabled

FALSE (0):

By default, Host behaviour is assumed. This means:

• IsRouter flag is not set in Neighbour Advertisements.

• If accept_ra is TRUE (default), transmit Router Solicitations.
• If accept_ra is TRUE (default), accept Router Advertisements (and do autoconfiguration).
• If accept_redirects is TRUE (default), accept Redirects.

TRUE (1):

If local forwarding is enabled, Router behaviour is assumed. This means exactly the reverse from the above:

• IsRouter flag is set in Neighbour Advertisements.

• Router Solicitations are not sent unless accept_ra is 2.
• Router Advertisements are ignored unless accept_ra is 2.
• Redirects are ignored. Default: 0 (disabled) if global forwarding is disabled (default), otherwise 1 (enabled).

/etc/sysctl.d/ip_forwarding.conf

   1 ### ENABLE IPv4 FORWARDING GLOBALLY ON ALL INTERFACES                                                                                                   
   2 ### ALSO:
   3 ### * SET isRouter IN NEIGHBOUR ADVERTISMENTS
   4 ### * DISCARD ROUTER ADVERTICEMENTS
   5 ### * DISABLE SLAAC                                                                                                                                     
   6 ### * DON'T HONOR REDIRECTS                                                                                                                             
   7 net.ipv4.conf.all.forwarding = 1                                                                                                                        
   8 net.ipv6.conf.all.forwarding = 1                                                                                                                        

Enable the configuration

   1 sysctl -p /etc/sysctl.d/ip_forwarding.conf

Control the configuration

   1 sysctl -a |grep -E 'net\.ipv[46]\.conf\.[^.]+\.?forwarding'

Virtual Memory

• https://www.kernel.org/doc/Documentation/sysctl/vm.txt

• You should use the newer sysfs interface, while the procfs-interface is kept for backwards compatibility.

Take a look on the info provided by the pseudo-filesystems exported by the kernel concerning the virtual memory management.

   1 watch -n1 -- cat /proc/meminfo
   2 watch -n1 -- 'grep -rH "" /sys/kernel/mm/hugepages'
   3 watch -n1 -- 'grep -rH "" /sys/kernel/mm/transparent_hugepage'

Swappiness

• path: /proc/sys/vm/swappiness

• sysctl key: vm.swappiness

• default: 60
• configuration:

  1. at boottime via sysctl

  2. at runtime via procfs

/proc/sys/vm/swappiness

   1 # cat /proc/sys/vm/swappiness
   2 60
   3 # echo 5 > /proc/sys/vm/swappiness
   4 # cat /proc/sys/vm/swappiness
   5 5

/etc/sysctl.d/vm.conf

   1 vm.swappiness = 5

Apply configuration via sysctl.

   1 # sysctl --system
   2 * Applying /etc/sysctl.d/30-baloo-inotify-limit.conf ...
   3 fs.inotify.max_user_watches = 524288
   4 * Applying /etc/sysctl.d/30-postgresql-shm.conf ...
   5 * Applying /etc/sysctl.d/30-tracker.conf ...
   6 fs.inotify.max_user_watches = 65536
   7 * Applying /usr/lib/sysctl.d/50-coredump.conf ...
   8 kernel.core_pattern = |/lib/systemd/systemd-coredump %P %u %g %s %t 9223372036854775808 %e
   9 * Applying /etc/sysctl.d/99-sysctl.conf ...
  10 * Applying /etc/sysctl.d/vm.conf ...
  11 vm.swappiness = 5
  12 vm.dirty_background_ratio = 8
  13 vm.dirty_expire_centisecs = 3000
  14 vm.dirty_ratio = 32
  15 vm.dirty_writeback_centisecs = 500
  16 * Applying /etc/sysctl.conf ...

Memory over-commitment

https://www.kernel.org/doc/html/latest/vm/overcommit-accounting.html

• sysctl key: vm.overcommit_memory

• path: /proc/sys/vm/overcommit_memory

• default: 0
• configuration:

  1. at boottime via sysctl

  2. at runtime via procfs

• usage:
  • redis demands it with

    WARNING overcommit_memory is set to 0!
    Background save may fail under low memory condition.
    To fix this issue add 'vm.overcommit_memory = 1' to /etc/sysctl.conf and
    then reboot or run the command 'sysctl vm.overcommit_memory=1'
    for this to take effect.

/etc/sysctl.d/vm.conf

   1 vm.overcommit_memory = 1

Hugepages

Hugepages are a optimization of the memory management targeting the Translation Lookaside Buffer (TLB), which is a fast buffer and limited in the CPU, that maps virtual addresses to physical addresses. Less entries in the TLB because of bigger page sizes, mean less page misses during the runtime.

Transparent hugepages

https://www.kernel.org/doc/html/latest/admin-guide/mm/transhuge.html

khugepaged scans the memory in intervals and defrags and collapses large areas to a hugepages.

Currently THP only works for anonymous memory mappings and tmpfs/shmem. But in the future it can expand to other filesystems. See also tmpfs with systemd

• path: /sys/kernel/mm/transparent_hugepage

• default: 0
• configuration:

  1. at boottime via sysfsutils

  2. at runtime via sysfs

• usage:
  • redis demands it with

    WARNING you have Transparent Huge Pages (THP) support enabled in your kernel.
    This will create latency and memory usage issues with Redis.
    To fix this issue run the command 'echo madvise > /sys/kernel/mm/transparent_hugepage/enabled' as root,
    and add it to your /etc/rc.local in order to retain the setting after a reboot.
    Redis must be restarted after THP is disabled (set to 'madvise' or 'never').

/etc/sysfs.d/transparent_hugepage.conf

   1 kernel/mm/transparent_hugepage = madvise

Explicit hugepages

https://www.kernel.org/doc/html/latest/admin-guide/mm/hugetlbpage.html

• path: /sys/kernel/mm/hugepages/

• default: 0
• configuration:

  1. at boottime via sysctl

  2. at runtime via sysfs

  3. at runtime via procfs /proc/sys/vm/*huge*

This example configures 2048 hugepages, each 2MiB in size, which may be allocated dynamically on top to the fixed number of hugepages (0).

/etc/sysctl.d/hugepages.conf

   1 #vm.hugetlb_shm_group = 0
   2 #vm.nr_hugepages = 0
   3 #vm.nr_hugepages_mempolicy = 0
   4 vm.nr_overcommit_hugepages = 2048

grep -rH "" /sys/kernel/mm/hugepages

   1 /sys/kernel/mm/hugepages/hugepages-2048kB/free_hugepages:0
   2 /sys/kernel/mm/hugepages/hugepages-2048kB/resv_hugepages:0
   3 /sys/kernel/mm/hugepages/hugepages-2048kB/surplus_hugepages:0
   4 /sys/kernel/mm/hugepages/hugepages-2048kB/nr_hugepages_mempolicy:0
   5 /sys/kernel/mm/hugepages/hugepages-2048kB/nr_hugepages:0
   6 /sys/kernel/mm/hugepages/hugepages-2048kB/nr_overcommit_hugepages:2048
   7 /sys/kernel/mm/hugepages/hugepages-1048576kB/free_hugepages:0
   8 /sys/kernel/mm/hugepages/hugepages-1048576kB/resv_hugepages:0
   9 /sys/kernel/mm/hugepages/hugepages-1048576kB/surplus_hugepages:0
  10 /sys/kernel/mm/hugepages/hugepages-1048576kB/nr_hugepages_mempolicy:0
  11 /sys/kernel/mm/hugepages/hugepages-1048576kB/nr_hugepages:0
  12 /sys/kernel/mm/hugepages/hugepages-1048576kB/nr_overcommit_hugepages:0

zswap

• https://www.kernel.org/doc/html/latest/vm/zswap.html

• https://www.kernel.org/doc/Documentation/vm/zswap.txt

• https://en.wikipedia.org/wiki/Zswap

• https://lwn.net/Articles/537422/

• https://wiki.archlinux.org/index.php/Zswap

Zswap is a lightweight compressed cache for swap pages. It takes pages that are in the process of being swapped out and attempts to compress them into a dynamically allocated RAM-based memory pool. zswap basically trades CPU cycles for potentially reduced swap I/O. This trade-off can also result in a significant performance improvement if reads from the compressed cache are faster than reads from a swap device.

• path: /sys/module/zswap/parameters

• default: N
• configuration:

  1. at boottime via sysfsutils

  2. at runtime via sysfs

grep -R . /sys/module/zswap/parameters

   1 /sys/module/zswap/parameters/same_filled_pages_enabled:Y
   2 /sys/module/zswap/parameters/enabled:N
   3 /sys/module/zswap/parameters/max_pool_percent:20
   4 /sys/module/zswap/parameters/compressor:lzo
   5 /sys/module/zswap/parameters/zpool:zbud

/etc/sysfs.d/zswap.conf

   1 module/zswap/parameters/enabled = 1

CPU scaling governor

Available scaling governors

   1 # cat /sys/devices/system/cpu/cpu*/cpufreq/scaling_available_governors|uniq
   2 conservative ondemand userspace powersave performance schedutil 

Default is ondemand

Set governor

   1 echo "performance" \
   2      |sudo tee /sys/devices/system/cpu/cpu*/cpufreq/scaling_governor
   3 echo "powersave" \
   4      |sudo tee /sys/devices/system/cpu/cpu*/cpufreq/scaling_governor
   5 echo "ondemand" \
   6      |sudo tee /sys/devices/system/cpu/cpu*/cpufreq/scaling_governor

Finetune CPU frequency with

   1 # grep -H "" /sys/devices/system/cpu/cpu*/cpufreq/scaling_*_freq
   2 /sys/devices/system/cpu/cpu0/cpufreq/scaling_cur_freq:600000
   3 /sys/devices/system/cpu/cpu0/cpufreq/scaling_max_freq:1500000
   4 /sys/devices/system/cpu/cpu0/cpufreq/scaling_min_freq:600000
   5 /sys/devices/system/cpu/cpu1/cpufreq/scaling_cur_freq:600000
   6 /sys/devices/system/cpu/cpu1/cpufreq/scaling_max_freq:1500000
   7 /sys/devices/system/cpu/cpu1/cpufreq/scaling_min_freq:600000
   8 /sys/devices/system/cpu/cpu2/cpufreq/scaling_cur_freq:600000
   9 /sys/devices/system/cpu/cpu2/cpufreq/scaling_max_freq:1500000
  10 /sys/devices/system/cpu/cpu2/cpufreq/scaling_min_freq:600000
  11 /sys/devices/system/cpu/cpu3/cpufreq/scaling_cur_freq:600000
  12 /sys/devices/system/cpu/cpu3/cpufreq/scaling_max_freq:1500000
  13 /sys/devices/system/cpu/cpu3/cpufreq/scaling_min_freq:600000

Task delay accounting

https://www.kernel.org/doc/Documentation/accounting/delay-accounting.rst

iotop complains that:
CONFIG_TASK_DELAY_ACCT not enabled in kernel, cannot determine SWAPIN and IO %

So CONFIG_TASK_DELAY_ACCT is not configured?

   1 grep CONFIG_TASK_DELAY_ACCT /boot/config-$(uname -r)
   2 CONFIG_TASK_DELAY_ACCT=y

You may add the option delayacct in /etc/default/grub to GRUB_CMDLINE_LINUX_DEFAULT.

Enable it at runtime

   1 sysctl kernel.task_delayacct
   2 sysctl kernel.task_delayacct=1

For persistence

   1 echo 'kernel.task_delayacct = 1' > /etc/sysctl.d/80-task_delayacct.conf
   2 sysctl --system

IO-Scheduler

• On a hypervisor the scheduler bfq seems to be reasonable.

• On a VM with no disk or controller pass-through none should be used. This avoids optimizing the queues twice, which is inefficient and contra-productive. The hypervisor will optimize the io-request anyway.

Make alternative schedulers available

BLK-MQ is nowadays broadly available and enabled in distributions. Using multiple queues on multicore systems with fast storage promises some performance gains.

But when I took a look on available schedulers only "mq-deadline" and "none" were available.

   1 cat /sys/class/block/sda/queue/scheduler
   2 [mq-deadline] none

This is because these scheduler are shipped as a kernel module and need to be loaded first into the kernel via modprobe.

   1 ls /lib/modules/$(uname -r)/kernel/block
   2 bfq.ko  kyber-iosched.ko

Modules may be loaded manually:

   1 modprobe bfq
   2 modprobe kyber-iosched

Modules may also be loaded automatically at boot-time via /etc/modules.

   1 for SCHEDULER in bfq kyber-iosched; do
   2         if \! grep -q "$SCHEDULER" /etc/modules && \
   3            \! grep -qr "$SCHEDULER" /etc/modules-load.d; then
   4                 echo "$SCHEDULER" >> /etc/modules
   5         fi
   6 done

Set IO-Scheduler permanently

kernel-cmdline

• Method seems not to be working any longer.

• Service affecting.

/etc/default/grub

   1 GRUB_CMDLINE_LINUX_DEFAULT="quiet elevator=$SCHEDULER"

Refresh grub config and reboot.

   1 update-grub2
   2 systemctl reboot

udev-rule

• Works at run- and at boot-time!

• More selective because disks may be filtered with a regex.

/etc/udev/rules.d/60-persistent-storage-scheduler.rules

   1 # GUEST SHOULD NOT REORDER STORAGE REQUESTS -> NONE or NOOP
   2 ACTION=="add|change", KERNEL=="[sv]d[a-z]", ATTR{queue/scheduler}="none"
   3 # HYPERVISOR SHOULD NOT REORDER STORAGE REQUESTS IN PARALLEL QUEUES
   4 #ACTION=="add|change", KERNEL=="sd[ab]", ATTR{queue/scheduler}="bfq"
   5 

Reload udev-rules

Reload will probably happen automatically but the "trigger" is necessary.

   1 udevadm control --reload-rules && udevadm trigger

Drop FS Cache

   1 echo 3 | tee /proc/sys/vm/drop_caches

Hardening

Disable TCP Timestamping

• https://cloudshark.io/articles/tcp-timestamp-option/

• https://netsense.ch/blog/tcp-timestamps/

• https://www.scip.ch/?labs.20150305

• IETF RFC1323 TCP Extensions for High Performance (obsoleted)

• IETF RFC7323 TCP Extensions for High Performance

   1 hping3 -S -p 22 --tcp-timestamp $DESTINATION
   2 
   3 1 root@libertas /home/tobias/Downloads # hping3 -S -p 22 --tcp-timestamp www.rockstable.it
   4 HPING www.rockstable.it (bridge 178.63.149.226): S set, 40 headers + 0 data bytes
   5 len=56 ip=178.63.149.226 ttl=53 DF id=0 sport=22 flags=SA seq=0 win=65160 rtt=24.2 ms
   6   TCP timestamp: tcpts=2031225761
   7 
   8 len=56 ip=178.63.149.226 ttl=53 DF id=0 sport=22 flags=SA seq=1 win=65160 rtt=19.8 ms
   9   TCP timestamp: tcpts=2031226761
  10   HZ seems hz=1000
  11   System uptime seems: 23 days, 12 hours, 13 minutes, 46 seconds

Disable temporarily

   1 echo 0 > /proc/sys/net/ipv4/tcp_timestamps

Disable persitent
{{{/etc/sysctl.d/tcp.conf

   1 net.ipv4.tcp_timestamps=0