networking

WCH (in full WinChipHead) CH350G (Nanjing Qinheng Microelectronics Co., Ltd)
- http://www.wch-ic.com/products/category/1.html
- CH340DS1.PDF
- 50bps to 2Mbps
- 12MHz clock
- compatible to 3.3V and 5V
FTDI FT232 RL (Future Technology Devices International Limited)
- FT232RL
- DS_FT232R.pdf
- 300baud to 3Mbaud
- 48MHz clock (multiplied from 12MHz)
- compatible to 1.8V up to 5.25V
- FTDIgate?!
Silicon Labs CP2102
- USB to UART Bridge
- CP2102.pdf
- 300bps to 1Mbps
- 48 MHz clock
- compatible to 3.3V (5V devices will probably still recognize 3.3V)
Prolific PL2303
- Prolific USB to Serial/UART/RS232/Printer
- Prolific PL2303GS USB to Serial Bridge Controller
- PL2303GS product brochure 20190125.pdf
- 1bps to 12Mbps
- compatible to 3.3V and 5V
- 96 MHz clock

SPI/I2C/JTAG adapter

USB 2.0 Hi-Speed to MPSSE cable (Multi-Protocol Synchronous Serial Engine)

RJ-45 repair clips

Give a cable a second life! :-D

Best practices

Network Byte Order

The network byte order is Big Endian (most network protocols, Motorola, IBM Mainframes like System/360|370 and ESA/390 and z/Architecture). The most significant BYTE (MSB) is stored at the lower address or sent first (Big Startian).

Examples:

ISO long date YYYY-MM-DD or
Ordinary English numbers 1234 "one thousand two hundred and thirty four".

In opposite Little Endian (most Processor architectures, Intel, RS-232) the least significant BYTE (MSB) is stored at the lower address or sent first (Little Startian).

Examples:

German long date DD.MM.YYYY or
German number lower than 100 like 21 "einundzwanzig"

0x12345678 on a big endian 32bit machine is
0x78563412 on a little endian 32bit machine
and vice versa.

Linux device names

The length of device names, aliases and alternative interface names is defined in
github torvalds/linux master include/uapi/linux/if.h

   1 #if __UAPI_DEF_IF_IFNAMSIZ
   2 #define IFNAMSIZ        16
   3 #endif /* __UAPI_DEF_IF_IFNAMSIZ */
   4 #define IFALIASZ        256
   5 #define ALTIFNAMSIZ     128
   6 #include <linux/hdlc/ioctl.h>
   7

Predictable device names

Let's predict some names.

Bus:Device.Function (BDF)

lspci |grep -i eth

   1 00:1f.6 Ethernet controller: Intel Corporation Ethernet Connection (2) I219-LM

PCI-address 00:1f.6

BUS: 0
Device/Slot: 31 = 1*16¹ + 15*16⁰
Function: 6

So the interface name is:
enp0s31f6 = "en" + "p" + "0" + s + "31" + "f" + "6"

lspci |grep -i eth

   1 07:00.0 Ethernet controller: Intel Corporation I210 Gigabit Network Connection (rev 03)

PCI-address 07:00.0

BUS: 7
Device/Slot: 0 = 0*16⁰
Function: 0

So the interface name is:
enp0s31f6 = "en" + "p" + "7" + s + "0" + "f" + "0"

Models OSI and TCP/IP

Physical Layer

Definitions and abbreviations

SFD
: Start Frame Delimiter
: 10101011 LSB 213 0xD5
: Occurs after 7 octets of preamble
: provide byte-level synchronization
: Immeadiately followed by the Ethernet Frame specifically the destination MAC
: Wiki EN Preamble and start frame delimiter
Preamble
: 56-bit (seven-bytes/octets) pattern of alternating 1 and 0 bits
: provides bit-level synchronization
: 10101010 LSB 85 0x55
: Full preamble with Start Frame Delimiter
10101010 10101010 10101010 10101010 10101010 10101010 10101011
: Wiki EN Preamble and start frame delimiter
MII
: media independent interface
: more specifically MII, GMII, RGMII, SGMII, XGMII
: Interface between MAC (chip) and PHY (chip)
: Wiki EN Media-independent interface
PHY
: PHYsical (layer) transceiver (circuitry)
: Wiki EN Ethernet physical transceiver
: Provides analog signal physical access to the link

Data Link

Logical Link Control (LLC)

Wiki EN Logical Link Control
L2 sublayer
provides flow control and multiplexing for the logical link

Medium Access Control (MAC)

Medium access control
L2 sublayer
provides flow control and multiplexing for the transmission medium

Address Resolution Protocol (ARP)

The Address Resolution Protocol (ARP) is a communication protocol used for discovering the link layer address, such as a MAC address, associated with a given internet layer address, typically an IPv4 address. This mapping is a critical function in the Internet protocol suite. ARP was defined in 1982 by RFC 826, which is Internet Standard STD 37.

Show ARP

The commands arp option -a is different between Linux and BSD.

On BSD it means show all entries.
On Linux it means show entries in BSD style.

Display ARP table sorted (on BSD machines)

   1 LANG=C arp -a \
   2         | sed -r -e 's/^([[:alnum:].?-]+) +\(([0-9.]+)\)/\2 \1/' \
   3                 -e 's/ at / /' \
   4                 -e 's/ on / /' \
   5                 -e 's/ (expires in|permanent ).*//' \
   6         | sort -V \
   7         | column -t

Alternatively

   1 arp -a | sort -k 2V | column -t

Flush ARP Cache

This is useful on so many systems. When a MAC address changes, e. g. when the gateway is exchanged, this MAC needs to be purged from the cache, in order to trigger a new ARP resolution.

Delete single cache entry

   1 arp -d $FQDN
   2 arp -d $IP_ADDRESS
   3 arp -d $MAC_ADDRESS

Delete all entries in BSD

   1 arp -d -a

Delete all entries in Linux

   1 ip neighbor flush

Delete all entries in Linux by device

   1 DEVICE="br0"
   2 ip neighbor flush dev "$DEVICE"

Delete all entries in Linux with a shell script

   1 DEVICE="br0"
   2 for ip in $(awk '/([[:digit:]]\.)+.*('"$DEVICE"')/ {print $1}' /proc/net/arp); do  
   3       echo arp -d $ip ;
   4 done

MAC addresses

48 bit MAC addresses

48 bit MAC addresses, derived from EUI-48 (Extended Unique Identifier 48), consist of 6 octets. The first 3 octets are called Organizationally Unique Identifier (OUI) and the second 3 octets are called the (Network Interface Controller (NIC) specific) manufacturer-selected extension identifier. The least significant bits of the first octet of the OUI, signal M (I/G) bit Unicast (0) / Multicast(1) and globally unique (OUI enforced) (0) / locally administered (1).

    0  1  2  3  4  5  6  7  0  1  2  3  4  5  6  7
  +--+--+--+--+--+--+--+--+--+--+--+--+--+--+--+--+
  | .  .  .  .  Z  Y  X  M| .  .  .  .  .  .  .  .| octets 0+1
  +--+--+--+--+--+--+--+--+--+--+--+--+--+--+--+--+
  | .  .  .  .  .  .  .  .| .  .  .  .  .  .  .  .| octets 2+3
  +--+--+--+--+--+--+--+--+--+--+--+--+--+--+--+--+
  | .  .  .  .  .  .  .  .| .  .  .  .  .  .  .  .| octets 4+5
  +--+--+--+--+--+--+--+--+--+--+--+--+--+--+--+--+

  M … Unicast (0)   vs. multicast (1) [I/G bit] (Individual/Group address)
  X … Universal (0) vs. local     (1) [U/L bit] (Universally/locally administered address)
  Y + Z … SLAP quadrands

Structured Local Address Plan (SLAP)

SLAP quadrants as simple text table in comments

  SLAP      Y    Z    ZYXM  second     SLAP local                 SLAP local
  Quadrant  bit  bit        hex digit  identifier type            identifier
  00        0     0   0010  2          Administratively Assigned  AAI
  10        1     0   0110  6          Reserved                   Reserved
  01        0     1   1010  A          Extended Local             ELI
  11        1     1   1110  E          Standard Assigned          SAI

         Y = 0  Y = 1
  Z = 0  AAI    Reserved
  Z = 1  ELI    SAI

SLAP Quadrant	Y bit	Z bit	ZYXM	second hex digit	SLAP local identifier type	SLAP local identifier
`00`	`0`	`0`	`0010`	`2`	Administratively Assigned	AAI
`10`	`1`	`0`	`0110`	`6`	Reserved	Reserved
`01`	`0`	`1`	`1010`	`A`	Extended Local	ELI
`11`	`1`	`1`	`1110`	`E`	Standard Assigned	SAI

	Y = 0	Y = 1
Z = 0	AAI	Reserved
Z = 1	ELI	SAI

IEEE Registration Authority (IEEE RA)

IEEE Registration Authority (IEEE RA)

IEEE RA hexadecimal (hex) representation are octets separated by hyphens

IEEE RA base-16 notation is just hex digits without separators

Organizationally Unique Identifier (OUI)

Second hex digit = 0

ZYXM = 0000

: Z = 0, Y = 0 - Administratively assigned (AAI)
: X (U/L) = 0 - Universally administered (OUI)
: M (I/G) = 0 - Unicast (Individual Address)

24-bit (three octet) sequence.

Globally unique

: Derived MACs are burned into NICs to identify the devices.

Organizationally Unique Identifier 36 bit (OUI-36)

Second hex digit = C

ZYXM = 1100

: Z = 1, Y = 1 Standard Assigned Idenitifier (SAI)
: X (U/L) = 0 - Universally administered (OUI)
: M (I/G) = 0 - Unicast (Individual Address)

36-bit (four-and-one-half-octet) sequence.

Created by the IEEE RA by concatenating 12 bits to a 24-bit IEEE-reserved base OUI, concatenating these 12 bits after the least significant bit of Octet 2 (3rd).

Base OUI will not be assigned to another organization or otherwise be used as an OUI.

An assignee of an OUI-36 shall not truncate the OUI-36 to use as an OUI because the IEEE RA will use the base OUI to assign OUI-36 values to multiple organizations.

Company ID (CID)

Second hex digit = A

ZYXM = 1010

: Z = 1, Y = 0 Extended Local Idenitifier (ELI)
: X (U/L) = 1 Locally administered (CID)
: M (I/G) = 0 Unicast (Individual address)

24-bit (three octet) sequence.

Local

IEEE Registration Authority (RA) assigns CIDs, all in SLAP 01

Local uniqueness needs to be ensured by the administrator

Extended Unique Identifier (EUI)

Second hex digit = 0,4,8,C

ZYXM = ??00

: X (U/L) = 0 - Universally administered (OUI)
: M (I/G) = 0 - Unicast (Individual Address)

either a 48-bit Extended Unique Identifier (EUI-48) or a 64-bit Extended Unique Identifier (EUI-64)

Address space also includes protocol identifiers

In an EUI created by extending an OUI, the OUI is the initial (most significant) three octets.

In an EUI created by extending an OUI-36, the OUI-36 is the initial (most significant) four and a half octets.

Like ELI but with OUI or OUI-36

All local unicast

Second hex digit = 2,6,A,E

ZYXM = ??10

: X (U/L) = 1 - Locally administered (CID)
: M (I/G) = 0 - Unicast (Individual Address)

The 4 SLAP quadrants

Administratively Assigned Identifier (AAI)

Second hex digit = 2

ZYXM = 0010

: Z = 0, Y = 0 Standard Assigned Idenitifier (SAI)
: X (U/L) = 1 Locally administered (CID)
: M (I/G) = 0 Unicast (Individual address)

Administrators who wish to assign local MAC addresses in an arbitrary fashion (for example, randomly) and yet maintain compatibility with other assignment protocols operating under the SLAP on the same LAN may assign a local MAC address as AAI.

Reserved quadrant

Second hex digit = 6

ZYXM = 0110

: Z = 0, Y = 1 Reserved
: X (U/L) = 1 Locally administered (CID)
: M (I/G) = 0 Unicast (Individual address)

Reserved quadrant can be used like AAI, with reservations:

: May be administratively used and assigned in accordance with the considerations specified for AAI usage, without effect on SLAP assignments. However, administrators should be cognizant of possible future specifications, that would render administrative assignment incompatible with the SLAP.

Extended Local Identifier (ELI)

Second hex digit = A

ZYXM = 0110

: Z = 1, Y = 0 Extended Local Identifier (ELI)
: X (U/L) = 1 Locally administered (CID)
: M (I/G) = 0 Unicast (Individual address)

ELI-48 is a 48-bit ELI and the ELI-64 is a 64-bit ELI.

Created by concatenation from a CID, which comprises the initial three (most significant) octets.

Like EUI but with CID

All CID assignments are with M (I/G) set to 0.

: Since all CID assignments made by the IEEE RA have the M bit equal to 0, an ELI created as an extended identifier from an assigned CID has I/G=0 and is thus, when used as a MAC address, an individual address. The assignee of a CID may assign local group MAC addresses by extending a modified version of the assigned CID by setting the M bit to 1 (so that I/G=1). The resulting extended identifier is an ELI.
: Second hex digit = B

Local addresses are not globally unique, and a network administrator is responsible for assuring that any local addresses assigned are unique within the span of use. (Uniqueness of local addresses typically does not need to extend beyond a router.)

IEEE 802c reserves 4 CIDs for the local administrator

Standard Assigned Identifier (SAI)

Second hex digit = E

ZYXM = 1110

: Z = 1, Y = 1 Standard Assigned Identifier (SAI)
: X (U/L) = 1 Locally administered (CID)
: M (I/G) = 0 Unicast (Individual address)

MAC addresses for Anycast

When generating MAC addresses e.g. for Anycast IPs, you could set BITs ZYXM to 0010 and thus the 2nd nibble of octet 1 to 2. IMHO for Anycast addresses it is a good idea to just set nibble two of your chassis base mac to 2, to make the system identifiable by a glance on its MAC (less significant octets).

mac2private.sh

I wrote a script which takes or generates MACs and/or manipulates the bits. It's not my pretties one, but does the job. In heredocs (cat <<- EOH) tabs are prepended instead of spaces or the output will look wierd. Has support for input macs, prefixes, randomization, …
git.rockstable.it - mac2private.sh
mac2private.sh

   1 #!/bin/bash
   2 
   3 ### RFC 9542 - Special First Octet Bits
   4 ### https://www.rfc-editor.org/rfc/rfc9542.html#name-special-first-octet-bits
   5 ### https://standards.ieee.org/wp-content/uploads/import/documents/tutorials/eui.pdf
   6 
   7 PROGRAM="${0#*/}"
   8 
   9 ### DEFAULTS
  10 ADMIN=true
  11 EXT_LOCAL=false
  12 UNIVERSAL=true
  13 LOCAL=true
  14 RESERVED=false
  15 STANDARD=false
  16 INDIVIDUAL=true
  17 VERBOSE=false
  18 RANDOMIZE=false
  19 OUT_SEP=":"
  20 
  21 declare -a MAC_MASK_BIN_ARRAY MAC_MASK_HEX_ARRAY
  22 
  23 BIT_M=0
  24 BIT_X=0
  25 BIT_Y=0
  26 BIT_Z=0
  27 
  28 
  29 ### FUNCTIONS
  30 base2base() {
  31         local BYTE="${3^^}"
  32         echo "obase=$2; ibase=$1; $BYTE" | bc
  33 }
  34 
  35 usage() {
  36         cat <<-EOH
  37                 $PROGRAM [OPTIONS] [--] MAC1 [MAC2 [… MACN]]
  38 
  39         where OPTIONS are:
  40           '-a'|'--admin-assigned'             AAI (Bit 4 Z=0, bit 5 Y=0, bit 6 U/L] X=1).
  41           '-e'|'--extended-local'                  ELI (Bit 4 Z=1, bit 5 Y=0, bit 6 U/L] X=1).
  42           '-g'|'--group'                      Derive an group (multicast) MAC (bit 7 [I/G] M=1).
  43           '-h'|'--help'                       Show this page.
  44           '-i'|'--individual'                 Derive a individual (unicast) MAC (bit 7 [I/G] M=0) [DEFAULT].
  45           '-o char'|'--out-separator char'    Separate MAC octets by char (Default: ':').
  46                                               Values: any char or "cisco".
  47           '-l'|'--local'                      Derive a locally administered MAC (bit 6 [U/L] X=0).
  48           '-n[count]'|'--new-random[=count]'  Derive [count] new random MACs.
  49                                               Prefix is honored. Please take care of the syntax.
  50           '-p prefix'|'--prefix prefix'       Use a specified EUI-48 prefix.
  51           '-r'|'--reserved'                   Reserved (Bit 4 Z=0, bit 5 Y=1, bit 6 U/L] X=1).
  52           '-s'|'--standard-assigned'          SAI      (Bit 4 Z=1, bit 5 Y=1, bit 6 U/L] X=1).
  53           '-u'|'--universal'                  Derive an universally administered (global) MAC (bit 6 [U/L] X=0).
  54           '-v'|'--verbose'                    Print more detailed information.
  55 
  56           -- break option parsing
  57           [] marks optional parameters
  58 
  59           Either at least one MAC or the option switch for new-random must be given.
  60 
  61         First MAC octet has some special bits:
  62 
  63          * RFC 9542 - Special First Octet Bits
  64            https://www.rfc-editor.org/rfc/rfc9542.html#name-special-first-octet-bits
  65          * Guidelines for Use of Extended Unique Identifier (EUI),
  66            Organizationally Unique Identifier (OUI), and Company ID (CID)
  67            https://standards.ieee.org/wp-content/uploads/import/documents/tutorials/eui.pdf
  68          * IEEE Std 802c: What’s New and Useful in the Overview and Architecture - 2017-09-12 Roger Marks
  69            https://mentor.ieee.org/802.11/dcn/17/11-17-1466-00-0000-local-mac-addresses-in-the-overview-and-architecture-based-on-ieee-std-802c.pptx
  70 
  71             0  1  2  3  4  5  6  7  0  1  2  3  4  5  6  7
  72           +--+--+--+--+--+--+--+--+--+--+--+--+--+--+--+--+
  73           | .  .  .  .  Z  Y  X  M| .  .  .  .  .  .  .  .| octets 0+1
  74           +--+--+--+--+--+--+--+--+--+--+--+--+--+--+--+--+
  75           | .  .  .  .  .  .  .  .| .  .  .  .  .  .  .  .| octets 2+3
  76           +--+--+--+--+--+--+--+--+--+--+--+--+--+--+--+--+
  77           | .  .  .  .  .  .  .  .| .  .  .  .  .  .  .  .| octets 4+5
  78           +--+--+--+--+--+--+--+--+--+--+--+--+--+--+--+--+
  79 
  80           M … Unicast (0)   vs. multicast (1) [I/G bit] (Individual/Group address)
  81           X … Universal (0) vs. local     (1) [U/L bit] (Universally/locally administered address)
  82           Y + Z … SLAP quadrands
  83 
  84           SLAP      Y    Z    ZYXM  second     SLAP local                 SLAP local
  85           Quadrant  bit  bit        hex digit  identifier type            identifier
  86           00        0     0   0010  2          Administratively Assigned  AAI
  87           10        1     0   0110  6          Reserved                   Reserved
  88           01        0     1   1010  A          Extended Local             ELI
  89           11        1     1   1110  E          Standard Assigned          SAI
  90 
  91                  Y = 0  Y = 1
  92           Z = 0  AAI    Reserved
  93           Z = 1  ELI    SAI
  94         EOH
  95 }
  96 
  97 
  98 # Note that we use "$@" to let each command-line parameter expand to a
  99 # separate word. The quotes around "$@" are essential!
 100 # We need TEMP as the 'eval set --' would nuke the return value of getopt.
 101 TEMP=$(getopt \
 102         -o 'aeghiln::o:p:rsuv' \
 103         --long 'admin-assigned,extended-local,group,help,individual,local' \
 104         --long 'out-separator:,prefix:,new-random::,reserved,standard-assigned,univeral,verbose' \
 105         -n "$PROGRAM" -- "$@")
 106 
 107 if [ $? -ne 0 ]; then
 108         echo 'Terminating...' >&2
 109         exit 1
 110 fi
 111 
 112 # Note the quotes around "$TEMP": they are essential!
 113 eval set -- "$TEMP"
 114 unset TEMP
 115 
 116 while true; do
 117         case "$1" in
 118                 '-a'|'--admin-assigned')
 119                         ADMIN=true
 120                         EXT_LOCAL=false
 121                         RESERVED=false
 122                         STANDARD=false
 123                         UNIVERSAL=false
 124                         shift
 125                         continue
 126                 ;;
 127                 '-e'|'--extended-local')
 128                         ADMIN=false
 129                         EXT_LOCAL=true
 130                         RESERVED=false
 131                         STANDARD=false
 132                         UNIVERSAL=false
 133                         shift
 134                         continue
 135                 ;;
 136                 '-g'|'--group')
 137                         INDIVIDUAL=false
 138                         shift
 139                         continue
 140                 ;;
 141                 '-h'|'--help')
 142                         usage
 143                         exit
 144                 ;;
 145                 '-i'|'--individual')
 146                         INDIVIDUAL=true
 147                         shift
 148                         continue
 149                 ;;
 150                 '-l'|'--local')
 151                         UNIVERSAL=false
 152                         shift
 153                         continue
 154                 ;;
 155                 '-n'|'--new-random')
 156                         RANDOMIZE=true
 157                         case "$2" in
 158                                 '')
 159                                         RANDOMIZE_COUNT=1
 160                                 ;;
 161                                 *)
 162                                         RANDOMIZE_COUNT="$2"
 163                                 ;;
 164                         esac
 165                         shift 2
 166                         continue
 167                 ;;
 168                 '-o'|'--out-separator')
 169                         OUT_SEP="$2"
 170                         shift 2
 171                         continue
 172                 ;;
 173                 '-p'|'--prefix')
 174                         PREFIX_RAW="$2"
 175                         shift 2
 176                         continue
 177                 ;;
 178                 '-r'|'--reserved')
 179                         ADMIN=false
 180                         EXT_LOCAL=false
 181                         RESERVED=true
 182                         STANDARD=false
 183                         UNIVERSAL=false
 184                         shift
 185                         continue
 186                 ;;
 187                 '-s'|'--standard-assigned')
 188                         ADMIN=false
 189                         EXT_LOCAL=false
 190                         RESERVED=false
 191                         STANDARD=true
 192                         UNIVERSAL=false
 193                         shift
 194                         continue
 195                 ;;
 196                 '-u'|'--universal')
 197                         UNIVERSAL=true
 198                         shift
 199                         continue
 200                 ;;
 201                 '-v'|'--verbose')
 202                         VERBOSE=true
 203                         shift
 204                         continue
 205                 ;;
 206                 '--')
 207                         shift
 208                         break
 209                 ;;
 210                 *)
 211                         echo 'Internal error!' >&2
 212                         exit 1
 213                 ;;
 214         esac
 215 done
 216 
 217 if [ ${#@} -le 0 ] && ! $RANDOMIZE; then
 218         echo "Nothing to do - no MAC given. Exiting …"
 219         exit 0
 220 fi
 221 
 222 
 223 ### PREFIX
 224 PREFIX="${PREFIX_RAW//[:.-]/}"
 225 PREFIX_LENGTH="${#PREFIX}"
 226 
 227 for ((i=0; i<6; i++)); do
 228         CHAR="$(( PREFIX_LENGTH - i*2))"
 229         if [ "$CHAR" -ge "2" ]; then
 230                 PREFIX_MASK_BIN_ARRAY+=("11111111")
 231         elif [ "$CHAR" -ge "1" ]; then
 232                 PREFIX_MASK_BIN_ARRAY+=("11110000")
 233         elif [ "$CHAR" -lt "1" ]; then
 234                 PREFIX_MASK_BIN_ARRAY+=("00000000")
 235         fi
 236 done
 237 
 238 
 239 for BYTE in "${PREFIX_MASK_BIN_ARRAY[@]}"; do
 240         PREFIX_MASK_HEX_ARRAY+=("$(base2base 2 10 "$BYTE" \
 241                 | xargs printf '%02X')")
 242 done
 243 
 244 for (( i=0; i<PREFIX_LENGTH; i=i+2)); do
 245         if [ $((PREFIX_LENGTH-i)) -ge 2 ]; then
 246                 PREFIX_CHAR="${PREFIX:$i:2}"
 247         else
 248                 PREFIX_CHAR="${PREFIX:$i:1}0"
 249         fi
 250 
 251         PREFIX_HEX_ARRAY+=("$PREFIX_CHAR")
 252 done
 253 
 254 ### MAC MASKING
 255 if $ADMIN; then
 256         BIT_Y=0
 257         BIT_Z=0
 258 elif $EXT_LOCAL; then
 259         BIT_Y=0
 260         BIT_Z=1
 261 elif $RESERVED; then
 262         BIT_Y=1
 263         BIT_Z=0
 264 elif $STANDARD; then
 265         BIT_Y=1
 266         BIT_Z=1
 267 fi
 268 
 269 if ! $INDIVIDUAL; then
 270         BIT_M=1
 271 fi
 272 
 273 if ! $UNIVERSAL; then
 274         BIT_X=1
 275 fi
 276 
 277 for ((i=0; i<6; i++)); do
 278         MAC_MASK_BIN_ARRAY_AND+=("11111111")
 279         MAC_MASK_BIN_ARRAY_OR+=("00000000")
 280 done
 281 MAC_MASK_BIN_ARRAY_AND[0]="11110000"
 282 MAC_MASK_BIN_ARRAY_OR[0]="0000$BIT_Z$BIT_Y$BIT_X$BIT_M"
 283 
 284 for BYTE in "${MAC_MASK_BIN_ARRAY[@]}"; do
 285         MAC_MASK_HEX_ARRAY+=("$(base2base 2 10 "$BYTE" | xargs printf '%02X')")
 286 done
 287 
 288 for BYTE in "${MAC_MASK_BIN_ARRAY_AND[@]}"; do
 289         MAC_MASK_HEX_ARRAY_AND+=("$(base2base 2 10 "$BYTE" | xargs printf '%02X')")
 290 done
 291 for BYTE in "${MAC_MASK_BIN_ARRAY_OR[@]}"; do
 292         MAC_MASK_HEX_ARRAY_OR+=("$(base2base 2 10 "$BYTE" | xargs printf '%02X')")
 293 done
 294 
 295 if $RANDOMIZE; then
 296         MAC_COUNT="$RANDOMIZE_COUNT"
 297 else
 298         MAC_COUNT="${#*}"
 299 fi
 300 
 301 for ((c=1; c<MAC_COUNT+1; c++)); do
 302         unset MAC_HEX_ARRAY MAC_BIN_ARRAY MAC_NEW_BIN_ARRAY MAC_NEW_HEX_ARRAY MAC_NEW MAC_MASKED_HEX_ARRAY
 303         MAC_RAW="${!c}"
 304 
 305         if $RANDOMIZE; then
 306                 MAC_HEX="$(dd if=/dev/urandom count=1 bs=6 status=none| xxd -p)"
 307         else
 308                 MAC_HEX="$(echo -n "$MAC_RAW" | sed 's/[:.-]//g')"
 309         fi
 310 
 311 
 312         for ((i=0; i<12; i=i+2)); do
 313                 MAC_HEX_ARRAY+=("${MAC_HEX:$i:2}")
 314         done
 315 
 316         ### APPLY PREFIX
 317         for ((i=0; i<6; i++)); do
 318                 BYTE="0x${MAC_HEX_ARRAY[$i]}"
 319                 MASK="0x${PREFIX_MASK_HEX_ARRAY[$i]}"
 320                 HEX="$(printf -- '%02X' "$((BYTE&~MASK))")"
 321                 MAC_MASKED_HEX_ARRAY+=("$HEX")
 322         done
 323 
 324         for ((i=0; i<6; i++)); do
 325                 PREFIX_BYTE="0x${PREFIX_HEX_ARRAY[$i]}"
 326                 MAC_BYTE="0x${MAC_MASKED_HEX_ARRAY[$i]}"
 327                 HEX="$(printf -- '%02X' "$((PREFIX_BYTE|MAC_BYTE))")"
 328                 MAC_HEX_ARRAY[$i]="$HEX"
 329         done
 330 
 331         ### APPLY MAC MASKs
 332         for ((i=0; i<6; i++)); do
 333                 BYTE="0x${MAC_HEX_ARRAY[$i]}"
 334                 MASK_AND="0x${MAC_MASK_HEX_ARRAY_AND[$i]}"
 335                 MASK_OR="0x${MAC_MASK_HEX_ARRAY_OR[$i]}"
 336                 HEX="0x$(printf -- '%02X' "$((BYTE&MASK_AND))")"
 337                 HEX="$(printf -- '%02X' "$((HEX|MASK_OR))")"
 338                 MAC_NEW_HEX_ARRAY+=("$HEX")
 339         done
 340 
 341         ### CREATE BINARY REPRESENTION
 342         for BYTE in "${MAC_HEX_ARRAY[@]}"; do
 343                 MAC_BIN_ARRAY+=("$(base2base 16 2 "$BYTE" | xargs printf '%08d' )")
 344         done
 345 
 346         for BYTE in "${MAC_NEW_HEX_ARRAY[@]}"; do
 347                 MAC_NEW_BIN_ARRAY+=("$(base2base 16 2 "$BYTE" | xargs printf '%08d' )")
 348         done
 349 
 350         ### FORMAT OUTPUT
 351         if [ "$OUT_SEP" == "cisco" ]; then
 352                         MAC_NEW+="${MAC_NEW_HEX_ARRAY[0]}"
 353                         MAC_NEW+="${MAC_NEW_HEX_ARRAY[1]}"
 354                 for ((i=2; i<6; i=i+2)); do
 355                         MAC_NEW+=".${MAC_NEW_HEX_ARRAY[$i]}"
 356                         MAC_NEW+="${MAC_NEW_HEX_ARRAY[$i+1]}"
 357                 done
 358         else
 359                 MAC_NEW="$(echo "${MAC_NEW_HEX_ARRAY[*]}" | sed "s/ /$OUT_SEP/g")"
 360         fi
 361 
 362         if $VERBOSE; then
 363                 cat <<-EOF
 364 
 365                         PREFIX:                  '$PREFIX'
 366                         PREFIX_LENGTH:           '$PREFIX_LENGTH'
 367                         PREFIX_HEX_ARRAY:        '${PREFIX_HEX_ARRAY[*]}'
 368                         PREFIX_MASK_BIN_ARRAY:   '${PREFIX_MASK_BIN_ARRAY[*]}'
 369                         PREFIX_MASK_HEX_ARRAY:   '${PREFIX_MASK_HEX_ARRAY[*]}'
 370                         PREFIX_MASK_BIN_ARRAY:   '${PREFIX_MASK_BIN_ARRAY[*]}'
 371                         PREFIX_MASK_HEX_ARRAY:   '${PREFIX_MASK_HEX_ARRAY[*]}'
 372                         MAC_RAW:                 '$MAC_RAW'
 373                         MAC_HEX:                 '$MAC_HEX'
 374                         MAC_BIN_ARRAY:           '${MAC_BIN_ARRAY[*]}'
 375                         MAC_MASK_BIN_ARRAY_AND:  '${MAC_MASK_BIN_ARRAY_AND[*]}'
 376                         MAC_MASK_BIN_ARRAY_OR:   '${MAC_MASK_BIN_ARRAY_OR[*]}'
 377                         MAC_NEW_BIN_ARRAY:       '${MAC_NEW_BIN_ARRAY[*],,}'
 378                         MAC_HEX_ARRAY:           '${MAC_HEX_ARRAY[*],,}'
 379                         MAC_MASK_HEX_ARRAY_AND:  '${MAC_MASK_HEX_ARRAY_AND[*],,}'
 380                         MAC_MASK_HEX_ARRAY_OR:   '${MAC_MASK_HEX_ARRAY_OR[*],,}'
 381                         MAC_NEW_HEX_ARRAY:       '${MAC_NEW_HEX_ARRAY[*],,}'
 382                         MAC_NEW:                 '${MAC_NEW,,}'
 383                 EOF
 384         else
 385                 echo "${MAC_NEW,,}"
 386         fi
 387 done

Local database

   1 aptitude install ieee-data

MAC-addresses are derived from the OUI prefixes.

   1 /usr/share/ieee-data/iab.csv
   2 /usr/share/ieee-data/iab.txt
   3 /usr/share/ieee-data/mam.csv
   4 /usr/share/ieee-data/mam.txt
   5 /usr/share/ieee-data/oui.csv    ### <--
   6 /usr/share/ieee-data/oui.txt    ### <--
   7 /usr/share/ieee-data/oui36.csv  ### <--
   8 /usr/share/ieee-data/oui36.txt  ### <--
   9 
  10 /usr/share/nmap/nmap-mac-prefixes

Spanning Tree Protocol Family

Ensure a loop free topology

Algorhyme

Radia Perlman penned this poem while she developed Spanning Tree.

I think that I shall never see
A graph more lovely than a tree.
A tree whose crucial property
Is loop-free connectivity.
A tree that must be sure to span
So packets can reach every LAN.
First, the root must be selected.
By ID, it is elected.
Least-cost paths from root are traced.
In the tree, these paths are placed.
A mesh is made by folks like me,
Then bridges find a spanning tree.

--
Radia Perlman

Planning

Root bridge
Root ports
Designated ports
Blocking/non-sesignated ports

Spanning Tree Protocol (STP)

Standards

IEEE 802.1d-1990 - STP
IEEE 802.1w-2003 - RSTP
- later IEEE 802.1D-2004
IEEE 802.1s-2003 - MSTP
- later IEEE 802.1Q-2014

Definitions

Bridge-ID (BID)

is 8 Byte long (2 Byte bridge priority, 6 Byte MAC address).

: When the bridge priority is equal the lower MAC address is elected.

Bridge Priority

is basically a 4bit number and locally assigned system ID extension (12 bits)

has only 16 values,

p * 2¹², where 0 ≤ p ≤ 2⁴

Default most of the time is 32768 = 8 * 2¹².

Root Bridge

of the spanning tree is the bridge with the smallest (lowest) bridge ID.

There can only be one root bridge in a spanning tree.

When initializing the protocol all bridges send out BPDUs, with themselves as root bridge.

After convergence only the root bridge generates BPDUs. Other devices only forward BPDUs.

Has no root port.

All the ports on the root bridge are designated ports.

Designated Bridge

Has exactly one root port

Device responsible to forwards frames to a LAN segment

Bridge Protocol Data Units (BPDUs)

are sent
- from the unique source MAC-address of a switch port
- to STP destination multicast MAC-address 01:80:C2:00:00:00 or 01:00:0C:CC:CC:CD in case of Cisco Per VLAN Spanning Tree (PVST)
2 types
- Configuration BPDU
- Topology Change Notification (TCN) BPDU

STP port states (4)

Blocking

: BPDUs are received and processed
: Frames are not forwarded
: This port would cause a topology loop
: May transit to forwarding state on failure of another link
: There is no link whose both sides are in state blocking.

Listening

: BPDUs are received and processed
: Frames are not forwarded
: MAC table is not populated
: May return to blocking

Learning

: BPDUs are received and processed
: Frames are not forwarded
: MAC table is populated
: May return to blocking

Forwarding

: BPDUs are received and processed
: Frames are forwarded
: May return to blocking

Disabled

: Not strictly part of STP
: Manually disabled switch port

Blocking -> Listening -> Learning -> Forwarding

STP Port roles (2)

Root

: A forwarding port that is the best port from non-root bridge to root bridge
: Port on which a device received the optimum configuration BPDU.

Designated

: A forwarding port for every LAN segment

Disabled

: Not strictly part of STP, a network administrator can manually disable a port

STP timers

Hello-Timer Default: 2s
Forward-Delay Default: 15s (1x Listening, 1x Learning)
Maximum Age Default: 20s

The root bridge sets the timer values and distributes these in Configuration BPDUs.

When a new device is attached it takes 2x Forward-Delay timers (default: 30s) to transit to the state Forwarding.

On failure of a link there is a convergence time of Maximum Age + 2 * Forward-Delay = 50s.

STP path cost

Lower path cost win
- calculated on bandwidth
  - The lower bandwidth, the higher cost
    - with STP originally 1Gbit/s divided by bandwidth
    - with RSTP 20Tbit/s divided by bandwidth
- admin can influence the path cost
- path cost to the root bridge add up along the path (across the switches)
If there are multiple upstream/designated bridges with equal cost to the root bridge,
the lower sender bridge-id wins
If there are multiple designated ports (no lag) to an upstream/designated bridge with equal cost to the root bridge,
the lower designated port-id wins
- Port ID = priority (4 bits) + ID (Interface number) (12 bits)
  - the default port priority is 128.
  - may be influenced by the admin
If this still has equal costs finally the local lowest Port ID wins.

Rapid STP (RSTP)

Standard IEEE 802.1D-2004 incorporates RSTP and obsoletes the original STP standard
Backwards-compatible with standard STP
Significantly faster convergence by introduction of new behaviours and new port roles
- usually responds to changes within 3 Hello times
Handshake between switches to determine if a rapid transition to the forwarding state is possible
1. RTSP bridges propagate their superior root bridge information to their designated ports
2. Receiving bridges
  1. sets all other ports to discarding.
  2. responds to this BPDUs with an BPDU with the agreement flag set.
3. Sending Bridge now knows, that it can transition directly to forwarding state, bypassing listening state.
Allows configuration and autodetection of edge ports, which directly transition to forwarding. Edge-ports transition to non-edge ports, when BPDUs are detected.

RSTP port roles (4)

Root

: A forwarding port that is the best port from non-root bridge to root bridge

Designated

: A forwarding port for every LAN segment

Alternate

: An alternate path to the root bridge.
: This path is different from using the root port.

Backup

: A backup/redundant path to a segment where another bridge port already connects

Disabled

: Not strictly part of STP, a network administrator can manually disable a port

RSTP port states (3)

Discarding

: BPDUs are received and processed
: Frames are not forwarded
: This port would cause a topology loop
: May transit to forwarding state on failure of another link

Learning

: BPDUs are received and processed
: Frames are not forwarded
: MAC table is populated
: May return to blocking

Forwarding

: BPDUs are received and processed
: Frames are forwarded
: May return to blocking

Disabled

: Not strictly part of STP
: manually disabled switch port

RSTP Timers

Hello-Timer Default: 2s
Forward-Delay Default: 15s (1x Listening, 1x Learning)
Maximum Age Default: 3x Hello-Timer (6s)

RSTP link types

Point-to-point

Full-duplex connection (switch)

Edge

if they are attached to a LAN that has no other bridges attached

Point-to-point port defined as non-invasive

depending on vendor, admin-edge or portfast mode

transition directly to the forwarding state

Avoids timing problems with DHCP (especially with conventional STP)

BPDUs are still observed

BPDU-guard should be active (-> error disabled)

Edge ports may be assumed by default (if BPDU -> non-edge).

: spanning-tree portfast default
: Every non-trunk port is considered edge.

RSTP improves convergence on point-to-point links by reducing the Max-Age time to 3 times Hello interval, removing the STP listening state, and exchanging a handshake between two switches to quickly transition the port to forwarding state.

Shared

Half-duplex connection (hub)

Handling like in STP

BPDU-filter

Simply removes BPDUs from traffic.
Maybe between two companies, that have their own distinct spanning tree and should not be influenced by another.

Per-VLAN-Spanning Tree (PV-STP)

Cisco proprietary
- Limited support across switch vendors
- Compatibility issues between
  - PVSTP(+) vendor implementations
  - compatible protocols like VLAN Spanning Tree Protocol (VSTP)
bridge system ID extension carries VLAN ID
Based on simple STP
One Spanning Tree instance per VLAN
- Adds significat overhead
- Consumes CPU-time on the networking device
VLAN encapsulation
- PVSTP uses (Cisco proprietary) Inter-Switch Link (ISL)
- PVSTP+ uses IEEE 802.1q
PVST+ can tunnel across an MSTP region.
Rapid PVSTP(+) (RPVSTP(+)) is based on RSTP instead.
Bridge Priority is altered.
- The VLAN ID is added to the defined priority. so 2 VLANs don not have the same bridge priority.

Multiple STP (MSTP)

Standard: IEEE 802.1s
Compatible to RSTP and thus to STP.
Bridge system ID extension carries the MSTP instance number
Allows usage of multiple Multiple spanning tree instances (MSTI)
- Instance 0 can be also seen as a catch-all.
Allows mapping multiple VLANs to a MSTI (based on mapping-tables)
- Allow load-sharing across links that otherwise would be blocked
- Reduces number of instances in comparison to PVSTP+ and therefor reduce the load to the processor.
  - Todays routers and switches should hold enough computing power to drive hundreds of STP instances. So RPVSTP+ is viable option again.
A switched network may be devided into multiple regions, with independent spanning trees.

Common Spanning Tree (CST)

: connects all MST regions in a switched network

Internal Spanning Tree (IST)

: runs in an MST region
: also named MSTI 0, a special MSTI to which all VLANs are mapped by default

Common and Internal Spanning Tree (CIST)

: connects all devices in a switched network
: consists of the ISTs in all MST regions and the CST

Regional root

: of the IST or a and MSTI within an MST region
: different MSTIs in a MST region may have different regional root bridges

Common root bridge

: root bridge of the CIST

MSTP Port roles (6)

6 Port states

Root port

: forwards data to the root bridge

Designated port

: forwards data to the designated bridge for a downstream network segment or device

Boundary port

port that connects a MST region to

: another MST region or
: a network-segment running STP, or RSTP

Master port

: root port (of a region) on the CIST to the common root bridge

Alternate port

: Backup port for a root port and master port
: Does not forward frames
: Takes over when the root port or master port has failed
: Starts forwarding without delay

Backup port

: Backup port of a designated port
: Starts forwarding without delay

Disabled port

: admin down in every MSTI

A port may have different roles in different MSTIs.

MSTP Port states (3)

Discarding
Learning
Forwarding

A port may have different states in different MSTIs.

Notes

Never use simple old STP.

Hold-down time (50s) is not reconcilable with high-availability.
If STP is in default configuration (with bridge-priority 32768) the election is based on the mac address. The root bridge bridge may be located in a inefficient position, e.g. far away from the router. If a STP-protocol is used in a network, it must be planned and configured carefully!
Different implementations of a standard are not guaranteed to work, due for example to differences in default timer settings.
Try to use (multi-chassis) link-aggregations ((MC)-LAGs) where possible, to avoid blocking redundant ports and enhance bandwidth.
Make sure to configure BPDU-guard to protect your network from malicious bridges (especially on ports of link type edge).
Edge-ports should be used
- to allow fast transition of terminal devices.
- in conjunction with BPDU-guard.
You may use a BPDU-filter to discard BPDUs from adjacent switching infrastructures, when it's clear that a loop can never be established.
Use loop detection on the edge of the network.

VLANs

IEEE 802.1Q
IEEE 802.1ad
- stacked VLANs, QinQ or double tagging
- Double tagging

Native VLAN

Native VLAN
: Frames of the native VLAN are not tagged on a trunk.
: Frames that arrive untagged on a trunk are put into the native VLAN.
: Native VLAN is usually by default VLAN 1, but can be changed on a trunk-base.
: On access ports the native VLAN loses its meaning.
: Some vendors also refer to a native VLAN as primary VLAN. This maybe the case, when native VLAN is never mentioned. The term "primary VLAN" unfortunately is ambiguous. So please do not mix it up with the primary VLAN of a private VLAN infrastructure (with secondary isolated, community VLANs and promiscuous, isolated and community ports like on Cisco devices). That's a different concept.

Network

Maximum Transmission Unit (MTU)

Aka Maximum Transfer Unit
Wiki EN Ethernet frame
Wiki EN Jumbo frame
Wiki EN Frame check sequence
Please also see
- #Path MTU discovery
- #MSS Clamping
kernel.org Doc - IP Sysctl
lwn.net - So you think you understand IP fragmentation?
IETF RFC791
https://wiki.wireshark.org/Ethernet
You should know the MTU, before you need to configure it.

MTU and related definitions

Maximum Transmission Unit (MTU)

In computer networking, the maximum transmission unit (MTU) is the size of the largest protocol data unit (PDU) that can be communicated in a single network layer transaction.

The MTU relates to, but is not identical to the maximum frame size that can be transported on the data link layer, e.g., Ethernet frame.

Packages larger than MTU need to be fragmented.

Packages with the flag "do not fragment" must not be fragmented and are instead discarded and answered with an ICMP "package to large".

Path MTU (PMTU)

The Internet Protocol defines the path MTU of an Internet transmission path as the smallest MTU supported by any of the hops on the path between a source and destination.

Put another way, the path MTU is the largest packet size that can traverse this path without suffering fragmentation.

Fragmentation can happen anywhere along a path.

A path across the network is not necessarily stable and can change any time (due to maintenance, outage, shorter ways, less cost,…). Fragmentation is a complex topic.

IP MTU

The MTUs reported by Linux or BSD in the output of ip is also called the IP MTU.

It excludes the headers of the encapsulating frame on layer 2.

So a frame on the wire will be greater.

Common numbers to be choosen as the IP MTU

: 1500 as stated by IEEE 802.3
: 9000 as recommended by Joint Engineering Team of Internet2

Most vendors nowadays support an MTU of 9216 Bytes

Routers often set the Layer 2 MTU and derive the IP MTU.

Typical headers that are "outside" of the IP MTU

Ethernet 18 Bytes
- 14 Bytes header
  - 2x MAC 6 Byte
  - Ether Type: 2 Byte 0x0800
- 4 Bytes trailer with the Frame Check Sequence (FCS)
  - (The FCS may not be shown in wireshark, because it is not supplied by underlying mechanisms.)
Optionally (Wiki EN) IEEE 802.1Q VLAN/QoS tag 4 Byte
- 2 Byte Tag protocol identifier (TPID) 0x8100
- 2 Byte Tag control information (TCI)
  - 3 bit Priority code point (PCP)
  - 1 bit Drop eligible indicator (DEI)
  - 12 bit VLAN identifier (VID)

Headers that are inside IP MTU

Optionally (Wiki EN) PPPoE 8 Byte
IPv4 header is greater or equal to 20 Byte in size.
- 1 Byte Version and header length
- 1 Byte Differentiated Services Field
- 2 Byte Total length
- 2 Byte Identification
- 2 Byte fragmentation field and fragmentation offset
- 1 Byte TTL
- 1 Byte Protocol
- 2 Byte Header checksum
- 2x 4 Byte source and destination address
Optionally ICMPv4 echo request header 8 Byte
- Type 1 Byte
- Code 1 Byte
- Checksum 2 Byte
- Identifier 2 Byte
- Sequence number 2 Byte

Typical resulting numbers

1472 - Max. size of a ICMPv4 Echo-Request on MTU 1500
1492 - DSL
- 1464 - Max. size of a ICMPv4 Echo-Request over DSL

Some notes

IPv6 required minimum PMTU: 1280
MTU may be detected via LLDP (TLV subtype 4)

MTUs an transceivers

Transceivers usually don't have an MTU. They only encode/decode the signals and stream the data to the PHY of the NIC or the switch/router ASIC. They do not process the Ethernet or PPPoE headers. Instead the MAC sublayer of the PHY introduces this limitations.

Path MTU cache

Be careful when adjusting MTUs.

A discovered PMTU is cached by linux for 600s (10min). Other OSs may do this as well.

Please see man ip-route for more information concerning the following commands.

The current value of the cache for a given prefix can be retrieved with
```
   1 ip route get to "$TARGET"
```
The full table can be inspected with
```
   1 ip route show cached
   2 ip route show table cache
   3 ip route show table cache root "$PREFIX"
```
This table can be flushed with
```
   1 # ip route flush cached
   2 ip route flush cached root "$PREFIX"
```

Take a look on the path MTU while adjusting the MTU settings in certain point of the network.

   1 TARGETS=(192.0.2.1 192.0.2.2)
   2 while sleep 5; do
   3         clear;
   4         for TARGET in "${TARGETS[@]}"; do
   5                 echo
   6                 ip route get to "$TARGET";
   7                 ping -M do -c 1 -w 1 -s 1472 "$TARGET";
   8         done
   9 done

Determine Path MTU on the shell

Traceroute

   1 traceroute -F --mtu -w 1 "$TARGET"

Tracepath

   1 tracepath -b "$TARGET"

A shell script

   1 for SIZE in $(seq 9000 1 8950) $(seq 1500 -1 1450); do
   2         echo -e "\nSize: '$SIZE'";
   3         ping -M do -c 1 -w 1 -s "$SIZE" "$TARGET";
   4         if [ "$?" -eq 0 ]; then
   5                 echo "The determined mtu is '$SIZE'."
   6                 break
   7         fi
   8         echo $?
   9 #        sleep 0.3;  
  10 done

Path MTU discovery

Links

Please see also #MSS Clamping

Properties

MSS is a field in the TCP-Header of a SYN-packet.
MSS cannot be altered during the connection.
MSS ≤ MTU - IP-Header (≥20Byte) - TCP-Header (≥20Byte)
With a smaller MSS the fragmentation can be avoided, but the overhead (header portion) increases with the number of packets.
Static routes may be used to set outgoing MSS.
PMTU discovery uses UDP random ports from the #ephemeral port range (32768-60999).
PMTU discovery uses ICMP packets for signaling failure, which also causes an extra wait of a round triptime, when the MTU on the path does not match.
- IPv4 - Fragmentation Needed (Type3, Code4)
- IPv6 - Packet Too Big (Type2, Code0)

In OPNsense there is a option System: Settings: Optimizations
net.inet.udp.blackhole - Do not send ICMP port unreachable messages for closed UDP ports
that is enabled by default(1). This could potentially disturb PMTU discovery and traceroute. At least the OPNsense does not show up in the output of tracepath.

For a successful tracepath UDP ephemeral ports (32768-60999) have to be opened. This corresponds to a firewall rule like the following

   1 Interface: !WAN (exclude WAN)
   2 Source: RFC1918 OR SPECIFIC PUBLIC NETWORK
   3 Source Port: ANY
   4 Destination: ANY
   5 Protocol: UDP
   6 Destination Port: 32768-60999
   7 Action: Allow
   8 Log: False
   9 Description: Allow PTMU discovery
  10 #Optimal:
  11 IP-Flags: DF (Do-Not-Fragment)

Please also make sure not to bind this rule to the WAN interface.

First-hop redundancy protocols (FHRP)

Wikipedia EN - First-hop redundancy protocol

Protocols: CARP, VRRP, HSRP, GLBP, ESRP, R-SMLT, NSRP

FHRP pitfalls

Many switches (like D-Link DGS-series switches (e.g. DGS-1210-16 Rev. A1, Netgear MS510TXM and probably this list is exhausting) seam to fail ARP in combination with CARP/VRRP gateways in default configuration.

In such cases the default gateway cannot be resolved to a MAC-address and routing fails generally. ARP resolution within the same network may still work and can be used to connect to and manage the device. This may be diagnosed using telnet and the command debug info (D-Link). The MAC of the CARP/VRRP address is only in the MAC-forwarding table, but missing in the host arp-table (mgmtVlan).

# WIP - exact feature has to be determined

Requirements to switches for using FHRP
https://docs.netgate.com/pfsense/en/latest/highavailability/index.html#switch-layer-2-concernsSwitch/docs.netgate.com - High Availability Layer 2 Concerns

Trouble Shooting

One hack that worked for IPMI interfaces. IPMI#LAN

Common Address Redundancy Protocol (CARP)

CARP is a secure, free alternative to the Virtual Router Redundancy Protocol (VRRP) and the Hot Standby Router Protocol (HSRP).

CARP allows multiple hosts on the same local network to share a set of IP addresses. Its primary purpose is to ensure that these addresses are always available, but in some configurations carp can also provide load balancing functionality.

Alternatives: CARP, VRRP, HSRP, GLBP, ESRP, R-SMLT, NSRP

Please also see ARP as it is closely related
#Address Resolution Protocol (ARP)

CARP numbers

VRRP and CARP use the same IANA number assignments.

Please see
#VRRP numbers

CARP sniffing

CARP Sniffing

   1 tcpdump -nepi igb0 -T carp carp
   2 16:05:05.024541 00:00:5e:00:01:01 > 01:00:5e:00:00:12, ethertype IPv4 (0x0800), length 70: 192.168.255.21 > 224.0.0.18: CARPv2-advertise 36: vhid=1 advbase=1 advskew=0 authlen=7 counter=14868393346680216794
   3 16:05:06.041066 00:00:5e:00:01:01 > 01:00:5e:00:00:12, ethertype IPv4 (0x0800), length 70: 192.168.255.21 > 224.0.0.18: CARPv2-advertise 36: vhid=1 advbase=1 advskew=0 authlen=7 counter=14868393346680216795
   4 16:05:07.056317 00:00:5e:00:01:01 > 01:00:5e:00:00:12, ethertype IPv4 (0x0800), length 70: 192.168.255.21 > 224.0.0.18: CARPv2-advertise 36: vhid=1 advbase=1 advskew=0 authlen=7 counter=14868393346680216796
   5 16:05:08.056781 00:00:5e:00:01:01 > 01:00:5e:00:00:12, ethertype IPv4 (0x0800), length 70: 192.168.255.21 > 224.0.0.18: CARPv2-advertise 36: vhid=1 advbase=1 advskew=0 authlen=7 counter=14868393346680216797

This is a typical ARP resolution for CARP addresses. You can see VRID = 0x01 (hex) in the last byte of the source MAC address. tcpdump -nepi igb0 arp

   1 tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
   2 listening on igb0, link-type EN10MB (Ethernet), capture size 262144 bytes
   3 18:37:10.376202 bc:2e:48:02:bf:bc > 00:00:5e:00:01:01, ethertype ARP (0x0806), length 60: Request who-has 192.168.255.20 tell 192.168.255.1, length 46
   4 18:37:10.376214 3c:ec:ef:ce:22:32 > bc:2e:48:02:bf:bc, ethertype ARP (0x0806), length 42: Reply 192.168.255.20 is-at 00:00:5e:00:01:01, length 28
   5 18:37:42.118744 bc:2e:48:02:bf:bc > 00:00:5e:00:01:01, ethertype ARP (0x0806), length 60: Request who-has 192.168.255.20 tell 192.168.255.1, length 46
   6 18:37:42.118751 3c:ec:ef:ce:22:32 > bc:2e:48:02:bf:bc, ethertype ARP (0x0806), length 42: Reply 192.168.255.20 is-at 00:00:5e:00:01:01, length 28

In the end the MAC address is never changed. But the frames may come in on another port of a switch, which may yield some trouble. The switch may filter it.

Virtual Router Redundancy Protocol (VRRP)

Alternatives: CARP, VRRP, HSRP, GLBP, ESRP, R-SMLT, NSRP

VRRP is using Cisco patents. All patents have expired in the mean time.

Implementation levels:

frrouting.org implements VRRP3 RFC5798 (recommended)
keepalived.org implements VRRP2 RFC2338
sourceforge.net vrrpd implements VRRP2 RFC2338

VRRP numbers

VRRP and CARP use the same IANA number assignments.

VRRP was assigned IPv4 multicast address 224.0.0.18 and IPv6 multicast address ff02::12.

IANA - IPv4 Multicast Address Space Registry

The OUI 00-00-5E has been assigned to IANA.
This includes 2**24 EUI-48 multicast identifiers 01-00-5E.

IANA Unicast 48-bit MAC Addresses

Addresses	Usage	Reference
`00-01-00` to `00-01-FF`	VRRP (Virtual Router Redundancy Protocol)	IETF RFC5798
`00-02-00` to `00-02-FF`	VRRP IPv6 (Virtual Router Redundancy Protocol IPv6)	IETF RFC5798

…

IANA Multicast 48-bit MAC Addresses

Addresses	Usage	Reference
`00-00-00` to `7F-FF-FF`	IPv4 Multicast	RFC1112

…

The 23 lower order bits of a multicast ip address are mapped to the respective bits of the MAC-multicast-address.

IETF RFC1112 - Host Extensions for IP Multicasting - Section 6.4 Extensions to an Ethernet Local Network Module

So the VRRP
IPv4 Multicast MAC address is 01:00:5e:00:00:12
IPv6 Multicast MAC address is 33:33:00:00:00:12.

So the VRRP Unicast MAC address is

IETF RFC5798 - Virtual Router Redundancy Protocol (VRRP) Version 3 for IPv4 and IPv6 - Virtual Router MAC Address
- IPv4 case: 00-00-5E-00-01-{VRID} (in hex, in Internet-standard bit- order) The first three octets are derived from the IANA's Organizational Unique Identifier (OUI). The next two octets (00-01) indicate the address block assigned to the VRRP for IPv4 protocol. {VRID} is the VRRP Virtual Router Identifier. This mapping provides for up to 255 IPv4 VRRP routers on a network.
  IPv6 case: 00-00-5E-00-02-{VRID} (in hex, in Internet-standard bit- order)

VRRPD

sourceforge.net vrrpd implements RFC2338
- Development has moved to
  github.com fredbcode/Vrrpd
  - This repository has been archived by the owner on Apr 14, 2023. It is now read-only.

So, the project is probably dead, the latest version in Debian is old as the hills and fails communicating with netlink.

IMHO: Don't use it.

However …

Install

   1 apt install vrrpd

This installed version 1.0-2+b2 of the package on Debian 11 (Bookworm).

vrrpd is a very minimal package with just the binary, some docs and the man-page.

Run it (as root)

   1 vrrpd -i bridge -v 11 -p 1 -d 1 -f /run -a "pw/0x12345678" 192.168.182.31
   2 RTNETLINK answers: Network is unreachable
   3 Can not talk with netlink interface...

Same without bridge and bond (active-backup) on the bare interface.

vrrpd was not working, because it was unable to set the IP-address when also propagating a virtual MAC (default behaviour without -n), whereas with -n set the IP-address correctly, undesireably was propagating the permanent hardware address.

tcpdump -nepi bridge 'vrrp'

   1 tcpdump: verbose output suppressed, use -v[v]... for full protocol decode
   2 listening on bridge, link-type EN10MB (Ethernet), snapshot length 262144 bytes
   3 10:11:44.951089 00:00:5e:00:01:0b > 01:00:5e:00:00:12, ethertype IPv4 (0x0800), length 54: 192.168.182.16 > 224.0.0.18: VRRPv2, Advertisement, vrid 11, prio 1, authtype simple, intvl 1s, length 20
   4 10:11:45.952245 00:00:5e:00:01:0b > 01:00:5e:00:00:12, ethertype IPv4 (0x0800), length 54: 192.168.182.16 > 224.0.0.18: VRRPv2, Advertisement, vrid 11, prio 1, authtype simple, intvl 1s, length 20
   5 10:11:46.985574 00:00:5e:00:01:0b > 01:00:5e:00:00:12, ethertype IPv4 (0x0800), length 54: 192.168.182.16 > 224.0.0.18: VRRPv2, Advertisement, vrid 11, prio 1, authtype simple, intvl 1s, length 20
   6 10:11:48.001304 00:00:5e:00:01:0b > 01:00:5e:00:00:12, ethertype IPv4 (0x0800), length 54: 192.168.182.16 > 224.0.0.18: VRRPv2, Advertisement, vrid 11, prio 1, authtype simple, intvl 1s, length 20

Keepalived

keepalived.org
github.com acassen/keepalived
man keepalived
man keepalived.conf

Install keepalived

   1 apt install keepalived

Keepalived doesn't start if the config file does not exist or is empty.

Configure keepalived

There are some nice sample configs in
/usr/share/doc/keepalived/sample

Distribute the config to all nodes and select the node specific interface if it differs /etc/keepalived/keepalived.conf

   1 ! Configuration File for keepalived
   2 
   3 vrrp_instance VI_0 {
   4     interface enp2s0
   5     virtual_router_id 10
   6     priority 50
   7     advert_int 1
   8     authentication {
   9         auth_type PASS
  10         auth_pass iu5zoh7S
  11     }
  12     virtual_ipaddress {
  13         172.16.1.1
  14     }
  15 }

Adjust the firewall to allow VRRP announcements

   1 ### ALLOW MULTICAST FOR KEEPALIVED
   2 iptables -t filter -A INPUT -i enp1s0  -d 224.0.0.18/32 -p vrrp -j ACCEPT
   3 iptables -t filter -I OUTPUT -o enp1s0 -d 224.0.0.18/32 -p vrrp -j ACCEPT
   4 iptables -t filter -A INPUT -i enp2s0  -d 224.0.0.18/32 -p vrrp -j ACCEPT
   5 iptables -t filter -I OUTPUT -o enp2s0 -d 224.0.0.18/32 -p vrrp -j ACCEPT

Follow the logs journalctl -fn 20 -u keepalived

   1 Mar 21 12:01:20 router1 Keepalived[11720]: Starting Keepalived v2.0.19 (10/19,2019)
   2 Mar 21 12:01:20 router1 Keepalived[11720]: WARNING - keepalived was build for newer Linux 5.4.166, running on Linux 5.4.0-126-generic #142-Ubuntu SMP Fri Aug 26 12:12:57 UTC 2022
   3 Mar 21 12:01:20 router1 Keepalived[11720]: Command line: '/usr/sbin/keepalived' '--dont-fork'
   4 Mar 21 12:01:20 router1 Keepalived[11720]: Opening file '/etc/keepalived/keepalived.conf'.
   5 Mar 21 12:01:20 router1 Keepalived[11720]: Starting VRRP child process, pid=11734
   6 Mar 21 12:01:20 router1 Keepalived_vrrp[11734]: Registering Kernel netlink reflector
   7 Mar 21 12:01:20 router1 Keepalived_vrrp[11734]: Registering Kernel netlink command channel
   8 Mar 21 12:01:20 router1 Keepalived_vrrp[11734]: Opening file '/etc/keepalived/keepalived.conf'.
   9 Mar 21 12:01:20 router1 Keepalived_vrrp[11734]: Registering gratuitous ARP shared channel
  10 Mar 21 12:01:20 router1 Keepalived_vrrp[11734]: (VI_0) Entering BACKUP STATE (init)
  11 Mar 21 12:01:23 router1 Keepalived_vrrp[11734]: (VI_0) Entering MASTER STATE
  12 ### SEND SIGUSR2 TO PID OF KEEPALIVED
  13 Mar 21 12:17:13 router1 Keepalived_vrrp[11734]: Printing VRRP stats for process(11734) on signal
  14 ### ALLOW MULTICAST TRAFFIC
  15 Mar 21 12:29:45 router1 Keepalived_vrrp[11734]: (VI_0) Entering BACKUP STATE
  16 Mar 21 12:31:02 router1 Keepalived_vrrp[11734]: Printing VRRP stats for process(11734) on signal

Get statistics
killall -USR2 keepalived

This command writes cat /tmp/keepalived.stats

   1 VRRP Instance: VI_0
   2   Advertisements:
   3     Received: 31482
   4     Sent: 1698
   5   Became master: 1
   6   Released master: 1
   7   Packet Errors:
   8     Length: 0
   9     TTL: 0
  10     Invalid Type: 0
  11     Advertisement Interval: 0
  12     Address List: 0
  13   Authentication Errors:
  14     Invalid Type: 0
  15     Type Mismatch: 0
  16     Failure: 0
  17   Priority Zero:
  18     Received: 0
  19     Sent: 0

Only the master sends VRRPv2 Advertisements to the network. If both hosts send advertisements, they may not be able to communicate with each other -> check firewalls tcpdump -ni eth0 vrrp

   1 tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
   2 listening on eth0, link-type EN10MB (Ethernet), capture size 262144 bytes
   3 21:29:49.835107 IP 172.16.1.2 > 224.0.0.18: VRRPv2, Advertisement, vrid 10, prio 100, authtype simple, intvl 1s, length 20
   4 21:29:50.835236 IP 172.16.1.2 > 224.0.0.18: VRRPv2, Advertisement, vrid 10, prio 100, authtype simple, intvl 1s, length 20
   5 21:29:51.835370 IP 172.16.1.2 > 224.0.0.18: VRRPv2, Advertisement, vrid 10, prio 100, authtype simple, intvl 1s, length 20
   6 21:29:52.835500 IP 172.16.1.2 > 224.0.0.18: VRRPv2, Advertisement, vrid 10, prio 100, authtype simple, intvl 1s, length 20
   7 21:29:53.835633 IP 172.16.1.2 > 224.0.0.18: VRRPv2, Advertisement, vrid 10, prio 100, authtype simple, intvl 1s, length 20
   8 21:29:54.835764 IP 172.16.1.2 > 224.0.0.18: VRRPv2, Advertisement, vrid 10, prio 100, authtype simple, intvl 1s, length 20
   9 21:29:55.835905 IP 172.16.1.2 > 224.0.0.18: VRRPv2, Advertisement, vrid 10, prio 100, authtype simple, intvl 1s, length 20
  10 21:29:56.836045 IP 172.16.1.2 > 224.0.0.18: VRRPv2, Advertisement, vrid 10, prio 100, authtype simple, intvl 1s, length 20
  11 21:29:57.836180 IP 172.16.1.2 > 224.0.0.18: VRRPv2, Advertisement, vrid 10, prio 100, authtype simple, intvl 1s, length 20
  12 ^C
  13 9 packets captured
  14 9 packets received by filter
  15 0 packets dropped by kernel

Free Range Routing (FRR)

FRRouting (FRR) is a free and open source Internet routing protocol suite for Linux and Unix platforms. It implements BGP, OSPF, RIP, IS-IS, PIM, LDP, BFD, Babel, PBR, OpenFabric and VRRP, with alpha support for EIGRP and NHRP.

Install

   1 apt install frr frr-pythontools frr-doc

The frr.service should have come up.

A basic setup should be performed
https://frrouting.readthedocs.io/en/latest/setup.html

FRR Crashlogs and Logbuffers

Paths are not affected by configurtration options:

Crashlogs in plaintext
/var/tmp/frr/<daemon>[-<instance>].<pid>/crashlog
Log-Buffers may contain unwritten logs which are null-byte terminated
/var/tmp/frr/<daemon>[-<instance>].<pid>/logbuf.<tid> and may be read with
tr '\0' '\n' < /var/tmp/frr/zebra.577515/logbuf.577519|less

FRR - Summoning daemons

The watchfrr, zebra and staticd daemons are always started.

vrrp is still disabled

   1 vtysh -c "show vrrp"
   2 vrrpd is not running

To enable the service set vrrpd=yes and reload the daemon.

/etc/frr/daemons

   1 # This file tells the frr package which daemons to start.
   2 #
   3 # Sample configurations for these daemons can be found in
   4 # /usr/share/doc/frr/examples/.
   5 #
   6 # ATTENTION:
   7 #
   8 # When activating a daemon for the first time, a config file, even if it is
   9 # empty, has to be present *and* be owned by the user and group "frr", else
  10 # the daemon will not be started by /etc/init.d/frr. The permissions should
  11 # be u=rw,g=r,o=.
  12 # When using "vtysh" such a config file is also needed. It should be owned by
  13 # group "frrvty" and set to ug=rw,o= though. Check /etc/pam.d/frr, too.
  14 #
  15 # The watchfrr, zebra and staticd daemons are always started.
  16 #
  17 bgpd=no
  18 ospfd=no
  19 ospf6d=no
  20 ripd=no
  21 ripngd=no
  22 isisd=no
  23 pimd=no
  24 pim6d=no
  25 ldpd=no
  26 nhrpd=no
  27 eigrpd=no
  28 babeld=no
  29 sharpd=no
  30 pbrd=no
  31 bfdd=no
  32 fabricd=no
  33 vrrpd=yes
  34 pathd=no
  35 
  36 #
  37 # If this option is set the /etc/init.d/frr script automatically loads
  38 # the config via "vtysh -b" when the servers are started.
  39 # Check /etc/pam.d/frr if you intend to use "vtysh"!
  40 #
  41 vtysh_enable=yes
  42 zebra_options="  -A 127.0.0.1 -s 90000000"
  43 bgpd_options="   -A 127.0.0.1"
  44 ospfd_options="  -A 127.0.0.1"
  45 ospf6d_options=" -A ::1"
  46 ripd_options="   -A 127.0.0.1"
  47 ripngd_options=" -A ::1"
  48 isisd_options="  -A 127.0.0.1"
  49 pimd_options="   -A 127.0.0.1"
  50 pim6d_options="  -A ::1"
  51 ldpd_options="   -A 127.0.0.1"
  52 nhrpd_options="  -A 127.0.0.1"
  53 eigrpd_options=" -A 127.0.0.1"
  54 babeld_options=" -A 127.0.0.1"
  55 sharpd_options=" -A 127.0.0.1"
  56 pbrd_options="   -A 127.0.0.1"
  57 staticd_options="-A 127.0.0.1"
  58 bfdd_options="   -A 127.0.0.1"
  59 fabricd_options="-A 127.0.0.1"
  60 vrrpd_options="  -A 127.0.0.1"
  61 pathd_options="  -A 127.0.0.1"
  62 
  63 
  64 # If you want to pass a common option to all daemons, you can use the
  65 # "frr_global_options" variable.
  66 #
  67 #frr_global_options=""
  68 
  69 
  70 # The list of daemons to watch is automatically generated by the init script.
  71 # This variable can be used to pass options to watchfrr that will be passed
  72 # prior to the daemon list.
  73 #
  74 # To make watchfrr create/join the specified netns, add the the "--netns"
  75 # option here. It will only have an effect in /etc/frr/<somename>/daemons, and
  76 # you need to start FRR with "/usr/lib/frr/frrinit.sh start <somename>".
  77 #
  78 #watchfrr_options=""
  79 
  80 
  81 # configuration profile
  82 #
  83 #frr_profile="traditional"
  84 #frr_profile="datacenter"
  85 
  86 
  87 # This is the maximum number of FD's that will be available.  Upon startup this
  88 # is read by the control files and ulimit is called.  Uncomment and use a
  89 # reasonable value for your setup if you are expecting a large number of peers
  90 # in say BGP.
  91 #
  92 #MAX_FDS=1024
  93 
  94 
  95 # For any daemon, you can specify a "wrap" command to start instead of starting
  96 # the daemon directly. This will simply be prepended to the daemon invocation.
  97 # These variables have the form daemon_wrap, where 'daemon' is the name of the
  98 # daemon (the same pattern as the daemon_options variables).
  99 #
 100 # Note that when daemons are started, they are told to daemonize with the `-d`
 101 # option. This has several implications. For one, the init script expects that
 102 # when it invokes a daemon, the invocation returns immediately. If you add a
 103 # wrap command here, it must comply with this expectation and daemonize as
 104 # well, or the init script will never return. Furthermore, because daemons are
 105 # themselves daemonized with -d, you must ensure that your wrapper command is
 106 # capable of following child processes after a fork() if you need it to do so.
 107 #
 108 # If your desired wrapper does not support daemonization, you can wrap it with
 109 # a utility program that daemonizes programs, such as 'daemonize'. An example
 110 # of this might look like:
 111 #
 112 # bgpd_wrap="/usr/bin/daemonize /usr/bin/mywrapper"
 113 #
 114 # This is particularly useful for programs which record processes but lack
 115 # daemonization options, such as perf and rr.
 116 #
 117 # If you wish to wrap all daemons in the same way, you may set the "all_wrap"
 118 # variable.
 119 #
 120 #all_wrap=""
 121

FRR - reload the daemon

Reloading FRR depends on frr-pythontools whjich provides /usr/lib/frr/frr-reload.py

   1 systemctl reload frr.service

vrrpd should now be started (shows no error any more).

   1 systemctl status frr.service
   2 ● frr.service - FRRouting
   3      Loaded: loaded (/lib/systemd/system/frr.service; enabled; preset: enabled)
   4      Active: active (running) since Wed 2023-04-26 14:37:26 CEST; 12s ago
   5        Docs: https://frrouting.readthedocs.io/en/latest/setup.html
   6     Process: 589952 ExecStart=/usr/lib/frr/frrinit.sh start (code=exited, status=0/SUCCESS)
   7    Main PID: 589962 (watchfrr)
   8      Status: "FRR Operational"
   9       Tasks: 9 (limit: 76893)
  10      Memory: 14.3M
  11         CPU: 184ms
  12      CGroup: /system.slice/frr.service
  13              ├─589962 /usr/lib/frr/watchfrr -d -F traditional zebra staticd vrrpd
  14              ├─589975 /usr/lib/frr/zebra -d -F traditional -A 127.0.0.1 -s 90000000
  15              ├─589980 /usr/lib/frr/staticd -d -F traditional -A 127.0.0.1
  16              └─589983 /usr/lib/frr/vrrpd -d -F traditional -A 127.0.0.1
  17 
  18 Apr 26 14:37:26 libertas watchfrr[589962]: [YFT0P-5Q5YX] Forked background command [pid 589963]: /usr/lib/frr/watchfrr.sh restart all
  19 Apr 26 14:37:26 libertas zebra[589975]: [VTVCM-Y2NW3] Configuration Read in Took: 00:00:00

The query for vrrp also shows no error any more

   1 vtysh -c "show vrrp"

FRR - PAM authentication

/etc/pam.d/frr

   1 # Any user may call vtysh but only those belonging to the group frrvty can
   2 # actually connect to the socket and use the program.
   3 auth    sufficient      pam_permit.so

FRR - vtysh

vtysh

   1 Hello, this is FRRouting (version 8.4.2).
   2 Copyright 1996-2005 Kunihiro Ishiguro, et al.
   3 
   4 libertas# 
   5 libertas# 
   6   add               Add registration
   7   clear             Reset functions
   8   configure         Configuration from vty interface
   9   copy              Copy from one file to another
  10   debug             Debugging functions
  11   disable           Turn off privileged mode command
  12   enable            Turn on privileged mode command
  13   end               End current mode and change to enable mode
  14   exit              Exit current mode and down to previous mode
  15   find              Find CLI command matching a regular expression
  16   graceful-restart  Graceful Restart commands
  17   list              Print command list
  18   mtrace            Multicast trace route to multicast source
  19   no                Negate a command or set its defaults
  20   output            Direct vtysh output to file
  21   ping              Send echo messages
  22   quit              Exit current mode and down to previous mode
  23   rpki              Control rpki specific settings
  24   show              Show running system information
  25   terminal          Set terminal line parameters
  26   traceroute        Trace route to destination
  27   watchfrr          Watchfrr Specific sub-command
  28   write             Write running configuration to memory, network, or terminal

Here we go!

FRR vrrpd

docs.frrouting.org VRRP

FRR uses MACVLAN interfaces to realize the virtual MAC requirement of the VRRP specification. Unfortunately FRR cannot yet create these interfaces itself. Create a MACVLAN device attached to enp8s0 using iproute2 utilities.

   1 ip link add vrrp4-2-1 link enp8s0 addrgenmode random type macvlan mode bridge
   2 ip link set dev vrrp4-2-1 address 00:00:5e:00:01:05
   3 ip addr add 10.0.2.16/24 dev vrrp4-2-1
   4 ip link set dev vrrp4-2-1 up
   5 
   6 ip link add vrrp6-2-1 link enp8s0 addrgenmode random type macvlan mode bridge
   7 ip link set dev vrrp6-2-1 address 00:00:5e:00:02:05
   8 ip addr add 2001:db8::370:7334/64 dev vrrp6-2-1
   9 ip link set dev vrrp6-2-1 up

You may also delete the interface with the following commands

   1 ip link del vrrp4-2-1 link enp8s0
   2 ip link del vrrp6-2-1 link enp8s0

You may also choose a more permanent solutioi in
/etc/network/interfaces

   1 auto vrrp4-2-1
   2 iface vrrp4-2-1 inet static
   3         pre-up ip link add vrrp4-2-1 link enp8s0 addrgenmode random type macvlan mode bridge
   4         hwaddress 00:00:5e:00:01:05
   5         address 192.168.182.31/24
   6         post-down ip link del vrrp4-2-1 link enp8s0
   7 
   8 iface vrrp6-2-1 inet6 static
   9         pre-up ip link add vrrp6-2-1 link enp8s0 addrgenmode random type macvlan mode bridge
  10         hwaddress 00:00:5e:00:02:05
  11         address 2001:db8::370:7334/64
  12         post-down ip link del vrrp6-2-1 link enp8s0

Now bring up/down the interfaces

   1 ifup vrrp4-2-1
   2 ifup vrrp6-2-1

Configure the virtual IP in vtysh

   1 vtysh 
   2                                                                                                                                                                                                                                                                                    
   3 Hello, this is FRRouting (version 8.4.2).
   4 Copyright 1996-2005 Kunihiro Ishiguro, et al.
   5                                                                                                                                                                                                                                                                                    
   6 libertas# 
   7 libertas# configure 
   8 libertas(config)# interface enp8s0 
   9 libertas(config-if)# vrrp 5 version 3
  10 libertas(config-if)# vrrp 5 priority 100
  11 libertas(config-if)# vrrp 5 advertisement-interval 1000
  12 libertas(config-if)# vrrp 5 shutdown
  13 libertas(config-if)# vrrp 5 ip 192.168.182.31
  14 libertas(config-if)# exit
  15 libertas(config)# exit
  16 libertas# show vrrp 
  17 
  18  Virtual Router ID                       5                   
  19  Protocol Version                        3                   
  20  Autoconfigured                          No                  
  21  Shutdown                                Yes                 
  22  Interface                               enp8s0              
  23  VRRP interface (v4)                     vrrp4-2-1           
  24  VRRP interface (v6)                     None                
  25  Primary IP (v4)                                             
  26  Primary IP (v6)                         ::                  
  27  Virtual MAC (v4)                        00:00:5e:00:01:05   
  28  Virtual MAC (v6)                        00:00:5e:00:02:05   
  29  Status (v4)                             Initialize          
  30  Status (v6)                             Initialize          
  31  Priority                                100                 
  32  Effective Priority (v4)                 100                 
  33  Effective Priority (v6)                 100                 
  34  Preempt Mode                            Yes                 
  35  Accept Mode                             Yes                 
  36  Advertisement Interval                  1000 ms             
  37  Master Advertisement Interval (v4) Rx   0 ms (stale)        
  38  Master Advertisement Interval (v6) Rx   0 ms (stale)        
  39  Advertisements Tx (v4)                  0                   
  40  Advertisements Tx (v6)                  0                   
  41  Advertisements Rx (v4)                  0                   
  42  Advertisements Rx (v6)                  0                   
  43  Gratuitous ARP Tx (v4)                  0                   
  44  Neigh. Adverts Tx (v6)                  0                   
  45  State transitions (v4)                  0                   
  46  State transitions (v6)                  0                   
  47  Skew Time (v4)                          0 ms                
  48  Skew Time (v6)                          0 ms                
  49  Master Down Interval (v4)               0 ms                
  50  Master Down Interval (v6)               0 ms                
  51  IPv4 Addresses                          1                   
  52  ..................................      192.168.182.31      
  53  IPv6 Addresses                          0                   
  54 
  55 libertas# 
  56 libertas# write terminal                                
  57 Building configuration...                      
  58                                                                     
  59 Current configuration:                                         
  60 !                                                 
  61 frr version 8.4.2                                  
  62 frr defaults traditional                        
  63 hostname libertas                             
  64 log syslog informational                         
  65 service integrated-vtysh-config                                                                                                          
  66 !                   
  67 interface enp8s0
  68  vrrp 5                                                      
  69  vrrp 5 ip 192.168.182.31                                    
  70 exit                                                         
  71 !                                                            
  72 end                                                          
  73 libertas# write file                                         
  74 Note: this version of vtysh never writes vtysh.conf          
  75 Building Configuration...                                    
  76 Integrated configuration saved to /etc/frr/frr.conf          
  77 [OK]                                                         
  78 libertas#

When issuing the command write file the following config was stored to
/etc/frr/frr.conf

   1 frr version 8.4.2
   2 frr defaults traditional
   3 hostname libertas
   4 log syslog informational
   5 service integrated-vtysh-config
   6 !
   7 interface enp8s0
   8  vrrp 5
   9  vrrp 5 ip 192.168.182.31
  10 exit
  11 !

Test

tcpdump -nepi enp8s0 'vrrp'

   1 15:24:51.666841 00:00:5e:00:01:05 > 01:00:5e:00:00:12, ethertype IPv4 (0x0800), length 46: 192.168.182.16 > 224.0.0.18: VRRPv3, Advertisement, vrid 5, prio 100, intvl 100cs, length 12
   2 15:24:52.667027 00:00:5e:00:01:05 > 01:00:5e:00:00:12, ethertype IPv4 (0x0800), length 46: 192.168.182.16 > 224.0.0.18: VRRPv3, Advertisement, vrid 5, prio 100, intvl 100cs, length 12
   3 15:24:53.667154 00:00:5e:00:01:05 > 01:00:5e:00:00:12, ethertype IPv4 (0x0800), length 46: 192.168.182.16 > 224.0.0.18: VRRPv3, Advertisement, vrid 5, prio 100, intvl 100cs, length 12
   4 15:24:54.667297 00:00:5e:00:01:05 > 01:00:5e:00:00:12, ethertype IPv4 (0x0800), length 46: 192.168.182.16 > 224.0.0.18: VRRPv3, Advertisement, vrid 5, prio 100, intvl 100cs, length 12
   5 15:24:55.667441 00:00:5e:00:01:05 > 01:00:5e:00:00:12, ethertype IPv4 (0x0800), length 46: 192.168.182.16 > 224.0.0.18: VRRPv3, Advertisement, vrid 5, prio 100, intvl 100cs, length 12

Router and the host itself can both ping the address.
ping 192.168.182.31

   1 PING 192.168.182.31 (192.168.182.31) 56(84) bytes of data.
   2 64 bytes from 192.168.182.31: icmp_seq=1 ttl=64 time=0.132 ms
   3 64 bytes from 192.168.182.31: icmp_seq=2 ttl=64 time=0.500 ms
   4 64 bytes from 192.168.182.31: icmp_seq=3 ttl=64 time=0.267 ms
   5 64 bytes from 192.168.182.31: icmp_seq=4 ttl=64 time=0.252 ms
   6 64 bytes from 192.168.182.31: icmp_seq=5 ttl=64 time=0.253 ms
   7 64 bytes from 192.168.182.31: icmp_seq=6 ttl=64 time=0.320 ms
   8 ^C
   9 --- 192.168.182.31 ping statistics ---
  10 6 packets transmitted, 6 received, 0% packet loss, time 5211ms
  11 rtt min/avg/max/mdev = 0.132/0.287/0.500/0.110 ms

It works.

Explicit Congestion Notification (ECN)

Please compare to

Explicit Congestion Notification (ECN) is an extension to the Internet Protocol and to the Transmission Control Protocol and is defined in RFC 3168 (2001). ECN allows end-to-end notification of network congestion without dropping packets. ECN is an optional feature that may be used between two ECN-enabled endpoints when the underlying network infrastructure also supports it.

Conventionally, TCP/IP networks signal congestion by dropping packets. When ECN is successfully negotiated, an ECN-aware router may set a mark in the IP header instead of dropping a packet in order to signal impending congestion. The receiver of the packet echoes the congestion indication to the sender, which reduces its transmission rate as if it detected a dropped packet.

Some OS defaults

OpenWRT: 0 disabled
Debian: 2 (on-demand)
Windows:
- Server 2012+: enabled (DCTCP)
- previous Windows versions and non-server versions: disabled
MacOS: enabled

ECN TCP initialization

Please see:

Already the SYN and SYN/ACK TCP segments have the reserved flag bits 8 (ECE [explicit congestion notification - echo] ) and/or 9 (CWR [explicit congestion notification - congestion window reduced]) set to negotiate if all hosts are ECN capable transports (ECTs). If the negotiation is successful in the ip header the least significant flag bits (10+11) of the field traffic class are set to 1 or 2. If congestion happens, a router on the path may set Congestion Experienced (CE) code point, which means both flags set (3) and the sender must reduce the pressure to the network.

ECN TCP Initialization sniffs

Neither host or at least HostA does not support ECN

   1 11:04:19.947303 IP HostA.53776 > HostB.https: Flags [S], seq 2068578816, win 42340, options [mss 1460,sackOK,TS val 2403502370 ecr 0,nop,wscale 12], length 0
   2 11:04:19.947585 IP HostB.https > HostA.53776: Flags [S.], seq 1151792898, ack 2068578817, win 65535, options [mss 1460,sackOK,TS val 2773609087 ecr 2403502370,nop,wscale 7], length 0
   3 11:04:19.947667 IP HostA.53776 > HostB.https: Flags [.], ack 1, win 11, options [nop,nop,TS val 2403502370 ecr 2773609087], length 0

HostA supports ECN, HostB not

   1 11:03:46.840333 IP HostA.44144 > HostB.https: Flags [SEW], seq 2208499727, win 42340, options [mss 1460,sackOK,TS val 2403469263 ecr 0,nop,wscale 12], length 0
   2 11:03:46.840542 IP HostB.https > HostA.44144: Flags [S.], seq 3316106916, ack 2208499728, win 65535, options [mss 1460,sackOK,TS val 2773575982 ecr 2403469263,nop,wscale 7], length 0
   3 11:03:46.840608 IP HostA.44144 > HostB.https: Flags [.], ack 1, win 11, options [nop,nop,TS val 2403469263 ecr 2773575982], length 0

Both hosts support ECN

   1 11:56:11.326494 IP HostA.35024 > HostB.https: Flags [SEW], seq 1721541561, win 42340, options [mss 1460,sackOK,TS val 2406613749 ecr 0,nop,wscale 12], length 0
   2 11:56:11.326722 IP HostB.https > HostA.35024: Flags [S.E], seq 96984504, ack 1721541562, win 65535, options [mss 1460,sackOK,TS val 2776720270 ecr 2406613749,nop,wscale 7], length 0
   3 11:56:11.326777 IP HostA.35024 > HostB.https: Flags [.], ack 1, win 11, options [nop,nop,TS val 2406613749 ecr 2776720270], length 0

ECN signalization in IP

Please see:
IETF RFC3168 - Section 5 - Explicit Congestion Notification in IP

Least significant flag bits (10+11) of the field traffic class in IP header.

   1       +-----+-----+
   2       | ECN FIELD |
   3       +-----+-----+
   4         ECT   CE         [Obsolete] RFC 2481 names for the ECN bits.
   5          0     0         Not-ECT
   6          0     1         ECT(1)
   7          1     0         ECT(0)
   8          1     1         CE

IPAM Documentation

Please use a IP address management / infrastructure resource planning tool to document your environments, like

Geo-Blocking

# TODO: Translate into english

Betrachtungen

Diskriminierung aufgrund der Herkunft
Mittel der Internet Zensur
VPNs, Anoymisierungen (wie z.B. TOR) und Proxies überwinden Geo-Blocking (spielend)
- Technisch Versierte wissen wie diese Werkzeuge einzusetzen sind.
- Ausgeschlossen werden einfache Nutzer ohne böswillige Motive.
Einschränkungen durch Geoblocking-Verordnung der EU
- Der EU-Binnenmarkt darf nicht geblockt werden.
- Bei falscher Implementierung unter Strafe gestellt
- Durchsetzung der Geoblocking-Verordnung durch Bundesnetzagentur
Es gibt meiner Meinung nach keine staatlichen Empfehlungen, welche Quellen (Kontinente, Länder) geblockt werden sollten.

Modalitäten

Kunde sollte durch den Dienstleister (wertungsfrei) auf die Implikationen hingewiesen werden.
Kunde muss die Einrichtung von Geo-Blocking explizit in Auftrag geben.
Kunde bestimmt die Inhalte der Allow- und Block-Listen.

Internet Protocol version 4 (IPv4)

Wiki EN Reserved IP addresses

IANA

In DNS make sure your NS, MX and A records for a given domain don't share a single IP-address.

Internet Protocol version 6 (IPv6)

IETF

IANA

Blog

Infoblox IPv6 Center of Excellence (COE) Blog

Best Current Practices

ripe.net - Best Common Practices - Best Current Operational Practice for Operators: IPv6 prefix assignment for end-users - persistent vs non-persistent, and what size to choose

Reasons for using IPv6

IPv4 address space is exhausted
Price increases quickly
New products, that need a IP-address, are possible
On fusions no readdressing is necessary, because the addresses are unique
Simplification

Abbreviations

CID: Company ID
DSCP: Differentiated Services Code Point
ECN: Explicit Congestion Notification
EUI: Extended Unique Identifier
IID: Modified EUI-64 interface identifiers
NAT64: Network Address Translation IPv6 to IPv4
NAT66: Network Address Translation IPv6 to IPv6
NPTv6: Network Prefix Translation IPv6-to-IPv6
ORCHIDv2: Overlay Routable Cryptographic Hash Identifiers Version 2
OUI: Organizationally Unique Identifier
RA: Route Advertisement
RS: Router Solicitation
SLAAC: Stateless Address Autoconfiguration
ULA: Unique Local Address

IPv6 Addresses

4bit are a nibble, which can be replaced by a hexadecimal digit -> 0-9a-f

Shortened	Full	Binary Prefix	Description
Addresses defined in RFC4291
`::/128`	`0:0:0:0:0:0:0:0`	`0:0:0:0:0:0:0:0`	Unspecified Address
`::1/128`	`0:0:0:0:0:0:0:1`	`0:0:0:0:0:0:0:1`	Loopback Address
`::AABB:CCDD`	`0:0:0:0:0:0:AABB:CCDD`	`0:0:0:0:0:0:`	IPv4-Compatible IPv6 Address (deprecated) where AABB:CCDD is a HEX encoded 32-bit IPv4 address
`::ffff:AABB:CCDD`	`0:0:0:0:0:ffff:AABB:CCDD`	`0:0:0:0:0:11111111 11111111:`	IPv4-Mapped IPv6 Address (RFC4038) where AABB:CCDD is a HEX encoded 32-bit IPv4 address
`2001::IID`		`00100000 00000001:`	Provider dependent address
`2001::/23`		`00100000 00000001:0000000`	IETF Protocol Assignments
`2001::/32`		`00100000 00000001:0:`	TEREDO
`2001:db8::/32`	`2001:db8:0:0:0:0:0:0/32`	`00100000 00000001:00001101 10111000:`	IETF RFC3849 - IPv6 Address Prefix Reserved for Documentation
`2001:1::1/128`	`2001:1:0:0:0:0:0:1/128`	`00100000 00000001:1:0:0:0:0:0:1`	Port Control Protocol Anycast
`2001:1::2/128`	`2001:1:0:0:0:0:0:2/128`	`00100000 00000001:1:0:0:0:0:0:2`	Traversal Using Relays around NAT Anycast
`2002::IID/16`	`2002:IID/16`	`00100000 00000010:`	6to4 addresses
`fc00::/7`	`fc00:IID`	`1111110`	IETF RFC4293 - Unique Local IPv6 Unicast Addresses
`fc00::/8`	`fc00:IID`	`11111100`	IETF RFC4293 - Unique Local IPv6 Unicast Addresses with local bit set to `0` is undefined and should not be used
`fd00::/8`	`fd00:IID`	`11111101`	IETF RFC4293 - Unique Local IPv6 Unicast Addresses with local bit set to `1`
`fe80::IID/64`	`fe80:0:0:0:IID`	`11111110 10`	Link-Local IPv6 Unicast Addresses
`fec0::IID/10`	`fec0:SNID:IID`	`11111110 11`	Site-Local IPv6 Unicast Addresses (deprecated in IETF RFC3879)
Multicast Addresses
`ffFS:GROUP-ID`	`ffFS:GROUP-ID`	`11111111 0RPTscop:`	Multicast Addresses where F defines flags and S defines the scope
`ff01::1`	`ff01:0:0:0:0:0:0:1`	`11111111 11111111:00000000 00000001:0:0:0:0:0:1`	ip6-allnodes (interface-local)
`ff02::1`	`ff02:0:0:0:0:0:0:1`	`11111111 11111111:00000000 00000010:0:0:0:0:0:1`	ip6-allnodes (link-local)
`ff02::1:ff00:0/104`	`ff02:0:0:0:0:1:ff00:0/104`	`11111111 11111111:00000000 00000010:0:0:0:00000000 00000001:11111111 000000000:0`	solicited-node multicast addresses (link-local)
`ff01::2`	`ff01:0:0:0:0:0:0:2`	`11111111 11111111:00000000 00000001:0:0:0:0:0:2`	ip6-allrouters (interface-local)
`ff02::2`	`ff02:0:0:0:0:0:0:2`	`11111111 11111111:00000000 00000010:0:0:0:0:0:2`	ip6-allrouters (link-local)
`ff02::5`	`ff02:0:0:0:0:0:0:5`	`11111111 11111111:00000000 00000010:0:0:0:0:0:5`	ip6-allspfrouters OSPFIGP (link-local)
`ff02::6`	`ff02:0:0:0:0:0:0:6`	`11111111 11111111:00000000 00000010:0:0:0:0:0:6`	ip6-alldrouters OSPFIGP Designated Routers (link-local)
`ff02::9`	`ff02:0:0:0:0:0:0:9`	`11111111 11111111:00000000 00000010:0:0:0:0:0:9`	ip6-allriprouters (link-local)
`ff02::a`	`ff02:0:0:0:0:0:0:a`	`11111111 11111111:00000000 00000010:0:0:0:0:0:a`	ip6-alleigrprouters (link-local)
`ff02::c`	`ff02:0:0:0:0:0:0:c`	`11111111 11111111:00000000 00000010:0:0:0:0:0:c`	Simple Service Discovery Protocol (link-local)
`ff02::ffXX:XXXX`	`ff02:0:0:0:0:1:ffXX:XXXX`	`11111111 11111111:00000000 00000010:0:0:0:0:11111111`	Solicited-Node Address, where XX:XXXX are low-order 24 bits taken of an address (unicast or anycast)
`ff05::2`	`ff05:0:0:0:0:0:0:2`	`11111111 11111111:00000000 00000101:0:0:0:0:0:2`	ip6-allrouters (site-local)
`ff05::c`	`ff05:0:0:0:0:0:0:c`	`11111111 11111111:00000000 00000101:0:0:0:0:0:c`	Simple Service Discovery Protocol (site-local)
Reserved Multicast Addresses
`ff00::/8`	`ff00:0:0:0:0:0:0:0`	`11111111 00000000:0:0:0:0:0:0:0`	reserved and shall never be assigned to any multicast group
`ff01::`	`ff01:0:0:0:0:0:0:0`	`11111111 00000001:0:0:0:0:0:0:0`	reserved and shall never be assigned to any multicast group
`ff02::`	`ff02:0:0:0:0:0:0:0`	`11111111 00000010:0:0:0:0:0:0:0`	reserved and shall never be assigned to any multicast group
`ff03::`	`ff03:0:0:0:0:0:0:0`	`11111111 00000011:0:0:0:0:0:0:0`	reserved and shall never be assigned to any multicast group
`ff04::`	`ff04:0:0:0:0:0:0:0`	`11111111 00000100:0:0:0:0:0:0:0`	reserved and shall never be assigned to any multicast group
`ff05::`	`ff05:0:0:0:0:0:0:0`	`11111111 00000101:0:0:0:0:0:0:0`	reserved and shall never be assigned to any multicast group
`ff06::`	`ff06:0:0:0:0:0:0:0`	`11111111 00000110:0:0:0:0:0:0:0`	reserved and shall never be assigned to any multicast group
`ff07::`	`ff07:0:0:0:0:0:0:0`	`11111111 00000111:0:0:0:0:0:0:0`	reserved and shall never be assigned to any multicast group
`ff08::`	`ff08:0:0:0:0:0:0:0`	`11111111 00001000:0:0:0:0:0:0:0`	reserved and shall never be assigned to any multicast group
`ff09::`	`ff09:0:0:0:0:0:0:0`	`11111111 00001001:0:0:0:0:0:0:0`	reserved and shall never be assigned to any multicast group
`ff0a::`	`ff0a:0:0:0:0:0:0:0`	`11111111 00001010:0:0:0:0:0:0:0`	reserved and shall never be assigned to any multicast group
`ff0b::`	`ff0b:0:0:0:0:0:0:0`	`11111111 00001011:0:0:0:0:0:0:0`	reserved and shall never be assigned to any multicast group
`ff0c::`	`ff0c:0:0:0:0:0:0:0`	`11111111 00001100:0:0:0:0:0:0:0`	reserved and shall never be assigned to any multicast group
`ff0d::`	`ff0d:0:0:0:0:0:0:0`	`11111111 00001101:0:0:0:0:0:0:0`	reserved and shall never be assigned to any multicast group
`ff0e::`	`ff0e:0:0:0:0:0:0:0`	`11111111 00001110:0:0:0:0:0:0:0`	reserved and shall never be assigned to any multicast group
`ff0f::`	`ff0f:0:0:0:0:0:0:0`	`11111111 00001111:0:0:0:0:0:0:0`	reserved and shall never be assigned to any multicast group

Multicast scopes

IETF RFC4291 - Multicast Addresses scope

scope is a 4-bit multicast scope value used to limit the scope of the multicast group.

The values are as follows:

0 reserved
1 Interface-Local scope
2 Link-Local scope
3 reserved
4 Admin-Local scope
5 Site-Local scope
6 (unassigned)
7 (unassigned)
8 Organization-Local scope
9 (unassigned)
A (unassigned)
B (unassigned)
C (unassigned)
D (unassigned)
E Global scope
F reserved

Hint for using link-local addresses:

When pinging a link-local address don't forget to specify the source interface -I $dev.

   1 ping -i eth0 fe80::5054:ff:fe70:67bc

You may also append the interface to the IP address separated with the percent notation.

   1 ping fe80::5054:ff:fe70:67bc%eth0
   2 ping fe80::5054:ff:fe70:67bc%2

Shortening

The following is an excerpt from
IETF RFC - A Recommendation for IPv6 Address Text Representation Section 4

4. A Recommendation for IPv6 Text Representation

A recommendation for a canonical text representation format of IPv6 addresses is presented in this section. The recommendation in this document is one that complies fully with [RFC4291], is implemented by various operating systems, and is human friendly. The recommendation in this section SHOULD be followed by systems when generating an address to be represented as text, but all implementations MUST accept and be able to handle any legitimate [RFC4291] format. It is advised that humans also follow these recommendations when spelling an address.

4.1. Handling Leading Zeros in a 16-Bit Field

Leading zeros MUST be suppressed. For example, 2001:0db8::0001 is not acceptable and must be represented as 2001:db8::1. A single 16- bit 0000 field MUST be represented as 0.

4.2. "::" Usage

4.2.1. Shorten as Much as Possible

The use of the symbol "::" MUST be used to its maximum capability. For example, 2001:db8:0:0:0:0:2:1 must be shortened to 2001:db8::2:1. Likewise, 2001:db8::0:1 is not acceptable, because the symbol "::" could have been used to produce a shorter representation 2001:db8::1.

4.2.2. Handling One 16-Bit 0 Field

The symbol "::" MUST NOT be used to shorten just one 16-bit 0 field. For example, the representation 2001:db8:0:1:1:1:1:1 is correct, but 2001:db8::1:1:1:1:1 is not correct.

4.2.3. Choice in Placement of "::"

When there is an alternative choice in the placement of a "::", the longest run of consecutive 16-bit 0 fields MUST be shortened (i.e., the sequence with three consecutive zero fields is shortened in 2001: 0:0:1:0:0:0:1). When the length of the consecutive 16-bit 0 fields are equal (i.e., 2001:db8:0:0:1:0:0:1), the first sequence of zero bits MUST be shortened. For example, 2001:db8::1:0:0:1 is correct representation.

4.3. Lowercase

The characters "a", "b", "c", "d", "e", and "f" in an IPv6 address MUST be represented in lowercase.

Examples:

In IPv4 1.0.0.1
00000001 00000000 00000000 00000001
may be shortened to 1.1
In IPv6 fe80:0000:0000:0000:0000:0000:0000:0001
1111111010000000 0000000000000000 0000000000000000 0000000000000000 0000000000000000 0000000000000000 0000000000000000 0000000000000001
shortens to fe80::1

Interface Identifiers (IID)

IETF RFC4291 - IPv6 Addressing Architecture - Appendix A: Creating Modified EUI-64 Format Interface Identifiers
IEEE Guidelines for Use of Extended Unique Identifier (EUI), Organizationally Unique Identifier (OUI), and Company ID (CID)
- Here's a local copy since I've seen many broken links before

For the structure of a MAC address
please see #MAC addresses

Deriving the IID

In IIDs the universal bit (X) of the MAC address is inverted.
Furthermore between the vendor (OUI) and the NIC-ID
ff:fe 11111111 11111110 is inserted.

   |0              1|1              3|3              4|4              6|
   |0              5|6              1|2              7|8              3|
   +----------------+----------------+----------------+----------------+
   |ccccccugcccccccc|cccccccc11111111|11111110mmmmmmmm|mmmmmmmmmmmmmmmm|
   +----------------+----------------+----------------+----------------+

Bash oneliner to calculate IID from MAC.

   1 MAC="01:23:45:aa:bb:cc";
   2 MAC_ARRAY=($(echo 0x$MAC |sed -r 's/[:-]/ 0x/g'))
   3 MAC_ARRAY[0]="$((${MAC_ARRAY[0]}^2))"
   4 printf "%02x%02x:%xff:fe%02x:%02x%02x" "${MAC_ARRAY[@]}"

Bash oneliner to calculate link-local address from MAC.

   1 MAC="01:23:45:aa:bb:cc";
   2 MAC_ARRAY=($(echo 0x$MAC |sed -r 's/[:-]/ 0x/g'))
   3 MAC_ARRAY[0]="$((${MAC_ARRAY[0]}^2))"
   4 printf "fe80::%x%02x:%xff:fe%02x:%x%02x/64" "${MAC_ARRAY[@]}"

6to4 addresses

Oneliner to generate a 6to4 address from an IPv4 address

   1 ipv4="1.2.3.4"; sla="5";
   2 printf "2002:%02x%02x:%02x%02x:%04x::1" \
   3         $(echo $ipv4 |tr "." " ") $sla

Unique Local Addresses (ULA)

IETF RFC4293 - Unique Local IPv6 Unicast Addresses

Provider independent address space
- Makes renumbering of the provider dependent addresses possible without disrupting the internal network.
Usable like any other global IPv6 unicast addresses
Well known prefix that can easily be filtered
Randomly generated with a high probability to be unique
- Merging site is possible without renumbering
- Routeable within and between a limited number of sites
- Not globally routable (actually filtered and rejected), because of their random bits their cannot be aggregated to shorter prefixes and would impair a great performance penalty in convergence of routing protocols.

Some important notes

3.2.1. Locally Assigned Global IDs
- Locally assigned Global IDs MUST be generated with a pseudo-random algorithm consistent with [RANDOM]. Section 3.2.2 describes a suggested algorithm. It is important that all sites generating Global IDs use a functionally similar algorithm to ensure there is a high probability of uniqueness. The use of a pseudo-random algorithm to generate Global IDs in the locally assigned prefix gives an assurance that any network numbered using such a prefix is highly unlikely to have that address space clash with any other network that has another locally assigned prefix allocated to it. This is a particularly useful property when considering a number of scenarios including networks that merge, overlapping VPN address space, or hosts mobile between such networks.
3.2.2. Sample Code for Pseudo-Random Global ID Algorithm
- The algorithm described below is intended to be used for locally assigned Global IDs. In each case the resulting global ID will be used in the appropriate prefix as defined in Section 3.2.
  1. Obtain the current time of day in 64-bit NTP format [NTP].
  2. Obtain an EUI-64 identifier from the system running this algorithm. If an EUI-64 does not exist, one can be created from a 48-bit MAC address as specified in [ADDARCH]. If an EUI-64 cannot be obtained or created, a suitably unique identifier, local to the node, should be used (e.g., system serial number).
  3. Concatenate the time of day with the system-specific identifier in order to create a key.
  4. Compute an SHA-1 digest on the key as specified in [FIPS, SHA1]; the resulting value is 160 bits.
  5. Use the least significant 40 bits as the Global ID.
  6. Concatenate fc00::/7, the L bit set to 1, and the 40-bit Global ID to create a Local IPv6 address prefix.
  This algorithm will result in a Global ID that is reasonably unique and can be used to create a locally assigned Local IPv6 address prefix.
4.1 Routing
- The default behavior of exterior routing protocol sessions between administrative routing regions must be to ignore receipt of and not
  advertise prefixes in the fc00::/7 block. A network operator may specifically configure prefixes longer than fc00::/7 for inter-site communication.
4.4. DNS Issues
- At the present time, AAAA and PTR records for locally assigned local IPv6 addresses are not recommended to be installed in the global DNS. …

ULA generation

github.com yoshi0808/ula-generator

   1 git clone https://github.com/yoshi0808/ula-generator.git
   2 cd ula-generator
   3 python3 ula_generator.py
   4 Input MAC address: 52:54:00:6c:69:60        
   5 
   6 ULA Prefix        -> fd30:6ab5:f9fa::/48
   7 First Subnet      -> fd30:6ab5:f9fa::/64
   8 Last Subnet       -> fd30:6ab5:f9fa:ffff::/64
   9 First IPv6 Address-> fd30:6ab5:f9fa::1/64

github.com ekline/ulagen

   1 git clone https://github.com/ekline/ulagen.git
   2 cd ulagen
   3 make
   4 impl/c/ulagen
   5 fd35:f14e:b221::/48
   6 impl/cpp/ulagen
   7 fdc3:2ae1:4fe1::/48

github.com adeverteuil/bash-ula-generator
```
   1 sudo apt install ntp
   2 git clone https://github.com/adeverteuil/bash-ula-generator
   3 cd bash-ula-generator
   4 bash gen-ula.sh
```
- Not recommended
  - Dependency ntp collides with systemd-timesyncd
  - ntpq command does not come back.

Linux Kernel options

There are numerous options supported by the Linux Kernel
kernel.org/doc latest /proc/sys/net/ipv6/* Variables

Suppress learning default route from route advertisement (ra)

kernel.org/doc latest Search for accept_ra_defrtr - BOOLEAN

Learn default router in Router Advertisement.

Functional default:

enabled if accept_ra is enabled.
disabled if accept_ra is disabled.

It is possible that your host has learned a default route from an internal interface.
ip -6 r |grep -e default -e ovs-pub2

   1 fe80::/64 dev ovs-pub2 proto kernel metric 256 pref medium
   2 default via fe80::1 dev enp7s0 metric 1024 onlink pref medium
   3 default via fe80::5054:ff:fece:d9b7 dev ovs-pub2 proto ra metric 1024 expires 1751sec hoplimit 64 pref medium

You may suppress this behaviour with
sysctl net.ipv6.conf.ovs-pub2.accept_ra_defrtr=0

   1 net.ipv6.conf.ovs-pub2.accept_ra_defrtr = 0

Reload the interface and check again

   1 ifdown ovs-pub2; ifup ovs-pub2
   2 ip -6 r |grep -e default -e ovs-pub2
   3 fe80::/64 dev ovs-pub2 proto kernel metric 256 pref medium
   4 default via fe80::1 dev enp7s0 metric 1024 onlink pref medium

To make it persistent

   1 cat > /etc/sysctl.d/net.ipv6.conf <<-EOF
   2 net.ipv6.conf.ovs-pub1.accept_ra_defrtr = 0
   3 net.ipv6.conf.ovs-pub2.accept_ra_defrtr = 0
   4 net.ipv6.conf.ovs-1a.accept_ra_defrtr = 0
   5 EOF

router advertisement daemon (radvd)

IETF Standards

Tools

IPv6 Toolkit
The ipv6calc Homepage
The Hacker Choice's IPv6 Attack Toolkit thc-ipv6

Internet Group Management Protocol (IGMP)

Wiki EN - Internet Group Management Protocol
IETF RFC1112 - Host Extensions for IP Multicasting
IETF RFC2236 - Internet Group Management Protocol, Version 2
man 8 netstat
man 8 ip-maddress
Similar to #Multicast Listener Discovery (MLD) in IPv6

Linux show all group memberships

   1 netstat -g
   2 ### WITH NUMERICAL OUTPUT
   3 netstat -gn

List multicast addresses

   1 ip maddr
   2 ### OR FOR A SPECIFIC DEVICE
   3 ip maddr show dev bridge

Internet Control Message Protocol version 6 (ICMPv6)

ICMPv6 Message Types

iana - Internet Control Message Protocol version 6 (ICMPv6) Parameters

ICMPv6 messages are grouped into two classes: error messages and informational messages. Error messages are identified as such by a zero in the high-order bit of their message Type field values. Thus, error messages have message types from 0 to 127; informational messages have message types from 128 to 255.

Neighbor Discovery Protocol (NDP)

The protocol defines five ICMPv6 packet types to perform functions for IPv6 similar to the #Address Resolution Protocol (ARP) and Internet Control Message Protocol (ICMP) Router Discovery and Router Redirect protocols for IPv4.

A solicited-node address is created by taking the least-significant 24 bits of a unicast or anycast address and appending them to the prefix ff02::1:ff00:0/104.

You can sniff NDP traffic longaside with ICMPv6.

   1 tcpdump -eni any icmp6

Multicast listener discovery (MLD)

Multicast Listener Discovery (MLD) is a component of the Internet Protocol Version 6 (IPv6) suite. MLD is used by IPv6 routers for discovering multicast listeners on a directly attached link, much like #Internet Group Management Protocol (IGMP) is used in IPv4. The protocol is embedded in ICMPv6 instead of using a separate protocol. MLDv1 is similar to IGMPv2 and MLDv2 similar to IGMPv3.

Multicast router discovery (MRD)

Multicast router discovery (MRD) provides a general mechanism for the discovery of multicast routers on an IP network. For IPv4 the mechanism is based on #Internet Group Management Protocol (IGMP), for IPv6 on #Multicast listener discovery (MLD).

ifmetric

0pointer.de ifmetric

ifmetric is a Linux tool for setting the metrics of all IPv4 routes attached to a given network interface at once. This may be used to change the priority of routing IPv4 traffic over the interface. Lower metrics correlate with higher priorities.

You may use this to e.g.

have a fallback interface in the same network or
control the default route based on the metric

Just add the keyword metric in the interface section in
/etc/network/interfaces

   1 auto enp8s0
   2 iface enp8s0 inet dhcp
   3         metric 10
   4 
   5 auto enp9s0
   6 iface enp9s0 inet dhcp
   7         metric 100

Take a lot at the routes
ip r

   1 default via 192.168.182.1 dev enp8s0 proto dhcp metric 10
   2 default via 192.168.182.1 dev enp9s0 proto dhcp metric 100 
   3 169.254.0.0/16 dev lo scope link metric 1000 
   4 192.168.182.0/24 dev enp8s0 proto kernel scope link src 192.168.182.16 metric 10
   5 192.168.182.0/24 dev enp9s0 proto kernel scope link src 192.168.182.162 metric 100

NetBIOS

is obsolete.
is not strictly a network layer protocol, this depends on its implementation.
may be implemented as
- NetBEUI (NetBEUI -> MAC)
- on top of IPX (NetBIOS -> IPX-address-> MAC)
- on top of TCP/IP (NetBIOS -> IP-address-> MAC)
is limited to a broadcast domain and is not routed.
causes a high amount of broadcast traffic.

Links

* IETF RFC1002 PROTOCOL STANDARD FOR A NetBIOS SERVICE ON A TCP/UDP TRANSPORT: DETAILED SPECIFICATIONS

NetBIOS name

reservations are broadcasted to the broadcast-address followed by a time the requestor listens for complains from the network (already registered). This procedure is repeated 4 times, until the name is considered reserved.
are 16Byte long.
may be "unique (U)" or a "group (G)".
- Please avoid assigning the name as a workgroup to a node. This may lead to problems on the node.
Microsoft's implementation of NetBIOS uses the 16th byte of the NetBIOS Name as a type field, which is also known as the NetBIOS-Suffix.

Patrick P. Yeung NetBIOS Suffix Code Table

Name	Number (HEX)	Type	Usage
`<computername>`	`00`	U	Workstation Service
`<computername>`	`01`	U	Messenger Service
`<computername>`	`03`	U	Messenger Service
`<computername>`	`06`	U	RAS Server Service
`<computername>`	`1f`	U	NetDDE Service
`<computername>`	`20`	U	File Server Service
`<computername>`	`21`	U	RAS Client Service
`<computername>`	`22`	U	Exchange Interchange
`<computername>`	`23`	U	Exchange Store
`<computername>`	`24`	U	Exchange Directory
`<computername>`	`30`	U	Modem Sharing Server Service
`<computername>`	`31`	U	Modem Sharing Client Service
`<computername>`	`43`	U	SMS Client Remote Control
`<computername>`	`44`	U	SMS Admin Remote Control Tool
`<computername>`	`45`	U	SMS Client Remote Chat
`<computername>`	`46`	U	SMS Client Remote Transfer
`<computername>`	`4c`	U	DEC Pathworks TCPIP Service
`<computername>`	`52`	U	DEC Pathworks TCPIP Service
`<computername>`	`87`	U	Exchange MTA
`<computername>`	`6a`	U	Exchange IMC
`<computername>`	`be`	U	Network Monitor Agent
`<computername>`	`bf`	U	Network Monitor Apps
`<username>`	`03`	U	Messenger Service
`<\\_MSBROWSE_>`	`01`	G	Master Browser
`<domain>`	`00`	G	Domain Name (Membership)
`<domain>`	`1b`	U	Domain Master Browser
`<domain>`	`1c`	G	Domain Controllers, Domain Logon Server
`<domain>`	`1d`	U	(Local) Master Browser
`<domain>`	`1e`	G	(Local) Browser Service Elections
`<INet~Services>`	`1c`	G	Internet Information Server
`<IS~Computer_name>`	`00`	U	Internet Information Server

`<computername>`	`[2b]`	U	Lotus Notes Server
`IRISMULTICAST`	`[2f]`	G	Lotus Notes
`IRISNAMESERVER`	`[33]`	G	Lotus Notes
`Forte_$ND800ZA`	`[20]`	U	DCA Irmalan Gateway Service

NetBIOS Name resolution order in MS Windows

Order of resolution

Cache
- contains static entries
  from LMHOSTS with the tag #PRE
WINS
Broadcast
LMHOSTS.SAM
DNS (if configured)

lmhosts

Static NetBIOS name resolution like /etc/hosts

LMHOSTS.SAM

   1 192.168.1.11    samba #PRE
   2 #INCLUDE        \\samba\public\lmhosts
   3

Keywords (case sensitive, labels are case-insensitive):

#PRE enforces lookup of hostname in LMHOSTS.SAM without resolution over the network.
- These entries are loaded into the NetBIOS cache on boot.
- The NetBIOS cache can be flushed and repopulated with nbtstat -R.
#INCLUDE allows including another file, which may also reside on a share.

WINS

Microsoft Docs - Windows Internet Name Service (WINS)

WINS is as obsolete as NetBIOS is.

There can only be one WINS server in a given network or the NetBIOS namespace is divided (even with multiple workgroups or domains). Replicating WINS servers may be used (Windows or samba4WINS).

WINS uses directed UDP unicast, which is routed across network boundaries. Please make sure the firewall is open on udp/137.

Names, which are registered in WINS, are resolved using directed udp-unicast, otherwise they are resolved using broadcasts.

WINS works without wait times for complaints and thus is faster and conserves resources.

Consider using WINS even in small networks to reduce NetBIOS broadcasts.

WINS names are registered on boot of a node (e.g. in case the WINS server was changed during runtime).

The reservation is only valid for a limited amount of time and has to be renewed in regular intervals. The intervals can be chosen by the client, but the server defines minimum and maximum boundries.

Via dhcp WINS server is distributed as netbios-name-servers. option netbios-node-type 8; ### DEFAULT may be set to explicitly define NetBIOS behaviour.

type	short name	long name	resolution
1	B-Node	broadcast	broadcast
2	P-Node	point-to-point	WINS
4	M-Node	mixed	1^st broadcast, 2^nd WINS
8	H-Node	hybrid	1^st WINS, 2^nd broadcast

Troubleshooting

Martians

A Martian packet is an IP packet seen on the public Internet that contains a source or destination address that is reserved for special-use by Internet Assigned Numbers Authority (IANA) as defined in RFC 1812, Appendix B Glossary (Martian Address Filtering). On the public Internet, such a packet either has a spoofed source address, and it cannot actually originate as claimed, or the packet cannot be delivered. The requirement to do this is found in RFC 1812, Section 5.3.7 (Martian Address Filtering). Martian packets commonly arise from IP address spoofing in denial-of-service attacks, but can also arise from network equipment malfunction or misconfiguration of a host. In Linux terminology, a martian packet is an IP packet received by the kernel on a specific interface, while routing tables indicate that the source IP is expected on another interface. The name is derived from packet from Mars, meaning that packet seems to be not of this Earth.

There are many reasons.

Asymmetric routing in a multi-homing setup
Same Broadcast-Domain multiple IP-adresses
Address-spoofing
…

Transport

Ephemeral Ports

Wiki EN Ephemeral Ports

Range

The Internet Assigned Numbers Authority (IANA) suggests the range 49152 to 65535 (2¹⁵ + 2¹⁴ to 2¹⁶−1) for dynamic or private ports.

Many Linux kernels use the port range 32768 to 61000. FreeBSD has used the IANA port range since release 4.6. Previous versions, including the Berkeley Software Distribution (BSD), use ports 1024 to 5000 as ephemeral ports.

View and customize ephemeral ports range

   1 cat /proc/sys/net/ipv4/ip_local_port_range
   2 32768   60999
   3 sysctl net.ipv4.ip_local_port_range
   4 net.ipv4.ip_local_port_range = 32768    60999

Microsoft Windows operating systems through XP use the range 1025–5000 as ephemeral ports by default. Windows Vista, Windows 7, and Server 2008 use the IANA range by default. Windows Server 2003 uses the range 1025–5000 by default, until Microsoft security update MS08-037 from 2008 is installed, after which it uses the IANA range by default. Windows Server 2008 with Exchange Server 2007 installed has a default port range of 1025–60000. In addition to the default range, all versions of Windows since Windows 2000 have the option of specifying a custom range anywhere within 1025–65535.

MSS Clamping

MSS clamping is a technique of TCP to reduce MSS (to MSS=MTU-40Byte) of a SYN package on a Customer Premisses Equipment (CPE), when the uplink has a lower MTU then the internal networks.
It has been developed to avoid slow #Path MTU discovery, which also often fails, e.g. when ICMP packages that signal failure ("Fragmentation needed." or "Package to big.") are block by a stateful package filter.
Since MSS clamping is usually done in software on the CPU, that's why MSS clamping should not be performed on a provider equipment (PE) router, This would open a vector for DoS attacks, who could send SYN packages to block a the bus to the CPU or eat up all CPU.
UDP has no equivalent to MSS clamping. To avoid extra roundtrip times and fragmentation
- DNS server
  - should set the UDP playload size to a lower value (1220Byte).
  - should respect a clients UDP payload size.
- DNS clients sohould set their UDP payload size to a lower value. Please see
  - IETF RFC4035 section 4.1
  - Recommended EDNS buffer size

In OpenWRT MSS clamping can be activated in

Menu Point: Network -> Firewall
Choose the "WAN" zone press the "Edit" button.
Enable "MSS Claming" an "Save" the change.
"Save and Apply" the change.

/etc/config/firewall

   1 config zone
   2         option name 'wan'
   3         option input 'REJECT'
   4         option output 'ACCEPT'
   5         option forward 'REJECT'
   6         option masq '1'
   7         option mtu_fix '1'
   8         list network 'wan'
   9         list network 'wan6'

Hardware info

PCI devices

   1 lspci -vv -s 00:1f.6

Tools

Avahi

Avahi is a free zero-configuration networking (zeroconf) implementation, including a system for multicast DNS/DNS-SD service discovery (like Apples proprietary "Bonjour").

avahi-discover

Show a real-time graphical browse list for mDNS/DNS-SD network services running on the local LAN using the Avahi daemon.

Very useful service discovery.

Install

   1 apt install avahi-discover

Use

   1 avahi-discover

avahi-ui-utils

bssh/bvnc/bshell browses for SSH/VNC servers on the local network, shows them in a GUI for the user to select one and finally calls ssh/vncviewer after a selection was made.

Install

   1 apt install avahi-ui-utils

Use

   1 bshell

avahi-utils

Avahi browsing, publishing and discovery utilities Avahi is a fully LGPL framework for Multicast DNS Service Discovery. It allows programs to publish and discover services and hosts running on a local network with no specific configuration. For example you can plug into a network and instantly find printers to print to, files to look at and people to talk to. This package contains several utilities that allow you to interact with the Avahi daemon, including publish, browsing and discovering services.

Install

   1 apt install avahi-utils

A hole set of tools

   1 avahi-browse
   2 avahi-publish
   3 avahi-resolve
   4 avahi-set-host-name
   5 avahi-browse-domains
   6 avahi-publish-address
   7 avahi-publish-service
   8 avahi-resolve-address
   9 avahi-resolve-host-name

Use

   1 avahi-browse -ar

dsniff

https://www.monkey.org/~dugsong/dsniff/

dsniff is a collection of tools for network auditing and penetration testing. dsniff, filesnarf, mailsnarf, msgsnarf, urlsnarf, and webspy passively monitor a network for interesting data (passwords, e-mail, files, etc.). arpspoof, dnsspoof, and macof facilitate the interception of network traffic normally unavailable to an attacker (e.g, due to layer-2 switching). sshmitm and webmitm implement active monkey-in-the-middle attacks against redirected SSH and HTTPS sessions by exploiting weak bindings in ad-hoc PKI.

arpspoof - Send out unrequested (and possibly forged) arp replies.
dnsspoof - forge replies to arbitrary DNS address / pointer queries on the Local Area Network.
dsniff - password sniffer for several protocols.
filesnarf - saves selected files sniffed from NFS traffic.
macof - flood the local network with random MAC addresses.
mailsnarf - sniffs mail on the LAN and stores it in mbox format.
msgsnarf - record selected messages from different Instant Messengers.
sshmitm - SSH monkey-in-the-middle. proxies and sniffs SSH traffic.
sshow - SSH traffic analyser.
tcpkill - kills specified in-progress TCP connections.
tcpnice - slow down specified TCP connections via "active" traffic shaping.
urlsnarf - output selected URLs sniffed from HTTP traffic in CLF.
webmitm - HTTP / HTTPS monkey-in-the-middle. transparently proxies.
webspy - sends URLs sniffed from a client to your local browser (requires libx11-6 installed). Please do not abuse this software.

Install

   1 apt install dsniff

ethtool

https://www.kernel.org/pub/software/network/ethtool/

Display or change Ethernet device settings. ethtool can be used to query and change settings such as speed, auto- negotiation and checksum offload on many network devices, especially Ethernet devices.

Gather info

Get basic link state (speed, duplex, MDI-X, link, autoneg …)

   1 # ethtool enp0s31f6 
   2 Settings for enp0s31f6:
   3         Supported ports: [ TP ]
   4         Supported link modes:   10baseT/Half 10baseT/Full 
   5                                 100baseT/Half 100baseT/Full 
   6                                 1000baseT/Full 
   7         Supported pause frame use: No
   8         Supports auto-negotiation: Yes
   9         Supported FEC modes: Not reported
  10         Advertised link modes:  10baseT/Half 10baseT/Full 
  11                                 100baseT/Half 100baseT/Full 
  12                                 1000baseT/Full 
  13         Advertised pause frame use: No
  14         Advertised auto-negotiation: Yes
  15         Advertised FEC modes: Not reported
  16         Speed: 1000Mb/s
  17         Duplex: Full
  18         Port: Twisted Pair
  19         PHYAD: 1
  20         Transceiver: internal
  21         Auto-negotiation: on
  22         MDI-X: on (auto)
  23         Supports Wake-on: pumbg
  24         Wake-on: g
  25         Current message level: 0x00000007 (7)
  26                                drv probe link
  27         Link detected: yes

Queries the specified network device for associated driver information.

   1 ethtool -i enp0s31f6

Queries the specified network device for the state of protocol offload and other features.

   1 # ethtool -k enp0s31f6
   2 Features for enp0s31f6:
   3 rx-checksumming: on
   4 tx-checksumming: on
   5         tx-checksum-ipv4: off [fixed]
   6         tx-checksum-ip-generic: on
   7         tx-checksum-ipv6: off [fixed]
   8         tx-checksum-fcoe-crc: off [fixed]
   9         tx-checksum-sctp: off [fixed]
  10 scatter-gather: on
  11         tx-scatter-gather: on
  12         tx-scatter-gather-fraglist: off [fixed]
  13 tcp-segmentation-offload: on
  14         tx-tcp-segmentation: on
  15         tx-tcp-ecn-segmentation: off [fixed]
  16         tx-tcp-mangleid-segmentation: off
  17         tx-tcp6-segmentation: on
  18 udp-fragmentation-offload: off
  19 generic-segmentation-offload: on
  20 generic-receive-offload: on
  21 large-receive-offload: off [fixed]
  22 rx-vlan-offload: on
  23 tx-vlan-offload: on
  24 ntuple-filters: off [fixed]
  25 receive-hashing: on
  26 highdma: on [fixed]
  27 rx-vlan-filter: off [fixed]
  28 vlan-challenged: off [fixed]
  29 tx-lockless: off [fixed]
  30 netns-local: off [fixed]
  31 tx-gso-robust: off [fixed]
  32 tx-fcoe-segmentation: off [fixed]
  33 tx-gre-segmentation: off [fixed]
  34 tx-gre-csum-segmentation: off [fixed]
  35 tx-ipxip4-segmentation: off [fixed]
  36 tx-ipxip6-segmentation: off [fixed]
  37 tx-udp_tnl-segmentation: off [fixed]
  38 tx-udp_tnl-csum-segmentation: off [fixed]
  39 tx-gso-partial: off [fixed]
  40 tx-sctp-segmentation: off [fixed]
  41 tx-esp-segmentation: off [fixed]
  42 tx-udp-segmentation: off [fixed]
  43 fcoe-mtu: off [fixed]
  44 tx-nocache-copy: off
  45 loopback: off [fixed]
  46 rx-fcs: off
  47 rx-all: off
  48 tx-vlan-stag-hw-insert: off [fixed]
  49 rx-vlan-stag-hw-parse: off [fixed]
  50 rx-vlan-stag-filter: off [fixed]
  51 l2-fwd-offload: off [fixed]
  52 hw-tc-offload: off [fixed]
  53 esp-hw-offload: off [fixed]
  54 esp-tx-csum-hw-offload: off [fixed]
  55 rx-udp_tunnel-port-offload: off [fixed]
  56 tls-hw-tx-offload: off [fixed]
  57 tls-hw-rx-offload: off [fixed]
  58 rx-gro-hw: off [fixed]
  59 tls-hw-record: off [fixed]

Alter

Restart auto-negotiation if enabled

   1 ethtool -r eth0

De/activate features of a nic

Examples:

scatter-gather (sg)
TCP-segmentation-offloading (tso)
generic-segmentation-offload (gro)

   1 # ethtool -K enp0s31f6 sg off tso off gro off
   2 Actual changes:
   3 scatter-gather: off
   4         tx-scatter-gather: off
   5 tcp-segmentation-offload: off
   6         tx-tcp-segmentation: off
   7         tx-tcp6-segmentation: off
   8 generic-segmentation-offload: off [requested on]
   9 generic-receive-offload: off

Post deactivation

   1 # ethtool -k enp0s31f6
   2 Features for enp0s31f6:
   3 rx-checksumming: on
   4 tx-checksumming: on
   5         tx-checksum-ipv4: off [fixed]
   6         tx-checksum-ip-generic: on
   7         tx-checksum-ipv6: off [fixed]
   8         tx-checksum-fcoe-crc: off [fixed]
   9         tx-checksum-sctp: off [fixed]
  10 scatter-gather: off
  11         tx-scatter-gather: off
  12         tx-scatter-gather-fraglist: off [fixed]
  13 tcp-segmentation-offload: off
  14         tx-tcp-segmentation: off
  15         tx-tcp-ecn-segmentation: off [fixed]
  16         tx-tcp-mangleid-segmentation: off
  17         tx-tcp6-segmentation: off
  18 udp-fragmentation-offload: off
  19 generic-segmentation-offload: off [requested on]
  20 generic-receive-offload: off
  21 large-receive-offload: off [fixed]
  22 rx-vlan-offload: on
  23 tx-vlan-offload: on
  24 ntuple-filters: off [fixed]
  25 receive-hashing: on
  26 highdma: on [fixed]
  27 rx-vlan-filter: off [fixed]
  28 vlan-challenged: off [fixed]
  29 tx-lockless: off [fixed]
  30 netns-local: off [fixed]
  31 tx-gso-robust: off [fixed]
  32 tx-fcoe-segmentation: off [fixed]
  33 tx-gre-segmentation: off [fixed]
  34 tx-gre-csum-segmentation: off [fixed]
  35 tx-ipxip4-segmentation: off [fixed]
  36 tx-ipxip6-segmentation: off [fixed]
  37 tx-udp_tnl-segmentation: off [fixed]
  38 tx-udp_tnl-csum-segmentation: off [fixed]
  39 tx-gso-partial: off [fixed]
  40 tx-sctp-segmentation: off [fixed]
  41 tx-esp-segmentation: off [fixed]
  42 tx-udp-segmentation: off [fixed]
  43 fcoe-mtu: off [fixed]
  44 tx-nocache-copy: off
  45 loopback: off [fixed]
  46 rx-fcs: off
  47 rx-all: off
  48 tx-vlan-stag-hw-insert: off [fixed]
  49 rx-vlan-stag-hw-parse: off [fixed]
  50 rx-vlan-stag-filter: off [fixed]
  51 l2-fwd-offload: off [fixed]
  52 hw-tc-offload: off [fixed]
  53 esp-hw-offload: off [fixed]
  54 esp-tx-csum-hw-offload: off [fixed]
  55 rx-udp_tunnel-port-offload: off [fixed]
  56 tls-hw-tx-offload: off [fixed]
  57 tls-hw-rx-offload: off [fixed]
  58 rx-gro-hw: off [fixed]
  59 tls-hw-record: off [fixed]

TX checksumming fails

If TCP TX checksumming fails this might be cause by the corresponding offloading feature. This is quite hard to figure out.

In tcpdump you'll see (invalid) on the end of the line as a hint. When you now increase verbosity by using
ethtool -nev -i any YOUR_FILTER
you'll see something like:

   1 18:47:58.895123 vnet4 P   ifindex 14 52:54:00:ac:35:62 ethertype IPv4 (0x0800), length 80: (tos 0x0, ttl 64, id 10990, offset 0, flags [DF], proto TCP (6), length 60)                                                                                                                                          
   2     178.63.149.229.36240 > 94.130.128.9.22: Flags [S], cksum 0x26df (incorrect -> 0xa99d), seq 597567343, win 64240, options [mss 1460,sackOK,TS val 2569734882 ecr 0,nop,wscale 7], length 0

The packet will be dropped.

There is also a small asymmetry depending on the inititating party, that might be noted.

When initiating from side A
you will see your [S] (SYN) package going out (with the corrupted checksum) and there will be no package with the corresponding [S.] (SYN,ACK).
When initiating from side B
you will see your [S] (SYN) package going out and you may receive a package with the corresponding [S.] (SYN,ACK) (and the corrupted checksum) and side B will not send a package with the corresponding [.] (ACK).

In wireshark you'll see nothing unless you have enabled the checksum verifcation. :-D

Right click -> Protocol settings -> Transmission Control Protocol -> Validate the TCP checksums if possible

Activate it, now! Saves lifetime.

You can disable TX checksum offloading with ethtool, e.g. for testing

   1 ethtool -k enp1s0 |grep tx
   2 ethtool -K enp1s0 tx off
   3 ethtool -k enp1s0 |grep tx

To disable TX checksum offloading across reboots there is an integration in ifupdown but not in ifupdown2.
/etc/network/interfaces

   1 # This file describes the network interfaces available on your system
   2 # and how to activate them. For more information, see interfaces(5).
   3 
   4 source /etc/network/interfaces.d/*
   5 
   6 # The loopback network interface
   7 auto lo
   8 iface lo inet loopback
   9 
  10 # The primary network interface
  11 auto enp1s0
  12 iface enp1s0 inet static
  13         address 178.63.149.228/28
  14         gateway 178.63.149.225
  15         # dns-* options are implemented by the resolvconf package, if installed
  16         dns-nameservers 178.63.149.225
  17         dns-search rockstable.it
  18         offload-tx off

Took me 4 endless perplexed hours, staring on pcaps.

hping3

http://www.hping.org/

hping3 is a network tool able to send custom TCP/IP packets and to display target replies like ping program does with ICMP replies. hping3 handle fragmentation, arbitrary packets body and size and can be used in order to transfer files encapsulated under supported protocols. Using hping3 you are able to perform at least the following stuff:

Test firewall rules
Advanced port scanning
Test net performance using different protocols, packet size, TOS (type of service) and fragmentation.
Path MTU discovery
Transferring files between even really fascist firewall rules.
Traceroute-like under different protocols.
Firewalk-like usage.
Remote OS fingerprinting.
TCP/IP stack auditing.
A lot of others.

It's also a good didactic tool to learn TCP/IP. hping3 is developed and maintained by <antirez@invece.org> and is licensed under GPL version 2. Development is open so you can send me patches, suggestion and affronts without inhibitions.

ip

Show interface statistics

   1 ip -s link show

iperf

perform network throughput tests

iperf is a tool for performing network throughput measurements. It can test either TCP or UDP throughput. To perform an iperf test the user must establish both a server (to discard traffic) and a client (to generate traffic).

There are 2 tools that call themselves iperf: iperf2 and iperf3 and both don't seem stall in development.

I tend to prefer iperf3.

iperf2

Let's start with the "original"

Hint: Sometimes iperf hangs and CTRL+\ helps.

Install

   1 aptitude install iperf

iperf is based on a client-server principle.

Some defaults:

time: 10s
port: 5001
protocol: tcp

Start server

   1 # iperf -s
   2 ------------------------------------------------------------
   3 Server listening on TCP port 5001
   4 TCP window size:  128 KByte (default)
   5 ------------------------------------------------------------
   6 [  4] local 192.168.0.12 port 5001 connected with 192.168.0.11 port 37720
   7 [ ID] Interval       Transfer     Bandwidth
   8 [  4]  0.0-10.0 sec  1.10 GBytes   941 Mbits/sec

Client connects to server an outputs some enhanced reporting

   1 % iperf -c remote-host -e
   2 ------------------------------------------------------------
   3 Client connecting to mail1, TCP port 5001 with pid 11333
   4 Write buffer size:  128 KByte
   5 TCP window size: 85.0 KByte (default)
   6 ------------------------------------------------------------
   7 [  3] local 192.168.0.11 port 37720 connected with 192.168.0.12 port 5001
   8 [ ID] Interval        Transfer    Bandwidth       Write/Err  Rtry    Cwnd/RTT
   9 [  3] 0.00-10.00 sec  1.10 GBytes   942 Mbits/sec  8989/0         46      395K/2481 us

Okay we got:

Bandwidth
Errors
Round Trip Time

iperf3

Install

   1 aptitude install iperf

Some defaults:

time: 10s
port: 5201
protocol: tcp

Some helpful switches:

-e, --enhancedreports use enhanced reporting giving more tcp/udp and traffic information
-m, --print_mss
-V, --ipv6_domain Set the domain to IPv6 (send packets over IPv6)

Or short: -emV

Start server and give more detailed output

   1 # iperf3 -s -V
   2 iperf3 -s -V
   3 iperf 3.6
   4 Linux remote-host 4.19.0-12-amd64 #1 SMP Debian 4.19.152-1 (2020-10-18) x86_64
   5 -----------------------------------------------------------
   6 Server listening on 5201
   7 -----------------------------------------------------------
   8 Time: Tue, 01 Dec 2020 11:35:55 GMT
   9 Accepted connection from 192.168.0.11, port 45606
  10       Cookie: u2x5xtyjazvllw7o67jtojde22xsdxvybe3y
  11       TCP MSS: 0 (default)
  12 [  5] local 192.168.0.12 port 5201 connected to 192.168.0.11 port 45608
  13 Starting Test: protocol: TCP, 1 streams, 131072 byte blocks, omitting 0 seconds, 10 second test, tos 0
  14 [ ID] Interval           Transfer     Bitrate
  15 [  5]   0.00-1.00   sec   107 MBytes   901 Mbits/sec
  16 [  5]   1.00-2.00   sec   112 MBytes   941 Mbits/sec
  17 [  5]   2.00-3.00   sec   112 MBytes   942 Mbits/sec
  18 [  5]   3.00-4.00   sec   112 MBytes   941 Mbits/sec
  19 [  5]   4.00-5.00   sec   112 MBytes   938 Mbits/sec
  20 [  5]   5.00-6.00   sec   112 MBytes   942 Mbits/sec
  21 [  5]   6.00-7.00   sec   112 MBytes   941 Mbits/sec
  22 [  5]   7.00-8.00   sec   112 MBytes   941 Mbits/sec
  23 [  5]   8.00-9.00   sec   112 MBytes   942 Mbits/sec
  24 [  5]   9.00-10.00  sec   112 MBytes   941 Mbits/sec
  25 [  5]  10.00-10.04  sec  4.52 MBytes   941 Mbits/sec
  26 - - - - - - - - - - - - - - - - - - - - - - - - -
  27 Test Complete. Summary Results:
  28 [ ID] Interval           Transfer     Bitrate
  29 [  5] (sender statistics not available)
  30 [  5]   0.00-10.04  sec  1.10 GBytes   937 Mbits/sec                  receiver
  31 CPU Utilization: local/receiver 5.1% (1.1%u/4.0%s), remote/sender 0.0% (0.0%u/0.0%s)
  32 rcv_tcp_congestion cubic
  33 iperf 3.6
  34 Linux remote-host 4.19.0-12-amd64 #1 SMP Debian 4.19.152-1 (2020-10-18) x86_64
  35 -----------------------------------------------------------
  36 Server listening on 5201
  37 -----------------------------------------------------------
  38 ^Ciperf3: interrupt - the server has terminated

Client connects to server an outputs some enhanced reporting

   1 % iperf3 -c remote-host -V
   2 iperf3 -c mail1 -V        
   3 iperf 3.6
   4 Linux hostname 4.19.0-12-amd64 #1 SMP Debian 4.19.152-1 (2020-10-18) x86_64
   5 Control connection MSS 1448
   6 Time: Tue, 01 Dec 2020 11:35:55 GMT
   7 Connecting to host remote-host, port 5201
   8       Cookie: u2x5xtyjazvllw7o67jtojde22xsdxvybe3y
   9       TCP MSS: 1448 (default)
  10 [  5] local 192.168.0.11 port 45608 connected to 192.168.0.12 port 5201
  11 Starting Test: protocol: TCP, 1 streams, 131072 byte blocks, omitting 0 seconds, 10 second test, tos 0
  12 [ ID] Interval           Transfer     Bitrate         Retr  Cwnd
  13 [  5]   0.00-1.00   sec   114 MBytes   956 Mbits/sec    0    437 KBytes       
  14 [  5]   1.00-2.00   sec   113 MBytes   944 Mbits/sec    0    460 KBytes       
  15 [  5]   2.00-3.00   sec   112 MBytes   940 Mbits/sec    0    460 KBytes       
  16 [  5]   3.00-4.00   sec   112 MBytes   940 Mbits/sec    0    460 KBytes       
  17 [  5]   4.00-5.00   sec   111 MBytes   934 Mbits/sec   11    386 KBytes       
  18 [  5]   5.00-6.00   sec   113 MBytes   947 Mbits/sec    0    402 KBytes       
  19 [  5]   6.00-7.00   sec   112 MBytes   938 Mbits/sec    0    443 KBytes       
  20 [  5]   7.00-8.00   sec   112 MBytes   939 Mbits/sec    0    445 KBytes       
  21 [  5]   8.00-9.00   sec   112 MBytes   940 Mbits/sec    0    445 KBytes       
  22 [  5]   9.00-10.00  sec   113 MBytes   949 Mbits/sec    0    447 KBytes       
  23 - - - - - - - - - - - - - - - - - - - - - - - - -
  24 Test Complete. Summary Results:
  25 [ ID] Interval           Transfer     Bitrate         Retr
  26 [  5]   0.00-10.00  sec  1.10 GBytes   943 Mbits/sec   11             sender
  27 [  5]   0.00-10.04  sec  1.10 GBytes   937 Mbits/sec                  receiver
  28 CPU Utilization: local/sender 3.0% (0.3%u/2.7%s), remote/receiver 5.1% (1.1%u/4.0%s)
  29 snd_tcp_congestion cubic
  30 rcv_tcp_congestion cubic
  31 
  32 iperf Done.

Seems to be more informative.

My Traceroute (mtr)

https://www.bitwizard.nl/mtr/

mtr combines the functionality of the 'traceroute' and 'ping' programs in a single network diagnostic tool.

As mtr starts, it investigates the network connection between the host mtr runs on and a user-specified destination host.

Install

   1 aptitude install mtr

A graphical live updating traceroute with some statistics.

   1 mtr hostname.domain.tld

A text live updating traceroute with some statistics. mtr -t hostname.domain.tld

   1                                      My traceroute  [v0.94]
   2 abcd.efghi.rockstable.org (192.168.182.16) -> www.rockstable.it         2020-11-12T15:44:46+0100
   3 Keys:  Help   Display mode   Restart statistics   Order of fields   quit
   4                                                         Packets               Pings
   5  Host                                                 Loss%   Snt   Last   Avg  Best  Wrst StDev
   6  1. abcd.efghi.rockstable.org                          0.0%    41    0.3   0.2   0.2   0.5   0.0
   7  2. ipABCDEFHI.dynamic.kabel-deutschland.de            0.0%    41    7.7  10.8   5.7  39.1   6.9
   8  3. 83-169-181-254-isp.superkabel.de                   0.0%    41    6.4   8.3   6.0  18.5   1.9
   9  4. ip5886c0f1.static.kabel-deutschland.de             0.0%    41    7.9   8.9   6.1  15.3   2.0
  10  5. 145.254.3.68                                       0.0%    41    8.1   8.8   6.1  14.2   1.9
  11  6. 145.254.2.179                                     39.0%    41   15.5  16.3  14.3  20.8   1.7
  12  7. 145.254.2.179                                     35.9%    40   16.1  16.5  13.7  21.6   2.2
  13  8. decix2-gw.hetzner.com                              0.0%    40   18.9  15.5  12.5  33.7   3.8
  14  9. core24.fsn1.hetzner.com                           94.9%    40   21.7  21.6  21.5  21.7   0.1
  15 10. ex9k1.dc14.fsn1.hetzner.com                        0.0%    40   25.3  19.2  17.2  25.3   1.9
  16 11. kvm2.rockstable.org                                0.0%    40   21.5  21.4  18.9  28.9   2.4
  17 12. www2.rockstable.it                                 0.0%    40   22.3  20.8  18.4  26.5   1.9

netstat

Install

   1 aptitude install net-tools

The package offers a nice small suite of (legacy) tools

   1 dpkg -L net-tools| grep bin/   
   2 /bin/netstat
   3 /sbin/ifconfig
   4 /sbin/ipmaddr
   5 /sbin/iptunnel
   6 /sbin/mii-tool
   7 /sbin/nameif
   8 /sbin/plipconfig
   9 /sbin/rarp
  10 /sbin/route
  11 /sbin/slattach
  12 /usr/sbin/arp

On Linux these toolsuite has been deprecated and there are a number of successors

netstat is nowadays replaced by ss. The name ss maybe derived from "socket status" like in FreeBSDs sockstat.
ifconfig -> ip
route -> ip
…

Important switches

-n don't resolve names (maybe slow)

Some important commands

   1 ### LISTENING PORTS
   2 % sudo netstat -tulpen
   3 [sudo] Passwort für tobias: 
   4 Aktive Internetverbindungen (Nur Server)
   5 Proto Recv-Q Send-Q Local Address           Foreign Address         State       Benutzer   Inode      PID/Program name    
   6 tcp        0      0 0.0.0.0:445             0.0.0.0:*               LISTEN      0          37224      3237/smbd           
   7 tcp        0      0 0.0.0.0:2049            0.0.0.0:*               LISTEN      0          30994      -                   
   8 tcp        0      0 0.0.0.0:3142            0.0.0.0:*               LISTEN      125        57443      1851/apt-cacher-ng  
   9 tcp        0      0 127.0.0.1:3306          0.0.0.0:*               LISTEN      135        54106      1915/mariadbd       
  10 tcp        0      0 0.0.0.0:139             0.0.0.0:*               LISTEN      0          37225      3237/smbd           
  11 tcp        0      0 0.0.0.0:9102            0.0.0.0:*               LISTEN      0          61478      2297/bareos-fd      
  12 tcp        0      0 192.168.122.1:53        0.0.0.0:*               LISTEN      0          31348      4064/dnsmasq        
  13 tcp        0      0 192.168.101.1:53        0.0.0.0:*               LISTEN      0          72039      3926/dnsmasq        
  14 tcp        0      0 192.168.100.1:53        0.0.0.0:*               LISTEN      0          50833      3797/dnsmasq        
  15 tcp        0      0 0.0.0.0:22              0.0.0.0:*               LISTEN      0          39934      1863/sshd: /usr/sbi 
  16 tcp        0      0 127.0.0.1:631           0.0.0.0:*               LISTEN      0          53186      1852/cupsd          
  17 tcp        0      0 127.0.0.1:5432          0.0.0.0:*               LISTEN      136        40049      1928/postgres       
  18 tcp        0      0 127.0.0.1:25            0.0.0.0:*               LISTEN      0          72985      3688/master         
  19 …
  20 
  21 ### KERNEL INTERFACE TABLE
  22 netstat -ian
  23 Kernel-Schnittstellentabelle
  24 Iface      MTU    RX-OK RX-ERR RX-DRP RX-OVR    TX-OK TX-ERR TX-DRP TX-OVR Flg
  25 bond0     1500  1915555      0   1122 0       1029535      0      0      0 BMmRU
  26 bridge    1500  1912759      0  50477 0       1028122      0      0      0 BMRU
  27 enp8s0    1500  1915555      0      0 0       1029535      0      0      0 BMsRU
  28 enp9s0    1500        0      0      0 0             0      0      0      0 BMU
  29 lo       65536    58342      0      0 0         58342      0      0      0 LRU
  30 …
  31 
  32 ### SHOW TCP/UDP SOCKETS (INCLUDING WAITING)
  33 netstat -tuna

nmap

Great tool for network discovery! Use it!

Zenmap example commands

Simple scan for list of ports nmap -p 80,443
Intense scan nmap -T4 -A -v
Intense scan plus UDP nmap -sS -sU -T4 -A -v
Intense scan, all TCP ports nmap -p 1-65535 -T4 -A -v
Intense scan, no ping nmap -T4 -A -v -Pn
Ping scan nmap -sn
Quick scan nmap -T4 -F
Quick scan plus nmap -sV -T4 -O -F --version-light
Quick traceroute nmap -sn --traceroute
Regular scan nmap
Slow comprehensive scan nmap -sS -sU -T4 -A -v -PE -PP -PS80,443 -PA3389 -PU40125 -PY -g 53 --script "default or (discovery and safe)"
Intense scan with alternative DNS on opnsense in shell "csh" with normal log-file
```
   1 nmap -T4 -A -v \
   2         --dns-servers 192.168.100.102 \
   3         -oN "nmap_intense_192.168.100.0_24_`date '+%F_%H%M'`.txt" \
   4         192.168.100.0/24
```

ping

missing capabilities

Quelle Debian-Forum

It is totally unneccessary to run ping with sudo.

Check ping capabilities

   1 getcap /bin/ping

ping capabilities should be

   1 /bin/ping = cap_net_raw+ep

add capability net_raw

   1 setcap cap_net_raw+ep /bin/ping

socat

# WIP

stunnel

Establish secure connections

http://www.stunnel.org/
man stunnel

The stunnel program is designed to work as TLS encryption wrapper between remote clients and local (inetd-startable) or remote servers. The concept is that having non-TLS aware daemons running on your system you can easily set them up to communicate with clients over secure TLS channels.

stunnel can be used to add TLS functionality to commonly used Inetd daemons like POP-2, POP-3, and IMAP servers, to standalone daemons like NNTP, SMTP and HTTP, and in tunneling PPP over network sockets without changes to the source code.

This product includes cryptographic software written by Eric Young (eay@cryptsoft.com)

As an example the following little snipped creates a secure tunnel which is bound to the loopback interface tcp/144 and spans over to the remote imap-server tcp/993. This has been used during the migration of imap mailboxes.

/etc/stunnel/imap_sync.conf

   1 [imaps]
   2 client = yes
   3 accept = localhost:144
   4 connect = remote-imap.server.tld:993
   5 ;verify = 2
   6 ;CApath = @sysconfdir/ssl/certs
   7 ;checkHost = imap.gmail.com
   8 ;OCSPaia = yes
   9 
  10 ; vim:ft=dosini

tcpdump

Incredibly useful tool. May be combined with wireshark when used to write a pcap-dump -w. This may also be used to capture remote via ssh and view it locally wireshark.
Please see: Wireshark Live captures

Some filter keywords to me remembered:

host
net
src
dst
port, sport, dport

tcpdump with layer 2 headers

When dealing with VLANs, tracing ARP or CARP and similar, it is incredibly useful to add option -e to the get some additional layer 2 information.

dump_multi.sh

tcpdump misses an option to display the interface the packet was received on. Here is a little wrapper script that simply starts multiple tcpdumps and prefixes the output with the interface name.

The script was mainly copied from this thread on
serverfault - how to display interface in tcpdump output flow

/usr/local/sbin/dump_multi.sh

   1 #!/bin/bash
   2 
   3 SELF="$(basename "$0")"
   4 
   5 declare -a INTERFACES
   6 
   7 ###ADD A STOP MARK TO THE POSITIONAL PARAMETERS
   8 STOPMARK="$(uuidgen)"
   9 set -- "$@" "$STOPMARK"
  10 
  11 usage () {
  12         cat <<-EOF
  13                 $SELF [Options]
  14                     Options:
  15                         -h|--help               Show this page
  16                         -i|--interface <arg>    Dump interface <arg>
  17 
  18                 Options of $SELF mask options of tcpdump.
  19                 Options of tcpdump are not documented here.
  20         EOF
  21 }
  22 
  23 while true; do
  24         case "$1" in
  25                 '-h'|'--help')
  26                         usage
  27                         shift
  28                         continue
  29                 ;;
  30                 '-i'|'--interface')
  31                         INTERFACES+=( "$2" ) 
  32                         shift 2 
  33                         continue
  34                 ;;
  35                 ### BREAK OPTION PARSING AFTER ONE FULL ITERATION
  36                 "$STOPMARK")
  37                         shift
  38                         break
  39                 ;;
  40                 *)
  41                         ### APPEND UNKNOWN OPTION TO THE END OF THE LIST
  42                         TMP1="$1"
  43                         shift
  44                         set -- "$@" "$TMP1"
  45                         unset TMP1
  46                 ;;
  47         esac
  48 done
  49 
  50 
  51 ### When this exits, exit all background processes:
  52 trap 'kill $(jobs -p) &> /dev/null && sleep 0.2 &&  echo ' EXIT
  53 
  54 ### Create one tcpdump output per interface and
  55 ### add an identifier to the beginning of each line:
  56 if [ "${#INTERFACES[@]}" -eq 1 ] \ 
  57         && [ "${INTERFACES[0]}" = "any" ];
  58 then
  59     for IFACE in $(ip l \
  60             |grep '^[0-9]:' \
  61             |grep ',UP' \
  62             |awk '{print $2}' \
  63             |sed 's/://')
  64     do
  65        tcpdump -l -i "$IFACE" -nn "$@" \ 
  66                |sed 's/^/[Iface: '"$IFACE"']    /' & 
  67     done
  68 elif [ "${#INTERFACES[@]}" -ge 1 ]; then
  69         for IFACE in "${INTERFACES[@]}"; do
  70                 tcpdump -l -i "$IFACE" "$@" \ 
  71                         |sed 's/^/[Iface: '"$IFACE"'] /' & 
  72         done
  73 fi
  74 
  75 # wait for CTRL+C
  76 wait

Use it like

   1 ### ON EVERY INTERFACE THAT IS UP
   2 dump_multi.sh -i any 'port …'
   3 ### ON A LIST OF INTERFACES
   4 dump_multi.sh -i lo -i eth0 -i eth1 -i bond0 -i br0 'port …'

Example: Dump DHCP/BOOTP traffic on a bridge, the attached bond and its slaves and determine the traffic flow.

   1 ./dump_multi.sh \
   2         -i bond0 -i enp8s0 -i enp9s0 -i bridge \
   3         'port (bootps or bootpc)'
   4 tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
   5 listening on bridge, link-type EN10MB (Ethernet), capture size 262144 bytes
   6 tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
   7 listening on bond0, link-type EN10MB (Ethernet), capture size 262144 bytes
   8 tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
   9 listening on enp9s0, link-type EN10MB (Ethernet), capture size 262144 bytes
  10 tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
  11 listening on enp8s0, link-type EN10MB (Ethernet), capture size 262144 bytes
  12 [Iface: bridge] 08:26:31.531753 IP 0.0.0.0.bootpc > 255.255.255.255.bootps: BOOTP/DHCP, Request from 52:54:00:d1:b6:9b (oui Unknown), length 300
  13 [Iface: bridge] 08:26:31.555936 IP 0.0.0.0.bootpc > 255.255.255.255.bootps: BOOTP/DHCP, Request from 52:54:00:d1:b6:9b (oui Unknown), length 300
  14 [Iface: bridge] 08:26:34.392529 IP 0.0.0.0.bootpc > 255.255.255.255.bootps: BOOTP/DHCP, Request from c0:d2:f3:e1:fb:7b (oui Unknown), length 315
  15 [Iface: bridge] 08:26:34.973608 IP 0.0.0.0.bootpc > 255.255.255.255.bootps: BOOTP/DHCP, Request from 52:54:00:d1:b6:9b (oui Unknown), length 300

whois

QUERY LIMIT

When experimenting, don't be to curious or you will be banned for a day.

   1 # whois -i nserver '195.201.246.253'                 
   2 % This is the RIPE Database query service.
   3 % The objects are in RPSL format.
   4 %
   5 % The RIPE Database is subject to Terms and Conditions.
   6 % See http://www.ripe.net/db/support/db-terms-conditions.pdf
   7 
   8 %ERROR:201: access denied for IP.ADD.RE.SS
   9 %
  10 % Queries from your IP address have passed the daily limit of controlled objects.
  11 % Access from your host has been temporarily denied.
  12 % For more information, see
  13 % http://www.ripe.net/data-tools/db/faq/faq-db/why-did-you-receive-the-error-201-access-denied
  14 
  15 % This query was served by the RIPE Database Query Service version 1.97.2 (HEREFORD)

The rate limit is applied to a whole /24 network of your ISP, so it eventually sums up with your neighbors.
Your probably safe with less than 60 queries per hour (t_interval ≥ 60s)
The rate limit is ~ 60 queries per hour for a temporary ban (48s < t_interval < 60s).
If you send more than 75 queries per hour (t_interval < 48s) can get banned permanantly. Whatever that means.
The rate limit applies to one whois-service. So if you query a mixed list of TLDs (using multiple whois-services) you may query a lot more.

Get information about server

Query server information

   1 # QUERY SUPPORTED TYPES
   2 whois -q types
   3 % This is the RIPE Database query service.
   4 % The objects are in RPSL format.
   5 %
   6 % The RIPE Database is subject to Terms and Conditions.
   7 % See http://www.ripe.net/db/support/db-terms-conditions.pdf
   8 
   9 inetnum
  10 inet6num
  11 as-block
  12 aut-num
  13 as-set
  14 route
  15 route6
  16 route-set
  17 inet-rtr
  18 filter-set
  19 peering-set
  20 rtr-set
  21 domain
  22 poetic-form
  23 poem
  24 mntner
  25 irt
  26 key-cert
  27 organisation
  28 role
  29 person
  30 
  31 % This query was served by the RIPE Database Query Service version 1.97.2 (BLAARKOP)
  32 
  33 # QUERY SERVER VERSION
  34 whois -q version
  35 % This is the RIPE Database query service.
  36 % The objects are in RPSL format.
  37 %
  38 % The RIPE Database is subject to Terms and Conditions.
  39 % See http://www.ripe.net/db/support/db-terms-conditions.pdf
  40 
  41 % whois-server-1.97.2
  42 % This query was served by the RIPE Database Query Service version 1.97.2 (BLAARKOP)
  43 
  44 # QUERY SERVER SOURCES
  45 whois -q sources

Querying

You can query the templates and the inverse keys with -t TYPE

   1 whois -t domain 
   2 % This is the RIPE Database query service.
   3 % The objects are in RPSL format.
   4 %
   5 % The RIPE Database is subject to Terms and Conditions.
   6 % See http://www.ripe.net/db/support/db-terms-conditions.pdf
   7 
   8 domain:         [mandatory]  [single]     [primary/lookup key]
   9 descr:          [optional]   [multiple]   [ ]
  10 org:            [optional]   [multiple]   [inverse key]
  11 admin-c:        [mandatory]  [multiple]   [inverse key]
  12 tech-c:         [mandatory]  [multiple]   [inverse key]
  13 zone-c:         [mandatory]  [multiple]   [inverse key]
  14 nserver:        [mandatory]  [multiple]   [inverse key]
  15 ds-rdata:       [optional]   [multiple]   [inverse key]
  16 remarks:        [optional]   [multiple]   [ ]
  17 notify:         [optional]   [multiple]   [inverse key]
  18 mnt-by:         [mandatory]  [multiple]   [inverse key]
  19 created:        [generated]  [single]     [ ]
  20 last-modified:  [generated]  [single]     [ ]
  21 source:         [mandatory]  [single]     [ ]
  22 
  23 % This query was served by the RIPE Database Query Service version 1.97.2 (BLAARKOP)
  24 
  25 ### OUTPUT THE TEMPLATE MORE VERBOSE
  26 whois -v domain
  27 % This is the RIPE Database query service.
  28 % The objects are in RPSL format.
  29 %
  30 % The RIPE Database is subject to Terms and Conditions.
  31 % See http://www.ripe.net/db/support/db-terms-conditions.pdf
  32 
  33 The domain class:
  34 
  35       A domain object represents a Top Level Domain (TLD) or
  36       other domain registrations. It is also used for Reverse
  37       Delegations.
  38 
  39 domain:         [mandatory]  [single]     [primary/lookup key]
  40 descr:          [optional]   [multiple]   [ ]
  41 org:            [optional]   [multiple]   [inverse key]
  42 admin-c:        [mandatory]  [multiple]   [inverse key]
  43 tech-c:         [mandatory]  [multiple]   [inverse key]
  44 zone-c:         [mandatory]  [multiple]   [inverse key]
  45 nserver:        [mandatory]  [multiple]   [inverse key]
  46 ds-rdata:       [optional]   [multiple]   [inverse key]
  47 remarks:        [optional]   [multiple]   [ ]
  48 notify:         [optional]   [multiple]   [inverse key]
  49 mnt-by:         [mandatory]  [multiple]   [inverse key]
  50 created:        [generated]  [single]     [ ]
  51 last-modified:  [generated]  [single]     [ ]
  52 source:         [mandatory]  [single]     [ ]
  53 
  54 The content of the attributes of the domain class are defined below:
  55 
  56 domain
  57 
  58    Domain name.
  59 
  60      Domain name as specified in RFC 1034 (point 5.2.1.2) with or
  61      without trailing dot (".").  The total length should not exceed
  62      254 characters (octets).
  63 
  64 descr
  65 
  66    A short description related to the object.
  67 
  68      A sequence of ASCII characters. 
  69 
  70 …

Techniques

Bonding

Linux Docs html Bonding
Linux Docs txt Bonding
Some general information is documented in the man-pages
man 5 interfaces

Please buy switches that support LACP and MC-LAG!

Link aggregation - IEEE 802.1AX (previously 802.3ad)

Link aggregation

Not to be confused with IEEE 801.3ad (QinQ).

Initial release of 802.3ad in 2000, fast adoption by vendors.
Formal transition of 802.3ad to IEEE 802.1AX-2008 on 3 November 2008.
Increases bandwidth of a interface beyond a single interface
Increases resilience (link-level redundancy)
- In a port-cable-port connection each part can fail and has redundancy.
Lowers costs to the price of the cable

A link aggregation group (LAG) is the collection of physical ports combined together.

Link Aggregation Control Protocol (LACP)

Within the IEEE specification, the Link Aggregation Control Protocol (LACP) provides a method to control the bundling of several physical ports together to form a single logical channel. LACP allows a network device to negotiate an automatic bundling of links by sending LACP packets to the peer (directly connected device that also implements LACP).

Linux bondig driver mode 4 802.3ad

Creates aggregation groups that share the same speed and duplex settings. Utilizes all slave network interfaces in the active aggregation group according to the 802.3ad specification. This mode is similar to the XOR mode above and supports the same balancing policies. The link is set up dynamically between two LACP-supporting peers.

LACP features and practical examples

Maximum number of bundled ports allowed in the port channel: Valid values are usually from 1 to 8.
LACP packets are sent with multicast group MAC address
01:80:c2:00:00:02 (01-80-c2-00-00-02)
During LACP detection period
- LACP packets are transmitted every second
- Keep-alive mechanism for link member: (default: slow = 30s, fast=1s)
LACP can have the port-channel load-balance mode
- link (link-id) Integer that identifies the member link for load balancing. The range is from 1 to 8 and the load balancing mode can be set-up based on traffic models.[9]
LACP mode
- Active: Enables LACP unconditionally.
- Passive: Enables LACP only when an LACP device is detected. (This is the default state)

Advantages over static configuration

Failover occurs automatically
- When a link fails and here is (for example) a media converter between the devices, a peer system will not perceive any connectivity problems. With static link aggregation, the peer would continue sending traffic down the link causing the connection to fail.
Dynamic configuration
- The device can confirm that the configuration at the other end can handle link aggregation. With Static link aggregation, a cabling or configuration mistake could go undetected and cause undesirable network behavior.

Practical notes

LACP works by sending frames (LACPDUs) down all links that have the protocol enabled. If it finds a device on the other end of the link that also has LACP enabled, it will also independently send frames along the same links enabling the two units to detect multiple links between themselves and then combine them into a single logical link. LACP can be configured in one of two modes: active or passive. In active mode it will always send LACPDUs along the configured links. In passive mode, however, it only reacts as "speak when spoken to", and therefore can be used as a way of controlling accidental loops (as long as the other device is in active mode).

Proprietary link aggregation

In addition to the IEEE link aggregation substandards, there are a number of proprietary aggregation schemes including Cisco's EtherChannel and Port Aggregation Protocol, Juniper's Aggregated Ethernet, AVAYA's Multi-Link Trunking, Split Multi-Link Trunking, Routed Split Multi-Link Trunking and Distributed Split Multi-Link Trunking, ZTE's "Smartgroup", Huawei's "Eth-Trunk", or Connectify's Speedify. Most high-end network devices support some kind of link aggregation, and software-based implementations – such as the *BSD lagg package, Linux bonding driver, Solaris dladm aggr, etc. – also exist for many operating systems.

Limitations

Single switch

With the modes balance-rr, balance-xor, broadcast and 802.3ad, all physical ports in the link aggregation group must reside on the same logical switch, which, in most common scenarios, will leave a single point of failure when the physical switch to which all links are connected goes offline.
The modes active-backup, balance-tlb, and balance-alb can also be set up with two or more switches. But after failover (like all other modes), in some cases, active sessions may fail (due to ARP problems) and have to be restarted.

Same link speed and duplex

In most LAGs, all the ports used in an aggregation consist of the same physical type, such as all copper ports (10/100/1000BASE‑T), all multi-mode fiber ports, or all single-mode fiber ports. However, all the IEEE standard requires is that each link be full duplex and all of them have an identical speed (10, 100, 1,000 or 10,000 Mbit/s). So fiber and copper port may be use long aside in the same LAG as long as they match in speed and duplex.
IMHO this may be very implementation specific, while some vendors allow it, others suspend slower interfaces.
If not obeyed
- Load-shareing may be suboptimal. Faster links are not to utilized to the full capacity. This may be influenced by selecting another load-sharing algorithm, which does not use persitence.
- TCP out-of-order delivery may occur. Especially in situations with higher load.
- Links may not be used at all if a faster link is member in the LACP LAG.

Ethernet aggregation mismatch

Aggregation mismatch refers to not matching the aggregation type on both ends of the link. Some switches do not implement the 802.1AX standard but support static configuration of link aggregation. Therefore, link aggregation between similarly statically configured switches will work but will fail between a statically configured switch and a device that is configured for LACP.

Multi-chassis link aggregation group (MC-LAG)

Multi-chassis link aggregation group

A multi-chassis link aggregation group (MLAG or MC-LAG) is a type of link aggregation group (LAG) with constituent ports that terminate on separate chassis, primarily for the purpose of providing redundancy in the event one of the chassis fails. The IEEE 802.1AX-2008 industry standard for link aggregation does not mention MC-LAG, but does not preclude it. Its implementation varies by vendor; notably, the protocol existing between the chassis is proprietary.

Advantages over LAG

node- and link-level redundancy
superior to spanning tree
- links of a MC-LAG don't need to be disabled to prevent loops

Bonding with ifupdown and ifenslave

It's recommended to configure bonding
via iproute2 (netlink) or sysfs,
the old ifenslave control utility is obsolete.

Make sure to not interfere with Network-Manager.

During the initial creation it is recommended to monitor the bond in a tmux session with

tail -f /var/log/messages to watch for bonding driver errors.
watch -n1 -- cat /proc/net/bonding/bond0 to check current bonding parameters.
watch -n1 -- ip a to check current networking amd interface state.

While the setup fails prepare a individual statement to reset the state to a clean starting point.

   1 for IFACE in bond0 enp8s0 enp9s0; do
   2         ifdown "$IFACE"
   3         ip l set down dev "$IFACE";
   4 done
   5 ip a del 192.168.182.16/24 dev enp8s0
   6 ip a del 192.168.182.208/24 dev enp9s0

Bonding with ifupdown requires the package ifenslave.

   1 apt install ifenslave

There is a bit of documentation in
/usr/share/doc/ifenslave/README.Debian.gz

The examples are found at
/usr/share/doc/ifenslave/examples

Whenever possible use LACP IEEE 802.3ad.

If using lacp you should set the rate to fast (every 1 second). The default is slow (every 30 seconds).

   1 # This file describes the network interfaces available on your system
   2 # and how to activate them. For more information, see interfaces(5).
   3 
   4 source /etc/network/interfaces.d/*
   5 
   6 # The loopback network interface
   7 auto lo
   8 iface lo inet loopback
   9 
  10 auto bond0
  11 iface bond0 inet dhcp
  12         bond-slaves     enp8s0 enp9s0
  13         bond-miimon     100
  14         mode            802.3ad
  15         lacp_rate       fast
  16 
  17 auto enp8s0
  18 iface enp8s0 inet manual
  19 
  20 auto enp9s0
  21 iface enp9s0 inet manual

This config is for LACP agnostic switches with apdaptive load-balancing.

/etc/network/interfaces

   1 # This file describes the network interfaces available on your system
   2 # and how to activate them. For more information, see interfaces(5).
   3 
   4 source /etc/network/interfaces.d/*
   5 
   6 # The loopback network interface
   7 auto lo
   8 iface lo inet loopback
   9 
  10 auto bond0
  11 iface bond0 inet dhcp
  12         bond-primary    enp8s0 enp9s0
  13         bond-slaves     enp8s0 enp9s0
  14         bond-miimon     100
  15         bond-mode       balance-alb

Bonding with Network-Manager

Well, i had huge trouble configuring a lacp bond with NetworkManager. Actually it didn't even work. Whereas ifupdown was trivial and ready in seconds on first try … I'm not yet convinced of NetworkManager.

It's easy to convert ifupdown config to Network-Manager in nm-connecttion-editor.

For every interface

open the interface to be converted
mark the interface to be started automatically
enable auto-negotiation
save and exit the interface

The auto-generated file from /run/NetworkManager/system-connections will be saved in /etc/NetworkManager/system-connections.

Unfortunately the option autoconnect-slaves=1 does not activate the slaves of the bonds. So it's important that the slave interfaces are marked to be started automatically (autoconnect=false is missing or autoconnect=true) or the bond won't come up. On the down-side the bond will also get up on boot even if the bond interface is marked as autoconnect=false.

/etc/NetworkManager/system-connections/bond0.nmconnection

   1 [connection]
   2 id=bond0
   3 uuid=9d6f2839-75ce-4b40-9f1a-2ca925af6dfc
   4 type=bond
   5 interface-name=bond0
   6 permissions=
   7 timestamp=1604305217
   8 
   9 [bond]
  10 downdelay=0
  11 miimon=100
  12 mode=balance-alb
  13 updelay=0
  14 
  15 [ipv4]
  16 dns-priority=100
  17 dns-search=
  18 method=auto
  19 
  20 [ipv6]
  21 addr-gen-mode=stable-privacy
  22 address1=fd93:56fb:daf7:0:2d8:61ff:fe2e:7979/64
  23 dns-priority=100
  24 dns-search=
  25 ip6-privacy=0

The default of Ethernet auto-negotiation is false.

/etc/NetworkManager/system-connections/enp8s0.nmconnection

   1 [connection]
   2 id=enp8s0
   3 uuid=1fc1fa53-f44e-48eb-8904-8cfd6a80950f
   4 type=ethernet
   5 interface-name=enp8s0
   6 master=bond0
   7 permissions=
   8 slave-type=bond
   9 timestamp=1604305049
  10 
  11 [ethernet]
  12 auto-negotiate=true
  13 mac-address=00:D8:61:2E:79:79
  14 mac-address-blacklist=

/etc/NetworkManager/system-connections/enp9s0.nmconnection

   1 [connection]
   2 id=enp9s0
   3 uuid=bb8eac2e-3b3f-4b96-8b49-72dc331ca521
   4 type=ethernet
   5 interface-name=enp9s0
   6 master=bond0
   7 permissions=
   8 slave-type=bond
   9 timestamp=1604305049
  10 
  11 [ethernet]
  12 auto-negotiate=true
  13 mac-address=00:D8:61:2E:79:7A
  14 mac-address-blacklist=

Bonding on VMware

VMware Docs 7.0 Index
VMware Docs 7.0 LACP-Support auf einem vSphere Distributed Switch
LACP is only supported in vSphere 5.1, 5.5 and 6.0 using vSphere Distributed Switches (VDS) or the Cisco Nexus 1000v.
A distributed virtual switch requires VMware vSphere Enterprise Plus licensing.

Bridging

Bridging with Network-Manager

The following script is mainly a condensed form of Christophers Blog Articel on How to create bridges on bonds with and without vlans using networkmanager

Create a bridge with the following script

   1 #!/bin/bash
   2 
   3 ### DEFINE BRIDGE
   4 BRIDGE=bridge
   5 BRIDGE_STP=no
   6 #BRIDGE_MTU=1500
   7 
   8 nmcli con add ifname "$BRIDGE" type bridge con-name "$BRIDGE"
   9 nmcli con modify "$BRIDGE" bridge.stp "$BRIDGE_STP"
  10 #nmcli con modify "$BRIDGE" 802-3-ethernet.mtu "$BRIDGE_MTU"
  11 
  12 ### DEFINE BONDsetup8
  13 BOND=bond0
  14 #BOND_SLAVE0=enp0s8
  15 #BOND_SLAVE1=enp0s8
  16 #BOND_MODE=active-backup
  17 #BOND_MTU=9000
  18 
  19 #nmcli con add type bond ifname "${BOND}" con-name "${BOND}"
  20 #nmcli con modify "${BOND}" bond.options mode="${BOND_MODE}"
  21 #nmcli con modify "${BOND}" 802-3-ethernet.mtu "${BOND_MTU}"
  22 #nmcli con add type ethernet con-name "${BOND}-slave-${BOND_SLAVE0}" ifname "${BOND_SLAVE0}" master "${BOND}"
  23 #nmcli con add type ethernet con-name "${BOND}-slave-${BOND_SLAVE1}" ifname "${BOND_SLAVE1}" master "${BOND}"
  24 #nmcli con modify "${BOND}-slave-${BOND_SLAVE0}" 802-3-ethernet.mtu "${BOND_MTU}"
  25 #nmcli con modify "${BOND}-slave-${BOND_SLAVE1}" 802-3-ethernet.mtu "${BOND_MTU}"
  26 
  27 ### ADD bond0 To bridge
  28 nmcli con modify "${BOND}" master "${BRIDGE}" slave-type bridge
  29 
  30 ### SHOW SOME INFO
  31 nmcli con
  32 ls /sys/class/net/bridge/brif/
  33 brctl show

Next is just a note and is yet unconfirmed. There may be some bonding modes, that don't work well with linux bridges, especially balance-alb.

VM on Bridge

I had some trouble getting traffic of a VM attached to a bridge over the enslaved bond. Here are my notes.

The following line allows traffic, but it's not the solution, only a hint to a the problem. Don't run your server with an uninitialized iptables stack in conjuction with other nftables hooks.

   1 echo 0 > /proc/sys/net/bridge/bridge-nf-call-iptables

According to kernel.org doc ip-sysctl

   1 bridge-nf-call-iptables - BOOLEAN
   2 
   3         1 : pass bridged IPv4 traffic to iptables’ chains.
   4         0 : disable this.
   5 
   6     Default: 1

This seems to mean that #nftables is circumvented, which is the default nowadays in Linux.

I wasn't aware that there is such an extensive nftables ruleset without having configured anything.
nft list ruleset

It turns out that the FORWARD policy of ipv4 traffic is set to drop.
nft list chain ip filter FORWARD

   1 table ip filter {
   2         chain FORWARD {
   3                 type filter hook forward priority filter; policy drop;
   4                 counter packets 1095 bytes 92945 jump DOCKER-USER
   5                 counter packets 1095 bytes 92945 jump DOCKER-ISOLATION-STAGE-1
   6                 oifname "docker0" ct state related,established counter packets 0 bytes 0 accept
   7                 oifname "docker0" counter packets 0 bytes 0 jump DOCKER
   8                 iifname "docker0" oifname != "docker0" counter packets 0 bytes 0 accept
   9                 iifname "docker0" oifname "docker0" counter packets 0 bytes 0 accept
  10                 counter packets 1095 bytes 92945 jump LIBVIRT_FWX
  11                 counter packets 1095 bytes 92945 jump LIBVIRT_FWI
  12                 counter packets 1095 bytes 92945 jump LIBVIRT_FWO
  13         }
  14 }

When setting this policy to accept the traffic of virtual machine passes. So this is the first runtime fix.

   1 nft chain ip filter FORWARD '{policy accept;}'

I still need a permanent solution. And i need to figure out, who set this policy to drop. I'll start by disabling docker at boot time and reboot.

   1 systemctl disable docker

And i was right - after reboot
nft list chain ip filter FORWARD

   1 table ip filter {
   2         chain FORWARD {
   3                 type filter hook forward priority filter; policy accept;
   4                 counter packets 0 bytes 0 jump LIBVIRT_FWX
   5                 counter packets 0 bytes 0 jump LIBVIRT_FWI
   6                 counter packets 0 bytes 0 jump LIBVIRT_FWO
   7         }
   8 }

The VM got a IP via DHCP.

Attempt 1

Revert bridge-nf-call-iptables to 1 and flushed nftables ruleset nft flush ruleset. And it also worked.
Restart nftables.service and it still worked.
Restart libvirtd.service and it still worked.
Restart docker.service and it still worked.
- Docker adds rules to nftables.
- Did not change the ipv4 forward policy to drop,
  because libvirt already enabled ip_forward.
Stuck - I only got one more confirmation, that the nftables stack is responsible.

Attempt 2

Rebooted and it fails.
Stopped docker.service - still fails
- Docker does not remove its rules from nftables.
- IPv4 forward policy is set to drop.
Disabled docker.service and rebooted - works.
Started docker.service - still works.
- Docker adds rules to nftables.
- Did not change the ipv4 forward policy to drop,
  because libvirt already enabled ip_forward.
Double check this by enabling and rebooting to exclude interference with the "runtime fix" from above.
- Confirmed Docker sets IPv4 forwarding policy to drop.

Related info

Debian bug report #865975
Debian bug report #903635
Docker.io README.Debian I haven't been able to work around this with the daemon config file either
Docker Docs - Container networking
Docker Docs - Docker and iptables
Docker Docs - Use bridge networks

How docker reasons this behaviour

The FORWARD chain policy is set to DROP by docker since 1.13. As of writing this I'm currently using docker.it 19.03.13+dfsg1-3 from Debian Bullseye.

Docker needs relies on package forwarding to make containers reachable.

If it's not enabled when bringing up the docker interfaces, docker enables forwarding and is by this reason also responsible for any security implications that arise from enabling forwarding and thus sets default policy to DROP. This may break other 3rd-party applications or VMs on the system, but it at least does not impose security threads.

I tend to agree with this idea. Other software should have configured their firewall rules as well.

Solution

In the end it's race condition between docker (which would set net.ipv4.ip_forward = 1 and set nf policy drop; unless net.ipv4.ip_forward was not enabled by docker) and libvirtd (which would set net.ipv4.ip_forward = 1).

A quick and permanent solution by this reason is enabling forwarding before bringing up docker (or anything else).
Linux#IPv4_Forwarding]

VLAN Subinterfaces

VLAN Subinterfaces with ifupdown

Can't be easier.

/etc/network/interfaces

   1 auto bond0.32
   2 iface bond0.32 inet dhcp
   3 
   4 auto bond0.40
   5 iface bond0.40 inet dhcp

NAT

NAT: Network Address Translation

The Two Types of NAT

This neat little explaination was found in the
netfilter.org Linux 2.4 NAT HOWTO

I divide NAT into two different types:
Source NAT (SNAT) and Destination NAT (DNAT).

Source NAT
: is when you alter the source address of the first packet:
i.e. you are changing where the connection is coming from.
: Source NAT is always done post-routing, just before the packet goes out onto the wire.
: Masquerading is a specialized form of SNAT.
Destination NAT
: is when you alter the destination address of the first packet:
i.e. you are changing where the connection is going to.
: Destination NAT is always done before routing, when the packet first comes off the wire.
: Port forwarding, load sharing, and transparent proxying are all forms of DNAT.

Wake on Lan

Enable "wake on lan (wol)" in UEFI/BIOS.

Install ethtool

   1 apt install ethtool

Check wol support
ethtool only reports Wake-on when executed with elevated privileges (probably based on the capabilities).

   1 DEVICES="$(ip l |grep -E '^[0-9]+:' |cut -d\  -f2 |sed 's/:$//')"
   2 for DEVICE in $(ip l |grep -E '^[0-9]+:'|cut -d\  -f2 |sed 's/:$//'); do
   3         echo "Device: '$DEVICE'";
   4         if WOL_SUPPORT="$(sudo ethtool "$DEVICE" 2>/dev/null| grep -i 'wake')"; then
   5 
   6                 echo "\tSupport: true\n$WOL_SUPPORT"
   7         else
   8                 echo -e "\tSupport: false"
   9         fi
  10 done

For comparison the table from the man page
man -P "less -p '^ *wol'" ethtool

   1            wol p|u|m|b|a|g|s|f|d...
   2                   Sets Wake-on-LAN options.  Not all devices support this.  The argument to this option is a string of characters specifying which options to enable.
   3 
   4                   p   Wake on PHY activity
   5                   u   Wake on unicast messages
   6                   m   Wake on multicast messages
   7                   b   Wake on broadcast messages
   8                   a   Wake on ARP
   9                   g   Wake on MagicPacket™
  10                   s   Enable SecureOn™ password for MagicPacket™
  11                   f   Wake on filter(s)
  12                   d   Disable  (wake  on nothing).  This option clears all previous
  13                       options.

Enable WOL non-persistently

   1 ethtool -s enp0s31f6 wol g

Enable WOL persistently

With network-manager

   1 nmcli connection modify enp8s0 802-3-ethernet.wake-on-lan magic
   2 nmcli connection modify enp9s0 802-3-ethernet.wake-on-lan magic

Wake target

With etherwake

Please see: man 1 etherwake
Default interface is eth0
etherwake: This program must be run as root.

   1 sudo etherwake -i bridge 00:1d:ec:10:57:ab

This generates the following magic packet

With wakeonlan

Please see: man 1 etherwake
The 'magic packet' consists of 6 times 0xFF followed by 16 times the hardware address of the NIC. This sequence can be encapsulated in any kind of packet. This script uses UDP packets.
Elevated privileges are not necessary.
By default sends the package
to the limited broadcast address 255.255.255.255
Destination port: Default: 9 (the discard port)

   1 $ wakeonlan 00:1d:ec:10:57:ab
   2 Sending magic packet to 255.255.255.255:9 with 00:1d:ec:10:57:ab

This generates the following magic packet

Configure interfaces and routing

This little snippet is useful to configure Debian networking with IPv4/6 and stattic routing.

/etc/network/interfaces

   1 # interfaces(5) file used by ifup(8) and ifdown(8)
   2 # Include files from /etc/network/interfaces.d:
   3 source /etc/network/interfaces.d/*
   4 
   5 ### OUTSIDE
   6 allow-hotplug enp7s0
   7 iface enp7s0 inet static
   8         address         94.130.128.9/26
   9         gateway         94.130.128.1
  10         #metric         256
  11         description     "OUTSIDE"
  12 
  13 iface enp7s0 inet6 static
  14         address         2a01:4f8:13b:3bf1::2/64
  15         gateway         fe80::1
  16         #metric         1024
  17         description     "OUTSIDE6"
  18 
  19 ### FAKE BRIDGES
  20 allow-hotplug ovs-trf1
  21 iface ovs-trf1 inet static
  22         address         172.16.255.1/24
  23         description     fake-bridge: transfer network host - fw
  24         post-up         ip route add 176.9.178.16/29 via 172.16.255.2 dev ovs-trf1
  25         post-down       ip route del 176.9.178.16/29 via 172.16.255.2 dev ovs-trf1
  26 
  27 iface ovs-trf1 inet6 static
  28         ### ULA-PREFIX  fd2f:763d:5b4d::1/48
  29         address         fd2f:763d:5b4d::1/64
  30         autoconf        1
  31         description     fake-bridge: transfer network host - fw
  32         ### OTHER MACHINE HAS MAC 52:54:00:65:89:5a BUT LOCAL FLAG IS INVERTED IN EUI-64 IID
  33         ### EXAMPLE WITH UNIQUE LOCAL ADDRESSES MANUALLY ASSIGNED
  34         #post-up        ip route add 2a01:4f8:13b:fb00::/56 via fd2f:763d:5b4d::2 dev ovs-trf1
  35         #post-down      ip route del 2a01:4f8:13b:fb00::/56 via fd2f:763d:5b4d::2 dev ovs-trf1
  36         ### EXAMPLE WITH UNIQUE LOCAL ADDRESSES AND SLAAC
  37         post-up         ip route add 2a01:4f8:13b:fb00::/56 via fd2f:763d:5b4d::5054:ff:fe65:895a dev ovs-trf1
  38         post-down       ip route del 2a01:4f8:13b:fb00::/56 via fd2f:763d:5b4d::5054:ff:fe65:895a dev ovs-trf1
  39         ### EXAMPLE WITH LINK-LOCAL ADDRESSES
  40         #post-up        ip route add 2a01:4f8:13b:fb00::/56 via fe80::5054:ff:fe65:895a dev ovs-trf1
  41         #post-down      ip route del 2a01:4f8:13b:fb00::/56 via fe80::5054:ff:fe65:895a dev ovs-trf1
  42 
  43 ### VLANS 500-999
  44 allow-hotplug ovs-pub1
  45 iface ovs-pub1 inet manual
  46         #address       178.63.149.225/28
  47         description         "fake-bridge: public dmz - public network1"
  48 
  49 allow-hotplug ovs-pub2
  50 iface ovs-pub2 inet manual
  51         #address       176.9.178.17/29
  52         description         "fake-bridge: public dmz - public network2"
  53 
  54 ### VLANS 1000-1499
  55 allow-hotplug ovs-1a
  56 iface ovs-1a inet static
  57         address       172.18.0.254/24
  58         description         "fake-bridge: public dmz - private network"

WiFi

Linux Wiki Wireless - About ath10k

Abbreviations

AP: Access Point
BSS: Basic Service Set (base station, AP)
FT: Fast transition
SSID: Service Set Identifier
WME: Wireless Multimedia Extensions (=WMM)
WMM: Wi-Fi Multimedia (=WME)

WPS

WPS is widely understood to be insecure. It should not be used.

vulernable to brute-force and dictionary attacks (in few hours)
Not very well supported
- Android removed WPS
- Linux desktop environments don't support it (gnome-shell seems to be an exception)

Wiki EN Wi-Fi Protected Setup

WiFi Optimization

I'm using OpenWRT most of the time. Here are some notes on configuring a secure and fast WiFi network. When changing options in the WiFi, it may be necessary to disconnect and reconnect to the SSID to make the changes work.

In WiFi: Device Configuration: General Setup
- Choose a channel that is not overcrowded with other SSIDs.
In WiFi: Device Configuration: Advanced Settings
- Always set the right country code
  to choose the correct frequency bands/channels.
- Don't check "Allow legacy 802.11b rates"
In WiFi: Interface Configuration: General Setup
- Enable "WMM Mode"
  Wiki EN Wi-Fi Multimedia based on the IEEE 802.11e provides basic Quality of service (QoS) features to IEEE 802.11 networks.
In WiFi: Interface Configuration: Security
- Don't use cipher set to 'auto'.
  Wiki EN Temporal Key Integrity Protocol (TKIP) was resolved to be deprecated by the IEEE in January 2009. TKIP limits is said limits the datarates.
- Check "Enable key reinstallation (KRACK) countermeasures"
- Enable IEEE "802.11r Fast Transition"
  to allow clients to move between 2.4GHz and 5GHz seamlessly. Please make sure to use an identical mobility domain in networks with the same SSID.

Other tuning tipps:

Devin Akin / December 28, 2014 - Top Ten Tune-up Tips

iw

List Wiki EN PHYs

   1 iw phy
   2 iw phy phy0 info

List devices

   1 iw dev
   2 iw dev wlan0 info
   3 iw wlan0 info

Get currently registered country (to avoid radar interference)

   1 iw reg get
   2 iw reg set DE

Hardware

linux-firmware_dl.sh

Download the firmware recursively and move firmware to its path in filesystem.

/usr/local/sbin/linux-firmware_dl.sh

   1 #!/bin/bash
   2 
   3 [ "$1" ] && GIT_DIR="$1" || exit 1
   4 [ "$2" ] && GIT_FILTER="$2"
   5 
   6 DIR_TMP_BASE="$(mktemp -d)"
   7 DIR_TMP="$DIR_TMP_BASE/$GIT_DIR"
   8 DIR_FW="/lib/firmware/$GIT_DIR"
   9 URL_GIT='https://git.kernel.org/pub/scm/linux/kernel/git/firmware/linux-firmware.git/plain'
  10 
  11 cleanup () {
  12         echo "Cleaning up"
  13         [ -d "$DIR_TMP" ] \
  14                 && echo "$DIR_TMP"|grep -q '^/tmp/' \
  15                 && rm -rvf "$DIR_TMP_BASE"
  16 }
  17 
  18 trap cleanup ERR
  19 
  20 mkdir -p "$DIR_TMP"
  21 cd "$DIR_TMP" || exit 1
  22 wget -r -nd -np -e robots=off \
  23                 -A README -A "$GIT_FILTER*.bin" \
  24                         "$URL_GIT/$GIT_DIR/"
  25 #wget --recursive --no-directories --no-parent -e robots=off \
  26 #       -A README -A "$GIT_FILTER*.bin" \
  27 #       "$URL_GIT/$GIT_DIR/"
  28 
  29 if [ "$?" -ne 0 ]; then
  30         echo "Download failed"
  31         exit 1
  32 fi
  33 
  34 ls -1 "$DIR_TMP"/*.tmp > /dev/null 2>&1 \
  35                 && rm "$DIR_TMP"/*.tmp
  36 cd ..
  37 [ -d "$DIR_FW" ] || mkdir "$DIR_FW"
  38 sudo mv "$DIR_TMP"/* "$DIR_FW"
  39 sudo chown -R 0.0 "$DIR_FW"
  40 sudo find "$DIR_FW" -type d -exec chmod 0755 {} \;
  41 sudo find "$DIR_FW" -type f -exec chmod 0644 {} \;
  42 
  43 cleanup

Use it like

   1 chmod u+x /usr/local/sbin/linux-firmware_dl.sh
   2 ### linux-firmware_dl.sh $SUBDIR $PREFIX
   3 linux-firmware_dl.sh rtw88
   4 ### OR FOR LATEST VEGA64 FIRMWARE
   5 #linux-firmware_dl.sh amdgpu vega10_
   6

Looks like this

   1 ll -d /lib/firmware/rtw88/*
   2 -rw-r--r-- 1 root root   1087 Mai 26 14:14 /lib/firmware/rtw88/README
   3 -rw-r--r-- 1 root root  28884 Mai 26 14:14 /lib/firmware/rtw88/rtw8723d_fw.bin
   4 -rw-r--r-- 1 root root 137896 Mai 26 14:14 /lib/firmware/rtw88/rtw8821c_fw.bin
   5 -rw-r--r-- 1 root root 150984 Mai 26 14:14 /lib/firmware/rtw88/rtw8822b_fw.bin
   6 -rw-r--r-- 1 root root 189152 Mai 26 14:14 /lib/firmware/rtw88/rtw8822c_fw.bin
   7 -rw-r--r-- 1 root root 138720 Mai 26 14:14 /lib/firmware/rtw88/rtw8822c_wow_fw.bin

Debian RTL8822BE/RTL8822CE

Works with Linux 4.19 with old driver rtlwifi
- uses firmware:
  - /lib/firmware/rtlwifi/rtl8822befw.bin, which is packed in firmware-realtek
Fails since Linux 5.2 with the new driver rtwpci -> rtw88
- uses firmware:
  - rtw88/rtw8822b_fw.bin
  - rtw88/rtw8822c_wow_fw.bin
  - rtw88/rtw8822c_fw.bin

So you need the firmware rtw88/rtw8822b_fw.bin

Debian Bug #945172 suggests to link the files.

I think it's better to get the most recent version from
git.kernel.org linux-firmware

You may use the following script to download rtw88 directory to /lib/firmware
#linux-firmware_dl.sh

Install the latest backports-kernel and reboot or just update your initramfs
update-initramfs -k all -u

I guess your WiFi works now.

And one day … Debian's firmware-packages will be refreshed and overwrite the contents of this manually created directory.

Working WiFi with rtw88 on Linux 5.6
lspci -vvs 04:00.0

   1 04:00.0 Network controller: Realtek Semiconductor Co., Ltd. RTL8822BE 802.11a/b/g/n/ac WiFi adapter
   2         Subsystem: Lenovo ThinkPad E595
   3         Physical Slot: 0
   4         Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx+
   5         Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
   6         Latency: 0, Cache Line Size: 32 bytes
   7         Interrupt: pin A routed to IRQ 80
   8         Region 0: I/O ports at 2000 [size=256]
   9         Region 2: Memory at d0600000 (64-bit, non-prefetchable) [size=64K]
  10         Capabilities: [40] Power Management version 3
  11                 Flags: PMEClk- DSI- D1+ D2+ AuxCurrent=375mA PME(D0+,D1+,D2+,D3hot+,D3cold+)
  12                 Status: D0 NoSoftRst+ PME-Enable- DSel=0 DScale=0 PME-
  13         Capabilities: [50] MSI: Enable+ Count=1/1 Maskable- 64bit+
  14                 Address: 00000000fee00000  Data: 0000
  15         Capabilities: [70] Express (v2) Endpoint, MSI 00
  16                 DevCap: MaxPayload 128 bytes, PhantFunc 0, Latency L0s <4us, L1 <64us
  17                         ExtTag- AttnBtn- AttnInd- PwrInd- RBE+ FLReset- SlotPowerLimit 0.000W
  18                 DevCtl: CorrErr- NonFatalErr- FatalErr- UnsupReq-
  19                         RlxdOrd+ ExtTag- PhantFunc- AuxPwr- NoSnoop-
  20                         MaxPayload 128 bytes, MaxReadReq 512 bytes
  21                 DevSta: CorrErr- NonFatalErr- FatalErr- UnsupReq- AuxPwr+ TransPend-
  22                 LnkCap: Port #0, Speed 2.5GT/s, Width x1, ASPM L0s L1, Exit Latency L0s <2us, L1 <64us
  23                         ClockPM+ Surprise- LLActRep- BwNot- ASPMOptComp-
  24                 LnkCtl: ASPM L1 Enabled; RCB 64 bytes Disabled- CommClk+
  25                         ExtSynch- ClockPM+ AutWidDis- BWInt- AutBWInt-
  26                 LnkSta: Speed 2.5GT/s (ok), Width x1 (ok)
  27                         TrErr- Train- SlotClk+ DLActive- BWMgmt- ABWMgmt-
  28                 DevCap2: Completion Timeout: Not Supported, TimeoutDis+, NROPrPrP-, LTR+
  29                          10BitTagComp-, 10BitTagReq-, OBFF Via message/WAKE#, ExtFmt-, EETLPPrefix-
  30                          EmergencyPowerReduction Not Supported, EmergencyPowerReductionInit-
  31                          FRS-, TPHComp-, ExtTPHComp-
  32                          AtomicOpsCap: 32bit- 64bit- 128bitCAS-
  33                 DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis-, LTR+, OBFF Disabled
  34                          AtomicOpsCtl: ReqEn-
  35                 LnkCtl2: Target Link Speed: 5GT/s, EnterCompliance- SpeedDis-
  36                          Transmit Margin: Normal Operating Range, EnterModifiedCompliance- ComplianceSOS-
  37                          Compliance De-emphasis: -6dB
  38                 LnkSta2: Current De-emphasis Level: -3.5dB, EqualizationComplete-, EqualizationPhase1-
  39                          EqualizationPhase2-, EqualizationPhase3-, LinkEqualizationRequest-
  40         Capabilities: [100 v2] Advanced Error Reporting
  41                 UESta:  DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
  42                 UEMsk:  DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
  43                 UESvrt: DLP+ SDES+ TLP- FCP+ CmpltTO- CmpltAbrt- UnxCmplt- RxOF+ MalfTLP+ ECRC- UnsupReq- ACSViol-
  44                 CESta:  RxErr- BadTLP- BadDLLP- Rollover- Timeout- AdvNonFatalErr-
  45                 CEMsk:  RxErr- BadTLP- BadDLLP- Rollover- Timeout- AdvNonFatalErr+
  46                 AERCap: First Error Pointer: 00, ECRCGenCap+ ECRCGenEn- ECRCChkCap+ ECRCChkEn-
  47                         MultHdrRecCap- MultHdrRecEn- TLPPfxPres- HdrLogCap-
  48                 HeaderLog: 00000000 00000000 00000000 00000000
  49         Capabilities: [148 v1] Device Serial Number 00-e0-4c-ff-fe-b8-22-01
  50         Capabilities: [158 v1] Latency Tolerance Reporting
  51                 Max snoop latency: 1048576ns
  52                 Max no snoop latency: 1048576ns
  53         Capabilities: [160 v1] L1 PM Substates
  54                 L1SubCap: PCI-PM_L1.2+ PCI-PM_L1.1+ ASPM_L1.2+ ASPM_L1.1+ L1_PM_Substates+
  55                           PortCommonModeRestoreTime=30us PortTPowerOnTime=60us
  56                 L1SubCtl1: PCI-PM_L1.2- PCI-PM_L1.1+ ASPM_L1.2- ASPM_L1.1+
  57                            T_CommonMode=0us LTR1.2_Threshold=0ns
  58                 L1SubCtl2: T_PwrOn=60us
  59         Kernel driver in use: rtw_pci
  60         Kernel modules: rtwpci

modinfo rtwpci rtw88

   1 filename:       /lib/modules/5.6.0-1-amd64/kernel/drivers/net/wireless/realtek/rtw88/rtwpci.ko
   2 license:        Dual BSD/GPL
   3 description:    Realtek 802.11ac wireless PCI driver
   4 author:         Realtek Corporation
   5 alias:          pci:v000010ECd0000C822sv*sd*bc*sc*i*
   6 alias:          pci:v000010ECd0000B822sv*sd*bc*sc*i*
   7 depends:        mac80211,rtw88
   8 retpoline:      Y
   9 intree:         Y
  10 name:           rtwpci
  11 vermagic:       5.6.0-1-amd64 SMP mod_unload modversions 
  12 sig_id:         PKCS#7
  13 signer:         Debian Secure Boot CA
  14 sig_key:        A7:46:8D:EF
  15 sig_hashalgo:   sha256
  16 signature:      79:B8:A3:7B:68:2D:CD:38:76:CB:48:1C:56:D4:20:77:7D:97:3C:24:
  17                 F5:BE:84:25:31:34:EE:27:03:F8:13:41:49:BD:09:E3:A7:09:86:CB:
  18                 91:50:0A:E0:3F:CA:19:CC:2A:AF:56:CE:D0:A2:4C:E0:83:C6:8F:71:
  19                 C0:E3:A2:68:BA:F6:50:F6:FC:10:76:E4:08:94:65:33:37:0A:56:9C:
  20                 C3:F9:AF:97:FA:30:7F:10:7A:47:81:28:F2:79:B5:79:7F:AE:F6:58:
  21                 6F:E2:6B:F6:78:8C:9D:89:37:26:67:3A:57:ED:03:16:79:26:EA:D2:
  22                 91:D5:F0:8B:1C:4D:CC:56:97:EA:3D:4F:45:5F:B7:54:C2:26:08:71:
  23                 A1:01:FF:A9:7E:2F:61:CF:C2:A8:DA:1C:1B:2C:D3:60:4C:D6:53:1E:
  24                 00:8D:3A:09:14:BB:7A:A7:27:8C:E4:BB:C9:40:85:EB:FE:0B:18:0A:
  25                 76:39:F7:9F:70:FB:0B:DB:BA:33:BC:31:0F:C2:75:45:E1:11:1A:B4:
  26                 58:31:6E:26:CC:45:AE:AC:4D:67:5B:DE:CC:08:D8:01:49:D9:71:E8:
  27                 25:6C:C5:E8:DF:F7:DE:64:CE:34:00:5F:7A:3D:E6:8D:77:28:FD:BB:
  28                 6A:E5:83:41:61:46:0F:73:C7:21:F9:90:2F:5A:6D:93
  29 parm:           disable_msi:Set Y to disable MSI interrupt support (bool)
  30 
  31 filename:       /lib/modules/5.6.0-1-amd64/kernel/drivers/net/wireless/realtek/rtw88/rtw88.ko
  32 license:        Dual BSD/GPL
  33 description:    Realtek 802.11ac wireless core module
  34 author:         Realtek Corporation
  35 firmware:       rtw88/rtw8822b_fw.bin
  36 firmware:       rtw88/rtw8822c_wow_fw.bin
  37 firmware:       rtw88/rtw8822c_fw.bin
  38 depends:        mac80211,cfg80211
  39 retpoline:      Y
  40 intree:         Y
  41 name:           rtw88
  42 vermagic:       5.6.0-1-amd64 SMP mod_unload modversions 
  43 sig_id:         PKCS#7
  44 signer:         Debian Secure Boot CA
  45 sig_key:        A7:46:8D:EF
  46 sig_hashalgo:   sha256
  47 signature:      AD:30:FA:52:72:54:79:79:FC:7B:8A:52:92:19:F5:30:91:CD:F2:13:
  48                 00:8A:FD:8D:11:B2:94:FA:DB:4E:BF:B7:32:5D:EB:71:C6:27:81:34:
  49                 87:D9:59:7F:8F:32:6F:E6:2F:AF:F9:8F:EF:E2:E1:FF:39:EE:AD:EB:
  50                 BF:13:9C:CE:9A:F6:72:3A:8E:27:91:E4:98:60:48:4C:36:84:3E:90:
  51                 01:4D:4A:BA:7C:5E:D5:7B:7F:C0:3F:74:1C:C7:04:04:EC:9D:5D:55:
  52                 D6:CE:AE:2C:F6:8E:37:94:83:1B:D2:6D:34:17:DA:59:B0:57:68:6C:
  53                 A3:E6:5A:2D:3E:2D:FB:EA:C0:08:E1:0C:DE:64:1C:84:17:75:CD:C1:
  54                 0F:C5:C4:CE:97:E1:24:2E:57:F1:B8:EF:9E:8B:B0:C7:99:B6:1C:1D:
  55                 4D:AE:49:DE:BD:3B:40:65:74:C5:C8:DF:96:C2:40:DC:7B:23:6E:73:
  56                 20:52:E4:DF:E2:C1:86:D3:F0:C6:B4:6D:5E:12:97:09:EE:82:A6:5F:
  57                 E6:E0:69:95:9A:98:69:B8:F5:48:12:2F:4A:BB:5B:FD:3E:63:0E:A7:
  58                 D8:40:A8:55:E5:07:E8:81:EF:5E:36:3E:38:6F:D9:A5:75:BE:6E:D5:
  59                 F5:70:C2:AD:F5:4F:94:D4:D4:29:68:E6:31:FA:8D:6E
  60 parm:           lps_deep_mode:Deeper PS mode. If 0, deep PS is disabled (uint)
  61 parm:           support_bf:Set Y to enable beamformee support (bool)
  62 parm:           debug_mask:Debugging mask (uint)

Network-Manager

About

Please see

man 5 nm-settings for a description of settings and properties of NetworkManager connection profiles for nmcli
man 5 NetworkManager.conf for a description of the NetworkManager configuration file

Ecosystem

nmcli (command line interface)
nm-connection-editor (gui-editor
nm-applet (gui integration)
nm-online (check if online)

Authorization with policy kit

If you are in group sudo or netdev, you are allowed to modify system-connections.

/usr/share/polkit-1/rules.d/60-network-manager.rules

   1 polkit.addRule(function(action, subject) {
   2   if (action.id == "org.freedesktop.NetworkManager.settings.modify.system" &&
   3         subject.local && subject.active && 
   4         (subject.isInGroup ("sudo") || subject.isInGroup ("netdev"))) {
   5     return polkit.Result.YES;
   6   }
   7 });

Do not manage interfaces

You can configure Network-Manager to not manage the interfaces mentioned in
/etc/network/interfaces

   1 # This file describes the network interfaces available on your system
   2 # and how to activate them. For more information, see interfaces(5).
   3 
   4 source /etc/network/interfaces.d/*
   5 
   6 # The loopback network interface
   7 auto lo
   8 iface lo inet loopback
   9 
  10 auto enp1s0
  11 iface enp1s0 inet dhcp
  12 
  13 #auto enp2s0
  14 iface enp2s0 inet dhcp

Make sure you don't remove the line entirely or Network-Manager will grab the interface and spawn a dhclient for the interface.

/etc/NetworkManager/NetworkManager.conf

   1 [main]
   2 plugins=ifupdown,keyfile
   3 
   4 [ifupdown]
   5 managed=false

Here is the corresponding section from
man 5 NetworkManager.conf

   1 IFUPDOWN SECTION
   2        This section contains ifupdown-specific options and thus
   3        only has effect when using the ifupdown plugin.
   4 
   5        managed
   6            If set to true, then interfaces listed in
   7            /etc/network/interfaces are managed by
   8            NetworkManager. If set to false, then any interface
   9            listed in /etc/network/interfaces will be ignored by
  10            NetworkManager. Remember that NetworkManager
  11            controls the default route, so because the interface
  12            is ignored, NetworkManager may assign the default
  13            route to some other interface.
  14 
  15            The default value is false.

Manage interfaces

Something to remember …

Network-Manager controls the default route.

If you try to connect to a VPN with the default route assigned by a interface that is unmanaged by Network-Manager (Network-Manager does not control the default route), the following error message is logged.

   1 Oct 30 17:27:14 libertas NetworkManager[116637]: <info>  [1604075234.9052] audit: op="connection-activate" uuid="6bcc9142-242a-44c9-adff-d30601f41919" name="openvpn_connection" pid=5284 uid=1000 result="fail" reason="Could not find source connection."

When setting to managed=true,
the following error is logged activating bond0.

   1 # nmcli connection  up Ifupdown\ \(bond0\) 
   2 Fehler: Aktivierung der Verbindung ist gescheitert: No suitable device found for this connection (device docker0 not available because profile is not compatible with device (mismatching interface name)).

In ifupdown-managed mode Network-Manager automatically creates configurations for the devices listes in /etc/network/interfaces. These files are located in
ll /run/NetworkManager/system-connections

   1 -rw------- 1 root root 393 30. Oct 18:50 bond0.nmconnection
   2 -rw------- 1 root root 433 30. Oct 12:19 docker0.nmconnection
   3 -rw------- 1 root root 304 30. Oct 18:40 enp8s0.nmconnection
   4 -rw------- 1 root root 304 30. Oct 18:40 enp9s0.nmconnection
   5 -rw------- 1 root root 438 30. Oct 12:19 virbr0.nmconnection

The bonding configuration is not bad at all but is also not functionial. So i converted the dynamically generated config to a permanent config and adjusted it. Please see #Bonding with Network-Manager. Now that NM controls the default route, activating/… VPNs works simply fine!

Imported VPNs

Network-Manager imports various VPN-profiles, but does not set the file permissions on the private key tight enough.

NM stores certificates in subdirectories below
~/.local/share/networkmanagement/certificates/. Make sure you have the correct selinux context on this files, too.

   1 find ~/.local/share/networkmanagement/certificates/ \
   2         -type f -name private.key -exec chmod 600 {} \;

OpenVPN

The username may be case-sensitive.
You may disable getting pushed the default route in
VPN configuration dialogue > IPv4/6 > Button > (Check) "Use only for resources of this connection"

Disable MAC randomization

This is interessting e.g. if you want to give your raspberrypi a static IP address in your LAN.

/etc/NetworkManager/conf.d/100-disable-wifi-mac-randomization.conf

   1 [connection]
   2 wifi.mac-address-randomization=0
   3 
   4 [device]
   5 wifi.scan-rand-mac-address=no

Common network setups

single points of failures (SPOF)

Setup 1

Setup
- Uplink: SPOF
- Firewall: SPOF
- Switches: SPOF
- Hosts: SPOF
Attributes
- most simple and cheap setup
- maintenance always causes downtime
- no redundancy at all, let them fail
- cascading single points of failures
  - The longer the cascade, the higher the risk of failure.

Setup 2

Setup
- Uplink: SPOF
- Firewall:
  - redundant (active-standby)
  - maintenance without service interuption
- Switches: SPOF
- Hosts: SPOF
- Additional external switch: SPOF
  - gives some additional flexibility
  - distributes the uplink
  - probably unnecessary
  - additional spof
Attributes
- better external connectivity

Setup 3

Setup
- Uplink:
  - redundant
  - single isp handles failure of a uplink router
  - dual-isp setup with own Autonomous System (AS) and dynamic routing (BGP) possible
- Firewall:
  - redundant (active-standby)
  - maintenance without service interuption
- Switches: SPOF
- Hosts: SPOF
Comments
1. Setup may be distributed, e.g. to different buildings

Setup 4

Setup
- Uplink:
  - redundant
  - single isp handles failure of a uplink router
  - dual-isp setup with own Autonomous System (AS) and dynamic routing (BGP) possible
- Firewall:
  - redundant (active-standby)
  - maintenance without service interuption
- Switches:
  - redundant (STP)
  - no addtional performance (links blocked)
- Hosts: bond (active-backup)

Setup 5

Setup
- Uplink:
  - redundant
  - single isp handles failure of a uplink router
  - dual-isp setup with own Autonomous System (AS) and dynamic routing (BGP) possible
- Firewall:
  - redundant (active-standby)
  - maintenance without service interuption
  - bond(lacp)
- Switches:
  - redundant (STP)
  - no addtional performance (links blocked)
- Hosts: bond (active-backup)

Setup 6

Setup
- Uplink:
  - redundant
  - single isp handles failure of a uplink router
  - dual-isp setup with own Autonomous System (AS) and dynamic routing (BGP) possible
- Firewall:
  - redundant (active-standby)
  - maintenance without service interuption
  - bond(active-backup)
- Switches:
  - redundant (STP)
  - no addtional performance (links blocked)
- Hosts: bond (active-backup)

Setup 7

Setup
- Uplink:
  - redundant
  - single isp handles failure of a uplink router
  - dual-isp setup with own Autonomous System (AS) and dynamic routing (BGP) possible
- Firewall:
  - redundant (active-standby)
  - maintenance without service interuption
  - bond(lacp)
- Switches:
  - redundant (stack)
  - no wasted performance (lags with lacp)
- Hosts: bond (lacp)

Setup 8

Setup
- Uplink:
  - redundant
  - single isp handles failure of a uplink router
  - dual-isp setup with own Autonomous System (AS) and dynamic routing (BGP) possible
- Firewall:
  - redundant (active-standby)
  - maintenance without service interuption
  - bond(lacp)
- Switches:
  - redundant (peered)
  - no wasted performance (mc-lags with lacp)
- Hosts: bond (lacp)

Recommended modern approach

GNS3

Please see: networking/gns3

Routing non-local traffic in Azure (IPsec)

These are some notes for establishing a IPsec S2S tunnel between a OPNsense Appliance in Azure and a remote Azure Virtual Network Gateway just to have a endpoint to terminate the IPsec connection.

Hint: In Azure every thing takes its time. Even if the Azure Frontend tells you the operation has been successfully fulfilled, it may not yet be applied to the network. Be patient during your experiments.
- Applying route tables
- Enabling IP forwarding
- …
Create a Network Security Group (NGS) and bind it to the WAN interface of the OPNsense to allow incoming and outgoing traffic for
- ISAKMP (udp/500)
- IPsec NAT-T (udp/4500)
Azure does not really expect you to do low level networking e.g. use your own router with multiple interfaces. Networking is very different in Azure!
1. Routes are pushed to your virtual machine
  with DHCP option 121 classless-routes
  - DNS, DHCP, waagent WireServer endpoint 168.63.129.16 https://docs.microsoft.com/en-us/azure/virtual-network/what-is-ip-address-168-63-129-16
  - Default gateway
  - APIPA
2. I encountered a problem, where 168.63.129.16 was routed via the wrong the interface directly to a MAC.
  To resolve this delete all dhclient.leases and reboot the firewall.
  rm /var/db/dhclient.leases.hn*
3. Routes may also be set by the Azure Linux VM Agent
  https://docs.microsoft.com/en-us/azure/virtual-machines/extensions/agent-linux
4. It's not possible to create a "real" isolated networks.
  - There are always reserved IP-addresses in a subnet. As far as i know, the first addresses in the network cannot be reclaimed. (Untested so far) The default gateway may be changed using a custom user managed routing table. Networks smaller than /29 are not even possible.
    Azure Virtual Network FAQ
  - All network interfaces (NIC) attached to a VM deployed through the Resource Manager deployment model must be connected to a VNet.
    Azure Virtual Network FAQ
5. Only the first interface is pushed a default gateway.
6. Be aware of the strange MAC-addresses 12:34:56:78:9a:bc, which are not unique in a layer 2 broadcast domain.
Network security groups (NSG) are not important. I disassociated them from all NICs and subnets and was still able to ping though the IPsec tunnel of the OPNsense. Possible explanation: The packets are originating from a NIC in the vNet and therefor are allowed be the default ruleset.
Test connectivity and routing not directly with ping,
use a wrapping while loop. The OPNsense seems to remember the ping sequence number and keep the (established/related) state.
```
   1  DST_IP="10.255.252.5"
   2  while sleep 1; do ping -c4 -W2 $DST_IP; done
```
I recommend using a intermediate transport/transfer network “TRANSFER” between an Azure standard gateway and the OPNsense.
In the IP configurations “IP forwarding” needs to be enabled for each interface that should be allowed to receive traffic not destined for itself e.g. in the “TRANSFER” network. This can be understood as promiscious mode.
In the OPNsense a gateway “AZURE_GW” with least priority (255) in the transfer network “TRANSFER” with the first IP-address in the network needs to be created.
In the OPNsense a static route is created that directs traffic for the application network to the “AZURE_GW”.
A “route table” (RT) is necessary in every network that has machines attached, which need to reach the VPN-Pool or to a remote network. This RT needs the following entries:
- remote-network via OPNsense interface in network "TRANSFER"
- VPN pool via OPNsense interface in network "TRANSFER"
Some hint when using OPNsense (BI)NAT with IPsec traffic:
- The remote endpoint should use only NATed addresses in the IPsec local networks and the firewall ruleset.
- The local OPNsense endpoint should enter original addresses and NATed addresses to IPsec local networks and the firewall ruleset. This allows the traffic to be put on the IPsec interface, where the (BI)NAT rules are applied
  (bound to the interface "IPsec" -> late). Otherwise the OPNsense answers with “ICMP host unreachable”. For a OpenVPN remote access VPN unnated pool does not have to be included in a phase 2 SA, if NAT-rules are in place.

Using OPNsense in Azure as Default GW

Pro

More conventional approach
Single point of access to Azure
Additional control
- All traffic can be filtered by OPNsense firewall rules on the perimeter
Lower direct attacking surface to the UDP-TCP/IP-stacks
Better debugging of network flows
- Live-Logs
- tcpdump on cli, …
Additional features
- Intrusion Detection/Prevention System (IDPS)
NAT may be used
Better and more responsive GUI

Contra

More cloud native approach
Additional networking/routing hop
- Adds some latency
- Adds some complexity
Possible performance impact on high traffic
Some state/configuration has to be maintained in the OPNsense
If the micro-service application is designed for a zero-trust security model, filtering on the perimeter may not be necessary.
OPNsense is not redundant
The OPNsense approach, may not be as flexible when it comes to adding further Public IP-addresses
The firewall rule set on the OPNsense can be much simpler when not being installed as default gateway

Uncommon Ports

DATEV Ports

Name	Protocol	Description	Port Number
DATEV_4401	tcp;	DATEV Extended Security Tunnel Client	4401
DATEV_7279	tcp;	DATEV Zentrale Komponente für Sicherheits-Plugins	7279
DATEV_9919	tcp;	DATEV Sicherheitspaket; Hostprozess	9919
DATEV_10000	tcp;	DATEV Extended Security Tunnel Client	10000
DATEV_58431	tcp;	DATEV-Lizenzserver	58431
DATEV_58432	tcp;	DATEV-Lizenzserver	58432
DATEV_58451	tcp;	DATEV DFL-Services; Hosting HTTP	58451
DATEV_58452	tcp;	DATEV DFL-Services; Hosting HTTPS	58452