Rockstable Wiki:

networking

Trouble Shooting

In non-deterministic cases you may resort to
IETF RFC 2321 - RITA -- The Reliable Internetwork Troubleshooting Agent

Models OSI and TCP/IP

attachment:OSI.dot.svg

Address Resolution Protocol (ARP)

The Address Resolution Protocol (ARP) is a communication protocol used for discovering the link layer address, such as a MAC address, associated with a given internet layer address, typically an IPv4 address. This mapping is a critical function in the Internet protocol suite. ARP was defined in 1982 by RFC 826, which is Internet Standard STD 37.

Notes

Old D-Link DGS-series switches (like DGS-1210-16 Rev. A1) seam to fail ARP in combination with CARP/VRRP gateways. In such cases the default gateway cannot be resolved to a MAC-address and routing fails generally. ARP resolution within the same network may still work. This may be diagnosed using telnet and the command debug info. The MAC from the CARP/VRRP address is only in the MAC-forwarding table, but missing in the host arp-table (mgmtVlan).

# WIP - exact feature has to be determined

Neighbor Discovery Protocol (NDP)

MAC addresses

EUI-48 (Extended Unique Identifier 48)

3-octet OUI (Organizationally Unique Identifier)

Local database

   1 aptitude install ieee-data

   1 /usr/share/ieee-data/iab.csv
   2 /usr/share/ieee-data/iab.txt
   3 /usr/share/ieee-data/mam.csv
   4 /usr/share/ieee-data/mam.txt
   5 /usr/share/ieee-data/oui.csv
   6 /usr/share/ieee-data/oui.txt ### <-- MAC-ADRESSESS
   7 /usr/share/ieee-data/oui36.csv
   8 /usr/share/ieee-data/oui36.txt
   9 
  10 /usr/share/nmap/nmap-mac-prefixes

Spanning Tree Protocol

Algorhyme

Radia Perlman penned this poem while she developed Spanning Tree.

I think that I shall never see
A graph more lovely than a tree.
A tree whose crucial property
Is loop-free connectivity.
A tree that must be sure to span
So packets can reach every LAN.
First, the root must be selected.
By ID, it is elected.
Least-cost paths from root are traced.
In the tree, these paths are placed.
A mesh is made by folks like me,
Then bridges find a spanning tree.

--
Radia Perlman 

STP

Standards

Bridge-ID (BID) is 8 Byte long (2 Byte bridge priority, 6 Byte MAC address).

Bridge Priority
is basically a 4bit number and locally assigned system ID extension (12 bits)
has only 16 values,

p * 212, where 0 ≤ p ≤ 24

Root Bridge
of the spanning tree is the bridge with the smallest (lowest) bridge ID.

There can only be one root bridge in a spanning tree. ;-)

When initializing the protocol all bridges send out BPDUs, with themselves as root bridge.
After convergence only the root bridge generates BPDUs. Other devices only forward BPDUs.
Has no root port.
All the ports on the root bridge are designated ports
Designated Bridge
Has exactly one root port
Device responsible to forwards frames to a LAN segment

Bridge Protocol Data Units (BPDUs)

Port states (4)

Blocking
BPDUs are received and processed
Frames are not forwarded
This port would cause a topology loop
May transit to forwarding state on failure of another link
Listening
BPDUs are received and processed
Frames are not forwarded
MAC table is not populated
May return to blocking
Learning
BPDUs are received and processed
Frames are not forwarded
MAC table is populated
May return to blocking
Forwarding
BPDUs are received and processed
Frames are forwarded
May return to blocking
Disabled
Not strictly part of STP
Manually disabled switch port

Blocking -> Listening -> Learning -> Forwarding

Port roles (2)

Root
A forwarding port that is the best port from non-root bridge to root bridge
Port on which a device received the optimum configuration BPDU.
Designated
A forwarding port for every LAN segment
Disabled
Not strictly part of STP, a network administrator can manually disable a port

Timers

The root bridge sets the timer values and distributes these in Configuration BPDUs.

When a new device is attached it takes 2x Forward-Delay timers (default: 30s) to transit to the state Forwarding.

Path cost

Rapid STP

Port roles

Root
A forwarding port that is the best port from non-root bridge to root bridge
Designated
A forwarding port for every LAN segment
Alternate
An alternate path to the root bridge.
This path is different from using the root port.
Backup
A backup/redundant path to a segment where another bridge port already connects
Disabled
Not strictly part of STP, a network administrator can manually disable a port

port states (3)

Discarding
BPDUs are received and processed
Frames are not forwarded
This port would cause a topology loop
May transit to forwarding state on failure of another link
Learning
BPDUs are received and processed
Frames are not forwarded
MAC table is populated
May return to blocking
Forwarding
BPDUs are received and processed
Frames are forwarded
May return to blocking
Disabled
Not strictly part of STP
manually disabled switch port

Timers

Per-VLAN-Spanning Tree

Multiple STP

Port roles (6)

Root port
forwards data to the root bridge
Designated port
forwards data to the designated bridge for a downstream network segment or device
Boundary port
port that connects a MST region to
another MST region or
a network-segment running STP, or RSTP
Master port
root port (of a region) on the CIST to the common root bridge
Alternate port
Backup port for a root port and master port
Does not forward frames
Takes over when the root port or master port has failed
Starts forwarding without delay
Backup port
Backup port of a designated port
Starts forwarding without delay
Disabled port
admin down in every MSTI

A port may have different roles in different MSTIs.

Port states (3)

A port may have different states in different MSTIs.

Notes

Never use simple old STP.

Hold-down time (50s) is not reconcilable with high-availability.

If STP is in default configuration (with bridge-priority 32768) the election is based on the mac address. The root bridge bridge may be located in a inefficient position, e.g. far away from the router. If a STP-protocol is used in a network, it must be planned and configured carefully!

Different implementations of a standard are not guaranteed to work, due for example to differences in default timer settings.

Try to use (multi-chassis) link-aggregations ((MC)-LAGs) where possible, to avoid blocking redundant ports and enhance bandwidth.

Make sure to configure BPDU-guard to protect your network from malicious bridges.

Edge-ports should be used

You may use a BPDU-filter to discard BPDUs from adjacent switching infrastructures, when it's clear that a loop can never be established.

Use loop detection on the edge to the network.

Network

Common Address Redundancy Protocol (CARP)

Alternatives: VRRP, HSRP, GLBP, ESRP, R-SMLT, NSRP

Sniff carp

   1 tcpdump -npi vtnet2 -T carp

IP addresses

In DNS make sure your NS, MX and A records for a given domain don't share a single IP-address.

NetBIOS

* IETF RFC1002 PROTOCOL STANDARD FOR A NetBIOS SERVICE ON A TCP/UDP TRANSPORT: DETAILED SPECIFICATIONS

NetBIOS name

Patrick P. Yeung NetBIOS Suffix Code Table

Name

Number (HEX)

Type

Usage

<computername>

00

U

Workstation Service

<computername>

01

U

Messenger Service

<computername>

03

U

Messenger Service

<computername>

06

U

RAS Server Service

<computername>

1f

U

NetDDE Service

<computername>

20

U

File Server Service

<computername>

21

U

RAS Client Service

<computername>

22

U

Exchange Interchange

<computername>

23

U

Exchange Store

<computername>

24

U

Exchange Directory

<computername>

30

U

Modem Sharing Server Service

<computername>

31

U

Modem Sharing Client Service

<computername>

43

U

SMS Client Remote Control

<computername>

44

U

SMS Admin Remote Control Tool

<computername>

45

U

SMS Client Remote Chat

<computername>

46

U

SMS Client Remote Transfer

<computername>

4c

U

DEC Pathworks TCPIP Service

<computername>

52

U

DEC Pathworks TCPIP Service

<computername>

87

U

Exchange MTA

<computername>

6a

U

Exchange IMC

<computername>

be

U

Network Monitor Agent

<computername>

bf

U

Network Monitor Apps

<username>

03

U

Messenger Service

<\\_MSBROWSE_>

01

G

Master Browser

<domain>

00

G

Domain Name (Membership)

<domain>

1b

U

Domain Master Browser

<domain>

1c

G

Domain Controllers,
Domain Logon Server

<domain>

1d

U

(Local) Master Browser

<domain>

1e

G

(Local) Browser Service Elections

<INet~Services>

1c

G

Internet Information Server

<IS~Computer_name>

00

U

Internet Information Server

<computername>

[2b]

U

Lotus Notes Server

IRISMULTICAST

[2f]

G

Lotus Notes

IRISNAMESERVER

[33]

G

Lotus Notes

Forte_$ND800ZA

[20]

U

DCA Irmalan Gateway Service

NetBIOS Name resolution order in MS Windows

Order of resolution

  1. Cache
    • contains static entries

      from LMHOSTS with the tag #PRE

  2. WINS
  3. Broadcast
  4. LMHOSTS.SAM

  5. DNS (if configured)

lmhosts

Static NetBIOS name resolution like /etc/hosts

LMHOSTS.SAM

   1 192.168.1.11    samba #PRE
   2 #INCLUDE        \\samba\public\lmhosts
   3 

Keywords (case sensitive, labels are case-insensitive):

WINS

Microsoft Docs - Windows Internet Name Service (WINS)

WINS is as obsolete as NetBIOS is.

There can only be one WINS server in a given network or the NetBIOS namespace is divided (even with multiple workgroups or domains). Replicating WINS servers may be used (Windows or samba4WINS).

WINS uses directed UDP unicast, which is routed across network boundaries. Please make sure the firewall is open on udp/137.

Names, which are registered in WINS, are resolved using directed udp-unicast, otherwise they are resolved using broadcasts.

WINS works without wait times for complaints and thus is faster and conserves resources.

Consider using WINS even in small networks to reduce NetBIOS broadcasts.

WINS names are registered on boot of a node (e.g. in case the WINS server was changed during runtime).

The reservation is only valid for a limited amount of time and has to be renewed in regular intervals. The intervals can be chosen by the client, but the server defines minimum and maximum boundries.

Via dhcp WINS server is distributed as netbios-name-servers. option netbios-node-type 8; ### DEFAULT may be set to explicitly define NetBIOS behaviour.

type

short name

long name

resolution

1

B-Node

broadcast

broadcast

2

P-Node

point-to-point

WINS

4

M-Node

mixed

1st broadcast, 2nd WINS

8

H-Node

hybrid

1st WINS, 2nd broadcast

Transport

Ephemeral Ports

Range

The Internet Assigned Numbers Authority (IANA) suggests the range 49152 to 65535 (215 + 214 to 216−1) for dynamic or private ports.

Many Linux kernels use the port range 32768 to 61000. FreeBSD has used the IANA port range since release 4.6. Previous versions, including the Berkeley Software Distribution (BSD), use ports 1024 to 5000 as ephemeral ports.

View and customize ephemeral ports range

   1 cat /proc/sys/net/ipv4/ip_local_port_range
   2 32768   60999
   3 sysctl net.ipv4.ip_local_port_range
   4 net.ipv4.ip_local_port_range = 32768    60999

Microsoft Windows operating systems through XP use the range 1025–5000 as ephemeral ports by default. Windows Vista, Windows 7, and Server 2008 use the IANA range by default. Windows Server 2003 uses the range 1025–5000 by default, until Microsoft security update MS08-037 from 2008 is installed, after which it uses the IANA range by default. Windows Server 2008 with Exchange Server 2007 installed has a default port range of 1025–60000. In addition to the default range, all versions of Windows since Windows 2000 have the option of specifying a custom range anywhere within 1025–65535.

Hardware info

PCI devices

   1 lspci -vv -s 00:1f.6

Tools

dsniff

https://www.monkey.org/~dugsong/dsniff/

dsniff is a collection of tools for network auditing and penetration testing. dsniff, filesnarf, mailsnarf, msgsnarf, urlsnarf, and webspy passively monitor a network for interesting data (passwords, e-mail, files, etc.). arpspoof, dnsspoof, and macof facilitate the interception of network traffic normally unavailable to an attacker (e.g, due to layer-2 switching). sshmitm and webmitm implement active monkey-in-the-middle attacks against redirected SSH and HTTPS sessions by exploiting weak bindings in ad-hoc PKI.

Install

   1 apt install dsniff

ethtool

https://www.kernel.org/pub/software/network/ethtool/

Display or change Ethernet device settings. ethtool can be used to query and change settings such as speed, auto- negotiation and checksum offload on many network devices, especially Ethernet devices.

Gather info

Get basic link state (speed, duplex, MDI-X, link, autoneg …)

   1 # ethtool enp0s31f6 
   2 Settings for enp0s31f6:
   3         Supported ports: [ TP ]
   4         Supported link modes:   10baseT/Half 10baseT/Full 
   5                                 100baseT/Half 100baseT/Full 
   6                                 1000baseT/Full 
   7         Supported pause frame use: No
   8         Supports auto-negotiation: Yes
   9         Supported FEC modes: Not reported
  10         Advertised link modes:  10baseT/Half 10baseT/Full 
  11                                 100baseT/Half 100baseT/Full 
  12                                 1000baseT/Full 
  13         Advertised pause frame use: No
  14         Advertised auto-negotiation: Yes
  15         Advertised FEC modes: Not reported
  16         Speed: 1000Mb/s
  17         Duplex: Full
  18         Port: Twisted Pair
  19         PHYAD: 1
  20         Transceiver: internal
  21         Auto-negotiation: on
  22         MDI-X: on (auto)
  23         Supports Wake-on: pumbg
  24         Wake-on: g
  25         Current message level: 0x00000007 (7)
  26                                drv probe link
  27         Link detected: yes

Queries the specified network device for associated driver information.

   1 ethtool -i enp0s31f6

Queries the specified network device for the state of protocol offload and other features.

   1 # ethtool -k enp0s31f6
   2 Features for enp0s31f6:
   3 rx-checksumming: on
   4 tx-checksumming: on
   5         tx-checksum-ipv4: off [fixed]
   6         tx-checksum-ip-generic: on
   7         tx-checksum-ipv6: off [fixed]
   8         tx-checksum-fcoe-crc: off [fixed]
   9         tx-checksum-sctp: off [fixed]
  10 scatter-gather: on
  11         tx-scatter-gather: on
  12         tx-scatter-gather-fraglist: off [fixed]
  13 tcp-segmentation-offload: on
  14         tx-tcp-segmentation: on
  15         tx-tcp-ecn-segmentation: off [fixed]
  16         tx-tcp-mangleid-segmentation: off
  17         tx-tcp6-segmentation: on
  18 udp-fragmentation-offload: off
  19 generic-segmentation-offload: on
  20 generic-receive-offload: on
  21 large-receive-offload: off [fixed]
  22 rx-vlan-offload: on
  23 tx-vlan-offload: on
  24 ntuple-filters: off [fixed]
  25 receive-hashing: on
  26 highdma: on [fixed]
  27 rx-vlan-filter: off [fixed]
  28 vlan-challenged: off [fixed]
  29 tx-lockless: off [fixed]
  30 netns-local: off [fixed]
  31 tx-gso-robust: off [fixed]
  32 tx-fcoe-segmentation: off [fixed]
  33 tx-gre-segmentation: off [fixed]
  34 tx-gre-csum-segmentation: off [fixed]
  35 tx-ipxip4-segmentation: off [fixed]
  36 tx-ipxip6-segmentation: off [fixed]
  37 tx-udp_tnl-segmentation: off [fixed]
  38 tx-udp_tnl-csum-segmentation: off [fixed]
  39 tx-gso-partial: off [fixed]
  40 tx-sctp-segmentation: off [fixed]
  41 tx-esp-segmentation: off [fixed]
  42 tx-udp-segmentation: off [fixed]
  43 fcoe-mtu: off [fixed]
  44 tx-nocache-copy: off
  45 loopback: off [fixed]
  46 rx-fcs: off
  47 rx-all: off
  48 tx-vlan-stag-hw-insert: off [fixed]
  49 rx-vlan-stag-hw-parse: off [fixed]
  50 rx-vlan-stag-filter: off [fixed]
  51 l2-fwd-offload: off [fixed]
  52 hw-tc-offload: off [fixed]
  53 esp-hw-offload: off [fixed]
  54 esp-tx-csum-hw-offload: off [fixed]
  55 rx-udp_tunnel-port-offload: off [fixed]
  56 tls-hw-tx-offload: off [fixed]
  57 tls-hw-rx-offload: off [fixed]
  58 rx-gro-hw: off [fixed]
  59 tls-hw-record: off [fixed]

Alter

Restart auto-negotiation if enabled

   1 ethtool -r eth0

De/activate features of a nic

Examples:

   1 # ethtool -K enp0s31f6 sg off tso off gro off
   2 Actual changes:
   3 scatter-gather: off
   4         tx-scatter-gather: off
   5 tcp-segmentation-offload: off
   6         tx-tcp-segmentation: off
   7         tx-tcp6-segmentation: off
   8 generic-segmentation-offload: off [requested on]
   9 generic-receive-offload: off

Post deactivation

   1 # ethtool -k enp0s31f6
   2 Features for enp0s31f6:
   3 rx-checksumming: on
   4 tx-checksumming: on
   5         tx-checksum-ipv4: off [fixed]
   6         tx-checksum-ip-generic: on
   7         tx-checksum-ipv6: off [fixed]
   8         tx-checksum-fcoe-crc: off [fixed]
   9         tx-checksum-sctp: off [fixed]
  10 scatter-gather: off
  11         tx-scatter-gather: off
  12         tx-scatter-gather-fraglist: off [fixed]
  13 tcp-segmentation-offload: off
  14         tx-tcp-segmentation: off
  15         tx-tcp-ecn-segmentation: off [fixed]
  16         tx-tcp-mangleid-segmentation: off
  17         tx-tcp6-segmentation: off
  18 udp-fragmentation-offload: off
  19 generic-segmentation-offload: off [requested on]
  20 generic-receive-offload: off
  21 large-receive-offload: off [fixed]
  22 rx-vlan-offload: on
  23 tx-vlan-offload: on
  24 ntuple-filters: off [fixed]
  25 receive-hashing: on
  26 highdma: on [fixed]
  27 rx-vlan-filter: off [fixed]
  28 vlan-challenged: off [fixed]
  29 tx-lockless: off [fixed]
  30 netns-local: off [fixed]
  31 tx-gso-robust: off [fixed]
  32 tx-fcoe-segmentation: off [fixed]
  33 tx-gre-segmentation: off [fixed]
  34 tx-gre-csum-segmentation: off [fixed]
  35 tx-ipxip4-segmentation: off [fixed]
  36 tx-ipxip6-segmentation: off [fixed]
  37 tx-udp_tnl-segmentation: off [fixed]
  38 tx-udp_tnl-csum-segmentation: off [fixed]
  39 tx-gso-partial: off [fixed]
  40 tx-sctp-segmentation: off [fixed]
  41 tx-esp-segmentation: off [fixed]
  42 tx-udp-segmentation: off [fixed]
  43 fcoe-mtu: off [fixed]
  44 tx-nocache-copy: off
  45 loopback: off [fixed]
  46 rx-fcs: off
  47 rx-all: off
  48 tx-vlan-stag-hw-insert: off [fixed]
  49 rx-vlan-stag-hw-parse: off [fixed]
  50 rx-vlan-stag-filter: off [fixed]
  51 l2-fwd-offload: off [fixed]
  52 hw-tc-offload: off [fixed]
  53 esp-hw-offload: off [fixed]
  54 esp-tx-csum-hw-offload: off [fixed]
  55 rx-udp_tunnel-port-offload: off [fixed]
  56 tls-hw-tx-offload: off [fixed]
  57 tls-hw-rx-offload: off [fixed]
  58 rx-gro-hw: off [fixed]
  59 tls-hw-record: off [fixed]

hping3

hping3 is a network tool able to send custom TCP/IP packets and to display target replies like ping program does with ICMP replies. hping3 handle fragmentation, arbitrary packets body and size and can be used in order to transfer files encapsulated under supported protocols. Using hping3 you are able to perform at least the following stuff:

It's also a good didactic tool to learn TCP/IP. hping3 is developed and maintained by <antirez@invece.org> and is licensed under GPL version 2. Development is open so you can send me patches, suggestion and affronts without inhibitions.

   1 

   1 

iperf

perform network throughput tests

iperf is a tool for performing network throughput measurements. It can test either TCP or UDP throughput. To perform an iperf test the user must establish both a server (to discard traffic) and a client (to generate traffic).

There are 2 tools that call themselves iperf: iperf2 and iperf3 and both don't seem stall in development.

I tend to prefer iperf3.

iperf2

Let's start with the "original"

Hint: Sometimes iperf hangs and CTRL+\ helps.

Install

   1 aptitude install iperf

iperf is based on a client-server principle.

Some defaults:

Start server

   1 # iperf -s
   2 ------------------------------------------------------------
   3 Server listening on TCP port 5001
   4 TCP window size:  128 KByte (default)
   5 ------------------------------------------------------------
   6 [  4] local 192.168.0.12 port 5001 connected with 192.168.0.11 port 37720
   7 [ ID] Interval       Transfer     Bandwidth
   8 [  4]  0.0-10.0 sec  1.10 GBytes   941 Mbits/sec

Client connects to server an outputs some enhanced reporting

   1 % iperf -c remote-host -e
   2 ------------------------------------------------------------
   3 Client connecting to mail1, TCP port 5001 with pid 11333
   4 Write buffer size:  128 KByte
   5 TCP window size: 85.0 KByte (default)
   6 ------------------------------------------------------------
   7 [  3] local 192.168.0.11 port 37720 connected with 192.168.0.12 port 5001
   8 [ ID] Interval        Transfer    Bandwidth       Write/Err  Rtry    Cwnd/RTT
   9 [  3] 0.00-10.00 sec  1.10 GBytes   942 Mbits/sec  8989/0         46      395K/2481 us

Okay we got:

iperf3

Install

   1 aptitude install iperf

Some defaults:

Start server and give more detailed output

   1 # iperf3 -s -V
   2 iperf3 -s -V
   3 iperf 3.6
   4 Linux remote-host 4.19.0-12-amd64 #1 SMP Debian 4.19.152-1 (2020-10-18) x86_64
   5 -----------------------------------------------------------
   6 Server listening on 5201
   7 -----------------------------------------------------------
   8 Time: Tue, 01 Dec 2020 11:35:55 GMT
   9 Accepted connection from 192.168.0.11, port 45606
  10       Cookie: u2x5xtyjazvllw7o67jtojde22xsdxvybe3y
  11       TCP MSS: 0 (default)
  12 [  5] local 192.168.0.12 port 5201 connected to 192.168.0.11 port 45608
  13 Starting Test: protocol: TCP, 1 streams, 131072 byte blocks, omitting 0 seconds, 10 second test, tos 0
  14 [ ID] Interval           Transfer     Bitrate
  15 [  5]   0.00-1.00   sec   107 MBytes   901 Mbits/sec
  16 [  5]   1.00-2.00   sec   112 MBytes   941 Mbits/sec
  17 [  5]   2.00-3.00   sec   112 MBytes   942 Mbits/sec
  18 [  5]   3.00-4.00   sec   112 MBytes   941 Mbits/sec
  19 [  5]   4.00-5.00   sec   112 MBytes   938 Mbits/sec
  20 [  5]   5.00-6.00   sec   112 MBytes   942 Mbits/sec
  21 [  5]   6.00-7.00   sec   112 MBytes   941 Mbits/sec
  22 [  5]   7.00-8.00   sec   112 MBytes   941 Mbits/sec
  23 [  5]   8.00-9.00   sec   112 MBytes   942 Mbits/sec
  24 [  5]   9.00-10.00  sec   112 MBytes   941 Mbits/sec
  25 [  5]  10.00-10.04  sec  4.52 MBytes   941 Mbits/sec
  26 - - - - - - - - - - - - - - - - - - - - - - - - -
  27 Test Complete. Summary Results:
  28 [ ID] Interval           Transfer     Bitrate
  29 [  5] (sender statistics not available)
  30 [  5]   0.00-10.04  sec  1.10 GBytes   937 Mbits/sec                  receiver
  31 CPU Utilization: local/receiver 5.1% (1.1%u/4.0%s), remote/sender 0.0% (0.0%u/0.0%s)
  32 rcv_tcp_congestion cubic
  33 iperf 3.6
  34 Linux remote-host 4.19.0-12-amd64 #1 SMP Debian 4.19.152-1 (2020-10-18) x86_64
  35 -----------------------------------------------------------
  36 Server listening on 5201
  37 -----------------------------------------------------------
  38 ^Ciperf3: interrupt - the server has terminated

Client connects to server an outputs some enhanced reporting

   1 % iperf3 -c remote-host -V
   2 iperf3 -c mail1 -V        
   3 iperf 3.6
   4 Linux hostname 4.19.0-12-amd64 #1 SMP Debian 4.19.152-1 (2020-10-18) x86_64
   5 Control connection MSS 1448
   6 Time: Tue, 01 Dec 2020 11:35:55 GMT
   7 Connecting to host remote-host, port 5201
   8       Cookie: u2x5xtyjazvllw7o67jtojde22xsdxvybe3y
   9       TCP MSS: 1448 (default)
  10 [  5] local 192.168.0.11 port 45608 connected to 192.168.0.12 port 5201
  11 Starting Test: protocol: TCP, 1 streams, 131072 byte blocks, omitting 0 seconds, 10 second test, tos 0
  12 [ ID] Interval           Transfer     Bitrate         Retr  Cwnd
  13 [  5]   0.00-1.00   sec   114 MBytes   956 Mbits/sec    0    437 KBytes       
  14 [  5]   1.00-2.00   sec   113 MBytes   944 Mbits/sec    0    460 KBytes       
  15 [  5]   2.00-3.00   sec   112 MBytes   940 Mbits/sec    0    460 KBytes       
  16 [  5]   3.00-4.00   sec   112 MBytes   940 Mbits/sec    0    460 KBytes       
  17 [  5]   4.00-5.00   sec   111 MBytes   934 Mbits/sec   11    386 KBytes       
  18 [  5]   5.00-6.00   sec   113 MBytes   947 Mbits/sec    0    402 KBytes       
  19 [  5]   6.00-7.00   sec   112 MBytes   938 Mbits/sec    0    443 KBytes       
  20 [  5]   7.00-8.00   sec   112 MBytes   939 Mbits/sec    0    445 KBytes       
  21 [  5]   8.00-9.00   sec   112 MBytes   940 Mbits/sec    0    445 KBytes       
  22 [  5]   9.00-10.00  sec   113 MBytes   949 Mbits/sec    0    447 KBytes       
  23 - - - - - - - - - - - - - - - - - - - - - - - - -
  24 Test Complete. Summary Results:
  25 [ ID] Interval           Transfer     Bitrate         Retr
  26 [  5]   0.00-10.00  sec  1.10 GBytes   943 Mbits/sec   11             sender
  27 [  5]   0.00-10.04  sec  1.10 GBytes   937 Mbits/sec                  receiver
  28 CPU Utilization: local/sender 3.0% (0.3%u/2.7%s), remote/receiver 5.1% (1.1%u/4.0%s)
  29 snd_tcp_congestion cubic
  30 rcv_tcp_congestion cubic
  31 
  32 iperf Done.

Seems to be more informative.

My Traceroute (mtr)

https://www.bitwizard.nl/mtr/

mtr combines the functionality of the 'traceroute' and 'ping' programs in a single network diagnostic tool.

As mtr starts, it investigates the network connection between the host mtr runs on and a user-specified destination host.

Install

   1 aptitude install mtr

A graphical live updating traceroute with some statistics.

   1 mtr hostname.domain.tld

A text live updating traceroute with some statistics. mtr -t hostname.domain.tld

   1                                      My traceroute  [v0.94]
   2 abcd.efghi.rockstable.org (192.168.182.16) -> www.rockstable.it         2020-11-12T15:44:46+0100
   3 Keys:  Help   Display mode   Restart statistics   Order of fields   quit
   4                                                         Packets               Pings
   5  Host                                                 Loss%   Snt   Last   Avg  Best  Wrst StDev
   6  1. abcd.efghi.rockstable.org                          0.0%    41    0.3   0.2   0.2   0.5   0.0
   7  2. ipABCDEFHI.dynamic.kabel-deutschland.de            0.0%    41    7.7  10.8   5.7  39.1   6.9
   8  3. 83-169-181-254-isp.superkabel.de                   0.0%    41    6.4   8.3   6.0  18.5   1.9
   9  4. ip5886c0f1.static.kabel-deutschland.de             0.0%    41    7.9   8.9   6.1  15.3   2.0
  10  5. 145.254.3.68                                       0.0%    41    8.1   8.8   6.1  14.2   1.9
  11  6. 145.254.2.179                                     39.0%    41   15.5  16.3  14.3  20.8   1.7
  12  7. 145.254.2.179                                     35.9%    40   16.1  16.5  13.7  21.6   2.2
  13  8. decix2-gw.hetzner.com                              0.0%    40   18.9  15.5  12.5  33.7   3.8
  14  9. core24.fsn1.hetzner.com                           94.9%    40   21.7  21.6  21.5  21.7   0.1
  15 10. ex9k1.dc14.fsn1.hetzner.com                        0.0%    40   25.3  19.2  17.2  25.3   1.9
  16 11. kvm2.rockstable.org                                0.0%    40   21.5  21.4  18.9  28.9   2.4
  17 12. www2.rockstable.it                                 0.0%    40   22.3  20.8  18.4  26.5   1.9

netstat

Install netstat

   1 aptitude install net-tools

Some important commands

   1 ### LISTENING PORTS
   2 % sudo netstat -tulpen
   3 [sudo] Passwort für tobias: 
   4 Aktive Internetverbindungen (Nur Server)
   5 Proto Recv-Q Send-Q Local Address           Foreign Address         State       Benutzer   Inode      PID/Program name    
   6 tcp        0      0 0.0.0.0:445             0.0.0.0:*               LISTEN      0          37224      3237/smbd           
   7 tcp        0      0 0.0.0.0:2049            0.0.0.0:*               LISTEN      0          30994      -                   
   8 tcp        0      0 0.0.0.0:3142            0.0.0.0:*               LISTEN      125        57443      1851/apt-cacher-ng  
   9 tcp        0      0 127.0.0.1:3306          0.0.0.0:*               LISTEN      135        54106      1915/mariadbd       
  10 tcp        0      0 0.0.0.0:139             0.0.0.0:*               LISTEN      0          37225      3237/smbd           
  11 tcp        0      0 0.0.0.0:9102            0.0.0.0:*               LISTEN      0          61478      2297/bareos-fd      
  12 tcp        0      0 192.168.122.1:53        0.0.0.0:*               LISTEN      0          31348      4064/dnsmasq        
  13 tcp        0      0 192.168.101.1:53        0.0.0.0:*               LISTEN      0          72039      3926/dnsmasq        
  14 tcp        0      0 192.168.100.1:53        0.0.0.0:*               LISTEN      0          50833      3797/dnsmasq        
  15 tcp        0      0 0.0.0.0:22              0.0.0.0:*               LISTEN      0          39934      1863/sshd: /usr/sbi 
  16 tcp        0      0 127.0.0.1:631           0.0.0.0:*               LISTEN      0          53186      1852/cupsd          
  17 tcp        0      0 127.0.0.1:5432          0.0.0.0:*               LISTEN      136        40049      1928/postgres       
  18 tcp        0      0 127.0.0.1:25            0.0.0.0:*               LISTEN      0          72985      3688/master         
  19 
  20 
  21 ### KERNEL INTERFACE TABLE
  22 netstat -ian
  23 Kernel-Schnittstellentabelle
  24 Iface      MTU    RX-OK RX-ERR RX-DRP RX-OVR    TX-OK TX-ERR TX-DRP TX-OVR Flg
  25 bond0     1500  1915555      0   1122 0       1029535      0      0      0 BMmRU
  26 bridge    1500  1912759      0  50477 0       1028122      0      0      0 BMRU
  27 enp8s0    1500  1915555      0      0 0       1029535      0      0      0 BMsRU
  28 enp9s0    1500        0      0      0 0             0      0      0      0 BMU
  29 lo       65536    58342      0      0 0         58342      0      0      0 LRU
  30 
  31 
  32 ### SHOW TCP/UDP SOCKETS (INCLUDING WAITING)
  33 netstat -tuna

ping

missing capabilities

Quelle Debian-Forum

It is totally unneccessary to run ping with sudo.

Check ping capabilities

   1 getcap /bin/ping

ping capabilities should be

   1 /bin/ping = cap_net_raw+ep

add capability net_raw

   1 setcap cap_net_raw+ep /bin/ping

socat

tcpdump

Incredibly useful tool. May be combined with wireshark when used to write a pcap-dump -w.

Some filter keywords to me remembered:

dump_multi.sh

tcpdump misses an option to display the interface the packet was received on. Here is a little wrapper script that simply starts multiple tcpdumps and prefixes the output with the interface name.

The script was mainly copied from this thread on
serverfault - how to display interface in tcpdump output flow

/usr/local/sbin/dump_multi.sh

   1 #!/bin/bash
   2 
   3 SELF="$(basename "$0")"
   4 
   5 declare -a INTERFACES
   6 
   7 ###ADD A STOP MARK TO THE POSITIONAL PARAMETERS
   8 STOPMARK="$(uuidgen)"
   9 set -- "$@" "$STOPMARK"
  10 
  11 usage () {
  12         cat <<-EOF
  13                 $SELF [Options]
  14                     Options:
  15                         -h|--help               Show this page
  16                         -i|--interface <arg>    Dump interface <arg>
  17 
  18                 Options of $SELF mask options of tcpdump.
  19                 Options of tcpdump are not documented here.
  20         EOF
  21 }
  22 
  23 while true; do
  24         case "$1" in
  25                 '-h'|'--help')
  26                         usage
  27                         shift
  28                         continue
  29                 ;;
  30                 '-i'|'--interface')
  31                         INTERFACES+=( "$2" ) 
  32                         shift 2 
  33                         continue
  34                 ;;
  35                 ### BREAK OPTION PARSING AFTER ONE FULL ITERATION
  36                 "$STOPMARK")
  37                         shift
  38                         break
  39                 ;;
  40                 *)
  41                         ### APPEND UNKNOWN OPTION TO THE END OF THE LIST
  42                         TMP1="$1"
  43                         shift
  44                         set -- "$@" "$TMP1"
  45                         unset TMP1
  46                 ;;
  47         esac
  48 done
  49 
  50 
  51 ### When this exits, exit all background processes:
  52 trap 'kill $(jobs -p) &> /dev/null && sleep 0.2 &&  echo ' EXIT
  53 
  54 ### Create one tcpdump output per interface and
  55 ### add an identifier to the beginning of each line:
  56 if [ "${#INTERFACES[@]}" -eq 1 ] \ 
  57         && [ "${INTERFACES[0]}" = "any" ];
  58 then
  59     for IFACE in $(ip l \
  60             |grep '^[0-9]:' \
  61             |grep ',UP' \
  62             |awk '{print $2}' \
  63             |sed 's/://')
  64     do
  65        tcpdump -l -i "$IFACE" -nn "$@" \ 
  66                |sed 's/^/[Iface: '"$IFACE"']    /' & 
  67     done
  68 elif [ "${#INTERFACES[@]}" -ge 1 ]; then
  69         for IFACE in "${INTERFACES[@]}"; do
  70                 tcpdump -l -i "$IFACE" "$@" \ 
  71                         |sed 's/^/[Iface: '"$IFACE"'] /' & 
  72         done
  73 fi
  74 
  75 # wait for CTRL+C
  76 wait

Use it like

   1 ### ON EVERY INTERFACE THAT IS UP
   2 dump_multi.sh -i any 'port …'
   3 ### ON A LIST OF INTERFACES
   4 dump_multi.sh -i lo -i eth0 -i eth1 -i bond0 -i br0 'port …'

Example: Dump DHCP/BOOTP traffic on a bridge, the attached bond and its slaves and determine the traffic flow.

   1 ./dump_multi.sh \
   2         -i bond0 -i enp8s0 -i enp9s0 -i bridge \
   3         'port (bootps or bootpc)'
   4 tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
   5 listening on bridge, link-type EN10MB (Ethernet), capture size 262144 bytes
   6 tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
   7 listening on bond0, link-type EN10MB (Ethernet), capture size 262144 bytes
   8 tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
   9 listening on enp9s0, link-type EN10MB (Ethernet), capture size 262144 bytes
  10 tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
  11 listening on enp8s0, link-type EN10MB (Ethernet), capture size 262144 bytes
  12 [Iface: bridge] 08:26:31.531753 IP 0.0.0.0.bootpc > 255.255.255.255.bootps: BOOTP/DHCP, Request from 52:54:00:d1:b6:9b (oui Unknown), length 300
  13 [Iface: bridge] 08:26:31.555936 IP 0.0.0.0.bootpc > 255.255.255.255.bootps: BOOTP/DHCP, Request from 52:54:00:d1:b6:9b (oui Unknown), length 300
  14 [Iface: bridge] 08:26:34.392529 IP 0.0.0.0.bootpc > 255.255.255.255.bootps: BOOTP/DHCP, Request from c0:d2:f3:e1:fb:7b (oui Unknown), length 315
  15 [Iface: bridge] 08:26:34.973608 IP 0.0.0.0.bootpc > 255.255.255.255.bootps: BOOTP/DHCP, Request from 52:54:00:d1:b6:9b (oui Unknown), length 300

whois

QUERY LIMIT

When experimenting. don't be to curious or you will be banned for a day.

   1 # whois -i nserver '195.201.246.253'                 
   2 % This is the RIPE Database query service.
   3 % The objects are in RPSL format.
   4 %
   5 % The RIPE Database is subject to Terms and Conditions.
   6 % See http://www.ripe.net/db/support/db-terms-conditions.pdf
   7 
   8 %ERROR:201: access denied for IP.ADD.RE.SS
   9 %
  10 % Queries from your IP address have passed the daily limit of controlled objects.
  11 % Access from your host has been temporarily denied.
  12 % For more information, see
  13 % http://www.ripe.net/data-tools/db/faq/faq-db/why-did-you-receive-the-error-201-access-denied
  14 
  15 % This query was served by the RIPE Database Query Service version 1.97.2 (HEREFORD)

Get information about server

Query server information

   1 # QUERY SUPPORTED TYPES
   2 whois -q types
   3 % This is the RIPE Database query service.
   4 % The objects are in RPSL format.
   5 %
   6 % The RIPE Database is subject to Terms and Conditions.
   7 % See http://www.ripe.net/db/support/db-terms-conditions.pdf
   8 
   9 inetnum
  10 inet6num
  11 as-block
  12 aut-num
  13 as-set
  14 route
  15 route6
  16 route-set
  17 inet-rtr
  18 filter-set
  19 peering-set
  20 rtr-set
  21 domain
  22 poetic-form
  23 poem
  24 mntner
  25 irt
  26 key-cert
  27 organisation
  28 role
  29 person
  30 
  31 % This query was served by the RIPE Database Query Service version 1.97.2 (BLAARKOP)
  32 
  33 # QUERY SERVER VERSION
  34 whois -q version
  35 % This is the RIPE Database query service.
  36 % The objects are in RPSL format.
  37 %
  38 % The RIPE Database is subject to Terms and Conditions.
  39 % See http://www.ripe.net/db/support/db-terms-conditions.pdf
  40 
  41 % whois-server-1.97.2
  42 % This query was served by the RIPE Database Query Service version 1.97.2 (BLAARKOP)
  43 
  44 # QUERY SERVER SOURCES
  45 whois -q sources

Querying

You can query the templates and the inverse keys with -t TYPE

   1 whois -t domain 
   2 % This is the RIPE Database query service.
   3 % The objects are in RPSL format.
   4 %
   5 % The RIPE Database is subject to Terms and Conditions.
   6 % See http://www.ripe.net/db/support/db-terms-conditions.pdf
   7 
   8 domain:         [mandatory]  [single]     [primary/lookup key]
   9 descr:          [optional]   [multiple]   [ ]
  10 org:            [optional]   [multiple]   [inverse key]
  11 admin-c:        [mandatory]  [multiple]   [inverse key]
  12 tech-c:         [mandatory]  [multiple]   [inverse key]
  13 zone-c:         [mandatory]  [multiple]   [inverse key]
  14 nserver:        [mandatory]  [multiple]   [inverse key]
  15 ds-rdata:       [optional]   [multiple]   [inverse key]
  16 remarks:        [optional]   [multiple]   [ ]
  17 notify:         [optional]   [multiple]   [inverse key]
  18 mnt-by:         [mandatory]  [multiple]   [inverse key]
  19 created:        [generated]  [single]     [ ]
  20 last-modified:  [generated]  [single]     [ ]
  21 source:         [mandatory]  [single]     [ ]
  22 
  23 % This query was served by the RIPE Database Query Service version 1.97.2 (BLAARKOP)
  24 
  25 ### OUTPUT THE TEMPLATE MORE VERBOSE
  26 whois -v domain
  27 % This is the RIPE Database query service.                                                                                                        
  28 % The objects are in RPSL format.                                                                                                                 
  29 %                                                                                                                                                 
  30 % The RIPE Database is subject to Terms and Conditions.                                                                                           
  31 % See http://www.ripe.net/db/support/db-terms-conditions.pdf                                                                                      
  32                                                                                                                                                   
  33 The domain class:                                                                                                                                 
  34                                                                                                                                                   
  35       A domain object represents a Top Level Domain (TLD) or                                                                                      
  36       other domain registrations. It is also used for Reverse                                                                                     
  37       Delegations.                                                                                                                                
  38                                                                                                                                                   
  39 domain:         [mandatory]  [single]     [primary/lookup key]                                                                                    
  40 descr:          [optional]   [multiple]   [ ]                                                                                                     
  41 org:            [optional]   [multiple]   [inverse key]                                                                                           
  42 admin-c:        [mandatory]  [multiple]   [inverse key]                                                                                           
  43 tech-c:         [mandatory]  [multiple]   [inverse key]                                                                                           
  44 zone-c:         [mandatory]  [multiple]   [inverse key]                                                                                           
  45 nserver:        [mandatory]  [multiple]   [inverse key]                                                                                           
  46 ds-rdata:       [optional]   [multiple]   [inverse key]                                                                                           
  47 remarks:        [optional]   [multiple]   [ ]                                                                                                     
  48 notify:         [optional]   [multiple]   [inverse key]                                                                                           
  49 mnt-by:         [mandatory]  [multiple]   [inverse key]                                                                                           
  50 created:        [generated]  [single]     [ ]                                                                                                     
  51 last-modified:  [generated]  [single]     [ ]                                                                                                     
  52 source:         [mandatory]  [single]     [ ]
  53                                                                          
  54 The content of the attributes of the domain class are defined below:
  55                                                                          
  56 domain
  57                                                                                                                                                   
  58    Domain name.                                                                                                                                   
  59                                     
  60      Domain name as specified in RFC 1034 (point 5.2.1.2) with or
  61      without trailing dot (".").  The total length should not exceed
  62      254 characters (octets).
  63 
  64 descr               
  65 
  66    A short description related to the object.
  67 
  68      A sequence of ASCII characters. 
  69 
  70 

Techniques

Bonding

Buy switches that support LACP and MC-LAG!

Link aggregation

Not to be confused with IEEE 801.3ad (QinQ).

A link aggregation group (LAG) is the collection of physical ports combined together.

Within the IEEE specification, the Link Aggregation Control Protocol (LACP) provides a method to control the bundling of several physical ports together to form a single logical channel. LACP allows a network device to negotiate an automatic bundling of links by sending LACP packets to the peer (directly connected device that also implements LACP).

Linux bondig driver mode 4 802.3ad

Creates aggregation groups that share the same speed and duplex settings. Utilizes all slave network interfaces in the active aggregator group according to the 802.3ad specification. This mode is similar to the XOR mode above and supports the same balancing policies. The link is set up dynamically between two LACP-supporting peers.

LACP Features and practical examples
  1. Maximum number of bundled ports allowed in the port channel: Valid values are usually from 1 to 8.
  2. LACP packets are sent with multicast group MAC address

    01:80:c2:00:00:02 (01-80-c2-00-00-02)

  3. During LACP detection period
    • LACP packets are transmitted every second
    • Keep-alive mechanism for link member: (default: slow = 30s, fast=1s)
  4. LACP can have the port-channel load-balance mode
    • link (link-id) Integer that identifies the member link for load balancing. The range is from 1 to 8 and the load balancing mode can be set-up based on traffic models.[9]
  5. LACP mode
    • Active: Enables LACP unconditionally.
    • Passive: Enables LACP only when an LACP device is detected. (This is the default state)

Advantages over static configuration
  1. Failover occurs automatically: When a link fails and there is (for example) a media converter between the devices, a peer system will not perceive any connectivity problems. With static link aggregation, the peer would continue sending traffic down the link causing the connection to fail.

  2. Dynamic configuration: The device can confirm that the configuration at the other end can handle link aggregation. With Static link aggregation, a cabling or configuration mistake could go undetected and cause undesirable network behavior.

Practical notes

LACP works by sending frames (LACPDUs) down all links that have the protocol enabled. If it finds a device on the other end of the link that also has LACP enabled, it will also independently send frames along the same links enabling the two units to detect multiple links between themselves and then combine them into a single logical link. LACP can be configured in one of two modes: active or passive. In active mode it will always send LACPDUs along the configured links. In passive mode, however, it only reacts as "speak when spoken to", and therefore can be used as a way of controlling accidental loops (as long as the other device is in active mode).

In addition to the IEEE link aggregation substandards, there are a number of proprietary aggregation schemes including Cisco's EtherChannel and Port Aggregation Protocol, Juniper's Aggregated Ethernet, AVAYA's Multi-Link Trunking, Split Multi-Link Trunking, Routed Split Multi-Link Trunking and Distributed Split Multi-Link Trunking, ZTE's "Smartgroup", Huawei's "Eth-Trunk", or Connectify's Speedify. Most high-end network devices support some kind of link aggregation, and software-based implementations – such as the *BSD lagg package, Linux bonding driver, Solaris dladm aggr, etc. – also exist for many operating systems.

Limitations

Single Switch

Same Link Speed and Duplex

Ethernet aggregation mismatch

A multi-chassis link aggregation group (MLAG or MC-LAG) is a type of link aggregation group (LAG) with constituent ports that terminate on separate chassis, primarily for the purpose of providing redundancy in the event one of the chassis fails. The IEEE 802.1AX-2008 industry standard for link aggregation does not mention MC-LAG, but does not preclude it. Its implementation varies by vendor; notably, the protocol existing between the chassis is proprietary.

Advantages over LAG

Bonding with ifupdown and ifenslave

It's recommended to configure bonding
via iproute2 (netlink) or sysfs,
the old ifenslave control utility is obsolete.

Make sure to not interfere with Network-Manager. :-)

During the initial creation it is recommended to monitor the bond in a tmux session with

While the setup fails prepare a individual statement to reset the state to a clean starting point.

   1 for IFACE in bond0 enp8s0 enp9s0; do
   2         ifdown "$IFACE"
   3         ip l set down dev "$IFACE";
   4 done
   5 ip a del 192.168.182.16/24 dev enp8s0
   6 ip a del 192.168.182.208/24 dev enp9s0

Bonding with ifupdown requires the package ifenslave.

   1 apt install ifenslave

There is a bit of documentation in
/usr/share/doc/ifenslave/README.Debian.gz

The examples are found at
/usr/share/doc/ifenslave/examples

Whenever possible use LACP IEEE 802.3ad.

If using lacp you should set the rate to fast (every 1 second). The default is slow (every 30 seconds).

   1 # This file describes the network interfaces available on your system
   2 # and how to activate them. For more information, see interfaces(5).
   3 
   4 source /etc/network/interfaces.d/*
   5 
   6 # The loopback network interface
   7 auto lo
   8 iface lo inet loopback
   9 
  10 auto bond0
  11 iface bond0 inet dhcp
  12         bond-slaves     enp8s0 enp9s0
  13         bond-miimon     100
  14         mode            802.3ad
  15         lacp_rate       fast
  16 
  17 auto enp8s0
  18 iface enp8s0 inet manual
  19 
  20 auto enp9s0
  21 iface enp9s0 inet manual

This config is for LACP agnostic switches with apdaptive load-balancing.

/etc/network/interfaces

   1 # This file describes the network interfaces available on your system
   2 # and how to activate them. For more information, see interfaces(5).
   3 
   4 source /etc/network/interfaces.d/*
   5 
   6 # The loopback network interface
   7 auto lo
   8 iface lo inet loopback
   9 
  10 auto bond0
  11 iface bond0 inet dhcp
  12         bond-primary    enp8s0 enp9s0
  13         bond-slaves     enp8s0 enp9s0
  14         bond-miimon     100
  15         bond-mode       balance-alb

Bonding with Network-Manager

Well, i had huge trouble configuring a lacp bond with NetworkManager. Actually it didn't even work. Whereas ifupdown was trivial and ready in seconds on first try … I'm not yet convinced of NetworkManager.

It's easy to convert ifupdown config to Network-Manager in nm-connecttion-editor.

For every interface

  1. open the interface to be converted
  2. mark the interface to be started automatically
  3. enable auto-negotiation
  4. save and exit the interface

The auto-generated file from /run/NetworkManager/system-connections will be saved in /etc/NetworkManager/system-connections.

Unfortunately the option autoconnect-slaves=1 does not activate the slaves of the bonds. So it's important that the slave interfaces are marked to be started automatically (autoconnect=false is missing or autoconnect=true) or the bond won't come up. On the down-side the bond will also get up on boot even if the bond interface is marked as autoconnect=false.

/etc/NetworkManager/system-connections/bond0.nmconnection

   1 [connection]
   2 id=bond0
   3 uuid=9d6f2839-75ce-4b40-9f1a-2ca925af6dfc
   4 type=bond
   5 interface-name=bond0
   6 permissions=
   7 timestamp=1604305217
   8 
   9 [bond]
  10 downdelay=0
  11 miimon=100
  12 mode=balance-alb
  13 updelay=0
  14 
  15 [ipv4]
  16 dns-priority=100
  17 dns-search=
  18 method=auto
  19 
  20 [ipv6]
  21 addr-gen-mode=stable-privacy
  22 address1=fd93:56fb:daf7:0:2d8:61ff:fe2e:7979/64
  23 dns-priority=100
  24 dns-search=
  25 ip6-privacy=0

The default of Ethernet auto-negotiation is false.

/etc/NetworkManager/system-connections/enp8s0.nmconnection

   1 [connection]
   2 id=enp8s0
   3 uuid=1fc1fa53-f44e-48eb-8904-8cfd6a80950f
   4 type=ethernet
   5 interface-name=enp8s0
   6 master=bond0
   7 permissions=
   8 slave-type=bond
   9 timestamp=1604305049
  10 
  11 [ethernet]
  12 auto-negotiate=true
  13 mac-address=00:D8:61:2E:79:79
  14 mac-address-blacklist=

/etc/NetworkManager/system-connections/enp9s0.nmconnection

   1 [connection]
   2 id=enp9s0
   3 uuid=bb8eac2e-3b3f-4b96-8b49-72dc331ca521
   4 type=ethernet
   5 interface-name=enp9s0
   6 master=bond0
   7 permissions=
   8 slave-type=bond
   9 timestamp=1604305049
  10 
  11 [ethernet]
  12 auto-negotiate=true
  13 mac-address=00:D8:61:2E:79:7A
  14 mac-address-blacklist=

Bonding on VMware

Bridging

Bridging with Network-Manager

The following script is mainly a condensed form of Christophers Blog Articel on How to create bridges on bonds with and without vlans using networkmanager

Create a bridge with the following script

   1 #!/bin/bash
   2 
   3 ### DEFINE BRIDGE
   4 BRIDGE=bridge
   5 BRIDGE_STP=no
   6 #BRIDGE_MTU=1500
   7 
   8 nmcli con add ifname "$BRIDGE" type bridge con-name "$BRIDGE"
   9 nmcli con modify "$BRIDGE" bridge.stp "$BRIDGE_STP"
  10 #nmcli con modify "$BRIDGE" 802-3-ethernet.mtu "$BRIDGE_MTU"
  11 
  12 ### DEFINE BONDsetup8
  13 BOND=bond0
  14 #BOND_SLAVE0=enp0s8
  15 #BOND_SLAVE1=enp0s8
  16 #BOND_MODE=active-backup
  17 #BOND_MTU=9000
  18 
  19 #nmcli con add type bond ifname "${BOND}" con-name "${BOND}"
  20 #nmcli con modify "${BOND}" bond.options mode="${BOND_MODE}"
  21 #nmcli con modify "${BOND}" 802-3-ethernet.mtu "${BOND_MTU}"
  22 #nmcli con add type ethernet con-name "${BOND}-slave-${BOND_SLAVE0}" ifname "${BOND_SLAVE0}" master "${BOND}"
  23 #nmcli con add type ethernet con-name "${BOND}-slave-${BOND_SLAVE1}" ifname "${BOND_SLAVE1}" master "${BOND}"
  24 #nmcli con modify "${BOND}-slave-${BOND_SLAVE0}" 802-3-ethernet.mtu "${BOND_MTU}"
  25 #nmcli con modify "${BOND}-slave-${BOND_SLAVE1}" 802-3-ethernet.mtu "${BOND_MTU}"
  26 
  27 ### ADD bond0 To bridge
  28 nmcli con modify "${BOND}" master "${BRIDGE}" slave-type bridge
  29 
  30 ### SHOW SOME INFO
  31 nmcli con
  32 ls /sys/class/net/bridge/brif/
  33 brctl show

Next is just a note and is yet unconfirmed. There may be some bonding modes, that don't work well with linux bridges, especially balance-alb.

VM on Bridge

I had some trouble getting traffic of a VM attached to a bridge over the enslaved bond. Here are my notes.

The following line allows traffic, but it's not the solution, only a hint to a the problem. Don't run your server with an uninitialized iptables stack in conjuction with other nftables hooks.

   1 echo 0 > /proc/sys/net/bridge/bridge-nf-call-iptables

According to kernel.org doc ip-sysctl

   1 bridge-nf-call-iptables - BOOLEAN
   2 
   3         1 : pass bridged IPv4 traffic to iptables’ chains.
   4         0 : disable this.
   5 
   6     Default: 1

This seems to mean that #nftables is circumvented, which is the default nowadays in Linux.

I wasn't aware that there is such an extensive nftables ruleset without having configured anything.
nft list ruleset

It turns out that the FORWARD policy of ipv4 traffic is set to drop.
nft list chain ip filter FORWARD

   1 table ip filter {
   2         chain FORWARD {
   3                 type filter hook forward priority filter; policy drop;
   4                 counter packets 1095 bytes 92945 jump DOCKER-USER
   5                 counter packets 1095 bytes 92945 jump DOCKER-ISOLATION-STAGE-1
   6                 oifname "docker0" ct state related,established counter packets 0 bytes 0 accept
   7                 oifname "docker0" counter packets 0 bytes 0 jump DOCKER
   8                 iifname "docker0" oifname != "docker0" counter packets 0 bytes 0 accept
   9                 iifname "docker0" oifname "docker0" counter packets 0 bytes 0 accept
  10                 counter packets 1095 bytes 92945 jump LIBVIRT_FWX
  11                 counter packets 1095 bytes 92945 jump LIBVIRT_FWI
  12                 counter packets 1095 bytes 92945 jump LIBVIRT_FWO
  13         }
  14 }

When setting this policy to accept the traffic of virtual machine passes. So this is the first runtime fix.

   1 nft chain ip filter FORWARD '{policy accept;}'

I still need a permanent solution. And i need to figure out, who set this policy to drop. I'll start by disabling docker at boot time and reboot.

   1 systemctl disable docker

And i was right - after reboot
nft list chain ip filter FORWARD

   1 table ip filter {
   2         chain FORWARD {
   3                 type filter hook forward priority filter; policy accept;
   4                 counter packets 0 bytes 0 jump LIBVIRT_FWX
   5                 counter packets 0 bytes 0 jump LIBVIRT_FWI
   6                 counter packets 0 bytes 0 jump LIBVIRT_FWO
   7         }
   8 }

The VM got a IP via DHCP.

Attempt 1

  1. Revert bridge-nf-call-iptables to 1 and flushed nftables ruleset nft flush ruleset. And it also worked.

  2. Restart nftables.service and it still worked.

  3. Restart libvirtd.service and it still worked.

  4. Restart docker.service and it still worked.

    • Docker adds rules to nftables.
    • Did not change the ipv4 forward policy to drop,

      because libvirt already enabled ip_forward.

  5. Stuck - I only got one more confirmation, that the nftables stack is responsible.

Attempt 2

  1. Rebooted and it fails.
  2. Stopped docker.service - still fails

    • Docker does not remove its rules from nftables.
    • IPv4 forward policy is set to drop.
  3. Disabled docker.service and rebooted - works.

  4. Started docker.service - still works.

    • Docker adds rules to nftables.
    • Did not change the ipv4 forward policy to drop,

      because libvirt already enabled ip_forward.

  5. Double check this by enabling and rebooting to exclude interference with the "runtime fix" from above.
    • Confirmed Docker sets IPv4 forwarding policy to drop. :-(

Related info

How docker reasons this behaviour

The FORWARD chain policy is set to DROP by docker since 1.13. As of writing this I'm currently using docker.it 19.03.13+dfsg1-3 from Debian Bullseye.

Docker needs relies on package forwarding to make containers reachable.

If it's not enabled when bringing up the docker interfaces, docker enables forwarding and is by this reason also responsible for any security implications that arise from enabling forwarding and thus sets default policy to DROP. This may break other 3rd-party applications or VMs on the system, but it at least does not impose security threads.

I tend to agree with this idea. Other software should have configured their firewall rules as well.

Solution

In the end it's race condition between docker (which would set net.ipv4.ip_forward = 1 and set nf policy drop; unless net.ipv4.ip_forward was not enabled by docker) and libvirtd (which would set net.ipv4.ip_forward = 1).

A quick and permanent solution by this reason is enabling forwarding before bringing up docker (or anything else).
Linux#IPv4_Forwarding]

VLAN Subinterfaces

VLAN Subinterfaces with ifupdown

Can't be easier.

/etc/network/interfaces

   1 auto bond0.32
   2 iface bond0.32 inet dhcp
   3 
   4 auto bond0.40
   5 iface bond0.40 inet dhcp

nftables

Default rule set in
/etc/nftables.conf

   1 #!/usr/sbin/nft -f
   2 
   3 flush ruleset
   4 
   5 table inet filter {
   6         chain input {
   7                 type filter hook input priority 0;
   8         }
   9         chain forward {
  10                 type filter hook forward priority 0;
  11         }
  12         chain output {
  13                 type filter hook output priority 0;
  14         }
  15 }

Executable: /usr/sbin/nft

Important commands

   1 nft list ruleset
   2 nft flush ruleset
   3 nft add table inet filter
   4 nft add chain inet filter input { type filter hook input priority 0 \; policy drop }
   5 nft add rule inet filter input ct state established counter accept
   6 nft list ruleset

Hardware

linux-firmware_dl.sh

Download the firmware recursively and move firmware to its path in filesystem.

/usr/local/sbin/linux-firmware_dl.sh

   1 #!/bin/bash
   2 
   3 [ "$1" ] && GIT_DIR="$1" || exit 1
   4 [ "$2" ] && GIT_FILTER="$2"
   5 
   6 DIR_TMP_BASE="$(mktemp -d)"
   7 DIR_TMP="$DIR_TMP_BASE/$GIT_DIR"
   8 DIR_FW="/lib/firmware/$GIT_DIR"
   9 URL_GIT='https://git.kernel.org/pub/scm/linux/kernel/git/firmware/linux-firmware.git/plain'
  10 
  11 mkdir -p "$DIR_TMP"
  12 cd "$DIR_TMP" || exit 1
  13 wget -r -nd -np -e robots=off \
  14         -A README -A "$GIT_FILTER*.bin" \
  15         "$URL_GIT/$GIT_DIR/"
  16 #wget --recursive --no-directories --no-parent -e robots=off \
  17 #       -A README -A "$GIT_FILTER*.bin" \
  18 #       "$URL_GIT/$GIT_DIR/"
  19 ls -1 "$DIR_TMP"/*.tmp > /dev/null 2>&1 \
  20         && rm "$DIR_TMP"/*.tmp
  21 cd ..
  22 [ -d "$DIR_FW" ] || mkdir "$DIR_FW"
  23 sudo mv "$DIR_TMP"/* "$DIR_FW"
  24 sudo chown -R 0.0 "$DIR_FW"
  25 sudo find "$DIR_FW" -type d -exec chmod 0755 {} \;
  26 sudo find "$DIR_FW" -type f -exec chmod 0644 {} \;
  27 rm -rv "$DIR_TMP_BASE"

Use it like

   1 chmod u+x /usr/local/sbin/linux-firmware_dl.sh
   2 ### linux-firmware_dl.sh $SUBDIR $PREFIX
   3 linux-firmware_dl.sh rtw88
   4 ### OR FOR LATEST VEGA64 FIRMWARE
   5 #linux.firmware_dl.sh amdgpu vega10_
   6 

Looks like this

   1 ll -d /lib/firmware/rtw88/*
   2 -rw-r--r-- 1 root root   1087 Mai 26 14:14 /lib/firmware/rtw88/README
   3 -rw-r--r-- 1 root root  28884 Mai 26 14:14 /lib/firmware/rtw88/rtw8723d_fw.bin
   4 -rw-r--r-- 1 root root 137896 Mai 26 14:14 /lib/firmware/rtw88/rtw8821c_fw.bin
   5 -rw-r--r-- 1 root root 150984 Mai 26 14:14 /lib/firmware/rtw88/rtw8822b_fw.bin
   6 -rw-r--r-- 1 root root 189152 Mai 26 14:14 /lib/firmware/rtw88/rtw8822c_fw.bin
   7 -rw-r--r-- 1 root root 138720 Mai 26 14:14 /lib/firmware/rtw88/rtw8822c_wow_fw.bin

Debian RTL8822BE/RTL8822CE

So you need the firmware rtw88/rtw8822b_fw.bin

Debian Bug #945172 suggests to link the files.

I think it's better to get the most recent version from
git.kernel.org linux-firmware

You may use the following script to download rtw88 directory to /lib/firmware
#linux-firmware_dl.sh

Install the latest backports-kernel and reboot or just update your initramfs
update-initramfs -k all -u

I guess your WiFi works now.

And one day … Debian's firmware-packages will be refreshed and overwrite the contents of this manually created directory. ;-)

Working WiFi with rtw88 on Linux 5.6
lspci -vvs 04:00.0

   1 04:00.0 Network controller: Realtek Semiconductor Co., Ltd. RTL8822BE 802.11a/b/g/n/ac WiFi adapter
   2         Subsystem: Lenovo ThinkPad E595
   3         Physical Slot: 0
   4         Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx+
   5         Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
   6         Latency: 0, Cache Line Size: 32 bytes
   7         Interrupt: pin A routed to IRQ 80
   8         Region 0: I/O ports at 2000 [size=256]
   9         Region 2: Memory at d0600000 (64-bit, non-prefetchable) [size=64K]
  10         Capabilities: [40] Power Management version 3
  11                 Flags: PMEClk- DSI- D1+ D2+ AuxCurrent=375mA PME(D0+,D1+,D2+,D3hot+,D3cold+)
  12                 Status: D0 NoSoftRst+ PME-Enable- DSel=0 DScale=0 PME-
  13         Capabilities: [50] MSI: Enable+ Count=1/1 Maskable- 64bit+
  14                 Address: 00000000fee00000  Data: 0000
  15         Capabilities: [70] Express (v2) Endpoint, MSI 00
  16                 DevCap: MaxPayload 128 bytes, PhantFunc 0, Latency L0s <4us, L1 <64us
  17                         ExtTag- AttnBtn- AttnInd- PwrInd- RBE+ FLReset- SlotPowerLimit 0.000W
  18                 DevCtl: CorrErr- NonFatalErr- FatalErr- UnsupReq-
  19                         RlxdOrd+ ExtTag- PhantFunc- AuxPwr- NoSnoop-
  20                         MaxPayload 128 bytes, MaxReadReq 512 bytes
  21                 DevSta: CorrErr- NonFatalErr- FatalErr- UnsupReq- AuxPwr+ TransPend-
  22                 LnkCap: Port #0, Speed 2.5GT/s, Width x1, ASPM L0s L1, Exit Latency L0s <2us, L1 <64us
  23                         ClockPM+ Surprise- LLActRep- BwNot- ASPMOptComp-
  24                 LnkCtl: ASPM L1 Enabled; RCB 64 bytes Disabled- CommClk+
  25                         ExtSynch- ClockPM+ AutWidDis- BWInt- AutBWInt-
  26                 LnkSta: Speed 2.5GT/s (ok), Width x1 (ok)
  27                         TrErr- Train- SlotClk+ DLActive- BWMgmt- ABWMgmt-
  28                 DevCap2: Completion Timeout: Not Supported, TimeoutDis+, NROPrPrP-, LTR+
  29                          10BitTagComp-, 10BitTagReq-, OBFF Via message/WAKE#, ExtFmt-, EETLPPrefix-
  30                          EmergencyPowerReduction Not Supported, EmergencyPowerReductionInit-
  31                          FRS-, TPHComp-, ExtTPHComp-
  32                          AtomicOpsCap: 32bit- 64bit- 128bitCAS-
  33                 DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis-, LTR+, OBFF Disabled
  34                          AtomicOpsCtl: ReqEn-
  35                 LnkCtl2: Target Link Speed: 5GT/s, EnterCompliance- SpeedDis-
  36                          Transmit Margin: Normal Operating Range, EnterModifiedCompliance- ComplianceSOS-
  37                          Compliance De-emphasis: -6dB
  38                 LnkSta2: Current De-emphasis Level: -3.5dB, EqualizationComplete-, EqualizationPhase1-
  39                          EqualizationPhase2-, EqualizationPhase3-, LinkEqualizationRequest-
  40         Capabilities: [100 v2] Advanced Error Reporting
  41                 UESta:  DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
  42                 UEMsk:  DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
  43                 UESvrt: DLP+ SDES+ TLP- FCP+ CmpltTO- CmpltAbrt- UnxCmplt- RxOF+ MalfTLP+ ECRC- UnsupReq- ACSViol-
  44                 CESta:  RxErr- BadTLP- BadDLLP- Rollover- Timeout- AdvNonFatalErr-
  45                 CEMsk:  RxErr- BadTLP- BadDLLP- Rollover- Timeout- AdvNonFatalErr+
  46                 AERCap: First Error Pointer: 00, ECRCGenCap+ ECRCGenEn- ECRCChkCap+ ECRCChkEn-
  47                         MultHdrRecCap- MultHdrRecEn- TLPPfxPres- HdrLogCap-
  48                 HeaderLog: 00000000 00000000 00000000 00000000
  49         Capabilities: [148 v1] Device Serial Number 00-e0-4c-ff-fe-b8-22-01
  50         Capabilities: [158 v1] Latency Tolerance Reporting
  51                 Max snoop latency: 1048576ns
  52                 Max no snoop latency: 1048576ns
  53         Capabilities: [160 v1] L1 PM Substates
  54                 L1SubCap: PCI-PM_L1.2+ PCI-PM_L1.1+ ASPM_L1.2+ ASPM_L1.1+ L1_PM_Substates+
  55                           PortCommonModeRestoreTime=30us PortTPowerOnTime=60us
  56                 L1SubCtl1: PCI-PM_L1.2- PCI-PM_L1.1+ ASPM_L1.2- ASPM_L1.1+
  57                            T_CommonMode=0us LTR1.2_Threshold=0ns
  58                 L1SubCtl2: T_PwrOn=60us
  59         Kernel driver in use: rtw_pci
  60         Kernel modules: rtwpci

modinfo rtwpci rtw88

   1 filename:       /lib/modules/5.6.0-1-amd64/kernel/drivers/net/wireless/realtek/rtw88/rtwpci.ko
   2 license:        Dual BSD/GPL
   3 description:    Realtek 802.11ac wireless PCI driver
   4 author:         Realtek Corporation
   5 alias:          pci:v000010ECd0000C822sv*sd*bc*sc*i*
   6 alias:          pci:v000010ECd0000B822sv*sd*bc*sc*i*
   7 depends:        mac80211,rtw88
   8 retpoline:      Y
   9 intree:         Y
  10 name:           rtwpci
  11 vermagic:       5.6.0-1-amd64 SMP mod_unload modversions 
  12 sig_id:         PKCS#7
  13 signer:         Debian Secure Boot CA
  14 sig_key:        A7:46:8D:EF
  15 sig_hashalgo:   sha256
  16 signature:      79:B8:A3:7B:68:2D:CD:38:76:CB:48:1C:56:D4:20:77:7D:97:3C:24:
  17                 F5:BE:84:25:31:34:EE:27:03:F8:13:41:49:BD:09:E3:A7:09:86:CB:
  18                 91:50:0A:E0:3F:CA:19:CC:2A:AF:56:CE:D0:A2:4C:E0:83:C6:8F:71:
  19                 C0:E3:A2:68:BA:F6:50:F6:FC:10:76:E4:08:94:65:33:37:0A:56:9C:
  20                 C3:F9:AF:97:FA:30:7F:10:7A:47:81:28:F2:79:B5:79:7F:AE:F6:58:
  21                 6F:E2:6B:F6:78:8C:9D:89:37:26:67:3A:57:ED:03:16:79:26:EA:D2:
  22                 91:D5:F0:8B:1C:4D:CC:56:97:EA:3D:4F:45:5F:B7:54:C2:26:08:71:
  23                 A1:01:FF:A9:7E:2F:61:CF:C2:A8:DA:1C:1B:2C:D3:60:4C:D6:53:1E:
  24                 00:8D:3A:09:14:BB:7A:A7:27:8C:E4:BB:C9:40:85:EB:FE:0B:18:0A:
  25                 76:39:F7:9F:70:FB:0B:DB:BA:33:BC:31:0F:C2:75:45:E1:11:1A:B4:
  26                 58:31:6E:26:CC:45:AE:AC:4D:67:5B:DE:CC:08:D8:01:49:D9:71:E8:
  27                 25:6C:C5:E8:DF:F7:DE:64:CE:34:00:5F:7A:3D:E6:8D:77:28:FD:BB:
  28                 6A:E5:83:41:61:46:0F:73:C7:21:F9:90:2F:5A:6D:93
  29 parm:           disable_msi:Set Y to disable MSI interrupt support (bool)
  30 
  31 filename:       /lib/modules/5.6.0-1-amd64/kernel/drivers/net/wireless/realtek/rtw88/rtw88.ko
  32 license:        Dual BSD/GPL
  33 description:    Realtek 802.11ac wireless core module
  34 author:         Realtek Corporation
  35 firmware:       rtw88/rtw8822b_fw.bin
  36 firmware:       rtw88/rtw8822c_wow_fw.bin
  37 firmware:       rtw88/rtw8822c_fw.bin
  38 depends:        mac80211,cfg80211
  39 retpoline:      Y
  40 intree:         Y
  41 name:           rtw88
  42 vermagic:       5.6.0-1-amd64 SMP mod_unload modversions 
  43 sig_id:         PKCS#7
  44 signer:         Debian Secure Boot CA
  45 sig_key:        A7:46:8D:EF
  46 sig_hashalgo:   sha256
  47 signature:      AD:30:FA:52:72:54:79:79:FC:7B:8A:52:92:19:F5:30:91:CD:F2:13:
  48                 00:8A:FD:8D:11:B2:94:FA:DB:4E:BF:B7:32:5D:EB:71:C6:27:81:34:
  49                 87:D9:59:7F:8F:32:6F:E6:2F:AF:F9:8F:EF:E2:E1:FF:39:EE:AD:EB:
  50                 BF:13:9C:CE:9A:F6:72:3A:8E:27:91:E4:98:60:48:4C:36:84:3E:90:
  51                 01:4D:4A:BA:7C:5E:D5:7B:7F:C0:3F:74:1C:C7:04:04:EC:9D:5D:55:
  52                 D6:CE:AE:2C:F6:8E:37:94:83:1B:D2:6D:34:17:DA:59:B0:57:68:6C:
  53                 A3:E6:5A:2D:3E:2D:FB:EA:C0:08:E1:0C:DE:64:1C:84:17:75:CD:C1:
  54                 0F:C5:C4:CE:97:E1:24:2E:57:F1:B8:EF:9E:8B:B0:C7:99:B6:1C:1D:
  55                 4D:AE:49:DE:BD:3B:40:65:74:C5:C8:DF:96:C2:40:DC:7B:23:6E:73:
  56                 20:52:E4:DF:E2:C1:86:D3:F0:C6:B4:6D:5E:12:97:09:EE:82:A6:5F:
  57                 E6:E0:69:95:9A:98:69:B8:F5:48:12:2F:4A:BB:5B:FD:3E:63:0E:A7:
  58                 D8:40:A8:55:E5:07:E8:81:EF:5E:36:3E:38:6F:D9:A5:75:BE:6E:D5:
  59                 F5:70:C2:AD:F5:4F:94:D4:D4:29:68:E6:31:FA:8D:6E
  60 parm:           lps_deep_mode:Deeper PS mode. If 0, deep PS is disabled (uint)
  61 parm:           support_bf:Set Y to enable beamformee support (bool)
  62 parm:           debug_mask:Debugging mask (uint)

firewalld

A service daemon with D-Bus interface

Install the dynamic firewall configuration daemon

   1 apt install firewalld

Install the graphical configuration tool and systray applet

   1 apt install firewall-config firewall-applet

Gather some info

   1 ### AVAILABLE SERVICES
   2 root@ipa2 ~ # firewall-cmd --get-services |fold -s
   3 RH-Satellite-6 amanda-client amanda-k5-client amqp amqps apcupsd audit bacula 
   4 bacula-client bb bgp bitcoin bitcoin-rpc bitcoin-testnet bitcoin-testnet-rpc 
   5 bittorrent-lsd ceph ceph-mon cfengine cockpit condor-collector ctdb dhcp dhcpv6 
   6 dhcpv6-client distcc dns dns-over-tls docker-registry docker-swarm 
   7 dropbox-lansync elasticsearch etcd-client etcd-server finger freeipa-4 
   8 freeipa-ldap freeipa-ldaps freeipa-replication freeipa-trust ftp ganglia-client 
   9 ganglia-master git grafana gre high-availability http https imap imaps ipp 
  10 ipp-client ipsec irc ircs iscsi-target isns jenkins kadmin kdeconnect kerberos 
  11 kibana klogin kpasswd kprop kshell kube-apiserver ldap ldaps libvirt 
  12 libvirt-tls lightning-network llmnr managesieve matrix mdns memcache minidlna 
  13 mongodb mosh mountd mqtt mqtt-tls ms-wbt mssql murmur mysql nfs nfs3 nmea-0183 
  14 nrpe ntp nut openvpn ovirt-imageio ovirt-storageconsole ovirt-vmconsole plex 
  15 pmcd pmproxy pmwebapi pmwebapis pop3 pop3s postgresql privoxy prometheus 
  16 proxy-dhcp ptp pulseaudio puppetmaster quassel radius rdp redis redis-sentinel 
  17 rpc-bind rsh rsyncd rtsp salt-master samba samba-client samba-dc sane sip sips 
  18 slp smtp smtp-submission smtps snmp snmptrap spideroak-lansync spotify-sync 
  19 squid ssdp ssh steam-streaming svdrp svn syncthing syncthing-gui synergy syslog 
  20 syslog-tls telnet tentacle tftp tftp-client tile38 tinc tor-socks 
  21 transmission-client upnp-client vdsm vnc-server wbem-http wbem-https wsman 
  22 wsmans xdmcp xmpp-bosh xmpp-client xmpp-local xmpp-server zabbix-agent 
  23 zabbix-server
  24 root@ipa2 ~ # firewall-cmd --get-default-zone
  25 public
  26 root@ipa2 ~ # firewall-cmd --permanent --info-zone=public
  27 public
  28   target: default
  29   icmp-block-inversion: no
  30   interfaces:
  31   sources:
  32   services: cockpit dhcpv6-client ssh
  33   ports:
  34   protocols:
  35   masquerade: no
  36   forward-ports:
  37   source-ports:
  38   icmp-blocks:
  39   rich rules:
  40 root@ipa2 ~ # firewall-cmd --permanent --list-services
  41 cockpit dhcpv6-client ssh

Open the firewall for freeipa

   1 root@ipa2 ~ # firewall-cmd --add-service=freeipa-4
   2 success
   3 root@ipa2 ~ # firewall-cmd --permanent --add-service=freeipa-4
   4 success
   5 root@ipa2 ~ # firewall-cmd --permanent --list-services
   6 cockpit dhcpv6-client freeipa-4 ssh
   7 root@ipa2 ~ # firewall-cmd --permanent --service=freeipa-4 --get-includes
   8 http https kerberos kpasswd ldap ldaps
   9 root@ipa2 ~ # firewall-cmd --permanent --service=ldap --get-ports
  10 389/tcp
  11 
  12 root@ipa2 ~ # firewall-cmd --permanent --service=freeipa-replication --get-ports
  13 7389/tcp
  14 root@ipa2 ~ # firewall-cmd --permanent --add-service=freeipa-replication
  15 success

Open the firewall for DNS queries

   1 root@ipa2 ~ # firewall-cmd --add-service=dns
   2 success
   3 root@ipa2 /usr/lib64/bind # firewall-cmd --permanent --add-service=dns
   4 success

Test if it worked! It works when it's tested sucessfully! :-D

Network-Manager

About

Please see

Ecosystem

Authorization with policy kit

If you are in group sudo or netdev, you are allowed to modify system-connections.

/usr/share/polkit-1/rules.d/60-network-manager.rules

   1 polkit.addRule(function(action, subject) {
   2   if (action.id == "org.freedesktop.NetworkManager.settings.modify.system" &&
   3         subject.local && subject.active && 
   4         (subject.isInGroup ("sudo") || subject.isInGroup ("netdev"))) {
   5     return polkit.Result.YES;
   6   }
   7 });

Do not manage interfaces

You can configure Network-Manager to not manage the interfaces mentioned in
/etc/network/interfaces

   1 # This file describes the network interfaces available on your system
   2 # and how to activate them. For more information, see interfaces(5).
   3 
   4 source /etc/network/interfaces.d/*
   5 
   6 # The loopback network interface
   7 auto lo
   8 iface lo inet loopback
   9 
  10 auto enp1s0
  11 iface enp1s0 inet dhcp
  12 
  13 #auto enp2s0
  14 iface enp2s0 inet dhcp

Make sure you don't remove the line entirely or Network-Manager will grab the interface and spawn a dhclient for the interface.

/etc/NetworkManager/NetworkManager.conf

   1 [main]
   2 plugins=ifupdown,keyfile
   3 
   4 [ifupdown]
   5 managed=false

Here is the corresponding section from
man 5 NetworkManager.conf

   1 IFUPDOWN SECTION
   2        This section contains ifupdown-specific options and thus
   3        only has effect when using the ifupdown plugin.
   4 
   5        managed
   6            If set to true, then interfaces listed in
   7            /etc/network/interfaces are managed by
   8            NetworkManager. If set to false, then any interface
   9            listed in /etc/network/interfaces will be ignored by
  10            NetworkManager. Remember that NetworkManager
  11            controls the default route, so because the interface
  12            is ignored, NetworkManager may assign the default
  13            route to some other interface.
  14 
  15            The default value is false.

Manage interfaces

Something to remember …

Network-Manager controls the default route.

If you try to connect to a VPN with the default route assigned by a interface that is unmanaged by Network-Manager (Network-Manager does not control the default route), the following error message is logged.

   1 Oct 30 17:27:14 libertas NetworkManager[116637]: <info>  [1604075234.9052] audit: op="connection-activate" uuid="6bcc9142-242a-44c9-adff-d30601f41919" name="openvpn_connection" pid=5284 uid=1000 result="fail" reason="Could not find source connection."

When setting to managed=true,
the following error is logged activating bond0.

   1 # nmcli connection  up Ifupdown\ \(bond0\) 
   2 Fehler: Aktivierung der Verbindung ist gescheitert: No suitable device found for this connection (device docker0 not available because profile is not compatible with device (mismatching interface name)).

In ifupdown-managed mode Network-Manager automatically creates configurations for the devices listes in /etc/network/interfaces. These files are located in
ll /run/NetworkManager/system-connections

   1 -rw------- 1 root root 393 30. Oct 18:50 bond0.nmconnection
   2 -rw------- 1 root root 433 30. Oct 12:19 docker0.nmconnection
   3 -rw------- 1 root root 304 30. Oct 18:40 enp8s0.nmconnection
   4 -rw------- 1 root root 304 30. Oct 18:40 enp9s0.nmconnection
   5 -rw------- 1 root root 438 30. Oct 12:19 virbr0.nmconnection

The bonding configuration is not bad at all but is also not functionial. So i converted the dynamically generated config to a permanent config and adjusted it. Please see #Bonding with Network-Manager. Now that NM controls the default route, activating/… VPNs works simply fine!

Imported VPNs

Network-Manager imports various VPN-profiles, but does not set the file permissions on the private key tight enough.

NM stores certificates in subdirectories below
~/.local/share/networkmanagement/certificates/. Make sure you have the correct selinux context on this files, too.

   1 find ~/.local/share/networkmanagement/certificates/ \
   2         -type f -name private.key -exec chmod 600 {} \;

OpenVPN

   1 

Common network setups

single points of failures (SPOF)

Setup 1

attachment:setup1.svg

Setup 2

attachment:setup2.svg

Setup 3

attachment:setup3.svg

Setup 4

attachment:setup4.svg

Setup 5

attachment:setup5.svg

Setup 6

attachment:setup6.svg

Setup 7

attachment:setup7.svg

Setup 8

attachment:setup8.svg

Recommended modern approach

GNS3

Install

Install dependencies

   1 apt update
   2 apt install \
   3         python3-pip python3-pyqt5 python3-pyqt5.qtsvg \
   4         python3-pyqt5.qtwebsockets \
   5         qemu qemu-system-x86 qemu-utils libvirt-clients libvirt-daemon-system virtinst \
   6         wireshark xtightvncviewer apt-transport-https \
   7         ca-certificates curl gnupg2 software-properties-common

dynamips - Cisco 7200/3600/3725/3745/2600/1700 Router Emulator and

uBridge is a simple application to create user-land bridges between various technologies. Currently bridging between UDP tunnels, Ethernet and TAP interfaces is supported. Packet capture is also supported.

Import public key

   1 apt-key adv \
   2         --keyserver keyserver.ubuntu.com \
   3         --recv-keys F88F6D313016330404F710FC9A2FD067A2E3EF7B

/etc/apt/sources.list.d/gns3.list

   1 deb     http://ppa.launchpad.net/gns3/ppa/ubuntu focal main
   2 deb-src http://ppa.launchpad.net/gns3/ppa/ubuntu focal main
   3 
   4 deb     http://ppa.launchpad.net/gns3/ppa/ubuntu groovy main
   5 deb-src http://ppa.launchpad.net/gns3/ppa/ubuntu groovy main

/etc/apt/preferences.d/gns3

   1 Package:        *
   2 Pin:            origin ppa.launchpad.net
   3 Pin-Priority:   101

Install ubridge and dynamips (adjust ist to use the most recent version)

   1 aptitude install \
   2         gns3-server gns3-gui gns3-webclient-pack \
   3         dynamips ubridge

Join the groups

   1 for GROUP in kvm libvirt docker ubridge wireshark; do
   2         adduser tobias $GROUP;
   3 done

Configuration

Change default console

To allow resizing the font, i thought about changing the terminal emulator.

Edit > Preferences General > Console Applications (TAB)

   1 ### DEFAULT
   2 xterm -T "%d" -e "telnet %h %p"
   3 ## PLASMA KONSOLE
   4 konsole -e "telnet %h %p"

Rockstable Wiki: networking (last edited 2021-04-09 08:57:40 by RockStable)