#acl ContribGroup:read,write,revert,delete Default

= networking =

<<TableOfContents(4)>>

 * [[https://ieeexplore.ieee.org/browse/standards/get-program/page/series?id=68|IEEE Standards]]


== Trouble Shooting ==

In non-deterministic cases you may resort to<<BR>>
[[https://tools.ietf.org/html/rfc2321|IETF RFC 2321 - RITA -- The Reliable Internetwork Troubleshooting Agent]]


== Models OSI and TCP/IP ==

[[attachment:OSI.dot.svg|{{attachment:OSI.dot.svg}}]]


== Data Link ==


=== Address Resolution Protocol (ARP) ===

 * [[https://en.wikipedia.org/wiki/Address_Resolution_Protocol|Wiki EN Address Resolution Protocol]]
 * [[https://tools.ietf.org/html/rfc826|IETF RFC826 An Ethernet Address Resolution Protocol]]
 * [[https://tools.ietf.org/html/rfc5227|IETF RFC5227 IPv4 Address Conflict Detection]]
 * [[https://tools.ietf.org/html/rfc5494|IETF RFC5494 IANA Allocation Guidelines for the Address Resolution Protocol (ARP)]]

The Address Resolution Protocol (ARP) is a communication protocol
used for discovering the link layer address, such as a MAC address,
associated with a given internet layer address, typically an IPv4 address.
This mapping is a critical function in the Internet protocol suite.
ARP was defined in 1982 by RFC 826, which is Internet Standard STD 37. 

==== Notes ====

Old D-Link DGS-series switches (like DGS-1210-16 Rev. A1)
seam to fail ARP in combination with CARP/VRRP gateways.
In such cases the default gateway cannot be resolved
to a MAC-address and routing fails generally.
ARP resolution within the same network may still work.
This may be diagnosed using {{{telnet}}} and
the command {{{debug info}}}.
The MAC from the CARP/VRRP address is only in the MAC-forwarding table,
but missing in the host arp-table (mgmtVlan).

# WIP - exact feature has to be determined

=== Neighbor Discovery Protocol (NDP) ===

 * [[https://en.wikipedia.org/wiki/Neighbor_Discovery_Protocol|Wiki EN Neighbor Discovery Protocol]]
 * [[https://tools.ietf.org/html/rfc4861|IETF RFC4861 Neighbor Discovery for IP version 6 (IPv6)]]

=== MAC addresses ===

 * [[https://www.iana.org/assignments/ethernet-numbers/ethernet-numbers.xhtml|IANA Ethernet Numbers]]
 * [[https://tools.ietf.org/html/rfc7042#section-2.1.1|IETF RFC7042 IANA Considerations and IETF Protocol and Documentation Usage for IEEE 802 Parameters #EUI-48 Assignments under the IANA OUI]]
 * [[https://tools.ietf.org/html/rfc5798#page-28|IETF RFC5798 Virtual Router Redundancy Protocol (VRRP) Version 3 for IPv4 and IPv6 #Virtual Router MAC Address]]
 * [[https://www.iana.org/assignments/ieee-802-numbers/ieee-802-numbers.xhtml|IANA IEEE 802 Numbers]]
   * [[https://www.iana.org/assignments/ieee-802-numbers/ieee-802-numbers.xhtml#ieee-802-numbers-1|IANA ETHER TYPES]]
   * [[https://www.iana.org/assignments/ieee-802-numbers/ieee-802-numbers.xhtml#ieee-802-numbers-2|IANA ORGANIZATIONALLY UNIQUE IDENTIFIERS]]
   * [[https://www.iana.org/assignments/ieee-802-numbers/ieee-802-numbers.xhtml#ieee-802-numbers-3|Logical Link Control (LLC) Numbers]]
   * [[https://www.iana.org/assignments/ieee-802-numbers/ieee-802-numbers.xhtml#iana-lldp-tlv-subtypes|IANA Link Layer Discovery Protocol (LLDP) TLV Subtypes]]
 * [[]]

EUI-48 (Extended Unique Identifier 48)

3-octet OUI (Organizationally Unique Identifier)


==== Local database ====

{{{#!highlight bash
aptitude install ieee-data
}}}

{{{#!highlight bash
/usr/share/ieee-data/iab.csv
/usr/share/ieee-data/iab.txt
/usr/share/ieee-data/mam.csv
/usr/share/ieee-data/mam.txt
/usr/share/ieee-data/oui.csv
/usr/share/ieee-data/oui.txt ### <-- MAC-ADRESSESS
/usr/share/ieee-data/oui36.csv
/usr/share/ieee-data/oui36.txt

/usr/share/nmap/nmap-mac-prefixes
}}}

=== Spanning Tree Protocol ===

==== Algorhyme ====

Radia Perlman penned this poem
while she developed Spanning Tree.

{{{
I think that I shall never see
A graph more lovely than a tree.
A tree whose crucial property
Is loop-free connectivity.
A tree that must be sure to span
So packets can reach every LAN.
First, the root must be selected.
By ID, it is elected.
Least-cost paths from root are traced.
In the tree, these paths are placed.
A mesh is made by folks like me,
Then bridges find a spanning tree.

--
Radia Perlman 
}}}

==== STP ====

 * [[https://en.wikipedia.org/wiki/Spanning_Tree_Protocol|Wiki EN Spanning Tree Protocol]]
 * [[https://standards.ieee.org/standard/802_1D-2004.html|IEEE 802.1D-2004 - IEEE Standard for Local and metropolitan area networks: Media Access Control (MAC) Bridges]]

Standards
 * IEEE 802.1d-1990 - STP
 * IEEE 802.1w-2003 - RSTP
   * later IEEE 802.1D-2004
 * IEEE 802.1s-2003 - MSTP
   * later IEEE 802.1Q-2014

Bridge-ID (BID) is 8 Byte long
(2 Byte bridge priority, 6 Byte MAC address).

 Bridge Priority::
 :: is basically a 4bit number and
 locally assigned system ID extension (12 bits)
 :: has only 16 values,
 p * 2^12^, where 0 ≤ p ≤ 2^4^

 Root Bridge::
 :: of the spanning tree is the bridge
 with the smallest (lowest) bridge ID.
 :: There can only be one root bridge in a spanning tree. ;-)
 :: When initializing the protocol
 all bridges send out BPDUs,
 with themselves as root bridge.
 :: After convergence only the root bridge generates BPDUs.
 Other devices only forward BPDUs.
 :: Has no root port.
 :: All the ports on the root bridge are designated ports

 Designated Bridge::
 :: Has exactly one root port
 :: Device responsible to forwards frames to a LAN segment

Bridge Protocol Data Units (BPDUs)
 * are sent
   * from the unique source MAC-address of a switch port
   * to STP destination multicast MAC-address {{{01:80:C2:00:00:00}}}
   or {{{01:00:0C:CC:CC:CD}}} in case of Cisco Per VLAN Spanning Tree (PVST) :-)
 * 2 types
   * Configuration BPDU
   * Topology Change Notification (TCN) BPDU

Port states (4)
 Blocking::
   :: BPDUs are received and processed
   :: Frames are not forwarded
   :: This port would cause a topology loop
   :: May transit to forwarding state on failure of another link
 Listening::
   :: BPDUs are received and processed
   :: Frames are not forwarded
   :: MAC table is not populated
   :: May return to blocking
 Learning::
   :: BPDUs are received and processed
   :: Frames are not forwarded
   :: MAC table is populated
   :: May return to blocking
 Forwarding::
   :: BPDUs are received and processed
   :: Frames are forwarded
   :: May return to blocking
 Disabled::
   :: Not strictly part of STP
   :: Manually disabled switch port

Blocking -> Listening -> Learning -> Forwarding


Port roles (2)
 Root::
   :: A forwarding port that is the best port
   from non-root bridge to root bridge
   :: Port on which a device
   received the optimum configuration BPDU.
 Designated::
   ::  A forwarding port for every LAN segment
 Disabled::
   :: Not strictly part of STP,
   a network administrator can manually disable a port

Timers
 * Hello-Timer Default: 2s
 * Forward-Delay Default: 15s (1x Listening, 1x Learning)
 * Maximum Age Default: 20s

The root bridge sets the timer values and
distributes these in Configuration BPDUs.

When a new device is attached
it takes 2x Forward-Delay timers (default: 30s)
to transit to the state Forwarding.

Path cost
 * calculated on bandwidth
   * The lower bandwidth, the higher cost
     * with STP originally 1Gbit/s devided by bandwidth
     * with RSTP 20Tbit/s devided by bandwidth
 * admin can influence the path cost
 * path cost to the root bridge add up along the path
 (across the switches)
 * if there are multiple upstream/designated bridges
 with equal cost to the root bridge,
 the lower sender bridge-id wins
 * if there are multiple designated ports (no lag)
 to an upstream/designated bridges
 with equal cost to the root bridge,
 the lower designated port-id wins
   * Port ID = priority (4 bits) + ID (Interface number) (12 bits)
     * the default port priority is 128.
     * may be influenced by the admin
 * if this still has equal costs
 finally the local lowest Port ID wins


==== Rapid STP ====

 * Standard IEEE 802.1D-2004 incorporates RSTP
 and obsoletes the original STP standard
 * Backwards-compatible with standard STP
 * Significantly faster convergence
 by introduction of new behaviours and new port roles
   * usually responds to changes within 3 Hello times
 * Handshake between switches to determine if a rapid transition
 to the forwarding state is possible
   1. RTSP bridges propagate their superior root bridge
   information to their designated ports
   1. Receiving bridges
     1. sets all other ports to discarding.
     1. responds to this BPDUs with an BPDU with the agreement flag set.
   1. Sending Bridge now knows,
   that it can transition directly to forwarding state,
   bypassing listening state.
 * Allows configuration and autodetection of edge ports,
 which directly transition to forwarding.
 Edge-ports transition to non-edge ports,
 when BPDUs are detected.


Port roles
 Root::
   :: A forwarding port that is the best port
   from non-root bridge to root bridge
 Designated::
   ::  A forwarding port for every LAN segment
 Alternate::
   :: An alternate path to the root bridge.
   :: This path is different from using the root port.
 Backup::
   :: A backup/redundant path to a segment
   where another bridge port already connects
 Disabled::
   :: Not strictly part of STP,
   a network administrator can manually disable a port


port states (3)
 Discarding::
   :: BPDUs are received and processed
   :: Frames are not forwarded
   :: This port would cause a topology loop
   :: May transit to forwarding state on failure of another link
 Learning::
   :: BPDUs are received and processed
   :: Frames are not forwarded
   :: MAC table is populated
   :: May return to blocking
 Forwarding::
   :: BPDUs are received and processed
   :: Frames are forwarded
   :: May return to blocking
 Disabled::
   :: Not strictly part of STP
   :: manually disabled switch port


Timers
 * Hello-Timer Default: 2s
 * Forward-Delay Default: 15s (1x Listening, 1x Learning)
 * Maximum Age Default: 3x Hello-Timer (6s)

==== Per-VLAN-Spanning Tree ====

 * Cisco proprietary
   * Limited support across switch vendors
   * Compatibility issues between
     * PVSTP(+) vendor implementations
     * compatible protocols like VLAN Spanning Tree Protocol (VSTP)
 * bridge system ID extension carries VLAN ID
 * Based on simple STP
 * One Spanning Tree instance per VLAN
   * Adds significat overhead
   * Consumes CPU-time on the networking device
 * VLAN encapsulation
   * PVSTP uses Inter-Switch Link (ISL)
   * PVSTP+ uses IEEE 802.1q
 * PVST+ can tunnel across an MSTP region.
 * Rapid PVSTP(+) (RPVSTP(+)) is based on RSTP instead.

==== Multiple STP ====

 * Compatible to RSTP and thus to STP.
 * Bridge system ID extension carries the MSTP instance number
 * Allows usage of multiple Multiple spanning tree instances (MSTI)
 * Allows mapping multiple VLANs to a MSTI (based on mapping-tables)
   * Allow load-sharing across links that otherwise would be blocked
   * Reduces number of instances in comparision to PVSTP+
 * A switched network may be devided into multiple regions,
 with independant spanning trees.


 Common Spanning Tree (CST)::
   :: connects all MST regions in a switched network
 Internal Spanning Tree (IST)::
   :: runs in an MST region
   :: also named MSTI 0,
   a special MSTI to which all VLANs are mapped by default
 Common and Internal Spanning Tree (CIST)::
   :: connects all devices in a switched network
   :: consists of the ISTs in all MST regions and the CST
 Regional root::
   :: of the IST or a and MSTI within an MST region
   :: different MSTIs in a MST region
   may have different regional root bridges
 Common root bridge::
   :: root bridge of the CIST

Port roles (6)
 Root port::
   :: forwards data to the root bridge
 Designated port::
   :: forwards data to the designated bridge for a downstream network segment or device
 Boundary port::
   :: port that connects a MST region to
     :: another MST region or
     :: a network-segment running STP, or RSTP
 Master port::
   :: root port (of a region) on the CIST to the common root bridge
 Alternate port::
   :: Backup port for a root port and master port
   :: Does not forward frames
   :: Takes over when the root port or master port has failed
   :: Starts forwarding without delay
 Backup port::
   :: Backup port of a designated port
   :: Starts forwarding without delay
 Disabled port::
   :: admin down in every MSTI

A port may have different roles in different MSTIs.

Port states (3)
 * Discarding 
 * Learning
 * Forwarding

A port may have different states in different MSTIs.


==== Notes ====

'''Never use simple old STP.'''

Hold-down time (50s) is not reconcilable with high-availability.

If STP is in default configuration
(with bridge-priority 32768)
the election is based on the mac address.
The root bridge bridge may be located in a inefficient position,
e.g. far away from the router.
If a STP-protocol is used in a network,
it must be planned and configured carefully!

Different implementations of a standard are not guaranteed to work,
due for example to differences in default timer settings.

Try to use (multi-chassis) link-aggregations ((MC)-LAGs) where possible,
to avoid blocking redundant ports and enhance bandwidth.

Make sure to configure BPDU-guard to protect your network
from malicious bridges.

Edge-ports should be used
 * to allow fast transition of terminal devices.
 *  in conjunction with BPDU-guard.

You may use a BPDU-filter to discard BPDUs
from adjacent switching infrastructures,
when it's clear that a loop can never be established.

Use loop detection on the edge to the network.


== Network ==

=== Common Address Redundancy Protocol (CARP) ===

 * [[https://en.wikipedia.org/wiki/Common_Address_Redundancy_Protocol|Wiki EN Common Address Redundancy Protocol]]

Alternatives: VRRP, HSRP, GLBP, ESRP, R-SMLT, NSRP

Sniff carp 
{{{#!highlight bash
tcpdump -npi vtnet2 -T carp
}}}

=== IP addresses ===

 * [[https://en.wikipedia.org/wiki/Reserved_IP_addresses|Wiki EN Reserved IP addresses]]

In DNS make sure your NS, MX and A records for a given domain
don't share a single IP-address.


=== NetBIOS ===

 * is obsolete.
 * is not strictly a network layer protocol,
 this depends on its implementation.
 * may be implemented as
   * NetBEUI (NetBEUI -> MAC)
   * on top of IPX (NetBIOS -> IPX-address-> MAC)
   * on top of TCP/IP (NetBIOS -> IP-address-> MAC)
 * is limited to a broadcast domain and
 is not routed.
 * causes a high amount of broadcast traffic.

==== Links ====

* [[https://tools.ietf.org/html/rfc1002|IETF RFC1002 PROTOCOL STANDARD FOR A NetBIOS SERVICE ON A TCP/UDP TRANSPORT: DETAILED SPECIFICATIONS]]

==== NetBIOS name ====

 * reservations are broadcasted to the broadcast-address
 followed by a time the requestor listens
 for complains from the network (already registered).
 This procedure is repeated 4 times,
 until the name is considered reserved.
 * are 16Byte long.
 * may be ''unique'' (U) or a ''group'' (G).
   * Please avoid assigning the name as a workgroup to a node.
   This may lead to problems on the node.
 * Microsoft's implementation of NetBIOS uses
 the 16th byte of the NetBIOS Name as a type field,
 which is also known as the NetBIOS-Suffix.

[[http://www.pyeung.com/pages/microsoft/winnt/netbioscodes.html|Patrick P. Yeung NetBIOS Suffix Code Table]]

|| '''Name'''           || '''Number (HEX)''' || '''Type''' || '''Usage''' ||
|| {{{<computername>}}} || {{{00}}} || U || Workstation Service ||
|| {{{<computername>}}} || {{{01}}} || U || Messenger Service ||
|| {{{<computername>}}} || {{{03}}} || U || Messenger Service ||
|| {{{<computername>}}} || {{{06}}} || U || RAS Server Service ||
|| {{{<computername>}}} || {{{1f}}} || U || NetDDE Service ||
|| {{{<computername>}}} || {{{20}}} || U || File Server Service ||
|| {{{<computername>}}} || {{{21}}} || U || RAS Client Service ||
|| {{{<computername>}}} || {{{22}}} || U || Exchange Interchange ||
|| {{{<computername>}}} || {{{23}}} || U || Exchange Store ||
|| {{{<computername>}}} || {{{24}}} || U || Exchange Directory ||
|| {{{<computername>}}} || {{{30}}} || U || Modem Sharing Server Service ||
|| {{{<computername>}}} || {{{31}}} || U || Modem Sharing Client Service ||
|| {{{<computername>}}} || {{{43}}} || U || SMS Client Remote Control ||
|| {{{<computername>}}} || {{{44}}} || U || SMS Admin Remote Control Tool ||
|| {{{<computername>}}} || {{{45}}} || U || SMS Client Remote Chat ||
|| {{{<computername>}}} || {{{46}}} || U || SMS Client Remote Transfer ||
|| {{{<computername>}}} || {{{4c}}} || U || DEC Pathworks TCPIP Service ||
|| {{{<computername>}}} || {{{52}}} || U || DEC Pathworks TCPIP Service ||
|| {{{<computername>}}} || {{{87}}} || U || Exchange MTA ||
|| {{{<computername>}}} || {{{6a}}} || U || Exchange IMC ||
|| {{{<computername>}}} || {{{be}}} || U || Network Monitor Agent ||
|| {{{<computername>}}} || {{{bf}}} || U || Network Monitor Apps ||
|| {{{<username>}}}     || {{{03}}} || U || Messenger Service ||
|| {{{<\\_MSBROWSE_>}}} || {{{01}}} || G || Master Browser ||
|| {{{<domain>}}}       || {{{00}}} || G || Domain Name (Membership) ||
|| {{{<domain>}}}       || {{{1b}}} || U || Domain Master Browser ||
|| {{{<domain>}}}       || {{{1c}}} || G || '''Domain Controllers''', <<BR>> Domain Logon Server ||
|| {{{<domain>}}}       || {{{1d}}} || U || (Local) Master Browser ||
|| {{{<domain>}}}       || {{{1e}}} || G || (Local) Browser Service Elections ||
|| {{{<INet~Services>}}} || {{{1c}}} || G || Internet Information Server ||
|| {{{<IS~Computer_name>}}} || {{{00}}} || U || Internet Information Server ||
||||||||||
|| {{{<computername>}}} || {{{[2b]}}} || U || Lotus Notes Server ||
|| {{{IRISMULTICAST}}}  || {{{[2f]}}} || G || Lotus Notes ||
|| {{{IRISNAMESERVER}}} || {{{[33]}}} || G || Lotus Notes ||
|| {{{Forte_$ND800ZA}}} || {{{[20]}}} || U || DCA Irmalan Gateway Service ||

==== NetBIOS Name resolution order in MS Windows ====

Order of resolution
 1. Cache
   * contains static entries
   from LMHOSTS with the tag {{{#PRE}}}
 1. WINS
 1. Broadcast
 1. {{{LMHOSTS.SAM}}}
 1. DNS (if configured)

==== lmhosts ====

Static NetBIOS name resolution like {{{/etc/hosts}}}

{{{LMHOSTS.SAM}}}
{{{#!highlight bash
192.168.1.11    samba #PRE
#INCLUDE        \\samba\public\lmhosts
}}}

Keywords (case sensitive, labels are case-insensitive):
 * {{{#PRE}}} enforces lookup of hostname in LMHOSTS.SAM
 without resolution over the network.
   * These entries are loaded into the NetBIOS cache on boot.
   * The NetBIOS cache can be flushed and repopulated with {{{nbtstat -R}}}.
 * {{{#INCLUDE}}} allows including another file,
 which may also reside on a share.



==== WINS ====

[[https://docs.microsoft.com/en-us/windows-server/networking/technologies/wins/wins-top|Microsoft Docs - Windows Internet Name Service (WINS)]]

WINS is as obsolete as NetBIOS is.

There can only be one WINS server in a given network or
the NetBIOS namespace is divided
(even with multiple workgroups or domains).
Replicating WINS servers may be used
(Windows or samba4WINS).

WINS uses directed UDP unicast,
which is routed across network boundaries.
Please make sure the firewall is open on {{{udp/137}}}.

Names, which are registered in WINS,
are resolved using directed udp-unicast,
otherwise they are resolved using broadcasts.

WINS works without wait times for complaints and
thus is faster and conserves resources.

Consider using WINS even in small networks to reduce NetBIOS broadcasts.

WINS names are registered on boot of a node
(e.g. in case the WINS server was changed during runtime).

The reservation is only valid for a limited amount of time and
has to be renewed in regular intervals.
The intervals can be chosen by the client,
but the server defines minimum and maximum boundries.

Via dhcp WINS server is distributed as {{{netbios-name-servers}}}.
{{{option netbios-node-type 8; ### DEFAULT}}} may be set
to explicitly define NetBIOS behaviour.

||'''type'''||'''short name'''||'''long name'''||'''resolution'''||
||1 || B-Node || broadcast || broadcast||
||2 || P-Node || point-to-point || WINS||
||4 || M-Node || mixed || 1^st^ broadcast, 2^nd^ WINS||
||8 || H-Node || hybrid || 1^st^ WINS, 2^nd^ broadcast||


== Transport ==

=== Ephemeral Ports ===

 * [[https://en.wikipedia.org/wiki/Ephemeral_port|Wiki EN Ephemeral Ports]]

Range

The Internet Assigned Numbers Authority (IANA) suggests
the range 49152 to 65535 (2^15^ + 2^14^ to 2^16^−1) for dynamic or private ports.

Many Linux kernels use the port range 32768 to 61000.
FreeBSD has used the IANA port range since release 4.6.
Previous versions, including the Berkeley Software Distribution (BSD),
use ports 1024 to 5000 as ephemeral ports.

View and customize ephemeral ports range
{{{#!highlight bash
cat /proc/sys/net/ipv4/ip_local_port_range
32768   60999
sysctl net.ipv4.ip_local_port_range
net.ipv4.ip_local_port_range = 32768    60999
}}}

Microsoft Windows operating systems through XP
use the range 1025–5000 as ephemeral ports by default.
Windows Vista, Windows 7, and Server 2008 use the IANA range by default.
Windows Server 2003 uses the range 1025–5000 by default,
until Microsoft security update MS08-037 from 2008 is installed,
after which it uses the IANA range by default.
Windows Server 2008 with Exchange Server 2007 installed
has a default port range of 1025–60000.
In addition to the default range,
all versions of Windows since Windows 2000
have the option of specifying
a custom range anywhere within 1025–65535.


== Hardware info ==

PCI devices
{{{#!highlight bash
lspci -vv -s 00:1f.6
}}}


== Tools ==

=== dsniff ===

[[https://www.monkey.org/~dugsong/dsniff/]]

dsniff is a collection of tools for network auditing and penetration testing.
dsniff, filesnarf, mailsnarf, msgsnarf, urlsnarf, and webspy
passively monitor a network for interesting data
(passwords, e-mail, files, etc.).
arpspoof, dnsspoof, and macof
facilitate the interception of network traffic
normally unavailable to an attacker (e.g, due to layer-2 switching).
sshmitm and webmitm implement active monkey-in-the-middle attacks
against redirected SSH and HTTPS sessions
by exploiting weak bindings in ad-hoc PKI. 

 * arpspoof  - Send out unrequested (and possibly forged) arp replies.
 * dnsspoof  - forge replies to arbitrary DNS address / pointer queries
 on the Local Area Network.
 * dsniff    - password sniffer for several protocols.
 * filesnarf - saves selected files sniffed from NFS traffic.
 * macof     - flood the local network with random MAC addresses.
 * mailsnarf - sniffs mail on the LAN and stores it in mbox format.
 * msgsnarf  - record selected messages from different Instant Messengers.
 * sshmitm   - SSH monkey-in-the-middle. proxies and sniffs SSH traffic.
 * sshow     - SSH traffic analyser.
 * tcpkill   - kills specified in-progress TCP connections.
 * tcpnice   - slow down specified TCP connections via "active"
 traffic shaping.
 * urlsnarf  - output selected URLs sniffed from HTTP traffic in CLF.
 * webmitm   - HTTP / HTTPS monkey-in-the-middle. transparently proxies.
 * webspy    - sends URLs sniffed from a client to your local browser
 (requires libx11-6 installed).

 Please do not abuse this software.

Install 
{{{#!highlight bash
apt install dsniff
}}}


=== ethtool ===

[[https://www.kernel.org/pub/software/network/ethtool/]]

Display or change Ethernet device settings.
ethtool can be used to query and change settings such as
speed, auto- negotiation and checksum offload
on many network devices, especially Ethernet devices.

==== Gather info ====

Get basic link state
(speed, duplex, MDI-X, link, autoneg …)
{{{#!highlight bash
# ethtool enp0s31f6 
Settings for enp0s31f6:
        Supported ports: [ TP ]
        Supported link modes:   10baseT/Half 10baseT/Full 
                                100baseT/Half 100baseT/Full 
                                1000baseT/Full 
        Supported pause frame use: No
        Supports auto-negotiation: Yes
        Supported FEC modes: Not reported
        Advertised link modes:  10baseT/Half 10baseT/Full 
                                100baseT/Half 100baseT/Full 
                                1000baseT/Full 
        Advertised pause frame use: No
        Advertised auto-negotiation: Yes
        Advertised FEC modes: Not reported
        Speed: 1000Mb/s
        Duplex: Full
        Port: Twisted Pair
        PHYAD: 1
        Transceiver: internal
        Auto-negotiation: on
        MDI-X: on (auto)
        Supports Wake-on: pumbg
        Wake-on: g
        Current message level: 0x00000007 (7)
                               drv probe link
        Link detected: yes
}}}

Queries the specified network device
for associated driver information.
{{{#!highlight bash
ethtool -i enp0s31f6
}}}

Queries the specified network device
for the state of protocol offload and other features.
{{{#!highlight bash
# ethtool -k enp0s31f6
Features for enp0s31f6:
rx-checksumming: on
tx-checksumming: on
        tx-checksum-ipv4: off [fixed]
        tx-checksum-ip-generic: on
        tx-checksum-ipv6: off [fixed]
        tx-checksum-fcoe-crc: off [fixed]
        tx-checksum-sctp: off [fixed]
scatter-gather: on
        tx-scatter-gather: on
        tx-scatter-gather-fraglist: off [fixed]
tcp-segmentation-offload: on
        tx-tcp-segmentation: on
        tx-tcp-ecn-segmentation: off [fixed]
        tx-tcp-mangleid-segmentation: off
        tx-tcp6-segmentation: on
udp-fragmentation-offload: off
generic-segmentation-offload: on
generic-receive-offload: on
large-receive-offload: off [fixed]
rx-vlan-offload: on
tx-vlan-offload: on
ntuple-filters: off [fixed]
receive-hashing: on
highdma: on [fixed]
rx-vlan-filter: off [fixed]
vlan-challenged: off [fixed]
tx-lockless: off [fixed]
netns-local: off [fixed]
tx-gso-robust: off [fixed]
tx-fcoe-segmentation: off [fixed]
tx-gre-segmentation: off [fixed]
tx-gre-csum-segmentation: off [fixed]
tx-ipxip4-segmentation: off [fixed]
tx-ipxip6-segmentation: off [fixed]
tx-udp_tnl-segmentation: off [fixed]
tx-udp_tnl-csum-segmentation: off [fixed]
tx-gso-partial: off [fixed]
tx-sctp-segmentation: off [fixed]
tx-esp-segmentation: off [fixed]
tx-udp-segmentation: off [fixed]
fcoe-mtu: off [fixed]
tx-nocache-copy: off
loopback: off [fixed]
rx-fcs: off
rx-all: off
tx-vlan-stag-hw-insert: off [fixed]
rx-vlan-stag-hw-parse: off [fixed]
rx-vlan-stag-filter: off [fixed]
l2-fwd-offload: off [fixed]
hw-tc-offload: off [fixed]
esp-hw-offload: off [fixed]
esp-tx-csum-hw-offload: off [fixed]
rx-udp_tunnel-port-offload: off [fixed]
tls-hw-tx-offload: off [fixed]
tls-hw-rx-offload: off [fixed]
rx-gro-hw: off [fixed]
tls-hw-record: off [fixed]
}}}


==== Alter ====

Restart auto-negotiation if enabled 
{{{#!highlight bash
ethtool -r eth0
}}}

===== De/activate features of a nic =====

Examples:
 * scatter-gather (sg)
 * TCP-segmentation-offloading (tso)
 * generic-segmentation-offload (gro)

{{{#!highlight bash
# ethtool -K enp0s31f6 sg off tso off gro off
Actual changes:
scatter-gather: off
        tx-scatter-gather: off
tcp-segmentation-offload: off
        tx-tcp-segmentation: off
        tx-tcp6-segmentation: off
generic-segmentation-offload: off [requested on]
generic-receive-offload: off
}}}

Post deactivation
{{{#!highlight bash
# ethtool -k enp0s31f6
Features for enp0s31f6:
rx-checksumming: on
tx-checksumming: on
        tx-checksum-ipv4: off [fixed]
        tx-checksum-ip-generic: on
        tx-checksum-ipv6: off [fixed]
        tx-checksum-fcoe-crc: off [fixed]
        tx-checksum-sctp: off [fixed]
scatter-gather: off
        tx-scatter-gather: off
        tx-scatter-gather-fraglist: off [fixed]
tcp-segmentation-offload: off
        tx-tcp-segmentation: off
        tx-tcp-ecn-segmentation: off [fixed]
        tx-tcp-mangleid-segmentation: off
        tx-tcp6-segmentation: off
udp-fragmentation-offload: off
generic-segmentation-offload: off [requested on]
generic-receive-offload: off
large-receive-offload: off [fixed]
rx-vlan-offload: on
tx-vlan-offload: on
ntuple-filters: off [fixed]
receive-hashing: on
highdma: on [fixed]
rx-vlan-filter: off [fixed]
vlan-challenged: off [fixed]
tx-lockless: off [fixed]
netns-local: off [fixed]
tx-gso-robust: off [fixed]
tx-fcoe-segmentation: off [fixed]
tx-gre-segmentation: off [fixed]
tx-gre-csum-segmentation: off [fixed]
tx-ipxip4-segmentation: off [fixed]
tx-ipxip6-segmentation: off [fixed]
tx-udp_tnl-segmentation: off [fixed]
tx-udp_tnl-csum-segmentation: off [fixed]
tx-gso-partial: off [fixed]
tx-sctp-segmentation: off [fixed]
tx-esp-segmentation: off [fixed]
tx-udp-segmentation: off [fixed]
fcoe-mtu: off [fixed]
tx-nocache-copy: off
loopback: off [fixed]
rx-fcs: off
rx-all: off
tx-vlan-stag-hw-insert: off [fixed]
rx-vlan-stag-hw-parse: off [fixed]
rx-vlan-stag-filter: off [fixed]
l2-fwd-offload: off [fixed]
hw-tc-offload: off [fixed]
esp-hw-offload: off [fixed]
esp-tx-csum-hw-offload: off [fixed]
rx-udp_tunnel-port-offload: off [fixed]
tls-hw-tx-offload: off [fixed]
tls-hw-rx-offload: off [fixed]
rx-gro-hw: off [fixed]
tls-hw-record: off [fixed]
}}}

=== hping3 ===

 * [[http://www.hping.org/]]

hping3 is a network tool able to send custom TCP/IP packets and
to display target replies like ping program does with ICMP replies.
hping3  handle  fragmentation, arbitrary packets body and size and
can be used in order to transfer files encapsulated under supported protocols.
Using hping3 you are able to perform at least the following stuff:

 * Test firewall rules
 * Advanced port scanning
 * Test net performance using different protocols,
 packet size, TOS (type of service) and fragmentation.
 * Path MTU discovery
 * Transferring files between even really fascist firewall rules.
 * Traceroute-like under different protocols.
 * Firewalk-like usage.
 * Remote OS fingerprinting.
 * TCP/IP stack auditing.
 * A lot of others.

It's also a good didactic tool to learn TCP/IP. 
hping3 is developed and maintained by <<MailTo(antirez@invece.org)>> and
is licensed under GPL version 2.
Development is open so you can send me patches,
suggestion and affronts without inhibitions.


{{{}}}
{{{#!highlight bash
}}}

{{{#!highlight bash
}}}


=== iperf ===

perform network throughput tests

iperf is a tool for performing network throughput measurements.
It can test either TCP or UDP throughput.
To perform an iperf test
the user must establish both
a server (to discard traffic) and
a client (to generate traffic).

There are 2 tools that call themselves iperf:
iperf2 and iperf3
and both don't seem stall in development.
 * [[https://sourceforge.net/projects/iperf2/|iperf2 Homepage]]
 * [[http://software.es.net/iperf/|iperf3 Homepage]]

I tend to prefer iperf3.

==== iperf2 ====

Let's start with the "original"

Hint:
Sometimes iperf hangs and {{{CTRL+\}}} helps.

Install
{{{#!highlight bash
aptitude install iperf
}}}

iperf is based on a client-server principle.

Some defaults:
 * time: 10s
 * port: 5001
 * protocol: tcp

Start server
{{{#!highlight bash
# iperf -s
------------------------------------------------------------
Server listening on TCP port 5001
TCP window size:  128 KByte (default)
------------------------------------------------------------
[  4] local 192.168.0.12 port 5001 connected with 192.168.0.11 port 37720
[ ID] Interval       Transfer     Bandwidth
[  4]  0.0-10.0 sec  1.10 GBytes   941 Mbits/sec
}}}

Client connects to server an outputs some enhanced reporting
{{{#!highlight bash
% iperf -c remote-host -e
------------------------------------------------------------
Client connecting to mail1, TCP port 5001 with pid 11333
Write buffer size:  128 KByte
TCP window size: 85.0 KByte (default)
------------------------------------------------------------
[  3] local 192.168.0.11 port 37720 connected with 192.168.0.12 port 5001
[ ID] Interval        Transfer    Bandwidth       Write/Err  Rtry    Cwnd/RTT
[  3] 0.00-10.00 sec  1.10 GBytes   942 Mbits/sec  8989/0         46      395K/2481 us
}}}

Okay we got:
 * Bandwidth
 * Errors
 * Round Trip Time

==== iperf3 ====

Install
{{{#!highlight bash
aptitude install iperf
}}}

Some defaults:
 * time: 10s
 * port: 5201
 * protocol: tcp

Start server and give more detailed output
{{{#!highlight bash
# iperf3 -s -V
iperf3 -s -V
iperf 3.6
Linux remote-host 4.19.0-12-amd64 #1 SMP Debian 4.19.152-1 (2020-10-18) x86_64
-----------------------------------------------------------
Server listening on 5201
-----------------------------------------------------------
Time: Tue, 01 Dec 2020 11:35:55 GMT
Accepted connection from 192.168.0.11, port 45606
      Cookie: u2x5xtyjazvllw7o67jtojde22xsdxvybe3y
      TCP MSS: 0 (default)
[  5] local 192.168.0.12 port 5201 connected to 192.168.0.11 port 45608
Starting Test: protocol: TCP, 1 streams, 131072 byte blocks, omitting 0 seconds, 10 second test, tos 0
[ ID] Interval           Transfer     Bitrate
[  5]   0.00-1.00   sec   107 MBytes   901 Mbits/sec
[  5]   1.00-2.00   sec   112 MBytes   941 Mbits/sec
[  5]   2.00-3.00   sec   112 MBytes   942 Mbits/sec
[  5]   3.00-4.00   sec   112 MBytes   941 Mbits/sec
[  5]   4.00-5.00   sec   112 MBytes   938 Mbits/sec
[  5]   5.00-6.00   sec   112 MBytes   942 Mbits/sec
[  5]   6.00-7.00   sec   112 MBytes   941 Mbits/sec
[  5]   7.00-8.00   sec   112 MBytes   941 Mbits/sec
[  5]   8.00-9.00   sec   112 MBytes   942 Mbits/sec
[  5]   9.00-10.00  sec   112 MBytes   941 Mbits/sec
[  5]  10.00-10.04  sec  4.52 MBytes   941 Mbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
Test Complete. Summary Results:
[ ID] Interval           Transfer     Bitrate
[  5] (sender statistics not available)
[  5]   0.00-10.04  sec  1.10 GBytes   937 Mbits/sec                  receiver
CPU Utilization: local/receiver 5.1% (1.1%u/4.0%s), remote/sender 0.0% (0.0%u/0.0%s)
rcv_tcp_congestion cubic
iperf 3.6
Linux remote-host 4.19.0-12-amd64 #1 SMP Debian 4.19.152-1 (2020-10-18) x86_64
-----------------------------------------------------------
Server listening on 5201
-----------------------------------------------------------
^Ciperf3: interrupt - the server has terminated
}}}


Client connects to server an outputs some enhanced reporting
{{{#!highlight bash
% iperf3 -c remote-host -V
iperf3 -c mail1 -V        
iperf 3.6
Linux hostname 4.19.0-12-amd64 #1 SMP Debian 4.19.152-1 (2020-10-18) x86_64
Control connection MSS 1448
Time: Tue, 01 Dec 2020 11:35:55 GMT
Connecting to host remote-host, port 5201
      Cookie: u2x5xtyjazvllw7o67jtojde22xsdxvybe3y
      TCP MSS: 1448 (default)
[  5] local 192.168.0.11 port 45608 connected to 192.168.0.12 port 5201
Starting Test: protocol: TCP, 1 streams, 131072 byte blocks, omitting 0 seconds, 10 second test, tos 0
[ ID] Interval           Transfer     Bitrate         Retr  Cwnd
[  5]   0.00-1.00   sec   114 MBytes   956 Mbits/sec    0    437 KBytes       
[  5]   1.00-2.00   sec   113 MBytes   944 Mbits/sec    0    460 KBytes       
[  5]   2.00-3.00   sec   112 MBytes   940 Mbits/sec    0    460 KBytes       
[  5]   3.00-4.00   sec   112 MBytes   940 Mbits/sec    0    460 KBytes       
[  5]   4.00-5.00   sec   111 MBytes   934 Mbits/sec   11    386 KBytes       
[  5]   5.00-6.00   sec   113 MBytes   947 Mbits/sec    0    402 KBytes       
[  5]   6.00-7.00   sec   112 MBytes   938 Mbits/sec    0    443 KBytes       
[  5]   7.00-8.00   sec   112 MBytes   939 Mbits/sec    0    445 KBytes       
[  5]   8.00-9.00   sec   112 MBytes   940 Mbits/sec    0    445 KBytes       
[  5]   9.00-10.00  sec   113 MBytes   949 Mbits/sec    0    447 KBytes       
- - - - - - - - - - - - - - - - - - - - - - - - -
Test Complete. Summary Results:
[ ID] Interval           Transfer     Bitrate         Retr
[  5]   0.00-10.00  sec  1.10 GBytes   943 Mbits/sec   11             sender
[  5]   0.00-10.04  sec  1.10 GBytes   937 Mbits/sec                  receiver
CPU Utilization: local/sender 3.0% (0.3%u/2.7%s), remote/receiver 5.1% (1.1%u/4.0%s)
snd_tcp_congestion cubic
rcv_tcp_congestion cubic

iperf Done.
}}}

Seems to be more informative.


=== My Traceroute (mtr) ===

[[https://www.bitwizard.nl/mtr/]]

mtr combines the functionality of the 'traceroute' and 'ping' programs in a single network diagnostic tool.

As mtr starts, it investigates the network connection between the host mtr runs on and a user-specified destination host.

Install
{{{#!highlight bash
aptitude install mtr
}}}

A graphical live updating traceroute with some statistics.
{{{#!highlight bash
mtr hostname.domain.tld
}}}

A text live updating traceroute with some statistics.
{{{mtr -t hostname.domain.tld}}}
{{{#!highlight bash
                                     My traceroute  [v0.94]
abcd.efghi.rockstable.org (192.168.182.16) -> www.rockstable.it         2020-11-12T15:44:46+0100
Keys:  Help   Display mode   Restart statistics   Order of fields   quit
                                                        Packets               Pings
 Host                                                 Loss%   Snt   Last   Avg  Best  Wrst StDev
 1. abcd.efghi.rockstable.org                          0.0%    41    0.3   0.2   0.2   0.5   0.0
 2. ipABCDEFHI.dynamic.kabel-deutschland.de            0.0%    41    7.7  10.8   5.7  39.1   6.9
 3. 83-169-181-254-isp.superkabel.de                   0.0%    41    6.4   8.3   6.0  18.5   1.9
 4. ip5886c0f1.static.kabel-deutschland.de             0.0%    41    7.9   8.9   6.1  15.3   2.0
 5. 145.254.3.68                                       0.0%    41    8.1   8.8   6.1  14.2   1.9
 6. 145.254.2.179                                     39.0%    41   15.5  16.3  14.3  20.8   1.7
 7. 145.254.2.179                                     35.9%    40   16.1  16.5  13.7  21.6   2.2
 8. decix2-gw.hetzner.com                              0.0%    40   18.9  15.5  12.5  33.7   3.8
 9. core24.fsn1.hetzner.com                           94.9%    40   21.7  21.6  21.5  21.7   0.1
10. ex9k1.dc14.fsn1.hetzner.com                        0.0%    40   25.3  19.2  17.2  25.3   1.9
11. kvm2.rockstable.org                                0.0%    40   21.5  21.4  18.9  28.9   2.4
12. www2.rockstable.it                                 0.0%    40   22.3  20.8  18.4  26.5   1.9
}}}

=== netstat ===

Install netstat 
{{{#!highlight bash
aptitude install net-tools
}}}

 * {{{-n}}} don't resolve names (maybe slow)

Some important commands
{{{#!highlight bash
### LISTENING PORTS
% sudo netstat -tulpen
[sudo] Passwort für tobias: 
Aktive Internetverbindungen (Nur Server)
Proto Recv-Q Send-Q Local Address           Foreign Address         State       Benutzer   Inode      PID/Program name    
tcp        0      0 0.0.0.0:445             0.0.0.0:*               LISTEN      0          37224      3237/smbd           
tcp        0      0 0.0.0.0:2049            0.0.0.0:*               LISTEN      0          30994      -                   
tcp        0      0 0.0.0.0:3142            0.0.0.0:*               LISTEN      125        57443      1851/apt-cacher-ng  
tcp        0      0 127.0.0.1:3306          0.0.0.0:*               LISTEN      135        54106      1915/mariadbd       
tcp        0      0 0.0.0.0:139             0.0.0.0:*               LISTEN      0          37225      3237/smbd           
tcp        0      0 0.0.0.0:9102            0.0.0.0:*               LISTEN      0          61478      2297/bareos-fd      
tcp        0      0 192.168.122.1:53        0.0.0.0:*               LISTEN      0          31348      4064/dnsmasq        
tcp        0      0 192.168.101.1:53        0.0.0.0:*               LISTEN      0          72039      3926/dnsmasq        
tcp        0      0 192.168.100.1:53        0.0.0.0:*               LISTEN      0          50833      3797/dnsmasq        
tcp        0      0 0.0.0.0:22              0.0.0.0:*               LISTEN      0          39934      1863/sshd: /usr/sbi 
tcp        0      0 127.0.0.1:631           0.0.0.0:*               LISTEN      0          53186      1852/cupsd          
tcp        0      0 127.0.0.1:5432          0.0.0.0:*               LISTEN      136        40049      1928/postgres       
tcp        0      0 127.0.0.1:25            0.0.0.0:*               LISTEN      0          72985      3688/master         
…

### KERNEL INTERFACE TABLE
netstat -ian
Kernel-Schnittstellentabelle
Iface      MTU    RX-OK RX-ERR RX-DRP RX-OVR    TX-OK TX-ERR TX-DRP TX-OVR Flg
bond0     1500  1915555      0   1122 0       1029535      0      0      0 BMmRU
bridge    1500  1912759      0  50477 0       1028122      0      0      0 BMRU
enp8s0    1500  1915555      0      0 0       1029535      0      0      0 BMsRU
enp9s0    1500        0      0      0 0             0      0      0      0 BMU
lo       65536    58342      0      0 0         58342      0      0      0 LRU
…

### SHOW TCP/UDP SOCKETS (INCLUDING WAITING)
netstat -tuna
}}}

=== ping ===

==== missing capabilities ====

[[https://debianforum.de/forum/viewtopic.php?t=156254|Quelle Debian-Forum]]

It is totally unneccessary to run ping with sudo.

Check ping capabilities 
{{{#!highlight bash
getcap /bin/ping
}}}

ping capabilities should be
{{{#!highlight bash
/bin/ping = cap_net_raw+ep
}}}

add capability net_raw 
{{{#!highlight bash
setcap cap_net_raw+ep /bin/ping
}}}

=== socat ===


=== tcpdump ===

Incredibly useful tool.
May be combined with wireshark
when used to write a pcap-dump {{{-w}}}.

Some filter keywords to me remembered:
 * host
 * net
 * src
 * dst
 * port

==== dump_multi.sh ====

tcpdump misses an option
to display the interface the packet was received on.
Here is a little wrapper script
that simply starts multiple tcpdumps and
prefixes the output with the interface name.

The script was mainly copied from this thread on<<BR>>
[[https://serverfault.com/questions/224698/how-to-display-interface-in-tcpdump-output-flow|serverfault - how to display interface in tcpdump output flow]]

{{{/usr/local/sbin/dump_multi.sh}}}
{{{#!highlight bash
#!/bin/bash

SELF="$(basename "$0")"

declare -a INTERFACES

###ADD A STOP MARK TO THE POSITIONAL PARAMETERS
STOPMARK="$(uuidgen)"
set -- "$@" "$STOPMARK"

usage () {
        cat <<-EOF
                $SELF [Options]
                    Options:
                        -h|--help               Show this page
                        -i|--interface <arg>    Dump interface <arg>

                Options of $SELF mask options of tcpdump.
                Options of tcpdump are not documented here.
        EOF
}

while true; do
        case "$1" in
                '-h'|'--help')
                        usage
                        shift
                        continue
                ;;
                '-i'|'--interface')
                        INTERFACES+=( "$2" ) 
                        shift 2 
                        continue
                ;;
                ### BREAK OPTION PARSING AFTER ONE FULL ITERATION
                "$STOPMARK")
                        shift
                        break
                ;;
                *)
                        ### APPEND UNKNOWN OPTION TO THE END OF THE LIST
                        TMP1="$1"
                        shift
                        set -- "$@" "$TMP1"
                        unset TMP1
                ;;
        esac
done


### When this exits, exit all background processes:
trap 'kill $(jobs -p) &> /dev/null && sleep 0.2 &&  echo ' EXIT

### Create one tcpdump output per interface and
### add an identifier to the beginning of each line:
if [ "${#INTERFACES[@]}" -eq 1 ] \ 
        && [ "${INTERFACES[0]}" = "any" ];
then
    for IFACE in $(ip l \
            |grep '^[0-9]:' \
            |grep ',UP' \
            |awk '{print $2}' \
            |sed 's/://')
    do
       tcpdump -l -i "$IFACE" -nn "$@" \ 
               |sed 's/^/[Iface: '"$IFACE"']    /' & 
    done
elif [ "${#INTERFACES[@]}" -ge 1 ]; then
        for IFACE in "${INTERFACES[@]}"; do
                tcpdump -l -i "$IFACE" "$@" \ 
                        |sed 's/^/[Iface: '"$IFACE"'] /' & 
        done
fi

# wait for CTRL+C
wait
}}}

Use it like
{{{#!highlight bash
### ON EVERY INTERFACE THAT IS UP
dump_multi.sh -i any 'port …'
### ON A LIST OF INTERFACES
dump_multi.sh -i lo -i eth0 -i eth1 -i bond0 -i br0 'port …'
}}}

Example:
Dump DHCP/BOOTP traffic on a bridge, the attached bond and its slaves
and determine the traffic flow.
{{{#!highlight bash
./dump_multi.sh \
        -i bond0 -i enp8s0 -i enp9s0 -i bridge \
        'port (bootps or bootpc)'
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on bridge, link-type EN10MB (Ethernet), capture size 262144 bytes
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on bond0, link-type EN10MB (Ethernet), capture size 262144 bytes
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on enp9s0, link-type EN10MB (Ethernet), capture size 262144 bytes
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on enp8s0, link-type EN10MB (Ethernet), capture size 262144 bytes
[Iface: bridge] 08:26:31.531753 IP 0.0.0.0.bootpc > 255.255.255.255.bootps: BOOTP/DHCP, Request from 52:54:00:d1:b6:9b (oui Unknown), length 300
[Iface: bridge] 08:26:31.555936 IP 0.0.0.0.bootpc > 255.255.255.255.bootps: BOOTP/DHCP, Request from 52:54:00:d1:b6:9b (oui Unknown), length 300
[Iface: bridge] 08:26:34.392529 IP 0.0.0.0.bootpc > 255.255.255.255.bootps: BOOTP/DHCP, Request from c0:d2:f3:e1:fb:7b (oui Unknown), length 315
[Iface: bridge] 08:26:34.973608 IP 0.0.0.0.bootpc > 255.255.255.255.bootps: BOOTP/DHCP, Request from 52:54:00:d1:b6:9b (oui Unknown), length 300
}}}


=== whois ===

==== QUERY LIMIT ====

When experimenting.
don't be to curious or
you will be banned for a day.
{{{#!highlight bash
# whois -i nserver '195.201.246.253'                 
% This is the RIPE Database query service.
% The objects are in RPSL format.
%
% The RIPE Database is subject to Terms and Conditions.
% See http://www.ripe.net/db/support/db-terms-conditions.pdf

%ERROR:201: access denied for IP.ADD.RE.SS
%
% Queries from your IP address have passed the daily limit of controlled objects.
% Access from your host has been temporarily denied.
% For more information, see
% http://www.ripe.net/data-tools/db/faq/faq-db/why-did-you-receive-the-error-201-access-denied

% This query was served by the RIPE Database Query Service version 1.97.2 (HEREFORD)
}}}


==== Get information about server ====

Query server information
{{{#!highlight bash
# QUERY SUPPORTED TYPES
whois -q types
% This is the RIPE Database query service.
% The objects are in RPSL format.
%
% The RIPE Database is subject to Terms and Conditions.
% See http://www.ripe.net/db/support/db-terms-conditions.pdf

inetnum
inet6num
as-block
aut-num
as-set
route
route6
route-set
inet-rtr
filter-set
peering-set
rtr-set
domain
poetic-form
poem
mntner
irt
key-cert
organisation
role
person

% This query was served by the RIPE Database Query Service version 1.97.2 (BLAARKOP)

# QUERY SERVER VERSION
whois -q version
% This is the RIPE Database query service.
% The objects are in RPSL format.
%
% The RIPE Database is subject to Terms and Conditions.
% See http://www.ripe.net/db/support/db-terms-conditions.pdf

% whois-server-1.97.2
% This query was served by the RIPE Database Query Service version 1.97.2 (BLAARKOP)

# QUERY SERVER SOURCES
whois -q sources
}}}

==== Querying ====

You can query the templates and
the inverse keys with {{{-t TYPE}}}
{{{#!highlight bash
whois -t domain 
% This is the RIPE Database query service.
% The objects are in RPSL format.
%
% The RIPE Database is subject to Terms and Conditions.
% See http://www.ripe.net/db/support/db-terms-conditions.pdf

domain:         [mandatory]  [single]     [primary/lookup key]
descr:          [optional]   [multiple]   [ ]
org:            [optional]   [multiple]   [inverse key]
admin-c:        [mandatory]  [multiple]   [inverse key]
tech-c:         [mandatory]  [multiple]   [inverse key]
zone-c:         [mandatory]  [multiple]   [inverse key]
nserver:        [mandatory]  [multiple]   [inverse key]
ds-rdata:       [optional]   [multiple]   [inverse key]
remarks:        [optional]   [multiple]   [ ]
notify:         [optional]   [multiple]   [inverse key]
mnt-by:         [mandatory]  [multiple]   [inverse key]
created:        [generated]  [single]     [ ]
last-modified:  [generated]  [single]     [ ]
source:         [mandatory]  [single]     [ ]

% This query was served by the RIPE Database Query Service version 1.97.2 (BLAARKOP)

### OUTPUT THE TEMPLATE MORE VERBOSE
whois -v domain
% This is the RIPE Database query service.                                                                                                        
% The objects are in RPSL format.                                                                                                                 
%                                                                                                                                                 
% The RIPE Database is subject to Terms and Conditions.                                                                                           
% See http://www.ripe.net/db/support/db-terms-conditions.pdf                                                                                      
                                                                                                                                                  
The domain class:                                                                                                                                 
                                                                                                                                                  
      A domain object represents a Top Level Domain (TLD) or                                                                                      
      other domain registrations. It is also used for Reverse                                                                                     
      Delegations.                                                                                                                                
                                                                                                                                                  
domain:         [mandatory]  [single]     [primary/lookup key]                                                                                    
descr:          [optional]   [multiple]   [ ]                                                                                                     
org:            [optional]   [multiple]   [inverse key]                                                                                           
admin-c:        [mandatory]  [multiple]   [inverse key]                                                                                           
tech-c:         [mandatory]  [multiple]   [inverse key]                                                                                           
zone-c:         [mandatory]  [multiple]   [inverse key]                                                                                           
nserver:        [mandatory]  [multiple]   [inverse key]                                                                                           
ds-rdata:       [optional]   [multiple]   [inverse key]                                                                                           
remarks:        [optional]   [multiple]   [ ]                                                                                                     
notify:         [optional]   [multiple]   [inverse key]                                                                                           
mnt-by:         [mandatory]  [multiple]   [inverse key]                                                                                           
created:        [generated]  [single]     [ ]                                                                                                     
last-modified:  [generated]  [single]     [ ]                                                                                                     
source:         [mandatory]  [single]     [ ]
                                                                         
The content of the attributes of the domain class are defined below:
                                                                         
domain
                                                                                                                                                  
   Domain name.                                                                                                                                   
                                    
     Domain name as specified in RFC 1034 (point 5.2.1.2) with or
     without trailing dot (".").  The total length should not exceed
     254 characters (octets).

descr               

   A short description related to the object.

     A sequence of ASCII characters. 

…
}}}


== Techniques ==

=== Bonding ===

 * [[https://www.kernel.org/doc/html/latest/networking/bonding.html|Linux Docs html Bonding]]
 * [[https://www.kernel.org/doc/Documentation/networking/bonding.txt|Linux Docs txt Bonding]]
 * Some general information is documented in the man-pages<<BR>>
 {{{man 5 interfaces}}}

'''Buy switches that support LACP and MC-LAG!'''

==== Link aggregation - IEEE 802.1AX (previously 802.3ad) ====

Link aggregation
 * [[https://en.wikipedia.org/wiki/Link_aggregation|Wiki En Link aggregation]]
 * [[https://en.wikipedia.org/wiki/Link_aggregation#Link_Aggregation_Control_Protocol|Wiki En Link Aggregation Control Protocol (LACP)]]
 * [[https://en.wikipedia.org/wiki/Out-of-order_delivery|Wiki En Out-of-order delivery]]

''Not to be confused with IEEE 801.3ad (QinQ).''

 * Initial release of 802.3ad in 2000, fast adoption by vendors.
 * Formal transition of 802.3ad to IEEE 802.1AX-2008 on 3 November 2008.
 * Increases bandwidth of a interface beyond a single interface
 * Increases resilience (link-level redundancy)
  * In a port-cable-port connection each part can fail and has redundancy.
 * Lowers costs to the price of the cable

A link aggregation group (LAG) is the collection of physical ports combined together. 

===== Link Aggregation Control Protocol (LACP) =====

Within the IEEE specification,
the Link Aggregation Control Protocol (LACP) provides a method
to control the bundling of several physical ports together
to form a single logical channel.
LACP allows a network device
to negotiate an automatic bundling of links
by sending LACP packets to the peer
(directly connected device that also implements LACP). 

====== Linux bondig driver mode 4 802.3ad ======

Creates aggregation groups that ''share the same speed and duplex settings''.
Utilizes all slave network interfaces in the active aggregator group
according to the 802.3ad specification.
This mode is similar to the XOR mode above and
supports the same balancing policies.
The link is set up dynamically between two LACP-supporting peers.


====== LACP Features and practical examples ======

 1. Maximum number of bundled ports allowed in the port channel:
 Valid values are usually from 1 to 8.
 1. LACP packets are sent with multicast group MAC address
 {{{01:80:c2:00:00:02 (01-80-c2-00-00-02)}}}
 1. During LACP detection period
   * LACP packets are transmitted every second
   * Keep-alive mechanism for link member:
   (default: slow = 30s, fast=1s)
 1. LACP can have the port-channel load-balance mode
   * link (link-id) Integer that identifies the member link for load balancing.
   The range is from 1 to 8 and the load balancing mode
   can be set-up based on traffic models.[9]
 1. LACP mode
   * Active: Enables LACP unconditionally.
   * Passive: Enables LACP only when an LACP device is detected. (This is the default state)

====== Advantages over static configuration ======

 1. '''Failover occurs automatically''':
 When a link fails and
 there is (for example) a media converter between the devices,
 a peer system will not perceive any connectivity problems.
 With static link aggregation,
 the peer would continue sending traffic down the link
 causing the connection to fail.
 1. '''Dynamic configuration''':
 The device can confirm that the configuration
 at the other end can handle link aggregation.
 With Static link aggregation,
 a cabling or configuration mistake could go undetected and
 cause undesirable network behavior.

====== Practical notes ======

LACP works by sending frames (LACPDUs) down all links
that have the protocol enabled.
If it finds a device on the other end of the link
that also has LACP enabled,
it will also independently send frames along the same links
enabling the two units to detect multiple links between themselves and
then combine them into a single logical link.
LACP can be configured in one of two modes:
active or passive.
In active mode it will always send LACPDUs along the configured links.
In passive mode, however,
it only reacts as "speak when spoken to",
and therefore can be used as a way of controlling accidental loops
(as long as the other device is in active mode).

====== Proprietary link aggregation ======

In addition to the IEEE link aggregation substandards,
there are a number of proprietary aggregation schemes including
Cisco's EtherChannel and Port Aggregation Protocol,
Juniper's Aggregated Ethernet, AVAYA's Multi-Link Trunking,
Split Multi-Link Trunking, Routed Split Multi-Link Trunking and
Distributed Split Multi-Link Trunking, ZTE's "Smartgroup",
Huawei's "Eth-Trunk", or Connectify's Speedify.
Most high-end network devices support some kind of link aggregation,
and software-based implementations –
such as the *BSD lagg package, Linux bonding driver,
Solaris dladm aggr, etc. – also exist for many operating systems. 

====== Limitations ======

Single Switch
 * With the modes ''balance-rr, balance-xor, broadcast and 802.3ad'',
 all physical ports in the link aggregation group ''must reside on the same logical switch'',
 which, in most common scenarios,
 will leave a ''single point of failure''
 when the physical switch to which all links are connected goes offline.
 * The modes ''active-backup, balance-tlb, and balance-alb''
 can also be set up with ''two or more switches''.
 But ''after failover'' (like all other modes), in some cases,
 ''active sessions may fail (due to ARP problems)'' and
 have to be restarted.

Same Link Speed and Duplex
 * In most implementations,
 all the ports used in an aggregation consist of the same physical type,
 such as all copper ports (10/100/1000BASE‑T),
 all multi-mode fiber ports, or all single-mode fiber ports.
 However, all ''the IEEE standard requires is that each link be full duplex and
 all of them have an identical speed'' (10, 100, 1,000 or 10,000 Mbit/s). 


Ethernet aggregation mismatch
 * Aggregation mismatch refers to not matching the aggregation type on both ends of the link.
 Some switches do not implement the 802.1AX standard but support static configuration of link aggregation.
 Therefore, link aggregation between similarly statically configured switches will work
 but ''will fail between a statically configured switch and a device that is configured for LACP''. 


==== Multi-chassis link aggregation group (MC-LAG) ====

 * [[https://en.wikipedia.org/wiki/Multi-chassis_link_aggregation_group|Multi-chassis link aggregation group]]
##  * [[|]]

A multi-chassis link aggregation group (MLAG or MC-LAG)
is a type of link aggregation group (LAG) with constituent ports
that terminate on separate chassis,
primarily for the purpose of providing redundancy
in the event one of the chassis fails.
The IEEE 802.1AX-2008 industry standard for link aggregation does not mention MC-LAG,
but does not preclude it.
Its implementation varies by vendor;
notably, the protocol existing between the chassis is proprietary.

===== Advantages over LAG =====

 * node- and link-level redundancy
 * superior to spanning tree
   * links of a MC-LAG don't need to be disabled to prevent loops



==== Bonding with ifupdown and ifenslave ====

{{{
It's recommended to configure bonding
via iproute2 (netlink) or sysfs,
the old ifenslave control utility is obsolete.
}}}

Make sure to not interfere with Network-Manager. :-)

During the initial creation it is recommended to monitor the bond in a {{{tmux}}} session with
 * {{{tail -f /var/log/messages}}} to watch for bonding driver errors.
 * {{{watch -n1 -- cat  /proc/net/bonding/bond0}}} to check current bonding parameters.
 * {{{watch -n1 -- ip a}}} to check current networking amd interface state.


While the setup fails
prepare a individual statement to reset the state
to a clean starting point.
{{{#!highlight bash
for IFACE in bond0 enp8s0 enp9s0; do
        ifdown "$IFACE"
        ip l set down dev "$IFACE";
done
ip a del 192.168.182.16/24 dev enp8s0
ip a del 192.168.182.208/24 dev enp9s0
}}}


Bonding with ifupdown requires the package {{{ifenslave}}}.
{{{#!highlight bash
apt install ifenslave
}}}

There is a bit of documentation in<<BR>>
{{{/usr/share/doc/ifenslave/README.Debian.gz}}}

The examples are found at<<BR>>
{{{/usr/share/doc/ifenslave/examples}}}


Whenever possible use LACP IEEE 802.3ad.

If using lacp you should set the rate to fast (every 1 second).
The default is slow (every 30 seconds).
{{{#!highlight bash
# This file describes the network interfaces available on your system
# and how to activate them. For more information, see interfaces(5).

source /etc/network/interfaces.d/*

# The loopback network interface
auto lo
iface lo inet loopback

auto bond0
iface bond0 inet dhcp
        bond-slaves     enp8s0 enp9s0
        bond-miimon     100
        mode            802.3ad
        lacp_rate       fast

auto enp8s0
iface enp8s0 inet manual

auto enp9s0
iface enp9s0 inet manual
}}}


This config is for LACP agnostic switches
with apdaptive load-balancing.

{{{/etc/network/interfaces}}}
{{{#!highlight bash
# This file describes the network interfaces available on your system
# and how to activate them. For more information, see interfaces(5).

source /etc/network/interfaces.d/*

# The loopback network interface
auto lo
iface lo inet loopback

auto bond0
iface bond0 inet dhcp
        bond-primary    enp8s0 enp9s0
        bond-slaves     enp8s0 enp9s0
        bond-miimon     100
        bond-mode       balance-alb
}}}

==== Bonding with Network-Manager ====

Well, i had huge trouble configuring a lacp bond with NetworkManager.
Actually it didn't even work.
Whereas ifupdown was trivial and ready in seconds on first try …
I'm not yet convinced of NetworkManager.

It's easy to convert ifupdown config to Network-Manager
in {{{nm-connecttion-editor}}}.

For every interface
 1. open the interface to be converted
 1. mark the interface to be started automatically
 1. enable auto-negotiation
 1. save and exit the interface

The auto-generated file from
{{{/run/NetworkManager/system-connections}}}
will be saved in
{{{/etc/NetworkManager/system-connections}}}.

Unfortunately the option {{{autoconnect-slaves=1}}}
does not activate the slaves of the bonds.
So it's important that the slave interfaces
are marked to be started automatically
({{{autoconnect=false}}} is missing or {{{autoconnect=true}}}) or
the bond won't come up.
On the down-side the bond will also get up on boot
even if the bond interface is marked as {{{autoconnect=false}}}.


{{{/etc/NetworkManager/system-connections/bond0.nmconnection}}}
{{{#!highlight bash
[connection]
id=bond0
uuid=9d6f2839-75ce-4b40-9f1a-2ca925af6dfc
type=bond
interface-name=bond0
permissions=
timestamp=1604305217

[bond]
downdelay=0
miimon=100
mode=balance-alb
updelay=0

[ipv4]
dns-priority=100
dns-search=
method=auto

[ipv6]
addr-gen-mode=stable-privacy
address1=fd93:56fb:daf7:0:2d8:61ff:fe2e:7979/64
dns-priority=100
dns-search=
ip6-privacy=0
}}}

The default of Ethernet auto-negotiation is false.

{{{/etc/NetworkManager/system-connections/enp8s0.nmconnection}}}
{{{#!highlight bash
[connection]
id=enp8s0
uuid=1fc1fa53-f44e-48eb-8904-8cfd6a80950f
type=ethernet
interface-name=enp8s0
master=bond0
permissions=
slave-type=bond
timestamp=1604305049

[ethernet]
auto-negotiate=true
mac-address=00:D8:61:2E:79:79
mac-address-blacklist=
}}}

{{{/etc/NetworkManager/system-connections/enp9s0.nmconnection}}}
{{{#!highlight bash
[connection]
id=enp9s0
uuid=bb8eac2e-3b3f-4b96-8b49-72dc331ca521
type=ethernet
interface-name=enp9s0
master=bond0
permissions=
slave-type=bond
timestamp=1604305049

[ethernet]
auto-negotiate=true
mac-address=00:D8:61:2E:79:7A
mac-address-blacklist=
}}}

==== Bonding on VMware ====

 * [[https://docs.vmware.com/de/VMware-vSphere/index.html|VMware Docs 7.0 Index]]
 * [[https://docs.vmware.com/de/VMware-vSphere/7.0/com.vmware.vsphere.networking.doc/GUID-0D1EF5B4-7581-480B-B99D-5714B42CD7A9.html|VMware Docs 7.0 LACP-Support auf einem vSphere Distributed Switch]]

 * LACP is only supported in vSphere 5.1, 5.5 and 6.0
 using vSphere Distributed Switches (VDS) or
 the Cisco Nexus 1000v.
 * '''A distributed virtual switch requires VMware vSphere Enterprise Plus licensing.'''
 

=== Bridging ===

==== Bridging with Network-Manager ====

The following script is mainly a condensed form of 
[[https://blog.christophersmart.com/2020/08/29/how-to-create-bridges-on-bonds-with-and-without-vlans-using-networkmanager/|Christophers Blog Articel on How to create bridges on bonds with and without vlans using networkmanager]]

Create a bridge with the following script
{{{#!highlight bash
#!/bin/bash

### DEFINE BRIDGE
BRIDGE=bridge
BRIDGE_STP=no
#BRIDGE_MTU=1500

nmcli con add ifname "$BRIDGE" type bridge con-name "$BRIDGE"
nmcli con modify "$BRIDGE" bridge.stp "$BRIDGE_STP"
#nmcli con modify "$BRIDGE" 802-3-ethernet.mtu "$BRIDGE_MTU"

### DEFINE BONDsetup8
BOND=bond0
#BOND_SLAVE0=enp0s8
#BOND_SLAVE1=enp0s8
#BOND_MODE=active-backup
#BOND_MTU=9000

#nmcli con add type bond ifname "${BOND}" con-name "${BOND}"
#nmcli con modify "${BOND}" bond.options mode="${BOND_MODE}"
#nmcli con modify "${BOND}" 802-3-ethernet.mtu "${BOND_MTU}"
#nmcli con add type ethernet con-name "${BOND}-slave-${BOND_SLAVE0}" ifname "${BOND_SLAVE0}" master "${BOND}"
#nmcli con add type ethernet con-name "${BOND}-slave-${BOND_SLAVE1}" ifname "${BOND_SLAVE1}" master "${BOND}"
#nmcli con modify "${BOND}-slave-${BOND_SLAVE0}" 802-3-ethernet.mtu "${BOND_MTU}"
#nmcli con modify "${BOND}-slave-${BOND_SLAVE1}" 802-3-ethernet.mtu "${BOND_MTU}"

### ADD bond0 To bridge
nmcli con modify "${BOND}" master "${BRIDGE}" slave-type bridge

### SHOW SOME INFO
nmcli con
ls /sys/class/net/bridge/brif/
brctl show
}}}

Next is just a note and is yet unconfirmed.
There may be some bonding modes,
that don't work well with linux bridges,
especially {{{balance-alb}}}.



==== VM on Bridge ====

I had some trouble getting traffic
of a VM attached to a bridge over the enslaved bond.
Here are my notes.

The following line allows traffic,
but it's not the solution,
only a hint to a the problem.
Don't run your server with
an uninitialized iptables stack
in conjuction with other nftables hooks.
{{{#!highlight bash
echo 0 > /proc/sys/net/bridge/bridge-nf-call-iptables
}}}

According to
[[https://www.kernel.org/doc/html/latest/networking/ip-sysctl.html#proc-sys-net-bridge-variables|kernel.org doc ip-sysctl]]
{{{#!highlight bash
bridge-nf-call-iptables - BOOLEAN

        1 : pass bridged IPv4 traffic to iptables’ chains.
        0 : disable this.

    Default: 1
}}}

This seems to mean that [[#nftables]] is circumvented,
which is the default nowadays in Linux.

I wasn't aware
that there is such an extensive nftables ruleset
without having configured anything.<<BR>>
{{{nft list ruleset}}}

It turns out that the FORWARD policy of ipv4 traffic
is set to {{{drop}}}.<<BR>>
{{{nft list chain ip filter FORWARD}}}
{{{#!highlight bash
table ip filter {
        chain FORWARD {
                type filter hook forward priority filter; policy drop;
                counter packets 1095 bytes 92945 jump DOCKER-USER
                counter packets 1095 bytes 92945 jump DOCKER-ISOLATION-STAGE-1
                oifname "docker0" ct state related,established counter packets 0 bytes 0 accept
                oifname "docker0" counter packets 0 bytes 0 jump DOCKER
                iifname "docker0" oifname != "docker0" counter packets 0 bytes 0 accept
                iifname "docker0" oifname "docker0" counter packets 0 bytes 0 accept
                counter packets 1095 bytes 92945 jump LIBVIRT_FWX
                counter packets 1095 bytes 92945 jump LIBVIRT_FWI
                counter packets 1095 bytes 92945 jump LIBVIRT_FWO
        }
}
}}}

When setting this policy to accept
the traffic of virtual machine passes.
So this is the first runtime fix.<<BR>>
{{{#!highlight bash
nft chain ip filter FORWARD '{policy accept;}'
}}}

I still need a permanent solution.
And i need to figure out,
who set this policy to drop.
I'll start by disabling docker at boot time and reboot.
{{{#!highlight bash
systemctl disable docker
}}}

And i was right - after reboot<<BR>>
{{{nft list chain ip filter FORWARD}}}
{{{#!highlight bash
table ip filter {
        chain FORWARD {
                type filter hook forward priority filter; policy accept;
                counter packets 0 bytes 0 jump LIBVIRT_FWX
                counter packets 0 bytes 0 jump LIBVIRT_FWI
                counter packets 0 bytes 0 jump LIBVIRT_FWO
        }
}
}}}

The VM got a IP via DHCP.


Attempt 1
 1. Revert {{{bridge-nf-call-iptables}}} to 1 and
 flushed nftables ruleset {{{nft flush ruleset}}}.
 And it also worked.
 1. Restart {{{nftables.service}}} and it still worked.
 1. Restart {{{libvirtd.service}}} and it still worked.
 1. Restart {{{docker.service}}} and it still worked.
   * Docker adds rules to nftables.
   * Did not change the ipv4 forward policy to drop,
   because libvirt already enabled {{{ip_forward}}}.
 1. Stuck - I only got one more confirmation,
 that the nftables stack is responsible.

Attempt 2
 1. Rebooted and it fails.
 1. Stopped {{{docker.service}}} - still fails
   * Docker does not remove its rules from nftables.
   * IPv4 forward policy is set to drop.
 1. Disabled {{{docker.service}}} and rebooted - works.
 1. Started {{{docker.service}}} - still works.
   * Docker adds rules to nftables.
   * Did not change the ipv4 forward policy to drop,
   because libvirt already enabled {{{ip_forward}}}.
 1. Double check this by enabling and rebooting
 to exclude interference with the "runtime fix" from above.
   * '''Confirmed''' Docker sets IPv4 forwarding policy to drop. :-(

Related info
 * [[https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=865975|Debian bug report #865975]]
 * [[https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=903635|Debian bug report #903635]]
 * [[file:///usr/share/doc/docker.io/README.Debian|Docker.io README.Debian]]
 I haven't been able to work around this with the daemon config file either
 * [[https://docs.docker.com/config/containers/container-networking/|Docker Docs - Container networking]]
 * [[https://docs.docker.com/network/iptables/|Docker Docs - Docker and iptables]]
 * [[https://docs.docker.com/network/bridge/|Docker Docs - Use bridge networks]]


===== How docker reasons this behaviour =====

The FORWARD chain policy is set to DROP by docker since 1.13.
As of writing this I'm currently using
{{{docker.it 19.03.13+dfsg1-3}}} from Debian Bullseye.

Docker needs relies on package forwarding
to make containers reachable.

If it's not enabled when bringing up the docker interfaces,
docker enables forwarding and
is by this reason also responsible for any security implications
that arise from enabling forwarding and
thus sets default policy to DROP.
This may break other 3rd-party applications or VMs on the system,
but it at least does not impose security threads.

I tend to agree with this idea.
Other software should have configured their firewall rules as well.

===== Solution =====

In the end it's race condition between
docker (which would set {{{net.ipv4.ip_forward = 1}}} and
set nf {{{policy drop;}}} unless {{{net.ipv4.ip_forward}}} was not enabled by docker) and
libvirtd (which would set {{{net.ipv4.ip_forward = 1}}}).

A quick and permanent solution by this reason is
enabling forwarding before bringing up docker (or anything else).<<BR>>
[[Linux#IPv4_Forwarding]]]


=== VLAN Subinterfaces ===

==== VLAN Subinterfaces with ifupdown ====

Can't be easier.

{{{/etc/network/interfaces}}}
{{{#!highlight json
auto bond0.32
iface bond0.32 inet dhcp

auto bond0.40
iface bond0.40 inet dhcp
}}}


== nftables ==

 * [[https://wiki.nftables.org]]
 * [[file:///usr/share/doc/nftables/README.Debian]]


Default rule set in<<BR>>
{{{/etc/nftables.conf}}}
{{{#!highlight json
#!/usr/sbin/nft -f

flush ruleset

table inet filter {
        chain input {
                type filter hook input priority 0;
        }
        chain forward {
                type filter hook forward priority 0;
        }
        chain output {
                type filter hook output priority 0;
        }
}
}}}

Executable: {{{/usr/sbin/nft}}}

Important commands
{{{#!highlight bash
nft list ruleset
nft flush ruleset
nft add table inet filter
nft add chain inet filter input { type filter hook input priority 0 \; policy drop }
nft add rule inet filter input ct state established counter accept
nft list ruleset
}}}





== Hardware ==

=== linux-firmware_dl.sh ===

Download the firmware recursively and
move firmware to its path in filesystem.

{{{/usr/local/sbin/linux-firmware_dl.sh}}}
{{{#!highlight bash
#!/bin/bash

[ "$1" ] && GIT_DIR="$1" || exit 1
[ "$2" ] && GIT_FILTER="$2"

DIR_TMP_BASE="$(mktemp -d)"
DIR_TMP="$DIR_TMP_BASE/$GIT_DIR"
DIR_FW="/lib/firmware/$GIT_DIR"
URL_GIT='https://git.kernel.org/pub/scm/linux/kernel/git/firmware/linux-firmware.git/plain'

mkdir -p "$DIR_TMP"
cd "$DIR_TMP" || exit 1
wget -r -nd -np -e robots=off \
        -A README -A "$GIT_FILTER*.bin" \
        "$URL_GIT/$GIT_DIR/"
#wget --recursive --no-directories --no-parent -e robots=off \
#       -A README -A "$GIT_FILTER*.bin" \
#       "$URL_GIT/$GIT_DIR/"
ls -1 "$DIR_TMP"/*.tmp > /dev/null 2>&1 \
        && rm "$DIR_TMP"/*.tmp
cd ..
[ -d "$DIR_FW" ] || mkdir "$DIR_FW"
sudo mv "$DIR_TMP"/* "$DIR_FW"
sudo chown -R 0.0 "$DIR_FW"
sudo find "$DIR_FW" -type d -exec chmod 0755 {} \;
sudo find "$DIR_FW" -type f -exec chmod 0644 {} \;
rm -rv "$DIR_TMP_BASE"
}}}

Use it like
{{{#!highlight bash
chmod u+x /usr/local/sbin/linux-firmware_dl.sh
### linux-firmware_dl.sh $SUBDIR $PREFIX
linux-firmware_dl.sh rtw88
### OR FOR LATEST VEGA64 FIRMWARE
#linux.firmware_dl.sh amdgpu vega10_
}}}

Looks like this
{{{#!highlight bash
ll -d /lib/firmware/rtw88/*
-rw-r--r-- 1 root root   1087 Mai 26 14:14 /lib/firmware/rtw88/README
-rw-r--r-- 1 root root  28884 Mai 26 14:14 /lib/firmware/rtw88/rtw8723d_fw.bin
-rw-r--r-- 1 root root 137896 Mai 26 14:14 /lib/firmware/rtw88/rtw8821c_fw.bin
-rw-r--r-- 1 root root 150984 Mai 26 14:14 /lib/firmware/rtw88/rtw8822b_fw.bin
-rw-r--r-- 1 root root 189152 Mai 26 14:14 /lib/firmware/rtw88/rtw8822c_fw.bin
-rw-r--r-- 1 root root 138720 Mai 26 14:14 /lib/firmware/rtw88/rtw8822c_wow_fw.bin
}}}

=== Debian RTL8822BE/RTL8822CE ===

 * Works with Linux 4.19 with old driver {{{rtlwifi}}}
   * uses firmware:
     * {{{/lib/firmware/rtlwifi/rtl8822befw.bin}}},
     which is packed in {{{firmware-realtek}}}
 * Fails since Linux 5.2 with the new driver {{{rtwpci}}} -> {{{rtw88}}}
   * uses firmware:
     * {{{rtw88/rtw8822b_fw.bin}}}
     * {{{rtw88/rtw8822c_wow_fw.bin}}}
     * {{{rtw88/rtw8822c_fw.bin}}}

So you need the firmware {{{rtw88/rtw8822b_fw.bin}}}

[[https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=945172|Debian Bug #945172]] suggests to link the files.

I think it's better to get the most recent version from <<BR>>
[[https://git.kernel.org/pub/scm/linux/kernel/git/firmware/linux-firmware.git|git.kernel.org linux-firmware]]

You may use the following script to download {{{rtw88}}} directory to {{{/lib/firmware}}}<<BR>>
[[#linux-firmware_dl.sh]]

Install the latest backports-kernel and reboot or
just update your initramfs <<BR>>
{{{update-initramfs -k all -u}}}

I guess your !WiFi works now.

And one day … Debian's firmware-packages will be refreshed and
overwrite the contents of this manually created directory. ;-)

Working !WiFi with {{{rtw88}}} on Linux 5.6 <<BR>>
{{{lspci -vvs 04:00.0}}}
{{{#!highlight bash
04:00.0 Network controller: Realtek Semiconductor Co., Ltd. RTL8822BE 802.11a/b/g/n/ac WiFi adapter
        Subsystem: Lenovo ThinkPad E595
        Physical Slot: 0
        Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx+
        Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
        Latency: 0, Cache Line Size: 32 bytes
        Interrupt: pin A routed to IRQ 80
        Region 0: I/O ports at 2000 [size=256]
        Region 2: Memory at d0600000 (64-bit, non-prefetchable) [size=64K]
        Capabilities: [40] Power Management version 3
                Flags: PMEClk- DSI- D1+ D2+ AuxCurrent=375mA PME(D0+,D1+,D2+,D3hot+,D3cold+)
                Status: D0 NoSoftRst+ PME-Enable- DSel=0 DScale=0 PME-
        Capabilities: [50] MSI: Enable+ Count=1/1 Maskable- 64bit+
                Address: 00000000fee00000  Data: 0000
        Capabilities: [70] Express (v2) Endpoint, MSI 00
                DevCap: MaxPayload 128 bytes, PhantFunc 0, Latency L0s <4us, L1 <64us
                        ExtTag- AttnBtn- AttnInd- PwrInd- RBE+ FLReset- SlotPowerLimit 0.000W
                DevCtl: CorrErr- NonFatalErr- FatalErr- UnsupReq-
                        RlxdOrd+ ExtTag- PhantFunc- AuxPwr- NoSnoop-
                        MaxPayload 128 bytes, MaxReadReq 512 bytes
                DevSta: CorrErr- NonFatalErr- FatalErr- UnsupReq- AuxPwr+ TransPend-
                LnkCap: Port #0, Speed 2.5GT/s, Width x1, ASPM L0s L1, Exit Latency L0s <2us, L1 <64us
                        ClockPM+ Surprise- LLActRep- BwNot- ASPMOptComp-
                LnkCtl: ASPM L1 Enabled; RCB 64 bytes Disabled- CommClk+
                        ExtSynch- ClockPM+ AutWidDis- BWInt- AutBWInt-
                LnkSta: Speed 2.5GT/s (ok), Width x1 (ok)
                        TrErr- Train- SlotClk+ DLActive- BWMgmt- ABWMgmt-
                DevCap2: Completion Timeout: Not Supported, TimeoutDis+, NROPrPrP-, LTR+
                         10BitTagComp-, 10BitTagReq-, OBFF Via message/WAKE#, ExtFmt-, EETLPPrefix-
                         EmergencyPowerReduction Not Supported, EmergencyPowerReductionInit-
                         FRS-, TPHComp-, ExtTPHComp-
                         AtomicOpsCap: 32bit- 64bit- 128bitCAS-
                DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis-, LTR+, OBFF Disabled
                         AtomicOpsCtl: ReqEn-
                LnkCtl2: Target Link Speed: 5GT/s, EnterCompliance- SpeedDis-
                         Transmit Margin: Normal Operating Range, EnterModifiedCompliance- ComplianceSOS-
                         Compliance De-emphasis: -6dB
                LnkSta2: Current De-emphasis Level: -3.5dB, EqualizationComplete-, EqualizationPhase1-
                         EqualizationPhase2-, EqualizationPhase3-, LinkEqualizationRequest-
        Capabilities: [100 v2] Advanced Error Reporting
                UESta:  DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
                UEMsk:  DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
                UESvrt: DLP+ SDES+ TLP- FCP+ CmpltTO- CmpltAbrt- UnxCmplt- RxOF+ MalfTLP+ ECRC- UnsupReq- ACSViol-
                CESta:  RxErr- BadTLP- BadDLLP- Rollover- Timeout- AdvNonFatalErr-
                CEMsk:  RxErr- BadTLP- BadDLLP- Rollover- Timeout- AdvNonFatalErr+
                AERCap: First Error Pointer: 00, ECRCGenCap+ ECRCGenEn- ECRCChkCap+ ECRCChkEn-
                        MultHdrRecCap- MultHdrRecEn- TLPPfxPres- HdrLogCap-
                HeaderLog: 00000000 00000000 00000000 00000000
        Capabilities: [148 v1] Device Serial Number 00-e0-4c-ff-fe-b8-22-01
        Capabilities: [158 v1] Latency Tolerance Reporting
                Max snoop latency: 1048576ns
                Max no snoop latency: 1048576ns
        Capabilities: [160 v1] L1 PM Substates
                L1SubCap: PCI-PM_L1.2+ PCI-PM_L1.1+ ASPM_L1.2+ ASPM_L1.1+ L1_PM_Substates+
                          PortCommonModeRestoreTime=30us PortTPowerOnTime=60us
                L1SubCtl1: PCI-PM_L1.2- PCI-PM_L1.1+ ASPM_L1.2- ASPM_L1.1+
                           T_CommonMode=0us LTR1.2_Threshold=0ns
                L1SubCtl2: T_PwrOn=60us
        Kernel driver in use: rtw_pci
        Kernel modules: rtwpci
}}}

{{{modinfo rtwpci rtw88}}}
{{{#!highlight bash
filename:       /lib/modules/5.6.0-1-amd64/kernel/drivers/net/wireless/realtek/rtw88/rtwpci.ko
license:        Dual BSD/GPL
description:    Realtek 802.11ac wireless PCI driver
author:         Realtek Corporation
alias:          pci:v000010ECd0000C822sv*sd*bc*sc*i*
alias:          pci:v000010ECd0000B822sv*sd*bc*sc*i*
depends:        mac80211,rtw88
retpoline:      Y
intree:         Y
name:           rtwpci
vermagic:       5.6.0-1-amd64 SMP mod_unload modversions 
sig_id:         PKCS#7
signer:         Debian Secure Boot CA
sig_key:        A7:46:8D:EF
sig_hashalgo:   sha256
signature:      79:B8:A3:7B:68:2D:CD:38:76:CB:48:1C:56:D4:20:77:7D:97:3C:24:
                F5:BE:84:25:31:34:EE:27:03:F8:13:41:49:BD:09:E3:A7:09:86:CB:
                91:50:0A:E0:3F:CA:19:CC:2A:AF:56:CE:D0:A2:4C:E0:83:C6:8F:71:
                C0:E3:A2:68:BA:F6:50:F6:FC:10:76:E4:08:94:65:33:37:0A:56:9C:
                C3:F9:AF:97:FA:30:7F:10:7A:47:81:28:F2:79:B5:79:7F:AE:F6:58:
                6F:E2:6B:F6:78:8C:9D:89:37:26:67:3A:57:ED:03:16:79:26:EA:D2:
                91:D5:F0:8B:1C:4D:CC:56:97:EA:3D:4F:45:5F:B7:54:C2:26:08:71:
                A1:01:FF:A9:7E:2F:61:CF:C2:A8:DA:1C:1B:2C:D3:60:4C:D6:53:1E:
                00:8D:3A:09:14:BB:7A:A7:27:8C:E4:BB:C9:40:85:EB:FE:0B:18:0A:
                76:39:F7:9F:70:FB:0B:DB:BA:33:BC:31:0F:C2:75:45:E1:11:1A:B4:
                58:31:6E:26:CC:45:AE:AC:4D:67:5B:DE:CC:08:D8:01:49:D9:71:E8:
                25:6C:C5:E8:DF:F7:DE:64:CE:34:00:5F:7A:3D:E6:8D:77:28:FD:BB:
                6A:E5:83:41:61:46:0F:73:C7:21:F9:90:2F:5A:6D:93
parm:           disable_msi:Set Y to disable MSI interrupt support (bool)

filename:       /lib/modules/5.6.0-1-amd64/kernel/drivers/net/wireless/realtek/rtw88/rtw88.ko
license:        Dual BSD/GPL
description:    Realtek 802.11ac wireless core module
author:         Realtek Corporation
firmware:       rtw88/rtw8822b_fw.bin
firmware:       rtw88/rtw8822c_wow_fw.bin
firmware:       rtw88/rtw8822c_fw.bin
depends:        mac80211,cfg80211
retpoline:      Y
intree:         Y
name:           rtw88
vermagic:       5.6.0-1-amd64 SMP mod_unload modversions 
sig_id:         PKCS#7
signer:         Debian Secure Boot CA
sig_key:        A7:46:8D:EF
sig_hashalgo:   sha256
signature:      AD:30:FA:52:72:54:79:79:FC:7B:8A:52:92:19:F5:30:91:CD:F2:13:
                00:8A:FD:8D:11:B2:94:FA:DB:4E:BF:B7:32:5D:EB:71:C6:27:81:34:
                87:D9:59:7F:8F:32:6F:E6:2F:AF:F9:8F:EF:E2:E1:FF:39:EE:AD:EB:
                BF:13:9C:CE:9A:F6:72:3A:8E:27:91:E4:98:60:48:4C:36:84:3E:90:
                01:4D:4A:BA:7C:5E:D5:7B:7F:C0:3F:74:1C:C7:04:04:EC:9D:5D:55:
                D6:CE:AE:2C:F6:8E:37:94:83:1B:D2:6D:34:17:DA:59:B0:57:68:6C:
                A3:E6:5A:2D:3E:2D:FB:EA:C0:08:E1:0C:DE:64:1C:84:17:75:CD:C1:
                0F:C5:C4:CE:97:E1:24:2E:57:F1:B8:EF:9E:8B:B0:C7:99:B6:1C:1D:
                4D:AE:49:DE:BD:3B:40:65:74:C5:C8:DF:96:C2:40:DC:7B:23:6E:73:
                20:52:E4:DF:E2:C1:86:D3:F0:C6:B4:6D:5E:12:97:09:EE:82:A6:5F:
                E6:E0:69:95:9A:98:69:B8:F5:48:12:2F:4A:BB:5B:FD:3E:63:0E:A7:
                D8:40:A8:55:E5:07:E8:81:EF:5E:36:3E:38:6F:D9:A5:75:BE:6E:D5:
                F5:70:C2:AD:F5:4F:94:D4:D4:29:68:E6:31:FA:8D:6E
parm:           lps_deep_mode:Deeper PS mode. If 0, deep PS is disabled (uint)
parm:           support_bf:Set Y to enable beamformee support (bool)
parm:           debug_mask:Debugging mask (uint)
}}}


== firewalld ==

 * [[http://www.firewalld.org]]
 * [[https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/8/html/configuring_and_managing_networking/using-and-configuring-firewalld_configuring-and-managing-networking|Red Hat Customer Portal Using and configuring firewalld]]

A service daemon with D-Bus interface

 * User configuration: {{{/etc/firewalld}}}
 * Presets: {{{/usr/lib/firewalld}}}
 * Logdir: {{{/var/log/firewalld}}}

Install the dynamic firewall configuration daemon
{{{#!highlight bash
apt install firewalld
}}}

Install the graphical configuration tool and systray applet
{{{#!highlight bash
apt install firewall-config firewall-applet
}}}

Gather some info
{{{#!highlight bash
### AVAILABLE SERVICES
root@ipa2 ~ # firewall-cmd --get-services |fold -s
RH-Satellite-6 amanda-client amanda-k5-client amqp amqps apcupsd audit bacula 
bacula-client bb bgp bitcoin bitcoin-rpc bitcoin-testnet bitcoin-testnet-rpc 
bittorrent-lsd ceph ceph-mon cfengine cockpit condor-collector ctdb dhcp dhcpv6 
dhcpv6-client distcc dns dns-over-tls docker-registry docker-swarm 
dropbox-lansync elasticsearch etcd-client etcd-server finger freeipa-4 
freeipa-ldap freeipa-ldaps freeipa-replication freeipa-trust ftp ganglia-client 
ganglia-master git grafana gre high-availability http https imap imaps ipp 
ipp-client ipsec irc ircs iscsi-target isns jenkins kadmin kdeconnect kerberos 
kibana klogin kpasswd kprop kshell kube-apiserver ldap ldaps libvirt 
libvirt-tls lightning-network llmnr managesieve matrix mdns memcache minidlna 
mongodb mosh mountd mqtt mqtt-tls ms-wbt mssql murmur mysql nfs nfs3 nmea-0183 
nrpe ntp nut openvpn ovirt-imageio ovirt-storageconsole ovirt-vmconsole plex 
pmcd pmproxy pmwebapi pmwebapis pop3 pop3s postgresql privoxy prometheus 
proxy-dhcp ptp pulseaudio puppetmaster quassel radius rdp redis redis-sentinel 
rpc-bind rsh rsyncd rtsp salt-master samba samba-client samba-dc sane sip sips 
slp smtp smtp-submission smtps snmp snmptrap spideroak-lansync spotify-sync 
squid ssdp ssh steam-streaming svdrp svn syncthing syncthing-gui synergy syslog 
syslog-tls telnet tentacle tftp tftp-client tile38 tinc tor-socks 
transmission-client upnp-client vdsm vnc-server wbem-http wbem-https wsman 
wsmans xdmcp xmpp-bosh xmpp-client xmpp-local xmpp-server zabbix-agent 
zabbix-server
root@ipa2 ~ # firewall-cmd --get-default-zone
public
root@ipa2 ~ # firewall-cmd --permanent --info-zone=public
public
  target: default
  icmp-block-inversion: no
  interfaces:
  sources:
  services: cockpit dhcpv6-client ssh
  ports:
  protocols:
  masquerade: no
  forward-ports:
  source-ports:
  icmp-blocks:
  rich rules:
root@ipa2 ~ # firewall-cmd --permanent --list-services
cockpit dhcpv6-client ssh
}}}

Open the firewall for freeipa
{{{#!highlight bash
root@ipa2 ~ # firewall-cmd --add-service=freeipa-4
success
root@ipa2 ~ # firewall-cmd --permanent --add-service=freeipa-4
success
root@ipa2 ~ # firewall-cmd --permanent --list-services
cockpit dhcpv6-client freeipa-4 ssh
root@ipa2 ~ # firewall-cmd --permanent --service=freeipa-4 --get-includes
http https kerberos kpasswd ldap ldaps
root@ipa2 ~ # firewall-cmd --permanent --service=ldap --get-ports
389/tcp
…
root@ipa2 ~ # firewall-cmd --permanent --service=freeipa-replication --get-ports
7389/tcp
root@ipa2 ~ # firewall-cmd --permanent --add-service=freeipa-replication
success
}}}

Open the firewall for DNS queries
{{{#!highlight bash
root@ipa2 ~ # firewall-cmd --add-service=dns
success
root@ipa2 /usr/lib64/bind # firewall-cmd --permanent --add-service=dns
success
}}}

'''Test if it worked!
It works when it's tested sucessfully! :-D'''

== Network-Manager ==

=== About ===

 * [[https://developer.gnome.org/NetworkManager/|Network-Manager Docs]]
 * [[https://wiki.gnome.org/Projects/NetworkManager|Network-Manager Homepage]]
 * [[https://gitlab.freedesktop.org/NetworkManager/NetworkManager|Network-Manager Source]]

Please see
 * '''{{{man 5 nm-settings}}}''' for a
 description of settings and properties of NetworkManager connection profiles for nmcli
 * {{{man 5 NetworkManager.conf}}} for a description of the NetworkManager configuration file


=== Ecosystem ===

 * nmcli (command line interface)
 * nm-connection-editor (gui-editor
 * nm-applet (gui integration)
 * nm-online (check if online)

=== Authorization with policy kit ===

If you are in group {{{sudo}}} or {{{netdev}}},
you are allowed to modify system-connections.

{{{/usr/share/polkit-1/rules.d/60-network-manager.rules}}}
{{{#!highlight bash
polkit.addRule(function(action, subject) {
  if (action.id == "org.freedesktop.NetworkManager.settings.modify.system" &&
        subject.local && subject.active && 
        (subject.isInGroup ("sudo") || subject.isInGroup ("netdev"))) {
    return polkit.Result.YES;
  }
});
}}}

=== Do not manage interfaces ===

You can configure Network-Manager
to not manage the interfaces mentioned in<<BR>>
{{{/etc/network/interfaces}}}
{{{#!highlight bash
# This file describes the network interfaces available on your system
# and how to activate them. For more information, see interfaces(5).

source /etc/network/interfaces.d/*

# The loopback network interface
auto lo
iface lo inet loopback

auto enp1s0
iface enp1s0 inet dhcp

#auto enp2s0
iface enp2s0 inet dhcp
}}}

Make sure you don't remove the line entirely or
Network-Manager will grab the interface and
spawn a dhclient for the interface.

{{{/etc/NetworkManager/NetworkManager.conf}}}
{{{#!highlight bash
[main]
plugins=ifupdown,keyfile

[ifupdown]
managed=false
}}}

Here is the corresponding section from<<BR>>
{{{man 5 NetworkManager.conf}}}
{{{#!highlight bash
IFUPDOWN SECTION
       This section contains ifupdown-specific options and thus
       only has effect when using the ifupdown plugin.

       managed
           If set to true, then interfaces listed in
           /etc/network/interfaces are managed by
           NetworkManager. If set to false, then any interface
           listed in /etc/network/interfaces will be ignored by
           NetworkManager. Remember that NetworkManager
           controls the default route, so because the interface
           is ignored, NetworkManager may assign the default
           route to some other interface.

           The default value is false.
}}}


=== Manage interfaces ===

Something to remember …

'''Network-Manager controls the default route.'''


If you try to connect to a VPN
with the default route assigned by a interface that is unmanaged by Network-Manager
(Network-Manager does not control the default route),
the following error message is logged.
{{{#!highlight bash
Oct 30 17:27:14 libertas NetworkManager[116637]: <info>  [1604075234.9052] audit: op="connection-activate" uuid="6bcc9142-242a-44c9-adff-d30601f41919" name="openvpn_connection" pid=5284 uid=1000 result="fail" reason="Could not find source connection."
}}}


When setting to {{{managed=true}}},<<BR>>
the following error is logged activating {{{bond0}}}.
{{{#!highlight bash
# nmcli connection  up Ifupdown\ \(bond0\) 
Fehler: Aktivierung der Verbindung ist gescheitert: No suitable device found for this connection (device docker0 not available because profile is not compatible with device (mismatching interface name)).
}}}

In ifupdown-managed mode
Network-Manager automatically creates configurations
for the devices listes in {{{/etc/network/interfaces}}}.
These files are located in<<BR>>
{{{ll /run/NetworkManager/system-connections}}}
{{{#!highlight bash
-rw------- 1 root root 393 30. Oct 18:50 bond0.nmconnection
-rw------- 1 root root 433 30. Oct 12:19 docker0.nmconnection
-rw------- 1 root root 304 30. Oct 18:40 enp8s0.nmconnection
-rw------- 1 root root 304 30. Oct 18:40 enp9s0.nmconnection
-rw------- 1 root root 438 30. Oct 12:19 virbr0.nmconnection
}}}

The bonding configuration is not bad at all but is also not functionial.
So i converted the dynamically generated config to a permanent config and adjusted it.
Please see [[#Bonding with Network-Manager]].
Now that NM controls the default route,
activating/… VPNs works simply fine!


=== Imported VPNs ===

Network-Manager imports various VPN-profiles,
but does not set the file permissions
on the private key tight enough.

NM stores certificates in subdirectories below<<BR>>
{{{~/.local/share/networkmanagement/certificates/}}}.
Make sure you have the correct selinux context on this files, too.
{{{#!highlight bash
find ~/.local/share/networkmanagement/certificates/ \
        -type f -name private.key -exec chmod 600 {} \;
}}}



==== OpenVPN ====

 * The username may be case-sensitive. 
 * You may disable getting pushed the default route in
 VPN configuration dialogue > IPv4/6 > Button > (Check) "Use only for resources of this connection"


{{{}}}
{{{#!highlight bash
}}}



== Common network setups ==

single points of failures (SPOF)

=== Setup 1 ===

 * Setup
   * Uplink: SPOF
   * Firewall: SPOF
   * Switches: SPOF
   * Hosts: SPOF
 * Attributes
   * most simple and cheap setup
   * maintenance always causes downtime
   * no redundancy at all, let them fail
   * cascading single points of failures
     * The longer the cascade,
     the higher the risk of failure.

[[attachment:setup1.svg|{{attachment:setup1.svg}}]]

=== Setup 2 ===

 * Setup
   * Uplink: SPOF
   * Firewall:
     * redundant (active-standby)
     * maintenance without service interuption
   * Switches: SPOF
   * Hosts: SPOF
   * Additional external switch: SPOF
      * gives some additional flexibility
      * distributes the uplink
      * probably unnecessary
      * additional spof
 * Attributes
   * better external connectivity
   

[[attachment:setup2.svg|{{attachment:setup2.svg}}]]

=== Setup 3 ===

 * Setup
   * Uplink:
     * redundant
     * single isp handles failure of a uplink router
     * dual-isp setup with own Autonomous System (AS) and
     dynamic routing (BGP) possible
   * Firewall:
     * redundant (active-standby)
     * maintenance without service interuption
   * Switches: SPOF
   * Hosts: SPOF
 * Comments
   1. Setup may be distributed,
   e.g. to different buildings


[[attachment:setup3.svg|{{attachment:setup3.svg}}]]

=== Setup 4 ===

 * Setup
   * Uplink:
     * redundant
     * single isp handles failure of a uplink router
     * dual-isp setup with own Autonomous System (AS) and
     dynamic routing (BGP) possible
   * Firewall:
     * redundant (active-standby)
     * maintenance without service interuption
   * Switches:
     * redundant (STP)
     * no addtional performance (links blocked)
   * Hosts: bond (active-backup)

[[attachment:setup4.svg|{{attachment:setup4.svg}}]]

=== Setup 5 ===

 * Setup
   * Uplink:
     * redundant
     * single isp handles failure of a uplink router
     * dual-isp setup with own Autonomous System (AS) and
     dynamic routing (BGP) possible
   * Firewall:
     * redundant (active-standby)
     * maintenance without service interuption
     * bond(lacp)
   * Switches:
     * redundant (STP)
     * no addtional performance (links blocked)
   * Hosts: bond (active-backup)

[[attachment:setup5.svg|{{attachment:setup5.svg}}]]

=== Setup 6 ===

 * Setup
   * Uplink:
     * redundant
     * single isp handles failure of a uplink router
     * dual-isp setup with own Autonomous System (AS) and
     dynamic routing (BGP) possible
   * Firewall:
     * redundant (active-standby)
     * maintenance without service interuption
     * bond(active-backup)
   * Switches:
     * redundant (STP)
     * no addtional performance (links blocked)
   * Hosts: bond (active-backup)

[[attachment:setup6.svg|{{attachment:setup6.svg}}]]

=== Setup 7 ===

 * Setup
   * Uplink:
     * redundant
     * single isp handles failure of a uplink router
     * dual-isp setup with own Autonomous System (AS) and
     dynamic routing (BGP) possible
   * Firewall:
     * redundant (active-standby)
     * maintenance without service interuption
     * bond(lacp)
   * Switches:
     * redundant (stack)
     * no wasted performance (lags with lacp)
   * Hosts: bond (lacp)

[[attachment:setup7.svg|{{attachment:setup7.svg}}]]

=== Setup 8 ===

 * Setup
   * Uplink:
     * redundant
     * single isp handles failure of a uplink router
     * dual-isp setup with own Autonomous System (AS) and
     dynamic routing (BGP) possible
   * Firewall:
     * redundant (active-standby)
     * maintenance without service interuption
     * bond(lacp)
   * Switches:
     * redundant (peered)
     * no wasted performance (mc-lags with lacp)
   * Hosts: bond (lacp)

[[attachment:setup8.svg|{{attachment:setup8.svg}}]]

Recommended modern approach


== GNS3 ==

 * [[https://www.gns3.com/]]
 * [[https://docs.gns3.com/docs/]]
 * [[https://github.com/GNS3]]
 * [[https://github.com/GNS3/dynamips]]
 * [[https://github.com/GNS3/ubridge]]
 * [[]]
 * [[]]

=== Install ===

Install dependencies
{{{#!highlight bash
apt update
apt install \
        python3-pip python3-pyqt5 python3-pyqt5.qtsvg \
        python3-pyqt5.qtwebsockets \
        qemu qemu-system-x86 qemu-utils libvirt-clients libvirt-daemon-system virtinst \
        wireshark xtightvncviewer apt-transport-https \
        ca-certificates curl gnupg2 software-properties-common
}}}

dynamips - Cisco 7200/3600/3725/3745/2600/1700 Router Emulator and

uBridge is a simple application to create user-land bridges between various technologies.
Currently bridging between UDP tunnels, Ethernet and TAP interfaces is supported. Packet capture is also supported.

Import public key
{{{#!highlight bash
apt-key adv \
        --keyserver keyserver.ubuntu.com \
        --recv-keys F88F6D313016330404F710FC9A2FD067A2E3EF7B
}}}

{{{/etc/apt/sources.list.d/gns3.list}}}
{{{#!highlight bash
deb     http://ppa.launchpad.net/gns3/ppa/ubuntu focal main
deb-src http://ppa.launchpad.net/gns3/ppa/ubuntu focal main

deb     http://ppa.launchpad.net/gns3/ppa/ubuntu groovy main
deb-src http://ppa.launchpad.net/gns3/ppa/ubuntu groovy main
}}}

{{{/etc/apt/preferences.d/gns3}}}
{{{#!highlight bash
Package:        *
Pin:            origin ppa.launchpad.net
Pin-Priority:   101
}}}

Install ubridge and dynamips
(adjust ist to use the most recent version)
{{{#!highlight bash
aptitude install \
        gns3-server gns3-gui gns3-webclient-pack \
        dynamips ubridge
}}}

Join the groups
{{{#!highlight bash
for GROUP in kvm libvirt docker ubridge wireshark; do
        adduser tobias $GROUP;
done
}}}

=== Configuration ===

==== Change default console ====

To allow resizing the font,
i thought about changing the terminal emulator.

Edit > Preferences
General > Console Applications (TAB)
{{{#!highlight bash
### DEFAULT
xterm -T "%d" -e "telnet %h %p"
## PLASMA KONSOLE
konsole -e "telnet %h %p"
}}}