OPNsense
Contents
-
OPNsense
- About
- Important notice
- Essential packages
- Preparing a USB-Stick
- USB-Stick with config
- Extensions
- Aliases and names
- DNS rebind attack prevention
- Firewall services
- Show device information
- IPMI configuration
- ON VMware
- Wireguard
- siproxd
- MultiWAN
- DNS
- configd
- Upgrades
- WAF
- FreeBSD
- Downgrade a package
- OPNcentral
About
OPNsense® is an open source, easy-to-use and easy-to-build HardenedBSD based firewall and routing platform.
OPNsense includes most of the features available in expensive commercial firewalls, and more in many cases. It brings the rich feature set of commercial offerings with the benefits of open and verifiable sources.
Important notice
When you virtualized the OPNsense appliance, you can easily attach and detach NICs, maybe even live. But you need to keep an eye on the interface order. OPNsense does not determine its interfaces based on MAC addresses, it uses device names. This means the order of the physical interfaces matters a lot!
Imagine you remove the interface vtnet0 from the OPNsense, than any interface name is decremented by one and every OPNsense interface will be assigned a wrong physical (para-virtualized) interface.
You may escape this situation only by logging into a TTY and create an intermediate configuration that allows network connectivity on the shell. So make sure you have shell access and can elevate privileges to configure the network! As an example you may take down an IP from a interface and configure it on another interface. Here is a simple example:
To disable the firewall and access the web-interface you can disable the firewalling entirely with
pfctl -d
Essential packages
Install some essential packages
Preparing a USB-Stick
- Download the compressed image from
Plugin the USB-stick and determine the device with lsblk.
- Unzip the image and write it to the stick on the fly.
USB-Stick with config
Practice your recovery skills!1!!
Computers will fail. It's only a matter of time. :-D
Plugin the USB-stick and determine the device with lsblk.
- Create a new GPT partitioning table
- Create a vfat filesystem (with an all-caps LABEL)
1 mkfs.vfat -n CONFIG /dev/sdc1
- Mount and prepare recovery config
When booting interrupt the processes when asked and enter the device name from the list (like da0). OPNsense will then mount the filesystem and load the config. Test the firewall config before continuing it is possible that you imported the config from the live medium.
When entering a password please mind that you will probably use a en_US keyboard layout.
To login in as root or installer the root password of the old config is needed. It is easier if you know the root password of the config to be imported. But if you have a login with sudo capabilities. You may login to the shell, change the passwords at runtime and exit the shells to return to the login prompt.
Now you may login to the installer with the new and known password.
The boot order may have changed during the installation. :-D
You'll have to install previous extensions after installation.
Extensions
There are some realy nice extensions
- os-git-backup Track config changes using git
- os-lldpd LLDP allows you to know exactly on which port is a server
- os-smart SMART tools
os-wireguard WireGuard VPN service
Themes
- os-theme-cicada The cicada theme - dark grey
- os-theme-rebellion A suitably dark theme
- os-theme-tukan The tukan theme - blue/white
- os-theme-vicuna The vicuna theme - dark anthrazit
Aliases and names
I've introduced some "rules" for myself to make distributed firewalling more understandable.
- Prefix your interfaces with a string that is specific for your firewall instance
(like DC1_) to make your names more readable. This leads to automatically created aliases that are named "DC1_DMZ network" or "DC1_WAN address", which are also unique on multiple firewalls.
- You shouldn't create aliases for locally networks, that have automatically created aliases. This may leads to situations, where an interface network changes but the corresponding firewall ruleset does not and the firewall blocks legitimate traffic. In any case this increases effort for ruleset maintenance.
- Use the description field for the fqdn/hostname, when creating an alias for a host, when possible.
DNS rebind attack prevention
Just add the hostnames, which should be allowed to
System: Settings: Administration -> Alternative Hostnames
System: Einstellungen: Verwaltung -> Alternative Hostnamen
Firewall services
Firewall offers:
Protocol |
Port |
Service |
Description |
tcp |
22 |
ssh |
Secure Shell |
udp |
53 |
unbound |
Domain Name Resolution |
tcp |
53 |
unbound |
Domain Name Resolution |
tcp |
953 |
unbound |
rndc Dynamic DNS |
udp |
123 |
ntpd |
Network Time Synchronization |
tcp |
80 |
lighttpd |
Webinterface (http) |
tcp |
443 |
lighttpd |
Webinterface (https) |
udp |
546 |
dhcp6c |
DHCP Client |
udp |
547 |
dhcpd |
DHCP Server |
udp |
500 |
charon |
IKEv2 |
udp |
4500 |
charon |
IPsec |
udp |
1194 |
openvpn |
OpenVPN Server |
Show listening ports
1 sockstat -l
Show device information
Show PCI information of the NICs
Example from Supermicro X11SDV-4C-TP8F
pciconf -lv |grep -A1 -B3 network
1 igb0@pci0:101:0:0: class=0x020000 rev=0x01 hdr=0x00 vendor=0x8086 device=0x1521 subvendor=0x15d9 subdevice=0x1521
2 vendor = 'Intel Corporation'
3 device = 'I350 Gigabit Network Connection'
4 class = network
5 subclass = ethernet
6 igb1@pci0:101:0:1: class=0x020000 rev=0x01 hdr=0x00 vendor=0x8086 device=0x1521 subvendor=0x15d9 subdevice=0x1521
7 vendor = 'Intel Corporation'
8 device = 'I350 Gigabit Network Connection'
9 class = network
10 subclass = ethernet
11 igb2@pci0:101:0:2: class=0x020000 rev=0x01 hdr=0x00 vendor=0x8086 device=0x1521 subvendor=0x15d9 subdevice=0x1521
12 vendor = 'Intel Corporation'
13 device = 'I350 Gigabit Network Connection'
14 class = network
15 subclass = ethernet
16 igb3@pci0:101:0:3: class=0x020000 rev=0x01 hdr=0x00 vendor=0x8086 device=0x1521 subvendor=0x15d9 subdevice=0x1521
17 vendor = 'Intel Corporation'
18 device = 'I350 Gigabit Network Connection'
19 class = network
20 subclass = ethernet
21 --
22 ixl0@pci0:181:0:0: class=0x020000 rev=0x04 hdr=0x00 vendor=0x8086 device=0x37d2 subvendor=0x15d9 subdevice=0x37d2
23 vendor = 'Intel Corporation'
24 device = 'Ethernet Connection X722 for 10GBASE-T'
25 class = network
26 subclass = ethernet
27 ixl1@pci0:181:0:1: class=0x020000 rev=0x04 hdr=0x00 vendor=0x8086 device=0x37d2 subvendor=0x15d9 subdevice=0x37d2
28 vendor = 'Intel Corporation'
29 device = 'Ethernet Connection X722 for 10GBASE-T'
30 class = network
31 subclass = ethernet
32 ixl2@pci0:181:0:2: class=0x020000 rev=0x04 hdr=0x00 vendor=0x8086 device=0x37d0 subvendor=0x15d9 subdevice=0x37d0
33 vendor = 'Intel Corporation'
34 device = 'Ethernet Connection X722 for 10GbE SFP+'
35 class = network
36 subclass = ethernet
37 ixl3@pci0:181:0:3: class=0x020000 rev=0x04 hdr=0x00 vendor=0x8086 device=0x37d0 subvendor=0x15d9 subdevice=0x37d0
38 vendor = 'Intel Corporation'
39 device = 'Ethernet Connection X722 for 10GbE SFP+'
40 class = network
41 subclass = ethernet
Show description and firmware version NICs
Extract further device information
1 sysctl -a \
2 |grep -E 'dev.(igb|ix|em).*.(%desc|fw_version)' \
3 |sort -n
4 dev.igb.0.%desc: Intel(R) I350 (Copper)
5 dev.igb.0.fw_version: EEPROM V1.63-0 eTrack 0x800009fa
6 dev.igb.1.%desc: Intel(R) I350 (Copper)
7 dev.igb.1.fw_version: EEPROM V1.63-0 eTrack 0x800009fa
8 dev.igb.2.%desc: Intel(R) I350 (Copper)
9 dev.igb.2.fw_version: EEPROM V1.63-0 eTrack 0x800009fa
10 dev.igb.3.%desc: Intel(R) I350 (Copper)
11 dev.igb.3.fw_version: EEPROM V1.63-0 eTrack 0x800009fa
12 dev.ixl.0.%desc: Intel(R) Ethernet Connection X722 for 10GBASE-T - 2.3.2-k
13 dev.ixl.0.fw_version: fw 4.1.59148 api 1.9 nvm 4.11 etid 80002044 oem 1.265.0
14 dev.ixl.1.%desc: Intel(R) Ethernet Connection X722 for 10GBASE-T - 2.3.2-k
15 dev.ixl.1.fw_version: fw 4.1.59148 api 1.9 nvm 4.11 etid 80002044 oem 1.265.0
16 dev.ixl.2.%desc: Intel(R) Ethernet Connection X722 for 10GbE SFP+ - 2.3.2-k
17 dev.ixl.2.fw_version: fw 4.1.59148 api 1.9 nvm 4.11 etid 80002044 oem 1.265.0
18 dev.ixl.3.%desc: Intel(R) Ethernet Connection X722 for 10GbE SFP+ - 2.3.2-k
19 dev.ixl.3.fw_version: fw 4.1.59148 api 1.9 nvm 4.11 etid 80002044 oem 1.265.0
There is much more to harvest in sysctl -a.
Show connected media
/usr/local/sbin/ifmedia.sh
Set script execuable
chmod a+x /usr/local/sbin/ifmedia.sh
ifmedia.sh
1 igb0: media: Ethernet autoselect (1000baseT <full-duplex>)
2 igb1: media: Ethernet autoselect (1000baseT <full-duplex>)
3 igb2: media: Ethernet autoselect
4 igb3: media: Ethernet autoselect
5 ixl0: media: Ethernet autoselect
6 ixl1: media: Ethernet autoselect
7 ixl2: media: Ethernet autoselect
8 ixl3: media: Ethernet autoselect
9 lo0:
10 enc0:
11 pflog0:
12 pfsync0:
13 lagg0: media: Ethernet autoselect
14 lagg0_vlan2: media: Ethernet autoselect
15 lagg0_vlan6: media: Ethernet autoselect
16 lagg0_vlan9: media: Ethernet autoselect
17 lagg0_vlan12: media: Ethernet autoselect
18 lagg0_vlan14: media: Ethernet autoselect
19 lagg0_vlan1: media: Ethernet autoselect
20 lagg0_vlan3: media: Ethernet autoselect
21 ovpns1:
IPMI configuration
Thomas-Krenn Wiki - IPMI Konfiguration unter Linux mittels ipmitool
Please also see:
man ipmitool
To configure IPMI from the OS level gain elevated privileges and load the kernel module.
List channels
1 foreach CHANNEL ( `seq 0 1 5` )
2 echo "Channel: '$CHANNEL'"
3 ipmitool channel info "$CHANNEL" |sed 's/ +$//'
4 end
5 Channel: '0'
6 Channel 0x0 info:
7 Channel Medium Type : IPMB (I2C)
8 Channel Protocol Type : IPMB-1.0
9 Session Support : session-less
10 Active Session Count : 0
11 Protocol Vendor ID : 7154
12 Channel: '1'
13 Channel 0x1 info:
14 Channel Medium Type : 802.3 LAN
15 Channel Protocol Type : IPMB-1.0
16 Session Support : multi-session
17 Active Session Count : 0
18 Protocol Vendor ID : 7154
19 Volatile(active) Settings
20 Alerting : enabled
21 Per-message Auth : enabled
22 User Level Auth : enabled
23 Access Mode : always available
24 Non-Volatile Settings
25 Alerting : enabled
26 Per-message Auth : enabled
27 User Level Auth : enabled
28 Access Mode : always available
29 Channel: '2'
30 IPMI command failed: Invalid data field in request
31 Unable to Get Channel Info
32 Channel: '3'
33 Channel 0x3 info:
34 Channel Medium Type : Serial/Modem
35 Channel Protocol Type : IPMB-1.0
36 Session Support : single-session
37 Active Session Count : 0
38 Protocol Vendor ID : 7154
39 Channel: '4'
40 IPMI command failed: Invalid data field in request
41 Unable to Get Channel Info
42 Channel: '5'
43 Channel 0x5 info:
44 Channel Medium Type : IPMB (I2C)
45 Channel Protocol Type : IPMB-1.0
46 Session Support : session-less
47 Active Session Count : 0
48 Protocol Vendor ID : 7154
Show IPMI lan configuration The channel is "1" but may be omitted, here.
1 ipmitool lan print
2 Set in Progress : Set Complete
3 Auth Type Support : NONE MD2 MD5 PASSWORD
4 Auth Type Enable : Callback : MD2 MD5 PASSWORD
5 : User : MD2 MD5 PASSWORD
6 : Operator : MD2 MD5 PASSWORD
7 : Admin : MD2 MD5 PASSWORD
8 : OEM : MD2 MD5 PASSWORD
9 IP Address Source : DHCP Address
10 IP Address : 0.0.0.0
11 Subnet Mask : 0.0.0.0
12 MAC Address : 3c:ec:ef:12:34:56
13 SNMP Community String : public
14 IP Header : TTL=0x00 Flags=0x00 Precedence=0x00 TOS=0x00
15 BMC ARP Control : ARP Responses Enabled, Gratuitous ARP Disabled
16 Default Gateway IP : 192.168.1.1
17 Default Gateway MAC : 00:00:00:00:00:00
18 Backup Gateway IP : 0.0.0.0
19 Backup Gateway MAC : 00:00:00:00:00:00
20 802.1q VLAN ID : Disabled
21 802.1q VLAN Priority : 0
22 RMCP+ Cipher Suites : 1,2,3,6,7,8,11,12
23 Cipher Suite Priv Max : XaaaXXaaaXXaaXX
24 : X=Cipher Suite Unused
25 : c=CALLBACK
26 : u=USER
27 : o=OPERATOR
28 : a=ADMIN
29 : O=OEM
30 Bad Password Threshold : 3
31 Invalid password disable: yes
32 Attempt Count Reset Int.: 300
33 User Lockout Interval : 300
Alter BMC LAN configuration
Interface assignments
IP interfaces can easily be moved between different physically or logical devices using the webinterface. But please keep in mind, that CARP VIPs may not migrate properly, when reassigning the devices.
In my specific case the LAN interface was moved from igb0 to VLAN1 on LAGG1. But the carp VIP stayed on igb0. Therefor the routing table still pointed igb0 for the migrated network and frames and packets were send out to the wrong interface (like ARP).
ON VMware
CARP
When using VMware with CARP.
Allow "Forged Transmits"/"Gefälschte Übertragungen" in the port-group.
This allows frames with a crafted MAC like 00:00:5e:00:01:14 to be forwarded over the vSwitch (and reach the backup node). Otherwise both firewalls are in state "master" (split-brain).
Allow "Promiscuous Mode" in the port-proup.
- To receive frames for the crafted CARP-MAC the OPNsense need to put the respective NIC into promiscuous mode (within the guest). This makes only sense if the VMware port-group also allows promiscuous mode.
Enable Net.ReversePathFwdCheckPromisc=1 on you hosts
$HOSTNAME > Configure > Advanced Settings -> Edit (Button) Please see: https://kb.vmware.com/s/article/59235
- It is not necessary to reboot the host. It's sufficient to reboot the guest or (live) migrate the guest to ahost with the option enabled.
Rationale:
When using a VMware vSwitch with multiple uplinks without Bonding (like LACP), multicast and broadcast frames (BUM-traffic) are flooded back to the vSwitch. The OPNsense that just got "master" and sends out its keep-alives, receives its own frames and falls back to state "backup". The IP-interface is flapping up and down in the log.
Please check: tail -f /var/log/system/latest.log
The webguis will show both firewalls in state "backup".
- Allow "CARP" in the firewall ruleset on the respective interface.
Install VMware-tools
Install extension os-vmware.
Wireguard
If you have important site-to-site connections running that should not be interrupted, you should establish separate listeners. This avoids unnecessary down-times, because misconfiguration happens and the wireguard interface won't come up until the problem has been fixed.
The Wireguard interface may not get up if
- the destination address is not resolveable
- Keep that in mind when forwarding DNS-traffic over wireguard
- the Wireguard "allowed_ip" address collides with a local address
For the Wireguard interfaces to be query-able by DNS, a access list has to be added to the Unbound service.
SERVICES: UNBOUND DNS: ACCESS LISTS
OPNsense creates a firewall interface group "WireGuard (group)". So there's no need to manually create one. The processing order can be looked up here.
OPNsense Docs - Firewall Processing order
Basically:
- Floating
- Group
- Interface
siproxd
Siproxd is only a proxy for old PBXs, which strictly require source port 5060. With siproxd multiple phones can connect to the remote PBX over a Masquerading firewall.
Please see:
You probably don't need this plugin.
MultiWAN
Notes
- A load-balancing setup with multiple Gateways in Tier 1, does not cope well with sticky connections. Random Websites do not load.
With tcpdump you can see a packet coming in to LAN, but not leaving on WAN1 or WAN2.
- Be careful with groups when using MultiWAN.
DNS
Dynamic DNS by DHCP
Works like a charm!
Services: Unbound DNS: General
Register DHCP leases: trueServices: Unbound DNS: General
Register DHCP static mappings: trueServices: Unbound DNS: General
Register DHCP leases: trueIn the zone e.g. Services: DHCPv4: [LAN]
Dynamic DNS -> Enable registration of DHCP client names in DNS: trueIn the zone e.g. Services: DHCPv4: [LAN]
Domain name: intern.dezentrale.space
configd
List configd actions
1 configctl configd actions list
Restart ssh-daemon
1 configctl openssh restart
Upgrades
Security Updates
The bi-weekly security upgrades should be installed automatically using the cron-job Automatic firmware updates, which is factory provided under
System: Settings: Cron.
This is generally a good idea!
You should think a about redundancy and fast storage (like NVME-SSDs).
Major Upgrades
OPNsense demands for a major upgrade twice a year. Please see:
Multiple Major Upgrades
First step is always to backup your configuration.
- Perform the upgrades serially in order.
Latest security release -> major upgrade -> Latest security release -> major upgrade (-> Latest security release -> major upgrade -> …)
- Use a live medium
- Create a live medium with the target release
- boot the live-medium and load the config during startup
- persist the config when it performs well
ALLOW_RISKY_MAJOR_UPGRADE
If you want to drive firewall in a more hazardous way, there is also a way to automate upgrades across major releases
OPNsense Release Note 17.1 “Eclectic Eagle” 17.1.11 (July 25, 2017)
https://github.com/opnsense/core/commit/93072dd807d0115a0586f9cb6345079e930e174f
firmware: cron-parameter ALLOW_RISKY_MAJOR_UPGRADE for auto-update
There is now a switch for the brave, which allows for targeted major release upgrades and release name agnostic major upgrades. Please see:
/usr/local/etc/rc.firmware.subr
So, it basically allows for timed and generally automatic major upgrades.
Perform a targeted upgrade
ALLOW_RISKY_MAJOR_UPGRADE
Use on your own risk.
OPNsense calls it a support nightmare. :-D
ALLOW_RISKY_MAJOR_UPGRADE Cron Schedule
How should a cron schedule look like for usage with ALLOW_RISKY_MAJOR_UPGRADE?
WARNING!
This is a first untested attempt to a definition.
All untested. You have been warned. No liability. ;-)
Assumptions:
- A major upgrade also installs the latest security release and updates plugins. TODO: Proof this.
Constraints:
- Primary is im CARP preemption mode
- Secondary prior to primary
The upgrade performs a random delay of up to -r 1500s (25min) when updating.
- Which can probably be changed
by adding another option -r 1 or whatever value you have in mind to parameters
- Which can probably be changed
- The upgrades of primary and secondary should not overlap, to avoid outage. There should be some margin.
- Upgrade time depends on resources (especially on fast CPU and storage)
- Daily security upgrades are much faster than major upgrades.
- Daily security upgrades may end up in a reboot.
- Major upgrades end up in a reboot.
- The time specification is bound to the cron format
- OPNsense relies on cron written by Paul Vixie
- There is a specialty when limiting both DoM and DoW
they are processed with logical OR instead of AND
Please see man 5 crontab for datails of the time spec. Short version:
# m h tdm mon tdw
- Firewall upgrades and reboots should not interfere with production and backup window.
- Primay and secondary should not be updated on the same day.
- IMHO friday, saturday and sunday
are not a good idea for ALLOW_RISKY_MAJOR_UPGRADE – poor people on standby.
node |
cron timespec |
command |
Secondary Standby |
0 1 * * * |
Firmware update check |
Primary Active |
0 2 * * * |
Firmware update check |
Secondary Standby |
5 1 * * * |
Automatic firmware update |
Primary Active |
5 2 * * * |
Automatic firmware update |
Secondary Standby |
5 3 1-7,15-21 * */4 |
ALLOW_RISKY_MAJOR_UPGRADE |
Primary Active |
5 3 8-14,22-31 * */4 |
ALLOW_RISKY_MAJOR_UPGRADE |
The meaning:
- Firmware update check prior to everything.
- Automatic firmware update with one hour delay between nodes and an offset of 5min to Firmware update check.
- Major upgrades on alternating weeks and if it's every 4th day of the week (thursday).
WAF
The WAF plugin is based on nginx with the naxsi (Nginx Anti XSS & SQL Injection) plugin.
- Please keep in mind,
that if no default_server is specified, the first server is chosen as the default server.
Nginx docs - How nginx processes a request
FreeBSD
Change keyboard mapping
Please see this great post
Tobias Koschinski - FreeBSD: Set keyboard layout
To change the keyboard layout
You may enable the wiki comments to view a list of layouts
Scrolling
On the TTY please use Scroll Lock and navigate with PageUp/Down or the arrow keys Up/Down.
In Linux you usually would use Shift + PageUP/Down or Shift + Up/Down.
Show device information
Show device info via (Common Access Method)
1 camcontrol identify /dev/ada0
Downgrade a package
Here on the example of nginx
1 opnsense-revert -r 22.1.8 nginx
OPNcentral
- OPNcentral ist eine Art Plugin für OPNsense zur Verwaltung mehrerer Firewalls
- Business License auf OPNcentral und angeschlossenen Clients (zwei für einen Cluster) notwendig.
- Getestete Upgrades
- Unterstützung der Entwicklung
- Recht beschränkt
Features
- Zentraler Status
- Ressourcen Verbrauch
- von laufenden Diensten
- Zentrale Firmware Upgrades
- OPNcentral muss eine neuere Version als der Client haben
- Achtung bei produktiven Systemen
- Synchronisiserung von
- Sollte selektiv ausgewählt werden
- Sinvoll sind: Aliases, Authentisierungsserver, Zertifikate, Floating ACLs, …
- Lokale nicht zentral verwaltete Aliase werden gelöscht?