• foreman

foreman

### WIP

About

Foreman is a complete lifecycle management tool for physical and virtual servers.

• Provision from anywhere

  • Bare metal (MaaS), Amazon EC2, Google Compute Engine, OpenStack, Libvirt, oVirt, VMware, and many other providers allow you to manage a hybrid cloud through Foreman.

• Configuration
  • An external node classifier, hiera-like parameters, and reports monitoring for Puppet, Salt and Chef are included. Completely ready to tweak host groups in your data center.
• Monitor hosts
  • Foreman reports will tell you exactly what happened in your nodes, and alert you when things go awry. See in your dashboard which hosts are healthy, and which ones are outdated.

Links

• https://www.theforeman.org/

• https://docs.theforeman.org/

• Foreman.org - Quick Start <- Use this guide

• docs.theforeman.org - Provisioning_Guide

• Foreman Redmine

• Foreman Forum

Prerequisites

• Lot's of patience and ability to suffer and fail hard.
• Use a Red Hat derivation as base OS, because the variety of plugins available is much richer.

  • docs.theforeman.org - Installing Foreman nightly server with Katello nightly plugin on Enterprise Linux #Supported Operating Systems

• You'll need quite some resources:
  • CPU: 4
  • RAM: At least 8GiB
  • Storage:

    • RootFS /: at least 20GiB

    • Variable data /var: at least 64GiB

  • NICs: There will be some traffic, when you are managing the whole provisioning with foreman. In a virtualized environment the 10Gbit/s virtio is quite okay. But think about attaching foreman to the network in a performant way. There should be at least 4Gbit/s available on the machine. It's an important system, use LACP-Bonds or other measures to facilitate more than one interface for redundancy and performance.
• Plan the networking layout
  • Zoning, IP addressing, DNS, Naming
• The foreman-installer needs puppet.

  • With CentOS 7/8 (Stream) you may install puppet from
    https://yum.puppet.com/

  • With Debian 10/11 you may install puppet from
    https://apt.puppet.com/

Installation

Prepare the installation

Firewalling

Configure the firewall for Foreman and its proxies.

Correct /etc/hosts

Bind hostname to correct interface in
/etc/hosts

   1 #127.0.1.1      foreman1.server.dezentrale.space        foreman1
   2 192.168.12.3    foreman1.server.dezentrale.space        foreman1

Repositories and dependencies on Debian 11

Stop right here and use CentOS 8 Stream.

Create a VM snapshot.

Make puppet available as described in the Quick Start.

Enable Puppet's 7.x repository

   1 apt-get -y install ca-certificates
   2 cd /tmp && wget https://apt.puppet.com/puppet7-release-bullseye.deb
   3 apt-get install /tmp/puppet7-release-bullseye.deb

Enable the Foreman repositories

   1 sudo wget https://deb.theforeman.org/foreman.asc -O /etc/apt/trusted.gpg.d/foreman.asc
   2 echo "deb http://deb.theforeman.org/ bullseye 3.2" | sudo tee /etc/apt/sources.list.d/foreman.list
   3 echo "deb http://deb.theforeman.org/ plugins 3.2" | sudo tee -a /etc/apt/sources.list.d/foreman.list
   4 apt-get update

Install the foreman-installer

   1 apt install foreman-installer

Repositories and dependencies on CentOS 8 Stream

Create a VM snapshot.

Make puppet available as described in the Quick Start.

Enable Puppet's 7.x repository

   1 dnf install https://yum.puppet.com/puppet7-release-el-8.noarch.rpm

Enable module ruby 2.7

   1 dnf module reset ruby
   2 dnf module enable ruby:2.7

Enable the Foreman, Katello and Ansible repositories

   1 dnf install https://yum.theforeman.org/releases/3.2/el8/x86_64/foreman-release.rpm
   2 dnf install https://yum.theforeman.org/katello/4.4/katello/el8/x86_64/katello-repos-latest.rpm
   3 dnf install centos-release-ansible-29

Install the foreman-installer

   1 dnf config-manager --set-enabled powertools
   2 dnf module enable postgresql:13
   3 dnf install foreman-installer-katello

Create and distribute ssh pubkeys

Generate a ssh-key for root and distribute it to the proxies

   1 ssh-keygen -b 4096

Install Foreman

Install Foreman with the plugin Katello

   1 foreman-installer -l INFO -i -v --foreman-hsts-enabled=false
   2 
   3 foreman-installer -l INFO -i -v \
   4         --scenario katello \
   5         --enable-foreman-plugin-ansible \
   6         --enable-foreman-plugin-discovery \
   7         --enable-foreman-plugin-setup

Wait for a loong time …
and Don't forget to note the final instructions of the foreman-installer

   1 …
   2   * Foreman is running at https://foreman1.server.dezentrale.space
   3       Initial credentials are admin / __secret_initial_password
   4   * To install an additional Foreman proxy on separate machine continue by running:
   5 
   6       foreman-proxy-certs-generate --foreman-proxy-fqdn "$FOREMAN_PROXY" --certs-tar "/root/$FOREMAN_PROXY-certs.tar"
   7   * Foreman Proxy is running at https://foreman1.server.dezentrale.space:9090
   8 
   9   The full log is at /var/log/foreman-installer/katello.log
  10 2022-06-06 16:54:58 [INFO  ] [post] All hooks in group post finished
  11 foreman-installer -l INFO -i -v --scenario katello     899,57s user 103,08s system 53% cpu 31:03,32 total

Configure the host firewall

Allow queries to the web-interface.

   1 firewall-cmd --permanent --add-service=http{,s}

Install Foreman proxy

Foreman cannot handle multiple proxy.urls. For more information please see
Foreman - Provisioning Guide 7.4. Discovery Templates and Snippets Settings - Paragraph Rendering the Smart Proxy’s Host Name
Therefor the smart proxy should be named equally in every subnet - something like foreman-proxy or fproxy.

Make sure your foreman server can resolve your proxy!

Generate a certificate for the smart proxy

   1 /opt/puppetlabs/bin/puppetserver ca generate --certname fproxy.hw4f.dezentrale.space 
   2 Successfully saved private key for fproxy.hw4f.dezentrale.space to /etc/puppetlabs/puppet/ssl/private_keys/fproxy.hw4f.dezentrale.space.pem
   3 Successfully saved public key for fproxy.hw4f.dezentrale.space to /etc/puppetlabs/puppet/ssl/public_keys/fproxy.hw4f.dezentrale.space.pem
   4 Successfully submitted certificate request for fproxy.hw4f.dezentrale.space
   5 Error:
   6     Signed certificate fproxy.hw4f.dezentrale.space could not be found on the CA
   7 Successfully signed certificate request for fproxy.hw4f.dezentrale.space
   8 Successfully saved certificate for fproxy.hw4f.dezentrale.space to /etc/puppetlabs/puppet/ssl/certs/fproxy.hw4f.dezentrale.space.pem

Copy the files to the smart proxy

   1 for KEY in \
   2         /etc/puppetlabs/puppet/ssl/{private_keys,certs}/fproxy.hw4f.dezentrale.space.pem \
   3         /etc/puppetlabs/puppet/ssl/certs/ca.pem;
   4 do
   5         scp "$KEY" "root@192.168.33.3:$KEY" 
   6 done                                                                                                                                  

Adjust owner and group of the files.

   1 chown foreman-proxy:foreman-proxy \
   2         /etc/puppetlabs/puppet/ssl/certs/foreman-proxy-hw4f1.hw4f.dezentrale.space.pem
   3         /etc/puppetlabs/puppet/ssl/private_keys/foreman-proxy-hw4f1.hw4f.dezentrale.space.pem

Also select foreman_plugin_setup and foreman_plugin_discovery.

Get the oauth key and secret of your foreman.

   1 foreman-rake config |grep oauth_consumer |sed -r 's/(oauth_consumer_.*): (.+)$/ \1=\2/'

Install a foreman smart proxy in the desired network

   1 foreman-installer \
   2   --no-enable-foreman \
   3   --no-enable-foreman-cli \
   4   --no-enable-foreman-plugin-puppet \
   5   --no-enable-foreman-cli-puppet \
   6   --enable-puppet \
   7   --puppet-server-ca=false \
   8   --puppet-server-foreman-url="https://foreman1.server.dezentrale.space" \
   9   --foreman-proxy-puppetca=false \
  10   --foreman-proxy-puppet-group="puppet" \
  11   --foreman-proxy-manage-puppet-group=true \
  12   --foreman-proxy-trusted-hosts="foreman1.server.dezentrale.space" \
  13   --enable-foreman-proxy \  
  14   --foreman-proxy-tftp=true \
  15   --foreman-proxy-tftp-servername=192.168.33.3 \
  16   --foreman-proxy-dhcp=true \
  17   --foreman-proxy-dhcp-interface=ens18 \    
  18   --foreman-proxy-dhcp-gateway=192.168.33.1 \
  19   --foreman-proxy-dhcp-nameservers="192.168.33.3" \
  20   --foreman-proxy-dhcp-range "192.168.33.100 192.168.33.199" \
  21   --foreman-proxy-dns=true \ 
  22   --foreman-proxy-dns-interface=ens18 \
  23   --foreman-proxy-dns-zone=hw4f.dezentrale.space \
  24   --foreman-proxy-dns-reverse=33.168.192.in-addr.arpa \
  25   --foreman-proxy-dns-forwarders=192.168.33.1 \
  26   --foreman-proxy-oauth-consumer-key="$oauth_consumer_key" \
  27   --foreman-proxy-oauth-consumer-secret="$oauth_consumer_secret"

Foreman discovery

Foreman Discovery 18.0 manual

Make sure the discovery image is in place

   1 foreman-installer \
   2   --enable-foreman-proxy-plugin-discovery \
   3   --foreman-proxy-plugin-discovery-install-images=true \
   4   #--foreman-proxy-plugin-discovery-source-url=https://downloads.theforeman.org/discovery/releases/3.5/
   5 

In Administer > Settings > (Tab) Provisioning change Default PXE global template entry to discovery. The global default is empty and the template defaults to local boot.

Katello

Create, organize, and manage local yum, deb, and puppet repositories. Sync remote repositories or upload content directly to build a library of content that serves as the basis for building custom builds of your content.

• https://katello.org/

• theforeman.org - Katello nightly Documentation (slightly outdated)

• github.com Katello/katello

Looks like Katello is only available for Red Hat derivatives.

You can find a subscription-manager for Debian and Ubuntu at
https://apt.atix.de/

File locations

• Provisioning templates

  • Standard (do not modify, as they may be overwritten on update)
    /usr/share/foreman/app/views/unattended/provisioning_templates

Configure

1. Administer > Users

   1. Create your personal user

2. Administer > User Groups

   1. Create user group "admins"
   2. Under roles mark check box "Administrator"
   3. Include your user in the group "admins"

3. Hosts > All hosts

   1. Check the foreman server

4. Infrastructure > Smart Proxies

   1. Make sure a proxy is available for the target network
   2. Check that the desired functions are available (DHCP, DNS, TFTP, Discovery)

5. Infrastructure > Organizations

   1. Create your beloved organization

6. Infrastructure > Locations

   1. Create your beloved location

7. Infrastructure > Domains

   1. Create Domain
   2. Assign DNS Proxy, so A-Records can be deployed on the subnets DNS-Server (if available on the smart proxy for the domain)
   3. Assign to organization and location

8. Configure > (Puppet) Environments

   1. Puppets default is lowercase "production"
   2. Assign to organization and location

9. Infrastructure > Subnets

   1. Create Subnet
   2. Assign DHCP, TFTP, Reverse DNS(, Template) and Discovery Proxy
   3. Assign to organization and location

10. Hosts > Operating Systems

    1. Create the OS you want to provision

11. Configure > Host Group

    1. Create a host group to act as a set of defaults
12. Create host

Trouble shooting

PXE menu entry does not respond

When selecting a PXE menu entry nothing happens.

Probably the corresponding file to downloaded from tftp does not exist.

Smart Proxy shows DHCP related errors on Debian Bullseye

Debian isc-dhcp-server in version 4.4.1-2.3 has a bug.
Debian Bug #995242 - isc-dhcp-server: omshell returns inconsistent results or segfaults

As a workaround upgrade to a version >= 4.4.3-1, e.g. from SID.

Pin SID low
/etc/apt/preferences.d/zz_releases 

   1 Package: *
   2 Pin: release o=Debian,a=unstable
   3 Pin-Priority: 120
   4 
   5 Package: *
   6 Pin: release o=Debian,a=stable
   7 Pin-Priority: 500

Check the priorities

   1 isc-dhcp-server:
   2   Installiert:           4.4.3-2
   3   Installationskandidat: 4.4.3-2
   4   Versionstabelle:
   5  *** 4.4.3-2 500
   6         120 http://ftp2.de.debian.org/debian sid/main amd64 Packages
   7         100 /var/lib/dpkg/status
   8      4.4.1-2.3 500
   9         500 http://ftp2.de.debian.org/debian bullseye/main amd64 Packages

And install isc-dhcp-server from SID using aptitude. You will have to upgrade glibc dependencies, too.