LetsEncrypt

Contents

LetsEncrypt
1. About
2. DNS Certification Authority Authorization (CAA) Resource Record
Certbot

About

Let’s Encrypt is a free, automated, and open certificate authority (CA), run for the public’s benefit. It is a service provided by the Internet Security Research Group (ISRG).

We give people the digital certificates they need in order to enable HTTPS (SSL/TLS) for websites, for free, in the most user-friendly way we can. We do this because we want to create a more secure and privacy-respecting Web.

You can read about our most recent year in review by downloading our annual report.

The key principles behind Let’s Encrypt are:

Free: Anyone who owns a domain name can use Let’s Encrypt to obtain a trusted certificate at zero cost.
Automatic: Software running on a web server can interact with Let’s Encrypt to painlessly obtain a certificate, securely configure it for use, and automatically take care of renewal.
Secure: Let’s Encrypt will serve as a platform for advancing TLS security best practices, both on the CA side and by helping site operators properly secure their servers.
Transparent: All certificates issued or revoked will be publicly recorded and available for anyone to inspect.
Open: The automatic issuance and renewal protocol is published as an open standard that others can adopt.
Cooperative: Much like the underlying Internet protocols themselves, Let’s Encrypt is a joint effort to benefit the community, beyond the control of any one organization.

We have a page with more detailed information about how the Let’s Encrypt CA works.

https://letsencrypt.org/
About Let's Encrypt
Let's Encrypt Challenge-types
https://www.abetterinternet.org/ Homepage der Internet Security Research Group (ISRG)

Thanks a lot for this service making this Internet a bit more secure and trustworthy.

DNS Certification Authority Authorization (CAA) Resource Record

DNS#DNS Certification Authority Authorization (CAA) Resource Record

Certbot

About

https://certbot.eff.org/

Protocols

ACME v1 (Deprecated in June 2020)
ACME v2 (RFC 8555)

LetsEncypt source IP addresses

This two links state, that in order to enhance the security of the challenge multiple addresses are used to verify the request. A list of source ip addresses is not provided to make it even harder. This means that a open and public connection needs to be established to use LetsEncrypt.

If a public connection is not necessary for other reasons, i would suggest to not use LetsEncrypt and instead use a certificate of another certificate authority. This allows valid cryptography and does not expose the service to the internet.

Installation

As standalone (e.g. postfix without webserver)

   1 aptitude install ssl-cert certbot python-certbot-apache

For apache2

   1 aptitude install ssl-cert certbot python-certbot-apache

For nginx

   1 aptitude install ssl-cert certbot python-certbot-nginx

Configuration

Prepare DNS records first! This may be an A or CNAME record.

Retrieve certificate for postfix

   1 certbot certonly --standalone \
   2         --rsa-key-size 4096 \
   3         -m "hostmaster@rockstable.it" \
   4         -d "mx1.rockstable.it" \
   5         --no-eff-email \
   6         --agree-tos

Retrieve certificate for apache2

   1 certbot certonly --apache \
   2         --rsa-key-size 4096 \
   3         -m "hostmaster@rockstable.it" \
   4         -d "mx1.rockstable.it" \
   5         -d "mail.rockstable.it" \
   6         --no-eff-email \
   7         --agree-tos
   8 Saving debug log to /var/log/letsencrypt/letsencrypt.log
   9 Plugins selected: Authenticator apache, Installer apache
  10 Obtaining a new certificate
  11 Performing the following challenges:
  12 http-01 challenge for mail.rockstable.it
  13 http-01 challenge for mx1.rockstable.it
  14 Waiting for verification...
  15 Cleaning up challenges
  16 
  17 IMPORTANT NOTES:
  18  - Congratulations! Your certificate and chain have been saved at:
  19    /etc/letsencrypt/live/mx1.rockstable.it/fullchain.pem
  20    Your key file has been saved at:
  21    /etc/letsencrypt/live/mx1.rockstable.it/privkey.pem
  22    Your cert will expire on 2019-09-06. To obtain a new or tweaked
  23    version of this certificate in the future, simply run certbot
  24    again. To non-interactively renew *all* of your certificates, run
  25    "certbot renew"
  26  - If you like Certbot, please consider supporting our work by:
  27 
  28    Donating to ISRG / Let's Encrypt:   https://letsencrypt.org/donate
  29    Donating to EFF:                    https://eff.org/donate-le

Retrieve certificate for nginx

   1 certbot certonly --nginx \
   2         --rsa-key-size 4096 \
   3         -m "hostmaster@rockstable.it" \
   4         -d "git.rockstable.it" \
   5         -d "www3.rockstable.it" \
   6         --no-eff-email \
   7         --agree-tos
   8 Saving debug log to /var/log/letsencrypt/letsencrypt.log
   9 Plugins selected: Authenticator nginx, Installer nginx
  10 Obtaining a new certificate
  11 Performing the following challenges:
  12 http-01 challenge for git.rockstable.it
  13 http-01 challenge for www3.rockstable.it
  14 Waiting for verification...
  15 Cleaning up challenges
  16 
  17 IMPORTANT NOTES:
  18  - Congratulations! Your certificate and chain have been saved at:
  19    /etc/letsencrypt/live/git.rockstable.it/fullchain.pem
  20    Your key file has been saved at:
  21    /etc/letsencrypt/live/git.rockstable.it/privkey.pem
  22    Your cert will expire on 2019-10-02. To obtain a new or tweaked
  23    version of this certificate in the future, simply run certbot
  24    again. To non-interactively renew *all* of your certificates, run
  25    "certbot renew"
  26  - Your account credentials have been saved in your Certbot
  27    configuration directory at /etc/letsencrypt. You should make a
  28    secure backup of this folder now. This configuration directory will
  29    also contain certificates and private keys obtained by Certbot so
  30    making regular backups of this folder is ideal.
  31  - If you like Certbot, please consider supporting our work by:
  32 
  33    Donating to ISRG / Let's Encrypt:   https://letsencrypt.org/donate
  34    Donating to EFF:                    https://eff.org/donate-le

Some considerations concerning access

Change group of directories /etc/letsencrypt/{live,archive} to ssl-cert.
Set "Setgid" bit and permit access to group.
```
   1 find /etc/letsencrypt/{live,archive} -type d \
   2         -exec chgrp ssl-cert {} \; \
   3         -exec chmod 2710 {} \;
```
Files in /etc/letsencrypt/$DOMAIN are only linked relatively to files in ../../archive/$DOMAIN. So it is sufficient to change permissions of /etc/letsencrypt/archive/$DOMAIN/*.
Change group of private keys in /etc/letsencrypt/archive to "ssl-cert" and permit read-access to group.
```
   1 find /etc/letsencrypt/archive/ -type f -name 'privkey*.pem' \
   2         -exec chgrp ssl-cert {} \; \
   3         -exec chmod 640 {} \;
```
Add all daemons that don't have a "primary/master"
process running under uid "0" to group ssl-cert to allow access to certificates.
```
   1 adduser daemon-user ssl-cert
```

For example it's unnecessary to add www-data, postfix and dovecot, since they have a "master" process running under "root" and "root" can do anything.

   1 ps faux \
   2         | egrep -A 3 'root.*(/usr/sbin/apache2|/usr/lib/postfix/sbin/master \
   3         |/usr/sbin/dovecot)' \
   4         | grep -v grep
   5 root       453  0.0  0.0   4308  3200 ?        Ss   Jun06   0:00 /usr/sbin/dovecot -F
   6 dovecot    802  0.0  0.0   3952  1096 ?        S    Jun06   0:00  \_ dovecot/anvil
   7 root     13273  0.0  0.0   4328  3176 ?        S    Jun07   0:00  \_ dovecot/log
   8 root     13275  0.0  0.1   6188  4500 ?        S    Jun07   0:00  \_ dovecot/config
   9 --
  10 root       508  0.0  0.2  15264  9980 ?        Ss   Jun06   0:06 /usr/sbin/apache2 -k start
  11 www-data 22005  0.0  0.1  15044  4556 ?        S    11:52   0:00  \_ /usr/sbin/apache2 -k start
  12 www-data 22006  0.0  0.4 2310216 17016 ?       Sl   11:52   0:00  \_ /usr/sbin/apache2 -k start
  13 www-data 22007  0.0  0.4 2310284 18996 ?       Sl   11:52   0:00  \_ /usr/sbin/apache2 -k start
  14 --
  15 root     15775  0.0  0.1  43472  4372 ?        Ss   Jun07   0:00 /usr/lib/postfix/sbin/master -w
  16 postfix  15777  0.0  0.2  43880  8636 ?        S    Jun07   0:00  \_ qmgr -l -t unix -u
  17 postfix  15815  0.0  0.2  44116  9756 ?        S    Jun07   0:00  \_ tlsmgr -l -t unix -u -c
  18 postfix  21276  0.0  0.1  43828  7424 ?        S    11:38   0:00  \_ pickup -l -t unix -u -c

Now change paths to certificates and keys of applications to point to the newly acquired.

   1 postfix/main.cf:smtpd_tls_cert_file          = /etc/letsencrypt/live/mx1.rockstable.it/fullchain.pem
   2 postfix/main.cf:smtpd_tls_key_file           = /etc/letsencrypt/live/mx1.rockstable.it/privkey.pem
   3 dovecot/conf.d/10-ssl.conf:ssl_cert = </etc/letsencrypt/live/mx1.rockstable.it/fullchain.pem
   4 dovecot/conf.d/10-ssl.conf:ssl_key = </etc/letsencrypt/live/mx1.rockstable.it/privkey.pem
   5 apache2/sites-available/roundcube_ssl.conf:             SSLCertificateFile      /etc/letsencrypt/live/mx1.rockstable.it/fullchain.pem
   6 apache2/sites-available/roundcube_ssl.conf:             SSLCertificateKeyFile   /etc/letsencrypt/live/mx1.rockstable.it/privkey.pem

Restart 'em -> DONE.
Systemd certbot.timer is waiting - ready for renewal -> FINE.

   1 systemctl status certbot.timer
   2 ● certbot.timer - Run certbot twice daily
   3    Loaded: loaded (/lib/systemd/system/certbot.timer; enabled; vendor preset: enabled)
   4    Active: active (waiting) since Thu 2019-06-06 20:59:23 CEST; 1 day 15h ago
   5   Trigger: Sat 2019-06-08 22:29:31 CEST; 9h left
   6 
   7 Warning: Journal has been rotated since unit was started. Log output is incomplete or unavailable.

Renewal hooks

/usr/local/sbin/action_service.sh

   1 #!/bin/bash
   2 SELF="$(basename $0|cut -f1 -d.)"
   3 ACTION="${SELF%%_*}"
   4 SERVICE="${SELF##*_}"
   5 
   6 VERBOSE=false
   7 
   8 ### SANITIZE
   9 if ! grep -qE -e "^(start|stop|restart|reload|status)" \
  10                 <<< "$ACTION"; then
  11         "$VERBOSE" && echo "Error - action filtered: '$ACTION'"
  12         exit 1
  13 fi
  14 
  15 if [ -z "$SERVICE" ]; then
  16         "$VERBOSE" && echo "Error - no service specified: '$SERVICE'"
  17         exit 2
  18 fi
  19 
  20 if [ "$ACTION" ] && [ "$SERVICE" ]; then
  21         systemctl "$ACTION" "$SERVICE".service
  22         "$VERBOSE" && systemctl status "$SERVICE".service
  23 fi

Link hooks

   1 chmod u+x /usr/local/sbin/action_service.sh
   2 cd /etc/letsencrypt/renewal-hooks/post
   3 ln -s /usr/local/sbin/action_service.sh restart_apache2
   4 ln -s /usr/local/sbin/action_service.sh restart_ejabberd
   5 ln -s /usr/local/sbin/action_service.sh restart_postfix
   6 ln -s /usr/local/sbin/action_service.sh restart_dovecot

Change domains of a certificate

Get the certificate names by issuing the following command:

   1 certbot certificates

With the option --cert-name you can change a certificate's properties like domains or purposes. This command is special since it additionally requests wildcard-SANs, therefore --manual --preferred-challenges dns

   1 certbot certonly \
   2         --cert-name "jabber.rockstable.it" \
   3         --manual \
   4         --rsa-key-size 4096 \
   5         --preferred-challenges dns \
   6         -m "hostmaster@rockstable.it" \
   7         -d "jabber.rockstable.it" \
   8         -d "jabber1.rockstable.it" \
   9         -d "*.jabber.rockstable.it" \
  10         -d "*.jabber1.rockstable.it" \
  11         --no-eff-email \
  12         --agree-tos
  13 Saving debug log to /var/log/letsencrypt/letsencrypt.log
  14 Plugins selected: Authenticator manual, Installer None
  15 
  16 - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
  17 You are updating certificate jabber.rockstable.it to include new domain(s):
  18 + *.jabber.rockstable.it
  19 + *.jabber1.rockstable.it
  20 
  21 You are also removing previously included domain(s):
  22 (None)
  23 
  24 Did you intend to make this change?
  25 - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
  26 (U)pdate cert/(C)ancel: U
  27 Renewing an existing certificate
  28 Performing the following challenges:
  29 dns-01 challenge for jabber.rockstable.it
  30 dns-01 challenge for jabber1.rockstable.it
  31 dns-01 challenge for jabber.rockstable.it
  32 dns-01 challenge for jabber1.rockstable.it
  33 
  34 - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
  35 NOTE: The IP of this machine will be publicly logged as having requested this
  36 certificate. If you're running certbot in manual mode on a machine that is not
  37 your server, please ensure you're okay with that.
  38 
  39 Are you OK with your IP being logged?
  40 - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
  41 (Y)es/(N)o: Y
  42 
  43 - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
  44 Please deploy a DNS TXT record under the name
  45 _acme-challenge.jabber.rockstable.it with the following value:
  46 
  47 XTXyH8KOM5jSCaI6MCZSPRE3wP-8VU7KQ_bwcwZ1W4s
  48 
  49 Before continuing, verify the record is deployed.
  50 - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
  51 Press Enter to Continue
  52 
  53 - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
  54 Please deploy a DNS TXT record under the name
  55 _acme-challenge.jabber1.rockstable.it with the following value:
  56 
  57 P_HWFlmOfqAXP98dRHibUu5KWuxMPYtY4cHfD3gd9QE
  58 
  59 Before continuing, verify the record is deployed.
  60 (This must be set up in addition to the previous challenges; do not remove,
  61 replace, or undo the previous challenge tasks yet. Note that you might be
  62 asked to create multiple distinct TXT records with the same name. This is
  63 permitted by DNS standards.)
  64 
  65 - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
  66 Press Enter to Continue
  67 
  68 - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
  69 Please deploy a DNS TXT record under the name
  70 _acme-challenge.jabber.rockstable.it with the following value:
  71 
  72 ACprK1TjjqXru0Qxffm2QZKfT1X-RVx1E9SQVjQaDTg
  73 
  74 Before continuing, verify the record is deployed.
  75 (This must be set up in addition to the previous challenges; do not remove,
  76 replace, or undo the previous challenge tasks yet. Note that you might be
  77 asked to create multiple distinct TXT records with the same name. This is
  78 permitted by DNS standards.)
  79 
  80 - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
  81 Press Enter to Continue
  82 
  83 - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
  84 Please deploy a DNS TXT record under the name
  85 _acme-challenge.jabber1.rockstable.it with the following value:
  86 
  87 xdz9ruXUFf5C6oDgtx1cF3H54N3sLhlJ-ZSVaf55saI
  88 
  89 Before continuing, verify the record is deployed.
  90 (This must be set up in addition to the previous challenges; do not remove,
  91 replace, or undo the previous challenge tasks yet. Note that you might be
  92 asked to create multiple distinct TXT records with the same name. This is
  93 permitted by DNS standards.)
  94 
  95 - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
  96 Press Enter to Continue
  97 Waiting for verification...
  98 Cleaning up challenges
  99 
 100 IMPORTANT NOTES:
 101  - Congratulations! Your certificate and chain have been saved at:
 102    /etc/letsencrypt/live/jabber.rockstable.it/fullchain.pem
 103    Your key file has been saved at:
 104    /etc/letsencrypt/live/jabber.rockstable.it/privkey.pem
 105    Your cert will expire on 2020-01-30. To obtain a new or tweaked
 106    version of this certificate in the future, simply run certbot
 107    again. To non-interactively renew *all* of your certificates, run
 108    "certbot renew"
 109  - If you like Certbot, please consider supporting our work by:
 110 
 111    Donating to ISRG / Let's Encrypt:   https://letsencrypt.org/donate
 112    Donating to EFF:                    https://eff.org/donate-le

Wildcard certificates

Always quote your domain-names to prevent shell expansion of the globber *.
When integrating the TXT-records in your zone, don't forget to increase serial. The TXT-records are only valid once.
A wildcard dns-name may only contain a single wildcard character and the resource record has to start with it
IETF: The Role of Wildcards in the Domain Name System Interesting Sections: 2.1.1, 4.5
```
   1 An unexpected error occurred:
   2 Error creating new order :: Cannot issue for "*.jabber*.rockstable.org": DNS name had more than one wildcard
```
For wildcard certs the only method to challenge the request is dns.
There are several
Certbot wildcard dns-plugins to assist the DNS challenge, but only --manual is installed. Many distributions ship them as separate packages.

Wildcard certificate with RFC2136

For a long time i was renewing the wildcard certificates for my jabber node manually.

This time is over now

Install the plugin and certbot

   1 aptitude install python3-certbot-dns-rfc2136

On your DNS server create a tsig key to be used during authentication of your server with the DNS service (bind9 nin this case).

   1 DIR_KEYS="/var/lib/bind/keys/tsig"
   2 [ -d "$DIR_KEYS" ] || mkdir "$DIR_KEYS"
   3 cd "$DIR_KEYS"
   4 HNAME="jabber1.rockstable.it."
   5 dnssec-keygen -a HMAC-SHA512 -b 512 -n HOST "$HNAME"

Define the key in your Bind9 configuration
/etc/bind/named.conf.auth

   1 key jabber1_rockstable_it {
   2         algorithm       hmac-sha512;
   3         secret          "kWIvE5mFlr7QOaTw/iYpl9NS/NaqtMH4WV4pF/NBzvW3j/YtGDJru+8p+WkCP3vb9lkQArtkUhA0iuS4swR4Eg==";
   4 };

And define a update-policy that granularily allows to update specificy TXT RRs only.

   1 view external {
   2         match-clients {
   3                  key DNS_REPLICATOR_EXTERNAL;    
   4                  key jabber1_rockstable_it;      
   5                 !key DNS_REPLICATOR_INTERNAL;   
   6                 !localhost;    
   7                 !nets_int;
   8                 !nets_ext;
   9                 any;
  10         };
  11 
  12         zone "rockstable.org" {
  13                 type                    master;
  14                 masterfile-format       text;                                   # (text|raw)
  15                 file                    "/var/lib/bind/zones/db.org.rockstable";
  16                 journal                 "/var/lib/bind/journal/db.org.rockstable.jnl";  # string ;
  17                 update-policy {
  18                         grant local-ddns                zonesub ANY;
  19                         //grant DNS_REPLICATOR          zonesub ANY;
  20                         grant DNS_REPLICATOR_EXTERNAL   zonesub ANY;
  21                         grant jabber1_rockstable_it     name _acme-challenge.jabber.rockstable.org. TXT;
  22                         grant jabber1_rockstable_it     name _acme-challenge.jabber1.rockstable.org. TXT;
  23                 };
  24                 // SOME ZONE CONFIGURATION OMITTED TO SHORTEN THE LISTING
  25         };
  26         zone "rockstable.it" {
  27                 type                    master;
  28                 masterfile-format       text;                                   # (text|raw)
  29                 file                    "/var/lib/bind/zones/db.it.rockstable";
  30                 journal                 "/var/lib/bind/journal/db.it.rockstable.jnl";   # string ;
  31                 update-policy {
  32                         grant local-ddns                zonesub ANY;
  33                         //grant DNS_REPLICATOR          zonesub ANY;
  34                         grant DNS_REPLICATOR_EXTERNAL   zonesub ANY;
  35                         grant jabber1_rockstable_it name _acme-challenge.jabber.rockstable.it. TXT;
  36                         grant jabber1_rockstable_it name _acme-challenge.jabber1.rockstable.it. TXT;
  37                 };
  38                 // SOME ZONE CONFIGURATION OMITTED TO SHORTEN THE LISTING
  39         };
  40 };
  41 
  42 view internal {
  43         match-clients           {
  44                  key DHCP_UPDATER;
  45                  key DNS_REPLICATOR_INTERNAL;
  46                 !key DNS_REPLICATOR_EXTERNAL;
  47                 !key jabber1_rockstable_it;
  48                 localhost;
  49                 nets_int;
  50                 nets_ext;
  51         }
  52         zone rockstable.org {
  53                 //…
  54         };
  55         zone rockstable.it {
  56                 in-view external;
  57         };
  58 };

Check the bind9 config and reload bind9

   1 named-checkconf
   2 systemctl reload bind9.service

On the server that is about to request a certifikate, create a file
/etc/letsencrypt/rfc2136.ini

   1 # Target DNS server
   2 dns_rfc2136_server = 195.201.246.253
   3 # Target DNS port
   4 dns_rfc2136_port = 53
   5 # TSIG key name
   6 dns_rfc2136_name = jabber1_rockstable_it
   7 # TSIG key secret
   8 dns_rfc2136_secret = kWIvE5mFlr7QOaTw/iYpl9NS/NaqtMH4WV4pF/NBzvW3j/YtGDJru+8p+WkCP3vb9lkQArtkUhA0iuS4swR4Eg==
   9 # TSIG key algorithm
  10 dns_rfc2136_algorithm = HMAC-SHA512

Make sure you are using an authoritative server. Certbot checks it! Please see #Trouble Shooting

Secure the filesystem permissions to the file

   1 FILE="/etc/letsencrypt/rfc2136.ini"
   2 chown 0.0 "$FILE"
   3 chmod 0640 "$FILE"

Check if the DNS server is reachable (in the case via TCP)

   1 nc -vz 178.63.149.225 53
   2 Connection to 178.63.149.225 53 port [tcp/domain] succeeded!

Please be patient during requesting the certificate. Certbot waits be defautl for 60s for the new DNS records to propagate.

   1 certbot certonly \
   2         --cert-name "jabber.rockstable.it" \
   3         --preferred-challenges dns \
   4         --dns-rfc2136 \
   5         --dns-rfc2136-credentials /etc/letsencrypt/rfc2136.ini \
   6         --dns-rfc2136-propagation-seconds 60 \
   7         --rsa-key-size 4096 \
   8         -m "hostmaster@rockstable.it" \
   9         -d "jabber.rockstable.it" \
  10         -d "jabber1.rockstable.it" \
  11         -d "*.jabber.rockstable.it" \
  12         -d "*.jabber1.rockstable.it" \
  13         --no-eff-email \
  14         --agree-tos \
  15         --dry-run

First test it with dry-runs.

SUCCESS!

   1 …
   2 IMPORTANT NOTES:
   3  - Congratulations! Your certificate and chain have been saved at:
   4 …

Carry on analogous with the other domain.

Here is the renewal file for comparision
/etc/letsencrypt/renewal/jabber.rockstable.it

   1 # renew_before_expiry = 30 days
   2 version = 1.12.0
   3 archive_dir = /etc/letsencrypt/archive/jabber.rockstable.it
   4 cert = /etc/letsencrypt/live/jabber.rockstable.it/cert.pem
   5 privkey = /etc/letsencrypt/live/jabber.rockstable.it/privkey.pem
   6 chain = /etc/letsencrypt/live/jabber.rockstable.it/chain.pem
   7 fullchain = /etc/letsencrypt/live/jabber.rockstable.it/fullchain.pem
   8 
   9 # Options used in the renewal process
  10 [renewalparams]
  11 account = __redacted
  12 rsa_key_size = 4096
  13 authenticator = dns-rfc2136
  14 server = https://acme-v02.api.letsencrypt.org/directory
  15 pref_challs = dns-01,
  16 dns_rfc2136_credentials = /etc/letsencrypt/rfc2136.ini

Trouble Shooting

iana.org DNS Header Flags

I had some problems with the plugin. The plugin guesses and queries some SOA-Record and checks if the "Authoritative Answer" (AA) flags is set. https://github.com/certbot/certbot/blob/master/certbot-dns-rfc2136/certbot_dns_rfc2136/_internal/dns_rfc2136.py#L223

The plugin alsow disables the "Recursion Desired" (RD) flag in this queries. So a recursive resolver won't answer, if the cache is not populated. https://github.com/certbot/certbot/blob/master/certbot-dns-rfc2136/certbot_dns_rfc2136/_internal/dns_rfc2136.py#L213

Bind9 sets the AA flag if the zone is defined and is not declared to be of type "mirror". https://github.com/isc-projects/bind9/blob/main/lib/ns/query.c#L5576

I my very specific case the reason, why the name server did not respond with AA set was, a miss-configuration of my views. The zone that should be updated ("rockstable.it") was only defined in the external view and not in the internal view. The DNS-client queried in the internal view, where the zone is not defined and recursion was used to resolve the name from the external view.

There are two work-arounds:

if bind9 < 9.10, use a zone transfer from one view to the other
if bind9 >=9.10, use the 'in-view' statement to define a zone in one view by efficiently referencing the zone in another view. Make sure the zone is defined in the config file first, before referencing it.

Please compare to ISC Knowledge Base - How do I share a dynamic zone between multiple views?

Simple as that
/etc/bind/named.conf.local

view external {

…
};

view internal {
        zone rockstable.it {
                in-view external;
        };
};

Once i configured the in-view

Move cert to other server

Revoke and delete certificate

   1 certbot revoke \
   2         --cert-name "rockstable.it" \
   3         --reason "affiliationchanged" \
   4         --delete-after-revoke

Undeleted certificated are tried to be renewed.

Certbot on Debian Jessie

Certbot version in Debian Jessie is 0.10.2-1~bpo8+1 and is furthermore only available via backports. Whis version only supports ACMEv1 which got deprecaded in June 2020. Here's a way to get a modern certbot running.
certbot.eff.org: Debian Jessie Apache

If you've got a older version of certbot installed, please make a snapshot of the machine and a backup of the directories:

/etc/letsencrypt
/etc/cron.d/certbot

You can get the certbot packages for Debian Jessie Backports from the Debian archive, but the signature is expired. :-D

   1 ##deb   http://archive.debian.org/debian/               jessie-backports main contrib non-free
   2

Wget the wrapper script/entry point that takes care of installation, updates and command invokation.

   1 wget -O /usr/local/bin/certbot-auto \
   2         https://dl.eff.org/certbot-auto
   3 chown root:root /usr/local/bin/certbot-auto
   4 chmod 0755 /usr/local/bin/certbot-auto

Install OS dependencies (python, virtualenv, compiler, headers …) and the certbot itself to /opt/eff.org/certbot/venv

   1 certbot-auto --install-only

Try it

   1 certbot-auto --help
   2 certbot-auto renew --apache --dry-run

Remove the old certbot

   1 apt remove certbot

Without systemd

Create a cron file triggerd twice a day /etc/cron.d/certbot-auto

   1 # Eventually, this will be an opportunity to validate certificates
   2 # haven't been revoked, etc.  Renewal will only occur if expiration
   3 # is within 30 days.
   4 SHELL=/bin/sh
   5 PATH=/usr/local/sbin:/usr/local/bin:/sbin:/bin:/usr/sbin:/usr/bin
   6 
   7 0 */12 * * * root test -x /usr/local/bin/certbot -a \! -d /run/systemd/system && perl -e 'sleep int(rand(3600))' && certbot-auto -q renew

With systemd

Create a service file to be triggered by a timer.

/lib/systemd/system/certbot-auto.service

   1 [Unit]
   2 Description=Certbot
   3 Documentation=file:///usr/share/doc/python-certbot-doc/html/index.html
   4 Documentation=https://letsencrypt.readthedocs.io/en/latest/
   5 [Service]
   6 Type=oneshot
   7 ExecStart=/usr/local/bin/certbot-auto -q renew
   8 PrivateTmp=true

/lib/systemd/system/certbot-auto.timer

   1 [Unit]
   2 Description=Run certbot twice daily
   3 
   4 [Timer]
   5 OnCalendar=*-*-* 00,12:00:00
   6 RandomizedDelaySec=43200
   7 Persistent=true
   8 
   9 [Install]
  10 WantedBy=timers.target

Reload systemd and enable timer

   1 systemctl daemon-reload
   2 systemctl enable certbot-auto.timer

Certbot in Docker

WIP

Certbot docs Running with Docker

Idea

So, I decided to try docker for this.

This is not the most sophisticated setup, but it actually works.

A volume that contains the whole filesystem of /etc/letsencrypt is mapped to the container "certbot", which updates the content of the directory.
Mount the volume (readonly) e.g. to container with "haproxy". This container now has access to the certificates generated by certbot.
Configure haproxy to redirect acme-challenges to the host.
Create a wrapper script on the host that is triggered by a systemd-timer for renewal.
When the renewal has acutally been performed,
also trigger a reload of the haproxy config sending signal HUP to the process.

Roadmap

Integrate this into docker compose
Trigger reload via haproxy dataplane API
Secure certificates better
…

Implementation

Prepare a file that contains some configuration
/srv/docker/certbot/certbot.default

   1 # -----------------------------------
   2 # CERTBOT
   3 # -----------------------------------
   4 CERTBOT_PORT="8090"
   5 #CERTBOT_KEYSIZE="4096"
   6 CERTBOT_MAIL="hostmaster@your.domain.tld"
   7 CERTBOT_DOMAINS=(
   8         "your.domain.tld"
   9         "other.domain.tld"
  10         )
  11 CERTBOT_HOOK_POST="docker ps |grep haproxy && docker kill -s HUP haproxy"

/srv/docker/certbot/certbot_wrapper.sh

   1 #!/bin/bash
   2 
   3 ### DEFAULTS
   4 CERTBOT_ENV="certbot.default"
   5 
   6 ### INIT
   7 PROGRAM="${0#*/}"
   8 PREFIX="${0%/*}"
   9 CERTBOT_AGREE="--agree-tos"
  10 CERTBOT_NOMAIL="--no-eff-email"
  11 CERTBOT_HOOK_POST=""
  12 CERTBOT_KEYSIZE="4096"
  13 CERTBOT_NEW=false
  14 CERTBOT_RENEW=false
  15 CERTBOT_CONCAT=false
  16 CERTBOT_PORT="8090"
  17 CERTBOT_MAIL=""
  18 VERBOSE=false
  19 unset DOMAINS
  20 declare -a CERTBOT_DOMAINS
  21 
  22 ### DETERMINE ABSOLUTE PREFIX PATH
  23 READLINK="/bin/readlink"
  24 REALPATH="/bin/realpath"
  25 if [ "$READLINK" ]; then
  26         PREFIX_ABS="$(readlink -f "$PREFIX")"
  27 elif [ "$REALPATH" ]; then
  28         PREFIX_ABS="$(realpath -e "$PREFIX")"
  29 else
  30         cat <<-EOF
  31                 Cannot determine prefix path.
  32                 The binaries 'readlink' and 'realpath'
  33                 are both missing.
  34                 Exiting…
  35         EOF
  36         exit 1
  37 fi
  38 
  39 usage () {
  40         cat <<-EOF
  41 
  42                 $PROGRAM [WRAPPER_OPTIONS] -- [CERTBOT_OPTIONS]
  43 
  44                 [] … Optional
  45 
  46                 WRAPPER_OPTIONS:
  47                 -c|--concat             Concatenate private key, certificate and
  48                                         the certificate chain to a bundle.
  49                 -e|--env "env.file"     Path to the defaults env.file
  50                 -l|--listen PORT        Bind certbot container to PORT
  51                                         Useful for boot-strapping,
  52                                         when no reverse proxy is running.
  53                 -h|--help               Display this page
  54                 -p|--prefix "PATH"      Set the working prefix for the wrapper
  55                 -v|--verbose            Print a more detailed output
  56 
  57                 [--]                    Stop parsing wrapper options
  58 
  59                 [CERTBOT_OPTIONS]       These options are passed to certbot directly.
  60                                         Please see 'certbot --help', 'certbot -h all',
  61                                         https://certbot.eff.org/docs/index.html
  62                                         or the man page.
  63         EOF
  64 }
  65 
  66 # Note that we use "$@" to let each command-line parameter expand to a
  67 # separate word. The quotes around "$@" are essential!
  68 # We need TEMP as the 'eval set --' would nuke the return value of getopt.
  69 TEMP=$(getopt \
  70         -o 'ce:hl:np:v' \
  71         --long 'concat,debug,env:,help,listen:,new,prefix:,verbose' \
  72         -n "$PROGRAM" -- "$@")
  73 
  74 if [ $? -ne 0 ]; then
  75         echo 'Terminating...' >&2
  76         exit 1
  77 fi
  78 
  79 # Note the quotes around "$TEMP": they are essential!
  80 eval set -- "$TEMP"
  81 unset TEMP
  82 
  83 while true; do
  84         case "$1" in
  85                 '-c'|'--concat')
  86                         CERTBOT_CONCAT=true
  87                         shift
  88                         continue
  89                 ;;
  90                 '-e'|'--env')
  91                         CERTBOT_ENV="$2"
  92                         shift 2
  93                         continue
  94                 ;;
  95                 '-h'|'--help')
  96                         usage
  97                         exit 0
  98                         shift
  99                         continue
 100                 ;;
 101                 '-l'|'--listen')
 102                         CERTBOT_PORT="$2"
 103                         shift 2
 104                         continue
 105                 ;;
 106                 '-n'|'--new')
 107                         CERTBOT_NEW=true
 108                         shift
 109                         continue
 110                 ;;
 111                 '-p'|'--prefix')
 112                         PREFIX="$2"
 113                         shift 2
 114                         continue
 115                 ;;
 116                 '-v'|'--verbose')
 117                         VERBOSE=true
 118                         shift
 119                         continue
 120                 ;;
 121                 '--')
 122                         shift
 123                         break
 124                 ;;
 125                 *)
 126                         echo 'Internal error!' >&2
 127                         exit 1
 128                 ;;
 129         esac
 130 done
 131 
 132 ### LOAD DEFAULTS
 133 if [ -f "$CERTBOT_ENV" ]; then
 134         source "$CERTBOT_ENV"
 135 else
 136         echo "Environment file '$CERTBOT_ENV' does not exist. Exiting …" \ >&2
 137         exit 2
 138 fi
 139 
 140 
 141 if $VERBOSE; then
 142         echo "CERTBOT_OPTIONS:"
 143         for ARG; do
 144                 echo "-> $ARG"
 145         done
 146 fi
 147 
 148 
 149 ### PASS REMAINING ARGUMENTS TO CERTBOT
 150 CERTBOT_ARGS="$@"
 151 
 152 
 153 ### SANITY CHECKS
 154 if [ "$CERTBOT_PORT" ]; then
 155         DOCKER_OPTIONS="-p $CERTBOT_PORT:80"
 156 fi
 157 
 158 
 159 ### PROCESS DOMAINS
 160 FIRST=true
 161 for DOMAIN in "${CERTBOT_DOMAINS[@]}"; do
 162         if $FIRST; then
 163                 FIRST=false
 164                 DOMAIN_PRIMARY="$DOMAIN"
 165         else
 166                 DOMAINS+=" "
 167         fi
 168         DOMAINS+="-d $DOMAIN"
 169 done
 170 
 171 ### MAIN
 172 DOCKER_CMD="docker run -it --rm --name certbot \
 173         -v "$PREFIX_ABS/etc/letsencrypt:/etc/letsencrypt" \
 174         -v "$PREFIX_ABS/var/lib/letsencrypt:/var/lib/letsencrypt" \
 175         $DOCKER_OPTIONS \
 176         certbot/certbot"
 177 
 178 if $CERTBOT_NEW; then
 179         CERTBOT_ARGS="certonly \
 180                         --standalone \
 181                         --rsa-key-size "$CERTBOT_KEYSIZE" \
 182                         -m "$CERTBOT_MAIL" \
 183                         $CERTBOT_NOMAIL \
 184                         $CERTBOT_AGREE \
 185                         $DOMAINS \
 186                         $CERTBOT_ARGS"
 187         CERTBOT_CONCAT=true
 188 fi
 189 
 190 if $VERBOSE; then
 191         echo "$CERTBOT_ARGS"
 192 fi
 193 
 194 $DOCKER_CMD $CERTBOT_ARGS
 195 RETURN="$?"
 196 
 197 if $VERBOSE; then
 198         echo "Return value: '$RETURN'"
 199 fi
 200 
 201 ### HAPROXY NEEDS A CERTIFICATE BUNDLE WITH THE PRIVATE KEY PREPENDED
 202 export PREFIX_CERT="$PREFIX_ABS/etc/letsencrypt/live/$DOMAIN_PRIMARY"
 203 export PATH_CERT="$PREFIX_CERT/fullchain.pem"
 204 export PATH_PRIVATE="$PREFIX_CERT/privkey.pem"
 205 export PATH_BUNDLE="$PREFIX_CERT/bundle.pem"
 206 
 207 if $CERTBOT_CONCAT &&
 208         [ -e "$PATH_PRIVATE" ] &&
 209         [ -e "$PATH_CERT" ]; then
 210         bash -c \
 211                 'umask 0033
 212                 cat \
 213                         "$PATH_PRIVATE" \
 214                         "$PATH_CERT" \
 215                         > "$PATH_BUNDLE"'
 216 fi
 217 
 218 ### RELAX FILESYSTEM PERMISSIONS
 219 chmod 0755 "$PREFIX_ABS/etc/letsencrypt/live"
 220 
 221 ### EXECUTE HOOK POST
 222 eval "$CERTBOT_HOOK_POST"

Sorry, tabstops are broken.

Usage

   1 cd /srv/docker/certbot
   2 ./certbot_wrapper.sh -h
   3 
   4 certbot_wrapper.sh [WRAPPER_OPTIONS] -- [CERTBOT_OPTIONS]
   5 
   6 [] … Optional
   7 
   8 WRAPPER_OPTIONS:
   9 -c|--concat             Concatenate private key, certificate and
  10                         the certificate chain to a bundle.
  11 -e|--env "env.file"     Path to the defaults env.file
  12 -l|--listen PORT        Bind certbot container to PORT
  13                         Useful for boot-strapping,
  14                         when no reverse proxy is running.
  15 -h|--help               Display this page
  16 -p|--prefix "PATH"      Set the working prefix for the wrapper
  17 -v|--verbose            Print a more detailed output
  18 
  19 --                      Stop parsing wrapper options
  20 
  21 [CERTBOT_OPTIONS]       These options are passed to certbot directly.
  22                         Please see 'certbot --help', 'certbot -h all',
  23                         https://certbot.eff.org/docs/index.html
  24                         or the man page.

Obtain a certificate

When you don't have a reverse proxy running you can simply bind the certbot container to port 80.

   1 cd /srv/docker/certbot
   2 ./certbot_wrapper.sh -n -l 80 -- --dry-run
   3 ./certbot_wrapper.sh -l 80 -n

Revoke a certificate

   1 ./certbot_wrapper.sh -v -- \
   2         revoke --delete-after-revoke \
   3         --cert-name your.domain.tld

You may need to remove the directory etc/letsencrypt/live/your.domain.tld manually because of the concatenated bundle.pem.

Renew a certificate

   1 ./certbot_wrapper.sh -v -- \
   2         --dry-run renew

Automatically renew the certificate

/srv/docker/certbot/certbot_docker.service

   1 [Unit]                         
   2 Description=Certbot Docker
   3 Documentation=file:///usr/share/doc/python-certbot-doc/html/index.html
   4 Documentation=https://certbot.eff.org/docs
   5 [Service]
   6 Type=oneshot
   7 ExecStart=/srv/docker/certbot/certbot_wrapper.sh -c -- renew
   8 PrivateTmp=true

/srv/docker/certbot/certbot_docker.timer

   1 [Unit]                         
   2 Description=Run certbot twice daily
   3 
   4 [Timer]
   5 OnCalendar=*-*-* 00,12:00:00
   6 RandomizedDelaySec=43200       
   7 Persistent=true
   8 
   9 [Install]
  10 WantedBy=timers.target

Enable the timer for the renewal in systemd

   1 ln -s /srv/docker/certbot/certbot_docker.service \
   2         /lib/systemd/system/certbot_docker.service
   3 ln -s /srv/docker/certbot/certbot_docker.timer \
   4         /lib/systemd/system/certbot_docker.timer 
   5 systemctl daemon-reload
   6 systemctl enable certbot_docker.timer
   7 Created symlink /etc/systemd/system/timers.target.wants/certbot_docker.timer → /srv/docker/certbot/certbot_docker.timer.
   8 systemctl status certbot_docker.timer
   9 Created symlink /etc/systemd/system/certbot_docker.timer → /srv/docker/certbot/certbot_docker.timer.
  10 ● certbot_docker.timer - Run certbot twice daily
  11      Loaded: loaded (/srv/docker/certbot/certbot_docker.timer; enabled; vendor preset: enabled)
  12      Active: active (waiting) since Mon 2021-11-01 11:36:16 CET; 58min ago
  13     Trigger: Tue 2021-11-02 04:36:57 CET; 16h left
  14    Triggers: ● certbot_docker.service
  15 
  16 Nov 01 11:36:16 hostname systemd[1]: Started Run certbot twice daily.

Configurations

Services

Misc

Navigation

LetsEncrypt

About

DNS Certification Authority Authorization (CAA) Resource Record

Certbot

About

LetsEncypt source IP addresses

Installation

Configuration

Retrieve certificate for postfix

Retrieve certificate for apache2

Retrieve certificate for nginx

Some considerations concerning access

Renewal hooks

Change domains of a certificate

Wildcard certificates

Wildcard certificate with RFC2136

Trouble Shooting

Move cert to other server

Certbot on Debian Jessie

Without systemd

With systemd

Certbot in Docker

Idea

Roadmap

Implementation

Usage

Obtain a certificate

Revoke a certificate

Renew a certificate

Automatically renew the certificate