Apache2

Contents

Apache2

About

HTTP
https://httpd.apache.org/
Apache HTTP Server Project
Apache2 Docs
The Apache Software Foundation (ASF)
- The Apache Software Foundation (ASF) exists to provide software for the public good. We believe in the power of community over code, known as The Apache Way. Thousands of people around the world contribute to ASF open source projects every day.
- Apache projects by name
- Apache Project Logos

Recent Version

It's strongly recommended to use the latest version of Apache.
Apache httpd 2.2 End-of-Life 2018-01-01
- As previously announced, the Apache HTTP Server Project has discontinued all development and patch review of the 2.2.x series of releases. The Apache HTTP Server Project had long committed to provide maintenance releases of the 2.2.x flavor through June of 2017. The final release 2.2.34 was published in July 2017, and no further evaluation of bug reports or security risks will be considered or published for 2.2.x releases.
Apache 2.4 new features
- Tends to utilize less memory than Apache2.2.
- Initial (while still experimental) support for the fast speedy successor http2.0.
- Additionally has "conf-enabled" to "mods-enabled" and "sites-enabled", which leaves a more consistent impression.
- Define statements simplify complex configurations.
- Macro support (i.e. for interesting vHost-Configuration).
- many more …
Apache 2.6 new features

Defaults and environment variables

The default config is located in
/etc/apache2/envvars or
/etc/sysconfig/httpd

   1 # envvars - default environment variables for apache2ctl
   2 
   3 # this won't be correct after changing uid
   4 unset HOME
   5 
   6 # for supporting multiple apache2 instances
   7 if [ "${APACHE_CONFDIR##/etc/apache2-}" != "${APACHE_CONFDIR}" ] ; then
   8         SUFFIX="-${APACHE_CONFDIR##/etc/apache2-}"
   9 else
  10         SUFFIX=
  11 fi
  12 
  13 # Since there is no sane way to get the parsed apache2 config in scripts, some
  14 # settings are defined via environment variables and then used in apache2ctl,
  15 # /etc/init.d/apache2, /etc/logrotate.d/apache2, etc.
  16 export APACHE_RUN_USER=www-data
  17 export APACHE_RUN_GROUP=www-data
  18 # temporary state file location. This might be changed to /run in Wheezy+1
  19 export APACHE_PID_FILE=/var/run/apache2$SUFFIX/apache2.pid
  20 export APACHE_RUN_DIR=/var/run/apache2$SUFFIX
  21 export APACHE_LOCK_DIR=/var/lock/apache2$SUFFIX
  22 # Only /var/log/apache2 is handled by /etc/logrotate.d/apache2.
  23 export APACHE_LOG_DIR=/var/log/apache2$SUFFIX
  24 
  25 ## The locale used by some modules like mod_dav
  26 export LANG=C
  27 ## Uncomment the following line to use the system default locale instead:
  28 #. /etc/default/locale
  29 
  30 export LANG
  31 
  32 ## The command to get the status for 'apache2ctl status'.
  33 ## Some packages providing 'www-browser' need '--dump' instead of '-dump'.
  34 #export APACHE_LYNX='www-browser -dump'
  35 
  36 ## If you need a higher file descriptor limit, uncomment and adjust the
  37 ## following line (default is 8192):
  38 #APACHE_ULIMIT_MAX_FILES='ulimit -n 65536'
  39 
  40 ## If you would like to pass arguments to the web server, add them below
  41 ## to the APACHE_ARGUMENTS environment.
  42 #export APACHE_ARGUMENTS=''
  43 
  44 ## Enable the debug mode for maintainer scripts.
  45 ## This will produce a verbose output on package installations of web server modules and web application
  46 ## installations which interact with Apache
  47 #export APACHE2_MAINTSCRIPT_DEBUG=1
  48

Tools

apache2ctl

Also apachectl or apache2ctl

Help

   1 apache2ctl -h

Show parsed vhost and run settings

   1 apache2ctl -S

Server-Status #mod_status

   1 watch -n1 -- 'apache2ctl status'
   2 Every 1,0s: apache2ctl status
   3 
   4 Apache Server Status for localhost (via 127.0.0.1)
   5 
   6 Server Version: Apache/2.4.17 (Debian) mod_fastcgi/mod_fastcgi-SNAP-0910052141
   7     OpenSSL/1.0.2d
   8 Server MPM: prefork
   9 Server Built: 2015-11-23T18:52:09
  10 
  11 -------------------------------------------------------------------------------
  12 
  13 Current Time: Sunday, 13-Dec-2015 23:29:29 CET
  14 Restart Time: Sunday, 13-Dec-2015 14:46:46 CET
  15 Parent Server Config. Generation: 34
  16 Parent Server MPM Generation: 33
  17 Server uptime: 8 hours 42 minutes 43 seconds
  18 Server load: 0.05 0.03 0.05
  19 Total accesses: 620 - Total Traffic: 1.1 MB
  20 CPU Usage: u.03 s.02 cu0 cs0 - .000159% CPU load
  21 .0198 requests/sec - 38 B/second - 1940 B/request
  22 1 requests currently being processed, 6 idle workers
  23 
  24 ____W__.....
  25 ............
  26 ............
  27 
  28 Scoreboard Key:
  29 "_" Waiting for Connection, "S" Starting up, "R" Reading Request,
  30 "W" Sending Reply, "K" Keepalive (read), "D" DNS Lookup,
  31 "C" Closing connection, "L" Logging, "G" Gracefully finishing,
  32 "I" Idle cleanup of worker, "." Open slot with no current process

Graceful restart webserver
This lets apache2 processes answer their pending jobs and restart them afterwards.

   1 apache2ctl graceful

Stop webserver

   1 apache2ctl stop

Start webserver

   1 apache2ctl start

The systemd.units internally also use this commands. But the bare apache2ctl commands should not be used by the administrator, if apache2 is governed by SystemD, because it supervises the status of the daemon and may lose track.

List modules

   1 apache2ctl -M
   2 Loaded Modules:
   3  core_module (static)
   4  so_module (static)
   5  watchdog_module (static)
   6  http_module (static)
   7  log_config_module (static)
   8  logio_module (static)
   9  version_module (static)
  10  unixd_module (static)
  11  access_compat_module (shared)
  12  actions_module (shared)
  13  alias_module (shared)
  14  auth_basic_module (shared)
  15  authn_core_module (shared)
  16  authn_file_module (shared)
  17  authz_core_module (shared)
  18  authz_groupfile_module (shared)
  19  authz_host_module (shared)
  20  authz_user_module (shared)
  21  autoindex_module (shared)
  22  cgid_module (shared)
  23  deflate_module (shared)
  24  dir_module (shared)
  25  env_module (shared)
  26  fastcgi_module (shared)
  27  filter_module (shared)
  28  headers_module (shared)
  29  http2_module (shared)
  30  include_module (shared)
  31  mime_module (shared)
  32  mpm_event_module (shared)
  33  negotiation_module (shared)
  34  reqtimeout_module (shared)
  35  rewrite_module (shared)
  36  setenvif_module (shared)
  37  socache_shmcb_module (shared)
  38  ssl_module (shared)
  39  status_module (shared)

htpasswd

Apache Docs - Password Formats
IETF RFC 2307 - An Approach for Using LDAP as a Network Information Service
- Definition of password hashing schemes like '{SSHA}'
Please also see
#mod_auth_basic

Tools to create the hashes

htpasswd
slappasswd from the package slapd supports RFC 2307 schemes of type: '{CRYPT}', '{MD5}', '{SMD5}', '{SSHA}', and '{SHA}'.
- The default is '{SSHA}'
- Example without trailing newline:
  slappasswd -nh '{CRYPT}'
doveadm pw from the package dovecot supports RFC 2307 schemes of type:
- Dovecot Docs - Password Schemes

openssl passwd

Generating an example htpasswd file (hidden in comment)

Generated example htpasswd ./htpasswd

   1 ### EXAMPLE HTPASSWD FILE
   2 user0:{PLAIN}test:plain with rcf2303 password scheme
   3 user1:test:plain (only on windows and netshare)
   4 user2:$2y$05$dJDW0ldoauWBLrhHO/GZLuKokgm2Bxkj7t1m7gcuXH0yT4bTfDGRG:bcrypt cost 5
   5 user3:$2y$17$/aXF3PCSJtLw3QmZMmhveeS38tBnaSaVkVWPTBGdXINSMPjb44h6K:bcrypt cost 17
   6 user4:2ngE8Y5cYdcD2:crypt
   7 user5:$apr1$ZnNtuBUv$LTiVaUmllAZ5TkyY.d3IA1:apr1
   8 user6:{SHA}qUqP5cyxm6YcTAhz05Hph5gvu9M=:sha1
   9 user7:$5$iLr9g/vYGEE8ZHXc$3qR0fYTYlZqLH/oAgLK0xwUreKeMyZeqZ4P46mNr3H.:SHA-256 crypt()
  10 user8:$6$bw/Mi3L/FeHnMv0K$vOi6FxnRIfSCy6aQcWUSHKgALQpmWZAqLYMJsa6OBW...A9.Um/4WttMPbYC9QIdx9mfXZuVnF2DHcPUFzkcU/:SHA-512 crypt()

Hardening

Apache2.4: Security Tips

Basic Security

Forbid any access, can later be allowed in another directory/location directive.
Restrict server-version published on error-pages top the minor version of the server.
Show email-link to the webmaster on the error page reporting problems.
Forbid tracing, sniffing and nesting iframes using extended HTTP-X-Headers-Fields.

/etc/apache2/conf-enabled/security.conf

   1 #
   2 # Disable access to the entire file system except for the directories that
   3 # are explicitly allowed later.
   4 #
   5 # This currently breaks the configurations that come with some web application
   6 # Debian packages.
   7 #
   8 <Directory />
   9     AllowOverride None
  10     Require all denied
  11     ### Don't use the old access directives anymore with 2.4
  12     ### subsequent stuff will fail
  13     #Order Deny,Allow
  14     #Deny from all
  15 </Directory>
  16 
  17 
  18 # Changing the following options will not really affect the security of the
  19 # server, but might make attacks slightly more difficult in some cases.
  20 
  21 #
  22 # ServerTokens
  23 # This directive configures what you return as the Server HTTP response
  24 # Header. The default is 'Full' which sends information about the OS-Type
  25 # and compiled in modules.
  26 # Set to one of:  Full | OS | Minimal | Minor | Major | Prod
  27 # where Full conveys the most information, and Prod the least.
  28 #ServerTokens Minimal
  29 ServerTokens Minor
  30 #ServerTokens Full
  31 
  32 #
  33 # Optionally add a line containing the server version and virtual host
  34 # name to server-generated pages (internal error documents, FTP directory
  35 # listings, mod_status and mod_info output etc., but not CGI generated
  36 # documents or custom error documents).
  37 # Set to "EMail" to also include a mailto: link to the ServerAdmin.
  38 # Set to one of:  On | Off | EMail
  39 #ServerSignature Off
  40 ServerSignature Email
  41 
  42 #
  43 # Allow TRACE method
  44 #
  45 # Set to "extended" to also reflect the request body (only for testing and
  46 # diagnostic purposes).
  47 #
  48 # Set to one of:  On | Off | extended
  49 TraceEnable Off
  50 #TraceEnable On
  51 
  52 #
  53 # Forbid access to version control directories
  54 #
  55 # If you use version control systems in your document root, you should
  56 # probably deny access to their directories. For example, for subversion:
  57 #
  58 #<DirectoryMatch "/\.svn">
  59 #   Require all denied
  60 #</DirectoryMatch>
  61 
  62 #
  63 # Setting this header will prevent MSIE from interpreting files as something
  64 # else than declared by the content type in the HTTP headers.
  65 # Requires mod_headers to be enabled.
  66 #
  67 Header set X-Content-Type-Options: "nosniff"
  68 
  69 #
  70 # Setting this header will prevent other sites from embedding pages from this
  71 # site as frames. This defends against clickjacking attacks.
  72 # Requires mod_headers to be enabled.
  73 #
  74 Header set X-Frame-Options: "sameorigin"
  75 
  76 
  77 # vim: syntax=apache ts=4 sw=4 sts=4 sr noet

Additional X-Headers

/etc/apache2/conf-available/x-headers.conf

   1 ### Additional Header Fields
   2 
   3 ### X-Frame-Options
   4 # Provides Clickjacking protection. Values: deny - no rendering
   5 # within a frame, sameorigin - no rendering if origin mismatch,
   6 # allow-from: DOMAIN - allow rendering if framed by frame loaded
   7 # from DOMAIN
   8 Header set X-Frame-Options: deny
   9 
  10 ### X-XSS-Protection
  11 # This header enables the Cross-site scripting (XSS) filter built into
  12 # most recent web browsers. It's usually enabled by default anyway,
  13 # so the role of this header is to re-enable the filter for this
  14 # particular website if it was disabled by the user.
  15 Header set X-XSS-Protection: "1;mode=block"
  16 
  17 ### HTTP Strict-Transport-Security (HSTS)
  18 # enforces secure (HTTP over SSL/TLS) connections to the server. This
  19 # reduces impact of bugs in web applications leaking session data through
  20 # cookies and external links and defends against Man-in-the-middle attacks.
  21 # HSTS also disables the ability for user's to ignore SSL negotiation warnings. 
  22 #Header set Strict-Transport-Security: max-age=16070400; includeSubDomains

   1 a2enconf x-headers.conf
   2 Enabling conf x-headers.
   3 To activate the new configuration, you need to run:
   4   service apache2 reload

Test the headers

You can check the HTTP-Header-Fields in Browser using

Built in Webkonsole
- F12 -> Network Analysis || Menu -> Web Developer -> Network Analysis
- Click on a HTTP-Request
the extension "Live HTTP Headers".
- In URL-Field (Ctrl+L) -> (Lock) Symbol left of URL -> Button "more information" -> Header

Crypto-Hardening

Before using this config, please generate Diffie-Hellman
openssl#Generate_or_renew_files_with_Diffie-Hellman-Parameters

/etc/apache2/conf-available/ssl_hardening.conf

   1 <IfModule mod_ssl.c>
   2         ### Mozilla SSL Configuration Generator
   3         ### https://mozilla.github.io/server-side-tls/ssl-config-generator/
   4 
   5         ### Mozilla recommendations for
   6         # Apache: 2.4.28
   7         # OpenSSL: 1.1.1b
   8 
   9         # HSTS (mod_headers is required) (15768000 seconds = 6 months)
  10         Header always set Strict-Transport-Security "max-age=15768000;includeSubDomains"
  11         #Header always set Strict-Transport-Security "max-age=15768000;includeSubDomains;preload"
  12 
  13         # https://wiki.mozilla.org/Security/Server_Side_TLS
  14         # MODERN CONFIGURATION, TWEAK TO YOUR NEEDS
  15         ### THERE ARE SOME WEAK CIPHERS (ACCORODING TO SSL-LABS)
  16         ### IF THIS WEAKER CIPHERS IN CBC-MODE ARE MISSING IE AND SAFARI BREAK
  17         #   ECDHE-ECDSA-AES256-SHA384:ECDHE-RSA-AES256-SHA384: \
  18         #   ECDHE-ECDSA-AES128-SHA256:ECDHE-RSA-AES128-SHA256"
  19         ### DO NOT APPEND "@STRENGHT" THIS WOULD PULL WEAKER-CIPHERS TO THE FRONT
  20         ### FIXME: BREAK THIS LOG LINE (SOME ANY IDEAS?)
  21         Define CIPHER_LIST              ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305:ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-SHA384:ECDHE-RSA-AES256-SHA384:ECDHE-ECDSA-AES128-SHA256:ECDHE-RSA-AES128-SHA256
  22 
  23         SSLCipherSuite                  ${CIPHER_LIST}
  24         SSLHonorCipherOrder             on
  25         ### Enabling compression causes security issues in most setups (the so called CRIME attack).
  26         SSLCompression                  off
  27         SSLSessionTickets               off
  28 
  29         SSLOptions                      +StrictRequire
  30         SSLVerifyDepth                  1
  31         SSLOpenSSLConfCmd               DHParameters    "/etc/ssl/dhparam/dhparam_4096.pem"
  32 
  33         # OCSP Stapling, only in httpd 2.3.3 and later
  34         SSLUseStapling                          on
  35         SSLStaplingResponderTimeout             5
  36         SSLStaplingReturnResponderErrors        off
  37         SSLStaplingCache                        shmcb:/var/run/ocsp(128000)
  38 
  39 
  40         ### FURTHER ADJUSTMENTS
  41         SSLProtocol                     TLSv1.3 +TLSv1.2
  42 
  43         SSLProxyEngine                  on
  44         SSLProxyCheckPeerCN             off
  45         SSLProxyCheckPeerExpire         off
  46         #SSLProxyCipherSuite            ALL:!ADH:!RC4:+HIGH:+MEDIUM
  47         SSLProxyCipherSuite             ${CIPHER_LIST}
  48 
  49         # Add vhost name to log entries:
  50         LogFormat "%h %l %u %t \"%r\" %>s %b \"%{Referer}i\" \"%{User-agent}i\"" vhost_combined
  51         LogFormat "%v %h %l %u %t \"%r\" %>s %b" vhost_common
  52 </IfModule>
  53 
  54 # vim: syntax=apache ts=4 sw=4 sts=4 sr noet

   1 a2enmod ssl
   2 a2enconf ssl_hardening

Test crypto

https://testssl.sh/ is a on premises solution that can e.g. be run via configuration management.
Qualys provides a good online scanner for http(s):
https://www.ssllabs.com/ssltest/

Performance Tuning

Apache 2.4 Docs - Apache Performance Tuning
Please also see #mod_mpm_event

Caching

Please also see #mod_cache

Virtual Hosts

I prefer using virtual hosts than the server context.

The webserver decides based on the header field "HOST" of the http-request or the TLS SNI which "vHost" should serve the response.

If another domain is configured on the server (e.g. for webmail), it's possible to configure each domain diffently with much more flexibilty. Parameters tuned in the server configuration are inherited, but can be overridden in the vHost-configuration.

NameVirtualHost

Interesting concept:

Use the Host header from the request to determine matches against the VirtualHost.
UseCanonicalName=(No)|Yes|DNS
influences population of ServerName and ServerPort.

By default name-based wirtual hosts are not used
grep -ri NameVirtualHost /etc/apache2.

So the variables SERVER_NAME and SERVER_PORT are not populated. I recommend using define statements to make them reusable on the top of your <VirtualHost>.
Example snippet

   1         Define       SERVER_NAME     hostname.domain.tld
   2         Define       SERVER_PORT     8080
   3         Define       USER_HOME       /home/template
   4         Define       HTTPD_ROOT      ${USER_HOME}/Apache2
   5         Define       APACHE_LOG_DIR  ${HTTPD_ROOT}/logs
   6 
   7         ServerName   ${SERVER_NAME}
   8 
   9         DocumentRoot htdocs/${SERVER_NAME}_${SERVER_PORT}

Crypto-vHost

CypherPunks do encrypt - anything.

So a configuration may be derived from the template /etc/apache/sites-available/default-ssl.conf to /etc/apache/sites-available/mail.rockstable.org.conf.

   1 cp /etc/apache/sites-available/{default-ssl.conf,mail.rockstable.org_443.conf}

Configure the

ServerName
ServerAlias
ServerAdmin
redirection to webmailer
inclusion of the configuration files of the application.

/etc/apache2/sites-available/mail.rockstable.org_443.conf

   1 <IfModule mod_ssl.c>
   2         <VirtualHost *:443>
   3                 Define          SERVER_NAME             mail.rockstable.org
   4                 ServerName      ${SERVER_NAME}
   5                 ServerAlias     mail.rockstable.it
   6                 ServerAdmin     webmaster@rockstable.it
   7 
   8                 #DocumentRoot /var/www/html
   9 
  10                 # Available loglevels: trace8, ..., trace1, debug, info, notice, warn,
  11                 # error, crit, alert, emerg.
  12                 # It is also possible to configure the loglevel for particular
  13                 # modules, e.g.
  14                 #LogLevel info ssl:warn
  15 
  16                 ErrorLog        ${APACHE_LOG_DIR}/${SERVER_NAME}_error.log
  17                 CustomLog       ${APACHE_LOG_DIR}/${SERVER_NAME}_access.log combined
  18 
  19 
  20                 ### SSL-CONFIG
  21                 # For most configuration files from conf-available/, which are
  22                 # enabled or disabled at a global level, it is possible to
  23                 # include a line for only one particular virtual host. For example the
  24                 # following line enables the CGI configuration for this host only
  25                 # after it has been globally disabled with "a2disconf".
  26                 #Include conf-available/serve-cgi-bin.conf
  27 
  28                 #   SSL Engine Switch:
  29                 #   Enable/Disable SSL for this virtual host.
  30                 SSLEngine on
  31 
  32                 #   A self-signed (snakeoil) certificate can be created by installing
  33                 #   the ssl-cert package. See
  34                 #   /usr/share/doc/apache2/README.Debian.gz for more info.
  35                 #   If both key and certificate are stored in the same file, only the
  36                 #   SSLCertificateFile directive is needed.
  37                 SSLCertificateFile      /etc/ssl/certs/ssl-cert-snakeoil.pem
  38                 SSLCertificateKeyFile   /etc/ssl/private/ssl-cert-snakeoil.key
  39 
  40                 #   Server Certificate Chain:
  41                 #   Point SSLCertificateChainFile at a file containing the
  42                 #   concatenation of PEM encoded CA certificates which form the
  43                 #   certificate chain for the server certificate. Alternatively
  44                 #   the referenced file can be the same as SSLCertificateFile
  45                 #   when the CA certificates are directly appended to the server
  46                 #   certificate for convinience.
  47                 #SSLCertificateChainFile /etc/apache2/ssl.crt/server-ca.crt
  48 
  49                 #   Certificate Authority (CA):
  50                 #   Set the CA certificate verification path where to find CA
  51                 #   certificates for client authentication or alternatively one
  52                 #   huge file containing all of them (file must be PEM encoded)
  53                 #   Note: Inside SSLCACertificatePath you need hash symlinks
  54                 #                to point to the certificate files. Use the provided
  55                 #                Makefile to update the hash symlinks after changes.
  56                 #SSLCACertificatePath /etc/ssl/certs/
  57                 #SSLCACertificateFile /etc/apache2/ssl.crt/ca-bundle.crt
  58 
  59                 #   Certificate Revocation Lists (CRL):
  60                 #   Set the CA revocation path where to find CA CRLs for client
  61                 #   authentication or alternatively one huge file containing all
  62                 #   of them (file must be PEM encoded)
  63                 #   Note: Inside SSLCARevocationPath you need hash symlinks
  64                 #                to point to the certificate files. Use the provided
  65                 #                Makefile to update the hash symlinks after changes.
  66                 #SSLCARevocationPath /etc/apache2/ssl.crl/
  67                 #SSLCARevocationFile /etc/apache2/ssl.crl/ca-bundle.crl
  68 
  69                 #   Client Authentication (Type):
  70                 #   Client certificate verification type and depth.  Types are
  71                 #   none, optional, require and optional_no_ca.  Depth is a
  72                 #   number which specifies how deeply to verify the certificate
  73                 #   issuer chain before deciding the certificate is not valid.
  74                 #SSLVerifyClient require
  75                 #SSLVerifyDepth  10
  76 
  77                 #   SSL Engine Options:
  78                 #   Set various options for the SSL engine.
  79                 #   o FakeBasicAuth:
  80                 #        Translate the client X.509 into a Basic Authorisation.  This means that
  81                 #        the standard Auth/DBMAuth methods can be used for access control.  The
  82                 #        user name is the `one line' version of the client's X.509 certificate.
  83                 #        Note that no password is obtained from the user. Every entry in the user
  84                 #        file needs this password: `xxj31ZMTZzkVA'.
  85                 #   o ExportCertData:
  86                 #        This exports two additional environment variables: SSL_CLIENT_CERT and
  87                 #        SSL_SERVER_CERT. These contain the PEM-encoded certificates of the
  88                 #        server (always existing) and the client (only existing when client
  89                 #        authentication is used). This can be used to import the certificates
  90                 #        into CGI scripts.
  91                 #   o StdEnvVars:
  92                 #        This exports the standard SSL/TLS related `SSL_*' environment variables.
  93                 #        Per default this exportation is switched off for performance reasons,
  94                 #        because the extraction step is an expensive operation and is usually
  95                 #        useless for serving static content. So one usually enables the
  96                 #        exportation for CGI and SSI requests only.
  97                 #   o OptRenegotiate:
  98                 #        This enables optimized SSL connection renegotiation handling when SSL
  99                 #        directives are used in per-directory context.
 100                 #SSLOptions +FakeBasicAuth +ExportCertData +StrictRequire
 101                 <FilesMatch "\.(cgi|shtml|phtml|php)$">
 102                                 SSLOptions +StdEnvVars
 103                 </FilesMatch>
 104                 <Directory /usr/lib/cgi-bin>
 105                                 SSLOptions +StdEnvVars
 106                 </Directory>
 107 
 108                 #   SSL Protocol Adjustments:
 109                 #   The safe and default but still SSL/TLS standard compliant shutdown
 110                 #   approach is that mod_ssl sends the close notify alert but doesn't wait for
 111                 #   the close notify alert from client. When you need a different shutdown
 112                 #   approach you can use one of the following variables:
 113                 #   o ssl-unclean-shutdown:
 114                 #        This forces an unclean shutdown when the connection is closed, i.e. no
 115                 #        SSL close notify alert is send or allowed to received.  This violates
 116                 #        the SSL/TLS standard but is needed for some brain-dead browsers. Use
 117                 #        this when you receive I/O errors because of the standard approach where
 118                 #        mod_ssl sends the close notify alert.
 119                 #   o ssl-accurate-shutdown:
 120                 #        This forces an accurate shutdown when the connection is closed, i.e. a
 121                 #        SSL close notify alert is send and mod_ssl waits for the close notify
 122                 #        alert of the client. This is 100% SSL/TLS standard compliant, but in
 123                 #        practice often causes hanging connections with brain-dead browsers. Use
 124                 #        this only for browsers where you know that their SSL implementation
 125                 #        works correctly.
 126                 #   Notice: Most problems of broken clients are also related to the HTTP
 127                 #   keep-alive facility, so you usually additionally want to disable
 128                 #   keep-alive for those clients, too. Use variable "nokeepalive" for this.
 129                 #   Similarly, one has to force some clients to use HTTP/1.0 to workaround
 130                 #   their broken HTTP/1.1 implementation. Use variables "downgrade-1.0" and
 131                 #   "force-response-1.0" for this.
 132                 BrowserMatch "MSIE [2-6]" \
 133                                 nokeepalive ssl-unclean-shutdown \
 134                                 downgrade-1.0 force-response-1.0
 135                 # MSIE 7 and newer should be able to use keepalive
 136                 BrowserMatch "MSIE [17-9]" ssl-unclean-shutdown
 137 
 138                 ### Redirect any request URI to the webserver root URI.
 139                 #RedirectMatch ^/$ "https://${SERVER_NAME}/webmail"
 140                 RewriteEngine on
 141                 RewriteCond %{HTTP_HOST}        !=${SERVER_NAME}
 142                 RewriteRule ^/(.*)$             https://${SERVER_NAME}/$1 [R=301,L]
 143 
 144                 ### PUT YOUR APPLICATION SPECIFIC CONFIGURATION HERE
 145                 #…
 146                 #…
 147         </VirtualHost>
 148 </IfModule>
 149 
 150 # vim: syntax=apache ts=4 sw=4 sts=4 sr noet

Enable

   1 a2enmod rewrite ssl
   2 a2ensite mail.rockstable.org_443.conf
   3 Enabling site mail.rockstable.org_443.
   4 To activate the new configuration, you need to run:
   5   service apache2 reload

Redirection to https

Nobody wants to see always "403 Forbidden".

So for the reasons of usability and convenience i recommend configuring a simple port 80 plaintext-vHost listening on the same ServerName like the encrypted counterpart, whose only task is to redirect anything to the encrypted port.

   1 cp /etc/apache/sites-available/{000-default.conf,redirect_80.conf}

#mod_rewrite can handle conditions and makes sure that status requests and acme-challenges are not rewritten to https.

/etc/apache2/sites-available/redirect_80.conf

   1 <VirtualHost *:80>
   2     # The ServerName directive sets the request scheme, hostname and port that
   3     # the server uses to identify itself. This is used when creating
   4     # redirection URLs. In the context of virtual hosts, the ServerName
   5     # specifies what hostname must appear in the request's Host: header to
   6     # match this virtual host. For the default virtual host (this file) this
   7     # value is not decisive as it is used as a last resort host regardless.
   8     # However, you must set it for any further virtual host explicitly.
   9     Define      SERVER_NAME mail.rockstable.org
  10     ServerName  ${SERVER_NAME}
  11     ServerAlias mail.rockstable.it
  12     ServerAdmin     webmaster@rockstable.it
  13 
  14     DocumentRoot /var/www/html
  15 
  16     # Available loglevels: trace8, ..., trace1, debug, info, notice, warn,
  17     # error, crit, alert, emerg.
  18     # It is also possible to configure the loglevel for particular
  19     # modules, e.g.
  20     #LogLevel info ssl:warn
  21 
  22     ErrorLog    ${APACHE_LOG_DIR}/${SERVER_NAME}_error.log
  23     CustomLog   ${APACHE_LOG_DIR}/${SERVER_NAME}_access.log combined
  24 
  25     ### Redirect anything up to the encrypted vHost
  26     ### (breaks /server-status if certificate is invalid)
  27     #RedirectMatch  ^/(.*)$                 https://${SERVER_NAME}/$1
  28 
  29     ### Finally nothing leads around mod_rewrite
  30     RewriteEngine on
  31     RewriteCond %{REQUEST_URI}      !=/server-status
  32     RewriteCond %{REQUEST_URI}      !^/\.well-known/acme-challenge(/[[:alnum:]-]*)?$
  33     RewriteRule ^/(.*)$             https://%{HTTP_HOST}/$1 [R=301,L]
  34 </VirtualHost>
  35 
  36 # vim: syntax=apache ts=4 sw=4 sts=4 sr noet

Enable and restart

   1 a2enmod rewrite ssl
   2 a2ensite mail.rockstable.org_80.conf
   3 Enabling site mail.rockstable.org_80.
   4 To activate the new configuration, you need to run:
   5   service apache2 reload

Modules

mod_auth_basic

Apache Documentation

Apache Docs - Access Control
Apache Docs - Authentication and Authorization
Apache Docs - Apache Module mod_auth_basic
For information on how to create a password file please see:
HTTP/Apache2#htpasswd

In your directory configuration (or .htaccess) place the following lines

   1 <Directory /var/www/restricted
   2         AuthType Basic
   3         AuthName "Restricted Files"
   4         # (Following line optional)
   5         AuthBasicProvider file
   6         AuthUserFile "/etc/apache/passwd/resticted"
   7         Require valid-user
   8         #Require user example-user
   9 </Directory>

mod_authz_host

Apache Documentation

With Apache2.4 authorization directives order, allow, deny, satisfy got deprecated.

Be consequent!!

Use AuthMerging, AuthzProviderAlias, AuthzSendForbiddenOnFailure, Require, RequireAny, RequireAll, RequireNone instead!

Make sure to not use use the old syntax in /etc/apache2/conf-available/security.conf otherwise subsequent configuration like mod_status will fail.

mod_autoindex

To make directory indexing a little more appealing i'm using the project "Apaxy" - a theme with a prepared configuration.

Clone the project

   1 cd /usr/local/share/
   2 git clone https://github.com/AdamWhitcroft/Apaxy

To make the changes take effect for all indexes, the file etc/apache2/mods-enabled/autoindex.conf should be moved the the side to keep the defaults.

   1 mv /etc/apache2/mods-available/autoindex.conf{,.bak}

Than merge a copy of the Apaxy-.htaccess with the original file.

Added
- <IfModule>
- #vim modeline
- <Directory>
Changed
- IndexOptions
- IndexHeadInsert
- IndexIgnore
- Some formatting
- All relative URIs theme/ changed to /theme/

   1 ### vim
   2 :%s#[, ]theme/# /theme/#
   3 ### sed
   4 sed -ir 's#([, ])theme/#\1/theme/#' /etc/apache2/mods-available/autoindex.conf

Here the full changed Configuration
/etc/apache2/mods-available/autoindex.conf

   1 <IfModule mod_autoindex.c>
   2         # Directives controlling the display of server-generated directory listings.
   3 
   4         #
   5         # Apaxy by @adamwhitcroft
   6         #
   7         # For a full breakdown of the mod_autoindex module:
   8         # http://apache.org/docs/2.2/mod/mod_autoindex.html
   9         #
  10 
  11         Alias /theme /usr/local/share/Apaxy/apaxy/theme 
  12         <Directory /usr/local/share/Apaxy/apaxy/theme>
  13                 Require all granted
  14         </Directory>
  15 
  16         <Directory /usr/local/share/Apaxy/apaxy/theme/icons>
  17                 Require all granted
  18         </Directory>
  19 
  20         # INDEX OPTIONS
  21         #IndexOptions   +Charset=UTF-8 +FancyIndexing +IgnoreCase +FoldersFirst +XHTML +HTMLTable +SuppressRules +SuppressDescription +NameWidth=* +IconsAreLinks
  22         IndexOptions    Charset=UTF-8 FancyIndexing IgnoreCase FoldersFirst XHTML HTMLTable SuppressRules SuppressDescription NameWidth=* IconsAreLinks FoldersFirst
  23 
  24         # META VIEWPORT
  25         IndexHeadInsert "  <meta name=\"viewport\" content=\"width=device-width, initial-scale=1\" />"
  26 
  27         # HIDE /theme DIRECTORY
  28         #IndexIgnore .htaccess /theme
  29         IndexIgnore .??* *~ *# RCS CVS *,v *,t /theme
  30 
  31         #
  32         # GENERAL ICONS (BLANK, DIRECTORY, PARENT DIRECTORY)
  33         #
  34 
  35         AddIcon /theme/icons/blank.png ^^BLANKICON^^
  36         AddIcon /theme/icons/folder.png ^^DIRECTORY^^
  37         AddIcon /theme/icons/folder-home.png ..
  38 
  39         AddIconByType (TXT,/theme/icons/text.png) text/*
  40         AddIconByType (IMG,/theme/icons/image.png) image/*
  41         AddIconByType (SND,/theme/icons/audio.png) audio/*
  42         AddIconByType (VID,/theme/icons/video.png) video/*
  43 
  44         #
  45         # EXTENSION SPECIFIC ICONS
  46         #
  47 
  48         AddIcon /theme/icons/archive.png .7z .bz2 .cab .gz .tar
  49         AddIcon /theme/icons/audio.png .aac .aif .aifc .aiff .ape .au .flac .iff .m4a .mid .mp3 .mpa .ra .wav .wma .f4a .f4b .oga .ogg .xm .it .s3m .mod
  50         AddIcon /theme/icons/bin.png .bin .hex
  51         AddIcon /theme/icons/bmp.png .bmp
  52         AddIcon /theme/icons/c.png .c
  53         AddIcon /theme/icons/calc.png .xlsx .xlsm .xltx .xltm .xlam .xlr .xls .csv
  54         AddIcon /theme/icons/cd.png .iso
  55         AddIcon /theme/icons/cpp.png .cpp
  56         AddIcon /theme/icons/css.png .css .sass .scss
  57         AddIcon /theme/icons/deb.png .deb
  58         AddIcon /theme/icons/doc.png .doc .docx .docm .dot .dotx .dotm .log .msg .odt .pages .rtf .tex .wpd .wps
  59         AddIcon /theme/icons/draw.png .svg .svgz
  60         AddIcon /theme/icons/eps.png .ai .eps
  61         AddIcon /theme/icons/exe.png .exe
  62         AddIcon /theme/icons/gif.png .gif
  63         AddIcon /theme/icons/h.png .h
  64         AddIcon /theme/icons/html.png .html .xhtml .shtml .htm .URL .url
  65         AddIcon /theme/icons/ico.png .ico
  66         AddIcon /theme/icons/java.png .jar
  67         AddIcon /theme/icons/jpg.png .jpg .jpeg .jpe
  68         AddIcon /theme/icons/js.png .js .json
  69         AddIcon /theme/icons/markdown.png .md
  70         AddIcon /theme/icons/package.png .pkg .dmg
  71         AddIcon /theme/icons/pdf.png .pdf
  72         AddIcon /theme/icons/php.png .php .phtml
  73         AddIcon /theme/icons/playlist.png .m3u .m3u8 .pls .pls8
  74         AddIcon /theme/icons/png.png .png
  75         AddIcon /theme/icons/ps.png .ps
  76         AddIcon /theme/icons/psd.png .psd
  77         AddIcon /theme/icons/py.png .py
  78         AddIcon /theme/icons/rar.png .rar
  79         AddIcon /theme/icons/rb.png .rb
  80         AddIcon /theme/icons/rpm.png .rpm
  81         AddIcon /theme/icons/rss.png .rss
  82         AddIcon /theme/icons/script.png .bat .cmd .sh
  83         AddIcon /theme/icons/sql.png .sql
  84         AddIcon /theme/icons/tiff.png .tiff .tif
  85         AddIcon /theme/icons/text.png .txt .nfo
  86         AddIcon /theme/icons/video.png .asf .asx .avi .flv .mkv .mov .mp4 .mpg .rm .srt .swf .vob .wmv .m4v .f4v .f4p .ogv
  87         AddIcon /theme/icons/xml.png .xml
  88         AddIcon /theme/icons/zip.png .zip
  89         DefaultIcon /theme/icons/default.png
  90 
  91         #
  92         # THEME FILES
  93         #
  94 
  95         HeaderName /theme/header.html
  96         ReadmeName /theme/footer.html
  97         IndexStyleSheet /theme/style.css
  98 
  99 # vim: syntax=apache ts=4 sw=4 sts=4 sr noet

Dann noch Apache2 neustarten

   1 apache2ctl graceful

Wer jetzt noch Lust hat, sollte /usr/local/share/Apaxy/apaxy/theme/{header.html,footer.html,style.css} an den jeweiligen Geschmack anpassen.
Sieht jetzt besser aus.

mod_alias

Apache Documentation: https://httpd.apache.org/docs/2.4/mod/mod_alias.html
Offers Redirect, RedirectMatch, …
Allow delivery of "/icons" after disabling
them during the #Hardening in security.conf -> allow from all

/etc/apache2/mods-available/alias.conf

   1 <IfModule alias_module>
   2     # Aliases: Add here as many aliases as you need (with no limit). The format is
   3     # Alias fakename realname
   4     #
   5     # Note that if you include a trailing / on fakename then the server will
   6     # require it to be present in the URL.  So "/icons" isn't aliased in this
   7     # example, only "/icons/".  If the fakename is slash-terminated, then the
   8     # realname must also be slash terminated, and if the fakename omits the
   9     # trailing slash, the realname must also omit it.
  10     #
  11     # We include the /icons/ alias for FancyIndexed directory listings.  If
  12     # you do not use FancyIndexing, you may comment this out.
  13 
  14     Alias /icons/ "/usr/share/apache2/icons/"
  15 
  16     <Directory "/usr/share/apache2/icons">
  17         Options FollowSymlinks
  18         AllowOverride None
  19         Require all granted
  20         Order Allow,Deny
  21         Allow from All
  22     </Directory>
  23 
  24 </IfModule>
  25 
  26 # vim: syntax=apache ts=4 sw=4 sts=4 sr noet

mod_brotli

https://github.com/google/brotli
IETF RFC7932 - Brotli Compressed Data Format
Wiki EN Brotli
better compression than gzip

a2enmod brotli

   1 Enabling module brotli.
   2 To activate the new configuration, you need to run:
   3   systemctl restart apache2

mod_cache

Caching of static files greatly increases the web-site performance. I should have done it for this wiki way earlier. This would have saved a tremendous amount of time. :-P

Enable necessary apache2 modules

   1 a2enmod expires headers

/etc/apache2/conf-available/expires.conf

   1 <IfModule mod_expires.c>
   2 # Turn on the module.
   3 ExpiresActive on
   4 # Set the default expiry times.
   5 #ExpiresDefault "access plus 2 days"
   6 ExpiresByType image/jpg "access plus 1 month"
   7 ExpiresByType image/gif "access plus 1 month"
   8 ExpiresByType image/jpeg "access plus 1 month"
   9 ExpiresByType image/png "access plus 1 month"
  10 ExpiresByType text/css "access plus 1 month"
  11 ExpiresByType text/javascript "access plus 1 month"
  12 ExpiresByType application/javascript "access plus 1 month"
  13 #ExpiresByType application/x-shockwave-flash "access plus 1 month"
  14 #ExpiresByType text/css "now plus 1 month"
  15 ExpiresByType image/ico "access plus 1 month"
  16 ExpiresByType image/x-icon "access plus 1 month"
  17 ExpiresByType image/vnd.microsoft.icon "access plus 1 month"
  18 ExpiresByType font/woff "access plus 1 month"
  19 #ExpiresByType text/html "access plus 600 seconds"
  20 </IfModule>

Enable the selecttive caching

   1 a2enconf expires
   2 apache2ctl configtest
   3 systemctl reload apache2

mod_dnssd

http://0pointer.de/lennart/projects/mod_dnssd/

mod_dnssd is an Apache HTTPD module, which adds Zeroconf support via DNS-SD using Avahi.

   1 apt install libapache2-mod-dnssd
   2 a2enmod dnssd
   3 systemctl reload apache2

mod_http2

Apache2 Docs - mod_http2
https://http2.github.io/faq/
Feature still experimental and changing
Let's try is out!
http/2 itself does not require encryption
- but there is currently no browser that supports http/2 without TLSv1.2
- so encryption is kind of mandatory, but that's alright.

/etc/apache2/mods-available/http2.conf

   1 ## h2c  -> http2 cleartext
   2 ## h2   -> http2 on TLS
   3 
   4 Protocols h2c h2 http/1.1
   5 
   6 #H2 Direct Protocol Switch
   7 # Best in VirtualHost
   8 #H2Direct on for h2c, off for h2 protocol
   9 #H2Direct on
  10 
  11 #Maximum number of active streams per HTTP/2 session.
  12 #H2MaxSessionStreams 100
  13 
  14 #Maximum number of seconds h2 workers remain idle until shut down.
  15 #H2MaxWorkerIdleSeconds 600
  16 
  17 #Maximum number of worker threads to use per child process.
  18 #H2MaxWorkers n
  19 
  20 #Minimal number of worker threads to use per child process.
  21 #H2MinWorkers n
  22 
  23 #Require HTTP/2 connections to be "modern TLS" only
  24 #H2ModernTLSOnly on
  25 
  26 #H2 Server Push Switch
  27 #H2Push on
  28 
  29 #H2PushPriority
  30 #H2PushPriority * After 16
  31 
  32 #Serialize Request/Response Processing Switch
  33 #H2SerializeHeaders off
  34 
  35 #Number of Extra File Handles
  36 #H2SessionExtraFiles 5
  37 
  38 #Maximum amount of output data buffered per stream.
  39 #H2StreamMaxMemSize 65536
  40 
  41 #
  42 #H2TLSCoolDownSecs 1
  43 
  44 #
  45 #H2TLSWarmUpSize 1048576
  46 
  47 #H2 Upgrade Protocol Switch
  48 # Best in VirtualHost
  49 #H2Upgrade on for h2c, off for h2 protocol
  50 #H2Upgrade on|off
  51 
  52 #Size of Stream Window for upstream data.
  53 #H2WindowSize 65536

a2enmod http2

   1 Enabling module http2.
   2 To activate the new configuration, you need to run:
   3   service apache2 restart

After a reload the server response is "HTTP/2.0 200 OK", nice!

mod_info

Apache Documentation: https://httpd.apache.org/docs/2.4/mod/mod_info.html
You should disable this module for security reasons.
You may also gain this info using the commandline to
```
   1 for OPTION in -l -L -S -M -v -V "-t -D DUMP_INCLUDES"; do
   2         echo
   3         eval apache2ctl $OPTION;
   4 done
```

Enable module

   1 # a2enmod info
   2 Enabling module info.
   3 To activate the new configuration, you need to run:
   4   service apache2 restart

https://localhost/server-info

Disable module

   1 # a2dismod info 
   2 Module info disabled.
   3 To activate the new configuration, you need to run:
   4   systemctl restart apache2

mod_nss

https://fedorahosted.org/mod_nss/
mod_nss is an SSL provider derived from the mod_ssl module for the Apache web server that uses the Network Security Services (NSS) libraries. We started with mod_ssl and replaced the OpenSSL calls with NSS calls.
not really necessary

   1 root@mail ~log/apache2 # a2enmod nss
   2 Enabling module nss.
   3 To activate the new configuration, you need to run:
   4   service apache2 restart

mod_php5

Please consider using php5-fpm and mod_fastcgi, instead of mod_php5 (libapache2-mod-php5) with "suexec" or sth. comparable. It circumvents some security problems of the default configuration, where the interpreter of php is running in the context of "www-data", while it shouldn't be able to read and deliver some files (readable by the webserver) to the client.

mod_mpm_event

Apache Docs - Apache MPM event
Uses super-process to bind against Port 80 with root privileges, forks privilege seperated child processes, which do spawn a listener/supporter connections (which handles new and idle thread) and a amount of worker-threads (that do the work).
Worker derivate that handles idle connections by passing them to supporter threads and frees up system-resources and workers in that way.
Thanks to the setup with php5-fpm it works even with PHP applications, where with mod_php5 i formerly was forced to use mpm_prefork only.

   1 root@mail /etc/apache2 # a2dismod mpm_prefork
   2 Module mpm_prefork disabled.
   3 To activate the new configuration, you need to run:
   4   service apache2 restart
   5 root@mail /etc/apache2 # a2enmod mpm_event
   6 Considering conflict mpm_worker for mpm_event:
   7 Considering conflict mpm_prefork for mpm_event:
   8 Enabling module mpm_event.
   9 To activate the new configuration, you need to run:
  10   service apache2 restart
  11 root@mail /etc/apache2 # systemctl restart apache2.service
  12

Benchmarking mpm_event

Please verify your results with the following commands. Please be aware that the requests/second are only a average value.

   1 watch -n .1 -- apachectl status

ab can handle up to maybe 10 concurrent connections and generated ~1200 requests/second on my workstation (Ryzen Threadripper 1950X) before becoming to much limited by the cpu.

   1 for ((i = 1 ; i < 21; i++ )); do
   2         nice -n 0 ab -n 10000 -c $i https://localhost/server-status 2>&1 \
   3         |grep -A2 \
   4                 -e 'Concurrency Level:' \
   5                 -e 'Requests per second:'
   6         echo
   7 done
   8 
   9 …
  10 Concurrency Level:      10
  11 Time taken for tests:   8.452 seconds
  12 Complete requests:      10000
  13 --
  14 Requests per second:    1183.21 [#/sec] (mean)
  15 Time per request:       8.452 [ms] (mean)
  16 Time per request:       0.845 [ms] (mean, across all concurrent requests)
  17 …

So you'll need multiple ab processes to saturate your server as this tool is largely cpu bound and does not use multiple cores. Hint: This is a naiv benchmark!

ab_wrapper.sh

   1 #!/bin/bash
   2 
   3 ### INIT
   4 PROGRAM="${0#*/}"
   5 URL="https://localhost/server-status"
   6 CONCURRENCY=10
   7 INSTANCES=12
   8 NUMBER=10000
   9 
  10 settings () {
  11         cat <<-EOF
  12                 URL:            "$URL"
  13                 CONCURRENCY:    "$CONCURRENCY"
  14                 INSTANCES:      "$INSTANCES"
  15                 NUMBER:         "$NUMBER"
  16 
  17         EOF
  18 }
  19 
  20 usage () {
  21         cat <<-EOF
  22                 $PROGRAM [OPTIONS] [--] [URL]
  23 
  24                 Options:
  25                         -h|--help               Print this page
  26                         -c|--concurrency        Concurrent connections in ab
  27                         -i|--instances          Parallel instances of ab
  28                         -n|--number             Number of requests per instance
  29                         --                      End parameter parsing
  30 
  31                 [] optional
  32                 () mandatory
  33 
  34                 Defaults:
  35         EOF
  36         settings
  37 }
  38 
  39 # Note that we use "$@" to let each command-line parameter expand to a
  40 # separate word. The quotes around "$@" are essential!
  41 # We need TEMP as the 'eval set --' would nuke the return value of getopt.
  42 TEMP=$(getopt \
  43         -o 'hc:i:n:' \
  44         --long 'help,concurrency:,instances:,number:' \
  45         -n "$PROGRAM" -- "$@")
  46 
  47 if [ $? -ne 0 ]; then
  48         echo 'Terminating...' >&2
  49         exit 1
  50 fi
  51 
  52 # Note the quotes around "$TEMP": they are essential!
  53 eval set -- "$TEMP"
  54 unset TEMP
  55 
  56 while true; do
  57         case "$1" in
  58                 '-h'|'--help')
  59                         usage
  60                         exit
  61                 ;;
  62                 '-c'|'--concurrency')
  63                         CONCURRENCY="$2"
  64                         shift 2
  65                         continue
  66                 ;;
  67                 '-n'|'--number')
  68                         NUMBER="$2"
  69                         shift 2
  70                         continue
  71                 ;;
  72                 '-i'|'--instances')
  73                         INSTANCES="$2"
  74                         shift 2
  75                         continue
  76                 ;;
  77                 #'-c'|'--c-long')
  78                 #       # c has an optional argument. As we are in quoted mode,
  79                 #       # an empty parameter will be generated if its optional
  80                 #       # argument is not found.
  81                 #       case "$2" in
  82                 #               '')
  83                 #                       echo 'Option c, no argument'
  84                 #               ;;
  85                 #               *)
  86                 #                       echo "Option c, argument '$2'"
  87                 #               ;;
  88                 #       esac
  89                 #       shift 2
  90                 #       continue
  91                 #;;
  92                 '--')
  93                         shift
  94                         break
  95                 ;;
  96                 *)
  97                         echo 'Internal error!' >&2
  98                         exit 1
  99                 ;;
 100         esac
 101 done
 102 
 103 if [ "$1" ]; then
 104         URL="$1"
 105 fi
 106 
 107 ### SANITY CHECKS
 108 if ! grep -qE '^(https?://)?[^/]+/[^/]*?$' <<< "$URL"; then
 109         echo "URL '$URL' must contain an '/' in the URI. Exiting …"
 110         exit 1
 111 fi
 112 
 113 echo "Benchmark settings:"
 114 settings
 115 
 116 for ((k = 1 ; k <= INSTANCES; k++ )); do
 117         ### RESTART APACHE TO START CLEAN
 118         #systemctl restart apache2.service
 119 
 120         echo "### Benchmarking with $k instances of ab"
 121         RUN="$(for ((i = 0 ; i < k; i++ )); do
 122                 nice -n 0 ab -n "$NUMBER" -c "$CONCURRENCY" "$URL" &
 123         done)"
 124 
 125         REQ_SUM="$(<<< "$RUN" grep 'Requests per second:' \
 126                 |awk '{ sum += $4 } END { print sum }'
 127         )"
 128         REQ_MEAN="$(<<< "$RUN" grep 'Requests per second:' \
 129                 |awk '{ sum += $4 } END { if (NR > 0) print sum / NR }'
 130         )"
 131         TIME_MEAN="$(<<< "$RUN" grep 'Time taken for tests:' \
 132                 |awk '{ sum += $5 } END { if (NR > 0) print sum / NR }'
 133         )"
 134 
 135         PERF="$(bc <<< "$REQ_SUM * $k / $TIME_MEAN")"
 136 
 137         cat <<-EOF
 138 
 139                 #### Results:
 140                 Requests (sum):  '$REQ_SUM'
 141                 Requests (mean): '$REQ_MEAN'
 142                 Time (mean):     '$TIME_MEAN'
 143                 Performance:     '$PERF'
 144 
 145         EOF
 146 
 147         #### RAW DATA
 148         echo "$RUN"
 149         echo
 150 
 151         #while (pidof ab > /dev/null); do
 152         #       sleep 1;
 153         #done
 154 done

Run the script

   1 chmod a+x ./ab_wrapper.sh
   2 ./ab_wrapper.sh -h
   3 sudo ./ab_wrapper.sh |tee /tmp/bench_$(date +%F_%H%M).log
   4 grep -A4 '###' /tmp/bench_*.log

Tuning mpm_event

The mpm_event default config is something like this
/etc/apache2/mods-available/mpm_event.conf

   1 # event MPM
   2 # StartServers: initial number of server processes to start
   3 # MinSpareThreads: minimum number of worker threads which are kept spare
   4 # MaxSpareThreads: maximum number of worker threads which are kept spare
   5 # ThreadsPerChild: constant number of worker threads in each server process
   6 # MaxRequestWorkers: maximum number of worker threads
   7 # MaxConnectionsPerChild: maximum number of requests a server process serves
   8 
   9 <IfModule mpm_event_module>
  10     StartServers             2
  11     MinSpareThreads      25
  12     MaxSpareThreads      75
  13     ThreadLimit          64
  14     ThreadsPerChild      25
  15     MaxRequestWorkers     150
  16     MaxConnectionsPerChild   0
  17 </IfModule>
  18 
  19 # vim: syntax=apache ts=4 sw=4 sts=4 sr noet

Maybe lower values of ThreadsPerChild are more responsive when it comes to latency? The values above finally depend on this setting.
ServerLimit seams to have default value of 16.

Please verify your config with apache2ctl configtest first. If your ServerLimit is to low, it will output a warning like the following

   1 AH00515: WARNING: MaxRequestWorkers of 4096 would require 128 servers and would exceed ServerLimit of 2, decreasing to 64. To increase, please see the ServerLimit directive.
   2 Syntax OK

According to the Apache2 Docs for mpm_common setting the option needs a true restart of Apache2. apache2ctl graceful or apache2ctl restart won't work.

   1 systemctl restart apache2.service

With the defaults the script peaked

   1 ### Benchmarking 8 instances of ab
   2 
   3 #### Results:
   4 Requests (sum):  '6365.25'
   5 Requests (mean): '795.656'
   6 Time (mean):     '12.5682'
   7 Performance:     '4051'

16 processes with threads can easily saturate a 32 core cpu.

With the defaults, MaxRequestWorkers 8000 and ServerLimit 320 the script peaked

   1 ### Benchmarking 6 instances of ab
   2 
   3 #### Results:
   4 Requests (sum):  '3802.84'
   5 Requests (mean): '633.807'
   6 Time (mean):     '15.7777'
   7 Performance:     '1446'

So it seems not the be the best option since many resources are wasted in overhead.

With the defaults, MaxRequestWorkers 400 the scripts peaked

   1 --
   2 ### Benchmarking 8 instances of ab
   3 
   4 #### Results:
   5 Requests (sum):  '6393.74'
   6 Requests (mean): '799.218'
   7 Time (mean):     '12.5124'
   8 Performance:     '4087'

If you have large amounts of concurrent requests you may need a slightly more beefy configuration.
/etc/apache2/mods-available/mpm_event.conf

   1 <IfModule mpm_event_module>
   2     StartServers             2
   3     MinSpareThreads      32
   4     MaxSpareThreads      96
   5     ThreadLimit          96
   6     ThreadsPerChild      32
   7     MaxRequestWorkers     512
   8     MaxConnectionsPerChild   0
   9 </IfModule>

the scripts peaked

   1 ### Benchmarking 8 instances of ab
   2 
   3 #### Results:
   4 Requests (sum):  '6193.62'
   5 Requests (mean): '774.202'
   6 Time (mean):     '12.9165'
   7 Performance:     '3836'

My benchmark was with ab and the webserver on the same machine. Sure not optimal. But I think the more beefy config should be remembered for the near future.

mod_rewrite

Extremly powerful module.
Extremly complex and complicated to maintain.
Probably already enabled in your server.
Lot's of caveats and trip-wires.

Enable mod_rewrite

   1 # a2enmod rewrite
   2 Module rewrite already enabled

You may want to enable a higher log level when writing and testing rewrite rules

   1         # Available loglevels: trace8, ..., trace1, debug, info, notice, warn,
   2         # error, crit, alert, emerg.
   3         # It is also possible to configure the loglevel for particular
   4         # modules, e.g.
   5         #LogLevel info ssl:warn
   6         LogLevel info ssl:warn rewrite:trace8

mod_ssl

Please see openssl on first steps of preparation.

Apache Documentation: https://httpd.apache.org/docs/2.4/mod/mod_ssl.html#sslprotocol

For me it's import to serve a clean, secure and state of the art configuration. So hardening of the ssl-stack is a mandtory.
- Always have that latest security patches for (lib|open)ssl, (lib)gnutls installed.
- Use strong crypto and disable broken suites, ciphers and hashes and don't let yourself be compromised by old clients and compatibilty. They're not worth it. (See man 1 ciphers). Disable:
  - Protocols:
    - SSLv2 (gone with openssl 1.0.1)
    - SSLv3 (vulnerable against POODLE)
    - TLSv1.0 (vulnerable against BEAST/POODLE)
    - TLSv1.1 can be left in the list of protocols for reasons of compatibilty (no known "real" vulnerabilities, yet)
  - CipherSuites
    - eNULL (no encryption)
    - aNULL (no authentication (e.g. anonymous diffie hellmann) ( vulnerable against Man In The Middle))
    - EXPORT (weakened and designed to be broken (40 and 56bits))
    - LOW (40 and 56 bit without exports)
    - NEDIUM (128bit ciphers) (but, we want strong crypto)
  - Ciphers
    - (A)RC4 terminally broken
    - (AES128 - Tried to exclude it, but this breaks compatibility with Iceweasel, no way…)
  - Hashes
    - MD5 (terminally broken, definitely insecure)
    - SHA1 (not free of collisions)
  - Order the string by cipher strength -> @STRENGTH

Default with openssl is currently 'ALL:!EXPORT:!aNULL:!eNULL:!SSLv2' So mine is:

   1 SSLCipherSuite @STRENGTH:HIGH:!aNULL:!eNULL:!EXPORT:!LOW:!MEDIUM:!SSLv2:!SSLv3:!TLSv1:!MD5:!SHA
   2 SSLProtocol TLSv1.2 +TLSv1.1 -SSLv2 -SSLv3 -TLSv1

Verify your SSLCipherSuite-String using:

   1 root@mail /etc/apache2 # openssl ciphers cipherlist '@STRENGTH:HIGH:!aNULL:!eNULL:!EXPORT:!LOW:!MEDIUM:!SSLv2:!SSLv3:!TLSv1:!MD5:!SHA'
   2 ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-SHA384:ECDHE-ECDSA-AES256-SHA384:DHE-DSS-AES256-GCM-SHA384:DHE-RSA-AES256-GCM-SHA384:DHE-RSA-AES256-SHA256:DHE-DSS-AES256-SHA256:ECDH-RSA-AES256-GCM-SHA384:ECDH-ECDSA-AES256-GCM-SHA384:ECDH-RSA-AES256-SHA384:ECDH-ECDSA-AES256-SHA384:AES256-GCM-SHA384:AES256-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-SHA256:ECDHE-ECDSA-AES128-SHA256:DHE-DSS-AES128-GCM-SHA256:DHE-RSA-AES128-GCM-SHA256:DHE-RSA-AES128-SHA256:DHE-DSS-AES128-SHA256:ECDH-RSA-AES128-GCM-SHA256:ECDH-ECDSA-AES128-GCM-SHA256:ECDH-RSA-AES128-SHA256:ECDH-ECDSA-AES128-SHA256:AES128-GCM-SHA256:AES128-SHA256

/etc/apache2/mods-available/ssl.conf

   1 <IfModule mod_ssl.c>
   2 
   3     # Pseudo Random Number Generator (PRNG):
   4     # Configure one or more sources to seed the PRNG of the SSL library.
   5     # The seed data should be of good random quality.
   6     # WARNING! On some platforms /dev/random blocks if not enough entropy
   7     # is available. This means you then cannot use the /dev/random device
   8     # because it would lead to very long connection times (as long as
   9     # it requires to make more entropy available). But usually those
  10     # platforms additionally provide a /dev/urandom device which doesn't
  11     # block. So, if available, use this one instead. Read the mod_ssl User
  12     # Manual for more details.
  13     #
  14     SSLRandomSeed startup builtin
  15     SSLRandomSeed startup file:/dev/urandom 512
  16     SSLRandomSeed connect builtin
  17     SSLRandomSeed connect file:/dev/urandom 512
  18 
  19     ##
  20     ##  SSL Global Context
  21     ##
  22     ##  All SSL configuration in this context applies both to
  23     ##  the main server and all SSL-enabled virtual hosts.
  24     ##
  25 
  26     #
  27     #   Some MIME-types for downloading Certificates and CRLs
  28     #
  29     AddType application/x-x509-ca-cert .crt
  30     AddType application/x-pkcs7-crl .crl
  31 
  32     #   Pass Phrase Dialog:
  33     #   Configure the pass phrase gathering process.
  34     #   The filtering dialog program (`builtin' is a internal
  35     #   terminal dialog) has to provide the pass phrase on stdout.
  36     SSLPassPhraseDialog  exec:/usr/share/apache2/ask-for-passphrase
  37 
  38     #   Inter-Process Session Cache:
  39     #   Configure the SSL Session Cache: First the mechanism 
  40     #   to use and second the expiring timeout (in seconds).
  41     #   (The mechanism dbm has known memory leaks and should not be used).
  42     #SSLSessionCache         dbm:${APACHE_RUN_DIR}/ssl_scache
  43     SSLSessionCache     shmcb:${APACHE_RUN_DIR}/ssl_scache(512000)
  44     SSLSessionCacheTimeout  300
  45 
  46     #   Semaphore:
  47     #   Configure the path to the mutual exclusion semaphore the
  48     #   SSL engine uses internally for inter-process synchronization. 
  49     #   (Disabled by default, the global Mutex directive consolidates by default
  50     #   this)
  51     #Mutex file:${APACHE_LOCK_DIR}/ssl_mutex ssl-cache
  52 
  53 
  54     #   SSL Cipher Suite:
  55     #   List the ciphers that the client is permitted to negotiate. See the
  56     #   ciphers(1) man page from the openssl package for list of all available
  57     #   options.
  58     #   Enable only secure ciphers:
  59     SSLCipherSuite @STRENGTH:HIGH:!aNULL:!eNULL:!EXPORT:!LOW:!MEDIUM:!SSLv2:!SSLv3:!TLSv1:!MD5:!SHA
  60 
  61     # SSL server cipher order preference:
  62     # Use server priorities for cipher algorithm choice.
  63     # Clients may prefer lower grade encryption.  You should enable this
  64     # option if you want to enforce stronger encryption, and can afford
  65     # the CPU cost, and did not override SSLCipherSuite in a way that puts
  66     # insecure ciphers first.
  67     # Default: Off
  68     SSLHonorCipherOrder on
  69 
  70     #   The protocols to enable.
  71     #   Available values: all, SSLv3, TLSv1, TLSv1.1, TLSv1.2
  72     #   SSL v2  is no longer supported
  73     SSLProtocol TLSv1.2 +TLSv1.1 -SSLv2 -SSLv2 -TLSv1
  74 
  75     #   Allow insecure renegotiation with clients which do not yet support the
  76     #   secure renegotiation protocol. Default: Off
  77     #SSLInsecureRenegotiation on
  78 
  79     #   Whether to forbid non-SNI clients to access name based virtual hosts.
  80     #   Default: Off
  81     #SSLStrictSNIVHostCheck On
  82 
  83 </IfModule>
  84 
  85 # vim: syntax=apache ts=4 sw=4 sts=4 sr noet

In the next step, test config, enable mod_ssl and restart the server.

   1 root@mail /etc/apache2/mods-available # apache2ctl configtest
   2 Syntax OK
   3 root@mail /etc/apache2 # a2enmod ssl
   4 Considering dependency setenvif for ssl:
   5 Module setenvif already enabled
   6 Considering dependency mime for ssl:
   7 Module mime already enabled
   8 Considering dependency socache_shmcb for ssl:
   9 Enabling module socache_shmcb.
  10 Enabling module ssl.
  11 See /usr/share/doc/apache2/README.Debian.gz on how to configure SSL and create self-signed certificates.
  12 To activate the new configuration, you need to run:
  13   service apache2 restart

Test it

After these steps a vHost listening on Port 443 should be established, to test the configuration please see #Crypto-vHost.

Here are some utilities to test the Webserver:

Qualys Test: https://www.ssllabs.com/ssltest/
- I landed on Grade "A-" besides the untrusted (self-signed) certificate. More on hardening the crypto later. Minimum requirement should be somewhat above Grade "B", like google provides (for compatibilty reasons) does.

mod_status

Apache-Documentation: Apache2 docs mod_status

With all requests on port 80 being redirected to 443 a little tweak to the way the apache2-status is fetched is necessary. There are several possibilities:

The usage of mod_rewrite in the port 80 vhost circumvents this problems, but may offer slightly less performance.
The redirected vhost must (like below)
be part of the whitelist-list of mod_status -> allow-from. Once we've got here a valid certificate installed, the traffic is even encrypted.
- A RedirectMatch can be placed to push any URI up to https like documented in #Redirection to https.
- If we like we can change the APACHE_STATUSURL
  in /etc/apache2/envvars to the encrypted path. The Host must be replaced with the FQDN (the !commonName in the subject) since the certificate is not valid for "localhost". APACHE_STATUSURL="https://mail.rockstable.org/server-status"

   1 <IfModule mod_status.c>
   2     # Allow server status reports generated by mod_status,
   3     # with the URL of http://servername/server-status
   4     # Uncomment and change the "192.0.2.0/24" to allow access from other hosts.
   5 
   6     <Location /server-status>
   7         SetHandler server-status
   8         Require local
   9         #Require ip 192.0.2.0/24
  10         ### Don't use the old stuff anymore
  11         #order deny,allow
  12         #deny from   all
  13         #allow from  localhost
  14         #allow from  mail.rockstable.org
  15     </Location>
  16 
  17     # Keep track of extended status information for each request
  18     ExtendedStatus On
  19 
  20     # Determine if mod_status displays the first 63 characters of a request or
  21     # the last 63, assuming the request itself is greater than 63 chars.
  22     # Default: Off
  23     #SeeRequestTail On
  24 
  25     <IfModule mod_proxy.c>
  26         # Show Proxy LoadBalancer status in mod_status
  27         ProxyStatus On
  28     </IfModule>
  29 
  30 </IfModule>
  31 
  32 # vim: syntax=apache ts=4 sw=4 sts=4 sr noet

mod_suexec

suexec - Switch user before executing external programs

suexec is used by the Apache HTTP Server to switch to another user before executing CGI programs. In order to achieve this, it must run as root. Since the HTTP daemon normally doesn't run as root, the suexec executable needs the setuid bit set and must be owned by root. It should never be writable for any other person than root.

The developers recommend reading this documentation first:

There are two variants

apache2-suexec-pristine

Configured at compile time and faster.

/usr/lib/apache2/suexec-pristine -V

   1  -D AP_DOC_ROOT="/var/www"
   2  -D AP_GID_MIN=100
   3  -D AP_HTTPD_USER="www-data"
   4  -D AP_LOG_EXEC="/var/log/apache2/suexec.log"
   5  -D AP_SAFE_PATH="/usr/local/bin:/usr/bin:/bin"
   6  -D AP_UID_MIN=100
   7  -D AP_USERDIR_SUFFIX="public_html"

apache2-suexec-custom

Configurable and slower (filesystem IO).

/usr/lib/apache2/suexec-custom -V

   1  -D SUEXEC_CONFIG_DIR=/etc/apache2/suexec/
   2  -D AP_GID_MIN=100
   3  -D AP_LOG_EXEC="/var/log/apache2/suexec.log"
   4  -D AP_SAFE_PATH="/usr/local/bin:/usr/bin:/bin"
   5  -D AP_UID_MIN=100

/etc/apache2/suexec/www-data

   1 /var/www
   2 public_html/cgi-bin
   3 # The first two lines contain the suexec document root and the suexec userdir
   4 # suffix. If one of them is disabled by prepending a # character, suexec will
   5 # refuse the corresponding type of request.
   6 # This config file is only used by the apache2-suexec-custom package. See the
   7 # suexec man page included in the package for more details.

You may select the desired suexec implementation, when both modules are installed.

   1 # update-alternatives --config suexec
   2 There are 2 choices for the alternative suexec (providing /usr/lib/apache2/suexec).
   3 
   4   Selection    Path                              Priority   Status
   5 ------------------------------------------------------------
   6 * 0            /usr/lib/apache2/suexec-custom     10        auto mode
   7   1            /usr/lib/apache2/suexec-custom     10        manual mode
   8   2            /usr/lib/apache2/suexec-pristine   10        manual mode
   9 
  10 Press <enter> to keep the current choice[*], or type selection number: 2
  11 update-alternatives: using /usr/lib/apache2/suexec-pristine to provide /usr/lib/apache2/suexec (suexec) in manual mode
  12 # readlink -f /usr/lib/apache2/suexec
  13 /usr/lib/apache2/suexec-pristine

The suexec executable is at
ls -l /usr/lib/apache2/suexec*

   1 lrwxrwxrwx 1 root root        24 Mar  8 17:25 /usr/lib/apache2/suexec -> /etc/alternatives/suexec
   2 -r-sr-xr-- 1 root www-data 26856 Jan  5 15:49 /usr/lib/apache2/suexec-custom
   3 -r-sr-xr-- 1 root www-data 22760 Jan  5 15:49 /usr/lib/apache2/suexec-pristine

Filesystem permissions:

The file must be owned by root and the setuid bit must be set.
The webservers group www-data may execute suexec.
With suexec-pristine only the user www-data may execute suexec.

Setting tight unix-permissions is essential!

   1 chown root:www-data /usr/lib/apache2/suexec*
   2 chmod 4750 /usr/lib/apache2/suexec*

The Apache2 portion now knowns how to pass the arguments to suexec. cat /etc/apache2/mods-available/suexec.load

   1 LoadModule suexec_module /usr/lib/apache2/modules/mod_suexec.so

Enable the module

   1 # a2enmod suexec
   2 Enabling module suexec.
   3 To activate the new configuration, you need to run:
   4   systemctl restart apache2

Create the system user (uidNumber > 100) and system group (gidNumber > 100).

Than in your vHost configure

   1         <IfModule mod_suexec.c>
   2                 SuexecUserGroup target_user target_group
   3         </IfModule>

The CGI script and the containing directory passed to suexec, must be owned by target_user target_group.