Icinga2
Contents
- Icinga2
- Icingaweb2
About
Conferences:
Many thanks to Christian Stein of NETWAYS GmbH from Nürnberg Germany for your enlightening Webinars in German language !1!! For those how missed them, here's a link to the NETWAYS Icinga2 playlist.
Distributed Monitoring
Tiers:
- A master node has no parent node.
A master node is where you usually install IcingaWeb 2.
- A master node can combine executed checks from child nodes into backends and notifications.
- A satellite node has a parent and a child node.
- A satellite node may execute checks on its own or delegate check execution to child nodes.
- A satellite node can receive configuration for hosts/services, etc. from the parent node.
- A satellite node continues to run even if the master node is temporarily unavailable.
- An agent node only has a parent node.
- An agent node will either run its own configured checks or receive command execution events from the parent node.
Automation
Ansible
Two modules are available:
- icinga2_feature
- icinga2_host
Installation
1 /usr/share/icingaweb2/etc/schema
Debugging
To trace events in icinga2 you may enable the feature debuglog which creates the file /var/log/icinga2/icinga2.log.
Log Files:
/var/log/icinga2/icinga2.log
/var/log/icinga2/error.log
/var/log/icinga2/debug.log
/var/lib/icinga2/api/zones-stage/startup.log
Configure
Increase ulimits
Standard ulimit for max open files in Debian is 1024. Icinga2 will easily pass this threshold once you got some checks and plugins will raise the following error:
Too check the current limits of the process:
Icinga2 is not a login process and therefore will not honor pam-limits. You will need to raise the limits from within the execution path. Icinga2 has a environment files which is also sourced by systemd. Just add a line for the corresponding limit.
/etc/default/icinga2
Restart icinga2 and it will be fine.
1 systemctl restart icinga2
Prepare postgres
Create user, db and grant
Configure postgres authentication /etc/postgresql/11/main/pg_hba.conf
1 # DO NOT DISABLE!
2 # If you change this first entry you will need to make sure that the
3 # database superuser can access the database using some other method.
4 # Noninteractive access to all databases is required during automatic
5 # maintenance (custom daily cronjobs, replication, and similar tasks).
6 #
7 # Database administrative login by Unix domain socket
8 local all postgres peer
9
10 # TYPE DATABASE USER ADDRESS METHOD
11 local icinga2 icinga2 md5
12 local icingaweb2 icingaweb2 md5
13 local icingaweb2_director icingaweb2_director md5
14 local icingaweb2_x509 icingaweb2_x509 md5
15
16 # "local" is for Unix domain socket connections only
17 local all all peer
18 # IPv4 local connections:
19 host all all 127.0.0.1/32 md5
20 # IPv6 local connections:
21 host all all ::1/128 md5
22 # Allow replication connections from localhost, by a user with the
23 # replication privilege.
24 local replication all peer
25 host replication all 127.0.0.1/32 md5
26 host replication all ::1/128 md5
Reload postgres
1 systemctl reload postgres.service
Timezone
/etc/php/7.3/fpm/php.ini
Master
Icinga2 node wizard
1 root@mon1 /etc/icinga2 # icinga2 node wizard
2 Welcome to the Icinga 2 Setup Wizard!
3
4 We will guide you through all required configuration details.
5
6 Please specify if this is an agent/satellite setup ('n' installs a master setup) [Y/n]: n
7
8 Starting the Master setup routine...
9
10 Please specify the common name (CN) [mon1.1a.rockstable.it]:
11 Reconfiguring Icinga...
12 Checking for existing certificates for common name 'mon1.1a.rockstable.it'...
13 Certificates not yet generated. Running 'api setup' now.
14 Generating master configuration for Icinga 2.
15 Enabling feature api. Make sure to restart Icinga 2 for these changes to take effect.
16
17 Master zone name [master]:
18
19 Default global zones: global-templates director-global
20 Do you want to specify additional global zones? [y/N]:
21 Please specify the API bind host/port (optional):
22 Bind Host []:
23 Bind Port []:
24
25 Do you want to disable the inclusion of the conf.d directory [Y/n]:
26 Disabling the inclusion of the conf.d directory...
27 Checking if the api-users.conf file exists...
28
29 Done.
30
31 Now restart your Icinga 2 daemon to finish the installation!
Create a ApiUser /etc/icinga2/conf.d/api-users.conf
Options like bind_host and bind_port may be changed in /etc/icinga2/features-available/api.conf
Zones may be changed in /etc/icinga2/zones.conf
1 /*
2 * Generated by Icinga 2 node setup commands
3 * on 2019-12-03 08:15:47 +0100
4 */
5
6 object Zone "global-templates" {
7 global = true
8 }
9
10 object Zone "director-global" {
11 global = true
12 }
13
14 object Endpoint "mon1.mon1.rockstable.it" {
15 }
16
17 object Zone "master" {
18 endpoints = [ "mon1.mon1.rockstable.it" ]
19 }
Some notes on zones
Members of the same zone should have the same features enabled or won't fail over correctly.
From my experience by experimenting with zones, endpoints and the director, importing the global zones director-global or global-templates into the director are a big cumberstone. Please be patient, if you don't understand instantly.
They serve as an predefined interface between icinga2 parents and children as well as from the director to the icinga2 core for configuration. You need them in the director to e.g. get apply-rules working.
Their definition will become duplicate once you try to roll out the config via the API (director). This is a clash of config and Icinga2 cannot determine which on is to be used for boot-strapping. There are no priorities, merge strategies or something like that to resolve it. This is known as Bug #7530 and you will be notified by the director prior to the rollout.
When importing the zones
- Make sure to set these zones global in director, too.
Apply rules are deployed to director-global and the resulting services wouldn't get scheduled for distribution.
- Make sure the standard global zones are provided by
the icinga2-configuration file /etc/icinga2/zones.conf.
Make sure /etc/icinga2/zones.conf is included by /etc/icinga2/icinga2.conf.
Either your agents or your masters must have a host = entry to initiate the connection to the other side.
- Firewall must be set free in the corresponding direction
(probably master -> agent).
You may need to delete previously received config via icinga2 API
1 rm -r /var/lib/icinga2/api/{zones,zones-statging}/*- To resolve BUG #7530 in your environment,
you need to change the object_type of the database records of these zones from object to external_object. Icinga2 director will treat this type records as defined externally (e.g. by configuration management or package defaults) and won't role it out via "director-global", what previously led to duplicity.
Restart Icinga2
1 systemctl restart icinga2.service
You may now you your Icinga2#Director to push the configuration to the master and their children.
Satellite
Agent
Agent nodes also have their own unique zone. By convention you must use the FQDN for the zone name.
Decide between CSR-Auto-Signing and On-Demand-Signing.
CSR-Auto-Signing
Use the ticket on the client
On-Demand-Signing
On the client
On the master list pending certificate signing requests.
Sign the pending certificate signing request of your choice.
1 icinga2 ca sign "$HASH_SHA256"
Features
Icinga Data Output (IDO)
The IDO (Icinga Data Output) feature for Icinga 2 takes care of exporting all configuration and status information into a database. The IDO database is used by Icinga Web 2 as data backend.
If the masters are configured as a cluster, they negotiate which one is to be active. The active master connects and writes to the IDO, the others disconnect.
Postgres database schema can be found in: /usr/share/icinga2-ido-pgsql/schema/
Livestatus
The MK Livestatus project implements a query protocol (over a unix-socket) that lets users query their Icinga instance for status information. It can also be used to send commands.
The Livestatus component that is distributed as part of Icinga 2 is a re-implementation of the Livestatus protocol which is compatible with MK Livestatus.
Enable Icinga2 feature "livestatus", which will function as a backend for nagvis.
By default the Livestatus socket is available in /var/run/icinga2/cmd/livestatus.
API
- Listens on tcp/5665
- Serves as the endpoint for a director.
First configure API then enable it or Icinga2 will complain and fail.
Change hostname
Make sure you remember all the variables you touched during the change to have a way back.
Prepare DNS, IPs, interfaces for the new fqdn and test it. Maybe chose a non destructive way (side by side) and deconfigure the old configuration afterwards.
Create a new certificate on the master with the Icinga2 CA and distribute it to the target host.
1 cd /var/lib/icinga2/certs
2 ### CREATE CERTIFICATE SIGN REQUEST (CSR)
3 icinga2 pki new-cert \
4 --cn mon1.mon1.rockstable.it \
5 --key mon1.mon1.rockstable.it.key \
6 --csr mon1.mon1.rockstable.it.csr
7 ### SIGN CSR AND CREATE CERTIFICATE
8 icinga2 pki sign-csr \
9 --csr mon1.mon1.rockstable.it.csr \
10 --cert mon1.mon1.rockstable.it.crt
Change the zone configuration by adding a second endpoint with the new node name and assign the new endpoint to your "master" zone. If you are using the director, make sure the endpoint has address and port defined or you may end up triggering bug #7530, where director can't determine the version of the Icinga2-Core-API.
Change Icinga2 NodeName
/etc/icinga2/constants.conf
1 /**
2 * This file defines global constants which can be used in
3 * the other configuration files.
4 */
5
6 /* The directory which contains the plugins from the Monitoring Plugins project. */
7 const PluginDir = "/usr/lib/nagios/plugins"
8
9 /* The directory which contains the Manubulon plugins.
10 * Check the documentation, chapter "SNMP Manubulon Plugin Check Commands", for details.
11 */
12 const ManubulonPluginDir = "/usr/lib/nagios/plugins"
13
14 /* The directory which you use to store additional plugins
15 * which ITL provides user contributed command definitions for.
16 * Check the documentation, chapter "Plugins Contribution", for details.
17 */
18 const PluginContribDir = "/usr/lib/nagios/plugins"
19
20 /* Our local instance name. By default this is the server's hostname as returned by `hostname --fqdn`.
21 * This should be the common name from the API certificate.
22 */
23 //const NodeName = "mon1.1a.rockstable.it"
24 const NodeName = "mon1.mon1.rockstable.it"
25
26 /* Our local zone name. */
27 const ZoneName = "master"
28
29 /* Secret key for remote node tickets
30 * Generated with $(openssl rand -base64 30)
31 */
32 const TicketSalt = "TICKET_SALT"
Restart icinga2
1 systemctl restart icinga2
Custom Plugins
Some useful documentation for the Plugin-API:
https://icinga.com/docs/icinga2/latest/doc/05-service-monitoring/#plugin-api
https://icinga.com/docs/icinga2/latest/doc/05-service-monitoring/#performance-data-metrics
Custom Plugin for Website Load Time
This script is based on exchange.nagios.org Check-HTTP-Requisites, which was initially written by Syslab.com Gmbh. It is licensed under the The Zope Public License Ver.2.0 (ZPL-2.0).
This script follows the simple idea to measure the time to download all resources needed to display a website. It's python, which is available everywhere and it runs fine on Debian Buster.
The script got some improvements:
- Support for parallelity, http2 and some speed improvements
by usage of wget2.
- Support for Icinga2 metrics and therefore graphing.
- Bug fix for producing output.
/usr/lib/nagios/plugins/check_http_requisites.py
1 #!/usr/bin/python
2
3 ##############################################################################
4 #
5 # Copyright (c) 2007 Syslab.com.
6 #
7 # This software is subject to the provisions of the Zope Public License,
8 # Version 2.0 (ZPL). A copy of the ZPL should accompany this distribution.
9 # THIS SOFTWARE IS PROVIDED "AS IS" AND ANY AND ALL EXPRESS OR IMPLIED
10 # WARRANTIES ARE DISCLAIMED, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED
11 # WARRANTIES OF TITLE, MERCHANTABILITY, AGAINST INFRINGEMENT, AND FITNESS
12 # FOR A PARTICULAR PURPOSE
13 #
14 ##############################################################################
15
16 ### Changed: 2020-01-20 Tobias Stein
17 """%(program)s: retrieve a complete http request including all page assets.
18 Requires wget2 and time. Authentication is made zope specific.
19
20 usage: %(program)s [options]
21
22 Options:
23 -h/--help -- print this help text
24 -U/--url -- the URL you want to check
25 -w/--warn -- seconds threshold for the warning state.
26 -c/--crit -- seconds threshold for the critical state
27 -u/--user -- username if you want to check authenticated
28 -p/--passwd -- password
29 """
30
31 import getopt
32 import os
33 import shutil
34 import sys
35 from tempfile import *
36
37 import urllib
38
39 from StringIO import StringIO
40
41 STATE_OK = 0
42 STATE_WARN = 1
43 STATE_CRIT = 2
44 STATE_UNKNOWN = 3
45
46 __dbg__ = 0
47 __LOG__ = 0
48
49 ### Changed: 2020-01-20 Tobias Stein
50 #CMDLINE = "/usr/bin/time -f \"%e\" /usr/bin/wget "
51 CMDLINE = "/usr/bin/time -f \"%e\" /usr/bin/wget2 "
52 AUTHLINE = " --post-data '__ac_name=%s&__ac_password=%s' "
53 ### Changed: 2020-01-20 Tobias Stein
54 #PARAMS = " --page-requisites --delete-after -nd -nH -nv "
55 PARAMS = " --page-requisites --delete-after -nd -nH "
56 FILTER = " 2>&1 |tail -n3 |grep -E -e '^Downloaded:' -e '[0-9]+\.[0-9]+'"
57 TMPDIR = "/tmp"
58 LOGPATH = TMPDIR + "/nagios_check_http_requisites.log"
59 # No cookie support at the moment
60 COOKIELINE = " "
61 #--header='Accept-Encoding: gzip,deflate' "
62
63 def dbg(t,d):
64 if __dbg__ == 0:
65 return
66 print "DBG (%s): %s" % (t,d)
67
68 def log(txt):
69 if __LOG__:
70 fh = open(LOGPATH, 'w')
71 fh.write(txt)
72 fh.close()
73
74 def main():
75 out = StringIO()
76 out.write("START\n")
77 try:
78 opts, args = getopt.getopt(sys.argv[1:],
79 "h:U:w:c:u:p:",
80 ["help", "url=","warn=", "crit=", "user=", "passwd=" ]
81 )
82 except getopt.GetoptError, msg:
83 usage(sys.stderr, "\nIllegal argument")
84
85 url = ""
86 user = ""
87 passwd = ""
88
89 out.write("Options: %s\n" %str(opts))
90
91
92 for opt, arg in opts:
93 if opt in ("-h", "--help"):
94 usage(sys.stderr)
95 if opt in ("-U", "--url"):
96 url = arg
97 if opt in ("-u", "--user"):
98 user = arg
99 if opt in ("-p", "--passwd"):
100 passwd = arg
101 if opt in ("-w", "--warn"):
102 try:
103 warn = float(arg.strip())
104 except Exception,e:
105 usage(sys.stderr, e)
106 if opt in ("-c", "--crit"):
107 try:
108 crit = float(arg.strip())
109 except Exception,e:
110 usage(sys.stderr, e)
111
112 if url=='':
113 usage(sys.stderr, "\nError: No URL supplied")
114
115 if user.startswith('-'):
116 user = ''
117 if passwd.startswith('-'):
118 passwd = ''
119
120 result = getResult(url, warn, crit, user, passwd, out)
121
122 out.write('Result: %s\n' %str(result))
123 log(out.getvalue())
124
125 print result[1]
126 sys.exit(result[0])
127
128
129 def getResult(url, warn, crit, user, password, out):
130 try:
131 data = fetchCheck(url, user, password)
132 except Exception, fetch:
133 state = STATE_CRIT
134 info = "%s - Connection not possible" %state_text(2)
135 dbg("Error Fetching Data", fetch)
136 out.write('Connection Err: %s\n' %fetch)
137 return (state, info)
138 lines = data.split("\n")
139 if len(lines)<2:
140 return (STATE_UNKNOWN, "UNKNOWN - Unexpected data returned. Check Plugin")
141
142 #Geholt: 51 Dateien, 4,2M in 0,1s (40,1 MB/s)
143 elems = lines[-2].split(" ")
144 size = elems[3]
145 if size == 0:
146 return (STATE_CRITICAL, "CRITICAL - No data retrieved. Check URL")
147
148 num = elems[1]
149 if num == 0:
150 return (STATE_CRITICAL, "CRITICAL - No files retrieved. Check URL")
151 real_t = float(lines[-1].strip())
152 ### Changed: 2020-01-20 Tobias Stein
153 perfdata = "'time'=%ss;%s;%s 'size'=%sB 'number'=%s" \
154 % (real_t, warn, crit, size, num )
155
156 state = STATE_UNKNOWN
157 if real_t<warn:
158 state = STATE_OK
159 if real_t< crit and real_t>warn:
160 state = STATE_WARN
161 if real_t>crit:
162 state = STATE_CRIT
163 retdata = state_text(state) + \
164 " - Downloaded: %s bytes in %s files in %.2f seconds | %s" \
165 % (size, num, real_t, perfdata)
166 return (state, retdata)
167
168
169 def state_text(state):
170 if state==0:
171 return "OK"
172 if state==1:
173 return "WARNING"
174 if state==2:
175 return "CRITICAL"
176 if state==3:
177 return "UNKNOWN"
178
179
180 def usage(stream, msg=None):
181 if msg:
182 print >>stream, msg
183 print >>stream
184 program = os.path.basename(sys.argv[0])
185 print >>stream, __doc__ % {"program": program}
186 sys.exit(STATE_UNKNOWN)
187
188
189 def fetchCheck(url, user, password):
190 """ fetches a page including requisites using wget """
191 tmp = mkdtemp(dir=TMPDIR)
192
193 os.chdir(tmp)
194 if user != "":
195 AUTH = AUTHLINE % (user, password)
196 else:
197 AUTH =""
198 ### Changed: 2020-01-20 Tobias Stein
199 CMD = CMDLINE + AUTH + PARAMS + COOKIELINE + url + FILTER
200 #print CMD
201 std_out, std_in = os.popen4(CMD)
202 data = std_in.read()
203 try:
204 if os.access(tmp, os.F_OK):
205 RMCMD = "rm -f %s/*" % tmp
206 os.popen(RMCMD)
207 os.rmdir(tmp)
208 except Exception, e:
209 log("Could not remove tmp directory: " + tmp)
210 log("Error: %s" %e)
211
212 return data.strip()
213
214 if __name__ == "__main__":
215 main()
Icingaweb2
Apache2
Security
https://github.com/Icinga/icingaweb2/blob/master/doc/06-Security.md#filter-expressions
https://icinga.com/docs/icingaweb2/latest/modules/monitoring/doc/06-Security/
Icingaweb2 can manage users in groups and assigns roles to them. Roles gather a set of permissions and objects may be filtered by white- and blacklists. This system makes authorization very flexible and grants multitenancy.
Any filter expression that is allowed in the filtered view, is also an allowed filter expression. This means, that it is possible to define:
negations !,
wildcards *,
and even nested filter expressions ()
containing AND & and OR-Clauses |.
You may also use variables like $custom_var$.
Hints:
In my opinion there is a bug in parsing /etc/icingaweb2/roles.ini. Icingaweb2 can't cope well with ini-style comments (line starting with ^\s*;) and throws an error in the UI which says something like could not be converted to string. If you encounter this problem, just remove the comments.
- You need to logout and login to Icingaweb2 to apply a changed set of permissions.
Modules
Try using the modules from github, because:
- Debian Versions (even in Sid/Unstable) are outdated.
Modules in Debian have a bug in their installer /etc/icingweb2/enabledModules
- Not all modules are available in Debian package management.
List of modules
Modules Directory
Create a directory for the icingaweb2-modules from github, ensure correct filesystem-permissions. You'll need this directory e.g. for the module "director" and it's dependencies.
Add this module-directory (colon-separated) to /etc/icingaweb2/config.ini
Boxydash
If navigate to the menu-point "Boxy Dashboard" and you only see HTTP 404, the module may have a wrong URI-prefix called path_prefix in the code.
/etc/icingaweb2/modules/boxydash/config.ini
businessprocess
Create top-level views of your applications in a graphical editor. Rules express dependencies between existing hosts and services and let you alert on application level. Business processes are displayed in a tree or list overview.
Director
Welcome to the Icinga Director, the bleeding edge configuration tool for Icinga 2! Developed as an Icinga Web 2 module it aims to be your new favorite Icinga config deployment tool. Even if you prefer plain text files and manual configuration, chances are good that the Director will change your mind.
Dependencies
icingaweb2-module-incubator >= v0.5.0
icingaweb2-module-reactbundle >= v0.7.0
icingaweb2-module-ipl >= v0.3.0
Installation
Create a directory for the icingaweb2-modules from github, ensure correct filesystem-permissions and clone the git-repo tag.
Configuration
Prepare database:
- Create postgres-user "icingaweb2_director"
- Create postgres-database "icingaweb2_director" with owner "icingaweb2_director".
- Grant all privileges on "icingaweb2_director" to "icingaweb2_director".
Add a resource to /etc/icingaweb2/resources.ini.
Specify the resource "icingaweb2_director-pgsql-socket" as backend for icinga-director.
/etc/icingaweb2/modules/director/config.ini
Prepare database by importing the schema.
Background Daemon
The Icinga Director Background Daemon is available (and mandatory) since v1.7.0. It is responsible for various background tasks, including fully automated Import, Sync & Config Deployment Tasks.
Create a daemon-user
1 ### DEBIAN
2 adduser --system --ingroup icingaweb2,www-data \
3 --gecos "icingaweb2 director daemon user,," \
4 --home /var/lib/icingadirector \
5 --shell /bin/false \
6 icingadirector
7 ### RHEL
8 useradd -r -g icingaweb2 \
9 -d /var/lib/icingadirector \
10 -s /bin/false \
11 icingadirector
12 install -d -o icingadirector -g icingaweb2 -m 0750 /var/lib/icingadirector
A sample systemd-unit may be found with locate:
In my case: /opt/icingaweb2/modules/director/contrib/systemd/icinga-director.service
Copy systemd-unit to /etc/systemd/system and reload systemd.
The icinga-director daemon needs access to the files or whatever you want to import. Make sure to grant sufficient privileges for this daemon in order to be useful. In my case icinga-director also needs to be member of www-data, to read the json-files dropped by ansible to be read by icingaweb2. Icingaweb2 runs under php-fpm with uid/gid www-data and must also be able to read the files via group permissions. So icinga-director must be in the same group or you'll have to use extened POSIX ACLs.
Import and Sync
Importing data from external sources and deriving Icingae2 objects from this data may cause conflicts in between the director configuration and the running Icinga2 configuration. E.g. if you import the data from the Icinga2-Core via the API, you'll have the definitions twice - one from the core in files and one from the director deployed via the API.
To avoid these kind of issues i use the Fileshipper which is fed by "facts" from ansible in json-format. This is very powerful!1!!
EventDB (deprecated)
Current Version: 2.04 (March 28th 2012)
No need to install.
Fileshipper
Fileshipper doesn't allow to import files from various locations. Locations have to be defined and given a name first.
/etc/icingaweb2/modules/fileshipper/imports.ini
Graphite
https://icinga.com/docs/icinga2/latest/doc/09-object-types/#objecttype-graphitewriter
https://graphite.readthedocs.io/en/latest/config-webapp.html#apache-mod-wsgi
Follow the instructions of /usr/share/doc/graphite-web/README.Debian.
1 Debian graphite-web
2 -------------------
3
4 Graphite Webinterface should run as user `_graphite`.
5
6 For testing Graphite, you can start Graphite with the Django lightweight Web
7 server for development:
8
9 # su -s /bin/bash -c '/usr/bin/django-admin runserver --settings \
10 graphite.settings 0.0.0.0:8080' _graphite
11
12 The Django runserver is not recommend for production use!
13
14 You can choose any Django compatible Webserver. Upstream recommends Apache.
15
16 A WSGI file for Apache is included at /usr/share/graphite-web/graphite.wsgi.
17
18 The Apache configuration example is included into the graphite-web package:
19
20 /usr/share/graphite-web/apache2-graphite.conf
21
22
23 The Django `manage.py` is available via `graphite-manage` command.
24
25 The default configurations that is shipped with the graphite-web package is
26 using an sqlite database.
27
28 To create an sqlite database, schemas and default user run:
29
30 # su -s /bin/bash _graphite -c 'graphite-manage migrate --run-syncdb'
31
32 Change the owner of graphite.db / directory to _graphite:
33
34 # chown _graphite:_graphite /var/lib/graphite/graphite.db
35 # chmod 0600 /var/lib/graphite/graphite.db
36
37
38 Quick manual to use Graphite with Apache:
39 -----------------------------------------
40
41 # Install Apache module wsgi:
42 apt-get install libapache2-mod-wsgi-py3 apache2
43
44 # enable wsgi
45 a2enmod wsgi
46
47 # copy default Apache configuration:
48 cp /usr/share/graphite-web/apache2-graphite.conf \
49 /etc/apache2/sites-available/graphite-web.conf
50
51 # enable graphite-web
52 a2ensite graphite-web
53
54 # restart Apache
55 service apache2 restart
56
57
58 Graphite with MySQL:
59 -------------------
60
61 You can use Graphite also with MySQL. You need to edit the configuration in
62 /etc/graphite/local_setting.py and install the Python MySQLDB module
63 (python-mysqldb). After configuration and installing python-mysqldb use
64 graphite-manage syncdb to create the schema.
65
66 -- Jonas Genannt <jonas.genannt@capi2name.de> Mon, 03 Dec 2012 21:04:05 +0100
Graphite by default comes with a totally empty sqlite3 (0B) database
/var/lib/graphite/graphite.db
Initialize the database with the following command
1 su -s /bin/bash _graphite -c 'graphite-manage migrate --run-syncdb'
Simply bind graphite to 127.0.0.1 and set graphiteurl = http://localhost/
1 icinga2 feature enable graphite
Graphite troubleshooting
If you get
RuntimeError: populate() isn't reentrant
you may change
/usr/lib/python3/dist-packages/django/apps/registry.py line 84
To get to the root cause during initialization, because the other one masks it and you know nothing. Don't forget to change it back.
In my specific case it revealed that django can't write a log:
FileNotFoundError: [Errno 2] No such file or directory: '/usr/lib/python3/storage/log/webapp/info.log' Should actually be LOG_DIR.
This was because the /etc/graphite/local_settings.py was not imported.
In /usr/lib/python3/dist-packages/graphite/settings.py Debian appended /etc/graphite to sys.path
To overide the defaults, edit
/usr/share/graphite-web/graphite.wsgi
1 os.environ.setdefault('DJANGO_SETTINGS_MODULE', 'graphite.settings') # noqa
2 #os.environ.setdefault('GRAPHITE_SETTINGS_MODULE', 'graphite.local_settings') # noqa
3 ### ADDED THIS LINE SO "/etc/graphite/local_settings.py" IS READ
4 os.environ.setdefault('GRAPHITE_SETTINGS_MODULE', 'local_settings') # noqa
5
Now the config is being read. 
Adjust carbon retentions
By default carbon keeps a resolution of 60s for one day 60s:1d. This will probably not enough for your monitoring.
/etc/carbon/storage-schemas.conf
1 # Schema definitions for Whisper files. Entries are scanned in order,
2 # and first match wins. This file is scanned for changes every 60 seconds.
3 #
4 # Definition Syntax:
5 #
6 # [name]
7 # pattern = regex
8 # retentions = timePerPoint:timeToStore, timePerPoint:timeToStore, ...
9 #
10 # Remember: To support accurate aggregation from higher to lower resolution
11 # archives, the precision of a longer retention archive must be
12 # cleanly divisible by precision of next lower retention archive.
13 #
14 # Valid: 60s:7d,300s:30d (300/60 = 5)
15 # Invalid: 180s:7d,300s:30d (300/180 = 3.333)
16 #
17 # This retention is set at the time the first metric is sent.
18 # Changing this file will not affect already-created .wsp files.
19 # Use whisper-resize.py to change existing data files.
20
21 # Carbon's internal metrics. This entry should match what is specified in
22 # CARBON_METRIC_PREFIX and CARBON_METRIC_INTERVAL settings
23 [carbon]
24 pattern = ^carbon\.
25 retentions = 60:90d
26
27 [default]
28 pattern = .*
29 #retentions = 60s:1d
30 retentions = 60s:1w,5m:2w,15m:4w,30m:8w,1h:16w,2h:32w,4h:64w,8h:128w,1d:3y
Here's a command to resize all whisper-files with 3 processes in parallel.
1 ### DETERMINE RETENTION
2 grep retentions /etc/carbon/storage-schemas.conf \
3 |cut -d= -f2 |tr ',' ' '
4 ### STOP SERVICES
5 systemctl stop apache2.service
6 systemctl stop icinga2.service
7 systemctl stop carbon-cache.service
8 ### REWRITE TIME-SERIES
9 find /var/lib/graphite/whisper/ -type f -iname "*.wsp" \
10 |xargs -n 1 -P 3 -I{} \
11 sudo -u _graphite -- \
12 whisper-resize {} \
13 60s:1d 5m:1w 15m:2w 30m:4w 1h:8w 2h:16w 4h:32w 8h:64w 1d:128w
14 ### START SERVICES
15 systemctl start carbon-cache.service
16 systemctl start icinga2.service
17 systemctl start apache2.service
18 ### DELETE THE BACKUP FILES WHEN YOU ARE DONE
19 #find /var/lib/graphite/whisper/ -type f -iname "*.wsp.bak" -delete
20
Grafana
With the upgrade to 7.2.0 you need to have the Grafana image renderer installed.
grafana-image-renderer comes as a node.js application packed with chrome. It takes about 400MiB right after installation. In my opinion this is
SUPER-MASSIVE BLOAT WARE.
You can investigate the dependencies of chrome with
But hey let's take up some more space (121MiB) and satisfy the dependencies of chrome?!
To install the renderer.
Fine. At least it works.
Icinga PHP library (IPL)
Libraries useful for IcingaWeb2 modules.
Don't install the version from Debian Sid (v1.1.1) because it's too old for a current version icingaweb2-module-directory (depends on ipl >= v0.3.0).
Hints:
Do not use GIT master.
Installation
Incubator
This repository ships bleeding edge libraries useful for Icinga Web 2 modules. Please download the latest release and install it like any other module.
HINT: Do NOT install the GIT master, it will not work! Checking out a branch like stable/0.5.0 or a tag like v0.5.0 is fine.
Installation
NagVis
Configuration
RTFM!1!! /opt/icingaweb2/modules/nagvis/README.md
Enable icinga2 feature Livestatus
- Link css-file provided by icingaweb2-module-nagvis to nagvis.
Include some code right after <?php from README, to load-icingaweb2 modules in
/usr/share/nagvis/share/server/core/functions/index.php1 <?php 2 /** 3 * Icinga Web 2 integration 4 */ 5 use Icinga\Application\EmbeddedWeb; 6 7 require_once 'Icinga/Application/EmbeddedWeb.php'; 8 require_once EmbeddedWeb::start('/usr/share/icingaweb2', '/etc/icingaweb2') 9 ->getModuleManager() 10 ->getModule('nagvis') 11 ->getLibDir() . '/nagvis-includes/init.inc.php';
Configure Nagvis
/etc/nagvis/nagvis.ini.php1 [global] 2 // Use icingaweb2-modules for AAA 3 authmodule="CoreAuthModIcingaweb2" 4 authorisationmodule="CoreAuthorisationModIcingaweb2" 5 logonmodule="LogonIcingaweb2" 6 [paths] 7 // Change URI-prefix 8 htmlcgi="/icingaweb2" 9 10 [defaults] 11 // Use this defined backend as default 12 backend="icinga2_livestatus" 13 // Set custom CSS 14 stylesheet="icingaweb-nagvis-integration.css" 15 // Alter URLs 16 ; URL template for host object links 17 ;hosturl="[htmlcgi]/status.cgi?host=[host_name]" 18 hosturl="[htmlcgi]/monitoring/host/show?host=[host_name]" 19 ; URL template for hostgroup object links 20 ;hostgroupurl="[htmlcgi]/status.cgi?hostgroup=[hostgroup_name]" 21 hostgroupurl="[htmlcgi]/monitoring/list/hostgroups?hostgroup_name=[hostgroup_name]" 22 ; URL template for service object links 23 ;serviceurl="[htmlcgi]/extinfo.cgi?type=2&host=[host_name]&service=[service_description]" 24 serviceurl="[htmlcgi]/monitoring/service/show?host=[host_name]&service=[service_description]" 25 ; URL template for servicegroup object links 26 ;servicegroupurl="[htmlcgi]/status.cgi?servicegroup=[servicegroup_name]&style=detail" 27 servicegroupurl="[htmlcgi]/monitoring/list/servicegroups?servicegroup_name=[servicegroup_name]" 28 ; URL template for dynamic group object links (disabled by default) 29 ;dyngroupurl="" 30 ; URL template for aggregations object links (disabled by default) 31 ;aggrurl="" 32 ; URL template for nested map links 33 ;mapurl="[htmlbase]/index.php?mod=Map&act=view&show=[map_name]" 34 mapurl="[htmlcgi]/nagvis/show/map?map=[map_name]" 35 36 37 // Define Backend 38 [backend_icinga2_livestatus] 39 backendtype="mklivestatus" 40 socket="unix:/var/run/icinga2/cmd/livestatus"
Youtube Tutorial - Create your first graph
Reactbundle
This repository is an attempt to ship 3rd party libraries that might be useful for asynchronous PHP-based Icinga Web 2 modules. Please download the latest release and install it like any other module.
HINT: Do NOT install the GIT master, it will not work! Checking out a branch like stable/0.7.0 or a tag like v0.7.0 is fine.
Installation
PNP4Nagios (Deprecated)
Deprecation notice
This module has been renamed, please use the PNP module instead.
Toplevelview
Depends on package php-yaml. After installation php-fpm needs to be restarted.
x509 - Certificate Monitoring
Dependencies:
reactbundle (>= 0.4) (Icinga Web 2 module)
Icinga PHP Library (ipl) (>= 0.1) (Icinga Web 2 module)
- php-gmp
- OpenSSL
- MySQL or MariaDB
No Postgres
However we want this feature - so we'll install mariadb.