etbemon - User contributions [en-gb]

Monitors

2017-09-04T05:52:56Z

Etbe: How to write monitor scripts

=== How to Write a Monitor Script ===

A monitor script will take some command-line parameters specifying what to test, it will return 1 if there is a failure or 0 if everything is OK.

The first line of output is the "summary" that will be shown when you run "monshow" and further lines will be the details.

When things go OK the "summary" line should briefly report any useful information, such as an indication of when the test might fail in future. So a test for certificate expiry should report the next time a certificate will expire even when it's far enough in the future to not be an error. A test for disk space should report the amount of free space even when that's OK.

A monitor script that tests network things should be in the mon.d directory. A script that does local tests (which doesn't mean network tests to localhost) should be in mon-local.d. SE Linux and other security systems will give different permissions to scripts in mon-local.d.

=== Local Tests ===

==== Filesystems and Storage ====

[[btrfs.monitor]] - BTRFS filesystems

[[softraid.monitor]] - Linux Software RAID

[[zfs.monitor]] - ZFS filesystems

[[freespace.monitor]] - free space

==== Others ====

[[linux-temp.monitor]] - Linux system Temperature

[[loadavg.monitor]] - Unix load average

[[ps.monitor]] - Unix processes

=== Remote Tests ===

I've added stub links for all the monitor scripts currently included. Some of them may not be suitable going forward, EG I really doubt that we need 2 ping tests, 3 smtp tests, and 2 imap tests. Also some of them probably should be in the local section (maybe dialin). Finally some of them may be unsupportable, if a test doesn't appear to work, I can't test it, and I can't find someone to take ownership of it then I will consider removing it.

==== Core Network Tests ====

[[fping.monitor]]

[[traceroute.monitor]]

[[dns.monitor]]

[[dns-query.monitor]]

[[ntpdate.monitor]]

[[ping.monitor]]

[[up_rtt.monitor]]

[[tcp.monitor]]

[[telnet.monitor]]

[[trace.monitor]]

==== Mail Servers ====

[[imap.monitor]]

[[imapnew.monitor]]

[[pop3.monitor]]

[[smtp.monitor]]

[[smtpswaks.monitor]]

[[smtp3.monitor]]

==== Monitoring Mon ====

[[mon.monitor]]

[[seq.monitor]]

==== Other Tests ====

[[asyncreboot.monitor]]

[[cpqhealth.monitor]]

[[dialin.monitor]]

[[file_change.monitor]]

[[foundry-chassis.monitor]]

[[ftp.monitor]]

[[hpnp.monitor]]

[[http.monitor]]

[[http_tppnp.monitor]]

[[ldap.monitor]]

[[local-syslog.monitor]]

[[lpd.monitor]]

[[msql-mysql.monitor]]

[[na_quota.monitor]]

[[netappfree.monitor]]

[[nntp.monitor]]

[[phttp.monitor]]

[[process.monitor]]

[[radius.monitor]]

[[rd.monitor]]

[[reboot.monitor]]

[[remote.monitor]]

[[rpc.monitor]]

[[silkworm.monitor]]

[[snmpdiskspace.monitor]]

[[snmpvar.monitor]]

[[sslcert.monitor]]

[[xedia-ipsec-tunnel.monitor]]

Redundant Mon Servers

2017-08-28T00:46:36Z

Etbe: Created page with "For reliable operation you must be able to handle the failure of any part of the system, and that includes a Mon server. The best way to deal with this is to have two Mon serv..."

For reliable operation you must be able to handle the failure of any part of the system, and that includes a Mon server. The best way to deal with this is to have two Mon servers in different availability zones monitoring each other. Then if one Mon server dies or if it sends an alert that isn't received (maybe the path from the Mon server to the alerting system is blocked) then the other Mon server will notify you.

Here is a section of the configuration on servera, it has a service named monsrvbmon that watches serverb and reports monitoring errors for every service apart from monsrvamon. This prevents recursive error conditions.

service monserverb
description monitor serverb mon
interval 5m
monitor remote.monitor --bigsummary --failure_duration=1800 --exclude serverb:monservera serverb ;;
period
numalerts 1
alert mailxmpp.alert -x user@servera.example.com -m user@example.com
upalert mailxmpp.alert -x user@servera.example.com -m user@example.com

Here is a section from the configuration on serverb, exactly the same but with names reversed. You can copy the configuration to your own machines with s/servera/whatever/ s/serverb/whatever/ .

service monservera
description monitor servera mon
interval 5m
monitor remote.monitor --bigsummary --failure_duration=1800 --exclude servera:monserverb servera ;;
period
numalerts 1
alert mailxmpp.alert -x user@serverb.example.com -m user@example.com
upalert mailxmpp.alert -x user@serverb.example.com -m user@example.com

For the mailxmpp.alert I have different destination Jabber servers. Jabber servers can fail too so we need two Jabber servers for reliability. While services like Slack are generally quite reliable and popular for network monitoring I prefer to use my own Jabber servers to eliminate the possibility of network issues between my servers and Slack servers. But having both systems use Slack should be quite reliable.

I use failure duration of 1800 because in my case I don't expect to have a Mon server fail to give alerts and in the normal case I don't need an immediate second alert. If I had a very low failure duration then it would give a second alert before I could login to fix things. But if a fast reaction in all adverse cases is important then you want to just deal with double alerts.

Now this is just a small part of the monitoring for the other servers. Each server needs to monitor the other for pingability, ssh, etc. Ideally you will notice problems in a failing Mon server before Mon itself becomes unavailable.

This configuration relies on the --exclude option to remote.monitor that only worked correctly as of version 1.3.1. This will not work with earlier versions of Mon.

Main Page

2017-08-28T00:24:29Z

Etbe: added redundant mon servers

== Configuration ==

[[Monitors]] - documentation for all the monitor scripts.

[[Alerts]] - documentation for all the alert scripts.

[[Redundant Mon Servers]] - how to setup two mon servers to watch each other.

Alerts

2017-08-27T09:09:04Z

Etbe: Created page with "file.alert irc.alert mail.alert mailxmpp.alert netpage.alert qpage.alert snpp.alert trap.alert == Test/Development Alerts == alert.tem..."

[[file.alert]]

[[irc.alert]]

[[mail.alert]]

[[mailxmpp.alert]]

[[netpage.alert]]

[[qpage.alert]]

[[snpp.alert]]

[[trap.alert]]

== Test/Development Alerts ==

alert.template - template to use for writing your own alerts. Included in docs not in the alerts directory.

test.alert - send alerts to /var/log/mon/test.alert.log.

Main Page

2017-08-27T08:59:34Z

Etbe: add alerts page

== Configuration ==

[[Monitors]]

[[Alerts]]

Monitors

2017-08-27T08:58:34Z

Etbe: Created page with "=== Local Tests === ==== Filesystems and Storage ==== btrfs.monitor - BTRFS filesystems softraid.monitor - Linux Software RAID zfs.monitor - ZFS filesystems [..."

=== Local Tests ===

==== Filesystems and Storage ====

[[btrfs.monitor]] - BTRFS filesystems

[[softraid.monitor]] - Linux Software RAID

[[zfs.monitor]] - ZFS filesystems

[[freespace.monitor]] - free space

==== Others ====

[[linux-temp.monitor]] - Linux system Temperature

[[loadavg.monitor]] - Unix load average

[[ps.monitor]] - Unix processes

=== Remote Tests ===

I've added stub links for all the monitor scripts currently included. Some of them may not be suitable going forward, EG I really doubt that we need 2 ping tests, 3 smtp tests, and 2 imap tests. Also some of them probably should be in the local section (maybe dialin). Finally some of them may be unsupportable, if a test doesn't appear to work, I can't test it, and I can't find someone to take ownership of it then I will consider removing it.

==== Core Network Tests ====

[[fping.monitor]]

[[traceroute.monitor]]

[[dns.monitor]]

[[dns-query.monitor]]

[[ntpdate.monitor]]

[[ping.monitor]]

[[up_rtt.monitor]]

[[tcp.monitor]]

[[telnet.monitor]]

[[trace.monitor]]

==== Mail Servers ====

[[imap.monitor]]

[[imapnew.monitor]]

[[pop3.monitor]]

[[smtp.monitor]]

[[smtpswaks.monitor]]

[[smtp3.monitor]]

==== Monitoring Mon ====

[[mon.monitor]]

[[seq.monitor]]

==== Other Tests ====

[[asyncreboot.monitor]]

[[cpqhealth.monitor]]

[[dialin.monitor]]

[[file_change.monitor]]

[[foundry-chassis.monitor]]

[[ftp.monitor]]

[[hpnp.monitor]]

[[http.monitor]]

[[http_tppnp.monitor]]

[[ldap.monitor]]

[[local-syslog.monitor]]

[[lpd.monitor]]

[[msql-mysql.monitor]]

[[na_quota.monitor]]

[[netappfree.monitor]]

[[nntp.monitor]]

[[phttp.monitor]]

[[process.monitor]]

[[radius.monitor]]

[[rd.monitor]]

[[reboot.monitor]]

[[remote.monitor]]

[[rpc.monitor]]

[[silkworm.monitor]]

[[snmpdiskspace.monitor]]

[[snmpvar.monitor]]

[[sslcert.monitor]]

[[xedia-ipsec-tunnel.monitor]]

Main Page

2017-08-27T08:58:25Z

Etbe: Move monitors to a new page

== Configuration ==

[[Monitors]]

Freespace.monitor

2017-08-27T08:54:12Z

Etbe: more detail and an example

The freespace monitor takes a list of filesystems and monitors the freespace, for each filesystem you have FILESYSTEM:FREESPACE:FREEINODES. Here is an example:

freespace.monitor /:10%:10000 /home:20G:10%

The freespace can be a percentage or the amount of free space required in KiB (or MiB/GiB/TiB if appended with M/G/T). The free inodes can be a percentage or the number of Inodes, it defaults to 10% of Inodes if the filesystem reports free Inodes.

Freespace.monitor

2017-08-27T04:18:31Z

Etbe: Created page with "The freespace monitor takes a list of filesystems and monitors the freespace, for each filesystem you have FILESYSTEM:FREESPACE:FREEINODES"

The freespace monitor takes a list of filesystems and monitors the freespace, for each filesystem you have FILESYSTEM:FREESPACE:FREEINODES

Zfs.monitor

2017-08-27T03:46:43Z

Etbe: Created page with "The ZFS monitor takes a list of mount points for ZFS filesystems on the command-line and checks how many errors there have been via the "zpool status" command. If there have b..."

The ZFS monitor takes a list of mount points for ZFS filesystems on the command-line and checks how many errors there have been via the "zpool status" command. If there have been errors then it looks for a file named /var/lib/mon/zfs-$MOUNTPOINT.count and gives an alert if that file doesn't exist or if the number in that file is less than the number of errors. This is so that if a filesystem has a few errors and you think that's OK you can count that as being OK. If you are monitoring the root filesystem then /var/lib/mon/zfs-root.count will have the count.

Btrfs.monitor

2017-08-27T03:45:55Z

Etbe: Created page with "The BTRFS monitor takes a list of mount points for BTRFS filesystems on the command-line and checks how many errors there have been via the "btrfs device stats" command. If th..."

The BTRFS monitor takes a list of mount points for BTRFS filesystems on the command-line and checks how many errors there have been via the "btrfs device stats" command. If there have been errors then it looks for a file named /var/lib/mon/btrfs-$MOUNTPOINT.count and gives an alert if that file doesn't exist or if the number in that file is less than the number of errors. This is so that if a filesystem has a few errors and you think that's OK you can count that as being OK. If you are monitoring the root filesystem then /var/lib/mon/btrfs-root.count will have the count.

Softraid.monitor

2017-08-27T02:52:12Z

Etbe: Created page with "By default this monitor will check all Linux software RAIDs and alert on missing/failed block devices. It used to be named softraid2 but has been renamed in etbemon version 1...."

By default this monitor will check all Linux software RAIDs and alert on missing/failed block devices. It used to be named softraid2 but has been renamed in etbemon version 1.3.2. It takes command-line parameters of the RAID devices that must be present and will alert if they aren't there.

Main Page

2017-08-27T02:36:42Z

Etbe: adding sections to remote

<strong>MediaWiki has been installed.</strong>

Consult the [//meta.wikimedia.org/wiki/Help:Contents User's Guide] for information on using the wiki software.

== Configuration ==

=== Local Tests ===

==== Filesystems and Storage ====

[[btrfs.monitor]] - BTRFS filesystems

[[softraid.monitor]] - Linux Software RAID

[[zfs.monitor]] - ZFS filesystems

[[freespace.monitor]] - free space

==== Others ====

[[linux-temp.monitor]] - Linux system Temperature

[[loadavg.monitor]] - Unix load average

[[ps.monitor]] - Unix processes

=== Remote Tests ===

I've added stub links for all the monitor scripts currently included. Some of them may not be suitable going forward, EG I really doubt that we need 2 ping tests, 3 smtp tests, and 2 imap tests. Also some of them probably should be in the local section (maybe dialin). Finally some of them may be unsupportable, if a test doesn't appear to work, I can't test it, and I can't find someone to take ownership of it then I will consider removing it.

==== Core Network Tests ====

[[fping.monitor]]

[[traceroute.monitor]]

[[dns.monitor]]

[[dns-query.monitor]]

[[ntpdate.monitor]]

[[ping.monitor]]

[[up_rtt.monitor]]

[[tcp.monitor]]

[[telnet.monitor]]

[[trace.monitor]]

==== Mail Servers ====

[[imap.monitor]]

[[imapnew.monitor]]

[[pop3.monitor]]

[[smtp.monitor]]

[[smtpswaks.monitor]]

[[smtp3.monitor]]

==== Monitoring Mon ====

[[mon.monitor]]

[[seq.monitor]]

==== Other Tests ====

[[asyncreboot.monitor]]

[[cpqhealth.monitor]]

[[dialin.monitor]]

[[file_change.monitor]]

[[foundry-chassis.monitor]]

[[ftp.monitor]]

[[hpnp.monitor]]

[[http.monitor]]

[[http_tppnp.monitor]]

[[ldap.monitor]]

[[local-syslog.monitor]]

[[lpd.monitor]]

[[msql-mysql.monitor]]

[[na_quota.monitor]]

[[netappfree.monitor]]

[[nntp.monitor]]

[[phttp.monitor]]

[[process.monitor]]

[[radius.monitor]]

[[rd.monitor]]

[[reboot.monitor]]

[[remote.monitor]]

[[rpc.monitor]]

[[silkworm.monitor]]

[[snmpdiskspace.monitor]]

[[snmpvar.monitor]]

[[sslcert.monitor]]

[[xedia-ipsec-tunnel.monitor]]

Main Page

2017-08-27T02:29:29Z

Etbe: /* Remote Tests */

<strong>MediaWiki has been installed.</strong>

Consult the [//meta.wikimedia.org/wiki/Help:Contents User's Guide] for information on using the wiki software.

== Configuration ==

=== Local Tests ===

==== Filesystems and Storage ====

[[btrfs.monitor]] - BTRFS filesystems

[[softraid.monitor]] - Linux Software RAID

[[zfs.monitor]] - ZFS filesystems

[[freespace.monitor]] - free space

==== Others ====

[[linux-temp.monitor]] - Linux system Temperature

[[loadavg.monitor]] - Unix load average

[[ps.monitor]] - Unix processes

=== Remote Tests ===

I've added stub links for all the monitor scripts currently included. Some of them may not be suitable going forward, EG I really doubt that we need 2 ping tests, 3 smtp tests, and 2 imap tests. Also some of them probably should be in the local section (maybe dialin). Finally some of them may be unsupportable, if a test doesn't appear to work, I can't test it, and I can't find someone to take ownership of it then I will consider removing it.

[[asyncreboot.monitor]]

[[cpqhealth.monitor]]

[[dialin.monitor]]

[[dns.monitor]]

[[dns-query.monitor]]

[[file_change.monitor]]

[[foundry-chassis.monitor]]

[[fping.monitor]]

[[ftp.monitor]]

[[hpnp.monitor]]

[[http.monitor]]

[[http_tppnp.monitor]]

[[imap.monitor]]

[[imapnew.monitor]]

[[ldap.monitor]]

[[local-syslog.monitor]]

[[lpd.monitor]]

[[Makefile]]

[[mon.monitor]]

[[msql-mysql.monitor]]

[[na_quota.monitor]]

[[netappfree.monitor]]

[[nntp.monitor]]

[[ntpdate.monitor]]

[[phttp.monitor]]

[[ping.monitor]]

[[pop3.monitor]]

[[process.monitor]]

[[radius.monitor]]

[[rd.monitor]]

[[reboot.monitor]]

[[remote.monitor]]

[[rpc.monitor]]

[[rpc.monitor.c]]

[[seq.monitor]]

[[silkworm.monitor]]

[[smtp3.monitor]]

[[smtp.monitor]]

[[smtpswaks.monitor]]

[[snmpdiskspace.monitor]]

[[snmpvar.monitor]]

[[sslcert.monitor]]

[[tcp.monitor]]

[[telnet.monitor]]

[[trace.monitor]]

[[traceroute.monitor]]

[[up_rtt.monitor]]

[[xedia-ipsec-tunnel.monitor]]

Main Page

2017-08-27T02:23:02Z

Etbe: Add new sections