Plugins

Every DRSM check is a plugin: a folder with a small manifest and one BASH script. bin/drsm_run.sh finds the plugins each host needs, runs them, and writes the reports the dashboard and alerts use. Adding a check means adding a folder, not editing a core script.

Plugins included

These are the checks from bin/*_checks.sh and system_check.sh with the same messages and default levels. Their settings are listed in Thresholds and alerts.

Running checks with drsm_run.sh

$ ~/drsm/bin/drsm_run.sh -l             # list hosts and the plugins that would run
$ ~/drsm/bin/drsm_run.sh -n             # all hosts, no alerts
$ ~/drsm/bin/drsm_run.sh -n -H cloud1   # one host
$ ~/drsm/bin/drsm_run.sh -n -g dev      # the dev group
$ ~/drsm/bin/drsm_run.sh -m probes      # connectivity probes only

Exit code: 0 all good, 1 warnings, 2 errors.

Host files

One file per host in etc/hosts, named <host>.conf. The files are plain KEY=value lines, read but never run, so a typo cannot execute anything. Values with spaces go in double quotes.

HOST=cloud1
DESCRIPTION="AWS web server"
IMPACT="Public website is down"
GROUP=production
PLATFORM=linux
COLLECTOR=ssh
PROBES="network/ping network/https"
PLUGINS="storage/disk storage/raid core/cpu core/load core/memory network/nics core/users"

# Plugin settings: <category>_<name>_<SETTING>=value
storage_disk_WARN=85
storage_disk_SKIP="/mnt/scratch"
network_https_URL=https://www.example.com/
core_cpu_TIMEOUT=60

_defaults.conf is read before every host file, so put the common PLUGINS list and site wide settings there. Files starting with _ are not hosts. If network/ping fails the host is reported down and its remote plugins are skipped.

To convert systems.dat, dev_systems.dat and the custom levels in <host>_profile.sh scripts, run utils/migrate_systems_dat.sh.

Writing a plugin

$ ~/drsm/utils/new_plugin.sh local/myapp            # runs on the host
$ ~/drsm/utils/new_plugin.sh local/myapp_url probe  # runs from the monitor against the host

This creates the folder, a commented plugin.conf, and a script you can run by hand right away:

$ DRSM_WARN=50 bash ~/drsm/plugins/local/myapp/check.sh
$ DRSM_TARGET=cloud1 bash ~/drsm/plugins/local/myapp_url/probe.sh

Then add local/myapp to PLUGINS (or PROBES) in a host file. Site plugins live in plugins/local, which updates never overwrite.

Manifest

NAME=myapp
CATEGORY=local
DESCRIPTION="Order queue is draining"
PLATFORMS="linux"      # linux, windows, or any
MODE=remote            # remote runs check.sh on the host, probe runs probe.sh on the monitor
TIMEOUT=30             # seconds, enforced by the runner
REQUIRES="curl jq"     # commands that must exist, otherwise the plugin is UNKNOWN
NEEDS_SUDO=no          # informational: no, yes, or optional

# Every other key is a setting with its default
WARN=100
CRIT=500
QUEUE_DIR=/var/spool/myapp

Each setting reaches the script as DRSM_<KEY>. A host overrides it with <category>_<name>_<KEY>, for example local_myapp_WARN=200, and can set <category>_<name>_TIMEOUT.

Script

#!/bin/bash
source "${DRSM_LIB:-$(dirname "$0")/../../../lib}/drsm_plugin.sh"

WARN=$(drsm_setting WARN 100)
CRIT=$(drsm_setting CRIT 500)
QUEUE_DIR=$(drsm_setting QUEUE_DIR /var/spool/myapp)

count=$(find "${QUEUE_DIR}" -type f | wc -l)
if [ "${count}" -ge "${CRIT}" ]; then
    drsm_crit "${HOST} myapp queue has ${count} files"
elif [ "${count}" -ge "${WARN}" ]; then
    drsm_warn "${HOST} myapp queue has ${count} files"
else
    drsm_ok "${HOST} myapp queue checks good at ${count} files"
fi
drsm_metric myapp_queue_files "${count}"

drsm_exit

States and exit codes

Exit codes follow the Nagios plugin convention:

0 OK 1 WARNING 2 ERROR 3 UNKNOWN

• An ERROR - or WARNING - line counts even if the script exits 0.
• A plugin that runs past its TIMEOUT is stopped and reported as a WARNING.
• A missing REQUIRES command makes the plugin UNKNOWN, which shows in the report but does not alert.
• A plugin that prints nothing is reported as an ERROR.

How a run works

For each host the runner builds one job script with a line per plugin, its settings and its timeout. Probe jobs run on the monitor host. Remote jobs go through the host's collector:

• local runs the job in place from $DRSMHOME.
• ssh-linux copies lib/drsm_plugin.sh and the host's plugins to ~/drsm_agent as one tar stream, skipped when the checksum has not changed, then runs the whole job in one SSH session. Set DRSM_REMOTE_BASE for another directory.

Each plugin's output is a section of the host report:

=== storage/disk [WARNING] 0s ===
Disk usage checks for cloud1
WARNING - cloud1:/var is at 92% capacity.
METRIC disk_used_pct{mount="/var"}=92

Groups other than production add _<group> to the group report names. Alerts use bin/text_email_alert.sh with the same subjects and 4 hour throttle as before.

Files

bin/drsm_run.sh               plugin runner
lib/drsm_plugin.sh            helpers every plugin sources
lib/drsm_conf.sh              KEY=value reader for manifests and host files
lib/drsm_job.sh               runs plugins on the target, with timeouts
collectors/local.sh           runs plugins on the monitor host itself
collectors/ssh-linux.sh       copies and runs plugins on Linux hosts over SSH
plugins/<category>/<name>/    plugin.conf plus check.sh or probe.sh
plugins/local/                your own plugins, never overwritten by updates
etc/hosts/_defaults.conf      settings shared by all hosts
etc/hosts/<host>.conf         one file per host
utils/new_plugin.sh           creates a plugin from a template
utils/migrate_systems_dat.sh  converts systems.dat and profiles to host files

Roadmap

Later phases of the plugin design add more checks (SMART, mounts, backup age, TCP ports, TLS expiry, DNS, time sync, failed services), generic database, container, service and process plugins with a custom/command catch all, a Windows collector over OpenSSH and PowerShell, and configurable notifiers (email, SMS, Slack, ntfy, webhook) with alert on change.