Thresholds and alerts

Every check has default warning and error levels that suit most hosts. Change them for all hosts in _defaults.conf or for one host in its host file, then decide who gets the email and text alerts.

Plugin settings

Set a plugin's setting as <category>_<name>_<SETTING>=value in etc/hosts/_defaults.conf for every host, or in etc/hosts/<host>.conf for one host. Every plugin also takes TIMEOUT.

$ vi ~/drsm/etc/hosts/db1.conf

storage_disk_WARN=80
storage_disk_CRIT=90
storage_disk_SKIP="/tmp /usr1"
core_memory_MIN_AVAIL_MB=1024
core_load_WARN=16
core_users_ERROR_USERS="root"

Plugin	Setting	Default	Meaning
`core/cpu`	`WARN`, `CRIT`	85, 95	Total CPU use in percent
	`NUMPROCS`	100	Processes listed in the report
	`SAMPLE_SECS`	2	Seconds between the two `/proc/stat` samples
	`TEMPS`	yes	Report CPU temperatures when `lm_sensors` is installed
`core/load`	`WARN`, `CRIT`	2x, 4x CPUs	15 minute load average; empty scales with the CPU count
`core/memory`	`MIN_AVAIL_MB`	256	Error at or below this much available memory
	`SWAP_WARN_MB`	1024	Warning at or above this much swap in use
	`NUMPROCS`	100	Processes listed in the report
`core/users`	`ERROR_USERS`, `WARN_USERS`	none	User names that raise an error or warning when logged in
`storage/disk`	`WARN`, `CRIT`	90, 99	Space and inode use in percent
	`SKIP`	none	Mount points to skip
	`FSTYPES`	xfs, ext2/3/4, btrfs, zfs, f2fs, jfs, reiserfs	Local file system types to check
`storage/raid`	`MEGACLI`	`/opt/MegaRAID/MegaCli/MegaCli64`	LSI MegaRAID utility, needs sudo
`network/nics`	`NICS`	all active	Interfaces to check
	`MAX_DROPPED`, `MAX_COLLISIONS`, `MAX_ERRORS`	1000, 100, 100	Alert levels during the sample window
	`SAMPLE_SECS`	5	Sample window in seconds
`network/ping`	`COUNT`, `DEADLINE`	3, 30	Packets and deadline, any reply passes
`network/http`, `https`, `ftp`	`URL`	`<scheme>://<host>`	URL to test
	`CONNECT_TIMEOUT`, `MAX_TIME`, `RETRIES`	15, 30, 2	curl limits

A Web server that answers with an error page or a redirect still counts as up. An invalid HTTPS certificate counts as a failure.

Profiles for system_report.sh

Sites still on system_report.sh set levels in the $CONFIGdir/<host>_profile.sh script generated for each host. Edit the arguments inside the double quotes:

Line	Arguments
`disk_checks.sh '/tmp /usr1' 80 90`	Mounts to skip (or `NONE`), warning %, error %
`cpu_checks.sh 75 85 200`	Warning %, error %, processes listed
`load_checks.sh 100 250`	Warning and error 15 minute load
`memory_checks.sh 128 4096 300`	Error at MB available, warning at MB swap used, processes listed
`network_checks.sh '' 5000 500 500 10`	NICs (`''` for all), dropped, collisions, errors, sample seconds
`user_checks.sh 'usr1 usr2' 'usr4'`	Users that raise an error, users that raise a warning

The scripts are copied to $REMOTEdir on each host, drsm/health_check_scripts under the remote user's home by default. Delete a profile to have a new one generated. utils/migrate_systems_dat.sh carries these levels over to host files.

Email and text alerts

Alerts go to root on the monitor host until you add addresses. An alert for the same problem goes out at most once every 4 hours, so a host that stays down does not flood your inbox. Each connectivity alert includes the host's IMPACT line.

$ source ~/.drsm.sh
$ vi $DRSMHOME/etc/alert.email.list

export EMAILlist="root,ops@example.com"
export TEXTlist="5555555555@vtext.com"

$ $DRSMHOME/bin/test_alert.sh

For texts, use the carrier's email to text address, such as number@vtext.com (Verizon) or number@tmomail.net (T-Mobile).

Several carriers have shut down their email to text gateways, including AT&T (2025), Sprint, Nextel and Cingular. Check with your carrier before relying on text alerts.

DRSM uses the first mail, mailx or s-nail command it finds. Set MAILcmd in ~/.drsm.sh to use another.

Relay host

If your network has a mail relay, point postfix at it:

# vi /etc/postfix/main.cf

relayhost = 192.168.1.12

# systemctl restart postfix

Smart host

Without a relay, or on a cloud server where port 25 is blocked, send through an external mail account on port 587:

# vi /etc/postfix/main.cf

smtp_tls_security_level = encrypt
smtp_sasl_auth_enable = yes
smtp_sasl_security_options = noanonymous
smtp_sasl_password_maps = hash:/etc/postfix/relay_passwd
relayhost = [smtp.gmail.com]:587
smtp_generic_maps = hash:/etc/postfix/generic

# vi /etc/postfix/relay_passwd
[smtp.gmail.com]:587 USERNAME:PASSWORD

# chmod 600 /etc/postfix/relay_passwd
# postmap /etc/postfix/relay_passwd
# rm /etc/postfix/relay_passwd      # postmap made relay_passwd.db

# vi /etc/postfix/generic
root@localdomain.local  username@example.com
@localdomain.local      username@example.com

# postmap /etc/postfix/generic
# systemctl restart postfix
# tail -f /var/log/maillog

The generic map gives outgoing mail a valid From address. On Debian and Ubuntu the log is /var/log/mail.log.