Deployment setups

DRSM needs one monitor host and somewhere to serve the dashboard. They can be the same machine, a cluster, or a workstation paired with a PHP Web hosting account. These are the setups in use today.

Production: 3 node cluster

For production, DRSM runs on a 3 node cluster. Each node has the same DRSM install and the same sysadmin account and SSH key, so any node can run the checks. Only the node that holds the cluster's floating IP runs them, so every host is checked once and every alert is sent once.

1. Install DRSM on every node

   Create the sysadmin account with the same UID and GID (3500) on all three nodes, follow Install on each, and copy the same ~/.ssh/id_ed25519 key pair to each so the monitored hosts accept every node. Keep the three installs on the same DRSM release.

2. Share the Web directory

   Put $WWWdir (with reports and config) on the cluster's shared storage and serve it with Apache on the cluster address. The reports then follow the active node, and the dashboard shows the same history after a failover. Keep etc/hosts the same on every node, or on shared storage too.

3. Run the cron jobs on the active node

   crontabs/cluster_crontab_example.txt is written for /etc/cron.d, with the user name on each line. Have the cluster manager install it with the cluster IP, or start it from the same resource group, so it runs on one node at a time:

   # vi /etc/cron.d/drsm
   
   */30 * * * * sysadmin bash -c '/home/sysadmin/drsm/bin/drsm_run.sh -m probes &> /dev/null'
   15,45 * * * * sysadmin bash -c '/home/sysadmin/drsm/bin/drsm_run.sh &> /dev/null'
   15 13 * * mon-fri sysadmin bash -c '/home/sysadmin/drsm/bin/drsm_run.sh -e &> /dev/null'
   59 23 * * * sysadmin bash -c '/home/sysadmin/drsm/bin/archive.sh &> /dev/null'
   00 * * * *  sysadmin bash -c '/home/sysadmin/drsm/bin/purge.sh &> /dev/null'

4. Monitor the cluster itself

   Give the cluster address a host file with probes only, and give each node its own host file for health checks. If a node goes down you see it on the dashboard while the service address stays green.

   $ cat ~/drsm/etc/hosts/cluster.conf
   DESCRIPTION="Production cluster address"
   IMPACT="Production services are down"
   COLLECTOR=none
   PROBES="network/ping network/https"
   
   $ cat ~/drsm/etc/hosts/node1.conf
   DESCRIPTION="Cluster node 1"
   IMPACT="Cluster has one less node"

   With the older systems.dat, set is_cluster_ip to yes on the cluster address row.

Workstation with a hosted dashboard (OSA and OCA)

For OSA and OCA, DRSM runs on a workstation and the dashboard lives on the datareel.com website, a HostGator shared hosting account. The hosting account only needs PHP: it never runs checks and needs no SSH access to the servers. The workstation runs the checks and copies the report files up after each run.

1. Install DRSM on the workstation

   Use the development workstation config, which runs from your own account:

   $ cd ~/git/dr_system_monitor
   $ cp -p etc/drsm_dev.sh ~/.drsm.sh
   $ ./utils/install.sh

   Add a host file for each OSA and OCA server. Most of them are cloud servers, so read Cloud servers for SSH users and ping.

2. Build the dashboard package

   utils/make_web_demo.sh builds a copy of the dashboard that finds its own files, so it runs from any folder. -p is the URL path it is served from and -t the name in its top bar. One package per site:

   $ ~/drsm/utils/make_web_demo.sh -p /drsm/osa -t "OSA System Monitor" -o /tmp/drsm_osa
   $ ~/drsm/utils/make_web_demo.sh -p /drsm/oca -t "OCA System Monitor" -o /tmp/drsm_oca

   The paths above are examples; use the ones you want on the site.

3. Upload and extract it

   Upload drsm_dashboard.zip to the parent of the URL path, public_html/drsm for /drsm/osa, and extract it there. In cPanel: File Manager, open public_html, create the drsm folder, open it, Upload the zip, then right click it and Extract. The package unpacks as dashboard/, so rename that folder to osa.

4. Password protect it

   The reports list processes and logged in users. Build the packages with -a and add accounts to turn on the dashboard login, so your office network gets straight in and everyone else signs in. Or in cPanel open Directory Privacy, select the osa folder, turn on password protection and add a user. Do the same for oca.

5. Copy the reports up after each run

   The dashboard reads the same files as on the workstation, under its reports folder. HostGator's SSH listens on port 2222; add the account to ~/.ssh/config and copy your key with ssh-copy-id first. Then copy the reports with rsync after each run, from a small script that cron calls:

   $ vi ~/.ssh/config
   
   Host datareel-web
       HostName datareel.com
       User <cpanel user>
       Port 2222
   
   $ vi ~/bin/drsm_osa.sh
   
   #!/bin/bash
   # Run the checks, then copy the reports to the hosted dashboard
   ~/drsm/bin/drsm_run.sh
   rsync -az --delete --exclude=.htaccess \
       ~/public_html/sysadmin/reports/ \
       datareel-web:public_html/drsm/osa/reports/
   
   $ chmod 755 ~/bin/drsm_osa.sh
   $ crontab -e
   
   15,45 * * * * ~/bin/drsm_osa.sh &> /dev/null

   Replace ~/public_html/sysadmin/reports/ with your $REPORTdir. The --exclude=.htaccess keeps the package's rules that turn on archive listings and hide status.json. If one workstation watches both sites, give each its own host group (GROUP=osa, GROUP=oca) and run drsm_run.sh -g osa and -g oca.

Public live demo

The live demo at https://datareel.com/drsm/dashboard/ is the same package built with -s, which adds sample reports for 15 made up hosts. Nothing from the machine that builds it goes into the sample reports, and their times always show as recent.

$ ~/drsm/utils/make_web_demo.sh -p /drsm/dashboard -s -o /tmp/drsm_web

Upload /tmp/drsm_web/drsm_dashboard.zip to public_html/drsm and extract it. Rebuild and upload again to refresh the demo after a dashboard update.

Cloud servers

• Ping is blocked. Most cloud providers drop ICMP. Leave network/ping out of the host's PROBES. The health run reports an error when it cannot log in over SSH, so a down host still alerts. With the older system_check.sh, set can_ping to no and it tests the host with an SSH login instead.
• SSH user, port and key. Cloud images log in as ubuntu, ec2-user, rocky and so on. Set them in the DRSM user's ~/.ssh/config under the same name you used for the host:

  Host cloud1.example.com
      User ubuntu
      Port 22
      IdentityFile ~/.ssh/id_ed25519

• Steal time. The CPU check reports hypervisor steal time, which shows a noisy neighbor or an exhausted burst credit.
• Outbound mail. Most providers block port 25. Send alerts through a smart host on port 587, see Smart host.
• SSH options. Connections use BatchMode, a 15 second connect timeout so a down host cannot hang a run, and StrictHostKeyChecking=accept-new: a new host key is accepted on first contact and a changed key is refused. Change them with SSH_OPTS and SCP_OPTS in ~/.drsm.sh.
• Shared hosting. Accounts such as HostGator shared plans use a jailed shell, often on port 2222. CPU, load and process numbers there are for the whole shared server, not just your account, and a check whose command is missing there, such as smartctl without root, reports Unknown instead of failing.
• The monitor itself. To watch the monitor host, give it a host file with COLLECTOR=local, so no SSH is needed.