UPS + NUT: auto shutdown 3 Proxmox node and the NAS when the power cut (rainy season guide)

Rainy season in HCMC. Some weeks we have 2-3 power cuts, 10 to 40 minutes. My APC Back-UPS 1000VA can run my 186W lab for ~18 minutes (UPS says), so for long cuts everything must shut down cleanly by itself, even when I am at work or asleep. This is how I do it with NUT (Network UPS Tools).

Plan

• The UPS USB cable goes to one machine. That machine is the NUT primary (server)
• All other machines are secondary (clients) and listen over the network
• Rule: on battery for 3 minutes, OR battery low, then shut down everything
• Order: VMs → Proxmox nodes → NAS last (the VMs keep disks on the NAS)

Primary (the box with the USB cable)

TrueNAS has NUT built in (System Settings, Services, UPS) and you can do all of this in the UI. On a normal Debian or Proxmox host it looks like this:

apt install nut

# /etc/nut/nut.conf
MODE=netserver

# /etc/nut/ups.conf
[apc]
    driver = usbhid-ups
    port = auto
    desc = "Back-UPS 1000"

# /etc/nut/upsd.conf
LISTEN 0.0.0.0 3493

# /etc/nut/upsd.users
[monuser]
    password = change-me-please
    upsmon primary

# /etc/nut/upsmon.conf
MONITOR apc@localhost 1 monuser change-me-please primary
SHUTDOWNCMD "/sbin/shutdown -h +0"

Check:

$ upsc apc@localhost
battery.charge: 100
battery.runtime: 1080
ups.load: 31
ups.status: OL

OL = online, OB = on battery, LB = low battery. battery.runtime is in seconds.

Secondaries (each Proxmox node)

# /etc/nut/nut.conf
MODE=netclient

# /etc/nut/upsmon.conf
MONITOR apc@192.168.10.5 1 monuser change-me-please secondary
SHUTDOWNCMD "/sbin/shutdown -h +0"

Proxmox already shut down the VMs when the host shut down (set a shutdown timeout per VM in the VM options, default can be too long).

The 3 minute timer: upssched

By default NUT only shut down at low battery. I want earlier, because my UPS battery is 3 years old and I don't trust its "18 minutes". In upsmon.conf on the primary add:

NOTIFYCMD /sbin/upssched
NOTIFYFLAG ONBATT SYSLOG+WALL+EXEC
NOTIFYFLAG ONLINE SYSLOG+WALL+EXEC

/etc/nut/upssched.conf:

CMDSCRIPT /etc/nut/upssched-cmd
PIPEFN /run/nut/upssched.pipe
LOCKFN /run/nut/upssched.lock
AT ONBATT * START-TIMER onbatt 180
AT ONLINE * CANCEL-TIMER onbatt
AT LOWBATT * EXECUTE onbatt

/etc/nut/upssched-cmd (chmod +x):

#!/bin/sh
case $1 in
  onbatt) /sbin/upsmon -c fsd ;;
esac

upsmon -c fsd = "forced shutdown". The primary tells all secondaries to shut down, waits for them to disconnect, then shuts down itself.

First test failed

I pull the UPS plug from the wall (everything backed up first!). After 3 min, the NAS shut down in ~20 seconds, the Proxmox nodes were still shutting down VMs that have disks on the NFS share from the NAS. Two VMs had a bad time.

The reason: the primary waits for the secondaries to log out, but only for HOSTSYNC seconds (default 15). My nodes need ~2 minutes to stop all VMs. Fix in primary upsmon.conf:

HOSTSYNC 180

Also make sure your 3 minute timer + 3 minutes HOSTSYNC + shutdown time is less than the real battery runtime, with margin.

Coming back

In the BIOS of every machine: "Restore on AC power loss: Power On". Then when power come back, everything boot by itself. Put the NAS first in boot priority or add a delay to the Proxmox nodes, so NFS is ready before VMs start.

And replace the UPS battery every 3 years. Lead acid hate heat, and Saigon is heat.

hope this help :)