Project · reliability tool

Gateway Watchdog

A watchdog that monitors Hermes gateway user units for a Matrix connection failure newer than the last successful connection, writes timestamped JSON evidence before any restart, records post-restart health, and posts one evidence-backed recovery notification per event.

Boundary: This tool only restarts Hermes gateway units on the local machine. It does not touch external services, post to public channels, or modify credentials. Evidence is written to a local state directory with redacted credential-shaped values.

How it works

Checks systemd journal for the gateway unit's Matrix connection state.
Detects a connection failure that is newer than the last successful connection marker.
Writes a JSON evidence record (timestamp, event id, unit, error class, redacted trigger, last success, action, post-restart health).
Restarts the gateway unit.
Verifies post-restart health and posts a single recovery notification with the evidence path.

Safe verification

WATCHDOG_DRY_RUN=1 WATCHDOG_EVIDENCE_DIR=/tmp/watchdog-test ./gateway-watchdog.sh records evidence and previews the notification without restarting anything. tests/test-watchdog.sh supplies mocked systemd/journal/Matrix/rodion commands and exercises both dry-run and restart paths without touching live services.

Verification

Run ./tests/test-watchdog.sh. Inspect the local evidence directory for evidence files. Each restart event produces a dated JSON record that can be independently audited.