Skip to content
Steelframe v0.19.7

Agent sentinel

The coordinator keeps running after the database is invalidated. The agent logs the fatal and stays up, so systemd Restart= does not fire. Discord is stored in the database, so the agent cannot send the alert either.

The agent sentinel is a separate process. It is not the coordinator’s own loop. It watches the current MainPID of steelframe.service and restarts that unit when one of these is true:

  • The current PID’s journal says the database has been invalidated.
  • The unit stays inactive or failed longer than the grace period.
  • The unit is active, but its listen port refuses a TCP connection longer than the grace period.

HTTP status is not a health signal. A 200 from the status route can be a process whose database reads are failing, and a 401 only means the auth gate answered.

The first unhealthy sample restarts the agent once. Samples during the grace period after that restart do nothing: the new process is still booting. If the agent is still unhealthy after that, the sentinel sends one alert and does not restart again until it has seen the agent healthy. A later outage can restart again, and never more often than the cooldown (default 15 minutes). A failed restart is recorded the same way, so a broken unit is not restarted in a loop.

An optional public URL is alert-only. The sentinel waits one grace period after it first starts, then alerts if the probe fails or returns a 5xx. It does not restart the agent for that. 401 and 200 count as reachable.

Discord

Section titled “Discord”

The webhook is a mode-0600 file, or STEELFRAME_SENTINEL_WEBHOOK. The sentinel never reads it from the database, and its unit must not use EnvironmentFile= on agent.env (that file holds the master key).

On a system unit, put the file at /etc/steelframe/discord-webhook, owned by root, mode 0600. /etc/steelframe stays root-owned so the agent user cannot replace the file. On a user unit the sentinel runs as the same user as the agent; the file is ~/opt/steelframe/data/discord-webhook.

If no webhook is available, the sentinel can post as the Discord bot instead. Put the bot token in a separate mode-0600 file (/etc/steelframe/discord-bot-token on a system unit, ~/opt/steelframe/data/discord-sentinel-token on a user unit) and set STEELFRAME_SENTINEL_DISCORD_CHANNEL to the alert channel ID. The webhook is preferred when both are configured. The bot token file is never agent.env.

A group- or world-accessible file is ignored. The alert is still written to the sentinel journal. A failed post is retried on the next check. The message names the host and the reason. It does not include the webhook, the master key, or a token.

Install

Section titled “Install”

System unit (make -C infra/steelframe install-sentinel):

Terminal window
sudo install -d -m 0755 /etc/steelframe
sudo install -m 0755 infra/steelframe/agent-sentinel.sh /opt/steelframe/agent-sentinel.sh
sudo install -m 0644 infra/steelframe/steelframe-sentinel.service /etc/systemd/system/steelframe-sentinel.service
sudo systemctl daemon-reload
sudo systemctl enable --now steelframe-sentinel.service

User unit (the dogfood host):

Terminal window
install -m 0755 infra/steelframe/agent-sentinel.sh "$HOME/opt/steelframe/agent-sentinel.sh"
install -m 0644 infra/steelframe/steelframe-sentinel.user.service \
  "$HOME/.config/systemd/user/steelframe-sentinel.service"
systemctl --user daemon-reload
systemctl --user enable --now steelframe-sentinel.service

Set the public URL and the display name in a drop-in, not in the checked-in unit:

[Service]
Environment=STEELFRAME_SENTINEL_NAME=dogfood
Environment=STEELFRAME_SENTINEL_PUBLIC_URL=https://steelframe.example/api/status

agent install does not lay this unit down. Copying the script into /opt/steelframe survives an agent binary update; the update replaces the binary only.

Environment variables are listed in Deploy. Checks run every 30 seconds. The grace period is 120 seconds. The cooldown is 900 seconds.