Health checks

Health checks (safety checks) are the agent’s definition of a healthy environment. They are configured per environment and used in two places:

1. Delivery gates — staging_healthchecks and prod_healthchecks in task_lifecycle.
2. Production monitor — production_monitor samples the checks for every monitored environment each cycle and escalates to investigation when an environment crosses the failure threshold.

Manage them under Config → Health checks or the JSON API:

Method	Path
GET / POST	/api/health-checks
PATCH / DELETE	/api/health-checks/{id}

Check shape

Field	Meaning
name	Check name (shown on the Dashboard)
kind	http_health, ci_workflow, metric_threshold, or smoke_test
environment	staging or production
config	Kind-specific JSON
required	Whether failure blocks a promotion gate

http_health

{ "url": "https://staging.example.com/health", "timeout_secs": 10, "expected_status": 200 }

Passes when the response status equals expected_status.

ci_workflow

{ "repo": "org/repo", "workflow": "ci.yml", "branch": "staging", "timeout_secs": 20 }

Checks the latest run of a GitHub Actions workflow with gh run list. When branch is absent it defaults to the check’s environment, so a production check polls the production branch rather than staging.

metric_threshold

{ "url": "https://metrics.example.com/api", "json_path": "value", "threshold": 0.05, "comparison": "lt", "timeout_secs": 10 }

Reads a JSON number and compares it to threshold with comparison. A source of gcx/grafana routes the check through Grafana instead of raw HTTP.

smoke_test

{ "command": "curl -fsS https://staging.example.com/health", "timeout_secs": 30 }

Runs the command with sh -c and passes on exit code 0. The child runs in its own process group and is killed with its children on timeout.

How checks gate promotion

Both gates call run_safety_checks(environment) and evaluate promotion_gate_passed:

• Only a failed required check blocks promotion.
• Optional check failures never block.
• System failures never block. A missing gcx binary, an unconfigured Grafana credential, or a gh spawn/auth error is agent-side tooling, not environment health. It is recorded and logged (promotion gate ignoring system failure) but does not stall staging or auto-revert production. A Grafana connection failure (no route to host, refused, timeout) is not tooling: it is an environment health signal and escalates to an investigation.

Staging behavior:

• For staging_deploy_strategy=workflow, a green gate advances to merging_pr.
• For merge_pr, staging is the delivery target and the run completes there.
• staging_soak_minutes (default 0) holds staging after the gate first goes green so it proves itself in use before promotion.
• A red gate re-polls every 60s. A persistently red gate trips the step’s timeout and fails the run.

Production behavior:

• A green gate completes the run.
• A red gate re-polls. A persistently red gate crosses the revert deadline and the run transitions to reverting_pr, which creates a revert PR and triggers redeploy.

Monitored environments

monitor_environments is a comma-separated list (for example staging,production) of the environments the periodic production_monitor samples every cycle. Unset, it falls back to monitor_environment (default production). The primary monitor_environment stays the target for incidents and optimizations that need one environment.

When an environment crosses the failure threshold, Steelframe spawns one investigate_incidents run per degraded environment, each with that environment’s Grafana and kubectl credentials, so a staging incident never investigates with production’s cluster.

Gotchas

• With no checks configured for an environment, the gate passes (no safety checks configured).
• Required vs optional is per check; mark only true environment health as required.
• The Dashboard’s environment health uses the latest result per check name for every environment in monitor_environments (falling back to monitor_environment, default production), independent of the staging gate. Groups and checks carry an environment label when more than one is monitored.
• A ci_workflow check defaults its branch to the environment name, which assumes your production branch is named production; set branch explicitly if it is not.