Alerts, audit and webhooks

Operational signals are tenant-scoped. Workers evaluate metrics and budgets, create deduplicated alerts, and optionally enqueue signed webhook deliveries. Human incident changes and system actions remain separately visible in audit evidence.

Alert evaluation

The alerts.evaluate job averages CPU metrics over the last hour. A resource above 90% produces one open WARNING alert until it is acknowledged or resolved. Identical open alerts are skipped. When month-to-date spend crosses the configured budget threshold, the worker creates a budget notification and queues a budget.threshold webhook event.

metrics / cost rows -> alerts.evaluate
          |                 |
          |                 +--> alert / notification
          |                 +--> webhook.deliver (optional)
          v
incident operator -> acknowledge -> investigate -> mitigate -> resolve

Incidents use OPEN, INVESTIGATING, IDENTIFIED, MITIGATING, MONITORING, RESOLVED and CLOSED, with an append-only event timeline.

Audit evidence

Audit entries include organization, actor type (USER, SYSTEM or AI), action, category, target, target type, request ID, IP and metadata. Query /api/v1/audit by action, actor, target or time window. Audit failure must not break the primary operation, but an operation that lacks expected audit evidence should be investigated.

Webhook delivery

Creating a webhook returns its secret once. Delivery posts an envelope containing event ID, event type, organization, timestamp and payload. The raw JSON body is signed with HMAC-SHA256 in X-Devopsify-Signature; X-Devopsify-Event carries the event name.

Condition	Worker behavior
2xx	Mark active, reset failure count and finish
5xx, timeout or network error	Retry up to 3 attempts with 500 ms, 1 s and 2 s backoff
4xx	Stop retrying that delivery
Final failure	Mark FAILING, raise one open warning alert and retain the error

Demo mode simulates outbound webhook delivery and must not be described as an external notification. Production consumers should verify the signature over the exact raw request body and deduplicate by event ID.