Retries and failure recovery

Devopsify separates retryable transport work from irreversible intent. A retry must not create a second side effect or hide a partial result.

Worker behavior

With Redis, BullMQ provides distributed locks, per-queue concurrency, exponential retry with jitter and progress events. Without REDIS_URL, inline mode invokes the same handlers synchronously for demo and tests. Every execution is persisted as a BackgroundJobRecord; Redis queue state is not enough to recover an incident.

Job family	Recovery behavior
Cloud/resource sync	Retry provider pages; retain sync error and last successful observation
Provisioning	Revalidate plan/approval, lock workspace, persist artifacts and verify; never silently re-plan
Pipeline monitor	Poll status; stop on terminal failure and let deployment verification decide rollback availability
Agent command	Enforce deadline and typed result; timed out commands are not reported as succeeded
Alert evaluation	Deduplicate open alerts and budget triggers
Webhook delivery	Three attempts for transient failures; mark FAILING and create a warning alert on final failure

Idempotency and duplicate delivery

Mutating API requests may send an Idempotency-Key. The server records the key and returns the original result for a repeat. Worker enqueue also deduplicates through IdempotencyRecord; repeated job keys are no-ops. Provisioning apply is additionally keyed to the run ID so a repeated apply does not rerun OpenTofu.

request + key -> idempotency record
       | new                         | duplicate
       v                             v
persist intent -> enqueue        return original result
       |
worker lock -> revalidate -> side effect once -> verify -> persist result

Use one stable key for one intent. Do not generate a new key when a response is lost. For risk 3 actions, investigate the job record and audit log before any retry.

Recovery checklist

1. Capture x-request-id, organization, resource/run ID and idempotency key.
2. Inspect API response, BackgroundJobRecord, worker logs and audit event without exposing secrets.
3. Determine whether the operation reached a provider, agent or repository boundary.
4. Reconcile observed state before creating a new plan or retrying a mutation.
5. Resume only with the original idempotency key when the operation is explicitly safe to retry; otherwise create a new approved plan.
6. For partial success, preserve the original evidence, verify each target and use a compensating run or rollback path.