Kubernetes Drift Detection
Detect configuration drift between Git (ArgoCD/Flux) and live cluster state using an AI agent.
Tested against cursor-0.45+ · windsurf-1.6+ · claude-code-1.x · copilot-chat
You are a Kubernetes platform engineer. Your task is to detect configuration drift between the Git source of truth (ArgoCD or Flux) and live cluster state, then report it in a structured way. Context: - You have cluster-reader access to one or more namespaces or the whole cluster. - The Git repo contains the manifests the GitOps controller tracks (Application/HelmRelease specs). - You must never modify resources; this is a read-only audit. Steps: 1. Identify the GitOps-reconciled resources: run `kubectl get applications -A -o wide` (ArgoCD) or `kubectl get kustomizations -A -o wide` (Flux). 2. For each target namespace, snapshot live state: `kubectl get deploy,sts,ds,svc,cronjob,cm,secret -n <ns> -l app.kubernetes.io/managed-by=Helm -o name`. 3. Compare the live manifest hash with the last applied revision stored in the resource annotation (e.g. `argocd.argoproj.io/tracked-revision` or `kustomize.toolkit.fluxcd.io/revision`). 4. For each mismatch, run `kubectl rollout status deploy/<name> -n <ns>` and inspect `kubectl describe` events for out-of-band edits. 5. Verify whether drift is manual (a live `kubectl edit`) or remedial (a health-check auto-scaling change) by checking the `last-applied-configuration` annotation. Output format: - Markdown table with columns: namespace, resource, kind, git revision, live revision, drift type (manual/remedial), severity (low/medium/high). - One paragraph summarizing the highest-risk drift and whether GitOps will self-heal it. Constraints: - Only use read-only commands (get, describe, diff). Never apply, edit, or delete. - Do not include secrets or configmap values in the report, only metadata. - If a resource is out of security context (system namespaces), flag it and move on. Example: Input: Application `payments` in namespace `payments` tracked at revision `a1b2c3`, live Deployment image differs. Output: row [payments, api, Deployment, a1b2c3, a1b2c3+dirty, manual, high] + note: image drifted from `nginx:1.24` to `nginx:1.25-live`; GitOps will revert unless the manifest is updated.
Why this directive matters
Kubernetes drift is one of the most common sources of 'works locally, gives up in prod' incidents. When someone pushes a change directly to the cluster with a quick kubectl edit, the Git repository that GitOps controllers treat as the source of truth silently diverges. The next reconcile typically reverts that change, which is confusing when the revert lands in the middle of a deploy. This directive turns an AI agent into a drift-auditing companion: it inventories the GitOps-tracked workloads, snapshots live state, compares revisions, and classifies each difference as manual or remedial. Running it on a schedule gives a platform team a recurring report of where the cluster no longer matches Git. It is best used before an incident postmortem, during a migration, or as a routine hygiene check after a release. The output is deliberately read-only and human-reviewable, so it is safe to run even in tightly governed environments.
Test Command
kubectl get applications -A -o wide || kubectl get kustomizations -A -o widePrerequisites
- kubectl config with cluster access
- cluster-reader RBAC
- Git repo with ArgoCD/Flux manifests
Expected Outputs
- Markdown table of drift report
- Severity classification per resource
- Suggested remediation steps
References
- Kubernetes Documentation | Concepts: authoritative concepts for cluster, RBAC, and workload audits.
- Terraform Documentation: plan, state, and provider reference for IaC directives.
- AWS Documentation: IAM, Cost Explorer, and service reference for cloud directives.
- CNCF Landscape: cloud-native tooling context for multi-cloud directives.
Automate it past the prompt.
This intermediate directive is a manual, read-only run. Devopsify can run the same check continuously across your estate, with policy gates, approvals, and a retained audit trail.