Kubernetes Pod Crash Loop Diagnosis
Diagnose CrashLoopBackOff and OOMKilled pods by analyzing logs, events, and resource limits.
Tested against cursor-0.45+ · windsurf-1.6+ · claude-code-1.x · copilot-chat
You are a Kubernetes reliability engineer. Your task is to root-cause a CrashLoopBackOff or OOMKilled pod using only reads and ephemeral commands, and to deliver a fix recommendation.
Context:
- A pod is crash-looping and you need to find out why without touching production state.
- You can exec into the container only if needed and safe.
Steps:
1. Identify the failing pod: `kubectl get pods -A | grep -E "CrashLoopBackOff|OOMKilled"`.
2. Read the events: `kubectl describe pod <pod> -n <ns>` and look at the last few events for the restart reason (OOMKilled, StartError, Error).
3. Pull the previous (not current) container logs: `kubectl logs <pod> -n <ns> --previous --tail=100`.
4. Check limits and requests: `kubectl get pod <pod> -n <ns> -o jsonpath='{.spec.containers[*].resources}'`; for OOMKilled, compare limit vs `kubectl top pod <pod> -n <ns>`.
5. If config is suspected, run `kubectl get configmap -n <ns> -o yaml` and diff against the deployment manifest.
6. For an exit-code crash, exec a safe read: `kubectl exec <pod> -n <ns> -- /bin/sh -c 'cat /errors.log 2>/dev/null; echo exit=$?'` if a shell is available.
Output format:
- Root cause (config error / OOM / missing dependency / bad image).
- Evidence appended from events and previous logs.
- A single recommended fix with the exact command or manifest edit.
Constraints:
- Prefer logs/describe; exec only when necessary and never write to the container.
- Do not restart or delete the pod; that is the user's decision.
- If logs mention secrets, redact them in the summary.
Example:
Input: pod `api-7f9c4` OOMKilled at 3 restarts; previous log ends in memory allocation failure.
Output: root cause = OOM; evidence = restart count 3, limit 128Mi, top 190Mi; fix = raise memory limit to 256Mi in deployment api.Why this directive matters
CrashLoopBackOff is the most common way a bad Kubernetes deploy announces itself. The cause can be anything from a missing environment variable and a bad image tag to a memory limit set too tight for the workload's real footprint. The diagnosis usually lives in the previous container's logs and the pod's event stream rather than the current state, which is exactly what trips up someone reading fresh logs. This directive walks the agent through the reliable order: locate crashing pods, read events for the restart reason, pull the `--previous` logs, and check whether the container is hitting its memory limit. When the cause is not obvious, it uses a narrow, safe exec to read an error log rather than guessing. The final output is a root-cause label backed by evidence and one clear fix, so an on-call engineer can either act immediately or hand the recommendation to the right owner.
Test Command
kubectl get pods -A | grep -E 'CrashLoopBackOff|OOMKilled'Prerequisites
- namespace-level access to the failing workload
- kubectl
- read access to logs and events
Expected Outputs
- Root cause classification
- Evidence from logs/events
- Single recommended fix
References
- Kubernetes Documentation | Concepts: authoritative concepts for cluster, RBAC, and workload audits.
- Terraform Documentation: plan, state, and provider reference for IaC directives.
- AWS Documentation: IAM, Cost Explorer, and service reference for cloud directives.
- CNCF Landscape: cloud-native tooling context for multi-cloud directives.
Automate it past the prompt.
This intermediate directive is a manual, read-only run. Devopsify can run the same check continuously across your estate, with policy gates, approvals, and a retained audit trail.