CrashLoopBackOff

Why this matters

CrashLoopBackOff means a container is repeatedly starting and crashing. It usually points to a real bug or a bad configuration. Every restart wastes resources and delays user traffic; in production this often shows up as 5xx errors, timeouts, or failing background jobs.

Symptoms

• kubectl get pods shows STATUS CrashLoopBackOff for one or more pods.
• Service or ingress in front of the workload is returning 5xx or timeouts.
• Containers start, run for a few seconds, then exit with a non-zero code.
• Logs show the same error pattern every time the pod restarts.

Common root causes

• Application startup failures (missing env vars, invalid config, missing migrations).
• Crash on dependency connection (DB not reachable, message broker auth failure).
• Image mismatch vs expected config (new image expects different env/flags).
• Fatal probe configuration (liveness/readiness probe failing on a new path/port).
• Secrets or ConfigMaps missing, wrong key names, or wrong mount paths.

How KubeGraf helps

• Highlights CrashLooping pods in the namespace so you don't hunt through raw kubectl output.
• Shows restart counts, last state (e.g. Error, exit code), and recent events next to the pod.
• Lets you pivot quickly between pod logs, events, Deployment, ConfigMap, and Secret.
• Incident timeline view helps correlate: image deployed → config changed → probes failing → CrashLoopBackOff.

Step-by-step using KubeGraf UI

1. Confirm the problem in the right cluster/namespace

kubectl config current-context
kubectl get pods -n <namespace>

• Start KubeGraf Terminal UI with kubegraf.
• Ensure the context and namespace in KubeGraf match what you just checked.

2. Locate CrashLooping pods

• Open the Pods view for the affected namespace.
• Use filters to show only unhealthy pods (status CrashLoopBackOff / Error).
• Note restart count, age, and container name (if multi-container pod).

3. Inspect recent events and reasons

• From the pod details, open Events.
• Look for messages such as Back-off restarting failed container, probe failures, or image pull errors.

4. Inspect logs around the crash

2025-03-22T12:01:03Z ERROR app Failed to start HTTP server: DB_CONNECTION_STRING not set
2025-03-22T12:01:03Z ERROR app Exiting with code 1

• From the same pod, open Logs in KubeGraf and scroll to the last lines before exit.
• Capture the exact error message and exit code.

5. Check configuration linked to the pod

• From pod details, jump to its Deployment (or StatefulSet/Job).
• Review container image tag, env vars, and probes.
• Follow links to ConfigMaps and Secrets referenced by the pod and compare with what the app expects.

6. Use Incident Timeline / change history

• Open the Incident Timeline for this workload/namespace.
• Look for deploys, config updates, or probe changes just before the CrashLoopBackOff started.

7. Apply fix and watch recovery

• Typical fixes include reverting a bad config/Secret, fixing missing env vars, or correcting probe path/port.

kubectl rollout undo deployment/<name> -n <namespace>
kubectl edit configmap <config-name> -n <namespace>

• Use KubeGraf to watch new pods transition from CrashLoopBackOff to Running.

What to check next

• Are other pods in the same Deployment also impacted, or only one replica?
• Does the issue correlate with a specific node (node-local problem)?
• Is the CrashLoop only in one namespace or across multiple environments (dev/staging/prod)?

Common mistakes

• Debugging the wrong cluster/namespace because the kubeconfig context was not checked.
• Only looking at logs and ignoring Events (probe misconfig is often obvious there).
• Fixing a single pod manually instead of changing the Deployment/ConfigMap/Secret.
• Rolling back an image without rolling back the config that was changed at the same time.

Related issues

• Deployment rollout stuck / ProgressDeadlineExceeded
• Pods restarting after ConfigMap or Secret change
• "It was working yesterday" incidents

Expected outcome

After following this playbook you should:

• Identify whether the CrashLoopBackOff is due to configuration, code, or environment.
• Know which change introduced the failure and either roll back or fix forward safely.
• See pods return to Running and external symptoms (5xx, latency) disappear.

[ TODO: screenshot showing KubeGraf with a CrashLoopBackOff pod selected, logs + events visible. ]