K3s Pod stuck in terminating state

A pod is stuck in terminating state, possibly due to finalizers or resource cleanup issues.

Understanding K3s and Its Purpose

K3s is a lightweight Kubernetes distribution designed for resource-constrained environments and edge computing. It simplifies the deployment and management of Kubernetes clusters by reducing the complexity and resource requirements typically associated with Kubernetes. K3s is ideal for IoT devices, ARM processors, and other environments where traditional Kubernetes might be too heavy.

Identifying the Symptom: Pod Stuck in Terminating State

One common issue encountered in K3s is when a pod becomes stuck in a terminating state. This symptom is observed when a pod does not shut down as expected and remains in a 'Terminating' status indefinitely. This can prevent the deployment of new pods and disrupt the normal operation of your cluster.

Exploring the Issue: Why Pods Get Stuck

When a pod is stuck in the terminating state, it is often due to issues with finalizers or resource cleanup. Finalizers are used to ensure that specific cleanup tasks are completed before a resource is deleted. If a finalizer is not removed properly, it can cause the pod to remain in a terminating state. Additionally, issues with network policies or volume detachments can also lead to this problem.

Common Causes

  • Finalizers not being removed.
  • Network policies preventing proper shutdown.
  • Persistent volumes not detaching correctly.

Steps to Resolve the Issue

To resolve a pod stuck in the terminating state, follow these steps:

Step 1: Identify the Pod

First, identify the pod that is stuck by using the following command:

kubectl get pods --all-namespaces | grep Terminating

This command will list all pods in the terminating state across all namespaces.

Step 2: Check for Finalizers

Inspect the pod's YAML to check for any finalizers:

kubectl get pod <pod-name> -n <namespace> -o yaml

Look for the finalizers field. If present, it may be preventing the pod from terminating.

Step 3: Remove Finalizers

If finalizers are causing the issue, you can remove them by editing the pod:

kubectl patch pod <pod-name> -n <namespace> -p '{"metadata":{"finalizers":null}}'

This command will remove all finalizers, allowing the pod to terminate.

Step 4: Force Delete the Pod

If the pod still does not terminate, you can force delete it:

kubectl delete pod <pod-name> -n <namespace> --grace-period=0 --force

This command will forcefully remove the pod from the cluster.

Additional Resources

For more information on managing pods in Kubernetes, check out the official Kubernetes Pod Lifecycle documentation. If you are new to K3s, the K3s Documentation is a great place to start.

Master

K3s

in Minutes — Grab the Ultimate Cheatsheet

(Perfect for DevOps & SREs)

Most-used commands
Real-world configs/examples
Handy troubleshooting shortcuts
Your email is safe with us. No spam, ever.

Thankyou for your submission

We have sent the whitepaper on your email!
Oops! Something went wrong while submitting the form.

K3s

Cheatsheet

(Perfect for DevOps & SREs)

Most-used commands
Your email is safe with us. No spam, ever.

Thankyou for your submission

We have sent the whitepaper on your email!
Oops! Something went wrong while submitting the form.

MORE ISSUES

Made with ❤️ in Bangalore & San Francisco 🏢

Doctor Droid