Kube-probe Liveness probe failed: container not running

What is Kube-probe Liveness probe failed: container not running

Understanding Kube-probe

Kube-probe is a diagnostic tool used in Kubernetes to check the health of containers. It helps ensure that applications running within containers are functioning correctly by periodically sending requests to the container and checking the response. There are three types of probes: liveness, readiness, and startup. The liveness probe determines if a container is running and healthy, allowing Kubernetes to restart it if necessary.

Identifying the Symptom

When you encounter the error message 'Liveness probe failed: container not running', it indicates that the liveness probe has detected that the container is not in a running state. This can lead to Kubernetes attempting to restart the container, which may disrupt your application's availability.

Exploring the Issue

What Causes This Error?

The primary cause of this error is that the container has crashed or is not running as expected. This can happen due to various reasons such as application errors, resource constraints, or misconfigurations.

Impact of the Error

When a liveness probe fails, Kubernetes will attempt to restart the container. If the underlying issue is not resolved, this can lead to a crash loop, where the container repeatedly fails and restarts.

Steps to Fix the Issue

1. Investigate Container Logs

The first step in diagnosing the issue is to check the container logs for any error messages or stack traces that might indicate the cause of the crash. You can view the logs using the following command:

kubectl logs <pod-name> -c <container-name>

Replace <pod-name> and <container-name> with the actual names of your pod and container.

2. Check Resource Constraints

Ensure that your container has sufficient resources allocated. You can check the resource requests and limits defined in your pod specification:

kubectl describe pod <pod-name>

If the container is running out of memory or CPU, consider increasing the resource limits.

3. Review Probe Configuration

Verify that the liveness probe is correctly configured in your deployment. Check the probe's path, port, and initial delay settings. Incorrect configurations can lead to false negatives:

kubectl get deployment <deployment-name> -o yaml

Ensure the probe settings match the application's expected behavior.

4. Update or Rollback the Application

If a recent change caused the issue, consider rolling back to a previous stable version of your application. Use the following command to rollback:

kubectl rollout undo deployment/<deployment-name>

Alternatively, update the application to fix any bugs that might be causing the crash.

Additional Resources

For more information on configuring liveness probes, refer to the Kubernetes official documentation. If you are new to Kubernetes, the Kubernetes Basics tutorial is a great place to start.