K3s NodeNotReadyDueToDiskPressure

A node is not ready due to disk pressure, affecting pod scheduling.

Understanding K3s

K3s is a lightweight Kubernetes distribution designed for resource-constrained environments and edge computing. It simplifies the deployment and management of Kubernetes clusters by reducing the overhead and complexity associated with traditional Kubernetes installations. K3s is particularly popular for IoT and CI/CD environments due to its minimal resource requirements and ease of use.

Identifying the Symptom

One common issue encountered in K3s is the NodeNotReadyDueToDiskPressure condition. This symptom manifests when a node in the cluster is marked as 'NotReady' due to insufficient disk space. This can prevent new pods from being scheduled on the affected node, potentially impacting application availability and performance.

Observing the Error

When this issue occurs, you may notice the following:

The node status is reported as 'NotReady' in the cluster.
Pods may be stuck in a 'Pending' state due to scheduling constraints.
Logs may show warnings or errors related to disk pressure.

Explaining the Issue

The NodeNotReadyDueToDiskPressure condition is triggered when the available disk space on a node falls below a certain threshold. Kubernetes monitors resource usage on nodes, and when disk space is critically low, it marks the node as 'NotReady' to prevent further scheduling of pods. This is a protective measure to ensure that existing workloads are not disrupted by running out of disk space.

Root Cause Analysis

The root cause of this issue is typically one of the following:

Excessive log files or temporary data consuming disk space.
Large container images or persistent volumes occupying disk capacity.
Improper disk cleanup or maintenance routines.

Steps to Resolve the Issue

To resolve the NodeNotReadyDueToDiskPressure issue, follow these steps:

Step 1: Check Disk Usage

Log into the affected node and check the current disk usage:

df -h

Identify the partitions with high usage and determine what is consuming the space.

Step 2: Free Up Disk Space

Remove unnecessary files and logs. For example, clear old log files:

sudo journalctl --vacuum-time=2d

Consider removing unused Docker images and containers:

docker system prune -a

Step 3: Increase Disk Capacity

If freeing up space is not sufficient, consider increasing the disk capacity of the node. This may involve resizing the disk in your cloud provider or adding additional storage.

Step 4: Verify Node Status

After addressing the disk space issue, verify that the node status returns to 'Ready':

kubectl get nodes

The node should now be listed as 'Ready', and pods should be able to schedule successfully.

Further Reading and Resources

For more information on managing disk pressure in Kubernetes, refer to the official Kubernetes documentation. Additionally, explore K3s documentation for specific guidance on managing K3s clusters.

Master

K3s

debugging in Minutes

— Grab the Ultimate Cheatsheet

(Perfect for DevOps & SREs)

Most-used commands

Real-world configs/examples

Handy troubleshooting shortcuts

Thankyou for your submission

We have sent the cheatsheet on your email!

Oops! Something went wrong while submitting the form.

K3s

Cheatsheet

(Perfect for DevOps & SREs)

Most-used commands

Thankyou for your submission

We have sent the cheatsheet on your email!

Oops! Something went wrong while submitting the form.

MORE ISSUES

K3s PodFailedToWatch

A pod failed to watch, possibly due to misconfiguration or resource issues.

K3s PodFailedToWrite

A pod failed to write, possibly due to misconfiguration or resource issues.

K3s PodFailedToWait

A pod failed to wait, possibly due to misconfiguration or resource issues.

K3s PodFailedToVerify

A pod failed to verify, possibly due to misconfiguration or resource issues.

K3s PodFailedToValidate

A pod failed to validate, possibly due to misconfiguration or resource issues.

K3s PodFailedToUpdate

A pod failed to update, possibly due to misconfiguration or resource issues.

K3s PodFailedToAttachVolume

A pod failed to attach a volume, possibly due to incorrect volume configuration.

K3s PodFailedToStop

A pod failed to stop, possibly due to finalizers or resource cleanup issues.

K3s PodFailedToStartContainer

A pod failed to start a container, possibly due to misconfiguration or resource issues.

K3s PodFailedToPullImage

A pod failed to pull an image, possibly due to incorrect image name or lack of access.

K3s PodFailedToSchedule

A pod failed to schedule due to constraints or lack of resources.

K3s PodFailedToMountVolume

A pod failed to mount a volume, possibly due to incorrect volume configuration.

K3s NodeNotReadyDueToDiskPressure

A node is not ready due to disk pressure, affecting pod scheduling.

K3s PodFailedToStart

A pod failed to start due to misconfiguration or resource issues.

K3s PodFailedToCreate

A pod failed to create due to misconfiguration or resource issues.

K3s PodFailedToDelete

A pod failed to delete, possibly due to finalizers or resource cleanup issues.

K3s NodeNotRegistered

A node is not registered with the cluster, affecting pod scheduling.

K3s PodSecurityContextViolation

A pod violates security context constraints, preventing it from running.

K3s PodOOMKilled

A pod was killed due to out-of-memory conditions.

K3s PodLivenessProbeFailure

A pod's liveness probe is failing, causing the pod to be restarted.

K3s PodAffinityRulesNotSatisfied

Pod affinity or anti-affinity rules are not satisfied, preventing scheduling.

K3s PodReadinessProbeFailure

A pod's readiness probe is failing, affecting service availability.

K3s PodInitContainerFailure

An init container in a pod has failed, preventing the pod from starting.

K3s Pods are not being scheduled on a node due to a taint.

A node taint is preventing pod scheduling.

K3s Ingress resources are not accessible.

Ingress resources are not accessible, possibly due to misconfiguration.

K3s Service account token has expired, affecting pod authentication.

A service account token has expired.

K3s Pod cannot pull an image due to missing or incorrect image pull secret.

A pod is unable to access the required image because the image pull secret is either missing or incorrectly configured.

K3s PodSecurityPolicyViolation

A pod violates a PodSecurityPolicy, preventing it from being scheduled.

K3s Network policies are blocking traffic, affecting pod communication.

Network policies are blocking traffic, affecting pod communication.

K3s Kubelet service is not running on a node, affecting pod management.

The kubelet service is not running on a node.

K3s Pod stuck in terminating state

A pod is stuck in terminating state, possibly due to finalizers or resource cleanup issues.

K3s NodeOutOfDisk

A node has run out of disk space, affecting pod scheduling and operation.

K3s UnauthorizedAccess

Access is denied due to incorrect credentials or permissions.

K3s API requests are timing out

Network issues or high load on the API server

K3s PersistentVolumeClaimPending

A PersistentVolumeClaim is pending due to lack of available PersistentVolumes.

K3s Network is unavailable on a node, affecting pod communication.

Network configuration issues or connectivity problems.

K3s NodeNotFound

A specified node cannot be found, possibly due to incorrect node name or deletion.

K3s FailedScheduling

A pod cannot be scheduled due to constraints or lack of resources.

K3s High memory usage observed in K3s cluster.

Excessive memory usage by pods or system components, causing resource exhaustion.

K3s NodeDiskPressure

A node is experiencing disk pressure, leading to pod eviction or scheduling issues.

K3s FailedMount

A pod cannot mount a volume, possibly due to incorrect volume configuration.

K3s Pods are unable to resolve DNS names.

CoreDNS issues causing DNS resolution failure.

K3s Pods are being evicted unexpectedly.

Pods are evicted due to resource constraints or node pressure.

K3s High CPU Usage

Excessive CPU usage by pods or system components, leading to performance degradation.

K3s PodUnschedulable

A pod cannot be scheduled due to insufficient resources or node constraints.

K3s ImagePullBackOff

K3s is unable to pull the specified container image, possibly due to incorrect image name or lack of access.

K3s Communication failures between K3s components due to expired certificates.

K3s certificates have expired, causing communication failures between components.

K3s ServiceUnavailable

A service is not reachable, possibly due to misconfigured service or network policies.

K3s NodeNotReady

The node is not in a ready state, possibly due to network issues or resource constraints.

K3s CrashLoopBackOff

A pod is repeatedly crashing due to application errors or misconfiguration.

Backed by

Resources

Contact

Platform

Connect

Deep Sea Tech Inc. — Made with ❤️ in & 🏢

Doctor Droid

Join and start debugging.

Log in or create a free account to start debugging your production application