Debug Your Infrastructure
Get Instant Solutions for Kubernetes, Databases, Docker and more
Thank you! Your submission has been received!
Oops! Something went wrong while submitting the form.
Pod Stuck in CrashLoopBackOff
Database connection timeout
Docker Container won't Start
Kubernetes ingress not working
Redis connection refused
CI/CD pipeline failing
OpenSearch Node Disk Watermark Exceeded
Disk usage on a node has exceeded the high watermark threshold.
Understanding OpenSearch and Its Purpose
OpenSearch is a powerful, open-source search and analytics suite that is used for a variety of applications, including log analytics, real-time application monitoring, and search. It is designed to be highly scalable and flexible, allowing users to ingest, search, visualize, and analyze data in real-time. OpenSearch is a community-driven project that originated from Elasticsearch and is maintained by a diverse group of contributors.
Symptom: Node Disk Watermark Exceeded
One of the common alerts you might encounter when using OpenSearch is the 'Node Disk Watermark Exceeded' alert. This alert is triggered when the disk usage on a node surpasses the high watermark threshold, indicating that the node is running low on disk space.
Details About the Alert
What Triggers This Alert?
The 'Node Disk Watermark Exceeded' alert is triggered when the disk usage on a node exceeds a predefined percentage of the total disk space. OpenSearch uses watermarks to manage disk usage and ensure that there is always enough space available for operations. The high watermark is typically set at 90% of the total disk space.
Impact of the Alert
When this alert is triggered, OpenSearch will stop allocating new shards to the affected node to prevent further disk usage. This can lead to degraded performance and, if not addressed, could result in data loss or cluster instability.
Steps to Fix the Alert
1. Identify the Affected Node
First, you need to identify which node is triggering the alert. You can do this by checking the OpenSearch logs or using the OpenSearch API. Run the following command to get the disk usage for each node:
GET _cat/nodes?v&h=name,disk.used_percent
This command will return a list of nodes with their respective disk usage percentages.
2. Free Up Disk Space
Once you have identified the affected node, you can take steps to free up disk space. This can be done by deleting old or unnecessary indices. Use the following command to delete an index:
DELETE /index_name
Ensure that you have backups of any important data before deleting indices.
3. Increase Disk Capacity
If freeing up space is not sufficient, consider increasing the disk capacity of the affected node. This may involve adding more storage to the existing node or migrating to a larger instance type if you are using a cloud provider.
4. Adjust Watermark Settings
If you frequently encounter this alert, you may need to adjust the watermark settings. You can do this by updating the cluster settings:
PUT _cluster/settings
{
"persistent": {
"cluster.routing.allocation.disk.watermark.high": "95%"
}
}
Adjust the percentage according to your needs, but be cautious not to set it too high, as this could lead to other issues.
Additional Resources
For more information on managing disk space in OpenSearch, you can refer to the OpenSearch Documentation. Additionally, the OpenSearch Blog provides insights and updates on best practices for managing your OpenSearch cluster.
Master
OpenSearch Node Disk Watermark Exceeded
debugging in Minutes
— Grab the Ultimate Cheatsheet
(Perfect for DevOps & SREs)
Most-used commands
Real-world configs/examples
Handy troubleshooting shortcuts
Oops! Something went wrong while submitting the form.
OpenSearch Node Disk Watermark Exceeded
Cheatsheet
(Perfect for DevOps & SREs)
Most-used commands
Oops! Something went wrong while submitting the form.
MORE ISSUES
Backed by
Deep Sea Tech Inc. — Made with ❤️ in Bangalore & San Francisco 🏢