OpenSearch Index Read-Only Mode

Understanding OpenSearch

OpenSearch is a powerful, open-source search and analytics suite derived from Elasticsearch. It is designed to provide users with a robust platform for searching, visualizing, and analyzing data in real-time. OpenSearch is widely used for log analytics, full-text search, and operational intelligence.

Symptom: Index Read-Only Mode

In OpenSearch, you might encounter a situation where an index is set to read-only mode. This is a common alert that indicates potential issues with your cluster's health, specifically related to disk space.

Details About the Alert

What Does Read-Only Mode Mean?

When an index is in read-only mode, it means that no new data can be written to it. This is often triggered automatically by OpenSearch when the disk space on a node falls below a certain threshold. This protective measure helps prevent data corruption and ensures the stability of the cluster.

Why Does This Happen?

The primary reason for an index being set to read-only mode is insufficient disk space. OpenSearch monitors disk usage and will set indices to read-only to prevent the node from running out of space completely, which could lead to more severe issues.

Steps to Fix the Alert

Step 1: Free Up Disk Space

Start by identifying and clearing unnecessary data from your nodes. This could involve deleting old indices or moving data to a different storage solution. You can use the following command to delete an index:

DELETE /your_index_name

Ensure you have backups before deleting any data.

Step 2: Change Index Back to Read-Write Mode

Once you have freed up sufficient disk space, you can change the index back to read-write mode using the following command:

PUT /your_index_name/_settings{ "index.blocks.read_only_allow_delete": null}

This command removes the read-only block from the index, allowing it to accept new data.

Step 3: Monitor Disk Usage

To prevent future occurrences, regularly monitor your disk usage. You can use OpenSearch's built-in monitoring tools or integrate with external solutions like Prometheus for more comprehensive monitoring.

Additional Resources

For more information on managing indices and disk space in OpenSearch, refer to the OpenSearch Documentation. Additionally, consider exploring Elasticsearch Index Modules for similar concepts applicable to OpenSearch.