OpenSearch: Cluster Blocks Read-Only

Posted on

OpenSearch can enforce read-only states on clusters or indices to protect against issues like low disk space or cluster instability. Understanding and resolving these blocks is crucial for maintaining a healthy and operational cluster. Below is a guide to address common scenarios.

How to Resolve cluster.blocks.read_only

The cluster.blocks.read_only setting typically occurs when OpenSearch detects a critical issue, such as running out of disk space. The cluster automatically sets itself to read-only to prevent further damage.
Steps to resolve:
  1. Free up disk space:
    • Delete unnecessary indices.
    • Archive old data.
    • Increase disk capacity.
  3. Clear the read-only block: Run the following command to remove the block after resolving the disk space issue:
PUT _cluster/settings
{
  "persistent": {
    "cluster.blocks.read_only": false
  }
}

Index-Level Read-Only Block

At the index level, OpenSearch can set specific indices to read-only, preventing writes. This is often triggered by disk watermarks or manual configuration.
Steps to resolve:
  1. Identify the affected indices using:
GET _cat/indices?v
  2.  Clear the block for an individual index:
PUT /<index-name>/_settings
{
  "index.blocks.read_only": false
}
  3.  If multiple indices are affected, use a wildcard:
PUT /*/_settings
{
  "index.blocks.read_only": false
}

How to Resolve cluster.blocks.read_only_allow_delete

This block allows only delete operations on indices, typically to help free up space when disk watermarks are exceeded.
Steps to resolve:
  1. Free up disk space as described earlier.
  2. Clear the read_only_allow_delete block with:
PUT _cluster/settings
{
  "persistent": {
    "cluster.blocks.read_only_allow_delete": false
  }
}

Index-Level Read-Only Delete Block

When indices are individually set to read_only_allow_delete, write operations are blocked except for deletions.
Steps to resolve:
  1. Check index settings:
GET /<index-name>/_settings
  2.  Remove the block for a specific index:
PUT /<index-name>/_settings
{
  "index.blocks.read_only_allow_delete": false
}
  3.  To remove the block from all indices:
PUT _all/_settings
{
  "index.blocks.read_only_allow_delete": false
}

Conclusion

The post OpenSearch: Cluster Blocks Read-Only appeared first on SOC Prime.

Related Posts