Dashboard shows Opensearch - Health: RED in vIDM

Products

• VCF Operations/Automation (formerly VMware Aria Suite)

Issue/Introduction

• vIDM system diagnostic dashboard shows failure for opensearch: Health in red status
• The analytics-service.log shows that the maximum number of shards are being used:

YYYY-MM-DD HH:MM:SS GMT ERROR <vIDM FQDN>:analytics (ForkJoinPool-1-worker-x) [-; ######-######-######-######;##.##.##.##.##; ######-######-######-######;-] com.vmware.idm.analytics.elasticsearch.ElasticSearchHelper - Unable to create index: vx_YYYY-MM-DD_action - {"root_cause":[{"type":"illegal_argument_exception","reason":"Validation Failed: 1: this action would add [xx] total shards, but this cluster currently has [3000]/[3000] maximum shards open;"}],"type":"illegal_argument_exception","reason":"Validation Failed: 1: this action would add [xx] total shards, but this cluster currently has [3000]/[3000] maximum shards open;"}

Environment

VMware Identity Manager 3.3.7

Cause

OpenSearch max shard count being exceeded.

Resolution

Login via SSH to each vIDM node and run the following command to increase the OpenSearch max shards count to 6500. A subsequent raise to 8200 may be necessary if the initial error in this KB is still observed after raising to 6500:

curl -X PUT localhost:9200/_cluster/settings -H "Content-Type: application/json" -d '{ "persistent":
{ "cluster.max_shards_per_node": "6500" }
}'

It may take 5-10 minutes for shards to re-allocate properly.

Additional Information

Please reference KB 325880 for more detail.