Unable to create index: v4_2023-03-04_action - {"root_cause":[{"type":"validation_exception","reason":"Validation Failed: 1: this action would add [10] total shards, but this cluster currently has [1000]/[1000] maximum shards open;"}],"type":"validation_exception","reason":"Validation Failed: 1: this action would add [10] total shards, but this cluster currently has [1000]/[1000] maximum shards open;"}
VMware Identity Manager 3.3.7
NB: Ensure a proper VM backup/snapshot is taken of each vIDM node prior to performing the steps below
1. Login via SSH to each vIDM node and run the following command to increase the OpenSearch max shards count to 6500. A subsequent raise to 8200 may be necessary if the initial error in this KB is still observed after raising to 6500:
curl -X PUT localhost:9200/_cluster/settings -H "Content-Type: application/json" -d '{ "persistent":
{ "cluster.max_shards_per_node": "6500" }
}'
It may take 5-10 minutes for shards to re-allocate properly.
2. Monitor the opensearch / shard status on each node with below command. For clustered deployments consisting of multiple VIDM nodes, the status should show Green with 0 Unassigned Shards:
watch curl http://localhost:9200/_cluster/health?pretty=true
3. If the above command status is Red/Yellow with UNASSIGNED shard value more than 0 on a clustered system, issue the following command to delete unassign shards:
On single node deployments status may be Yellow and show unassigns equal to the number of assigned shards, in which case you can skip this step and not delete unassigned shards.
curl -XGET http://localhost:9200/_cat/shards | grep UNASSIGNED | awk {'print $1'} | xargs -i curl -XDELETE "http://localhost:9200/{}"
4. Release locks (once for the cluster is enough - run on psql primary node)
/usr/sbin/hznAdminTool liquibaseOperations -forceReleaseLocks
5. Restart the main vIDM service - first on primary, wait a minute or two, then the other two nodes: service horizon-workspace restart
6. Check and confirm all issues reported above are resolved: reports now being populated correctly.
Max shards count for OpenSearch service will be increased on all nodes.