After upgrading to vIDM 3.3.7, user dashboard reporting and Active Directory sync log are not updating: the cluster currently has the maximum number of shards open

Products

• VMware Aria Suite

Issue/Introduction

Key symptom:

• The analytics-service.log file under /opt/vmware/horizon/workspace/logs shows that the maximum number of shards are being used:

  Unable to create index: v4_2023-03-04_action - {"root_cause":[{"type":"validation_exception","reason":"Validation Failed: 1: this action would add [10] total shards, but this cluster currently has [1000]/[1000] maximum shards open;"}],"type":"validation_exception","reason":"Validation Failed: 1: this action would add [10] total shards, but this cluster currently has [1000]/[1000] maximum shards open;"} 

 

Other possible symptoms:

• Active Directory sync log is not updating: unable to sync the directory with domain users. Only built-in administrators work.
  • You may receive an error similar to 'You don't have permission to perform the operation' when syncing a directory.
• User dashboard is not updating / reporting new user logins 
• The Reports > Audit Events shows no events
• Searching for users in the vIDM Administration console gives no results
• Dashboard within the Admin UI for all nodes contains elements which are not loading, so it appears blank.
• VIDM system diagnostic dashboard shows failure for opensearch: unassigned shards and yellow/red status
• vIDM Health Status might show an error with Integrated Components - Error retrieving component status
• vIDM Health Status might show an error with ACS Health - Application Deployment Status - Web Application Status - Error when connecting to the application.
• VRLCM shows failure for VIDM health, which is found to be caused by opensearch

 

Environment

VMware Identity Manager 3.3.7

Cause

• Elasticsearch was migrated to OpenSearch in version 3.3.7 which is used to receive and store audit records, sync logs and search data.
• During the upgrade OpenSearch is only allocated 1,000 shards by default.
• Validation failures then occur due to the Elasticsearch/OpenSearch max shard count being exceeded.
• Max shards need to be increased on each vIDM node (whether using a single node or 3 node cluster)

Resolution

NB: Ensure a proper VM backup/snapshot is taken of each vIDM node prior to performing the steps below

 

Resolution:

1. Login via SSH to each vIDM node and run the following command to increase the OpenSearch max shards count to 6500. A subsequent raise to 8200 may be necessary if the initial error in this KB is still observed after raising to 6500:

curl -X PUT localhost:9200/_cluster/settings -H "Content-Type: application/json" -d '{ "persistent":
{ "cluster.max_shards_per_node": "6500" }
}'

It may take 5-10 minutes for shards to re-allocate properly.


2. Monitor the opensearch / shard status on each node with below command. For clustered deployments consisting of multiple VIDM nodes, the status should show Green with 0 Unassigned Shards:

watch curl http://localhost:9200/_cluster/health?pretty=true

3. If the above command status is Red/Yellow with UNASSIGNED shard value more than 0 on a clustered system, issue the following command to delete unassign shards:
On single node deployments status may be Yellow and show unassigns equal to the number of assigned shards, in which case you can skip this step and not delete unassigned shards.

curl -XGET http://localhost:9200/_cat/shards | grep UNASSIGNED | awk {'print $1'} | xargs -i curl -XDELETE "http://localhost:9200/{}"

 

Important Note:

Health Status:
curl http://localhost:9200/_cluster/health?pretty=true 

Green: everything is good, there are enough nodes in the cluster to ensure at least 2 full copies of the data spread across the cluster.

Yellow: functioning, but there are not enough nodes in the cluster to ensure HA (eg, a single node cluster will always be in the yellow state because it can never have 2 copies of the data).
*for single Node - Elasticsearch/Opensearch will be yellow for a single node by default as it doesn't have a cluster. for single node its expected and it should not be a problem, if facing no issue in functionality.

Red: broken, unable to query existing data or store new data, typically due to not enough nodes in the cluster to function or out of disk space.


4.  Release locks (once for the cluster is enough - run on psql primary node)

/usr/sbin/hznAdminTool liquibaseOperations -forceReleaseLocks

5. Restart the main vIDM service - first on primary, wait a minute or two, then the other two nodes:

 service horizon-workspace restart

6. Check and confirm all issues reported above are resolved: reports now being populated correctly. 

Additional Information

Max shards count for OpenSearch service will be increased on all nodes.