After upgrading to vIDM 3.3.7, user dashboard reporting and Active Directory sync log are not updating: the cluster currently has the maximum number of shards open

Products

• VMware Aria Suite

Issue/Introduction

Key symptom:

• The analytics-service.log file under /opt/vmware/horizon/workspace/logs shows that the maximum number of shards are being used:

  Unable to create index: v4_2023-03-04_action - {"root_cause":[{"type":"validation_exception","reason":"Validation Failed: 1: this action would add [10] total shards, but this cluster currently has [1000]/[1000] maximum shards open;"}],"type":"validation_exception","reason":"Validation Failed: 1: this action would add [10] total shards, but this cluster currently has [1000]/[1000] maximum shards open;"} 

 

Other possible symptoms:

• Active Directory sync log is not updating: unable to sync the directory with domain users. Only built-in administrators work.
• User dashboard is not updating / reporting new user logins 
• The Reports > Audit Events shows no events
• Searching for users in the vIDM Administration console gives no results
• Dashboard within the Admin UI for all nodes contains elements which are not loading, so it appears blank.
• VIDM system diagnostic dashboard shows failure for opensearch: unassigned shards and yellow/red status
• vIDM Health Status might show an error with Integrated Components - Error retrieving component status
• vIDM Health Status might show an error with ACS Health - Application Deployment Status - Web Application Status - Error when connecting to the application.
• VRLCM shows failure for VIDM health, which is found to be caused by opensearch

 

Environment

VMware Identity Manager 3.3.7

Cause

• Elasticsearch was migrated to OpenSearch in version 3.3.7 which is used to receive and store audit records, sync logs and search data.
• During the upgrade OpenSearch is only allocated 1,000 shards by default.
• Validation failures then occur due to the Elasticsearch/OpenSearch max shard count being exceeded.
• Max shards need to be increased on each vIDM node (whether using a single node or 3 node cluster)

Resolution

NB: Ensure a proper VM backup/snapshot is taken of each vIDM node prior to performing the steps below

 

Resolution:

1. Login via SSH to each vIDM node and run the following command to increase the OpenSearch max shards count to 6500. A subsequent raise to 8200 may be necessary if the initial error in this KB is still observed after raising to 6500:

curl -X PUT localhost:9200/_cluster/settings -H "Content-Type: application/json" -d '{ "persistent":
{ "cluster.max_shards_per_node": "6500" }
}'

It may take 5-10 minutes for shards to re-allocate properly.


2. Monitor the opensearch / shard status on each node with below command. For clustered deployments consisting of multiple VIDM nodes, the status should show Green with 0 Unassigned Shards:

watch curl http://localhost:9200/_cluster/health?pretty=true

3. If the above command status is Red/Yellow with UNASSIGNED shard value more than 0 on a clustered system, issue the following command to delete unassign shards:
On single node deployments status may be Yellow and show unassigns equal to the number of assigned shards, in which case you can skip this step and not delete unassigned shards.

curl -XGET http://localhost:9200/_cat/shards | grep UNASSIGNED | awk {'print $1'} | xargs -i curl -XDELETE "http://localhost:9200/{}"

4.  Release locks (once for the cluster is enough - run on psql primary node)

/usr/sbin/hznAdminTool liquibaseOperations -forceReleaseLocks

5. Restart the main vIDM service - first on primary, wait a minute or two, then the other two nodes:

 service horizon-workspace restart

6. Check and confirm all issues reported above are resolved: reports now being populated correctly. 

Additional Information

Max shards count for OpenSearch service will be increased on all nodes.