User dashboard reporting and Active Directory sync logs do not update in VMware Identity Manager

Products

VCF Operations/Automation (formerly VMware Aria Suite)

Issue/Introduction

The analytics-service.log file under /opt/vmware/horizon/workspace/logs shows that the maximum number of shards are being used:

Unable to create index: v4_2023-03-04_action - {"root_cause":[{"type":"validation_exception","reason":"Validation Failed: 1: this action would add [10] total shards, but this cluster currently has [1000]/[1000] maximum shards open;"}],"type":"validation_exception","reason":"Validation Failed: 1: this action would add [10] total shards, but this cluster currently has [1000]/[1000] maximum shards open;"}

It was also noticed:

Active Directory sync log is not updating: unable to sync the directory with domain users. Only built-in administrators work. You may receive an error similar to You don't have permission to perform the operation when syncing a directory.
User dashboard is not updating / reporting new user logins.
The Reports > Audit Events shows no events.
Searching for users in the vIDM Administration console gives no results.
Dashboard within the Admin UI for all nodes contains elements which are not loading, so it appears blank.
VIDM system diagnostic dashboard shows failure for opensearch: unassigned shards and yellow/red status.
vIDM Health Status might show an error with Integrated Components - Error retrieving component status.
vIDM Health Status might show an error with ACS Health - Application Deployment Status - Web Application Status - Error when connecting to the application.
VRLCM shows failure for VIDM health, which is found to be caused by opensearch.
After providing valid Active Directory binding account credentials, you might encounter an error:
```
Connector communication failed with response: Circuit breaker...
```

Environment

VMware Identity Manager (vIDM) 3.3.7

Cause

In version 3.3.7, Elasticsearch was migrated to OpenSearch. By default, the migration allocates 1,000 shards per node. In active environments, this limit is frequently exceeded, causing validation failures during index creation.

Resolution

Note: Ensure a proper VM backup/snapshot is taken of each vIDM node prior to performing the steps below.

To resolve the issue, follow the steps below:

Login via SSH to each vIDM node and run the following command to increase the OpenSearch max shards count to 6500. A subsequent raise to 8200 may be necessary if the initial error in this KB is still observed after raising to 6500:
```
curl -X PUT localhost:9200/_cluster/settings -H "Content-Type: application/json" -d '{ "persistent":
{ "cluster.max_shards_per_node": "6500" }
}'
```
It may take 5-10 minutes for shards to re-allocate properly.
Monitor the opensearch / shard status on each node with the below command. For clustered deployments consisting of multiple VIDM nodes, the status should show Green with 0 Unassigned Shards:
```
watch curl http://localhost:9200/_cluster/health?pretty=true
```
If the above command status is Red/Yellow with an UNASSIGNED shard value more than 0 on a clustered system, issue the following command to delete unassigned shards.

On single node deployments, the status may be Yellow and show unassigned equal to the number of assigned shards, in which case you can skip this step and not delete unassigned shards.
```
curl -XGET http://localhost:9200/_cat/shards | grep UNASSIGNED | awk {'print $1'} | xargs -i curl -XDELETE "http://localhost:9200/{}"
```

follow by reindexing database

curl -X PUT localhost:9200/_cluster/settings -H "Content-Type: application/json" -d '{ "persistent": { "cluster.max_shards_per_node": "6500" } }'

Important Note:

Health Status:

curl http://localhost:9200/_cluster/health?pretty=true

Green: Everything is good, there are enough nodes in the cluster to ensure at least 2 full copies of the data spread across the cluster.
Yellow: Functioning, but there are not enough nodes in the cluster to ensure HA (e.g., a single node cluster will always be in the yellow state because it can never have 2 copies of the data).
- For single node: Elasticsearch/OpenSearch will be yellow for a single node by default as it doesn't have a cluster. For a single node, this is expected and it should not be a problem if there are no issues in functionality.
Red: Broken, unable to query existing data or store new data, typically due to not enough nodes in the cluster to function or being out of disk space.

Release locks (once for the cluster is enough - run on the psql primary node):
```
/usr/sbin/hznAdminTool liquibaseOperations -forceReleaseLocks
```
Restart the main vIDM service - first on the primary, wait a minute or two, then the other two nodes:
```
service horizon-workspace restart
```
Check and confirm all issues reported above are resolved: reports are now being populated correctly.