After upgrading to vIDM 3.3.7 the user dashboard reporting and Active Directory sync log is not updating

Products

• VMware

Issue/Introduction

Symptoms:

• Active Directory sync log is not updating.
• User dashboard is not updating / reporting new user logins 
• vIDM Health Status might show an error with Integrated Components - Error retrieving component status
• vIDM Health Status might show an error with ACS Health - Application Deployment Status - Web Application Status - Error when connecting to the application.
• The analytics-service.log file under /opt/vmware/horizon/workspace/logs shows max shards used:

  Unable to create index: v4_2023-03-04_action - {"root_cause":[{"type":"validation_exception","reason":"Validation Failed: 1: this action would add [10] total shards, but this cluster currently has [1000]/[1000] maximum shards open;"}],"type":"validation_exception","reason":"Validation Failed: 1: this action would add [10] total shards, but this cluster currently has [1000]/[1000] maximum shards open;"} 

Environment

VMware Identity Manager 3.3.x

Cause

Elasticsearch was migrated to OpenSearch in version 3.3.7 which is used to receive and store audit records, sync logs and search data.
During the upgrade OpenSearch is only allocated 1,000 shards by default. 
Max shards need to be increased on each vIDM node (whether using a one node or 3 node cluster)

Resolution

Ensure a proper backup/snapshot is taken on each vIDM prior to performing the action below

1. Login via SSH to each vIDM node and run the following command

curl -X PUT localhost:9200/_cluster/settings -H "Content-Type: application/json" -d '{ "persistent":
{ "cluster.max_shards_per_node": "6500" }
}'

2. Wait 5-10 minutes for shards to re-allocate properly and confirm reports are now being populated correctly.