VMware Identity Manager Connector may fail to communicate due to config-state.json corruption

Products

VCF Operations/Automation (formerly VMware Aria Suite)

Issue/Introduction

Corrupted config-state.json file results in Active Directory login failures
Sync connector fails with error - "Failed to parse response received from connector".
User receives error "400 bad request - invalid username / password" when using Aria Automation API to authenticate as a domain user
Connector Failed to load Auth Adapters | Auth Adapters are missing from VMware Identity Manager node as per this KB 322714
Aria Suite Lifecycle Manager shows the globalenvironment environment as healthy
Connector sync issue reported when checking health in VMware Identity Manager
The UI will not load and the URL will be redirected to https://<vIDMFQDN>/hc/error where it may show an error similar to Error: You do not have permission to access this page: /hc/####/authenticate/ or Error : See logs for details

Log files connector.log and horizon.log in /opt/vmware/horizon/workspace/logs/ show the issue as:

com.vmware.horizon.connector.utils.ServiceUtils - Failed to check service health: invalid MOL url

ERROR (pool-4-thread-1) [;;;] com.vmware.horizon.connector.admin.service.impl.StartupDirectoryConfigService - could not take the backup as config-state.json file size is empty

Environment

VMware Identity Manager 3.3.x
Aria Suite Lifecyle Manager 8.x
Aria Automation 8.x

Cause

The config-state.json file stores connector configurations. File system instability, disk space exhaustion, or interrupted write processes can lead to corruption, characterized by missing mandatory values such as in the idp section where null values are present instead.

Resolution

Prerequisites:

Before proceeding, follow the steps below:
Take a snapshot of the VMware Identity Manager Appliance(s).
1. If the UI will not load, and users are observing the /hc/error as shown in the introduction section above, verify the Bind User/Password of the directory are correct.
2. If a service account is being used in the configuration, its password may have expired.
3. Navigate to https://<vIDM_NODE_FQDN>/SAAS/login/0.
4. Test and Save the directory configuration once the password has been reset.
5. Skip ahead to step 8 below after receiving a successful Test and Save.
If the root partition is full, you first need to release disk space as detailed in KB 376193: Identity Manager root partition is full on one or more nodes in the cluster.

IMPORTANT:

The config-state.json file is located in:

/usr/local/horizon/conf/states/<TENANT_NAME>/<WORKER_ID>/

If there are multiple directories in vIDM, there will be a directory for each <WORKER_ID>.
The config-state.json in each <WORKER_ID> directory has to be checked.
Any affected file needs to be replaced with the latest stable version within its own directory.

Recovery:

SSH to each VMware Identity Manager Appliance(s) as the root user, change directory into the tenant directory, and list all worker id sub-directories and their content:
```
cd /usr/local/horizon/conf/states/<TENANT_NAME>/
ls -lR
```
Review this result to investigate how many files are corrupted, or need to be restored from backup, and their respective worker-id sub-directories.
Stop the horizon-workspace service only on the node where the config-state.json file is being recovered:
```
service horizon-workspace stop
```
If the current configuration file is not 0K in size, make a copy of it with the following command:
```
cp config-state.json config-state.json.1
```
Replace the current configuration file with a copy of the latest stable backup (if multiple versions are present, e.g. _v1, _v2, _v2.1, _v2.2, then use _v2):
```
cp -p config-state.json.backup_<latest-stable-version> config-state.json
```
Confirm the owner and group of the config-state.json is unchanged (horizon:www); if not, correct it with the following command:
```
chown horizon:www /usr/local/horizon/conf/states/<TENANT_NAME>/<WORKER_ID>/config-state.json
```
Confirm the permissions of the config-state.json are unchanged; if not, correct them with the following command:
```
chmod 640 /usr/local/horizon/conf/states/<TENANT_NAME>/<WORKER_ID>/config-state.json
```
NOTE:
- If the config-state.json file does not get updated with directory information even after restoring from the older versions, recreate the affected directory from the UI.
  - Navigate to Identity & Access Management > YOUR DIRECTORY
  - Make sure you copy all bind and sync settings values of the directory as those will be needed during recreation
  - Finally delete, and re-create it
Start the horizon-workspace service by running the following command:
```
service horizon-workspace start
```
In the UI, navigate to Identity & Access Management > YOUR DIRECTORY > Sync Settings, then run through each tab and save the setting:
Above, if the "Groups" tab refuses to save due to an error regarding the Bind DN: return to the directory settings page, enter the Bind DN password, validate & save, then come back to save the remaining tabs in Sync Settings.

To avoid the issue from re-occurring, please apply KB 376193: vIDM root partition is full on one node of the cluster.

```

Additional Information

Impact/Risks:

The config-state.json has been corrupted and needs to be restored. If not restored, the connectors are not going to work in the existing condition
Also Please see the following KB (396017) for the same overall error but for IDP configuration value is null in the vIDM node’s config-state.json file