Directory synchronization fails with error 'Connector Communication failed with Response'

Products

VCF Operations/Automation (formerly VMware Aria Suite)

Issue/Introduction

This article provides resolution when the config-state.json has corrupted or become blank or reset with default values.

Symptoms:

Running the Directory sync in vRealize Automation 7.x fails.
In vRealize Automation 7.x UI, you see error:
```
Connector Communication failed with Response
```
VIDM Users are unable to sign in to Aria Automation and receive the following error on login : Error See logs for details.
Testing connection in VIDM in Administration Console > Identity & Access Management > Select appropriate directory under Directory Name > Enter Bind User Password > Test connection fails with the error: Connector communication failed with response: for the connector
Running the Directory sync in VIDM Administration Console > Identity & Access Management > Select appropriate directory under Directory Name > Sync now fails
The config-state.json file located in /usr/local/horizon/conf/states/<TENANT_NAME>/<TENANT_ID> for one or more tenants has 0 byte file size or has default values

In the /storage/log/vmware/horizon/connector.log file of vRealize Automation, you see entries similar to:

ERROR (tomcat-http--14) [;;] com.vmware.horizon.common.api.token.SuiteToken - No keystore file or URL specified.
INFO (tomcat-http--14) [;;] com.vmware.horizon.common.api.token.SuiteToken - Suite token failed to initialize.
WARN (tomcat-http--14) [3002@#####;-;127.0.0.1] com.vmware.horizon.common.api.token.SuiteToken - SuiteToken revocation check failed. The SuiteTokenConfiguration.getRevokeCheckUrl was not set.
INFO (tomcat-http--14) [3002@#####;-;127.0.0.1] com.vmware.horizon.common.api.token.SuiteToken - Initializing keyStore for SuiteToken.
ERROR (tomcat-http--14) [3002@####;-;127.0.0.1] com.vmware.horizon.common.api.token.SuiteToken - No keystore file or URL specified.
INFO (tomcat-http--14) [3002@#####;-;127.0.0.1] com.vmware.horizon.common.api.token.SuiteToken - Suite token failed to initialize.
INFO (tomcat-http--14) [3002@######;-;127.0.0.1] com.vmware.horizon.connector.mvc.RestControllerInterceptor - Invalid suite token.

Note: The preceding log excerpts are only examples. Date, time, and environmental variables may vary depending on your environment.

Environment

VMware Identity Manager 3.3.7
VMware vRealize Automation

Cause

This issue occurs if the config-state.json file located in /usr/local/horizon/conf/states/<TENANT_NAME>/<TENANT_ID> has become corrupted, been emptied to a zero byte file, or has been reset with default values.
Example Path: /usr/local/horizon/conf/states/VSPHERE.LOCAL/3001/config-state.json

Resolution

Verify the primary tenant (VSPHERE.LOCAL) config-state.json for corruption even though the issue is observed in some secondary tenants.

Corruption can be confirmed If the files is of zero size or the data it contains is similar to:
{
  "isConfigured" : false,
  "version" : 15,
  "mol" : {
    "isConfigured" : null,
    "url" : null,
    "tenantId" : null,
    "clientId" : null,
    "clientSecret" : null,
    "metaData" : null,
......

Caution: This resolution is applicable only if config-state.json file has corrupted or become blank or reset with null values. Check all the tenants and ensure to follow this KB steps for all affected tenants.

To resolve the issue:

Take a snapshot of the vRealize Automation Appliance.
SSH to vRealize Automation Appliance using root credentials.
Change directory to the location of the config-state.json file by running the command:

cd /usr/local/horizon/conf/states/<TENANT_NAME>/<TENANT_ID>

For Example: cd /usr/local/horizon/conf/states/VSPHERE.LOCAL/3001
Back up current configuration file by running the command:

mv config-state.json config-state.json.1
Copy application backup of the configuration file by running the command:

cp -p config-state.json.backup_v1 config-state.json
Change the owner of the config-state.json to horizon user by running the command:

chown horizon:www /usr/local/horizon/conf/states/VSPHERE.LOCAL/3001/config-state.json
Change the permission of the config-state.json file by running the command:

chmod 640 /usr/local/horizon/conf/states/VSPHERE.LOCAL/3001/config-state.json
Restart vIDM/Workspace service by running the command:

service horizon-workspace restart
If the revert of file works, backup the backup copy:

cp config-state.json.backup_v1 config-state.json.backup_v2

Connector comes up after performing the above steps but the authentication for ad user using this connector are still fail.
The reason being the idp information might be lost as the connector was retrieved from the backup and the backup may not having the directory and idp information.

in the config-state.json file, you see the entries similar to:
  "idp" : {
    "isConfigured" : false,
    "host" : null,
    "tenantId" : null,
    "id" : null,
    "name" : null,
    "cert" : null,
    "key" : null
  }

To resolve this, Re-create the directory
1. login from local tenant admin.
2. Delete the directory.
3. Create the directory again.
  
  This will re-configure the idp for this connector.
In case of vRA, if the tenant admin configured for ad users are unable to access the director and getting the error User is not authorized to perform the task.

When a user or a group is removed from vIDM (either by deleting the directory or simply excluded from the sync for any reason) and then re-added again, vIDM considers it a different user/group. The vRA, however, is retaining the permissions for that user/group as vRA is identifying them by different means. The result is that the user seems to have all the permissions in vRA but is not able to perform any actions in vIDM.

To resolve this issue, see Tenant administrator loses permissions to manage items under Directories Management in vRealize Automation 7.x (2143798).