Remote VCenter gets disconnected unexpectedly after it crashed or after being restored, or after client VC's NDU upgrade.

Products

• VMware vCenter Server

Issue/Introduction

This article aims to assist users in recovering the service account credential that gets lost when the vCenter Server crashes, or after doing NDU upgrade.

Symptom 1:
In a Cross vCenter vSAN HCI Mesh, the health check will report a remote VC connectivity issue, and the user cannot perform Datastore source and Datastore management operations, such as destroying the datastore source, mounting or unmounting datastores of the remote VC.

When the client vCenter tries to connect to the server vCenter, the following error may be seen in the below log.

/var/log/vmware/vsan-health/vmware-vsan-health-service-6.log

####-##-##T##:##:##.###Z ERROR vsan-mgmt[19586] [VsanXvcPCManager::RefreshRemotePropertyCollectorInternal opID=7adf5445-W289] Cannot connect to remote VC: <vCenter FQDN>
return self._stub.InvokeMethod(self, info, args)
PyCppVmomi.vim.fault.NotAuthenticated: (vim.fault.NotAuthenticated) {
msg = "Received SOAP response fault from [<SSL(<io_obj p:0x00007f37d00e6860, h:78, <TCP '<IP ADDRESS> : 53792'>, <TCP '<IP ADDRESS> : 443'>>), /vsanHealth>]: createPropertyCollector\n",
object = 'vmodl.query.PropertyCollector:vsan-property-collector',
privilegeId = ''
}

Symptom 2:
For client vCenter after NDU upgrade, the 'Datastore Management' and 'Remote Datastores' info are missing from UI. And the following error may be seen in log file

/var/log/vmware/vsan-health/vmware-vsan-health-service.log:

####-##-##T##:##:##.###Z ERROR vsan-mgmt[70456] [VsanXvcPCManager::RefreshRemotePropertyCollectorInternal opID=23b9510f] Cannot connect to remote VC: <IP ADDRESS>
Traceback (most recent call last):
  File "bora/vsan/clusterconfig/vpxd/pyMoVsan/VsanXvcPCManager.py", line 102, in RefreshRemotePropertyCollectorInternal
  File "bora/vsan/xvc/DSConnMgr.py", line 47, in GetDSConnection
PyCppVmomi.vim.fault.FileFault: (vim.fault.FileFault) {
  msg = '',
  file = 'Failed to open key file /storage/vsan-health/xvchcimesh/xvcEncKey.7.dat'
}

Please refer to the screenshot below:

Environment

VMware vCenter Server 8.0.2
VMware vCenter Server 8.0.1

Cause

For the Symptom 1, when the server vCenter experiences an unexpected crash or is restored from an old backup, the service account credential of the client vCenter may be lost from persistent storage. As a result, the client vCenter may be unable to connect to the server vCenter.

For the Symptom 2, during the client vCenter NDU upgrade process, the key file "xvcEncKey.*.dat" is not copied to the target folder correctly, so when trying to query 'Datastore Management' and 'Remote Datastores', it cannot work well.

Resolution

Currently there is no resolution.

Workaround:

1. SSH to the client vCenter via root
2. Change directory

cd /usr/lib/vmware-vpx/vsan-health/bin

3. To reset the password, run:

./ServiceAccountCtl.sh -r -h '[service vcenter IP]' -u '[email protected]' -p '[service vcenter password]'

4. To verify whether resetting the password worked well, run:

./ServiceAccountCtl.sh -v -h '[service vcenter IP]'

5. To reset and verify the password at the same time, run:

./ServiceAccountCtl.sh -r -v -h '[service vcenter IP]' -u '[email protected]' -p '[service vcenter password]'

Note: Replace [email protected] with your local SSO admin user if the local SSO domain is different.