Use the server vCenter's stale info clean tool '/usr/lib/vmware-vpx/vsan-health/bin/CleanStaleInfoFromServer.sh' on the 80u1 product.

Products

• VMware vCenter Server

Issue/Introduction

In Cross vCenter HCI Mesh feature, we introduce a tool '/usr/lib/vmware-vpx/vsan-health/bin/CleanStaleInfoFromServer.sh' on vCenter. It can help clean up the stale client information on the vCenter when it is used as a server vCenter, which has server datastores mounted by some client clusters on client vCenters. when the client vCenter fades unexpectedly and is implacable. The user or GSS needs to use this tool to clean up the stale client information. So far, we don't have such a procedure supported in UI, and we think this is a rare case belonging to the scope of tech support. Such stale client information won't impact other clients in the mounting state, but it will occupy the budget of the clients allowed by the server vCenter/clusters, so we should clean up such stale client information when it happens.

How to manually run this tool, please refer to:
https://opengrok.eng.vmware.com/source/xref/main.perforce.1666/bora/vsan/xvc/util/CleanStaleInfoFromServer.md, this document is also packaged along with the binary tool on the vCenter under the folder '/usr/lib/vmware-vpx/vsan-health'.

Symptoms:

The server vCenter's stale information clean tool '/usr/lib/vmware-vpx/vsan-health/bin/CleanStaleInfoFromServer.sh' which bundled in the 80u1 product,  provides the function list cluster, list vCenter, delete cluster and delete vCenter. But this tool bundled in the  80u1 product, the function of listing cluster and listing vCenter works well. But for the function delete cluster and delete vCenter, it will throw some certificate issues.

The failed logs are as following:

Exceptions:
2023-01-09T11:50:41.216Z [ERROR vsanXvcLib.py::RunCreateDatastoreSource::167::MainThread] CreateDatastoreSource for 10.92.114.12: Got SSL verify fault: -----BEGIN CERTIFICATE-----
....
....
?-----END CERTIFICATE-----

Traceback (most recent call last):
File "/tmp/test-vpx_20230109-2534-1rerzmp/test-vpx-code/bora/vpx/tests/vsan/xvchcimesh/vsanXvcLib.py", line 163, in RunCreateDatastoreSource
vcInfo=tmpVcInfo))

File "/tmp/tmpwukg8ksc.test-vpx/pyVpx/pyVmomi/VmomiSupport.py", line 598, in <lambda>
self.f(*(self.args + (obj,) + args), **kwargs)

File "/tmp/tmpwukg8ksc.test-vpx/pyVpx/pyVmomi/VmomiSupport.py", line 388, in _InvokeMethod
return self._stub.InvokeMethod(self, info, args)

File "/tmp/tmpwukg8ksc.test-vpx/pyVpx/pyVmomi/SoapAdapter.py", line 1575, in InvokeMethod
raise obj # pylint: disable-msg=E0702

pyVmomi.VmomiSupport.vim.fault.VsanSslVerifyCertFault: (vim.fault.VsanSslVerifyCertFault) {...}
As this is a tool the user use directly, so when the user run this tool, this error message will show in the screen directly.

Environment

VMware vCenter Server 8.0
VMware vCenter Server 8.0.1

Resolution

VMware is aware of this issue and working to resolve this in a future release.

Workaround:
As a workaround follow below steps :
 

1. cd to the folder '/usr/lib/vmware-vpx/vsan-health/bin'

2. copy the file https://opengrok.eng.vmware.com/source/xref/main.perforce.1666/bora/vsan/xvc/util/CleanStaleInfoFromServer.py from main branch to server vCenter host.

3. edit the file CleanStaleInfoFromServer.sh, replace the line 8 'subCmd="python $ROOT_DIR/CleanStaleInfoFromServer.pyc $*;"' with 'subCmd="python $ROOT_DIR/CleanStaleInfoFromServer.py $*;"'