vCenter Autodeploy service is slow and unable to manage rules

Products

• VMware vCenter Server

Issue/Introduction

You are unable to load Auto Deploy page on vCenter UI.

Auto Deploy service stops working.

You find log entry similar to this in /var/log/vmware/rbd/rbd-syslog.log

####-##-##T##:##:##.#######+##:## err rbd  [123456]pluginmaster:ERROR:exception:rbdplugins.ruleset.vmwCacheRequest -- timed out
Traceback (most recent call last):
File "bora/install/vmvisor/autodeploy/site-packages/vmware/rbd/utils/pluginmaster.py", line 236, in _curry
File "bora/install/vmvisor/autodeploy/var/rbdplugins/ruleset.py", line 134, in vmwCacheRequest
File "bora/install/vmvisor/autodeploy/site-packages/vmware/rbd/vc_extension.py", line 629, in clientConnect
File "/usr/lib/vmware/site-packages/pyVmomi/VmomiSupport.py", line 612, in __call__
return self.f(*args, **kwargs)
File "/usr/lib/vmware/site-packages/pyVmomi/VmomiSupport.py", line 400, in _InvokeAccessor
return self._stub.InvokeAccessor(self, info)
File "/usr/lib/vmware/site-packages/pyVmomi/StubAdapterAccessorImpl.py", line 44, in InvokeAccessor
return self.InvokeMethod(mo, info, (prop, ))
File "/usr/lib/vmware/site-packages/pyVmomi/SoapAdapter.py", line 1558, in InvokeMethod
resp = conn.getresponse()
File "/usr/lib/python3.10/http/client.py", line 1375, in getresponse
response.begin()
File "/usr/lib/python3.10/http/client.py", line 318, in begin
version, status, reason = self._read_status()
File "/usr/lib/python3.10/http/client.py", line 279, in _read_status
line = str(self.fp.readline(_MAXLINE + 1), "iso-####-#")
File "/usr/lib/python3.10/socket.py", line 705, in readinto
return self._sock.recv_into(b)
TimeoutError: timed out
####-##-##T##:##:##.#######+##:## err rbd  [123456]cacher:ERROR:unable to satisfy cache request -- ('ruleset', 'liveRuleSetList', 1, 0, '####-##-## ##:##:##', 0, None, '####-##-## ##:##:##', None, None)

Environment

vCenter 8.0

Cause

This is due to exhaustion of the rbd-server process connection pool.  

Resolution

VMware engineering is aware of the issue. If you encounter this issue, please contact Broadcom Support.