The VCSA vSphere AutoDeploy service is experiencing slowness
search cancel

The VCSA vSphere AutoDeploy service is experiencing slowness

book

Article ID: 394999

calendar_today

Updated On:

Products

VMware vCenter Server VMware vCenter Server 8.0

Issue/Introduction

  • The VCSA vSphere AutoDeploy service is experiencing performance issues, leading to delays and issues with AutoDeploy operations in the UI pane.
  • Accessing the Auto Deploy section of the vSphere UI takes a considerable amount of time to load, particularly after making changes to any of the rules.
  • When attempting to activate or deactivate an AutoDeploy rule for further editing, users encounter the following error in the UI -  
    Operation failed! Live rule set not cached yet, please retry in a few seconds
  • Efforts to clear the Auto Deploy database and cache, as well as increase the Auto Deploy runtime cache size, do not resolve the issue.
  • grep -i "exception:rbdplugins.plainpxe.vmwCacheRequest -- timed out" /var/log/vmware/rbd/rbd-syslog.log | cat  
    yyyy-mm-ddThh:mm:ss.Z err rbd  [17904]pluginmaster:ERROR:exception:rbdplugins.plainpxe.vmwCacheRequest -- timed out
yyyy-mm-ddThh:mm:ss.Z err rbd  [17904]pluginmaster:ERROR:exception:rbdplugins.plainpxe.vmwCacheRequest -- timed out
yyyy-mm-ddThh:mm:ss.Z err rbd  [17904]pluginmaster:ERROR:exception:rbdplugins.plainpxe.vmwCacheRequest -- timed out
yyyy-mm-ddThh:mm:ss.Z err rbd  [126714]pluginmaster:ERROR:exception:rbdplugins.plainpxe.vmwCacheRequest -- timed out
yyyy-mm-ddThh:mm:ss.Z err rbd  [126714]pluginmaster:ERROR:exception:rbdplugins.plainpxe.vmwCacheRequest -- timed out
yyyy-mm-ddThh:mm:ss.Z err rbd  [1766587]pluginmaster:ERROR:exception:rbdplugins.plainpxe.vmwCacheRequest -- timed out
yyyy-mm-ddThh:mm:ss.Z err rbd  [1766587]pluginmaster:ERROR:exception:rbdplugins.plainpxe.vmwCacheRequest -- timed out
yyyy-mm-ddThh:mm:ss.Z err rbd  [1766587]pluginmaster:ERROR:exception:rbdplugins.plainpxe.vmwCacheRequest -- timed out
yyyy-mm-ddThh:mm:ss.Z err rbd  [1766587]pluginmaster:ERROR:exception:rbdplugins.plainpxe.vmwCacheRequest -- timed out
yyyy-mm-ddThh:mm:ss.Z err rbd  [16806]pluginmaster:ERROR:exception:rbdplugins.plainpxe.vmwCacheRequest -- timed out
yyyy-mm-ddThh:mm:ss.Z err rbd  [16806]pluginmaster:ERROR:exception:rbdplugins.plainpxe.vmwCacheRequest -- timed out
yyyy-mm-ddThh:mm:ss.Z err rbd  [290657]pluginmaster:ERROR:exception:rbdplugins.plainpxe.vmwCacheRequest -- timed out

 

  • The entries in the rbd-syslog.log are as follows -

    less var/log/vmware/rbd/rbd-syslog
    yyyy-mm-ddThh:mm:ss.Z err rbd  [2395781]pluginmaster:ERROR:exception:rbdplugins.ruleset.vmwCacheRequest -- timed out
Traceback (most recent call last):
File "bora/install/vmvisor/autodeploy/site-packages/vmware/rbd/utils/pluginmaster.py", line 236, in _curry
File "bora/install/vmvisor/autodeploy/var/rbdplugins/ruleset.py", line 134, in vmwCacheRequest
File "bora/install/vmvisor/autodeploy/site-packages/vmware/rbd/vc_extension.py", line 629, in clientConnect
File "/usr/lib/vmware/site-packages/pyVmomi/VmomiSupport.py", line 612, in __call__
return self.f(*args, **kwargs)
File "/usr/lib/vmware/site-packages/pyVmomi/VmomiSupport.py", line 400, in _InvokeAccessor
return self._stub.InvokeAccessor(self, info)
File "/usr/lib/vmware/site-packages/pyVmomi/StubAdapterAccessorImpl.py", line 44, in InvokeAccessor
return self.InvokeMethod(mo, info, (prop, ))
File "/usr/lib/vmware/site-packages/pyVmomi/SoapAdapter.py", line 1558, in InvokeMethod
resp = conn.getresponse()
File "/usr/lib/python3.10/http/client.py", line 1375, in getresponse
response.begin()
File "/usr/lib/python3.10/http/client.py", line 318, in begin
version, status, reason = self._read_status()
File "/usr/lib/python3.10/http/client.py", line 279, in _read_status
line = str(self.fp.readline(_MAXLINE + 1), "iso-8859-1")
File "/usr/lib/python3.10/socket.py", line 705, in readinto
return self._sock.recv_into(b)
TimeoutError: timed out
yyyy-mm-ddThh:mm:ss.Z err rbd  [2395781]cacher:ERROR:unable to satisfy cache request -- ('ruleset', 'liveRuleSetList', 1, 0, 'yyyy-mm-dd hh:mm:ss', 0, None, 'yyyy-mm-dd hh:mm:ss', None, None)
yyyy-mm-ddThh:mm:ss.Z err rbd  [2395781]pluginmaster:ERROR:exception:rbdplugins.plainpxe.vmwCacheRequest -- timed out
Traceback (most recent call last):
File "bora/install/vmvisor/autodeploy/site-packages/vmware/rbd/utils/pluginmaster.py", line 236, in _curry
File "bora/install/vmvisor/autodeploy/var/rbdplugins/plainpxe.py", line 782, in vmwCacheRequest
File "bora/install/vmvisor/autodeploy/site-packages/vmware/rbd/vc_extension.py", line 629, in clientConnect
File "/usr/lib/vmware/site-packages/pyVmomi/VmomiSupport.py", line 612, in __call__
return self.f(*args, **kwargs)
File "/usr/lib/vmware/site-packages/pyVmomi/VmomiSupport.py", line 400, in _InvokeAccessor
return self._stub.InvokeAccessor(self, info)
File "/usr/lib/vmware/site-packages/pyVmomi/StubAdapterAccessorImpl.py", line 44, in InvokeAccessor
return self.InvokeMethod(mo, info, (prop, ))
File "/usr/lib/vmware/site-packages/pyVmomi/SoapAdapter.py", line 1558, in InvokeMethod
resp = conn.getresponse()
File "/usr/lib/python3.10/http/client.py", line 1375, in getresponse
response.begin()
File "/usr/lib/python3.10/http/client.py", line 318, in begin
version, status, reason = self._read_status()
File "/usr/lib/python3.10/http/client.py", line 279, in _read_status
line = str(self.fp.readline(_MAXLINE + 1), "iso-8859-1")
File "/usr/lib/python3.10/socket.py", line 705, in readinto
return self._sock.recv_into(b)
TimeoutError: timed out

 

Environment

VMware vCenter Server 8.0.x

Cause

  • The issue was identified when the rbd-server process failed to respond to the rbd-cgi process during a request for the liveRuleSet, a local property that should always be accessible to the rbd-server.
  • This indicates that the problem lies with the connection establishment process, rather than the availability of the property itself.

Resolution

Shared below are the steps to be taken to resolve the AutoDeploy slowness issue -

  1. Create a backup of the vCenter VM (e.g., take a snapshot of the VM, preferably in a stopped state). If the snapshot is taken while the VM is running, ensure the memory is included.
  2. Stop the AutoDeploy service by issuing the following command through an SSH connection to vCenter in a bash shell -  
    service-control --stop rbd

  3. Download the file provided in the attachment. This attached file (rhttpproxy.json.in) must be used to replace the existing /usr/share/vmware-rbd/rhttpproxy.json.in file.

    To perform the replacement, use the following commands (assuming the attached file has been uploaded to the VCSA /root folder and you are logged in as root): 

    mv /usr/share/vmware-rbd/rhttpproxy.json.in /usr/share/vmware-rbd/rhttpproxy.json.in.org 
mv rhttpproxy.json.in /usr/share/vmware-rbd/
  4. Change the ownership of the new file to "deploy:deploy" with the following command -  
    chown deploy:deploy /usr/share/vmware-rbd/rhttpproxy.json.in
  5. Set the file as read-only for all with the following command  
     chmod 444 /usr/share/vmware-rbd/rhttpproxy.json.in
  6. Verify that the correct files (with the correct attributes) are present in the /usr/share/vmware-rbd/ folder by running 
    ls -al /usr/share/vmware-rbd/
Expected result - 

total 28
drwxr-xr-x  2 root   root   4096 Mar 30 13:41 .
drwxr-xr-x 54 root   root   4096 Mar 25 08:45 ..
-r--r--r--  1 deploy deploy 8625 Mar 30  2025 rhttpproxy.json.in
-r--r--r--  1 deploy deploy 6328 Mar 26 17:21 rhttpproxy.json.in.org
     
  7. Move the autodeploy-proxy.conf file from /etc/vmware-rhttpproxy/endpoints.conf.d/ to the /root folder - 
     mv /etc/vmware-rhttpproxy/endpoints.conf.d/autodeploy-proxy.conf ./
     
  8. Ensure there is no file named autodeploy-proxy.conf left in the /etc/vmware-rhttpproxy/endpoints.conf.d/ folder by running - 
    ls /etc/vmware-rhttpproxy/endpoints.conf.d/autodeploy-proxy.conf

ls: cannot access '/etc/vmware-rhttpproxy/endpoints.conf.d/autodeploy-proxy.conf': No such file or directory
  9. Start the AutoDeploy service with the following command - 
    service-control --start rbd
  10. Verify that the AutoDeploy UI is functioning properly by ensuring that all tabs display as expected.
  11. Check that only one connection is open (the LISTENING one) on the rbd-server-pipe by running -  
    netstat -anp | grep rbd-server-pipe | wc -l

1

Attachments

rhttpproxy.json.in get_app
Powered by Wolken Software