Exporting Auto Deploy Image task stuck in vCenter
search cancel

Exporting Auto Deploy Image task stuck in vCenter

book

Article ID: 382661

calendar_today

Updated On:

Products

VMware vCenter Server

Issue/Introduction

Auto Deploy fails to register ESXi host and the 'Export image' task is stuck in loop.

ESXi host boot is stuck at /vmw/rbd/host-register.

You may observe similar log snippets.

rbd-cgi.log

YYYY-MM-DDTHH:MM:SS [12782:Thread-2 (process_request_worker)]INFO:director:request -- Request[{'headers_out': {}, 'uri': '/vmw/rbd/tramp', 'environ': {'REQUEST_METHOD': 'GET', 'QUERY_STRING': ''}, 'rfile': <_io.BufferedReader name=18>, 'headers_in': <http.client.HTTPMessage object at 0x7f2799850a60>, 'remote_ip': 'xx.xx.xx.xx', 'request': <socket.socket fd=18, family=AddressFamily.AF_UNIX, type=SocketKind.SOCK_STREAM, proto=0, laddr=/var/run/vmware-rbd/rbd-cgi-pipe>}]
YYYY-MM-DDTHH:MM:SS [12782:Thread-2 (process_request_worker)]INFO:director: - - "GET /vmw/rbd/tramp HTTP/1.1" 200 -
YYYY-MM-DDTHH:MM:SS [12782:Thread-2 (process_request_worker)]INFO:director:request -- Request[{'headers_out': {}, 'uri': '/vmw/rbd/host-register', 'environ': {'REQUEST_METHOD': 'POST', 'QUERY_STRING': 'bootmac=xx.xx.xx.xx.xx.xx', 'CONTENT_TYPE': 'application/x-www-form-urlencoded', 'CONTENT_LENGTH': '962'}, 'rfile': <_io.BufferedReader name=18>, 'headers_in': <http.client.HTTPMessage object at 0x7f2799850eb0>, 'remote_ip': 'xx.xx.xx.xx', 'request': <socket.socket fd=18, family=AddressFamily.AF_UNIX, type=SocketKind.SOCK_STREAM, proto=0, laddr=/var/run/vmware-rbd/rbd-cgi-pipe>}]
YYYY-MM-DDTHH:MM:SS [12782:Thread-2 (process_request_worker)]INFO:director: - - "POST /vmw/rbd/host-register?bootmac=xx.xx.xx.xx.xx HTTP/1.1" 301 -


YYYY-MM-DDTHH:MM:SS [12782:Thread-2 (process_request_worker)]INFO:item_resolver:using rule -- 'rbd.PxeProfile:PxeProfile-22'
YYYY-MM-DDTHH:MM:SS [12782:Thread-2 (process_request_worker)]ERROR:lcmutil:Failed to infer vLCM config
Traceback (most recent call last):
  File "bora/install/vmvisor/autodeploy/site-packages/vmware/rbd/utils/lcmutil.py", line 238, in vlcmEnabledForPxeContext
Exception: PxeProfile not cached

 

rbd-cached.log

YYYY-MM-DDTHH:MM:SS [12783:CacheWorker-1]INFO:cache_inject:cacher responding to injection -- vlcm, https://vcenterlab.fdn.s1.cp.intactfc.com:9087/vum-filedownload/download?file=OFFLINE_BUNDLE_5213d5dc-8427-c177-a731-aaa4515f21d6.zip
YYYY-MM-DDTHH:MM:SS [12783:CacheWorker-2]ERROR:cacher:not enough cache space for item -- CachedItemContext[cacheRequest=('vlcm', 'domain-c44', None, 0, 'YYYY-MM-DDTHH:MM:SS', 0, None, 'YYYY-MM-DDTHH:MM:SS', None, None); cols={'id': 14273, 'plugin': 'vlcm', 'src_name': 'domain-c44', 'src_vc_id': None, 'lease_minutes': 0, 'dst_dir': '7c/29c972c29cd138e22dc91caf891123', 'dl_start_time': datetime.datetime(YYYY, MM, DD, HH, MM, SS)}; itemPaths={'/var/lib/rbd/cache/7c/29c972c29cd138e22dc91caf891123/desired_configuration.json': '/var/lib/rbd/cache/tmp/tmp_l1viali', '/var/lib/rbd/cache/7c/29c972c29cd138e22dc91caf891123/depot.zip': '/var/lib/rbd/cache/tmp/tmpwwgxcc20'}]
YYYY-MM-DDTHH:MM:SS [12783:CacheWorker-2]ERROR:pluginmaster:exception:rbdplugins.vlcm.vmwCacheRequest -- not enough cache space for item
Traceback (most recent call last):
  File "bora/install/vmvisor/autodeploy/site-packages/vmware/rbd/utils/pluginmaster.py", line 236, in _curry
  File "bora/install/vmvisor/autodeploy/var/rbdplugins/vlcm.py", line 149, in vmwCacheRequest
  File "bora/install/vmvisor/autodeploy/var/rbdplugins/vlcm.py", line 123, in _exportClusterImage
  File "bora/install/vmvisor/autodeploy/site-packages/vmware/rbd/waiter/cacher.py", line 230, in downloadUrl
  File "bora/install/vmvisor/autodeploy/site-packages/vmware/rbd/waiter/cacher.py", line 559, in reserveSpace
rbd.waiter.cacher.CacheException: not enough cache space for item

Environment

VMware vCenter Server 7.x
VMware vCenter Server 8.x

Cause

By default, the Auto Deploy Service cachesize_GB is set to 2GB which is increased to 4GB in newer releases.

The cache size is not sufficient for the OFFLINE_BUNDLE which is exported to the ESXi hosts during the register process.

Resolution

Increase the Auto Deploy and Image Builder Service cachesize_GB

  • Log in to the vCenter as local SSO administrator.
  • Navigate to Auto Deploy Menu.
  • Navigate to Configure.
  • Increase the VMware Auto Deploy Service cachesize_GB.
  • Increase the VMware Image Builder Service cachesize_GB.
  • Restart the vmware-rbd-watchdog and vmware-imagebuilder service on the vCenter for the changes to take effect.

Additional Information

Autodeploy and imagebuilder cache can be updated via their respective config files as well.
Service restart is needed for the changes to take effect.

RBD: /etc/vmware-rbd/autodeploy-setup.xml and edit the value for <maxSize> </maxSize>
Imagebuilder: /etc/vmware-imagebuilder/sca-config/imagebuilder-config.props and edit the value cacheSize_GB=

Note: Autodeploy cachesize cannot be more than half of autodeploy partition size.

Powered by Wolken Software