Exporting Auto Deploy Image task stuck in vCenter and failed with error inject vib in the cache: b"[CacheException('not enough cache space for item')
search cancel

Exporting Auto Deploy Image task stuck in vCenter and failed with error inject vib in the cache: b"[CacheException('not enough cache space for item')

book

Article ID: 382661

calendar_today

Updated On:

Products

VMware vCenter Server

Issue/Introduction

  • Auto Deploy fails to register ESXi host and the 'Export image' task is stuck in loop.
  • ESXi host boot is stuck at /vmw/rbd/host-register.
  • Error in /var/log/rbd-cgi.log in ESXi host
    [12782:Thread-2 (process_request_worker)]INFO:director:request -- Request[{'headers_out': {}, 'uri': '/vmw/rbd/tramp', 'environ': {'REQUEST_METHOD': 'GET', 'QUERY_STRING': ''}, 'rfile': <_io.BufferedReader name=18>, 'headers_in': <http.client.HTTPMessage object at 0x7f2799850a60>, 'remote_ip': '##.##.##.##', 'request': <socket.socket fd=18, family=AddressFamily.AF_UNIX, type=SocketKind.SOCK_STREAM, proto=0, laddr=/var/run/vmware-rbd/rbd-cgi-pipe>}]
[12782:Thread-2 (process_request_worker)]INFO:director: - - "GET /vmw/rbd/tramp HTTP/1.1" 200 -
[12782:Thread-2 (process_request_worker)]INFO:director:request -- Request[{'headers_out': {}, 'uri': '/vmw/rbd/host-register', 'environ': {'REQUEST_METHOD': 'POST', 'QUERY_STRING': 'bootmac=##.##.##.##', 'CONTENT_TYPE': 'application/x-www-form-urlencoded', 'CONTENT_LENGTH': '962'}, 'rfile': <_io.BufferedReader name=18>, 'headers_in': <http.client.HTTPMessage object at 0x7f2799850eb0>, 'remote_ip': '##.##.##.##', 'request': <socket.socket fd=18, family=AddressFamily.AF_UNIX, type=SocketKind.SOCK_STREAM, proto=0, laddr=/var/run/vmware-rbd/rbd-cgi-pipe>}]
[12782:Thread-2 (process_request_worker)]INFO:director: - - "POST /vmw/rbd/host-register?bootmac=##.##.##.## HTTP/1.1" 301 -


[12782:Thread-2 (process_request_worker)]INFO:item_resolver:using rule -- 'rbd.PxeProfile:PxeProfile-22'
[12782:Thread-2 (process_request_worker)]ERROR:lcmutil:Failed to infer vLCM config
Traceback (most recent call last):
  File "bora/install/vmvisor/autodeploy/site-packages/vmware/rbd/utils/lcmutil.py", line 238, in vlcmEnabledForPxeContext
Exception: PxeProfile not cached
  • Error in /var/log/rbd-cached.log in ESXi host
    [12783:CacheWorker-1]INFO:cache_inject:cacher responding to injection -- vlcm, https://vcenter.example.com:9087/vum-filedownload/download?file=OFFLINE_BUNDLE_5213d5dc-8427-c177-a731-aaa4515f21d6.zip
[12783:CacheWorker-2]ERROR:cacher:not enough cache space for item -- CachedItemContext[cacheRequest=('vlcm', 'domain-c44', None, 0, 'YYYY-MM-DDTHH:MM:SS', 0, None, 'YYYY-MM-DDTHH:MM:SS', None, None); cols={'id': 14273, 'plugin': 'vlcm', 'src_name': 'domain-c44', 'src_vc_id': None, 'lease_minutes': 0, 'dst_dir': '7c/29c9####################891123', 'dl_start_time': datetime.datetime(YYYY, MM, DD, HH, MM, SS)}; itemPaths={'/var/lib/rbd/cache/7c/29c9####################891123/desired_configuration.json': '/var/lib/rbd/cache/tmp/tmp_l1viali', '/var/lib/rbd/cache/7c/29c9####################891123/depot.zip': '/var/lib/rbd/cache/tmp/tm####20'}]
[12783:CacheWorker-2]ERROR:pluginmaster:exception:rbdplugins.vlcm.vmwCacheRequest -- not enough cache space for item
Traceback (most recent call last):
  File "bora/install/vmvisor/autodeploy/site-packages/vmware/rbd/utils/pluginmaster.py", line 236, in _curry
  File "bora/install/vmvisor/autodeploy/var/rbdplugins/vlcm.py", line 149, in vmwCacheRequest
  File "bora/install/vmvisor/autodeploy/var/rbdplugins/vlcm.py", line 123, in _exportClusterImage
  File "bora/install/vmvisor/autodeploy/site-packages/vmware/rbd/waiter/cacher.py", line 230, in downloadUrl
  File "bora/install/vmvisor/autodeploy/site-packages/vmware/rbd/waiter/cacher.py", line 559, in reserveSpace
rbd.waiter.cacher.CacheException: not enough cache space for item

Environment

VMware vCenter Server 7.x
VMware vCenter Server 8.x

Cause

By default, the Auto Deploy Service cachesize_GB is set to 2GB which is increased to 4GB in newer releases.

The cache size is not sufficient for the OFFLINE_BUNDLE which is exported to the ESXi hosts during the register process.

Resolution

Increase the Auto Deploy and Image Builder Service cachesize_GB

    1. Log in to the vCenter as local SSO administrator.
    2. Navigate to Auto Deploy Menu.
    3. Navigate to Configure.
    4. Increase the VMware Auto Deploy Service cachesize_GB.
    5. Increase the VMware Image Builder Service cachesize_GB and confirm the updated value in "/etc/vmware-rbd/autodeploy-setup.xml and /etc/vmware-imagebuilder/sca-config/imagebuilder-config.props"
    6. Restart the vmware-rbd-watchdog and vmware-imagebuilder service on the vCenter for the changes to take effect.

service-control --restart vmware-rbd-watchdog

service-control --restart vmware-imagebuilder

Additional Information

Autodeploy and imagebuilder cache can be updated via their respective config files as well.
Service restart is needed for the changes to take effect.

RBD: /etc/vmware-rbd/autodeploy-setup.xml and edit the value for <maxSize> </maxSize>
Imagebuilder: /etc/vmware-imagebuilder/sca-config/imagebuilder-config.props and edit the value cacheSize_GB=

Note: Autodeploy cachesize cannot be more than half of autodeploy partition size.

Powered by Wolken Software