Auto Deploy fails to register ESXi host and the 'Export image' task is stuck in loop.
ESXi host boot is stuck at /vmw/rbd/host-register.
You may observe similar log snippets.
rbd-cgi.log
YYYY-MM-DDTHH:MM:SS [12782:Thread-2 (process_request_worker)]INFO:director:request -- Request[{'headers_out': {}, 'uri': '/vmw/rbd/tramp', 'environ': {'REQUEST_METHOD': 'GET', 'QUERY_STRING': ''}, 'rfile': <_io.BufferedReader name=18>, 'headers_in': <http.client.HTTPMessage object at 0x7f2799850a60>, 'remote_ip': 'xx.xx.xx.xx', 'request': <socket.socket fd=18, family=AddressFamily.AF_UNIX, type=SocketKind.SOCK_STREAM, proto=0, laddr=/var/run/vmware-rbd/rbd-cgi-pipe>}]
YYYY-MM-DDTHH:MM:SS [12782:Thread-2 (process_request_worker)]INFO:director: - - "GET /vmw/rbd/tramp HTTP/1.1" 200 -
YYYY-MM-DDTHH:MM:SS [12782:Thread-2 (process_request_worker)]INFO:director:request -- Request[{'headers_out': {}, 'uri': '/vmw/rbd/host-register', 'environ': {'REQUEST_METHOD': 'POST', 'QUERY_STRING': 'bootmac=xx.xx.xx.xx.xx.xx', 'CONTENT_TYPE': 'application/x-www-form-urlencoded', 'CONTENT_LENGTH': '962'}, 'rfile': <_io.BufferedReader name=18>, 'headers_in': <http.client.HTTPMessage object at 0x7f2799850eb0>, 'remote_ip': 'xx.xx.xx.xx', 'request': <socket.socket fd=18, family=AddressFamily.AF_UNIX, type=SocketKind.SOCK_STREAM, proto=0, laddr=/var/run/vmware-rbd/rbd-cgi-pipe>}]
YYYY-MM-DDTHH:MM:SS [12782:Thread-2 (process_request_worker)]INFO:director: - - "POST /vmw/rbd/host-register?bootmac=xx.xx.xx.xx.xx HTTP/1.1" 301 -
YYYY-MM-DDTHH:MM:SS [12782:Thread-2 (process_request_worker)]INFO:item_resolver:using rule -- 'rbd.PxeProfile:PxeProfile-22'
YYYY-MM-DDTHH:MM:SS [12782:Thread-2 (process_request_worker)]ERROR:lcmutil:Failed to infer vLCM config
Traceback (most recent call last):
File "bora/install/vmvisor/autodeploy/site-packages/vmware/rbd/utils/lcmutil.py", line 238, in vlcmEnabledForPxeContext
Exception: PxeProfile not cached
rbd-cached.log
YYYY-MM-DDTHH:MM:SS [12783:CacheWorker-1]INFO:cache_inject:cacher responding to injection -- vlcm, https://vcenterlab.fdn.s1.cp.intactfc.com:9087/vum-filedownload/download?file=OFFLINE_BUNDLE_5213d5dc-8427-c177-a731-aaa4515f21d6.zip
YYYY-MM-DDTHH:MM:SS [12783:CacheWorker-2]ERROR:cacher:not enough cache space for item -- CachedItemContext[cacheRequest=('vlcm', 'domain-c44', None, 0, 'YYYY-MM-DDTHH:MM:SS', 0, None, 'YYYY-MM-DDTHH:MM:SS', None, None); cols={'id': 14273, 'plugin': 'vlcm', 'src_name': 'domain-c44', 'src_vc_id': None, 'lease_minutes': 0, 'dst_dir': '7c/29c972c29cd138e22dc91caf891123', 'dl_start_time': datetime.datetime(YYYY, MM, DD, HH, MM, SS)}; itemPaths={'/var/lib/rbd/cache/7c/29c972c29cd138e22dc91caf891123/desired_configuration.json': '/var/lib/rbd/cache/tmp/tmp_l1viali', '/var/lib/rbd/cache/7c/29c972c29cd138e22dc91caf891123/depot.zip': '/var/lib/rbd/cache/tmp/tmpwwgxcc20'}]
YYYY-MM-DDTHH:MM:SS [12783:CacheWorker-2]ERROR:pluginmaster:exception:rbdplugins.vlcm.vmwCacheRequest -- not enough cache space for item
Traceback (most recent call last):
File "bora/install/vmvisor/autodeploy/site-packages/vmware/rbd/utils/pluginmaster.py", line 236, in _curry
File "bora/install/vmvisor/autodeploy/var/rbdplugins/vlcm.py", line 149, in vmwCacheRequest
File "bora/install/vmvisor/autodeploy/var/rbdplugins/vlcm.py", line 123, in _exportClusterImage
File "bora/install/vmvisor/autodeploy/site-packages/vmware/rbd/waiter/cacher.py", line 230, in downloadUrl
File "bora/install/vmvisor/autodeploy/site-packages/vmware/rbd/waiter/cacher.py", line 559, in reserveSpace
rbd.waiter.cacher.CacheException: not enough cache space for item
VMware vCenter Server 7.x
VMware vCenter Server 8.x
By default, the Auto Deploy Service cachesize_GB is set to 2GB which is increased to 4GB in newer releases.
The cache size is not sufficient for the OFFLINE_BUNDLE which is exported to the ESXi hosts during the register process.
Increase the Auto Deploy and Image Builder Service cachesize_GB
Autodeploy and imagebuilder cache can be updated via their respective config files as well.
Service restart is needed for the changes to take effect.
RBD: /etc/vmware-rbd/autodeploy-setup.xml and edit the value for <maxSize> </maxSize>
Imagebuilder: /etc/vmware-imagebuilder/sca-config/imagebuilder-config.props and edit the value cacheSize_GB=
Note: Autodeploy cachesize cannot be more than half of autodeploy partition size.