VCF greenfield deployment failing at vCenter deployment stage due to host seeding error - Failed to get transaction lock
search cancel

VCF greenfield deployment failing at vCenter deployment stage due to host seeding error - Failed to get transaction lock

book

Article ID: 443692

calendar_today

Updated On:

Products

VMware SDDC Manager / VCF Installer

Issue/Introduction

  • The VCF Installer workflow fails at vCenter deploy stage
  • Seed host using an OEM custom image
  • vCenter OVA deploys with no issue and powers on
  • Deployment rolls back and retries 3 times before the failure
  • /var/log/vmware/vcf/domainmanager/domainmanager.log on the VCF Installer :

YYYY-MM-DDTHH:MM:SS DEBUG [vcf_dm,################################,####] [c.v.e.s.c.util.LocalProcessService,dm-exec-67]  Local command execution with timeout period 3 completed.
YYYY-MM-DDTHH:MM:SS ERROR [vcf_dm,################################,####] [c.v.e.s.o.model.error.ErrorFactory,dm-exec-67]  [L0T3CV] VCF_VSPHERE_VCENTER_INSTALLATION_FAILED vCenter installation failed. Check logs under /var/log/vmware/vcf/domainmanager/ci-installer-YYYY-MM-DDTHH:MM:SS for more details.
com.vmware.vcf.common.util.VsphereRuntimeException: vCenter installation failed. Check logs under /var/log/vmware/vcf/domainmanager/ci-installer-YYYY-MM-DDTHH:MM:SS for more details.
        at com.vmware.vcf.common.util.VsphereContractActionUtils.deployVC(VsphereContractActionUtils.java:291)
        at java.base/jdk.internal.reflect.DirectMethodHandleAccessor.invoke(DirectMethodHandleAccessor.java:103)
        at java.base/java.lang.reflect.Method.invoke(Method.java:580)

  • /var/log/vmware/vcf/domainmanager/ci-installer-xx-xx-xx-xx-xx-xx log on VCF Installer:

YYYY-MM-DDTHH:MM:SS - vCSACliInstallLogger - INFO - ================ [FAILED] Task: Running vSphere vLCM: Creating vSphere vLCM managed cluster. execution failed at HH:MM:SS ================
YYYY-MM-DDTHH:MM:SS - vCSACliInstallLogger - DEBUG - Task 'Running vSphere vLCM: Creating vSphere vLCM managed cluster.' execution failed because [Host seeding failed:(vmodl.MethodFault) {
   dynamicType = <unset>,
   dynamicProperty = (vmodl.DynamicProperty) [],
   msg = 'MethodFault.summary',
   faultCause = <unset>,
   faultMessage = (vmodl.LocalizableMessage) [
      (vmodl.LocalizableMessage) {
         dynamicType = <unset>,
         dynamicProperty = (vmodl.DynamicProperty) [],
         key = 'com.vmware.vcint.error_from_vlcm',
         arg = (vmodl.KeyAnyValue) [
            (vmodl.KeyAnyValue) {
               dynamicType = <unset>,
               dynamicProperty = (vmodl.DynamicProperty) [],
               key = 'vlcm_error',
               value = 'Error:\n   com.vmware.vapi.std.errors.error\nMessages:\n   com.vmware.vcIntegrity.lifecycle.ExtractDepotTask.HostExtractDepotFailed<Extraction of image from host <seed-host_FQDN> failed.>\n   com.vmware.vcIntegrity.lifecycle.EsxImage.LockingError<Remediation cannot begin because another process is updating the host.>\n'
            }
         ],
         message = "An internal error occurred: 'Error:\n   com.vmware.vapi.std.errors.error\nMessages:\n   com.vmware.vcIntegrity.lifecycle.ExtractDepotTask.HostExtractDepotFailed<Extraction of image from host <seed-host_FQDN> failed.>\n   com.vmware.vcIntegrity.lifecycle.EsxImage.LockingError<Remediation cannot begin because another process is updating the host.>\n'"
      }
   ]

 

  • /var/run/log/lifecycle.log on the ESX host :
    YYYY-MM-DDTHH:MM:SSZ -DEBUG lifecycle ####### [esx@####] HostSeeding:1197 Creating directory /var/vmware/lifecycle/hostSeed
    YYYY-MM-DDTHH:MM:SSZ -ERROR lifecycle ####### [esx@####] imagemanagerctl:402 Extract depot failed.
    YYYY-MM-DDTHH:MM:SSZ -ERROR lifecycle ####### [esx@####] imagemanagerctl:125 [NotADirectoryError]
    YYYY-MM-DDTHH:MM:SSZ +ERROR lifecycle ####### [esx@####] [Errno 20] Not a directory: '/var/vmware/lifecycle/hostSeed/reservedVibs'
    YYYY-MM-DDTHH:MM:SSZ -ERROR lifecycle ####### [esx@####] imagemanagerctl:130 Traceback (most recent call last):
    YYYY-MM-DDTHH:MM:SSZ -ERROR lifecycle ####### [esx@####] imagemanagerctl:130   File "/usr/lib/vmware/lifecycle/bin/imagemanagerctl.py", line 400, in depots
    YYYY-MM-DDTHH:MM:SSZ -ERROR lifecycle ####### [esx@####] imagemanagerctl:130     HostSeeding.InstalledImageInfo().extractDepot(task)
    YYYY-MM-DDTHH:MM:SSZ -ERROR lifecycle ####### [esx@####] imagemanagerctl:130   File "/lib64/python3.11/site-packages/vmware/esximage/ImageManager/HostSeeding.py", line 1215, in extractDepot
    YYYY-MM-DDTHH:MM:SSZ -ERROR lifecycle ####### [esx@####] imagemanagerctl:130     resVibCache = ReservedVibCache()
    YYYY-MM-DDTHH:MM:SSZ -ERROR lifecycle ####### [esx@####] imagemanagerctl:130   File "/lib64/python3.11/site-packages/vmware/esximage/ImageManager/HostSeeding.py", line 1333, in __init__
    YYYY-MM-DDTHH:MM:SSZ -ERROR lifecycle ####### [esx@####] imagemanagerctl:130     self._loadCachedVibs()
    YYYY-MM-DDTHH:MM:SSZ -ERROR lifecycle ####### [esx@####] imagemanagerctl:130   File "/lib64/python3.11/site-packages/vmware/esximage/ImageManager/HostSeeding.py", line 1390, in _loadCachedVibs
    YYYY-MM-DDTHH:MM:SSZ -ERROR lifecycle ####### [esx@####] imagemanagerctl:130     for fn in os.listdir(cachePath):
    YYYY-MM-DDTHH:MM:SSZ -ERROR lifecycle ####### [esx@####] imagemanagerctl:130 NotADirectoryError: [Errno 20] Not a directory: '/var/vmware/lifecycle/hostSeed/reservedVibs'
  • The VCF Installer identifies the /var/vmware/lifecycle/hostSeed/reservedVibs directory as a file.
  • All subsequent attempts to deploy the vCenter by the installer fails with process lock errors.
  • /var/run/log/lifecycle.log on the ESX host :
    YYYY-MM-DDTHH:MM:SSZ -ERROR lifecycle ####### [esx@####] LockFile:160 Failed to acquire lock /var/run/.esximg.pid: the file is currently locked by PID 2102197.
    YYYY-MM-DDTHH:MM:SSZ -ERROR lifecycle ####### [esx@####] LockFile:160 Failed to acquire lock /var/run/.esximg.pid: the file is currently locked by PID 2102197.
    YYYY-MM-DDTHH:MM:SSZ -ERROR lifecycle ####### [esx@####] LockFile:160 Failed to acquire lock /var/run/.esximg.pid: the file is currently locked by PID 2102197.
    YYYY-MM-DDTHH:MM:SSZ -ERROR lifecycle ####### [esx@####] HostSeeding:1163 Extract depot failed. Failed to get transaction lock: Another process is updating the ESX image. Please try again later.
    YYYY-MM-DDTHH:MM:SSZ -ERROR lifecycle ####### [esx@####] imagemanagerctl:402 Extract depot failed.
    YYYY-MM-DDTHH:MM:SSZ -ERROR lifecycle ####### [esx@####] imagemanagerctl:125 [LockingError]
    YYYY-MM-DDTHH:MM:SSZ +ERROR lifecycle ####### [esx@####] Another process is updating the ESX image. Please try again later.
    YYYY-MM-DDTHH:MM:SSZ -ERROR lifecycle ####### [esx@####] imagemanagerctl:130 Traceback (most recent call last):
    YYYY-MM-DDTHH:MM:SSZ -ERROR lifecycle ####### [esx@####] imagemanagerctl:130   File "/lib64/python3.11/site-packa1.

 

Environment

VCF 9.1.0

Cause

  • Host-seeding execution failed due to /var/vmware/lifecycle/hostSeed/reservedVibs existing as a plain file instead of the required directory. This initial failure subsequently orphaned a process lock file on the host, establishing a stale lock state that blocks all subsequent tasks.
  • This can be caused by low level corruption on the image used on the host.

Resolution

  • Re-image the ESX host with a newly downloaded/created image and retry the deployment.

 

Powered by Wolken Software