HA/FDM fails to restart a virtual machine with the error: vim.fault.NoCompatibleHost
search cancel

HA/FDM fails to restart a virtual machine with the error: vim.fault.NoCompatibleHost

book

Article ID: 343894

calendar_today

Updated On:

Products

VMware vSphere ESXi

Issue/Introduction

This article provides steps to avoid the vim.fault.NoCompatibleHost error when restarting the virtual machine.


Symptoms:

  • vSphere HA cluster experiences a host failure or a host isolation and one or more virtual machines have failed to failover.
  • The /var/log/fdm.log file, you see entries similar to:

    <YYYY-MM-DD>T<time></time>.377Z [FFF57400 verbose 'Default' opID=SWI-99a6d1dd] DrmFault: reason powerOnVm, vm /vmfs/volumes/4fbf41e5-########-####-##########60/vm009/vm009.vmx, host host-409, fault [N3Vim5Fault16NoCompatibleHostE:0x76fdcc0]
    <YYYY-MM-DD>T<time></time>.377Z [FFF57400 verbose 'Default' opID=SWI-99a6d1dd] FaultArgument: none
    <YYYY-MM-DD>T<time></time>.377Z [FFF57400 verbose 'Placement' opID=SWI-99a6d1dd] [DrmPE::InvokeDrsMultiplePasses] Pass3: use all compatible hosts
    <YYYY-MM-DD>T<time></time>.377Z [FFF57400 verbose 'Placement' opID=SWI-99a6d1dd] [DrmPE::InvokeDrsAlgorithmForPlacement] Calling mapVm to place 1 Vms
    <YYYY-MM-DD>T<time></time>.378Z [FFF57400 verbose 'Default' opID=SWI-99a6d1dd] DrmFault: reason powerOnVm, vm /vmfs/volumes/4fbf41e5-########-####-##########60/vm009/vm009.vmx, host host-409, fault [N3Vim5Fault16NoCompatibleHostE:0x7741998]
    <YYYY-MM-DD>T<time></time>.378Z [FFF57400 verbose 'Default' opID=SWI-99a6d1dd] FaultArgument: none
    <YYYY-MM-DD>T<time></time>.379Z [FFF57400 verbose 'Placement' opID=SWI-99a6d1dd] [PlacementManagerImpl::PlacementUpdateCb] No recommendation is generated
    <YYYY-MM-DD>T<time></time>.379Z [FFF57400 verbose 'Placement' opID=SWI-99a6d1dd] [PlacementManagerImpl::HandleNotPlacedVms] Reset Vm /vmfs/volumes/4fbf41e5-########-####-##########60/vm009/vm009.vmx, vim.fault.NoCompatibleHost
     
  • The Tasks and Events tab display entries similar to:
     
    Alert vSphere HA virtual machine failover failed
    Alarm 'vSphere HA virtual machine failover failed' changed from Gray to Red
    Not enough resources to failover this virtual machine. vSphere HA will retry when resources become available.

    Note: The preceding log excerpts are only examples. Date, time, and environmental variables may vary depending on your environment. 

Environment

  • VMware vSphere ESXi 8.0
  • VMware vSphere ESXi 7.0
  • VMware vSphere ESXi 6.7
  • VMware vSphere ESXi 6.5
  • VMware vSphere ESXi 6.0
  • VMware vSphere ESXi 5.5
  • VMware vSphere ESXi 5.1

Cause

In this example, an ISO image was mounted on the virtual machine, but the VMFS volume and ISO no longer exist.

To verify if an ISO image was mounted on the virtual machine, first determine the host with which the virtual machine is compatible. for more information, see Determining which hosts are compatible for HA protected virtual machines (2037002). For example, the virtual machine is compatible with and running on host-1208:
 
<host>
<key>1</key>
<name>host-1208</name>
</host>

<vm>
<key>7</key>
<name>/vmfs/volumes/4fbf41e5-########-####-##########60/vm009/vm009.vmx</name>
</vm>

<restartCompatibility>
<vm>7</vm>
<host>1</host>
</restartCompatibility>

The failover request (for example, RequestPlacement) is correctly ordered by the primary (for example, VmOperationsManager):

<YYYY-MM-DD>T<time></time>.900Z [48A81B90 verbose 'Policy' opID=SWI-6e28e512] [VmOperationsManager::RequestPlacement] VM ID /vmfs/volumes/4fbf41e5-########-####-##########60/vm009/vm009.vmx: Transitioned to 'Placement' state for 'VmFailover'

As indicated previously, the virtual machine is compatible and is running on one host, so the failover fails with the error NoCompatibleHost in the fdm logs. This issue occurs because the ISO image was mounted in the virtual machine, but the VMFS volume and ISO no longer exist:

server22.domain.local-YYYY-MM-DD--09.27/vmfs/volumes/4fbf41e5-########-####-##########60/vm009/vm009.vmx:
ide0:0.present = "true"
ide0:0.fileName = "/vmfs/volumes/4dde0d0f-6daac226-86d4-##########c2/iso/mysoftware.iso"
ide0:0.deviceType = "cdrom-image"

server22.domain.local-YYYY-MM-DD--09.27/var/run/log/hostd.log:
<YYYY-MM-DD>T<time></time>.809Z [67EA0B90 verbose 'vm:/vmfs/volumes/4fbf41e5-########-####-##########60/vm009/vm009.vmx'] Retrieved current VM state from foundry 7, 6
<YYYY-MM-DD>T<time></time>.454Z [67EA0B90 verbose 'vm:/vmfs/volumes/4fbf41e5-########-####-##########60/vm009/vm009.vmx'] Handling message _vmx2: Cannot connect file "/vmfs/volumes/4dde0d0f-########-####-##########c2/iso/mysoftware.iso" as a CD-ROM image: Could not find the file
<YYYY-MM-DD>T<time></time>.454Z [67EA0B90 info 'ha-eventmgr'] Event 21 : Warning message on vm009 on server22.example.local in ha-datacenter: Virtual device ide0:0 will start disconnected.
<YYYY-MM-DD>T<time></time>.455Z [67EA0B90 verbose 'vm:/vmfs/volumes/4fbf41e5-########-####-##########60/vm009/vm009.vmx'] sleepState value is 0
<YYYY-MM-DD>T<time></time>.471Z [67EA0B90 info 'ha-eventmgr' opID=3CAF5A58-00005C2E-a4-37-bf] Event 22 : vm009 on server22.example.local in ha-datacenter is powered on
<YYYY-MM-DD>T<time></time>.471Z [67EA0B90 info 'vm:/vmfs/volumes/4fbf41e5-########-####-##########60/vm009/vm009.vmx' opID=3CAF5A58-00005C2E-a4-37-bf] State Transition (VM_STATE_POWERING_ON -> VM_STATE_ON)

Resolution

Note: Also with vSphere 7.x /8.x in a vSphere HA cluster, a virtual machine residing on a local storage is marked as protected, but it cannot failover.

This is a known behavior by design and is currently being reviewed by VMware.

The protection state is not changed to reflect whether HA can currently restart a VM.
  • For example, we don't report a VM as unprotected if the VM is compatible with only one host, nor if the VM has the HA restart priority disabled.
To avoid this behavior:
  • Ensure that any virtual machine can run on any host in the cluster. All hosts should have access to the same virtual machine networks and datastores.
  • Ensure that virtual machines are located on shared storage, not local storage. If they are on local storage, they cannot be failed over in the case of a host failure.

For more information, see vSphere HA Checklist in vSphere Availability

If this issue occurs in your environment, try to migrate the virtual machine using vMotion. You can migrate the virtual machine that uses a virtual device backed by a device that is not accessible on the destination host using vMotion but you received a warning.

For example, you can migrate a virtual machine with a CD drive backed by the physical CD drive on the source host but you received a warning.

For more information, see:


Additional Information

Powered by Wolken Software