VM/VMs do not evacuate from a Host when Maintenance Mode task is triggered with the error message: "Module Nvman power on failed"
search cancel

VM/VMs do not evacuate from a Host when Maintenance Mode task is triggered with the error message: "Module Nvman power on failed"

book

Article ID: 410991

calendar_today

Updated On:

Products

VMware vSphere ESXi

Issue/Introduction

  • VM/VMs do not evacuate from one ESXi host to another when the Maintenance Mode task is triggered.
  • Maintenance mode fails as a VM failed to power on after vMotion.
  • DRS is set to "Fully Automated" and vSphere HA is enabled.
  • The VMs affected have no correlation to each other as it is random
  • The issue always happens when the host is placed in Maintenance Mode.
  • The error message is "Module Nvman power on failed. YYYY-MM-DDTHH:MM:SS The migration has exceeded the maximum switchover time of 100 second(s). ESX has preemptively failed the migration to allow the VM to continue running on the source. To avoid this failure, either increase the maximum allowable switchover time or wait until the VM is performing a less intensive workload. YYYY-MM-DDTHH:MM:SS The VM failed to resume on the destination during early power on."
  • Log snippet:
    vmware.*
    YYYY-MM-DDTHH:MM:SS In(05) vmx - DISK: Opening disks took 280794 ms.
    YYYY-MM-DDTHH:MM:SS No(00) vmx - PowerOnTiming: Module Disk took ######### us
    YYYY-MM-DDTHH:MM:SS Wa(03) vmx - USB: No USB controllers found.
    YYYY-MM-DDTHH:MM:SS In(05) vmx - OBJLIB-FILEBE : FileBEOpen: can't open '########-####-####-####-############.nvram' : Failed to lock the file (######).
    YYYY-MM-DDTHH:MM:SS In(05) vmx - OBJLIB-FILEBE : FileBEOpen: can't open '########-####-####-####-############.nvram' : Failed to lock the file (######).
    YYYY-MM-DDTHH:MM:SS Wa(03) vmx - NVRAMMGR: No valid NVRAM file found, will create default NVRAM.
    YYYY-MM-DDTHH:MM:SS Er(02) vmx - SNAPSHOT: SnapshotConfigInfoReadEx: Open error on '/vmfs/volumes/datastore_name/VM_name/VM_name.vmx': Failed to lock the file
    YYYY-MM-DDTHH:MM:SS Er(02) vmx - SNAPSHOT: SnapshotConfigInfoReadEx failed for file '/vmfs/volumes/datastore_name/VM_name/VM_name.vmx': Failed to lock the file (3)
    YYYY-MM-DDTHH:MM:SS In(05) vmx - Snapshot_VMXObjCreate: ConfigInfoRead failed with Failed to lock the file
    YYYY-MM-DDTHH:MM:SS Wa(03) vmx - NVRAMMGR: Failed to create default NVRAM file (########-####-####-####-############.nvram)
    YYYY-MM-DDTHH:MM:SS Wa(03) vmx - NVRAMMGR: Failed to create default NVRAM file (########-####-####-####-############.nvram)
    YYYY-MM-DDTHH:MM:SS In(05) vmx - Msg_Post: Error
    YYYY-MM-DDTHH:MM:SS In(05) vmx - [msg.nvram.open] NVRAM file open /vmfs/volumes/datastore_name/########-####-####-####-############.nvram (One of the parameters supplied is invalid).
    YYYY-MM-DDTHH:MM:SS In(05) vmx - ----------------------------------------
    YYYY-MM-DDTHH:MM:SS In(05) vmx - Module 'Nvman' power on failed.
    YYYY-MM-DDTHH:MM:SS No(00) vmx - PowerOnTiming: ALL took ######### us
    YYYY-MM-DDTHH:MM:SS In(05) vmx - VMX_PowerOn: ModuleTable_PowerOn = 0
    YYYY-MM-DDTHH:MM:SS In(05) vmx - MigrateSetStateFinished: type=2 new state=MIGRATE_FROM_VMX_FINISHED
    YYYY-MM-DDTHH:MM:SS In(05) vmx - MigrateSetState: Transitioning from state MIGRATE_FROM_VMX_CHECKPT (11) to MIGRATE_FROM_VMX_FINISHED (12).
    YYYY-MM-DDTHH:MM:SS In(05) vmx - Migrate: Caching migration error message list:
    YYYY-MM-DDTHH:MM:SS In(05) vmx - [msg.moduletable.powerOnFailed] Module 'Nvman' power on failed.
    YYYY-MM-DDTHH:MM:SS In(05) vmx - [msg.migrate.resume.fail] The VM failed to resume on the destination during early power on.
    YYYY-MM-DDTHH:MM:SS In(05) vmx - Msg_Post: Error
    YYYY-MM-DDTHH:MM:SS In(05) vmx - [msg.migrate.resume.fail] The VM failed to resume on the destination during early power on.
    YYYY-MM-DDTHH:MM:SS In(05) vmx - [msg.moduletable.powerOnFailed] Module 'Nvman' power on failed.
    YYYY-MM-DDTHH:MM:SS In(05) vmx - ----------------------------------------
    YYYY-MM-DDTHH:MM:SS In(05) vmx - Migrate: cleaning up migration state.
  • CBRC filter attach is not going through as seen below:
    vmkernel.*
    YYYY-MM-DDTHH:MM:SS Wa(180) vmkwarning: cpu##:#######)WARNING: cbrc_filter: CBRCFilterAttach:###: Unable to attach cbrc filter CBRC is disable- disabled VSCSI Handle #################
    YYYY-MM-DDTHH:MM:SS Wa(180) vmkwarning: cpu##:#######)WARNING: VSCSIFilter: ###: handle #################(GID:####)(vscsi0:0):Error attaching filter 'cbrc_filter' to VSCSI_Handle #x###f####a###: Failure
    YYYY-MM-DDTHH:MM:SS Wa(180) vmkwarning: cpu##:#######)WARNING: VSCSI: vm #######: ####: Attaching filter 'cbrc_filter' on scsi0:0 failed: Failure (#########)
    YYYY-MM-DDTHH:MM:SS Wa(180) vmkwarning: cpu##:#######)WARNING: cbrc_filter: CBRCFilterAttach:###: Unable to attach cbrc filter CBRC is disable- disabled VSCSI Handle #################
    YYYY-MM-DDTHH:MM:SS Wa(180) vmkwarning: cpu##:#######)WARNING: VSCSIFilter: ###: handle #################(GID:####)(vscsi0:0):Error attaching filter 'cbrc_filter' to VSCSI_Handle #x###f###e#ac#: Failure
    YYYY-MM-DDTHH:MM:SS Wa(180) vmkwarning: cpu12:3872322)WARNING: VSCSI: vm #######: ####: Attach

Environment

VMware vCenter Server 8.x

VMware ESXi Server 8.x

Cause

This is a result of the CID mismatch, which is a result of CBRC being disabled at host level. 

Resolution

  1. Enable CBRC at host level with VM powered on.
    1. Below are the steps to enable CBRC on host:
      1. In the vCenter Server UI select the ESXi host for which CBRC needs to be enabled:
        1. Click on "Configure" -> "System" -> "Advanced System Settings".
        2. In the "Advanced System Settings" menu click on "EDIT" for key "CBRC.Enable" provide Value as "true".
  • Workaround:
    • Rebooting the VM causing the issue resolves it temporarily. 

Additional Information

Note: Module NVMAN is not a typo as NVMAN stands for NVRAM manager.

Note: If for some reason the value changes the itself to the false option every few minutes then check the Horizon side.

Powered by Wolken Software