vMotion fails with error "Failed to start migration: Necessary module isn't loaded"

Products

• VMware vSphere ESXi

Issue/Introduction

• vMotion fails with the following error in /var/run/log/hostd.log : 
  
  YYYY-MM-DDTHH:MM:SSZZ In(166) Hostd[2100172]: [Originator@6876 sub=Vcsvc.VMotionDst.4984049820402878818] ResolveCb: Failed with fault: (vim.fault.GenericVmConfigFault) {
YYYY-MM-DDTHH:MM:SSZZ In(166) Hostd[2099622]: -->    faultMessage = (vmodl.LocalizableMessage) [
YYYY-MM-DDTHH:MM:SSZZ In(166) Hostd[2099622]: -->       (vmodl.LocalizableMessage) {
YYYY-MM-DDTHH:MM:SSZZ In(166) Hostd[2099622]: -->          key = "msg.checkpoint.initmigration",
YYYY-MM-DDTHH:MM:SSZZ In(166) Hostd[2099622]: -->          arg = (vmodl.KeyAnyValue) [
YYYY-MM-DDTHH:MM:SSZZ In(166) Hostd[2099622]: -->             (vmodl.KeyAnyValue) {
YYYY-MM-DDTHH:MM:SSZZ In(166) Hostd[2099622]: -->                key = "1",
YYYY-MM-DDTHH:MM:SSZZ In(166) Hostd[2099622]: -->                value = "Necessary module isn't loaded"
YYYY-MM-DDTHH:MM:SSZZ In(166) Hostd[2099622]: -->             }
YYYY-MM-DDTHH:MM:SSZZ In(166) Hostd[2099622]: -->          ],
YYYY-MM-DDTHH:MM:SSZZ In(166) Hostd[2099622]: -->          message = "Failed to start migration: Necessary module isn't loaded.
YYYY-MM-DDTHH:MM:SSZZ In(166) Hostd[2099622]: --> "
YYYY-MM-DDTHH:MM:SSZZ In(166) Hostd[2099622]: -->       }
YYYY-MM-DDTHH:MM:SSZZ In(166) Hostd[2099622]: -->    ],
YYYY-MM-DDTHH:MM:SSZZ In(166) Hostd[2099622]: -->    reason = "Failed to start migration: Necessary module isn't loaded.
YYYY-MM-DDTHH:MM:SSZZ In(166) Hostd[2099622]: --> ",
YYYY-MM-DDTHH:MM:SSZZ In(166) Hostd[2099622]: -->    msg = "Failed to start migration: Necessary module isn't loaded.
  
  
• During Host boot/reboot, following errors are observed :
  
  YYYY-MM-DDTHH:MM:SSZ cpu47:2097739)ALERT: Jumpstart plugin vrdma activation failed.
YYYY-MM-DDTHH:MM:SSZ cpu47:2097739)ALERT: Jumpstart plugin vmotion activation failed.
  
  

• /var/run/log/jumpstart-esxcli-stdout.log has the following entries : 
  
  

  YYYY-MM-DDTHH:MM:SSZZ vmkmod: VMKModLoad: Module nrdma is disabled and cannot be loaded.
YYYY-MM-DDTHH:MM:SSZZ vmkmod: VMKModLoad: Module nrdma_vmkapi_shim is disabled and cannot be loaded.
YYYY-MM-DDTHH:MM:SSZZ vmkmod: VMKModLoad: VMKernel_LoadKernelModule(vrdma): Invalid or missing namespace
YYYY-MM-DDTHH:MM:SSZZ Method invocation failed: execute system module load --module vrdma failed
YYYY-MM-DDTHH:MM:SSZZ vrdma: Unable to load module /usr/lib/vmware/vmkmod/vrdma: Invalid or missing namespace

  YYYY-MM-DDTHH:MM:SSZZ vmkmod: VMKModLoad: Module nrdma_vmkapi_shim is disabled and cannot be loaded.
YYYY-MM-DDTHH:MM:SSZZ vmkmod: VMKModLoad: VMKernel_LoadKernelModule(migrate): Unresolved symbol
YYYY-MM-DDTHH:MM:SSZZ Method invocation failed: execute system module load --module migrate failed

• Manually restarting the vMotion module resolves the issue until the next host reboot.

Cause

Due to manual intervention, the modules necessary for vMotion to load are disabled. As a result, both the vMotion and vRDMA modules fail to load leading to vMotion failures.

Resolution

To resolve the issue, enable the  nrdma and nrdma_vmkapi_shim modules :

• Put the host in maintenance mode.
• Take putty to the host after enabling the SSH service.
• Check the current status of the complaining modules (nrdma and nrdma_vmkapi_shim ) : esxcli system module list
• If they are disabled, run the following to enable nrdma and nrdma_vmkapi_shim :
  
  esxcli system module set --enabled=true --module=nrdma
esxcli system module set --enabled=true --module=nrdma_vmkapi_shim

• Verify the status and ensure the modules are now enabled : esxcli system module list
• Reboot the host and verify vMotion functionality.
  
  

Note : The disabled modules may vary depending on the environment. Please refer /var/run/log/jumpstart-esxcli-stdout.log to know which modules are in the disabled state and enable them accordingly to fix the issue.

Additional Information

Enabling and Disabling Native Drivers in ESXi