VM Migration Fails at 68% with 'Module NVMAN power on failed'

Products

• VMware vSphere ESXi

Issue/Introduction

VM live migration failure at 68%

Destination vmkernel logs:

2023-08-22T11:07:00.442Z cpu48:30315019)Fil3: 5010: Lock failed on file: XXXXX.vmx on vol 'XXXXX' with FD: <FD c39 r70>
2023-08-22T11:07:00.444Z cpu48:30315019)WARNING: Migrate: 6460: XXXXX D: Migration considered a failure by the VMX. It is most likely a timeout, but check the VMX log for the true error.
2023-08-22T11:07:00.444Z cpu48:30315019)Migrate: 102: XXXXX D: MigrateState: Failed
2023-08-22T11:07:00.444Z cpu48:30315019)WARNING: Migrate: 256: XXXXX D: Failed: Migration determined a failure by the VMX (0xbad0092) @0x42003a0b41bb
2023-08-22T11:07:00.444Z cpu48:30315019)VMotion: 7473: XXXXX D: Estimated network bandwidth 2990.698 MB/s before failure

 

Destination vpxa logs:

2023-08-22T11:08:12.863Z info vpxa[XXXXX] [Originator@6876 sub=Default opID=XXXXX-XXXXX-auto-hkce-h5:XXXXX-71-01-31-01] [VpxLRO] -- ERROR task-XXXXX -- -- vim.host.VMotionManager.initiateDestination:tracking: vim.fault.GenericVmConfigFault:

--> Result:

--> (vim.fault.GenericVmConfigFault) {

--> faultCause = (vmodl.MethodFault) null,

--> faultMessage = (vmodl.LocalizableMessage) [

--> (vmodl.LocalizableMessage) {

--> key = "msg.moduletable.powerOnFailed",

--> arg = (vmodl.KeyAnyValue) [

--> (vmodl.KeyAnyValue) {

--> key = "1",

--> value = "Nvman"

--> }

--> ],

--> message = "Module 'Nvman' power on failed. "

--> },

--> (vmodl.LocalizableMessage) {

--> key = "msg.migrate.resume.fail",

--> arg = <unset>,

--> message = "The VM failed to resume on the destination during early power on. "

--> },

--> (vmodl.LocalizableMessage) {

--> key = "faultTime",

--> arg = <unset>,

--> message = "2023-08-22T11:08:12.638714Z"

--> }

--> ],

--> reason = "Module 'Nvman' power on failed. "

--> msg = "Module 'Nvman' power on failed. "

--> }

--> Args:

-->

 

vmware.log

-------------------

2023-08-22T12:21:45.244Z In(05) vcpu-0 - Closing disk 'scsi0:0'

2023-08-22T12:21:45.244Z In(05) vcpu-0 - LWD: Closing disk 1760440B00 <================

2023-08-22T12:21:47.795Z Er(02) worker-29931522 - IOFIPC: Unable to connect to UDS at /var/run/vmwarelwd/daemon: No such file or directory

2023-08-22T12:21:47.795Z Er(02) worker-29931522 - IOFIPC: Error creating a connection to add to pool 'daemonId': No such file or directory

2023-08-22T12:21:47.795Z In(05) worker-29931522 - IOFIPC: Client connection failed in a previous attempt; delaying 65536ms before next attempt

2023-08-22T12:22:19.247Z Er(02) worker-29931525 - LWD: Failed to connect IPC client to serve request 17B3BBFED0 for disk 1760440B00; error: Connection timed out

2023-08-22T12:22:19.247Z Wa(03) vcpu-0 - LWD: Error sending CloseDisk IPC to daemon for disk 1760440B00: Connection timed out

2023-08-22T12:22:19.249Z In(05) vcpu-0 - LWD: Closed disk 1760440B00

2023-08-22T12:22:19.249Z In(05) vcpu-0 - LWD: LwdFilter_Exit while on disk 1760440B00

2023-08-22T12:22:53.251Z Er(02) worker-29931523 - LWD: Failed to connect IPC client to serve request 17A9A95480 for disk 0; error: Connection timed out

2023-08-22T12:22:53.251Z In(05) vcpu-0 - IOFIPC: IPC Service is no longer accepting connections on 109

2023-08-22T12:22:53.251Z Wa(03) vcpu-0 - IOFIPC: Unable to remove timer for server listening to 'filterId'

2023-08-22T12:22:53.251Z In(05) vcpu-0 - IOFIPC: IPC Service is no longer accepting connections on 109

2023-08-22T12:22:53.257Z In(05) vcpu-0 - IOFIPC: IPC management subsystem shut down

2023-08-22T12:22:53.257Z In(05) vcpu-0 - LWD: LwdFilter_MinimalExit while on disk 1760440B00  <============

2023-08-22T12:22:53.259Z In(05) vcpu-0 - DISKLIB-VMFS : "/vmfs/volumes/XXXXX/XXXXX/XXXXX_4-flat.vmdk" : closed.

 

Closing disk takes a long time and retries due to LWD failed to connect IPC client

 

 

 

Environment

VMware vSphere ESXi 7.0

Cause

The vm was protected with dp/lwd on source host in the past and the cluster where the VM is been migrated doesn't have the the dp service running. 

 

Resolution

Confirm the VMDK contains:

ddb.iofilters = "spif:vmwarelwd"

• Enable dp on those hosts by calling EPS (enable protection service on cluster) or starting dpd service on destination esx host[s] and attempting migration.
  
      Star the dpd service : /etc/init.d/dpd start

OR

• if the vm is no longer part of dp then unprotect dp/lwd on this VM: https://knowledge.broadcom.com/external/article?articleNumber=344906