Netapp SnapCenter Plug-In for VMware fails to restore a VM

Products

VMware vCenter Server

Issue/Introduction

Issue:

Related NetAppKB

Restoring a VM to the original location fails with "VM not found"
https://kb.netapp.com/data-mgmt/SnapCenter/SC_KBs/Restoring_a_VM_to_the_original_location_fails_with_VM_not_found

In the vpxd.log you see messages similar to below:

info vpxd[XXXXX] [Originator@XXXX sub=vpxLro opID=XXXXXXXX] [VpxLRO] -- BEGIN task-XXXX -- group-vXXXX -- vim.Folder.registerVm -- XXXXXXXX-XXXX-XXXX-XXXX-XXXXXXXXXXXX(XXXXXXXX-XXXX-XXXX-XXXX-XXXXXXXXXXXX) info vpxd[XXXXX] [Originator@XXXX sub=VdbOpJournal opID=XXXXXXXX] Added new journal id=XXX type=1 info vpxd[XXXXX] [Originator@XXXX sub=vpxLro opID=XXXXXXXX-01] [VpxLRO] -- BEGIN lro-XXXXXXXX -- group-vXXXX -- vim.Folder.registerVm -- XXXXXXXX-XXXX-XXXX-XXXX-XXXXXXXXXXXX(XXXXXXXX-XXXX-XXXX-XXXX-XXXXXXXXXXXX) warning vpxd[XXXXX] [Originator@XXXX sub=AuthorizeManager opID=XXXXXXXX] Privilege check for entity datacenter-XXXX, privilege System.Read made on null session. Backtrace N7Vmacore9ExceptionE(Privilege check in null session) --> [context]XXXXXXXXXXX[/context] info vpxd[XXXXX] [Originator@XXXX sub=Invt opID=XXXXXXXX-01-XXXXXXXX] Started call to update entity state moId on host; [vim.HostSystem:host-XXXX,<HOST_NAME>], store: (vm, 60), v: XXXXXXXX:vm-XXXX:XXXXXXXX, callId: 5 warning vpxd[XXXXX] [Originator@XXXX sub=vmmoVm opID=XXXXXXXX-01-XXXXXXXX] Setting VM [vim.VirtualMachine:vm-XXXX,<VM_NAME>] numCpu to default 1 warning vpxd[XXXXX] [Originator@XXXX sub=vmmoVm opID=XXXXXXXX-01-XXXXXXXX] Setting VM [vim.VirtualMachine:vm-XXXX,<VM_NAME> memory size to default 32MB warning vpxd[XXXXX] [Originator@XXXX sub=vmomi.soapStub[XX] opID=TaskLoop-host-XXXX] SOAP request returned HTTP failure; <<io_obj p:0x00007fcedd02b5c8, h:35, <UNIX ''>, <UNIX '/var/run/envoy-hgw/hgw-pipe'>>, /hgw/host-XXXX/vpxa>, method: waitForUpdates; code: 500(Internal Server Error) error vpxd[XXXXX] [Originator@XXXX sub=Invt opID=XXXXXXXX-01] Failed to update entity state moId on host; [vim.HostSystem:host-XXXX,<HOST_NAME>], store: (vm, 60), v: XXXXXXXX:vm-XXXX:XXXXXXXX, e: XXXXXXXX(Fault cause: vim.fault.InvalidState --> ) --> [context]XXXXXXXXXXX[/context]

Environment

vCenter 8.0U3

ESXi 8.0U3

Cause

In hostd.log you see unable to load configuration file, syntax error messages resulting in an invalid state.

In(166) Hostd[2099424]: [Originator@6876 sub=Libs opID=7f6159f1-01-e1-4a2c sid=52423bcf user=vpxuser:xxxxxx] VigorOffline_Init: Failed to initialize VIGOR offline: Unable to load configuration file '/vmfs/volumes/xxxxxxxxxxxxxxx/<vm>/<vm>.vmx'.

In(166) Hostd[2099424]: [Originator@6876 sub=Libs opID=7f6159f1-01-e1-4a2c sid=52423bcf user=vpxuser:xxxxxx] File "/vmfs/volumes/xxxxxxxxxxxxxxx/<vm>/<vm>.vmx" line 117: Syntax error.
Db(167) Hostd[2099424]: [Originator@6876 sub=Vmsvc.vm:/vmfs/volumes/xxxxxxxxxxxxxxxxx/<vm>/<vm>.vmx opID=7f6159f1-01-e1-4a2c sid=52423bcf user=vpxuser:xxxxx] LoadFromConfig translated error to vim.fault.InvalidVmConfig
Db(167) Hostd[2099424]: [Originator@6876 sub=Vmsvc.vm:/vmfs/volumes/xxxxxxxxxxxxxx/<vm>/<vm>.vmx opID=7f6159f1-01-e1-4a2c sid=52423bcf user=xxxxxxxx] LoadFromConfig message: Unable to load configuration file '/vmfs/volumes/xxxxxxxxxxxxxx/<vm>/<vm>.vmx'.

You also see the VM state transitions from NetApp trying to remove and recreate the VM ending with the VM in an invalid state

In(166) Hostd[2099427]: [Originator@6876 sub=Vmsvc.vm:/vmfs/volumes/xxxxxxxxxxxxxx/<vm>/<vm>.vmx opID=53913220-b2-4924 sid=52423bcf user=vpxuser:xxxxxx] State Transition (VM_STATE_ON -> VM_STATE_POWERING_OFF)
In(166) Hostd[2099422]: [Originator@6876 sub=Vmsvc.vm:/vmfs/volumes/xxxxxxxxxxxxxx/<vm>/<vm>.vmx] State Transition (VM_STATE_POWERING_OFF -> VM_STATE_OFF)
In(166) Hostd[2099408]: [Originator@6876 sub=Vmsvc.vm:/vmfs/volumes/xxxxxxxxxxxxxx/<vm>/<vm>.vmx opID=53913220-01-aa-4965 sid=52423bcf user=vpxuser:<no user>] State Transition (VM_STATE_OFF -> VM_STATE_RECONFIGURING)
In(166) Hostd[2099408]: [Originator@6876 sub=Vmsvc.vm:/vmfs/volumes/xxxxxxxxxxxxxx/<vm>/<vm>.vmx opID=53913220-01-aa-4965 sid=52423bcf user=vpxuser:<no user>] State Transition (VM_STATE_RECONFIGURING -> VM_STATE_OFF)
In(166) Hostd[2099423]: [Originator@6876 sub=Vmsvc.vm:/vmfs/volumes/xxxxxxxxxxxxxx/<vm>/<vm>.vmx opID=3de730d7-c2-49d0 sid=52423bcf user=vpxuser:xxxxxx] State Transition (VM_STATE_OFF -> VM_STATE_UNREGISTERING)
In(166) Hostd[2099423]: [Originator@6876 sub=Vmsvc.vm:/vmfs/volumes/xxxxxxxxxxxxxx/<vm>/<vm>.vmx opID=3de730d7-c2-49d0 sid=52423bcf user=vpxuser:xxxxxxxxxx] State Transition (VM_STATE_UNREGISTERING -> VM_STATE_GONE)
In(166) Hostd[2099424]: [Originator@6876 sub=Vmsvc.vm:/vmfs/volumes/xxxxxxxxxxxxxx/<vm>/<vm>.vmx opID=7f6159f1-01-e1-4a2c sid=52423bcf user=vpxuser:xxxxxxxxxx] State Transition (VM_STATE_INITIALIZING -> VM_STATE_INVALID_CONFIG)

Resolution

The customer did the following to resolve the issue:

Powered off the target VM and renamed the VM folder in the datastore

NetApp was able to restore all the VM files