Configuration file has unexpected changes and proxy virtual machine terminates during backup.

Products

• VMware vSphere ESXi

Issue/Introduction

This article provides workaround to avoid the Proxy VM from entering into an invalid state.

Symptoms:
Backup proxy goes to invalid state during backup.

• In the hostd.log file, you see entries similar to: 

YYYY-MM-DDTHH:MM:SSZ info hostd[15A03B70] [Originator@6876 sub=Vmsvc.vm:/vmfs/volumes/580692bf-#######-####-#######5e190/vm_name/vm_name.vmx opID=5953cf5e-3-a90a user=vpxuser:ADMIN\nwadmin_svc] State Transition (VM_STATE_ON -> VM_STATE_RECONFIGURING)
...
YYYY-MM-DDTHH:MM:SSZ error hostd[15640B70] [Originator@6876 sub=Vmsvc.vm:/vmfs/volumes/580692bf-#######-####-#######5e190/vm_name/vm_name.vmx opID=5953cf5e-3-a90a user=vpxuser:ADMIN\nwadmin_svc] Could not apply pre-reconfigure domain changes: Failed to add file policies to domain :171: world ID :0:Cannot allocate memory
...
YYYY-MM-DDTHH:MM:SSZ info hostd[15640B70] [Originator@6876 sub=Vmsvc.vm:/vmfs/volumes/580692bf-#######-####-#######5e190/vm_name/vm_name.vmx opID=5953cf5e-3-a90a user=vpxuser:ADMIN\nwadmin_svc] State Transition (VM_STATE_RECONFIGURING -> VM_STATE_ON)
...
YYYY-MM-DDTHH:MM:SSZ error hostd[15A44B70] [Originator@6876 sub=Vmsvc.vm:/vmfs/volumes/580692bf-#######-####-#######5e190/vm_name/vm_name.vmx] Expected permission (3) for /vmfs/volumes/5ac38771-f9105baf-f368-c25883f002d0/smf-app-comm-dev-054 not found in domain 171

• In the vmkernel.log file, you see entries similar to:

YYYY-MM-DDTHH:MM:SSZ cpu49:68763 opID=4c6a367c)World: 12235: VC opID d03205b-b9-17ed maps to vmkernel opID 4c6a367c
YYYY-MM-DDTHH:MM:SSZ cpu49:68763 opID=4c6a367c)WARNING: Heap: 3534: Heap domainHeap-54 already at its maximum size. Cannot expand.

• Increasing the FilePolicyMaxPolicies to 500 from default 256 returns the same issue: vsish -e set /config/VmkAccess/intOpts/FilePolicyMaxPolicies 500 
• Virtual Machine terminates with the error message similar to:
  
  Configuration file has unexpected changes. Virtual machine has been terminated. Unregister and re-register virtual machine if the configuration file is correct.
• To bring this VM up, you should restart the ESXi management agents or un-register and re-register the VM back to inventory in VC.

Note: The preceding log excerpts are only examples. Date, time, and environmental variables may vary depending on your environment.

Cause

A logical error leads to accumulation of entries in a kernel heap of a fixed size. For a proxy VM that performs numerous disk hot-add, eventually the heap space is exhausted. This results with incorrect policy for the VM and causes the management software to terminate the VM.

Resolution

This issue is resolved in ESXi 6.5 P03 and ESXi 6.7 U1. 


Workaround:
To work around this issue:

1. Migrate the proxy VM periodically to clear up stale security policy rules.
2. Reload the proxy VM after it is migrated by running the following command in the ESXi shell:

vim-cmd vmsvc/reload <vmid>

Where, <vmid> is the managed object ID of the VM on the host, and can be obtained using the vim-cmd vmsvc/getallvms command.