ESXi host becomes unresponsive after recovering VM's with Zerto

Products

• VMware vSphere ESXi 8.0
• VMware vSphere ESXi

Issue/Introduction

Though this issue was seen in an environment that used Zerto to recover VM's, this is not necessarily the only instance where this can happen, and the root cause was not able to be determined.

When recovering a VM, the hostd process becomes unresponsive and the host requires a reboot to resume normal functionality.

In the logs, you see an error in the vpxa.log similar to the following:

Er(163) Vpxa[2099642]: [Originator@6876 sub=vmomi.soapStub[4] opID=WFU-6d17d7a] Error deserializating SOAP response body; <<io_obj p:0x000000921cf72c00, h:27, <TCP '127.0.0.1 : 10793'>, <TCP '127.0.0.1 : 8307'>>, /sdk>, method: waitForUpdates, e:
Er(163) Vpxa[2099603]: --> Error returned by expat parser: not well-formed (invalid token)
Er(163) Vpxa[2099603]: -->
Er(163) Vpxa[2099603]: --> while parsing serialized value of type string
Er(163) Vpxa[2099603]: --> at line 7, column 2491
Er(163) Vpxa[2099603]: -->
Er(163) Vpxa[2099603]: --> while parsing property "searchDomain" of static type ArrayOfString
Er(163) Vpxa[2099603]: -->
Er(163) Vpxa[2099603]: --> while parsing serialized DataObject of type vim.net.DnsConfigInfo
Er(163) Vpxa[2099603]: --> at line 7, column 2028
Er(163) Vpxa[2099603]: -->
Er(163) Vpxa[2099603]: --> while parsing property "dnsConfig" of static type NetDnsConfigInfo
Er(163) Vpxa[2099603]: -->
Er(163) Vpxa[2099603]: --> while parsing serialized DataObject of type vim.vm.GuestInfo.StackInfo
Er(163) Vpxa[2099603]: --> at line 7, column 2019
Er(163) Vpxa[2099603]: -->
Er(163) Vpxa[2099603]: --> while parsing property "ipStack" of static type ArrayOfGuestStackInfo
Er(163) Vpxa[2099603]: -->
Er(163) Vpxa[2099603]: --> while parsing serialized DataObject of type vim.vm.GuestInfo
Er(163) Vpxa[2099603]: --> error parsing Any with xsiType GuestInfo
Er(163) Vpxa[2099603]: --> at line 7, column 326

You also see an error stating that vpxa can't connect to hostd:

Er(163) Vpxa[2100864]: [Originator@6876 sub=vpxaInvtHostCnx opID=WFU-7216bbb6] Can't connect to hostd. Shutting down...

Environment

ESXi 8.0

ESXi 9.0

Cause

This issue is caused by invalid characters in the DNS Search String value from within the Guest OS.  The invalid characters cause a parsing error that causes the issue.  The offending string was added by the Zerto recovery process, however this may not be the only way that the issue can be triggered.

 

Resolution

For a Windows-based OS, the Registry value "SearchList" in the following path in the registry should be examined:

HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\Tcpip\Parameters

However, the string may not appear invalid here.  To determine which strings to remove, you will need to examine the vimdump output file in a support bundle for the affected host.

The log file is located here within a support bundle:

commands/vmware-vimdump_-o----U-dcui.txt

Within this file, you can search for the following string:

searchDomain = (str)

One or more values of this property will be corrupt and need to be removed.

To identify which VM has the issue, look above the corrupt value for the hostName and ipAddress fields which will identify the VM.

This issue has not been observed with Linux, but that OS could still be affected.