ESXi 7.0 is disconnecting and reconnecting back to vCenter.

Products

VMware vSphere ESXi

Issue/Introduction

Symptoms:

ESXi is disconnecting and reconnecting back to vCenter.
vCenter Server shows ESX/ESXi host(s) as not responding
seeing the following within vpxa.log is located on the ESXi host in /var/run/log/vpxa.log

[YYYY-MM-DDTHH:MM:SS] warning vpxa[7513880] [Originator@6876 sub=IO.Connection] Failed to write buffer to stream; <io_obj p:0x0000007d12e169b8, h:14, <TCP '127.0.0.1 : 34970'>, <TCP '127.0.0.1 : 8307'>> e: 105(No buffer space available), async: true, duration: Omsec

[YYYY-MM-DDTHH:MM:SS] info vpxa[7513880] [Originator@6876 sub=IO.Http] Set user agent error; state: 3, <io_obj p:0x0000007d12e169b8, h:14, <TCP '127.0.0.1 : 34970'>, <TCP '127.0.0.1 : 8307'>>, N7Vmacore15SystemExceptionE(No buffer space available)
-- > [context]zKq7AVICAgAAABAEVQEJdnB4YQAA0kIXbGlidm1hY29yZS5zbwAAHKAtALP1MgBPVi4AzMctAFQDLgACET8B030AbGlicHRocmVhZC5zby4wAAJt0Q5saWJjLnNvLjYA[/context]

[YYYY-MM-DDTHH:MM:SS] error vpxa[7513868] [Originator@6876 sub=vpxaVmomi] Got exception when invoking VMOMI method; << last binding: << TCP '127.0.0.1 :34970'>, <TCP '127.0.0.1 : 8307'>>>, /sdk>, vim.vm.guest.GuestOperationsManager.GetProcessManager, N7Vmacore15SystemExceptionE(No buffer space available)

-- > [context]zKq7AVICAgAAABAEVQEJdnB4YQAA0kIXbGlidm1hY29yZS5zbwAAHKAtALP1MgBPVi4AzMctAFQDLgACET8B030AbGlicHRocmVhZC5zby4wAAJt0Q5saWJjLnNvLjYA[/context]

[YYYY-MM-DDTHH:MM:SS] error vpxa[7513880] [Originator@6876 sub=IO.Http] User agent failed to send request; <io_obj p:0x0000007d12e169b8, h :- 1, <TCP '127.0.0.1 : 34970'>, <TCP '127.0.0.1 : 8307'>>, N7Vmacore15SystemExceptionE(No buffer space available)

-- > [context]zKq7AVICAgAAABAEVQEJdnB4YQAA0kIXbGlidm1hY29yZS5zbwAAHKAtALP1MgBPVi4AzMctAFQDLgACET8B030AbGlicHRocmVhZC5zby4wAAJt0Q5saWJjLnNvLjYA[/context]

error vpxa[7513868] [Originator@6876 sub=hostdcnx] Exception in connecting to hostd - Shutting down; [17VpxaClientAdapter:0x0000007d12de1f40], N5Vmomi5Fault17HostCommunication9ExceptionE(Fault cause: vmodl. fault. HostCommunication

-> )

-- > [context]zKq7AVICAgAAABAEVQERdnB4YQAA0kIXbGlidm1hY29yZS5zbwABQjUYdnB4YQABiQwaAZ0NGgJcDBNsaWJ2bW9taS5zbwCDjVYVAWxpYnZpbS10eXBlcy5zbwAB1KgmAbK9JgE/xiYB1
doYAW/UJQEU2CUBMd0YAaseGAG41xcEXR0CbGliYy5zby42AAF1CRg=[/context]
[YYYY-MM-DDTHH:MM:SS] error vpxa[7513868] [Originator@6876 sub=vpxaInvtHostCnx] Can't connect to hostd. Shutting down ...
[YYYY-MM-DDTHH:MM:SS] info vpxa[7513868] [Originator@6876 sub=Default] [Vpxa] Shutting down now

seeing the following within syslog.log is located on the ESXi host in /var/run/log/syslog.log

syslog.0:[YYYY-MM-DDTHH:MM:SS] watchdog-vpxa[7511409]: '/usr/lib/vmware/vpxa/bin/vpxa ++min=0, swapscope=system -D /etc/vmware/vpxa' exited after 129 seconds 1
[ .. ]
syslog.1:[YYYY-MM-DDTHH:MM:SS] watchdog-vpxa[7495454]: '/usr/lib/vmware/vpxa/bin/vpxa ++min=0, swapscope=system -D /etc/vmware/vpxa' exited after 1 seconds (quick failure 6) 1

Within the vmkernel log seeing 'No buffer space available' /var/run/log/vmkernel.log

[YYYY-MM-DDTHH:MM:SS] cpu47:2098709)CMMDSNet: CMMDSNetSendtoUnicastChannels:1486: Throttled: 527060c6-f251-08bb-264d-92011463fbee: Failed to send to unicast host 'XX.XXx.X.XX;12321' on iface 'XX.XXx.X.XX': No buffer space available.

Environment

VMware vSphere ESXi 7.0

Cause

heap usage goes beyond 1024 MB.

Resolution

This will be fixed in the future release.

Workaround:

To workaround this issue, increase heap usage goes beyond 1024 MB. You can use below cli to set the heap size (please note that a reboot is required for the changes to take effect)

SSH to ESXi host
Run the below command to increase the heap value
```
esxcfg-advcfg -s 2048 /Net/TcpipHeapMax
```
Reboot the Host.