ESXi 7.0 is disconnecting and reconnecting back to vCenter.

Products

VMware vSphere ESXi

Issue/Introduction

Symptoms:

ESXi is disconnecting and reconnecting back to vCenter.
vCenter Server shows ESX/ESXi host(s) as not responding

seeing the following within vpxa.log is located on the ESXi host in /var/run/log/vpxa.log

2024-01-09T05:26:47.289Z warning vpxa[7513880] [Originator@6876 sub=IO.Connection] Failed to write buffer to stream; <io_obj p:0x0000007d12e169b8, h:14, <TCP '127.0.0.1 : 34970'>, <TCP '127.0.0.1 : 8307'>> e: 105(No buffer space available), async: true, duration: 0msec

2024-01-09T05:26:47.289Z info vpxa[7513880] [Originator@6876 sub=IO.Http] Set user agent error; state: 3, <io_obj p:0x0000007d12e169b8, h:14, <TCP '127.0.0.1 : 34970'>, <TCP '127.0.0.1 : 8307'>>, N7Vmacore15SystemExceptionE(No buffer space available)
--> [context]zKq7AVICAgAAABAEVQEJdnB4YQAA0kIXbGlidm1hY29yZS5zbwAAHKAtALP1MgBPVi4AzMctAFQDLgACET8BO30AbGlicHRocmVhZC5zby4wAAJt0Q5saWJjLnNvLjYA[/context]

2024-01-09T05:26:47.289Z error vpxa[7513868] [Originator@6876 sub=vpxaVmomi] Got exception when invoking VMOMI method; <<last binding: <<TCP '127.0.0.1 :34970'>, <TCP '127.0.0.1 : 8307'>>>, /sdk>, vim.vm.guest.GuestOperationsManager.GetProcessManager, N7Vmacore15SystemExceptionE(No buffer space available)

--> [context]zKq7AVICAgAAABAEVQEJdnB4YQAA0kIXbGlidm1hY29yZS5zbwAAHKAtALP1MgBPVi4AzMctAFQDLgACET8BO30AbGlicHRocmVhZC5zby4wAAJt0Q5saWJjLnNvLjYA[/context]

2024-01-09T05:26:47.289Z error vpxa[7513880] [Originator@6876 sub=IO.Http] User agent failed to send request; <io_obj p:0x0000007d12e169b8, h:-1, <TCP '127.0.0.1 : 34970'>, <TCP '127.0.0.1 : 8307'>>, N7Vmacore15SystemExceptionE(No buffer space available)

--> [context]zKq7AVICAgAAABAEVQEJdnB4YQAA0kIXbGlidm1hY29yZS5zbwAAHKAtALP1MgBPVi4AzMctAFQDLgACET8BO30AbGlicHRocmVhZC5zby4wAAJt0Q5saWJjLnNvLjYA[/context]

error vpxa[7513868] [Originator@6876 sub=hostdcnx] Exception in connecting to hostd - Shutting down; [17VpxaClientAdapter:0x0000007d12de1f40], N5Vmomi5Fault17HostCommunication9ExceptionE(Fault cause: vmodl.fault.HostCommunication

--> )

--> [context]zKq7AVICAgAAABAEVQERdnB4YQAA0kIXbGlidm1hY29yZS5zbwABQjUYdnB4YQABiQwaAZ0NGgJcDBNsaWJ2bW9taS5zbwCDjVYVAWxpYnZpbS10eXBlcy5zbwABlKgmAbK9JgE/xiYBl
doYAW/UJQEU2CUBMd0YAaseGAG4lxcEXR0CbGliYy5zby42AAF1CRg=[/context]
2024-01-09T05:26:47.291Z error vpxa[7513868] [Originator@6876 sub=vpxaInvtHostCnx] Can't connect to hostd. Shutting down...
2024-01-09T05:26:47.291Z info vpxa[7513868] [Originator@6876 sub=Default] [Vpxa] Shutting down now

seeing the following within syslog.log is located on the ESXi host in /var/run/log/syslog.log

syslog.0:2024-01-09T05:17:33.306Z watchdog-vpxa[7511409]: '/usr/lib/vmware/vpxa/bin/vpxa ++min=0,swapscope=system -D /etc/vmware/vpxa' exited after 129 seconds 1
[..]
syslog.1:2024-01-09T04:14:15.211Z watchdog-vpxa[7495454]: '/usr/lib/vmware/vpxa/bin/vpxa ++min=0,swapscope=system -D /etc/vmware/vpxa' exited after 1 seconds (quick failure 6) 1

Within the vmkernel log seeing 'No buffer space available' /var/run/log/vmkernel.log

2024-01-09T05:50:03.942Z cpu47:2098709)CMMDSNet: CMMDSNetSendtoUnicastChannels:1486: Throttled: 527060c6-f251-08bb-264d-92011463fbee: Failed to send to unicast host 'XX.XXx.X.XX;12321' on iface 'XX.XXx.X.XX': No buffer space available.

Environment

VMware vSphere ESXi 7.0

Cause

heap usage goes beyond 1024 MB.

Resolution

This will be fixed in the future release.

Workaround:

To workaround this issue, increase heap usage goes beyond 1024 MB. You can use below cli to set the heap size (please note that a reboot is required for the changes to take effect)

SSH to ESXi host
Run the below command to increase the heap value
```
esxcfg-advcfg -s 2048 /Net/TcpipHeapMax
```
Reboot the Host