ESXi become NO_RESPONSE due to temporary connection competition

Products

• VMware vCenter Server
• VMware vSphere ESXi

Issue/Introduction

• ESXi become NO_RESPONSE and come back soon.
• vpxd.log shows that the ESXi's vpxa service unavailable.
  

  YYYY-MM-DDTHH:MM:SS info vpxd[27487] [Originator@6876 sub=vmomi.soapStub[23825] opID=HB-host-###@56131-34c15589] SOAP request returned HTTP failure; <<io_obj p:0x00007fd2e422aee8, h:89, <UNIX ''>, <UNIX '/var/run/envoy-hgw/hgw-pipe'>>, /hgw/host-###/vpxa>, method: getChanges; code: 503(Service Unavailable); fault: (null)
  YYYY-MM-DDTHH:MM:SS error vpxd[27487] [Originator@6876 sub=Vmomi opID=HB-host-###@56131-34c15589] Got vmacore exception when invoking VMOMI method; <</hgw/host-###>, /vpxa>, vpxapi.VpxaService.getChanges, N7Vmacore4Http13HttpExceptionE(HTTP error response: Service Unavailable)
  --> [context]zKq7AVECAQAAACqAeAEbdnB4ZAAAQxxTbGlidm1hY29yZS5zbwAACBhCACk/QwCWmUoBIEIebGlidm1vbWkuc28AAT9kIQHqkCEBXwohggvnSwF2cHhkAIKzLEwBgiU4TAGCLTlMAYLv1UsBgucDTAEB19oaA7MTDGxpYnZweGFwaS10eXBlcy5zbwCCgAYxAYJgZjEBgrLQgAKC/eCAAoLx838CguDKgAIABOw3ABdFOADFD1EEsI4AbGlicHRocmVhZC5zby4wAAXf+g9saWJjLnNvLjYA[/context]
  YYYY-MM-DDTHH:MM:SS warning vpxd[27487] [Originator@6876 sub=InvtHostCnx opID=HB-host-###@56131-34c15589] Exception occurred during host sync; Host communication failed; [vim.HostSystem:host-###,host.example.com], e: N5Vmomi5Fault17HostCommunication9ExceptionE(Fault cause: vmodl.fault.HostCommunication
  --> )
  --> [context]zKq7AVECAQAAACqAeAEWdnB4ZAAAQxxTbGlidm1hY29yZS5zbwAACBhCACk/QwCWmUqBw40cAXZweGQAge6bKwGBfTpMAYHv1UsBgecDTAEC19oabGlidm1vbWkuc28AA7MTDGxpYnZweGFwaS10eXBlcy5zbwCBgAYxAYFgZjEBgbLQgAKB/eCAAoHx838CgeDKgAIABOw3ABdFOADFD1EEsI4AbGlicHRocmVhZC5zby4wAAXf+g9saWJjLnNvLjYA[/context]
  YYYY-MM-DDTHH:MM:SS info vpxd[27487] [Originator@6876 sub=QuickStats opID=HB-host-###@56131-34c15589] Host [vim.HostSystem:host-###,host.example.com] should not be polled
  YYYY-MM-DDTHH:MM:SS warning vpxd[27487] [Originator@6876 sub=MoHost opID=HB-host-###@56131-34c15589] host [vim.HostSystem:host-###,host.example.com] connection state changed to NO_RESPONSE

• ESXi's envoy returns 503 for the task. see envoy-access.log.
  

  YYYY-MM-DDTHH:MM:SS In(166) envoy-access[2099053]: POST /hgw/host-###/vpxa 503 upstream_reset_before_response_started{connection_termination} UC 1312 88 gzip 2 - - ###.###.###.###:#### HTTP/1.1 TLSv1.2 ###.###.###.###:#### ###.###.###.###:#### HTTP/1.1 - ###.###.###.###:#### "HB-host-###@56131-34c15589" "GetChangesVpxa"

• In vpxa.log, the previous heartbeat was closed by timed out at the same time.
  

  YYYY-MM-DDTHH:MM:SS In(166) Vpxa[2102366]: [Originator@6876 sub=HTTP.HTTPService.HttpConnection opID=HB-SpecSync-host-###@2434359-630642fb-2c] HTTP Connection has timed out while waiting for further requests;<io_obj p:0x000000816510e860, h:-1, <TCP '###.###.###.###:####'>, <TCP '###.###.###.###:####'>>, N7Vmacore16TimeoutExceptionE(Operation timed out: Stream: <io_obj p:0x000000816510e860, h:-1, <TCP '###.###.###.###:####'>, <TCP '###.###.###.###:####'>>, duration: 00:00:49.936238 (hh:mm:ss.us))
  YYYY-MM-DDTHH:MM:SS In(166) Vpxa[2099541]: --> [context]zKq7AVICAgAAANVUewEJdnB4YQAA88lHbGlidm1hY29yZS5zbwAAzh89AEsvLACUMiwAHtssAOD/LABLUFIBUngAbGlicHRocmVhZC5zby4wAAI/Ug9saWJjLnNvLjYA[/context]
  YYYY-MM-DDTHH:MM:SS In(166) Vpxa[2107459]: [Originator@6876 sub=HTTP.HTTPService.HttpConnection opID=###] HTTP Connection has timed out while waiting for further requests; <io_obj p:0x0000008165cbae00, h:-1, <TCP '###.###.###.###:####'>, <TCP '###.###.###.###:####'>>, N7Vmacore16TimeoutExceptionE(Operation timed out: Stream: <io_obj p:0x0000008165cbae00, h:-1, <TCP '###.###.###.###:####'>, <TCP '###.###.###.###:####'>>, duration: 00:00:49.939722 (hh:mm:ss.us))
  YYYY-MM-DDTHH:MM:SS In(166) Vpxa[2099541]: --> [context]zKq7AVICAgAAANVUewEJdnB4YQAA88lHbGlidm1hY29yZS5zbwAAzh89AEsvLACSMCwAHtssAOD/LABLUFIBUngAbGlicHRocmVhZC5zby4wAAI/Ug9saWJjLnNvLjYA[/context]

Cause

The time interval between the previous request and the failed request is approximately 49-50 seconds, which is roughly the same as the default "read" timeout for vpxa's HTTP service.
This is essentially a known race condition, which could cause vpxa to close the connection at the same time Envoy tries to use it.

Resolution

Broadcom is aware of this issue.

Please open a support request with Broadcom to obtain further details. Reference article: Creating and managing Broadcom support cases