ESXi hosts are marked as "Not Responding" and connecting gives an error message as "Error communicating to remote host"
search cancel

ESXi hosts are marked as "Not Responding" and connecting gives an error message as "Error communicating to remote host"

book

Article ID: 374019

calendar_today

Updated On:

Products

VMware vCenter Server 8.0

Issue/Introduction

  • ESXi hosts are marked as "Not Responding"
  • When connecting to ESXi host, error message come as "Error communicating to remote host".
  • ESXi Host UI is accessible.
  • Restarting vpxa and hostd services does not help.
  • In the /var/log/vmware/vpxd/vpxd.log on the vCenter Server you see entries similar to the following

     info vpxd[06945] [Originator@6876 sub=vpxLro opID=lziskjkj-mwq-h5:70002378-a4] [VpxLRO] -- FINISH lro-117771
     info vpxd[06052] [Originator@6876 sub=Heartbeat opID=HeartbeatModuleStart-7c30981f] Certificate not available, starting hostsync; [vim.HostSystem:host,<NAME OF ESXI>], msg: {srv: 1514140, gen: 158077, ct: 663903, bld: 22380479, cnx: 529f00fd-bc0a-5c43-aa04-bfaba808f5a1, ip: <XX.XX.XX.XX>}
     info vpxd[06059] [Originator@6876 sub=Vmomi opID=HostSync-host-53-1e123ff8] Creating SOAP stub adapter for /vpxa on <NAME OF ESXI>:443
     warning vpxd[07017] [Originator@6876 sub=IO.Connection opID=HostSync-host-53-1e123ff8] Failed to connect; <io_obj p:0x00007f04f806e038, h:54, <UNIX ''>, <UNIX '/var/run/envoy-hgw/hgw-pipe'>>, e: 2(No such file or directory), duration: 0msec
     warning vpxd[07017] [Originator@6876 sub=HttpConnectionPool-000000 opID=HostSync-host-53-1e123ff8] Failed to get pooled connection; <cs p:00007f04f804c5e0, PIPE:/var/run/envoy-hgw/hgw-pipe>, (null), duration: 0msec, N7Vmacore15SystemExceptionE(No such file or directory)
    --> [context]zKq7AVECAQAAADAhbQENdnB4ZAAAGdJTbGlidm1hY29yZS5zbwAAUhlDAIxBRACaWEsAAmI4AHOCOABHkzgA4aE4AObbNwD5NDgAk9BRAa6OAGxpYnB0aHJlYWQuc28uMAACL94PbGliYy5zby42AA==[/context]
     info vpxd[07017] [Originator@6876 sub=IO.Http opID=HostSync-host-53-1e123ff8] Set user agent error; state: 1, (null), N7Vmacore15SystemExceptionE(No such file or directory)
    --> [context]zKq7AVECAQAAADAhbQENdnB4ZAAAGdJTbGlidm1hY29yZS5zbwAAUhlDAIxBRACaWEsAAmI4AHOCOABHkzgA4aE4AObbNwD5NDgAk9BRAa6OAGxpYnB0aHJlYWQuc28uMAACL94PbGliYy5zby42AA==[/context]
     warning vpxd[06059] [Originator@6876 sub=vmomi.soapStub[69670] opID=HostSync-host-53-1e123ff8] Terminating invocation; <<cs p:00007f04f804c5e0, PIPE:/var/run/envoy-hgw/hgw-pipe>, /hgw/host-53/vpxa>, moref: vmodl.infra.VmodlNs:infra-vmodl-ns, method: selectCapabilities
    info vpxd[06059] [Originator@6876 sub=IO.Http opID=HostSync-host-53-1e123ff8] Terminating user agent request; (null), N7Vmacore17CanceledExceptionE(Operation was canceled)
    --> [context]zKq7AVECAQAAADAhbQEhdnB4ZAAAGdJTbGlidm1hY29yZS5zbwAAUhlDAIxBRADMWksA36NIARv/H2xpYnZtb21pLnNvAAFaLyABZZcdAd+XHQG5mB0BrXMdAeyZHYI42zcBdnB4ZACCfOA3AYLVCjgBguMNOAGCIhw4AYL9HTgBgqjBNwGCUew3AQF9NhoDpREMbGlidnB4YXBpLXR5cGVzLnNvAIJGLR4Bgv56HgGCp5ZlAoL5pmUCgiO+ZAKCvpBlAgDm2zcA+TQ4AJPQUQSujgBsaWJwdGhyZWFkLnNvLjAABS/eD2xpYmMuc28uNgA=[/context]
     error vpxd[07017] [Originator@6876 sub=IO.Http opID=HostSync-host-53-1e123ff8] User agent failed to send request; (null), N7Vmacore15SystemExceptionE(No such file or directory)
    --> [context]zKq7AVECAQAAADAhbQENdnB4ZAAAGdJTbGlidm1hY29yZS5zbwAAUhlDAIxBRACaWEsAAmI4AHOCOABHkzgA4aE4AObbNwD5NDgAk9BRAa6OAGxpYnB0aHJlYWQuc28uMAACL94PbGliYy5zby42AA==[/contex]

  • In the /var/run/log/hostd.log on the ESXi host you see entries similar to the following

    In(166) Hostd[35051706]: [Originator@6876 sub=SoapAdapter.HTTPService.HttpConnection] HTTP Connection read failed while waiting for further requests; <io_obj p:0x000000ad0c4b4878, h:-1, <TCP '127.0.0.1 : 8307'>,
    <TCP '127.0.0.1 : 56505'>>, N7Vmacore16TimeoutExceptionE(Operation timed out: Stream: <io_obj p:0x000000ad0c4b4878, h:-1, <TCP '127.0.0.1 : 8307'>, <TCP '127.0.0.1 : 56505'>>, duration: 00:00:49.498076 (hh:mm:ss.us))
    In(166) Hostd[35051675]: --> [context]zKq7AVICAgAAAP////8JaG9zdGQAAANZQWxpYnZtYWNvcmUuc28AAI4PNwCieicA0HsnAI4xKACwTCgAS9lJAYJ6AGxpYnB0aHJlYWQuc28uMAAC7y4PbGliYy5zby42AA==[/context]
    In(166) Hostd[35051687]: [Originator@6876 sub=SoapAdapter.HTTPService.HttpConnection] HTTP Connection read failed while waiting for further requests; <io_obj p:0x000000ad0c149d08, h:-1, <TCP '127.0.0.1 : 8307'>,
    <TCP '127.0.0.1 : 44123'>>, N7Vmacore16TimeoutExceptionE(Operation timed out: Stream: <io_obj p:0x000000ad0c149d08, h:-1, <TCP '127.0.0.1 : 8307'>, <TCP '127.0.0.1 : 44123'>>, duration: 00:00:48.504351 (hh:mm:ss.us))
    In(166) Hostd[35051675]: --> [context]zKq7AVICAgAAAP////8JaG9zdGQAAANZQWxpYnZtYWNvcmUuc28AAI4PNwCieicA0HsnAI4xKACwTCgAS9lJAYJ6AGxpYnB0aHJlYWQuc28uMAAC7y4PbGliYy5zby42AA==[/context]
    In(166) Hostd[35051692]: [Originator@6876 sub=SoapAdapter.HTTPService.HttpConnection] HTTP Connection read failed while waiting for further requests; <io_obj p:0x000000ad0c100898, h:-1, <TCP '127.0.0.1 : 8307'>,
    <TCP '127.0.0.1 : 55358'>>, N7Vmacore16TimeoutExceptionE(Operation timed out: Stream: <io_obj p:0x000000ad0c100898, h:-1, <TCP '127.0.0.1 : 8307'>, <TCP '127.0.0.1 : 55358'>>, duration: 00:00:49.510122 (hh:mm:ss.us))
    In(166) Hostd[35051675]: --> [context]zKq7AVICAgAAAP////8JaG9zdGQAAANZQWxpYnZtYWNvcmUuc28AAI4PNwCieicA0HsnAI4xKACwTCgAS9lJAYJ6AGxpYnB0aHJlYWQuc28uMAAC7y4PbGliYy5zby42AA==[/context]

     

  • In the /var/run/log/vpxa/vpxa.log on the ESXi host you see entries similar to the following

    In(166) Vpxa[35052040]: [Originator@6876 sub=HTTP.HTTPService.HttpConnection opID=671ca45f] HTTP Connection read failed while waiting for further requests; <io_obj p:0x000000229e625298, h:-1, <TCP '127.0.0.1 : 8089'>, <TCP '127.0.0.1 : 37980'>>, N7Vmacore16TimeoutExceptionE(Operation timed out: Stream: <io_obj p:0x000000229e625298, h:-1, <TCP '127.0.0.1 : 8089'>, <TCP '127.0.0.1 : 37980'>>, duration: 00:00:45.301158 (hh:mm:ss.us))
    In(166) Vpxa[35052019]: --> [context]zKq7AVICAgAAAL9/VQEJdnB4YQAAA1lBbGlidm1hY29yZS5zbwAAjg83AKJ6JwApfScAjjEoALBMKABL2UkBgnoAbGlicHRocmVhZC5zby4wAALvLg9saWJjLnNvLjYA[/context]
    In(166) Vpxa[35052045]: [Originator@6876 sub=HTTP.HTTPService.HttpConnection] HTTP Connection read failed while waiting for further requests; <io_obj p:0x000000229e8b5508, h:-1, <TCP '127.0.0.1 : 8089'>, <TCP '127.0.0.1 : 37983'>>, N7Vmacore16TimeoutExceptionE(Operation timed out: Stream: <io_obj p:0x000000229e8b5508, h:-1, <TCP '127.0.0.1 : 8089'>, <TCP '127.0.0.1 : 37983'>>, duration: 00:00:45.298440 (hh:mm:ss.us))
    In(166) Vpxa[35052019]: --> [context]zKq7AVICAgAAAL9/VQEJdnB4YQAAA1lBbGlidm1hY29yZS5zbwAAjg83AKJ6JwDQeycAjjEoALBMKABL2UkBgnoAbGlicHRocmVhZC5zby4wAALvLg9saWJjLnNvLjYA[/context]
    In(166) Vpxa[35052578]: [Originator@6876 sub=vpxaInvtHost opID=resourcePoolImpl.cpp:1680-5d8a4f67] Increment master gen. no to (925): ResourcePool:VpxaInvtHostResPoolListener::ConfigChanged
    In(166) Vpxa[35052040]: [Originator@6876 sub=vpxaInvtHost] Increment master gen. no to (928): Event:VpxaEventHostd::CheckQueuedEvents
    In(166) Vpxa[35052035]: [Originator@6876 sub=vpxaInvtHost opID=resourcePoolImpl.cpp:1680-1a3a9e64] Increment master gen. no to (933): ResourcePool:VpxaInvtHostResPoolListener::ConfigChanged
    In(166) Vpxa[35052034]: [Originator@6876 sub=vpxaInvtHost opID=WFU-472719c2] Increment master gen. no to (936): VmRuntime:GuestDiskChanged
    In(166) Vpxa[35052040]: [Originator@6876 sub=vpxaInvtHost] Increment master gen. no to (937): Event:VpxaEventHostd::CheckQueuedEvents
    In(166) Vpxa[35052032]: [Originator@6876 sub=vpxaInvtHost opID=WFU-56fdc2d6] Increment master gen. no to (941): VmRuntime:GuestDiskChanged

  • If the service vmware-envoy-hgn is in stopped state and is not starting then  in  /var/run/log/vmon/vmon.log on the vCenter you see entries similar to the following

        yyyy-mm-ddThh:mm:ss.ms In(05) host-2231 Received start request for envoy-hgw
    yyyy-mm-ddThh:mm:ss.ms In(05) host-2231 <envoy-hgw-prestart> Constructed command: /usr/bin/python /usr/lib/vmware-envoy-hgw/prestart-envoy-hgw.py
    yyyy-mm-ddThh:mm:ss.ms Wa(03) host-2231 <envoy-hgw> Service pre-start command's stderr: Traceback (most recent call last):
    yyyy-mm-ddThh:mm:ss.ms Wa(03)+ host-2231   File "/usr/lib/vmware-envoy-hgw/prestart-envoy-hgw.py", line 46, in <module>
    yyyy-mm-ddThh:mm:ss.ms Wa(03)+ host-2231
    yyyy-mm-ddThh:mm:ss.ms Wa(03) host-2231 <envoy-hgw> Service pre-start command's stderr:     main()
    yyyy-mm-ddThh:mm:ss.ms Wa(03)+ host-2231   File "/usr/lib/vmware-envoy-hgw/prestart-envoy-hgw.py", line 25, in main
    yyyy-mm-ddThh:mm:ss.ms Wa(03)+ host-2231     config = yaml.safe_load(file)
    yyyy-mm-ddThh:mm:ss.ms Wa(03)+ host-2231
    yyyy-mm-ddThh:mm:ss.ms Wa(03) host-2231 <envoy-hgw> Service pre-start command's stderr:   File "/usr/lib/python3.10/site-packages/yaml/__init__.py", line 162, in safe_load
    yyyy-mm-ddThh:mm:ss.ms Wa(03)+ host-2231
    yyyy-mm-ddThh:mm:ss.ms Wa(03) host-2231 <envoy-hgw> Service pre-start command's stderr:     return load(stream, SafeLoader)
    yyyy-mm-ddThh:mm:ss.ms Wa(03)+ host-2231   File "/usr/lib/python3.10/site-packages/yaml/__init__.py", line 112, in load
    yyyy-mm-ddThh:mm:ss.ms Wa(03)+ host-2231
    yyyy-mm-ddThh:mm:ss.ms Wa(03) host-2231 <envoy-hgw> Service pre-start command's stderr:     loader = Loader(stream)
    yyyy-mm-ddThh:mm:ss.ms Wa(03)+ host-2231   File "/usr/lib/python3.10/site-packages/yaml/loader.py", line 34, in __init__
    yyyy-mm-ddThh:mm:ss.ms Wa(03)+ host-2231     Reader.__init__(self, stream)
    yyyy-mm-ddThh:mm:ss.ms Wa(03)+ host-2231   File "/usr/lib/python3.10/site-packages/yaml/reader.py", line 85, in __init__
    yyyy-mm-ddThh:mm:ss.ms Wa(03)+ host-2231     self.determine_encoding()
    yyyy-mm-ddThh:mm:ss.ms Wa(03)+ host-2231   File "/usr/lib/python3.10/site-packages/yaml/reader.py", line 135, in determine_encoding
    yyyy-mm-ddThh:mm:ss.ms Wa(03)+ host-2231     self.update(1)
    yyyy-mm-ddThh:mm:ss.ms Wa(03)+ host-2231   File "/usr/lib/python3.10/site-packages/yaml/reader.py", line 169, in update
    yyyy-mm-ddThh:mm:ss.ms Wa(03)+ host-2231     self.check_printable(data)
    yyyy-mm-ddThh:mm:ss.ms Wa(03)+ host-2231
    yyyy-mm-ddThh:mm:ss.ms Wa(03) host-2231 <envoy-hgw> Service pre-start command's stderr:   File "/usr/lib/python3.10/site-packages/yaml/reader.py", line 143, in check_printable
    yyyy-mm-ddThh:mm:ss.ms Wa(03)+ host-2231
    yyyy-mm-ddThh:mm:ss.ms Wa(03) host-2231 <envoy-hgw> Service pre-start command's stderr:     raise ReaderError(self.name, position, ord(character),
    yyyy-mm-ddThh:mm:ss.ms Wa(03)+ host-2231
    yyyy-mm-ddThh:mm:ss.ms Wa(03) host-2231 <envoy-hgw> Service pre-start command's stderr: yaml.reader.ReaderError: unacceptable character #x0000: special characters are not allowed
    yyyy-mm-ddThh:mm:ss.ms Wa(03)+ host-2231   in "/etc/vmware-envoy-hgw/config.yaml", position 0
    yyyy-mm-ddThh:mm:ss.msWa(03)+ host-2231
    yyyy-mm-ddThh:mm:ss.ms Er(02) host-2231 <envoy-hgw> Service pre-start command failed with exit code 1.
    yyyy-mm-ddThh:mm:ss.ms host-2231 [ReadSvcSubStartupData] No startup information from envoy-hgw.



Cause

This issue occurs because the service  vmware-envoy-hgn is in stopped state or failed state and this service is used to communicate with the hostd and vpxa on the ESXi host. 

The /etc/vmware-envoy-hgw/config.yaml file may have other entries after VCSA VM was shut down abruptly.

Resolution

Follow the below steps to resolve this issue.

1.If vCenter was in failed state, fix fsck error by following the steps from KB - Failed to start file system check on /dev/disk..." error on Photon OS based virtual appliances.

2.Once Step 1 is completed if the vmware-envoy-hgn service is not starting up and vmon logs show "unacceptable character #x0000: special characters are not allowed",then from another working vCenter server using the same build,copy the file  "config.yaml"  from this location  "/etc/vmware-envoy-hgw" directory to the current vCenter Server using WinSCP tool. 

3.Once the file is copied, start the vmware-envoy-hgw service using the command service-control --start vmware-envoy-hgw

4.Login to vSphere Client, right click on ESXi host and connect.

 

 

 

Powered by Wolken Software