Not Responding" and reconnecting gives an error message as "An error occurred while communicating with the remote host"
search cancel

Not Responding" and reconnecting gives an error message as "An error occurred while communicating with the remote host"

book

Article ID: 374019

calendar_today

Updated On:

Products

VMware vCenter Server

Issue/Introduction

  • ESXi hosts are marked as "Not Responding"
  • Reconnecting the ESXi host, fails with below mentioned error:



  • ESXi Host UI and SSH are accessible.
  • Restarting vpxa and hostd services does not help.
  • In the /var/log/vmware/vpxd/vpxd.log on the vCenter Server you see entries similar to the following:

    info vpxd [Originator@6876 sub=vpxLro opID=lziskjkj-mwq-##:########-##] [VpxLRO] -- FINISH lro-117771
    info vpxd [Originator@6876 sub=Heartbeat opID=HeartbeatModuleStart-########] Certificate not available, starting hostsync; [vim.HostSystem:host,<NAME OF ESXI>], msg: {srv: 1514140, gen: 158077, ct: 663903, bld: 22380479, cnx: 529f00fd-bc0a-5c43-aa04-bfaba808f5a1, ip: <##.##.##.##>}
    info vpxd [Originator@6876 sub=Vmomi opID=HostSync-host-53-1e123ff8] Creating SOAP stub adapter for /vpxa on <NAME OF ESXI>:443
    warning vpxd[07017] [Originator@6876 sub=IO.Connection opID=HostSync-host-##-########] Failed to connect; <io_obj p:0x00007f04f806e038, h:54, <UNIX ''>, <UNIX '/var/run/envoy-hgw/hgw-pipe'>>, e: 2(No such file or directory), duration: 0msec
    warning vpxd[07017] [Originator@6876 sub=HttpConnectionPool-000000 opID=HostSync-host-53-1e123ff8] Failed to get pooled connection; <cs p:00007f04f804c5e0, PIPE:/var/run/envoy-hgw/hgw-pipe>, (null), duration: 0msec, N7Vmacore15SystemExceptionE(No such file or directory)
    --> [context]zKq7AVECAQAAADAhbQENdnB4ZAAAGdJTbGlidm1hY29yZS5zbwAAUhlDAIxBRACaWEsAAmI4AHOCOABHkzgA4aE4AObbNwD5NDgAk9BRAa6OAGxpYnB0aHJlYWQuc28uMAACL94PbGliYy5zby42AA==[/context]
    info vpxd[07017] [Originator@6876 sub=IO.Http opID=HostSync-host-53-1e123ff8] Set user agent error; state: 1, (null), N7Vmacore15SystemExceptionE(No such file or directory)

  • In the /var/run/log/hostd.log on the ESXi Host you see entries similar to the following:

    In(166) Hostd: [Originator@6876 sub=SoapAdapter.HTTPService.HttpConnection] HTTP Connection read failed while waiting for further requests; <io_obj p:0x000000ad0c4b4878, h:-1, <TCP '127.0.0.1 : 8307'>,
    <TCP '127.0.0.1 :'>>, N7Vmacore16TimeoutExceptionE(Operation timed out: Stream: <io_obj p:0x000000ad0c4b4878, h:-1, <TCP '127.0.0.1 : 8307'>, <TCP '127.0.0.1 : 56505'>>, duration: 00:00:49.498076 (hh:mm:ss.us))
    In(166) Hostd: --> [context]zKq7AVICAgAAAP////8JaG9zdGQAAANZQWxpYnZtYWNvcmUuc28AAI4PNwCieicA0HsnAI4xKACwTCgAS9lJAYJ6AGxpYnB0aHJlYWQuc28uMAAC7y4PbGliYy5zby42AA==[/context]
    In(166) Hostd: [Originator@6876 sub=SoapAdapter.HTTPService.HttpConnection] HTTP Connection read failed while waiting for further requests; <io_obj p:0x000000ad0c149d08, h:-1, <TCP '127.0.0.1 : 8307'>,
    <TCP '127.0.0.1 : 44123'>>, N7Vmacore16TimeoutExceptionE(Operation timed out: Stream: <io_obj p:0x000000ad0c149d08, h:-1, <TCP '127.0.0.1 : 8307'>, <TCP '127.0.0.1 : 44123'>>, duration: 00:00:48.504351 (hh:mm:ss.us))
    In(166) Hostd: --> [context]zKq7AVICAgAAAP////8JaG9zdGQAAANZQWxpYnZtYWNvcmUuc28AAI4PNwCieicA0HsnAI4xKACwTCgAS9lJAYJ6AGxpYnB0aHJlYWQuc28uMAAC7y4PbGliYy5zby42AA==[/context]
    In(166) Hostd: [Originator@6876 sub=SoapAdapter.HTTPService.HttpConnection] HTTP Connection read failed while waiting for further requests; <io_obj p:0x000000ad0c100898, h:-1, <TCP '127.0.0.1 : 8307'>,

  • In the /var/run/log/vpxa/vpxa.log on the ESXi host you see entries similar to the following:

    In(166) Vpxa[35052040]: [Originator@6876 sub=HTTP.HTTPService.HttpConnection opID=671ca45f] HTTP Connection read failed while waiting for further requests; <io_obj p:0x000000229e625298, h:-1, <TCP '127.0.0.1 : 8089'>, <TCP '127.0.0.1 : 37980'>>, N7Vmacore16TimeoutExceptionE(Operation timed out: Stream: <io_obj p:0x000000229e625298, h:-1, <TCP '127.0.0.1 : 8089'>, <TCP '127.0.0.1 : 37980'>>, duration: 00:00:45.301158 (hh:mm:ss.us))
    In(166) Vpxa[35052019]: --> [context]zKq7AVICAgAAAL9/VQEJdnB4YQAAA1lBbGlidm1hY29yZS5zbwAAjg83AKJ6JwApfScAjjEoALBMKABL2UkBgnoAbGlicHRocmVhZC5zby4wAALvLg9saWJjLnNvLjYA[/context]
    In(166) Vpxa[35052045]: [Originator@6876 sub=HTTP.HTTPService.HttpConnection] HTTP Connection read failed while waiting for further requests; <io_obj p:0x000000229e8b5508, h:-1, <TCP '127.0.0.1 : 8089'>, <TCP '127.0.0.1 : 37983'>>, N7Vmacore16TimeoutExceptionE(Operation timed out: Stream: <io_obj p:0x000000229e8b5508, h:-1, <TCP '127.0.0.1 : 8089'>, <TCP '127.0.0.1 : 37983'>>, duration: 00:00:45.298440 (hh:mm:ss.us))
    In(166) Vpxa[35052019]: --> [context]zKq7AVICAgAAAL9/VQEJdnB4YQAAA1lBbGlidm1hY29yZS5zbwAAjg83AKJ6JwDQeycAjjEoALBMKABL2UkBgnoAbGlicHRocmVhZC5zby4wAALvLg9saWJjLnNvLjYA[/context]
    In(166) Vpxa[35052578]: [Originator@6876 sub=vpxaInvtHost opID=resourcePoolImpl.cpp:1680-5d8a4f67] Increment master gen. no to (925): ResourcePool:VpxaInvtHostResPoolListener::ConfigChanged

  • If the service /var/log/vmware/vmware-envoy-hgw is in stopped state and is not starting, then under /var/log/vmware/vmon/vmon.log on the vCenter you see entries similar to the following:

YYYY-MM-DDTHH:MM:SSIn(05) host-#### Received start request for envoy-hgw
YYYY-MM-DDTHH:MM:SS In(05) host-#### <envoy-hgw-prestart> Constructed command: /usr/bin/python /usr/lib/vmware-envoy-hgw/prestart-envoy-hgw.py
YYYY-MM-DDTHH:MM:SS Wa(03) host-#### <envoy-hgw> Service pre-start command's stderr: Traceback (most recent call last):
YYYY-MM-DDTHH:MM:SS Wa(03)+ host-####File "/usr/lib/vmware-envoy-hgw/prestart-envoy-hgw.py", line 46, in <module>
YYYY-MM-DDTHH:MM:SS Wa(03) host-#### <envoy-hgw> Service pre-start command's stderr: main()
YYYY-MM-DDTHH:MM:SS Wa(03)+ host-####File "/usr/lib/vmware-envoy-hgw/prestart-envoy-hgw.py", line 25, in main
YYYY-MM-DDTHH:MM:SS Wa(03)+ host-#### config = yaml.safe_load(file)
YYYY-MM-DDTHH:MM:SS Wa(03) host-#### <envoy-hgw> Service pre-start command's stderr:File "/usr/lib/python3.10/site-packages/yaml/__init__.py", line 162, in safe_load
YYYY-MM-DDTHH:MM:SS Wa(03) host-#### <envoy-hgw> Service pre-start command's stderr: return load(stream, SafeLoader)
YYYY-MM-DDTHH:MM:SS Wa(03)+ host-####File "/usr/lib/python3.10/site-packages/yaml/__init__.py", line 112, in load
YYYY-MM-DDTHH:MM:SS Wa(03) host-#### <envoy-hgw> Service pre-start command's stderr: loader = Loader(stream)
YYYY-MM-DDTHH:MM:SS Wa(03)+ host-####File "/usr/lib/python3.10/site-packages/yaml/loader.py", line 34, in __init__
YYYY-MM-DDTHH:MM:SS Wa(03)+ host-#### Reader.__init__(self, stream)
YYYY-MM-DDTHH:MM:SS Wa(03)+ host-####File "/usr/lib/python3.10/site-packages/yaml/reader.py", line 85, in __init__
YYYY-MM-DDTHH:MM:SS Wa(03)+ host-#### self.determine_encoding()
YYYY-MM-DDTHH:MM:SS Wa(03)+ host-####File "/usr/lib/python3.10/site-packages/yaml/reader.py", line 135, in determine_encoding
YYYY-MM-DDTHH:MM:SS Wa(03)+ host-#### self.update(1)
YYYY-MM-DDTHH:MM:SS Wa(03)+ host-####File "/usr/lib/python3.10/site-packages/yaml/reader.py", line 169, in update
YYYY-MM-DDTHH:MM:SS Wa(03)+ host-#### self.check_printable(data)
YYYY-MM-DDTHH:MM:SS Wa(03) host-#### <envoy-hgw> Service pre-start command's stderr:File "/usr/lib/python3.10/site-packages/yaml/reader.py", line 143, in check_printable
YYYY-MM-DDTHH:MM:SS Wa(03) host-#### <envoy-hgw> Service pre-start command's stderr: raise ReaderError(self.name, position, ord(character),
YYYY-MM-DDTHH:MM:SS Wa(03) host-#### <envoy-hgw> Service pre-start command's stderr: yaml.reader.ReaderError: unacceptable character #x0000: special characters are not allowed
YYYY-MM-DDTHH:MM:SS Wa(03)+ host-####in "/etc/vmware-envoy-hgw/config.yaml", position 0
YYYY-MM-DDTHH:MM:SS Er(02) host-#### <envoy-hgw> Service pre-start command failed with exit code 1.
YYYY-MM-DDTHH:MM:SS host-#### [ReadSvcSubStartupData] No startup information from envoy-hgw.



Cause

  • This issue occurs because the service  vmware-envoy-hgw is in stopped state or failed state and this service is used to communicate with the hostd and vpxa on the ESXi host.
  • The /etc/vmware-envoy-hgw/config.yaml file may have other entries after VCSA VM was shut down abruptly.

Resolution

Follow the below steps to resolve this issue.

  1. If vCenter was in failed state, fix fsck error by following the steps from KB - "[FAILED] Failed to start File System Check on /dev/vg_root_0/lv_root0" error on Photon OS based virtual appliances

  2. Once Step 1 is completed if the vmware-envoy-hgw service is not starting up and vmon logs show "unacceptable character #x0000: special characters are not allowed", then from another working vCenter server using the same build,copy the file  "config.yaml"  from this location  "/etc/vmware-envoy-hgw" directory to the current vCenter Server using WinSCP tool.

  3. Once the file is copied, start the vmware-envoy-hgw service using below command:
    service-control --start vmware-envoy-hgw

  4. Login to vSphere Client, right click on ESXi host and connect.

Note: If above steps does not resolve the issue, then reboot the vCenter Server.

 



Powered by Wolken Software