Adding an ESXi host to the vCenter Server inventory fails with the error "Cannot contact host < IP_Address / FQDN > | Unable to push signed certificate to host "

Products

• VMware vCenter Server 8.0
• VMware vCenter Server 7.0

Issue/Introduction

• Adding the ESXi host to a vCenter fails after it presents the SSL thumbprint.
• Connection to ESXi host user interface also fails due to being unable to verify the SSL thumbprint.
• rhttpproxy logs contain entries similar to:: 

  2024-03-17THH:mm:ss.255Z Wa(164) Rhttpproxy[2133727]: [Originator@6876 sub=IO.Connection] Failed to SSL handshake; SSL(<io_obj p:0x000000d3da72d3a8, h:17, <TCP 'HOST IP Address : 443'>, <TCP 'vCenter IP Address : 49854'>>), e: 167773208(tlsv1 alert unknown ca), duration: 1058msec
2024-03-17THH:mm:ss.256Z Wa(164) Rhttpproxy[2133727]: [Originator@6876 sub=RhttpProxy] SSL Handshake failed for stream SSL(<io_obj p:0x000000d3da72d3a8, h:17, <TCP 'HOST IP Address : 443'>, <TCP 'vCenter IP Address : 49854'>>): N7Vmacore3Ssl12SSLExceptionE(SSL Exception: error:0A000418:SSL routines::tlsv1 alert unknown ca)
2024-03-17THH:mm:ss.257Z Wa(164) Rhttpproxy[2133676]: --> [context]zKq7AVICAgAAAEkBOQENcmh0dHBwcm94eQAAPbJEbGlidm1hY29yZS5zbwAAwnYhAEvZHwBjHBsARVkvAFvYLQDb2y0AAwcwAO6BLwAAni8AKxhBATt9AGxpYnB0aHJlYWQuc28uMAACfdEObGliYy5zby42AA==[/context]
2024-03-17THH:mm:ss.370Z Db(167) Rhttpproxy[2133726]: [Originator@6876 sub=Proxy Req 01906] The client closed the stream, not unexpectedly.
2024-03-17THH:mm:ss.928Z Db(167) Rhttpproxy[2133728]: [Originator@6876 sub=Proxy Req 01909] New proxy client SSL(<io_obj p:0x000000d39716ec48, h:14, <TCP 'HOST IP Address : 443'>, <TCP 'vCenter IP Address : 49858'>>)
2024-03-17THH:mm:ss.208Z Db(167) Rhttpproxy[2133726]: [Originator@6876 sub=Proxy Req 01910] New proxy client SSL(<io_obj p:0x000000d3da907c48, h:16, <TCP 'HOST IP Address : 443'>, <TCP 'vCenter IP Address : 52666'>>)
2024-03-17THH:mm:ss.522Z Db(167) Rhttpproxy[2134646]: [Originator@6876 sub=Proxy Req 01910] Resolved endpoint : [N7Vmacore4Http16LocalServiceSpecE:0x000000d39716eb40] _serverNamespace = /sdk action = Allow authenticationParams =  _port = 8307
2024-03-17THH:mm:ss.523Z Db(167) Rhttpproxy[2133772]: [Originator@6876 sub=IO.Connection] Attempting connection; <resolver p:0x000000d3da72e000, 'localhost:8307', next:<TCP '127.0.0.1 : 8307'>>, last e: 0(Success)
2024-03-17THH:mm:ss.523Z Db(167) Rhttpproxy[2134647]: [Originator@6876 sub=Proxy Req 01910] Connected to localhost:8307 (/sdk) over <io_obj p:0x000000d3dab264e8, h:17, <TCP '127.0.0.1 : 35233'>, <TCP '127.0.0.1 : 8307'>>
2024-03-17THH:mm:ss.568Z Wa(164) Rhttpproxy[2133727]: [Originator@6876 sub=Default] Proxy timed out writing to client. : Read timeout after approximately 50000ms. Closing stream SSL(<io_obj p:0x000000d39716ec48, h:14, <TCP 'HOST IP Address : 443'>, <TCP 'vCenter IP Address : 49858'>>)
2024-03-17THH:mm:ss.568Z Wa(164) Rhttpproxy[2134670]: [Originator@6876 sub=Proxy Req 01909] Error reading from client while waiting for header: N7Vmacore16TimeoutExceptionE(Operation timed out: Stream: SSL(<io_obj p:0x000000d39716ec48, h:-1, <TCP 'HOST IP Address : 443'>, <TCP 'vCenter IP Address : 49858'>>), duration: 00:00:48.639653 (hh:mm:ss.us))
2024-03-17THH:mm:ss.885Z Db(167) Rhttpproxy[2133726]: [Originator@6876 sub=Proxy Req 01910] The client closed the stream, not unexpectedly.

  Or :

  2024-10-20THH:mm:ss.982Z warning rhttpproxy[4131254] [Originator@6876 sub=IO.Connection] Failed to SSL handshake; SSL(<io_obj p:0x00000050e9f43a88, h:38, <TCP 'HOST-IP ADDRESS : 443'>, <TCP 'vCenter-IP ADDRESS : 60382'>>), e: 104(Connection reset by peer), duration: 51msec
2024-10-20THH:mm:ss.982Z warning rhttpproxy[4131254] [Originator@6876 sub=RhttpProxy] SSL Handshake failed for stream SSL(<io_obj p:0x00000050e9f43a88, h:38, <TCP 'HOST-IP ADDRESS : 443'>, <TCP 'vCenter-IP ADDRESS : 60382'>>): N7Vmacore15SystemExceptionE(Connection reset by peer: The connection is terminated by the remote end with a reset packet. Usually, this is a sign of a network problem,  timeout, or service overload.)
--> [context]zKq7AVICAgAAABAEVQENcmh0dHBwcm94eQAALaFCbGlidm1hY29yZS5zbwAA4mcdAC3RGwDSQhcAHKAtAMMHLABACSwAv1UuAMzHLQBUAy4AAhE/ATt9AGxpYnB0aHJlYWQuc28uMAACbdEObGliYy5zby42AA==[/context]
2024-10-20THH:mm:ss.985Z info rhttpproxy[4131254] [Originator@6876 sub=IO.Connection] Failed to shutdown socket; <io_obj p:0x00000050e9f43a88, h:38, <TCP 'HOST-IP ADDRESS : 443'>, <TCP 'vCenter-IP ADDRESS : 60382'>>, e: 104(shutdown: Connection reset by peer)

  
  2024-10-20THH:mm:ss.982Z warning rhttpproxy[4131254] [Originator@6876 sub=IO.Connection] Failed to SSL handshake; SSL(<io_obj p:0x00000050e9f43a88, h:38, <TCP 'HOST-IP ADDRESS : 443'>, <TCP 'vCenter-IP ADDRESS : 60382'>>), e: 104(Connection reset by peer), duration: 51msec
2024-10-20THH:mm:ss.982Z warning rhttpproxy[4131254] [Originator@6876 sub=RhttpProxy] SSL Handshake failed for stream SSL(<io_obj p:0x00000050e9f43a88, h:38, <TCP 'HOST-IP ADDRESS : 443'>, <TCP 'vCenter-IP ADDRESS : 60382'>>): N7Vmacore15SystemExceptionE(Connection reset by peer: The connection is terminated by the remote end with a reset packet. Usually, this is a sign of a network problem,  timeout, or service overload.)
--> [context]zKq7AVICAgAAABAEVQENcmh0dHBwcm94eQAALaFCbGlidm1hY29yZS5zbwAA4mcdAC3RGwDSQhcAHKAtAMMHLABACSwAv1UuAMzHLQBUAy4AAhE/ATt9AGxpYnB0aHJlYWQuc28uMAACbdEObGliYy5zby42AA==[/context]
2024-10-20THH:mm:ss.985Z info rhttpproxy[4131254] [Originator@6876 sub=IO.Connection] Failed to shutdown socket; <io_obj p:0x00000050e9f43a88, h:38, <TCP 'HOST-IP ADDRESS : 443'>, <TCP 'vCenter-IP ADDRESS : 60382'>>, e: 104(shutdown: Connection reset by peer)
2024-10-20THH:mm:ss.273Z warning rhttpproxy[4131096] [Originator@6876 sub=RhttpProxy] SSL Handshake failed for stream SSL(<io_obj p:0x00000050e9f43a88, h:19, <TCP 'HOST-IP ADDRESS : 443'>, <TCP 'vCenter-IP ADDRESS : 60414'>>): N7Vmacore3Ssl12SSLExceptionE(SSL Exception: error:140000DB:SSL routines:SSL routines:short read: The connection was closed by the remote end during handsha
ke.)
--> [context]zKq7AVICAgAAABAEVQENcmh0dHBwcm94eQAALaFCbGlidm1hY29yZS5zbwAA4mcdAC3RGwDSQhcAtqItAMMHLABACSwAv1UuAMzHLQBUAy4AAhE/ATt9AGxpYnB0aHJlYWQuc28uMAACbdEObGliYy5zby42AA==[/context]
2024-10-20THH:mm:ss.353Z warning rhttpproxy[2098623] [Originator@6876 sub=IO.Connection] Failed to SSL handshake; SSL(<io_obj p:0x00000050e9f06918, h:19, <TCP 'HOST-IP ADDRESS : 443'>, <TCP'vCenter-IP ADDRESS : 60430'>>), e: 335544539(short read), duration: 44msec
2024-10-20THH:mm:ss.354Z warning rhttpproxy[2098623] [Originator@6876 sub=RhttpProxy] SSL Handshake failed for stream SSL(<io_obj p:0x00000050e9f06918, h:19, <TCP 'HOST-IP ADDRESS : 443'>, <TCP 'vCenter-IP ADDRESS : 60430'>>): N7Vmacore3Ssl12SSLExceptionE(SSL Exception: error:140000DB:SSL routines:SSL routines:short read: The connection was closed by the remote end during handshake.)

   vCenter logs:

  
  2024-10-20THH:mm:ss.000Z warning vpxd[07695] [Originator@6876 sub=vmomi.soapStub[8] opID=OpID---OpID---OpID-ea-LicenseClientUnregisterHostAsync-618c8289] SOAP request returned HTTP failure; <SSL(<io_obj p:0x00007f20a88f6288, h:161, <TCP '127.0.0.1 : 50680'>, <TCP '127.0.0.1 : 443'>>), /ls/sdk>, method:unregisterEntity; code: 500(Internal Server Error)
2024-10-20THH:mm:ss.000Z warning vpxd[06868] [Originator@6876 sub=Vmomi opID=OpID---OpID---OpID-ea] VMOMI activation LRO failed; <<52f07951-6932-cb63-dfdb-9c9c40073de1, <TCP '127.0.0.1 : 8085'>, <TCP '127.0.0.1 : 59928'>>, group-h526, vim.Folder.addStandaloneHost>, N5Vmomi5Fault11SystemError9ExceptionE(Fault cause: vmodl.fault.SystemError
--> )
--> [context]zKq7AVECAQAAAG0mVQEZdnB4ZAAA9tg3bGlidm1hY29yZS5zbwAAjXgsAAtsLQAT6TIBwaFvdnB4ZAABhaNvAfprjQHjf40BDJiNgZjcHgGBzRUfAQGJo4sBuqmLgtXhBQFsaWJ2aW0tdHlwZXMuc28Age7IYQGB6ctgAYEq
zWABgVjcYAGBuwlgAYGGs2ABAKdJIwA1nyMAtGQ3A4d/AGxpYnB0aHJlYWQuc28uMAAELzYPbGliYy5zby42AA==[/context]
2024-10-20THH:mm:ss.000Z info vpxd[06868] [Originator@6876 sub=vpxLro opID=OpID---OpID---OpID-ea] [VpxLRO] -- FINISH task-1121664
2024-10-20THH:mm:ss.000Z info vpxd[06868] [Originator@6876 sub=Default opID=OpID---OpID---OpID-ea] [VpxLRO] -- ERROR task-1121664 -- group-h111 -- vim.Folder.add
StandaloneHost: vmodl.fault.SystemError:
--> Result:
--> (vmodl.fault.SystemError) {
-->    faultCause = (vmodl.MethodFault) null,
-->    faultMessage = <unset>,
-->    reason = "Unable to push signed certificate to host <host-FQDN / IP ADDRESS>"
-->    msg = ""
--> }
--> Args:
-->
--> Arg spec:
--> (vim.host.ConnectSpec) {
-->    hostName = "<host-FQDN / IP ADDRESS>",
-->    port = <unset>,
-->    sslThumbprint = "Thumb Print of the certificate (xx:xx:xx:xx:......:xx)",
-->    userName = "root",
-->    password = (not shown),
-->    vmFolder = 'FOLDER NAME',
-->    force = true,
-->    vimAccountName = "vpxuser",
-->    vimAccountPassword = (not shown),
-->    managementIp = <unset>,
-->    lockdownMode = "lockdownDisabled",
-->    hostGateway = (vim.host.GatewaySpec) null
--> }
--> Arg compResSpec:
-->
--> Arg addConnected:
--> true

   

Environment

ESXI and vCenter

Network ports are opened as required between the ESXi and vCenter

Cause

Improper MTU is configured in the network path.

The root cause relates to different packet sizes used for regular management traffic versus certificate distribution.

Basic connectivity checks use small TCP packets that can traverse the network successfully. However, pushing the SSL certificate requires a larger payload that may fail to transmit due to network configuration issues like MTU mismatches.

This creates a situation where the host appears reachable, but the certificate exchange needed for adding it to inventory or reaching its web interface cannot complete.

 

Resolution

Configure the MTUs properly.

A standard 1500 MTU size should be communicable between the ESXi to vCenter.

Additional Information

To Check the MTU

From vCenter:

ping -M do -s 1472 ESXI-Host IP

From  ESXi host:

ping -d -s 1472 vCenter IP