Edges/ESX hosts are in failed state with error "Failed to send HostConfig RPC to MPA"

Products

• VMware NSX

Issue/Introduction

ESXi hosts are in "Failed/Host Disconnected" status with below errors:

• "Host configuration: Failed to send the HostConfig message. 
[TN=TransportNode/<uuid>]. Reason: Failed to send HostConfig RPC to MPA TN:<uuid>. Error: Unable to reach client <tn-uuid>, application SwitchingVertical."
  
  
  
  
• "Failed to get response from NSX-SFHC component."
  
  

Environment

VMware NSX

Cause

"Error 3#######8-certificate verify failed" being noticed since the APH certificate is in revoked status, resulting in hosts and managers not being able to connect.

Resolution

To recover ESXi/Edges, following steps to be performed on all faulty nodes:

/opt/vmware/nsx-nestdb/bin/nestdb-cli

# To verify the entry of revoked certificates, if any.

get vmware.nsx.nestdb.CrlCertificatesCacheMsg

# To delete the entry.

delete vmware.nsx.nestdb.CrlCertificatesCacheMsg {"id":0}

# Restart nsx-proxy:

/etc/init.d/nsx-proxy restart

Additional Information

ESXi Host log locations to verify the behavior

/var/run/log/nsx-syslog:

YYYY-MM-DDTHH:MM:SS.780Z nsx-proxy[173598788]: NSX 173598788 - [nsx@6876 comp="nsx-esx" subcomp="nsx-proxy" s2comp="nsx-rpc" tid="173598836" level="ERROR" errorCode="RPC503"] RpcTransport[0]::RemoteService[31663ecf-####-4a2e-####-##########] Failed to resolve service: 3#######8-certificate verify failed

YYYY-MM-DDTHH:MM:SS.780Z nsx-proxy[173598788]: NSX 173598788 - [nsx@6876 comp="nsx-esx" subcomp="nsx-proxy" s2comp="nsx-net" tid="173598836" level="WARNING"] Certificate validation: couldn't find SHA256 digest '756#######1d8c3248e0########99173bb0e69574220fa###############23' in local trust store

YYYY-MM-DDTHH:MM:SS.780Z nsx-proxy[173598788]: NSX 173598788 - [nsx@6876 comp="nsx-esx" subcomp="nsx-proxy" s2comp="nsx-net" tid="173598836" level="ERROR" errorCode="NET1111"] Certificate validation failed: 18-self signed certificate Certificate:   <certificate data>

YYYY-MM-DDTHH:MM:SS.780Z nsx-proxy[173598788]: NSX 173598788 - [nsx@6876 comp="nsx-esx" subcomp="nsx-proxy" s2comp="nsx-net" tid="173598836" level="WARNING"] StreamConnection[9989 Connecting to ssl://#.#.#.#:1234 sid:9989] Couldn't connect to 'ssl://#.#.#.#:1234' (error: 3#######8-certificate verify failed)

YYYY-MM-DDTHH:MM:SS.780Z nsx-proxy[173598788]: NSX 173598788 - [nsx@6876 comp="nsx-esx" subcomp="nsx-proxy" s2comp="nsx-net" tid="173598836" level="WARNING"] StreamConnection[9989 Error to ssl://#.#.#.#:1234 sid:-1] Error 3#######8-certificate verify failed

...............

YYYY-MM-DDTHH:MM:SS.527Z nsx-proxy[170067301]: NSX 170067301 - [nsx@6876 comp="nsx-esx" subcomp="nsx-proxy" s2comp="nsx-rpc" tid="170067334" level="INFO"] RpcConnection[1373964 Connected to ssl://#.#.#.#:1234 0] Closing (remote certificates revoked)

YYYY-MM-DDTHH:MM:SS.528Z nsx-proxy[170067301]: NSX 170067301 - [nsx@6876 comp="nsx-esx" subcomp="nsx-proxy" s2comp="nsx-rpc" tid="170067334" level="INFO"] RpcConnection[1373964 Closed to ssl://#.#.#.#:1234 0] Notifying channels on connection down (remote certificates revoked)

YYYY-MM-DDTHH:MM:SS.565Z nsx-proxy[170067301]: NSX 170067301 - [nsx@6876 comp="nsx-esx" subcomp="nsx-proxy" s2comp="nsx-rpc" tid="170067334" level="INFO"] RpcConnection[1373965 Connected to ssl://#.#.#.#:1234 0] Closing (remote certificates revoked)