Unable to place ESXi host into maintenance mode from vCenter

Products

• VMware vCenter Server

Issue/Introduction

• vCenter certificates were recently renewed.
• Maintenance mode activity reports the following error
  Failed to enter namespaces maintenance mode due to Error: system_error Messages: vapi.send.failed
• In vCenter, /var/log/vmware/vpxd/vpxd.log, you would find the following entries: 

YYYY-MM-DDTHH:MM:SS.058Z error vpxd[08459] [Originator@6876 sub=MoHost opID=ml6yo4v7-3114020-auto-1uqsl-h5:7#####45-37] WCP enterMaintenanceMode vAPI returns error: Error:
-->    system_error
--> Messages:
-->    vapi.send.failed<Send of frame failed: N7Vmacore15SystemExceptionE(Connection refused: The remote service is not running, OR is overloaded, OR a firewall is rejecting connections.)
--> [context]zKq7AVECAQAAAG/ifgENdnB4ZAAAQxxTbGlidm1hY29yZS5zbwAACBhCACk/QwCWmUoAKHI4AJmSOABtozgAB7#########XRTgAxQ9RAbCOAGxpYnB0aHJlYWQuc28uMAAC3/oPbGliYy5zby42AA==[/context]>
-->

• In vCenter, /var/log/vmware/wcp/wcpsvc.log, you would find the following entries: 

YYYY-MM-DDTHH:MM:SS.852Z error wcp [nsxtlib/serverinfo.go:268] NSX extension does not contain type information: &{{} 0xc000e844e0 com.vmware.nsx.management.nsxt VMware  4.2.1.3  [{{} https://;; communications error to <DNS server IP address>#53: timed out:443 0xc000e84780 VMware VIP [[email protected]] 4C:87:AA:BB:CC:DD:2A:F2:50:44:53:AF:AA:BB:CC:DD:4F:DC:D3:22 -----BEGIN CERTIFICATE-----
.
.
-----END CERTIFICATE-----}] [] [] [] [] [] [] YYYY-MM-DD HH:MM:SS.498 +0000 UTC <nil> <nil> <nil> [] 0xc001002920 <nil>}
YYYY-MM-DDTHH:MM:SS.852Z error wcp [nsxtlib/serverinfo.go:281] Failed to parse NSX extension server URL; https://;; communications error to <DNS server IP address>#53: timed out:443, err: parse "https://;; communications error to <DNS server IP address>": invalid character " " in host name
YYYY-MM-DDTHH:MM:SS.852Z error wcp [vclib/nsx.go:52] Received error seeding nsxManagersCache: failed to parse NSX server url: parse "https://;; communications error to <DNS server IP address>": invalid character " " in host name
YYYY-MM-DDTHH:MM:SS.852Z panic wcp [cmd/main.go:475] Unable to initialize NSX extension monitor: failed to parse NSX server url: parse "https://;; communications error to <DNS server IP address>": invalid character " " in host name
YYYY-MM-DDTHH:MM:SS.852Z debug wcp [logger/trace.go:92] [ END ] [main.main:283] [492.084664ms] main
YYYY-MM-DDTHH:MM:SS.803Z info wcp [debug/debug.go:47] starting pprof at 127.0.0.1:8929
YYYY-MM-DDTHH:MM:SS.803Z debug wcp [logger/trace.go:77] [BEGIN] [main.main:283] main
YYYY-MM-DDTHH:MM:SS.803Z info wcp [cmd/main.go:285] Initializing Wcp Service. pid=1565704 build=25005939 change=15550359
YYYY-MM-DDTHH:MM:SS.820Z debug wcp [vclib/vc_client_factory.go:173] No CA bundles set
YYYY-MM-DDTHH:MM:SS.845Z debug wcp [ssolib/sts.go:100] Getting HOK signer; store: vpxd-extension, alias: vpxd-extension

• The command "service-control --status --all" reports wcp service in stopped state.

Environment

vCenter 8.x

Cause

The WCP service startup failure on vCenter caused the maintenance mode task to fail. 
The wcp service is in a stopped state because the NSX Manager is unable to establish a trust with vCenter due to a thumbprint mismatch.

Resolution

1. Log into NSX-T manager 
2. Click on System --> Fabric --> Compute Managers --> Select the vCenter
3. Click Edit, change the "Enable Trust" to No, and save the settings. 
4. Edit it again and then change the "Enable Trust" to Yes and save the settings. 
   
5. Start WCP service

Additional Information

ESXi maintenance mode task failed with an error message "The operation is not allowed in the current state. Host name cannot enter maintenance mode due to host latch failure."