ESXi maintenance mode task failed with an error message "The operation is not allowed in the current state. Host name cannot enter maintenance mode due to host latch failure."

Products

• VMware SDDC Manager

Issue/Introduction

• The operation to place an ESXi host into maintenance mode from vCenter UI may generate the following error message: "The operation is not allowed in the current state. Host name cannot enter maintenance mode due to host latch failure."
• The NSX Manager upgrade, initiated from the SDDC Manager, stalled because the 'Enter maintenance mode' task (initiated by com.vmware.nsx.management.nsxt) remained stuck at 0% in vCenter.
• Below error was observed in vCenter:

/var/log/vmware/vpxd/vpxd.log:

YYYY-MM-DDTHH:MM:SS error vpxd[05773] [Originator@6876 sub=MoHost opID=########] [CreateApiProvider: : errorCb] Providers stack failed: Error:
-- >    system_error
-- > Messages:
-- >    vapi. send. failed<Send of frame failed: N7Vmacore15SystemExceptionE (Connection refused: The remote service is not running, OR is overloaded, OR a firewall is rejecting connections.
) 
[context]zKq7AV#####################################uMAAC3/oPbGliYy5zby42AA == [/context]>
YYYY-MM-DDTHH:MM:SS error vpxd[05806] [Originator@6876 sub=MoHost opID=########] WCP enterMaintenanceMode vAPI returns error: Error:
-- >    system_error
-- > Messages:
-- >    vapi. send. failed<Send of frame failed: N7Vmacore15SystemExceptionE (Connection refused: The remote service is not running, OR is overloaded, OR a firewall is rejecting connections.
) 
-> [context]zKq7AV#####################################uMAAC3 == [/context]>
-- >
YYYY-MM-DDTHH:MM:SS info vpxd[05806] [Originator@6876 sub=MoHost opID=########] Waiting 60 secs then invoke WCP enterMaintenanctenanceMode vAPI

/var/log/vmware/wcp/wcpsvc.log

YYYY-MM-DDTHH:MM:SS error wcp [nsxtlib/serverinfo.go:247] Failed to parse NSX extension server URL; https:// ;; communications error to Unreachable DNS IP#53: timed out:443, err: parse "https:// ;; communications error to Unreachable DNS IP": invalid character " " in host name
YYYY-MM-DDTHH:MM:SS error wcp [vclib/nsx.go: 52] Received error seeding nsxManagersCache: failed to parse NSX server url: parse "https:// ;; communications error to  Unreachable DNS IP": invalid character " " in host name
YYYY-MM-DDTHH:MM:SS panic wcp [cmd/main.go:481] Unable to initialize NSX extension monitor: failed to parse NSX server url: parse "https:// ;; communications invalid character " " in host name

• The wcp service status was verified using the command, confirming that it was in a stopped state :
  • service-control --status --all
• The wcp service failed to start when the following command was executed :
  • service-control --start wcp

Environment

VMware vCenter Server 8.x

VMware Cloud Foundation 5.x

Cause

• The WCP service startup failure on vCenter caused the maintenance mode task to fail.
• The NSX Manager(s) were unable to connect to DNS because the DNS server's FQDN was resolving to an invalid IP, which consequently caused WCP service to fail.

Resolution

Follow the below procedure to diagnose and resolve the issue:

• Validate DNS Pointers:
  • Use the nslookup command to confirm that both NSX Manager and vCenter are correctly pointing to the designated DNS servers. 
• Use nslookup to confirm the DNS server's FQDN resolves to the correct IP (forward lookup) and that the IP resolves back to the FQDN (reverse lookup)
• Inspect DNS Records:
  • Verify that the forward and reverse DNS entries for vCenter and the NSX Manager(s) contain only valid characters and conform to correct formatting standards.
• Once the DNS configuration is corrected, start the wcp service by running: 
  
  • vmon-cli --start wcp

Additional Information

vCenter WCP service unable to start due to NSX-T FQDN invalid character invalid character in host name or unable to reach DNS server configured on NSX manager(s)