Error: "Failed to enter namespaces maintenance mode due to Error: system_error..." during ESXi maintenance mode task

Products

• VMware vCenter Server
• VMware vSphere ESXi

Issue/Introduction

When attempting to place an ESXi host into maintenance mode, the task becomes stuck at 2% and the following symptoms are observed:

• In the vSphere Client, the "Enter maintenance mode" task is stuck at 2% before failing with the following error:
  
  "Enter maintenance mode","","2%","Failed to enter namespaces maintenance mode due to Error: system_error Messages: vapi.send.failed<Send of frame failed: N7Vmacore15SystemExceptionE(Connection refused: The remote service is not running, OR is overloaded, OR a firewall is rejecting connections.)"
  
  
• The wcp (Cluster service) and vstats services on the vCenter Server are in a Stopped state.
  
  
• Manually attempting to start the wcp service fails with the following error: 
  
  Error executing start on service wcp. Details {
    "detail": [
        {
            "id": "install.ciscommon.service.failstart",
            "translatable": "An error occurred while starting service '%(0)s'",
            "args": [
                "wcp"
            ],
            "localized": "An error occurred while starting service 'wcp'"
  
  
• The vpxd log indicate a failure to communicate with the wcp service when the maintenance mode task is initiated.
  /var/log/vmware/vpxd/vpxd.log:
  
  YYYY-MM-DDThh:mm:ss error vpxd[#####] [Originator@6876 sub=MoHost opID=#####] WCP enterMaintenanceMode vAPI returns error: Error:
-->    system_error
--> Messages:
-->    vapi.send.failed<Send of frame failed: N7Vmacore15SystemExceptionE(Connection refused: The remote service is not running, OR is overloaded, OR a firewall is rejecting connections.)
  
  
• To check why the wcp service is offline, proceed with checking the vmon logs. They will show the service actively failing its pre-startup script.
  /var/log/vmware/vmon/vmon.log:
  
  YYYY-MM-DDThh:mm:ss info vmon[#####]: <wcp> Service state changed to STARTING
YYYY-MM-DDThh:mm:ss error vmon[#####]: <wcp> Start control script exited with code 1
YYYY-MM-DDThh:mm:ss error vmon[#####]: <wcp> Service pre-start failed.
  
  
• Reviewing the directory service logs reveals LDAP authentication failures for the machine account, confirming a credential mismatch.
  /var/log/vmware/vmdird/vmdird-syslog.log:
  
  YYYY-MM-DDThh:mm err vmdird t@140620962187008: SASLSessionStep: sasl error (-13) (SASL(-13): authentication failure: client evidence does not match what we calculated. Probably a password error)
YYYY-MM-DDThh:mm err vmdird t@#####: VmDirSendLdapResult: Request (Bind), Error (LDAP_INVALID_CREDENTIALS (49)), Message ((49)(SASL step failed.)), (0) socket (127.0.0.1)
YYYY-MM-DDThh:mm err vmdird t@#####: Bind Request Failed (127.0.0.1) error 49: Protocol version: 3, Bind DN: "cn=[HOSTNAME], ou=Domain Controllers, dc=vsphere, dc=local", Method: SASL
  
  
• Checking the VCSA registry confirms the machine account password exceeds 20 characters:
  /opt/likewise/bin/lwregshell list_values '[HKEY_THIS_MACHINE\services\vmdir]' | egrep -i "Password|dcAccountDN"
  
  Output: 
  +  "dcAccountDN"          REG_SZ          "cn=<VC_FQDN>,ou=Domain Controllers,dc=vsphere,dc=local"
+  "dcAccountOldPassword" REG_SZ          "####################"
+  "dcAccountPassword"    REG_SZ          "######################"
  
  

Environment

VMware vCenter Server

VMware vSphere ESXi

Cause

The vCenter Server machine account password exceeds the supported limit of 20 characters. This discrepancy typically occurs following a password policy update, leading to a mismatch between the stored credentials in the VMDIRD database and the password used by vCenter services. Consequently, the wcp service fails to authenticate and remains offline, which blocks ESXi hosts from entering maintenance mode.

Resolution

A valid snapshot of the vCenter Server is required before proceeding. For best practice refer: Snapshot Best practices for vCenter Server Virtual Machines

To resolve this issue, revert the SSO password policy to a supported length and synchronize the machine account credentials.

1. Revert SSO Password Policy

   1. Log in to the vSphere Client using the SSO administrator account (typically [e-mail]).
   2. Navigate to Administration > Single Sign On > Configuration.
   3. Select the Local Accounts tab, then select Password Policy.
   4. Click Edit and set the Maximum Length to 20 characters.
   5. Click Save.
      
      

2. Reset Machine Account Password

   1. Download the reset_machine_pw.sh script attached to the bottom of this KB article
   2. Using a file transfer utility (such as WinSCP), upload the script to the /tmp directory on the vCenter Server.
   3. Log in to the vCenter Server Appliance via SSH as root.
   4. Navigate to the /tmp directory and make the script executable:
      cd /tmp
chmod +x reset_machine_pw.sh
   5. Execute the script using command: ./reset_machine_pw.sh
      Note: This script will update the machine account password in both the registry and the directory service to a compliant 20-character string.
   6. Run the following command to restart all services: service-control --stop --all && service-control --start --all
   7. Verify that the wcp and vstats services are now in a Running state: service-control --status --all
   8. Retry placing the ESXi host into Maintenance Mode.