Entering ESXi maintenance mode fails with error: "Failed to enter namespaces maintenance"

Products

• VMware vCenter Server
• VMware vCenter Server 7.0

Issue/Introduction

• When placing an ESXi host into maintenance mode from the vSphere console, it may fail with the following message:
  
  Failed to enter namespaces maintenance mode due to Error: com.vmware.vapi.std.errors.unauthenticated Messages: vapi.security.authentication.invalid<Unable to authenticate user>. Retry XX


• The following may be seen in /var/log/vmware/vpxd/vpxd.log:

[YYYY-MM-DDTHH:MM:SS] error vpxd[57646] [Originator@6876 sub=MoHost opID=abcabcabcd-861-auto-ny-h5:70000172-1e] [Delete] Failed to delete vAPI session. Error:
--> Error:
-->  com.vmware.vapi.std.errors.unauthenticated
--> Messages:
-->  vapi.security.authentication.invalid<Unable to authenticate user>
[YYYY-MM-DDTHH:MM:SS] warning vpxd[57646] [Originator@6876 sub=MoHost opID=abcabcabcd-861-auto-ny-h5:70000172-1e] [Close] Host 'localhost' Failed to delete Session: N3Vim5Fault12InvalidLogin9ExceptionE(Fault cause: vim.fault.InvalidLogin

[YYYY-MM-DDTHH:MM:SS] warning vpxd[57646] [Originator@6876 sub=MoHost opID=abcabcabcd-861-auto-ny-h5:70000172-1e] [Invoke] Host 'localhost' Failed to acquire Session: N3Vim5Fault12InvalidLogin9ExceptionE(Fault cause: vim.fault.InvalidLogin

[YYYY-MM-DDTHH:MM:SS] info vpxd[57646] [Originator@6876 sub=MoHost opID=abcabcabcd-861-auto-ny-h5:70000172-1e] WCP enterMaintenanceMode vAPI returns error: Error:
-->  com.vmware.vapi.std.errors.unauthenticated
--> Messages:
-->  vapi.security.authentication.invalid<Unable to authenticate user>
[YYYY-MM-DDTHH:MM:SS] info vpxd[57646] [Originator@6876 sub=MoHost opID=abcabcabcd-861-auto-ny-h5:70000172-1e] Waiting 60 secs then invoke WCP enterMaintenanceMode vAPI


Environment

VMware vCenter Server 7.x

Cause

The error can be due to a missing service registration of 'tokenservice'.

Resolution

1. Take offline snapshots for all vCenters in the Enhanced Linked mode (ELM) cluster.
   
   
2. List all registered services:
   $ /usr/lib/vmware-lookupsvc/tools/lstool.py list --url http://localhost:7090/lookupservice/sdk --no-check-cert >> /tmp/service.txt
   
   
3. Check for the 'tokenservice' Service Registration:
   $ cat /tmp/service.txt |grep -i "service type: tokenservice" -A10 |egrep -i "Service Type|Service ID|URL"
   
   
4. Confirm there is a "Service Type: tokenservice" for each vCenter by looking at the associated URL line. There should be one tokenservice per vCenter.
   $ cat /tmp/service.txt |grep -i "service type: tokenservice" -A10 |egrep -i "Service Type|Service ID|URL"
   
   Output:
   Service Type: tokenservice
        Service ID: b5bfaaff-9b9e-464b-b64f-a2a3b4c5d6e7
                URL: https://vcsa1.example.com:443/tokenservice/vapi
Service Type: tokenservice
        Service ID: 1ca9e6a4-1e26-4466-bcfd-f0a7b1c1e7a4
                URL: https://vcsa2.example.com:443/tokenservice/vapi
   
   

If the service is missing, use the lsdoctor tool to register in a new 'tokenservice' registration.

Using the 'lsdoctor' Tool

• Use lsdoctor -r > Option 3 (Replace individual service)
• Enter the number associated for the token service when prompted

    5. Restart all vCenter Services
    $ service-control --stop --all && service-control --start --all

Additional Information

If the issue is caused due to 'WCP' solution user certificate, follow the steps in KB, "com.vmware.vapi.std.errors.unauthenticated" and "vapi.security.authentication.invalid" errors for the WCP service causing multiple workflow failures to regenerate the WCP solution user certificate.