ESXi maintenance mode fails "Failed to enter namespaces maintenance"

Products

• VMware vCenter Server
• VMware vCenter Server 7.0

Issue/Introduction

• When putting an ESXi host into maintenance mode from the vCenter console, it fails with the following message:
  
  Failed to enter namespaces maintenance mode due to Error: com.vmware.vapi.std.errors.unauthenticated Messages: vapi.security.authentication.invalid<Unable to authenticate user>. Retry XX


• The following is seen in vCenter - /var/log/vmware/vpxd/vpxd.log:

[YYYY-MM-DDTHH:MM:SS] error vpxd[57646] [Originator@6876 sub=MoHost opID=krlaondk-861-auto-ny-h5:70000172-1e] [Delete] Failed to delete vAPI session. Error:
--> Error:
-->  com.vmware.vapi.std.errors.unauthenticated
--> Messages:
-->  vapi.security.authentication.invalid<Unable to authenticate user>
[YYYY-MM-DDTHH:MM:SS] warning vpxd[57646] [Originator@6876 sub=MoHost opID=krlaondk-861-auto-ny-h5:70000172-1e] [Close] Host 'localhost' Failed to delete Session: N3Vim5Fault12InvalidLogin9ExceptionE(Fault cause: vim.fault.InvalidLogin

[YYYY-MM-DDTHH:MM:SS] warning vpxd[57646] [Originator@6876 sub=MoHost opID=krlaondk-861-auto-ny-h5:70000172-1e] [Invoke] Host 'localhost' Failed to acquire Session: N3Vim5Fault12InvalidLogin9ExceptionE(Fault cause: vim.fault.InvalidLogin

[YYYY-MM-DDTHH:MM:SS] info vpxd[57646] [Originator@6876 sub=MoHost opID=krlaondk-861-auto-ny-h5:70000172-1e] WCP enterMaintenanceMode vAPI returns error: Error:
-->  com.vmware.vapi.std.errors.unauthenticated
--> Messages:
-->  vapi.security.authentication.invalid<Unable to authenticate user>
[YYYY-MM-DDTHH:MM:SS] info vpxd[57646] [Originator@6876 sub=MoHost opID=krlaondk-861-auto-ny-h5:70000172-1e] Waiting 60 secs then invoke WCP enterMaintenanceMode vAPI


Environment

VMware vCenter Server 7.0.2

Cause

This can be caused due to a missing service registration of 'tokenservice'.

Resolution

1. Take offline snapshots for all vCenters in ELM.
   
   Note: If it is a standalone vCenter, a powered-on snapshot is sufficient.
   
   
2. List all registered services:
   
   /usr/lib/vmware-lookupsvc/tools/lstool.py list --url http://localhost:7090/lookupservice/sdk --no-check-cert >> /tmp/service.txt
   
   
3. Check for the 'tokenservice' Service Registration:
   
   cat /tmp/service.txt |grep -i "service type: tokenservice" -A10 |egrep -i "Service Type|Service ID|URL"
   
   
4. Confirm there is a "Service Type: tokenservice" for each vCenter by looking at the associated URL line. There should be one tokenservice per vCenter.
   
   Example
   
   cat /tmp/service.txt |grep -i "service type: tokenservice" -A10 |egrep -i "Service Type|Service ID|URL"
   
   Output
   
   Service Type: tokenservice
        Service ID: b5bf258f-9b9e-464b-b64f-a2c1b853d874
                URL: https://vcsa1.example.com:443/tokenservice/vapi
Service Type: tokenservice
        Service ID: 1ca9e6a4-1e26-4466-bcfd-c027b151f74a
                URL: https://vcsa2.example.com:443/tokenservice/vapi
   
   In the above example output, there are 2 vCenters in this ELM environment and each has their own tokenservice.
   
   
5. If the service is missing, use the lsdoctor tool found in the below KB to register in a new "token service" registration.
   • Using the 'lsdoctor' Tool
   • Use lsdoctor -r > Option 3 (Replace individual service)
   • Enter the number associated for the token service when prompted
     
     
6. Restart all vCenter Services
   
   service-control --stop --all && service-control --start --all

Additional Information

If the issue is caused due to 'WCP' solution user certificate, follow the steps in KB "com.vmware.vapi.std.errors.unauthenticated" and "vapi.security.authentication.invalid" errors for the WCP service causing multiple workflow failures to regenerate the certificate.