ESXi maintenance mode fails "Failed to enter namespaces maintenance"

Products

VMware vCenter Server VMware vCenter Server 7.0

Issue/Introduction

Symptoms:

When putting an ESXi host into maintenance mode from the vCenter console, it fails with the following message:
Failed to enter namespaces maintenance mode due to Error: com.vmware.vapi.std.errors.unauthenticated Messages: vapi.security.authentication.invalid<Unable to authenticate user>. Retry XX

vpxd.log:

2021-07-27T01:56:14.324Z error vpxd[57646] [Originator@6876 sub=MoHost opID=krlaondk-861-auto-ny-h5:70000172-1e] [Delete] Failed to delete vAPI session. Error:
--> Error:
-->  com.vmware.vapi.std.errors.unauthenticated
--> Messages:
-->  vapi.security.authentication.invalid<Unable to authenticate user>
2021-07-27T01:56:14.324Z warning vpxd[57646] [Originator@6876 sub=MoHost opID=krlaondk-861-auto-ny-h5:70000172-1e] [Close] Host 'localhost' Failed to delete Session: N3Vim5Fault12InvalidLogin9ExceptionE(Fault cause: vim.fault.InvalidLogin
[context]zKq7AVECAQAAAEcGEgEadnB4ZAAAUnE0bGlidm1hY29yZS5zbwAACaUpABGaKgAjZC8BD6ZfdnB4ZAABa3VtASTf0QHHzWMByKRfAUhMZQFaUGUBpPVnAVwJaAGRGGgCp7j2bGlidmltLXR5cGVzLnNvAIHZBCoBgXBdKAGBEpkoAYG7qCgBgVc0KAGBDgspAQDxQCAAAp0gACYANAOHfwBsaWJwdGhyZWFkLnNvLjAABL81D2xpYmMuc28uNgA=[/context]
2021-07-27T01:56:14.365Z warning vpxd[57646] [Originator@6876 sub=MoHost opID=krlaondk-861-auto-ny-h5:70000172-1e] [Invoke] Host 'localhost' Failed to acquire Session: N3Vim5Fault12InvalidLogin9ExceptionE(Fault cause: vim.fault.InvalidLogin
[context]zKq7AVECAQAAAEcGEgEZdnB4ZAAAUnE0bGlidm1hY29yZS5zbwAACaUpABGaKgAjZC8BD6ZfdnB4ZAABa3VtAQPi0QGORmUBVEtlAVpQZQGk9WcBXAloAZEYaAKnuPZsaWJ2aW0tdHlwZXMuc28AgdkEKgGBcF0oAYESmSgBgbuoKAGBVzQoAYEOCykBAPFAIAACnSAAJgA0A4d/AGxpYnB0aHJlYWQuc28uMAAEvzUPbGliYy5zby42AA==[/context]
2021-07-27T01:56:14.370Z info vpxd[57646] [Originator@6876 sub=MoHost opID=krlaondk-861-auto-ny-h5:70000172-1e] WCP enterMaintenanceMode vAPI returns error: Error:
-->  com.vmware.vapi.std.errors.unauthenticated
--> Messages:
-->  vapi.security.authentication.invalid<Unable to authenticate user>
2021-07-27T01:56:14.370Z info vpxd[57646] [Originator@6876 sub=MoHost opID=krlaondk-861-auto-ny-h5:70000172-1e] Waiting 60 secs then invoke WCP enterMaintenanceMode vAPI

Environment

VMware vCenter Server 7.0.2

Cause

This can be caused due to a missing service registration of ‘tokenservice’.

Resolution

1. Take Offline snapshots for all vCenter's in ELM
Note: If it is a standalone vCenter, you may take a powered-on snapshot.

2. List all registered services:

/usr/lib/vmware-lookupsvc/tools/lstool.py list --url http://localhost:7090/lookupservice/sdk --no-check-cert >> /tmp/service.txt

Check for the 'tokenservice' Service Registration

cat /tmp/service.txt |grep -i "service type: tokenservice" -A10 |egrep -i "Service Type|Service ID|URL"

Confirm there is a "Service Type: tokenservice" for each vCenter by looking at the associated URL line. There should be one tokenservice per vCenter.

Example:
cat /tmp/service.txt |grep -i "service type: tokenservice" -A10 |egrep -i "Service Type|Service ID|URL"

Output:
Service Type: tokenservice
Service ID: b5bf258f-9b9e-464b-b64f-a2c1b853d874
URL: https://vcsa1.example.com:443/tokenservice/vapi
Service Type: tokenservice
Service ID: 1ca9e6a4-1e26-4466-bcfd-c027b151f74a
URL: https://vcsa2.example.com:443/tokenservice/vapi

In the above example output, there are 2 vCenters in this ELM environment and each has their own tokenservice.

If the service is missing, use the lsdoctor tool found in the below KB to register in a new "token service" registration.
- * Using the 'lsdoctor' Tool (80469) - https://kb.vmware.com/s/article/80469
- Use lsdoctor -r > Option 3 (Replace individual service)
  KB article for lsdoctor :- 320837

Example of the prompt:

========================
0. Exit
1. Generate a template.
2. Replace all services with new services.
3. Replace individual service.
4. Restore services from backup file.
========================

Enter the number associated for the token service when prompted

Restart all vCenter Services

service-control --stop --all && Service-control --start -aal

Additional Information

If the issue is caused due to 'WCP' solution user certificate, please follow the steps in KB "com.vmware.vapi.std.errors.unauthenticated" and "vapi.security.authentication.invalid" errors for the WCP service causing multiple workflow failures to regenerate the certificate.