Unable to place the host in maintenance mode due to incorrect machine ID in solution users

Products

• VMware vCenter Server
• VMware vCenter Server 7.0
• VMware vCenter Server 8.0

Issue/Introduction

• Entering the host in maintenance mode fails with the following error in vCenter Server UI,

Failed to enter namespaces maintenance mode due to Error: com.vmware.vapi.std.errors.unauthorized Messages: vcenter.wcp.authorization.unauthorized<User is not authorized to perform this operation.> . Retry 53

vpxd.log:

2024-11-23T15:59:56.141Z info vpxd[52361] [Originator@6876 sub=MoHost opID=m2no7mr3-1822663-auto-132dl-h5:70167129-12] WCP enterMaintenanceMode vAPI returns e
rror: Error:
-->    com.vmware.vapi.std.errors.unauthorized
--> Messages:
-->    vcenter.wcp.authorization.unauthorized<User is not authorized to perform this operation.>
-->
2024-11-23T15:59:56.141Z info vpxd[52361] [Originator@6876 sub=MoHost opID=m2no7mr3-1822663-auto-132dl-h5:70167129-12] Waiting 60 secs then invoke WCP enterMaintenanceMode vAPI

 

• Received unauthorized errors for the solution users in wcpsvc logs,

wcpsvc.log:

Environment

VMware vCenter Server 7.x

VMware vCenter Server 8.x

Cause

This issue is due to in-correct machine ID in solution users in the vCenter Server.

1. List the current Solution users in the VCSA by running the following command:

/usr/lib/vmware-vmafd/bin/dir-cli service list

2. Execute the following command to identify the machine ID of the vCenter Server:

/usr/lib/vmware-vmafd/bin/vmafd-cli get-machine-id --server-name localhost

 

The machine ID from step 2 must match the solution users output from step 1, mismatch in the output states that it has incorrect machine ID in solution users.

Resolution

Perform the following steps to resolve this issue.

1. Take a no memory snapshot of the vCenter Server if it is a standalone server or else powered off snapshots of all vCenter Servers if they are in Enhanced Linked Mode (ELM) 

2. Connect the vCenter Server using JXplorer: Connect to JXplorer

3. Remove the solution user entries of the affected vCenter Server from JXplorer as follows,

• • vsphere --> Service Principals --. Right click and delete the solution user with incorrect machine ID

 

4. Once the incorrect solution users are removed re-create the solution users with the shell script by referring the KB: Recreate Solution Users for VCSA via Shell script

5. Restart the vCenter Server services: 

service-control --stop --all && service-control --start --all

6. Post recreating the solution users still the service registrations would hold the incorrect machine IDs which can be resolved by rebuilding service registrations using "python lsdoctor.py -r" script,

• https://knowledge.broadcom.com/external/article/320837/using-the-lsdoctor-tool.html

7. Execute the following commands to validate the machine ID and solution user, both should be same.

/usr/lib/vmware-vmafd/bin/vmafd-cli get-machine-id --server-name localhost

/usr/lib/vmware-vmafd/bin/dir-cli service list

8. Place the host in maintenance mode from the vCenter Server