Unable to place the host in maintenance mode due to incorrect machine ID in solution users

Products

VMware vCenter Server VMware vCenter Server 7.0 VMware vCenter Server 8.0

Issue/Introduction

Entering the host in maintenance mode fails with the following error in vCenter Server UI,

Failed to enter namespaces maintenance mode due to Error: com.vmware.vapi.std.errors.unauthorized Messages: vcenter.wcp.authorization.unauthorized<User is not authorized to perform this operation.> . Retry 53

vpxd.log:

YYYY-MM-DDTHH:MM:SS.141Z info vpxd[52361] [Originator@6876 sub=MoHost opID=m2no7mr3-xxxxx63-auto-132dl-h5:70167129-12] WCP enterMaintenanceMode vAPI returns e rror: Error:--> com.vmware.vapi.std.errors.unauthorized --> Messages: --> vcenter.wcp.authorization.unauthorized<User is not authorized to perform this operation.>-->

YYYY-MM-DDTHH:MM:SS.141Z info vpxd[52361] [Originator@6876 sub=MoHost opID=m2no7mr3-xxxxx63-auto-132dl-h5:70167129-12] Waiting 60 secs then invoke WCP enterMaintenanceMode vAPI

Received unauthorized errors for the solution users in wcpsvc logs,

wcpsvc.log:

YYYY-MM-DDTHH:MM:SS.973Z debug wcp [vapiauth/permission_validator.go:56] [opID=wcp-AuthzFilter] Validating permissions; operation: com. vmware. vcenter.namespace_management.nodes.enter_maintenance_mode, requested: map[{isOperation:false id:com.vmware.vcenter.namespace_management
resourceType: }: [] {isOperation:true id:com.vmware. vcenter.namespace_management.nodes.enter_maintenance_mode resourceType:}: [System.Anonymous]]

YYYY-MM-DDTHH:MM:SS.973Z debug wcp [namespace/authz.go:59] [opID=wcp-AuthzFilter] Checking privileges for username: vsphere.local\[email protected] groupnames: [[email protected] [email protected] [email protected] [email protected] [email protected]], resIDs: [{PermissionFolder global-permission}], privs: [System. Anonymous]

YYYY-MM-DDTHH:MM:SS.977Z error wcp [wcp/nodes.go:39] [opID=67xxxxx5c] Nodes API auth error: com.vmware.vapi.std.errors.unauthorized

Cause

This issue is due to in-correct machine ID in solution users in the vCenter Server.

List the current Solution users in the VCSA by running the following command:

/usr/lib/vmware-vmafd/bin/dir-cli service list

Execute the following command to identify the machine ID of the vCenter Server:

/usr/lib/vmware-vmafd/bin/vmafd-cli get-machine-id --server-name localhost

The machine ID from step 2 must match the solution users output from step 1, mismatch in the output states that it has incorrect machine ID in solution users.

Resolution

Perform the following steps to resolve this issue.

1. Take a no memory snapshot of the vCenter Server if it is a standalone server or else powered off snapshots of all vCenter Servers if they are in Enhanced Linked Mode (ELM)

2. Connect the vCenter Server using JXplorer: Connect to JXplorer

3. Remove the solution user entries of the affected vCenter Server from JXplorer as follows,

- vsphere --> Service Principals --. Right click and delete the solution user with incorrect machine ID

4. Once the incorrect solution users are removed re-create the solution users using “python lsdoctor.py -u” script using the lsdoctor tool from KB Using the 'lsdoctor' Tool

5. Restart the vCenter Server services:

service-control --stop --all && service-control --start --all

6. Post recreating the solution users still the service registrations would hold the incorrect machine IDs which can be resolved by rebuilding service registrations using "python lsdoctor.py -r" script from KB Using the 'lsdoctor' Tool

7. Execute the following commands to validate the machine ID and solution user, both should be same.

/usr/lib/vmware-vmafd/bin/vmafd-cli get-machine-id --server-name localhost

/usr/lib/vmware-vmafd/bin/dir-cli service list

8. Place the host in maintenance mode from the vCenter Server