Unable to add an upgraded host to vCenter/cluster with error "cannot complete the license assignment operation. license service on server is unavailable"

Products

• VMware vSphere ESXi
• VMware vCenter Server

Issue/Introduction

• vCenter is part of Enhanced linked mode.
  
  
• Attempting to add a host to the vCenter after a recent upgrade fails with the error "cannot complete the license assignment operation. license service on server is unavailable"

• /var/log/vmware/vmdird/vmdird-syslog.log

  yyyy-mm-ddTHH:MM:SS err vmdird t@139743522182912: InternalModifyEntry: VdirExecutePostModifyCommitPlugins - code (9114)
yyyy-mm-ddTHH:MM:SS err vmdird t@139743522182912: VmDirSendLdapResult: Request (Modify), Error (LDAP_UNWILLING_TO_PERFORM(53)), Message (Server in read-only mode), (0) socket (127.0.0.1)
yyyy-mm-ddTHH:MM:SS err vmdird t@139743949981440: VmDirSafeLDAPBindEx to (ldap://x.x.x.x:389) failed. SRP(9127)
yyyy-mm-ddTHH:MM:SS err vmdird t@139743949981440: VmDirSafeLDAPBindEx to (ldap://x.x.x.x:389) failed. SRP(9127)
yyyy-mm-ddTHH:MM:SS err vmdird t@139743949981440:_VmDirConsumePartner: vmdir is in READ_ONLY state
yyyy-mm-ddTHH:MM:SS err vmdird t@139743497004800:ParseSyncRequestControlVal: ServerId ########## has processed my USN (1621126), my max USN is (1619539)
yyyy-mm-ddTHH:MM:SS err vmdird t@139743497004800:ParseRequestControls: ParseSyncRequestControlVal failed.
yyyy-mm-ddTHH:MM:SS err vmdird t@139743949981440: VmDirSafeLDAPBindEx to (ldap://x.x.x.x:389) failed. SRP(9127)
yyyy-mm-ddTHH:MM:SS err vmdird t@139743949981440: VmDirSafeLDAPBindEx to (ldap://x.x.x.x:389) failed. SRP(9127)
yyyy-mm-ddTHH:MM:SS err vmdird t@139743949981440:VmDirConsumePartner: vmdir is in READ_ONLY state
yyyy-mm-ddTHH:MM:SS info vmdird t@139744241080064:VmDir State (3)
yyyy-mm-ddTHH:MM:SS info vmdird t@139744241080064:Srv RpcVmDirSetState: VmDir State (3)
yyyy-mm-ddTHH:MM:SS err vmdird t@139743497004800:ParseSyncRequestControlVal: ServerId ############ has processed my USN (1621126), my max USN is (1619539).
yyyy-mm-ddTHH:MM:SS err vmdird t@139743497004800:ParseSyncRequestControlVal: To prevent data loss, vmdir will be set to READ-ONLY mode , retVal [-1], errCode [53]
yyyy-mm-ddTHH:MM:SS info vmdird t@139743497004800: VmDir State (2)
yyyy-mm-ddTHH:MM:SS err vmdird t@139743497004800:ParseRequestControls: ParseSyncRequestControlVal failed.

• vpxd.log - /var/log/vmware/vpxd/vpxd.log 

yyyy-mm-ddTHH:MM:SS info vpxd[15501] [Originator@6876 sub=vpxLro opID=mc####k-11####1-auto-oqzy-h5:#####37-93-01-01] [VpxLRO] -- FINISH task-24693908
yyyy-mm-ddTHH:MM:SS info vpxd[15501] [Originator@6876 sub=Default opID=mc####k-11####1-auto-oqzy-h5:#####37-93-01-01] [VpxLRO] -- ERROR task-24693908 -- host -- Folder.addStandaloneHost: vim.fault.LicenseAssignmentFailed:
--> Result:
--> (vim.fault.LicenseAssignmentFailed) {
-->    faultCause = (vmodl.MethodFault) null,
-->    faultMessage = (vmodl.LocalizableMessage) [
-->       (vmodl.LocalizableMessage) {
-->          key = "com.vmware.vim.vc.license.error.licenseAssignmentFailedNoLicenseService",
-->          arg = (vmodl.KeyAnyValue) [
-->             (vmodl.KeyAnyValue) {
-->                key = "entityType",
-->                value = "hostEntity"
-->             },
-->             (vmodl.KeyAnyValue) {
-->                key = "entityDisplayName",
-->                value = "host_fqdn"
-->             }
-->          ],
-->          message = <unset>
-->       }
-->    ],
-->    reason = <unset>
-->    msg = ""
--> }
--> Args:

Environment

vCenter server 7.x and 8.x

Cause

This issue occurs when one of the vCenters in linked mode has VMDir in read-only mode. VMDir goes back to read-only mode after a few seconds even after setting it to normal using /usr/lib/vmware-vmdir/bin/vdcadmintool

 

Resolution

To check the status of VMdir:

• SSH to VCenter server appliance and run below command

/usr/lib/vmware-vmdir/bin/vdcadmintool

• Select option 6 to get vmdir state

To resolve the issue, proceed with the following steps:

• Take offline snapshots of all the vCenters in Enhanced linked mode before running the below command.
  
  
• SSH to the affected vCenter server appliance and run

     /usr/lib/vmware-vmdir/sbin/vmdird -c -m restore

• Restart vmdird service.

     service-control --restart vmdird

Additional Information

The VMDIR State can also be fixed using the Fix PSC script using the rebuild function - Fix PSC/vmdir inconsistencies using fixpsc python script