Error: One or more vCenter instances are not connected to a license server

Products

VCF Operations VMware Cloud Foundation

Issue/Introduction

A red banner appears in the vSphere Client after upgrading VMware Cloud Foundation (VCF) and vCenter Server to version 9.1. The warning indicates that vCenter instances are not connected to a license server, even after a standalone license server appliance is deployed and registered.
vCenter displays the banner: One or more vCenter instances are not connected to a license server. To avoid service interruption, verify that the vCenter instance is connected to a license server.
The vCenter cis-license.log contains errors similar to: #### ERROR ManagementAdapter #### [ops@#### threadName="vcf-licensing-assign-####"] - Error while assigning entitlement to VC: #### at [Redacted-VC-FQDN] java.lang.NullPointerException

YYYY-MM-DDTHH:MM:SS.815Z ERROR ManagementAdapter ##### [ops@#### threadId="######" threadName="vcf-licensing-assign-####" instanceId="##"] [(##) com.vmware.adapter.management.components.licensing.vcf.H.A] - Error while assigning entitlement to VC: #########-####-####-####-############### at [Redacted-VC-FQDN] java.lang.NullPointerException: Cannot invoke "com.vmware.vapi.bindings.Structure._hasTypeNameOf(java.lang.Class)" because "<local3>" is null
YYYY-MM-DDTHH:MM:SS.274Z INFO vcf-licensing-plugin ##### [ops@#### threadId="#####" threadName="pool-40-thread-4" operationId=""] [com.vmware.vrops.licensing.vcf.helper.EntitlementAutomaticReassignmentService.runAutomaticReassign] - Checking whether allocations [#########-####-####-####-###############, #########-####-####-####-###############] should be migrated.
YYYY-MM-DDTHH:MM:SS.279Z INFO vcf-licensing-plugin ##### [ops@#### threadId="#####" threadName="pool-40-thread-4" operationId=""] [com.vmware.vrops.licensing.vcf.helper.EntitlementAutomaticReassignmentService.runAutomaticReassign] - Starting migration for allocations [#########-####-####-####-###############, #########-####-####-####-###############]
YYYY-MM-DDTHH:MM:SS.279Z INFO vcf-licensing-plugin ##### [ops@#### threadId="######" threadName="pool-40-thread-4" operationId=""] [com.vmware.vrops.licensing.vcf.helper.EntitlementAutomaticReassignmentService.lambda$runAutomaticReassign$2] - Will migrate entitlement 6#########-####-####-####-############### to #########-####-####-####-###############
YYYY-MM-DDTHH:MM:SS.279Z INFO vcf-licensing-plugin ##### [ops@#### threadId="#####" threadName="pool-40-thread-4" operationId=""] [com.vmware.vrops.licensing.vcf.helper.EntitlementAutomaticReassignmentService.lambda$runAutomaticReassign$2] - Will migrate entitlement #########-####-####-####-############### to #########-####-####-####-###############
YYYY-MM-DDTHH:MM:SS.280Z INFO vcf-licensing-plugin ##### [ops@#### threadId="#####" threadName="pool-40-thread-4" operationId=""] [com.vmware.vrops.licensing.vcf.helper.EntitlementAutomaticReassignmentService.runAutomaticReassign] - Reassigning entitlements [#########-####-####-####-###############].

Environment

VCF 9.1.x
VCF Operations 9.1.x

Cause

This issue occurs when a stale LicenseEntity exists in the vCenter LDAP database, often due to an upgrade path where a vSAN (TiB) license was assigned without an active vSAN cluster, or when synchronization fails between the new 9.1 License Server and the vCenter instance.

Resolution

To manually clear the stale license entities, follow these steps:

Create a snapshot or backup of the vCenter Server Appliance.
Open an SSH session to the vCenter as the root user.
Execute the following command to list LicenseEntity and AssetEntity objects:
1. Note, paste the entire contents of the ldapsearch command on the vCenter bash command line.
  
  /opt/likewise/bin/ldapsearch -b "cn=LicenseService,cn=Services,dc=vsphere,dc=local" -D "cn=Administrator,cn=Users,dc=vsphere,dc=local" -s sub "(|(objectclass=vmwLicSvcAssetEntity)(objectclass=vmwLicSvcLicenseEntity))" cn vmwLicSvcAssetName vmwLicSvcAssetProductName vmwLicSvcAssetProductVersion vmwLicSvcAssetLicenseId vmwLicSvcLicenseId vmwLicSvcLicenseName vmwLicSvcLicenseSerialKeys -W
  
  When prompted, enter the LDAP Password for [email protected]
From the previous output, locate the LicenseEntity to remove:

In this example, we will remove,

dn: cn=LicenseEntity_#######-####-####-####-############,cn=LicenseService,cn =services,dc=vsphere,dc=local
cn: LicenseEntity_#######-####-####-####-############
vmwLicSvcLicenseName: 'VMware vSAN (TiB)'
vmwLicSvcLicenseSerialKeys: #######-####-####-####-############
1. Note: Paste the entire contents of the ldapmodify command to the second EOF on the vCenter bash command line. Make sure to have the correct 'dn' of the LicenseEntity to remove.
2. Note: Use notepad to prepare the ldapmodify with the correct dn...(The 4 separate lines below will need to be copied and paste as a whole and not separate lines into the SSH session)
  
  /opt/likewise/bin/ldapmodify -h localhost -D "cn=administrator,cn=users,dc=vsphere,dc=local" -W << EOF
  dn: cn=LicenseEntity_#######-####-####-####-############,cn=LicenseService,cn=services,dc=vsphere,dc=local
  changetype: delete
  EOF
  
  Example Output:
  
  Enter the LDAP Password when prompted. Note that the entry is deleted
  
  Enter LDAP Password:
  
  deleting entry "cn=LicenseEntity_#######-####-####-####-############cn=LicenseService,cn=services,dc=vsphere,dc=local"
  
  Verify the LicenseEntity was removed using an ldapsearch query:
  
  Note, paste the entire contents of the ldapsearch command on the vCenter bash command line..
  
  /opt/likewise/bin/ldapsearch -b "cn=LicenseService,cn=Services,dc=vsphere,dc=local" -D "cn=Administrator,cn=Users,dc=vsphere,dc=local" -s sub "(|(objectclass=vmwLicSvcAssetEntity)(objectclass=vmwLicSvcLicenseEntity))" cn vmwLicSvcAssetName vmwLicSvcAssetProductName vmwLicSvcAssetProductVersion vmwLicSvcAssetLicenseId vmwLicSvcLicenseId vmwLicSvcLicenseName vmwLicSvcLicenseSerialKeys -W
Verify from the vSphere Client UI -> Administration -> Licensing -> Licenses that the License is removed:

Note, a UI refresh may be needed. A reboot or restart of services is not necessary.

Additional Information

Note: Licensed assets using the license will become unlicensed and need to be relicensed accordingly.

This article provides an example to remove a "LicenseEntity". Removing the "AssetEntity" is not required or recommended.

CAUTION: This article shows how to remove a vCenter Server License. Do not remove the vCenter Server License if it is valid and in use. Only remove the desired stale or expired LicenseEntity.

How to remove a License Manually from vCenter using ldapmodify