Error: "One or more vCenter instances are not connected to a license server" after upgrading to VMware Cloud Foundation 9.1

Products

• VCF Operations
• VMware Cloud Foundation

Issue/Introduction

• After upgrading VMware Cloud Foundation (VCF) and vCenter Server to version 9.1, a red banner appears in the vSphere Client. The warning indicates that vCenter instances are not connected to a license server, even after a standalone license server appliance is successfully deployed and registered.
  
  
• The vCenter Server displays the following banner:  One or more vCenter instances are not connected to a license server. To avoid service interruption, verify that the vCenter instance is connected to a license server.
  
  
• Additionally, in vCenter /var/log/vmware/cis-license/license.log contains java.lang.NullPointerException errors similar to the following:
  
  

  #### ERROR ManagementAdapter #### [ops@#### threadName="vcf-licensing-assign-####"] - Error while assigning entitlement to VC: #### at [VC-FQDN] java.lang.NullPointerException

  YYYY-MM-DDTHH:MM:SS.815Z ERROR ManagementAdapter ##### [ops@#### threadId="######" threadName="vcf-licensing-assign-####" instanceId="##"] [(##) com.vmware.adapter.management.components.licensing.vcf.H.A] - Error while assigning entitlement to VC: #########-####-####-####-############### at [VC-FQDN] java.lang.NullPointerException: Cannot invoke "com.vmware.vapi.bindings.Structure._hasTypeNameOf(java.lang.Class)" because "<local3>" is null
  YYYY-MM-DDTHH:MM:SS.274Z INFO vcf-licensing-plugin ##### [ops@#### threadId="#####" threadName="pool-40-thread-4" operationId=""] [com.vmware.vrops.licensing.vcf.helper.EntitlementAutomaticReassignmentService.runAutomaticReassign] - Checking whether allocations [#########-####-####-####-###############, #########-####-####-####-###############] should be migrated.
  YYYY-MM-DDTHH:MM:SS.279Z INFO vcf-licensing-plugin ##### [ops@#### threadId="#####" threadName="pool-40-thread-4" operationId=""] [com.vmware.vrops.licensing.vcf.helper.EntitlementAutomaticReassignmentService.runAutomaticReassign] - Starting migration for allocations [#########-####-####-####-###############, #########-####-####-####-###############]
  YYYY-MM-DDTHH:MM:SS.280Z INFO vcf-licensing-plugin ##### [ops@#### threadId="#####" threadName="pool-40-thread-4" operationId=""] [com.vmware.vrops.licensing.vcf.helper.EntitlementAutomaticReassignmentService.runAutomaticReassign] - Reassigning entitlements [#########-####-####-####-###############].

Environment

• VMware Cloud Foundation 9.x
  
  
• VMware vCenter Server 9.x

Cause

This issue occurs when a stale LicenseEntity exists in the vCenter LDAP database, often due to an upgrade path where a vSAN (TiB) license was assigned without an active vSAN cluster, or when synchronization fails between the new 9.1 License Server and the vCenter instance.

Resolution

To manually clear the stale license entities, follow these steps:

1. Create an offline snapshot or backup of the vCenter Server Appliance.

2. Open an SSH session to the vCenter Server Appliance and log in as the root user.

3. Execute the following ldapsearch command to list the LicenseEntity and AssetEntity objects. Enter the LDAP password for [email protected] when prompted:
   
   /opt/likewise/bin/ldapsearch -b "cn=LicenseService,cn=Services,dc=vsphere,dc=local" -D "cn=Administrator,cn=Users,dc=vsphere,dc=local" -s sub "(|(objectclass=vmwLicSvcAssetEntity)(objectclass=vmwLicSvcLicenseEntity))" cn vmwLicSvcAssetName vmwLicSvcAssetProductName vmwLicSvcAssetProductVersion vmwLicSvcAssetLicenseId vmwLicSvcLicenseId vmwLicSvcLicenseName vmwLicSvcLicenseSerialKeys -W

4. From the output, locate the specific LicenseEntity that needs to be removed (e.g., the stale VMware vSAN license). Note the full dn (Distinguished Name) string. Example: 
   
   dn: cn=LicenseEntity_#######-####-####-####-############,cn=LicenseService,cn=services,dc=vsphere,dc=local
cn: LicenseEntity_#######-####-####-####-############
vmwLicSvcLicenseName: 'VMware vSAN (TiB)'
vmwLicSvcLicenseSerialKeys: #######-####-####-####-############
   
   
5. Use a text editor (like Notepad) to prepare the following ldapmodify command, replacing the dn line with the exact value retrieved in Step 4. Paste the block as a whole into the SSH session: 
   
   /opt/likewise/bin/ldapmodify -h localhost -D "cn=administrator,cn=users,dc=vsphere,dc=local" -W << EOF
dn: cn=LicenseEntity_#######-####-####-####-############,cn=LicenseService,cn=services,dc=vsphere,dc=local
changetype: delete
EOF

Example Output:
   Enter LDAP Password:
deleting entry "cn=LicenseEntity_#######-####-####-####-############cn=LicenseService,cn=services,dc=vsphere,dc=local"

6. Verify the LicenseEntity was successfully removed by running the ldapsearch command from Step 3 again. 

    
7. Restart the vmware-cis-license service.
   
   service-control --restart vmware-cis-license
   
   
8. Log in to the vSphere Client and navigate to Administration > Licensing > Licenses to confirm the license is removed.

Additional Information

• CAUTION: This article outlines how to remove a vCenter Server License. Do not remove a valid vCenter Server License that is currently in use. Only remove the targeted stale or expired LicenseEntity.

• This article provides steps to remove a LicenseEntity. Removing the AssetEntity is not required or recommended.

• Any licensed assets that were using the removed license will become unlicensed and must be relicensed accordingly.

• For further reference on manual license removal, see: How to remove a License Manually from vCenter using ldapmodify.