Unable to license vCenter server post upgrading VCF operations to 9.1, vCenter does not list in VCF Ops > manage > Licensing > Lincese & Registration > vCenter systems

Products

• VCF Operations

Issue/Introduction

After upgrading to VMware Cloud Foundation (VCF) Operations 9.1, vCenter systems may not appear in the Licensing management UI.
Attempts to register or sync licenses result in an internal server error.
This issue typically occurs when stale vSAN license entries exist within the vCenter LDAP database.

In the VCF Ops UI when recreating the adapter, we see transient error message
Internal Server Error

Logs state the following 
VCF Ops can't license or list the vCenter "Could not retrieve vcenter info"

 # /storage/log/vcops/log/vcf-licensing-plugin-<UUID>.log

<date && TIME> ERROR vcf-licensing-plugin 1200185 [ops@4413 threadId="284" threadName="DistTaskReconcileLicenseServerSchedule" operationId=""] [com.vmware.vrops.licensing.vcf.server.VcfLicensingServer.getVcenterInfo] - Could not retrieve vcenter info data.java.lang.NullPointerException: costUnitId is marked non-null but is null    at com.vmware.statsplatform.persistence.impl.vcenter.license.allocation.usage.asset.record.VcfEntitlementAssetUsageRecord$VcfEntitlementAssetUsageRecordBuilder.costUnitId(VcfEntitlementAssetUsageRecord.java:19) 

 

And that is because of an old license type, that was never properly removed from vCenter

# /var/log/vmware/cis-license/license.log

{urn:vri:com.vmware.license.asset:domain-c<TBD>-<UUID><UUID><UUID><UUID><UUID>
 instanceId: domain-c<TBD>
 scopeId: <UUID><UUID><UUID><UUID><UUID>
 name: <TBD>
 version: null
 productId: '<reference to VMware vSAN Desktop License>'
 cost: [concurrentUser=0, cpuPackage=0, vm=0]
 license: {urn:vri:com.vmware.license.license:<UUID><UUID><UUID><UUID><UUID> name: VDI [<license start>-XXXXX-XXXXX-XXXXX-<license end>]}
},

Environment

VMware Cloud Foundation Operations 9.1 [*upgraded environment]

Cause

The VCF Operations licensing plugin fails to parse license data when an LDAP entry for a vSAN asset exists without a defined cost unit.
This usually happens if a vSAN license was not completely removed during previous administrative actions. 

In clearer terms, every license has 2 important Properties:
Product family  - to what product is the license assignable - i.e. ESX, VSAN, vCenter, VCF Ops, etc.
Cost unit            - this is how the capacity of the license is measured, e.g. cores, TiBs (for vSAN storage) etc.

Simplified VCF Ops, detects a License that should not be present, and that can't be associated with a Product family, or with a Cost Unit.

Resolution

Follow these steps to manually remove the stale license entry from the vCenter LDAP database:

• Log in to the vCenter Server via SSH as root.
• Search for the stale vSAN license entries:
  

  /opt/likewise/bin/ldapsearch -LLL -h localhost -p 389 -x -b "dc=vsphere,dc=local" -D "cn=Administrator,cn=Users,dc=vsphere,dc=local" -s sub '(&(vmwLicSvcObjectClass=AssetEntity)(vmwLicSvcAssetProductName=VMware VSAN))' -W

• Identify the dn (Distinguished Name) of the stale asset (e.g., cn=AssetEntity_domain-####...)
• Create a temporary file to delete the entry or run the following command (replace the dn with your specific ID):
  

  /opt/likewise/bin/ldapmodify -h localhost -D "cn=administrator,cn=users,dc=vsphere,dc=local" -W << EOF
  dn: cn=AssetEntity_domain-####,cn=LicenseService,cn=services,dc=vsphere,dc=local
  changetype: delete
  EOF

• Restart the license service:
  

  # service-control --restart csi-license

• Wait 10 minutes for VCF Operations to synchronize.