Unable to license vCenter server post upgrading VCF operations to 9.1, vCenter does not list in VCF Ops > manage > Licensing > License & Registration > vCenter systems

Products

• VCF Operations

Issue/Introduction

• After upgrading to VMware Cloud Foundation (VCF) Operations 9.1, vCenter Server instances may fail to display in the Licensing management UI under Manage > Licensing > License & Registration > vCenter Systems
• Attempts to register or sync licenses result in an Internal Server Error.
• This issue typically occurs when stale vSAN license entries exist within the vCenter LDAP database.
  
  
  
• In the VCF Ops UI when recreating the adapter, we see transient error message : Internal Server Error
• As per the logs under /storage/log/vcops/log/vcf-licensing-plugin-<UUID>.log reports VCF Ops can't license or list the vCenter "Could not retrieve vCenter info"
  

  <date && TIME> ERROR vcf-licensing-plugin 1200185 [ops@4413 threadId="284" threadName="DistTaskReconcileLicenseServerSchedule" operationId=""] [com.vmware.vrops.licensing.vcf.server.VcfLicensingServer.getVcenterInfo] - Could not retrieve vcenter info data.java.lang.NullPointerException: costUnitId is marked non-null but is null    at com.vmware.statsplatform.persistence.impl.vcenter.license.allocation.usage.asset.record.VcfEntitlementAssetUsageRecord$VcfEntitlementAssetUsageRecordBuilder.costUnitId(VcfEntitlementAssetUsageRecord.java:19) 

• We can also see the events under /storage/log/vcops/log/vcf-licensing-plugin-<UUID>.log where the  NullPointerException is reported due to costUnitId property marked as null  
  

  YYYY-MM-DDtHH:MM:SS ERROR vcf-licensing-plugin 503165 [<USER_CONTEXT> threadId="584" threadName="ServerConnection on port 10000 Thread 5" operationId="<OPERATION_ID>"] [com.vmware.vrops.licensing.vcf.server.VcfLicensingServer.getVcenterInfo] - Could not retrieve vcenter info data.
  java.lang.NullPointerException: costUnitId is marked non-null but is null
      at com.vmware.statsplatform.persistence.impl.vcenter.license.allocation.usage.asset.record.VcfEntitlementAssetUsageRecord$VcfEntitlementAssetUsageRecordBuilder.costUnitId(VcfEntitlementAssetUsageRecord.java:19)
      at com.vmware.statsplatform.persistence.impl.vcenter.license.allocation.usage.asset.mapper.VcfEntitlementAssetLicensedOfDefaultCostUnitUsageRowMapper.mapRow(VcfEntitlementAssetLicensedOfDefaultCostUnitUsageRowMapper.java:40)
      at com.vmware.statsplatform.persistence.impl.vcenter.license.allocation.usage.asset.mapper.VcfEntitlementAssetLicensedOfDefaultCostUnitUsageRowMapper.mapRow(VcfEntitlementAssetLicensedOfDefaultCostUnitUsageRowMapper.java:18)
      at org.springframework.jdbc.core.RowMapperResultSetExtractor.extractData(RowMapperResultSetExtractor.java:94)
      at org.springframework.jdbc.core.RowMapperResultSetExtractor.extractData(RowMapperResultSetExtractor.java:61)
      at org.springframework.jdbc.core.JdbcTemplate$1.doInPreparedStatement(JdbcTemplate.java:733)
      at org.springframework.jdbc.core.JdbcTemplate.execute(JdbcTemplate.java:658)
      at org.springframework.jdbc.core.JdbcTemplate.query(JdbcTemplate.java:723)
      at org.springframework.jdbc.core.JdbcTemplate.query(JdbcTemplate.java:748)
      at org.springframework.jdbc.core.JdbcTemplate.query(JdbcTemplate.java:804)
      at org.springframework.jdbc.core.namedparam.NamedParameterJdbcTemplate.query(NamedParameterJdbcTemplate.java:218)
      at com.vmware.statsplatform.persistence.impl.vcenter.license.allocation.usage.VcfEntitlementAssetUsageService.getAssetsForVcIdsInLicensedOrDefaultCostUnit(VcfEntitlementAssetUsageService.java:882)
      at com.vmware.statsplatform.persistence.impl.vcenter.license.allocation.usage.VcfEntitlementVcenterUsageService.get(VcfEntitlementVcenterUsageService.java:633)
      at com.vmware.vrops.licensing.vcf.server.VcfLicensingServer.getVcenterInfo(VcfLicensingServer.java:1521)
      at jdk.internal.reflect.GeneratedMethodAccessor1336.invoke(Unknown Source)
      at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
      at java.base/java.lang.reflect.Method.invoke(Unknown Source)
      at com.integrien.alive.controller.Controller.lambda$createProxyHandler$2(Controller.java:1601)
      at jdk.proxy3/jdk.proxy3.$Proxy358.getVcenterInfo(Unknown Source)
      at jdk.internal.reflect.GeneratedMethodAccessor1336.invoke(Unknown Source)
      at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
      at java.base/java.lang.reflect.Method.invoke(Unknown Source)
      at com.vmware.vcops.platform.gemfire.GemfireFunction.invokeHandlerMethod(GemfireFunction.java:112)
      at com.vmware.vcops.platform.gemfire.GemfireFunction.execute(GemfireFunction.java:60)
      at com.vmware.vcops.platform.gemfire.GemfireFunctionHandler$FunctionHandler.execute(GemfireFunctionHandler.java:367)
      at com.vmware.vcops.platform.gemfire.GemfireFunctionHandler$TopGemfireFunction.execute(GemfireFunctionHandler.java:167)
      at org.apache.geode.internal.cache.tier.sockets.command.ExecuteFunction70.executeFunctionLocally(ExecuteFunction70.java:400)
      at org.apache.geode.internal.cache.tier.sockets.command.ExecuteFunction70.cmdExecute(ExecuteFunction70.java:261)
      at org.apache.geode.internal.cache.tier.sockets.BaseCommand.execute(BaseCommand.java:193)
      at org.apache.geode.internal.cache.tier.sockets.ServerConnection.doNormalMessage(ServerConnection.java:901)
      at org.apache.geode.internal.cache.tier.sockets.ServerConnection.doOneMessage(ServerConnection.java:1113)
      at org.apache.geode.internal.cache.tier.sockets.ServerConnection.run(ServerConnection.java:1394)
      at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
      at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
      at org.apache.geode.internal.cache.tier.sockets.AcceptorImpl.lambda$initializeServerConnectionThreadPool$3(AcceptorImpl.java:710)
      at org.apache.geode.logging.internal.executors.LoggingThreadFactory.lambda$newThread$0(LoggingThreadFactory.java:124)
      at java.base/java.lang.Thread.run(Unknown Source)

• A similar error may be observed in /var/log/vmware/cis-license/license.log when orphaned or obsolete license keys remain in vCenter
  

  {urn:vri:com.vmware.license.asset:domain-c<TBD>-<UUID><UUID><UUID><UUID><UUID>
   instanceId: domain-c<TBD>
   scopeId: <UUID><UUID><UUID><UUID><UUID>
   name: <TBD>
   version: null
   productId: '<reference to VMware vSAN Desktop License>'
   cost: [concurrentUser=0, cpuPackage=0, vm=0]
   license: {urn:vri:com.vmware.license.license:<UUID><UUID><UUID><UUID><UUID> name: VDI [<license start>-XXXXX-XXXXX-XXXXX-<license end>]}
  },

Environment

VMware Cloud Foundation Operations 9.1 [Upgraded environment]

Cause

• The VCF Operations licensing plugin fails to parse license data when an LDAP entry for a vSAN asset exists without a defined cost unit.
• If a legacy or unassigned vSAN license entry exists in the vCenter Server LDAP database without a defined cost unit, the licensing plugin encounters an exception when processing the asset records.
• During license reconciliation, the VCF Operations 9.1 licensing plugin requires every license asset entry to map to both a Product Family (e.g., ESX, vSAN) and a defined Cost Unit (e.g., cpuCore:16core for hosts or TiB for vSAN clusters).
• VCF Ops detects a license that should not be present and that cannot be associated with a Product family or a Cost Unit.

Note: VCF Operations processes all connected vCenter instances during reconciliation, a single stale entry on one vCenter Server will cause the list view to fail entirely, preventing all vCenter systems from appearing under: VCF Ops > Manage > License & Registration > vCenter Systems.

Resolution

Please contact Broadcom Technical Support to assist with steps to manually remove the stale license entry from the vCenter LDAP database.