Usage Meter 9.X reports "Incomplete collection" for the vCenter server with error 'API Exception occurred when querying SPBM profiles'

Products

• VMware vCenter Server

Issue/Introduction

• The Status for vCenter server  collection is reported as 'Incomplete collection' in Usage Meter UI
  
  
  
  
• Checking the notifications tab, the collection fails with : API Exception occurred when querying SPBM profiles
  
  
• Logs on Usage meter: /var/log/vmware-um-platform/collection-vcenter-main.log :
  
  <timestamp> WARN --- [collector-main-thread] c.v.u.v.spbm.SpbmCollectionStage         : Exception querying SPBM profiles for vCenter 20. com.vmware.um.collector.CollectionError: Spbm compliant VMs could not be fetched for profileId c24b38d1-xxxx-417a-94a2-xxxxxxxxxxxx for vCenter server: 20 
        at com.vmware.um.vccollector.spbm.SpbmServiceSession.getProfileAssociatedCompliantVMs(SpbmServiceSession.java:192) 
        at com.vmware.um.vccollector.spbm.SpbmCollectionStage.getVMsWithErasureCoding(SpbmCollectionStage.java:150) 
        at com.vmware.um.vccollector.spbm.SpbmCollectionStage.collectUsage(SpbmCollectionStage.java:91) 
        at com.vmware.um.vccollector.vsan.VsanCollectionStage.collectUsage(VsanCollectionStage.java:190) 
        at com.vmware.um.vccollector.VCCollector.collectStages(VCCollector.java:270) 
        at com.vmware.um.vccollector.VCCollector.collect(VCCollector.java:212) 
        at com.vmware.um.vccollector.VCCollector.collect(VCCollector.java:39) 
        at com.vmware.um.collector.CollectionHelper.collectFromServer(CollectionHelper.java:642) 
        at com.vmware.um.collector.CollectionHelper.collectFromServersWithReporting(CollectionHelper.java:808) 
        at com.vmware.um.collector.CollectionHelper.collectWithReporting(CollectionHelper.java:617) 
        at com.vmware.um.collector.SynchronousCollectionExecutor.triggerCollection(SynchronousCollectionExecutor.java:42) 
        at com.vmware.um.collector.AbstractCollectorComponent.handleCollectCommand(AbstractCollectorComponent.java:134) 
        at com.vmware.um.collector.AbstractCollectorComponent.handleCommand(AbstractCollectorComponent.java:55) 
        at com.vmware.um.collector.CollectorComponent.handleCommand(CollectorComponent.java:25) 
        at com.vmware.um.collector.CollectionHelper.processCommand(CollectionHelper.java:894) 
        at com.vmware.um.collector.AbstractCollector.processCommand(AbstractCollector.java:263) 
        at com.vmware.um.umcomponent.CommandRunner$CommandExecutionWrapper.execute(CommandRunner.java:397) 
        at com.vmware.um.umcomponent.CommandRunner.invokeCollector(CommandRunner.java:248) 
        at com.vmware.um.umcomponent.CommandRunner.lambda$executeCommand$1(CommandRunner.java:204) 
        at java.base/java.util.concurrent.CompletableFuture$AsyncRun.run(Unknown Source) 
        at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) 
        at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) 
        at java.base/java.lang.Thread.run(Unknown Source) 
Caused by: com.vmware.pbm.RuntimeFaultFaultMsg: 
        at java.base/jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) 
        at java.base/jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance(Unknown Source) 
        at java.base/jdk.internal.reflect.DelegatingConstructorAccessorImpl.newInstance(Unknown Source) 
        at java.base/java.lang.reflect.Constructor.newInstanceWithCaller(Unknown Source) 
        at java.base/java.lang.reflect.Constructor.newInstance(Unknown Source) 
        at com.sun.xml.ws.fault.SOAPFaultBuilder.createException(SOAPFaultBuilder.java:122) 
        at com.sun.xml.ws.client.sei.StubHandler.readResponse(StubHandler.java:223) 
        at com.sun.xml.ws.db.DatabindingImpl.deserializeResponse(DatabindingImpl.java:176) 
        at com.sun.xml.ws.db.DatabindingImpl.deserializeResponse(DatabindingImpl.java:263) 
        at com.sun.xml.ws.client.sei.SyncMethodHandler.invoke(SyncMethodHandler.java:88) 
        at com.sun.xml.ws.client.sei.SyncMethodHandler.invoke(SyncMethodHandler.java:61) 
        at com.sun.xml.ws.client.sei.SEIStub.invoke(SEIStub.java:131) 
        at jdk.proxy2/jdk.proxy2.$Proxy42.pbmCheckCompliance(Unknown Source) 
        at com.vmware.um.vccollector.spbm.SpbmServiceSession.getProfileAssociatedCompliantVMs(SpbmServiceSession.java:181) 
        ... 22 common frames omitted 
<timestamp>  WARN --- [collector-main-thread] c.vmware.um.collector.CollectionHelper   : Displaying warning notification: API Exception occurred when querying SPBM profiles. 
  
  
• Logs on the vcenter server : /var/log/vmware/vmware-sps/sps.log:

<timestamp> [pool-4-thread-1041] ERROR opId= com.vmware.pbm.compliance.impl.ComplianceManagerImpl - Exception occured. Entities: [(pbm.ServerObjectRef) {
dynamicType = null,
dynamicProperty = null,
...
}]
java.lang.NullPointerException: null

Environment

vCenter server 8.0

VCF Usage Meter 9.0

Cause

The issue occurs due to missing a profileId for a virtual machine in the spbmComplianceResultProvider for kv_value table.

Resolution

To resolve this issue we need to identify the virtual machine that has a null profile ID value.

Please raise a support request with Broadcom support by uploading the vCenter server and the Usage meter log bundle along with the SPS KV store dump.

• Take SSH to the vCenter server
• Execute the command to export the KV store output:

/opt/vmware/vpostgres/current/bin/pg_dump -U postgres -d VCDB -t cis_kv_keyvalue > /tmp/VCDBBackup

Upload these files to the Broadcom FTP for the support case.