Usage Meter shows 100% CPU utilization with intermittent collection failures

Products

• VMware Usage Meter

Issue/Introduction

• Usage Meter CPU utilization is 100%
• On Notification, collection failure message displayed such as
  • Partial VC collection failure at stage: Inventory. Exception while reporting usage data ERR_UM_CLIENT_TIMEOUT Unable to send message after 40 attempts
  • Incomplete collection due to failure at stagge: Inventory
    
    
• On collection log, "ERR_UM_CLIENT_TIMEOUT Unable to send message after 40 attempts" and "Server was overloaded" logged 
  
  
  2024-09-26 07:37:47.842 WARN --- [pool-1-thread-5] c.v.u.u.client.RequestManager : Server was overloaded, will retry later by timeout
  2024-09-26 07:37:50.842 INFO --- [RequestManager_vcenter_command_cli] c.v.u.u.c.RequestManager$RequestInfo : TRANSPORT - UM_CLIENT
  {"processing_time":120004,"request":"{action:str,authorization:str,command:str,data:{$schema:str,collectionId:num,itemCount:num,msg:str,status:str,time:num},priority:num,respond_id:str,timestamp:num,tracking_id:str}","create_time":1727336150837,"requests_num":40,"route_info":{"doRetry":true,"in":"agent.api.collection","transport":"NATS","timeout":3000},"responds":"[{data:null,errCode:str,errMsg:str}]","respond_time":[1727336270841]}
  2024-09-26 07:37:50.842 ERROR --- [pool-5-thread-1] c.vmware.um.collector.CollectionHelper : Exception while reporting usage data
  com.vmware.um.umconnection.err.UmApiException: ERR_UM_CLIENT_TIMEOUT Unable to send message after 40 attempts
          at com.vmware.um.umconnection.api.Collection.update(Collection.java:157)
          at com.vmware.um.collector.CollectionHelper.reportUsage(CollectionHelper.java:563)
          at com.vmware.um.vccollector.VCCollector.collectStages(VCCollector.java:286)
  :
          at java.base/java.lang.Thread.run(Unknown Source)
  2024-09-26 07:37:50.843 WARN --- [pool-5-thread-1] c.vmware.um.collector.CollectionHelper : Displaying warning notification: Incomplete collection due to failure at stage: Events.
  2024-09-26 07:37:50.843 ERROR --- [pool-5-thread-1] com.vmware.um.vccollector.VCCollector : Events stage raised exception com.vmware.um.collector.InternalError: Exception while reporting usage data
  com.vmware.um.collector.InternalError: Exception while reporting usage data
          at com.vmware.um.collector.CollectionHelper.reportUsage(CollectionHelper.java:572)
  :
  
  Caused by: com.vmware.um.umconnection.err.UmApiException: ERR_UM_CLIENT_TIMEOUT Unable to send message after 40 attempts
          at com.vmware.um.umconnection.api.Collection.update(Collection.java:157)
          at com.vmware.um.collector.CollectionHelper.reportUsage(CollectionHelper.java:563)
  :
  
  
  

Cause

This is known issue on Usage Meter 4.6 and later.

This issue  may happen when dss.hash file growned.

Resolution

There is no Resolution.

For workaround, follow steps below to recrate dss.hash and confirm that issue solved.

1. Login in Usage Meter appliance as usagemeter user
2. Navigate to /opt/vmware/cloudusagemetering/var/dss/hash
   
   
3. Rename the file: dss.hash 
       mv dss.hash dss.hash.bak
   
   
4. Restart the services:
   Navigate to: /opt/vmware/cloudusagemetering
   ./scripts/stop.sh All
   ./scripts/start.sh All
   
   
5. Wait a couple of minutes and check the status
   ./scripts/status.sh
6. Confirm that the file: dss.hash is recreated in: /opt/vmware/cloudusagemetering/var/dss/hash