Usage Meter shows 100% CPU utilization with intermittent collection failures
search cancel

Usage Meter shows 100% CPU utilization with intermittent collection failures

book

Article ID: 379463

calendar_today

Updated On:

Products

VMware Usage Meter

Issue/Introduction

  • Usage Meter CPU utilization is 100%
  • On Notification, collection failure message displayed such as
    • Partial VC collection failure at stage: Inventory. Exception while reporting usage data ERR_UM_CLIENT_TIMEOUT Unable to send message after 40 attempts
    • Incomplete collection due to failure at stagge: Inventory

  • On collection log, "ERR_UM_CLIENT_TIMEOUT Unable to send message after 40 attempts" and "Server was overloaded" logged 


    2024-09-26 07:37:47.842 WARN --- [pool-1-thread-5] c.v.u.u.client.RequestManager : Server was overloaded, will retry later by timeout
    2024-09-26 07:37:50.842 INFO --- [RequestManager_vcenter_command_cli] c.v.u.u.c.RequestManager$RequestInfo : TRANSPORT - UM_CLIENT
    {"processing_time":120004,"request":"{action:str,authorization:str,command:str,data:{$schema:str,collectionId:num,itemCount:num,msg:str,status:str,time:num},priority:num,respond_id:str,timestamp:num,tracking_id:str}","create_time":1727336150837,"requests_num":40,"route_info":{"doRetry":true,"in":"agent.api.collection","transport":"NATS","timeout":3000},"responds":"[{data:null,errCode:str,errMsg:str}]","respond_time":[1727336270841]}
    2024-09-26 07:37:50.842 ERROR --- [pool-5-thread-1] c.vmware.um.collector.CollectionHelper : Exception while reporting usage data
    com.vmware.um.umconnection.err.UmApiException: ERR_UM_CLIENT_TIMEOUT Unable to send message after 40 attempts
            at com.vmware.um.umconnection.api.Collection.update(Collection.java:157)
            at com.vmware.um.collector.CollectionHelper.reportUsage(CollectionHelper.java:563)
            at com.vmware.um.vccollector.VCCollector.collectStages(VCCollector.java:286)
    :
            at java.base/java.lang.Thread.run(Unknown Source)
    2024-09-26 07:37:50.843 WARN --- [pool-5-thread-1] c.vmware.um.collector.CollectionHelper : Displaying warning notification: Incomplete collection due to failure at stage: Events.
    2024-09-26 07:37:50.843 ERROR --- [pool-5-thread-1] com.vmware.um.vccollector.VCCollector : Events stage raised exception com.vmware.um.collector.InternalError: Exception while reporting usage data
    com.vmware.um.collector.InternalError: Exception while reporting usage data
            at com.vmware.um.collector.CollectionHelper.reportUsage(CollectionHelper.java:572)
    :

    Caused by: com.vmware.um.umconnection.err.UmApiException: ERR_UM_CLIENT_TIMEOUT Unable to send message after 40 attempts
            at com.vmware.um.umconnection.api.Collection.update(Collection.java:157)
            at com.vmware.um.collector.CollectionHelper.reportUsage(CollectionHelper.java:563)
    :


Cause

This is known issue on Usage Meter 4.6 and later.

This issue  may happen when dss.hash file growned.

Resolution

There is no Resolution.

For workaround, follow steps below to recrate dss.hash and confirm that issue solved.

  1. Login in Usage Meter appliance as usagemeter user
  2. Navigate to /opt/vmware/cloudusagemetering/var/dss/hash

  3. Rename the file: dss.hash 
        mv dss.hash dss.hash.bak

  4. Restart the services:
    Navigate to: /opt/vmware/cloudusagemetering
    ./scripts/stop.sh All
    ./scripts/start.sh All

  5. Wait a couple of minutes and check the status
    ./scripts/status.sh
  6. Confirm that the file: dss.hash is recreated in: /opt/vmware/cloudusagemetering/var/dss/hash