Usage Meter shows 100% CPU utilization with intermittent collection failures
book
Article ID: 379463
calendar_today
Updated On:
Products
VMware Usage Meter
Issue/Introduction
Usage Meter CPU utilization is 100%
On Notification, collection failure message displayed such as
Partial VC collection failure at stage: Inventory. Exception while reporting usage data ERR_UM_CLIENT_TIMEOUT Unable to send message after 40 attempts
Incomplete collection due to failure at stagge: Inventory
On collection log, "ERR_UM_CLIENT_TIMEOUT Unable to send message after 40 attempts" and "Server was overloaded" logged
2024-09-26 07:37:47.842 WARN --- [pool-1-thread-5] c.v.u.u.client.RequestManager : Server was overloaded, will retry later by timeout 2024-09-26 07:37:50.842 INFO --- [RequestManager_vcenter_command_cli] c.v.u.u.c.RequestManager$RequestInfo : TRANSPORT - UM_CLIENT {"processing_time":120004,"request":"{action:str,authorization:str,command:str,data:{$schema:str,collectionId:num,itemCount:num,msg:str,status:str,time:num},priority:num,respond_id:str,timestamp:num,tracking_id:str}","create_time":1727336150837,"requests_num":40,"route_info":{"doRetry":true,"in":"agent.api.collection","transport":"NATS","timeout":3000},"responds":"[{data:null,errCode:str,errMsg:str}]","respond_time":[1727336270841]} 2024-09-26 07:37:50.842 ERROR --- [pool-5-thread-1] c.vmware.um.collector.CollectionHelper : Exception while reporting usage data com.vmware.um.umconnection.err.UmApiException: ERR_UM_CLIENT_TIMEOUT Unable to send message after 40 attempts at com.vmware.um.umconnection.api.Collection.update(Collection.java:157) at com.vmware.um.collector.CollectionHelper.reportUsage(CollectionHelper.java:563) at com.vmware.um.vccollector.VCCollector.collectStages(VCCollector.java:286) : at java.base/java.lang.Thread.run(Unknown Source) 2024-09-26 07:37:50.843 WARN --- [pool-5-thread-1] c.vmware.um.collector.CollectionHelper : Displaying warning notification: Incomplete collection due to failure at stage: Events. 2024-09-26 07:37:50.843 ERROR --- [pool-5-thread-1] com.vmware.um.vccollector.VCCollector : Events stage raised exception com.vmware.um.collector.InternalError: Exception while reporting usage data com.vmware.um.collector.InternalError: Exception while reporting usage data at com.vmware.um.collector.CollectionHelper.reportUsage(CollectionHelper.java:572) :
Caused by: com.vmware.um.umconnection.err.UmApiException: ERR_UM_CLIENT_TIMEOUT Unable to send message after 40 attempts at com.vmware.um.umconnection.api.Collection.update(Collection.java:157) at com.vmware.um.collector.CollectionHelper.reportUsage(CollectionHelper.java:563) :
Cause
This is known issue on Usage Meter 4.6 and later.
This issue may happen when dss.hash file growned.
Resolution
There is no Resolution.
For workaround, follow steps below to recrate dss.hash and confirm that issue solved.
Login in Usage Meter appliance as usagemeter user
Navigate to /opt/vmware/cloudusagemetering/var/dss/hash
Rename the file: dss.hash mv dss.hash dss.hash.bak
Restart the services: Navigate to: /opt/vmware/cloudusagemetering ./scripts/stop.sh All ./scripts/start.sh All
Wait a couple of minutes and check the status ./scripts/status.sh
Confirm that the file: dss.hash is recreated in: /opt/vmware/cloudusagemetering/var/dss/hash