reportFatalErorr call with errorCode 'ERR_THREAD_WATCHER' and error Message 'Issue detected for UploaderThreadsWatcher' in Usage Meter

Products

• VMware Usage Meter

Issue/Introduction

Usage Meter has one or more issues as described below: 

• Constant restarts due to issues in watcher threads.
• Inability to check "Send Update to CPN" (test cloud connectivity)
• Inability to view Notifications page, including, but not limited to slow UI page rendering and error banners
• Large journal logs db file (/opt/vmware/cloudusagemetering/platform/data/journal-service.db)
• OOM (out of memory - java.lang.OutOfMemoryError) exceptions in /opt/vmware/cloudusagemetering/platform/log/vmware-um-journal.log fileusage
• You also see the below error in the vmware-um-journal.log 

ERROR --- [nginx-clojure-worker-1] com.vmware.um.umconnection.api.Journal : Unable to retrieve the Journal logs.
com.vmware.um.common.err.UmException: Unable to call service at https://localhost:8051/api/v2/journal/search-results: 500 - {"timestamp":"2024-03-12T09:47:47.256+00:00","status":500,"error":"Internal Server Error","path":"/api/v2/journal/search-results"}

 

Environment

vCloud Usage Meter 4.8 

VMware Usage Meter 9.0.1

Cause

Due to the current nature of the journal logs and Notifications UI page, in case there is a large number of journal log messages (more than 500k), spontaneous OOM exceptions can occur. In this case consider decreasing the retention policy of the journal messages from 3 years (by default) to 6 months.

Resolution

To resolve the issue, please follow the below:

• Login in vCloud Usage Meter Appliance Console (via SSH or using remote console) as usagemeter
• Backup /opt/vmware/cloudusagemetering/platform/bin/start-journal-service.sh file

cp /opt/vmware/cloudusagemetering/platform/bin/start-journal-service.sh /opt/vmware/cloudusagemetering/platform/bin/start-journal-service.sh.bk

• Add the below line in the start-journal-service.sh after the last export line

export JAVA_OPTS="${JAVA_OPTS} -Djournal-service.cleanup.retention-period=P6M"

• Reboot the appliance either through the vCenter or using the command below

sudo reboot

 

Additional Information

Also, you can increase the memory from 512M to 1536M in the same file /opt/vmware/cloudusagemetering/platform/bin/start-journal-service.sh