reportFatalErorr call with errorCode 'ERR_THREAD_WATCHER' and error Message 'Issue detected for UploaderThreadsWatcher' in vCloud Usage Meter 4.7/4.8

Products

• VMware vCloud Suite

Issue/Introduction

1. Constant restarts due to issues in watcher threads.
2. Inability to check "Send Update to CPN" (test cloud connectivity)
3. Inability to view Notifications page, including, but not limited to slow UI page rendering and error banners
4. Large journal logs db file (/opt/vmware/cloudusagemetering/platform/data/journal-service.db)
5. OOM (out of memory - java.lang.OutOfMemoryError) exceptions in /opt/vmware/cloudusagemetering/platform/log/vmware-um-journal.log fileusage

Environment

vCloud Usage Meter 4.7.x and 4.8.x

Cause

Due to the current nature of the journal logs and Notifications UI page, in case there is a large number of journal log messages (more than 500k), spontaneous OOM exceptions can occur.

In this case consider decreasing the retention policy of the journal messages from 3 years (by default) to 6 months.

Resolution

1. Login in vCloud Usage Meter Appliance Console (via SSH or using remote console) as usagemeter

2. Backup /opt/vmware/cloudusagemetering/platform/bin/start-journal-service.sh file

3. Add

      export JAVA_OPTS="${JAVA_OPTS} -Djournal-service.cleanup.retention-period=P6M"

     line inside /opt/vmware/cloudusagemetering/platform/bin/start-journal-service.sh after the last export line

          export JAVA_OPTS="${JAVA_OPTS} -D..."

4. Reboot the appliance either using vCenter or using the command below

          sudo reboot