VCF Usage Meter stops uploading consumption data

Products

• VMware Usage Meter

Issue/Introduction

VCF Usage Meter does not upload usage data to cloud after several (hourly) collection cycles.

When checked for Usage Meter status on the  VCF Business Services Console, it is not healthy.

If data upload or the authentication with VMware Cloud Services Console is impacted for any collection interval, tenant administrator and users receive an error message.

 

Encounter OutOfMemoryError in nginx and vmware-um-journal logs: 

YYYY-MM-DDTHH:MM:SSS.284Z ERROR 1232 --- [VCF Usage Meter Journal Service] [https-jsse-nio-127.0.0.1-8051-exec-7] o.a.c.c.C.[.[.[/].[dispatcherServlet]    : Servlet.service() for servlet [dispatcherServlet] in context with path [] threw exception [Request processing failed: java.lang.RuntimeException: java.lang.OutOfMemoryError: Java heap space] with root cause

java.lang.OutOfMemoryError: Java heap space
        at java.base/java.util.ArrayList.grow(Unknown Source)
        at java.base/java.util.ArrayList.addAll(Unknown Source)
        at org.springframework.data.domain.Chunk.<init>(Chunk.java:53)
        at org.springframework.data.domain.PageImpl.<init>(PageImpl.java:46)
        at org.springframework.data.support.PageableExecutionUtils.getPage(PageableExecutionUtils.java:60)
        at org.springframework.data.jpa.repository.support.SimpleJpaRepository.readPage(SimpleJpaRepository.java:715)
        at org.springframework.data.jpa.repository.support.SimpleJpaRepository.findAll(SimpleJpaRepository.java:456)
        at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(Unknown Source)
        at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
        at java.base/java.lang.reflect.Method.invoke(Unknown Source)
        at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:359)
        at org.springframework.data.repository.core.support.RepositoryMethodInvoker$RepositoryFragmentMethodInvoker.lambda$new$0(RepositoryMethodInvoker.java:277)
        at org.springframework.data.repository.core.support.RepositoryMethodInvoker$RepositoryFragmentMethodInvoker$$Lambda$2022/0x00007f171fdb7d50.invoke(Unknown Source)
        at org.springframework.data.repository.core.support.RepositoryMethodInvoker.doInvoke(RepositoryMethodInvoker.java:170)
        at org.springframework.data.repository.core.support.RepositoryMethodInvoker.invoke(RepositoryMethodInvoker.java:158)
        at org.springframework.data.repository.core.support.RepositoryComposition$RepositoryFragments.invoke(RepositoryComposition.java:516)
        at org.springframework.data.repository.core.support.RepositoryComposition.invoke(RepositoryComposition.java:285)
        at org.springframework.data.repository.core.support.RepositoryFactorySupport$ImplementationMethodExecutionInterceptor.invoke(RepositoryFactorySupport.java:628)
        at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:184)
        at org.springframework.data.repository.core.support.QueryExecutorMethodInterceptor.doInvoke(QueryExecutorMethodInterceptor.java:173)
        at org.springframework.data.repository.core.support.QueryExecutorMethodInterceptor.invoke(QueryExecutorMethodInterceptor.java:148)
        at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:184)
        at org.springframework.data.projection.DefaultMethodInvokingMethodInterceptor.invoke(DefaultMethodInvokingMethodInterceptor.java:70)
        at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:184)
        at org.springframework.transaction.interceptor.TransactionInterceptor$$Lambda$1620/0x00007f171fbbef28.proceedWithInvocation(Unknown Source)
        at org.springframework.transaction.interceptor.TransactionAspectSupport.invokeWithinTransaction(TransactionAspectSupport.java:379)
        at org.springframework.transaction.interceptor.TransactionInterceptor.invoke(TransactionInterceptor.java:119)
        at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:184)
        at org.springframework.dao.support.PersistenceExceptionTranslationInterceptor.invoke(PersistenceExceptionTranslationInterceptor.java:138)
        at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:184)
        at org.springframework.data.jpa.repository.support.CrudMethodMetadataPostProcessor$CrudMethodMetadataPopulatingMethodInterceptor.invoke(CrudMethodMetadataPostProcessor.java:165)

Environment

VCF Usage Meter 9.0.x

Cause

Due to 'Out of Memory' issue occurring during periodic endpoint metering.

Resolution

To resolve this issue, required to split the metering load to multiple usage meter instances.

However, if deploying new usage meter instance(s) are not readily possible, we can follow the workaround to increase java heap space within the same UM instance:

• Increase the overall UM appliance VM's memory with an additional 2GB.
• SSH to Usage Meter using usagemeter account.
• Edit below file using vi editor:
  "/opt/vmware/cloudusagemetering/platform/conf/schedule-executions/command.yml"
• Find property "extraJavaOpts" for vCenter collector and change the value from -Xmx1024M to -Xmx3072M.

Additional Information

Refer:

VMware Cloud Foundation Usage Meter 9.0 Configuration Limits

Cores were missing in the report due to the vCenter collection issue

vCenter Server status shows as "Credentials Verified" in Usage Meter