License service on vCenter runs out of memory and dumps java_pidxxxx.hprof

Products

• VMware vCenter Server

Issue/Introduction

• License service hit out of memory many times
• Many heap under file java_pidxxxxxxx.hprof were created
  
  • sample log with /var/log/vmware/cis-license/license-runtime.log.stdout:

java.lang.OutOfMemoryError: GC overhead limit exceeded
Dumping heap to /var/log/vmware/cis-license/java_pidxxxxxxx.hprof ...
Heap dump file created [xxxxxxxx bytes in 1.272 secs]
ERROR StatusConsoleListener An exception occurred processing Appender RollingLogfile
 org.apache.logging.log4j.core.appender.AppenderLoggingException: java.lang.OutOfMemoryError: GC overhead limit exceeded
• vsan-health service shows long w lock in logs, the listed opID was impacted and other task may be delayed 
  
  • sample log with /var/log/vmware/vsan-health/vsan-lock-debug.log:

YYYY-MM-DDThh:mm:ss.xxxZ WARNING vsanlock[13743] [VsanMgmtAdapters::_ExitInt opID=vsan-PC-xxxxxxx] Long w lock on host-nnnnn: wait 0.0s, locked 1801.07s
    
    

Environment

vCenter 7.0.x
vCenter 8.0.x

Cause

Too many unused license on vCenter

Resolution

To calculate the number of licenses present in the vCenter, run the following command on the LDIF file:

grep  vCenter.ldif | sort | uniq -c | wc -l

Remove all license keys that are no longer required, such as expired license keys or keys that you do not intend to assign.

Workaround:
    Increase the amount of memory assigned to service vmware-cis-license, for example to 250MB and restart vmware-cis-license.
    Please refer Manually increasing the heap memory of components in vCenter Server 7.x and 8.x to check the detail steps.
     

Additional Information

Managing licenses on ESXi hosts using the vSphere Web Client

How Do You Remove a License in the vSphere Client