Resolving /storage/log Disk Space Issues Caused by vmware-cis-license Java Heap Dump Files in vCenter Server

Products

• VMware vCenter Server

Issue/Introduction

vCenter Server may experience disk space issues in the /storage/log partition due to the accumulation of large cis-license Java heap dump files (java_pid*.hprof). This can lead to various operational problems, including failed operations and potential service disruptions.

Environment

- VMware vCenter Server 7.0 and newer

Cause

The cis-license service in vCenter Server may occasionally experience an Out of Memory (OOM) condition, causing it to generate large heap dump files in the /storage/log directory. These files can quickly consume available disk space if not managed properly.

Resolution

To clear the space and prevent future occurrences, follow these steps:

1. Check available disk space:
1. Log in to the vCenter Server Appliance Management Interface (VAMI).
2. Navigate to Monitor > Storage
3. Review the available space in the /storage/log partition.
2. Remove existing heap dump files:
1. Connect to the vCenter Server Appliance command line (usually through SSH)
2. Navigate to the /storage/log directory:
   cd /storage/log
3. List and remove any existing cis-license heap dump files:
   ls -lh cis-license/java_pid*.hprof
rm cis-license/java_pid*.hprof
3. Increase cis-license service memory allocation:
1. Check the current memory allocated to the cis-license service:
   cloudvm-ram-size -l | grep vmware-cis-license
2. Double the memory allocated to the cis-license service:
   cloudvm-ram-size -C <DoubleOutputFromStep1> vmware-cis-license
3. Restart the cis-license service:
   service-control --restart vmware-cis-license
4. Confirm that the memory allocated to the cis-license service changed:
   cloudvm-ram-size -l | grep vmware-cis-license
4. Monitor the situation:
1. Regularly check the /storage/log partition for new heap dump files and available disk space.

Note: Please take an offline snapshot or backup of VCenter (standalone or incase of linked mode, kindly take it for all VCenter in ELM).

Additional Information

• If the issue persists after increasing memory allocation, consider opening a support ticket with VMware for further assistance.
• Ensure that vCenter Server has adequate resources allocated, particularly if running in a resource-constrained environment.
• License service on vCenter runs out of memory and causes vsan health service long lock issue