Resolving /storage/log Disk Space Issues Caused by Infraprofile Java Heap Dump Files in vCenter Server

Products

• VMware vCenter Server

Issue/Introduction

vCenter server may experience disk space issues in the /storage/log partition due to the accumulation of large infraprofile Java heap dump files (java_pid*.hprof).

This can lead to various operational problems, including failed operations and potential service disruptions.

Environment

• VMware vCenter Server 7.0 and newer

Cause

The infraprofile service in vCenter Server may intermittently encounter an Out of Memory (OOM) condition.

When this occurs, the service generates large heap dump files in the /storage/log directory. If not proactively managed, these files can rapidly consume available disk space, potentially leading to broader system issues.

Resolution

Resolution:

• There is no resolution as of this writing.
• The development team is aware of the OOM issues and is working on long-term solutions, including improvements to session handling in the vAPI endpoint.

Work-around:

To clear the space and prevent future occurrences, follow these steps:

1. Check available disk space:
   1. Log in to the vCenter Server Appliance Management Interface (VAMI).
   2. Navigate to Monitor > Storage
   3. Review the available space in the /storage/log partition.
      
      
2. Remove existing heap dump files:
   1. Connect to the vCenter Server Appliance command line (usually through SSH)
   2. Navigate to the /storage/log directory:
      

      cd /storage/log/vmware

   3. List and remove any existing infraprofile heap dump files:

      ls -lh infraprofile/java_pid*.hprof && rm infraprofile/java_pid*.hprof

3. Increase infraprofile service memory allocation:
   1. Check the current memory allocated to the infraprofile service:

      cloudvm-ram-size -l | grep vmware-infraprofile

   2. Double the memory allocated to the infraprofile service:

      cloudvm-ram-size -C <DoubleOutputFromStep1> vmware-infraprofile

   3. Restart the infraprofile service:

      service-control --restart vmware-infraprofile

   4. Confirm that the memory allocated to the infraprofile service changed:

      cloudvm-ram-size -l | grep vmware-infraprofile

4. Monitor the situation:
   1. Regularly check the /storage/log partition for new heap dump files and available disk space.

Additional Information

If the issue persists after increasing memory allocation, consider opening a support ticket with VMware for further assistance.

Ensure that vCenter Server has adequate resources allocated, particularly if running in a resource-constrained environment.