Resolving /storage/log Disk Space Issues Caused by Infraprofile Java Heap Dump Files in vCenter Server

Products

• VMware vCenter Server

Issue/Introduction

vCenter Server may experience disk space issues in the /storage/log partition due to the accumulation of large infraprofile Java heap dump files (java_pid*.hprof). This can lead to various operational problems, including failed operations and potential service disruptions.

Environment

- VMware vCenter Server 7.0 and newer

Cause

The infraprofile service in vCenter Server may occasionally experience an Out of Memory (OOM) condition, causing it to generate large heap dump files in the /storage/log directory. These files can quickly consume available disk space if not managed properly.

Resolution

Resolution:

There is no resolution as of this writing. The development team is aware of the OOM issues and is working on long-term solutions, including improvements to session handling in the vAPI endpoint.

Work-around:

To clear the space and prevent future occurrences, follow these steps:

1. Check available disk space:
   1. Log in to the vCenter Server Appliance Management Interface (VAMI).
   2. Navigate to Monitor > Storage
   3. Review the available space in the /storage/log partition.
      
      
2. Remove existing heap dump files:
   1. Connect to the vCenter Server Appliance command line (usually through SSH)
   2. Navigate to the /storage/log directory:
      

      cd /storage/log

   3. List and remove any existing infraprofile heap dump files:

      ls -lh infraprofile/java_pid*.hprof
      rm infraprofile/java_pid*.hprof

3. Increase infraprofile service memory allocation:
   1. Check the current memory allocated to the infraprofile service:

      cloudvm-ram-size -l | grep vmware-infraprofile

   2. Check current memory allocated to the infraprofile service:

      cloudvm-ram-size -l | grep vmware-infraprofile

   3. Double the memory allocated to the infraprofile service:

      cloudvm-ram-size -C <DoubleOutputFromStep1> vmware-infraprofile

   4. Restart the infraprofile service:

      service-control --restart vmware-infraprofile

4. Monitor the situation:
   1. Regularly check the /storage/log partition for new heap dump files and available disk space.

Additional Information

• If the issue persists after increasing memory allocation, consider opening a support ticket with VMware for further assistance.
• Ensure that vCenter Server has adequate resources allocated, particularly if running in a resource-constrained environment.