Resolving /storage/log Disk Space Issues caused by Infraprofile Java Heap Dump Files in vCenter Server

Products

• VMware vCenter Server

Issue/Introduction

• vCenter server may experience disk space issues in the /storage/log partition due to the accumulation of large infraprofile Java heap dump files (java_pid*.hprof).
• This can lead to various operational problems, including failed operations and potential service disruptions.
• You might also receive alarm similar to "[Warning] Alarm Log Disk Exhaustion on vCenter on Folder Datacenters because vCenter Log File System Resource status changed from Green to Yellow on vCenter for continuous Disk utilization"
• Identify storage consumption or list files consuming storage space at /storage/log partition by following below mentioned steps;

VAMI UI

1. Log in to the vCenter Server Appliance Management Interface (VAMI).
2. Navigate to Monitor > Storage
3. Review the available space in the /storage/log partition.

 

vCenter CLI

• Take SSH session to vCenter server
• Login as root
• Navigate to /storage/log location: cd /storage/log/
• List storage space by directory: df -h

Environment

• VMware vCenter Server 7.x
• VMware vCenter Server 8.x

Cause

The infraprofile service in vCenter Server may intermittently encounter an Out of Memory (OOM) condition.

When this occurs, the service generates large heap dump files in the /storage/log directory. If not proactively managed, these files can rapidly consume available disk space, potentially leading to broader system issues.

Resolution

There is no resolution as of this writing. The development team is aware of the OOM issues and is working on long-term solutions, including improvements to session handling in the vAPI endpoint.

Work-around:

To clear the space and prevent future occurrences, follow these steps:

1. List the files consuming storage space
   
   
   1. # cd to /storage/log directory
   2. Type find . -type f -print0 | xargs -0 du -h | sort -rh | head -n 10
      
      

      root@<vcentername> [ /storage/log ]# find . -type f -print0 | xargs -0 du -h | sort -rh | head -n 10

      2.4G    ./vmware/vsphere-ui/java_pid####.hprof

      2.2G    ./vmware/vsphere-ui/java_pid####.hprof

      2.0G    ./vmware/vpostgres/postgresql.log

      1.8G    ./vmware/vsphere-ui/java_pid####.hprof

      1.7G    ./vmware/vsphere-ui/java_pid####.hprof

      287M    ./vmware/infraprofile/java_pid####.hprof

      175M    ./vmware/rbd/preUpgradeLogs.zip

      147M    ./vmware/vmware-sps/sps-access-log.YYYY-MM-DD.log

      122M    ./vmware/vc-ws1a-broker/nginx-error.log

      101M    ./vmware/vsphere-ui/logs/threadmonitor3.log

      Note: Isolate the files based on high consumption of storage space. In the above example, it shall be *.hprof files. Notice the location of these files differ, i.e. /vmware/vsphere-ui and /vmware/infraprofile/

2. Navigate to each of the above location and remove the files using 'rm' command.
   
   
3. Increase infraprofile service memory allocation:
   
   
   1. Check the current memory allocated to the infraprofile service:
      cloudvm-ram-size -l | grep vmware-infraprofile
      
      
   2. Double the memory allocated to the infraprofile service:
      cloudvm-ram-size -C <DoubleOutputFromStep1> vmware-infraprofile
      
      
   3. Restart the infraprofile service:
      service-control --restart vmware-infraprofile
      
      
   4. Confirm that the memory allocated to the infraprofile service changed:
      cloudvm-ram-size -l | grep vmware-infraprofile