/storage/log partition full due to Large Java dump files created under /storage/log/vmware/perfcharts in vCenter Server Appliance

Products

• VMware vCenter Server

Issue/Introduction

Symptoms:

• You see that the /storage/log directory is full and in critical state the appliance manager page in VCSA 6.5.
   
• You see that large Java dump files are created at /storage/log/vmware/perfcharts
   
• You see that the alerts are triggered in the vCenter Server VAMI page
   
• You notice that the /storage/log is almost full
   
  root@VCENTER [ ~ ]# df -h
  Filesystem                            Size  Used  Avail  Use%  Mounted on
  /dev/mapper/autodeploy_vg-autodeploy  9.8G   33M   9.2G    1%  /storage/autodeploy
  /dev/mapper/seat_vg-seat               25G  3.3G    20G   15%  /storage/seat
  /dev/mapper/db_vg-db                  9.8G  1.8G   7.5G   20%  /storage/db
  /dev/mapper/updatemgr_vg-updatemgr     99G  757M    93G    1%  /storage/updatemgr
  /dev/mapper/log_vg-log                 74G   67G   4.2G   95%  /storage/log
  /dev/mapper/core_vg-core               50G   52M    47G    1%  /storage/core
  /dev/sda1                             120M   28M    87M   24%  /boot
   
• When you browse to /storage/log/vmware/perfcharts directory, you see lots of Java dump (.hprof) files.
   

  root@VCENTER [ /storage/log/vmware/perfcharts ]# ls -l

  total 67433848

  -rw-r--r-- 1 perfcharts users 102100    May 19 19:03 java_error23079.log

  -rw-r--r-- 1 perfcharts users 106892    Mar 29 23:25 java_error45346.log

  -rw------- 1 perfcharts users 772752973 May 18 04:57 java_pid1290.hprof

  -rw------- 1 perfcharts users 775950352 Apr 23 22:59 java_pid13403.hprof

  -rw------- 1 perfcharts users 776215829 Apr 14 08:18 java_pid14007.hprof

  -rw------- 1 perfcharts users 776821858 Apr 10 16:00 java_pid14105.hprof

  -rw------- 1 perfcharts users 776148983 Apr 24 17:43 java_pid14437.hprof

  -rw------- 1 perfcharts users 775075757 Apr 30 03:48 java_pid14502.hprof

  -rw------- 1 perfcharts users 777414024 Mar  7 21:55 java_pid1463.hprof

  -rw------- 1 perfcharts users 777816472 Mar  3 14:35 java_pid15043.hprof

  -rw------- 1 perfcharts users 773803956 May 14 20:16 java_pid16276.hprof

  -rw------- 1 perfcharts users 776556635 Apr 19 04:38 java_pid17986.hprof

  -rw------- 1 perfcharts users 774637444 May  6 03:29 java_pid18867.hprof

  -rw------- 1 perfcharts users 776309973 Apr 17 14:34 java_pid19069.hprof
   
• Files are getting re-generated even after removing the java files by running the rm -rf /storage/log/vmware/perfcharts/java_pid*.hprof; command

Environment

VMware vCenter Server Appliance 6.5.x
VMware vCenter Server 6.5.x

Resolution

To resolve this issue upgrade to vCenter Server 6.5 U1d, available at VMware Download.

Note: Should the /storage/log partition be full but the symptoms in this KB don't match your issue, please see Troubleshooting disk space related issues: vCenter Server Appliance Maintenance Series (76563) for additional issues related to insufficient disk space. 

Workaround:
To workaround this issue:

1. Log in to vCenter Server appliance using SSH or console and enter the Bash shell
2. Take a backup of the /etc/vmware/service-layout.mfx file by running this command:

cp /etc/vmware/service-layout.mfx /etc/vmware/service-layout.mfx.bak;

3. Stop the vmware-perfcharts service by running this command:

service-control --stop vmware-perfcharts;

4. Open the service-layout.mfx file using a text editor by running this command:

vi service-layout.mfx;

5. Change CompressClassSize size from 94 to 64 for the row vmware-perfcharts
6. Save and close the service-layout.mfx file
7. Start the vmware-perfcharts service by running this command:

service-control --start vmware-perfcharts;

8. Now increase the allocated memory of the vmware-perfcharts service and restart it by running this command:

cloudvm-ram-size -C 1024 vmware-perfcharts;service-control --stop vmware-perfcharts;service-control --start vmware-perfcharts;

Additional Information

Troubleshooting vCenter Appliance /storage/log directory is 80% or more full