VCF Cloud Proxy exhaustion or /storage/log (catalina.out) full

Products

• VCF Operations

Issue/Introduction

Symptoms:

• VCF Operations triggers an alert:
  • Disk space on cloud proxy is low
  • At 90% the space will trigger Immediate(Yellow) alarms
  • At 95% the space will trigger Critical(Red) critical alarms
    
    
• The file /storage/log/vcops/log/casa/catalina.out-YYYYMMDDHHMMSS grows to approximately 16 GB and becomes the largest file under the /storage/log directory.
  
  
• The catalina.out-YYYYMMDDHHMMSS file may contain a large number of entries similar to the following:
  
  ERROR Failed to read file: /storage/db/vmware-vrops-cprc/status/cprc.health.check.status
ERROR No content to map due to end-of-input at [Source: REDACTED (`StreamReadFeature.INCLUDE_SOURCE_IN_LOCATION` disabled); line: 1]
  
  
• As the log file continues to grow, disk space may become exhausted, which can cause services on the Cloud Proxy to stop functioning.

Environment

VCF Operations 9.x
VCF Cloud Proxy 9.x

 

Cause

This issue occurs if the cprc.health.check.status file is either corrupted or empty

root@proxy_node [ /storage/db/vmware-vrops-cprc/status ]# ls -ltrh
total XX K
-rw-rw-r-- 1 admin admin    0 MMM DD xx:xx cprc.health.check.status

Resolution

Note : Before proceeding, create a snapshot of the Aria Operations cluster by following the instructions in Snapshot Creation in VMware Aria Operations (342576)

• Check if the root partition has sufficient free space. If disk space is low, perform a cleanup by following the steps outlined in Troubleshooting Storage Issues in Aria Operations (318408)
  
  
• Take SSH session into the affected Could Proxy node and verify the file "cprc.health.check.status" by navigating to below directory
  
  root@proxy_node [ /storage/db/vmware-vrops-cprc/status ]# ls -ltrh
total XX K
-rw-rw-r-- 1 admin admin    0 MMM  DD xx:xx cprc.health.check.status
  
  
• If the cprc.health.check.status file is empty or missing contents, copy the file contents from a healthy Cloud Proxy node.
  
  
• Monitor the catalina.out log and verify that the error messages are no longer being generated.

Additional Information

Cloud Proxy fails to upgrade and the health status appears as "Outdated Life Status" in the Aria Operations admin UI (409296)

VCF Cloud Proxy におけるディスク容量不足、または /storage/log（catalina.out）の肥大化による領域逼迫 (443392)