Error: VCFMS-HEALTH-002 - logging-operator-fluentd Readiness Probe Failure in VCF 9.1

Products

• VCF Operations

Issue/Introduction

• Upgrade prechecks in VMware Cloud Foundation (VCF) 9.1 fail for SDDC Lifecycle, Migration service engine, and Salt Master components. This failure is triggered by a platform health check error (VCFMS-HEALTH-002) indicating that the logging-operator-fluentd pod is in an InProgress state (0/1 Ready).
  
  
• The upgrade precheck fails with the following error:
  Platform Health Check Error [platform-statefulsets-core : 1 of 10 resources are not ok: logging-operator-fluentd: wrong resource state: InProgress - Ready: 0/1;] [VCFMS-HEALTH-002]


• When checking the pod status, only 1 of the 2 containers is running:
  root@<REDACTED_HOSTNAME> [ ~ ]# kubectl get pods --namespace vmsp-platform -o wide | grep logging-operator-fluentd
logging-operator-fluentd-0 1/2 Running <REDACTED_IP> <REDACTED_HOSTNAME> <none> <none>
logging-operator-fluentd-configcheck-e70bad5b 0/1 Completed <REDACTED_IP> <REDACTED_HOSTNAME> <none> <none>


• Logs indicate buffer flush failures:
  error_class=Errno::ECONNRESET error="Connection reset by peer"

Environment

• VMware Cloud Foundation (VCF) 9.1

Cause

The logging-operator-fluentd pod readiness probe, by default, enforces a limit of 10,000 files in the /buffers directory. When the destination endpoint (VCF Log management) becomes unreachable or encounters connection resets, log buffer chunks accumulate. When the file count exceeds 10,000, the readiness probe fails, causing the pod to restart and preventing the upgrade precheck from completing.

Resolution

To resolve this issue, clear the accumulated buffer files and restart all non-ready Fluentd pods in the VMSP control plane VM.

Note: Clearing the accumulated buffer results in the loss of unsent logs.

This precheck is disabled in future releases of VMware Cloud Foundation. Subscribe to this knowledge article to get updates on this issue.

1. Establish an SSH connection to the VMSP Control plane node using the vmware-system-user credentials.
   
   
2. Identify any non-ready pods by running the following command to check the status of all Fluentd pods:
   kubectl get pods -n vmsp-platform | grep logging-operator-fluentd
   
   
3. For each Fluentd pod that is not in a 2/2 Running state, clear the accumulated buffer files:
   kubectl exec -it <pod-name> -n vmsp-platform -- bash -c "find /buffers -type f -delete"

   (Note: Replace <pod-name> with the actual name of the non-ready pod. For example, if logging-operator-fluentd-1 and logging-operator-fluentd-2 are non-ready, run this command for each of them.)
   
   

4. After purging the buffers, delete the affected pods to re-initialize the logging services:
   kubectl delete pod <pod-name> -n vmsp-platform
   
   
5. Monitor the pods until they reach a 2/2 Running state:
   kubectl get pods -n vmsp-platform | grep logging-operator-fluentd
   
   
6. Once all pods are healthy, re-initiate the upgrade precheck via the VCF Operations Fleet Manager.

Note: Once the precheck is successful and the upgrade fails with the same error message, repeat Steps 2 - 6 on the VCF Automation component.