Error: VCFMS-HEALTH-002 - Management components upgrade fails with an Error "logging-operator-fluentd is in a wrong resource state"

Products

• VCF Operations
• VMware SDDC Manager / VCF Installer

Issue/Introduction

• VCF Management component upgrades fail for multiple components, including SDDC lifecycle, Salt Master, and VCF services runtime.
  
  
• The upgrade precheck or update task reports a Platform Health Check Error with ID VCFMS-HEALTH-002.
   
• Logs indicate that logging-operator-fluentd is in a wrong resource state: InProgress - Ready: 0/1.
  
  SDDC Build Service pod logs shows the following error message:
  
  Command:   kubectl logs vcf-sddc-build-service-sddcbuild-########-#### -n vcf-sddc-lcm | less
Log location is SOS bundle: /var/log/pods/vcf-sddc-lcm_vcf-sddc-build-service-sddcbuild-########-####-####/sddcbuild/##.log

YY-MM-DDTHH:MM:SSZ","id":"com.vmware.vcfms.system.name.HealthCheckFailure","name":{"default":"Platform Health Check","id":"com.vmware.vcfms.system.name.HealthCheckFailure","localized":"Platform Health Check"},"prechecks":[{"id":"VCFMS-HEALTH-002","issue":{"id":"com.vmware.vcfms.system.name.HealthCheckFailure","message":{"args":["platform-statefulsets-core : 1 of 10 resources are not ok: logging-operator-fluentd: wrong resource state: InProgress - Ready: 0/1;"]
  
  
• The "logging-operator-fluentd" pods remain in a 1/2 Running state, failing the readiness probe 
  
  Command:    kubectl get pods -A | grep fluentd
                      vmsp-platform   logging-operator-fluentd-0                                                1/2     Running
  
  

Environment

• VCF 9.1

 

Cause

The logging-operator-fluentd pods fail their readiness probes because the number of accumulated log buffer files has exceeded the default threshold (typically 10,000). When this limit is reached, the pod is marked as "Not Ready," which triggers a health check failure and blocks the VCF management update.

Resolution

To resolve this issue, increase the maximum buffer file threshold in the release template and restart the affected pods.

1. SSH into the VCF Management cluster's control plane VM:
   
   1. Log in to the VCF Operations interface at https://<vcf_operations_fqdn> as a user assigned the Administrator role.
   2. Navigate to Build > Lifecycle > VCF Management.
   3. On the Components tab, select the VCF services runtime.
   4. Scroll to the Nodes section.
   5. Identify a control plane node and note the IP address
   6. Login using vmware-system-user
   7. Switch to root using below command,
      su - 
      
      
2. Check the buffers for Fluentd pod by running the following commands
   kubectl exec -it -n vmsp-platform logging-operator-fluentd-0 -- bash
ls -l buffers/ | wc -l

Note: If the above command returns value above 20000, update the readiness probe to 50000.
   
   
3. Identify the current logging-operator version:
   kubectl get rt -n vmsp-platform | grep logging-operator
    
4. Edit the release template:
   kubectl edit rt logging-operator-#### -n vmsp-platform 
   
   
5. Update the buffer threshold: 
   Locate the parameter bufferFileNumberMax and change the value from 10000 to 50000.
 
6. Save and exit the editor (:wq!).
   
   
7. Delete the Fluentd pod to force it to reinitialize with the new configuration: 
   kubectl delete pod -n vmsp-platform logging-operator-fluentd-0
   
   
8. Verify the pod health: 
   Command to check:  kubectl get pods -A | grep fluentd
   
   Wait for the pod to return to a 2/2 Running (Ready) state.
   
   
9. Clear the health check cache by restarting the SDDC Manager services:
    /opt/vmware/vcf/operationsmanager/scripts/cli/sddcmanager_restart_services.sh
   
   
10. Retry the upgrade from the VCF Operations console.

Additional Information

Reference KB: https://knowledge.broadcom.com/external/article/438093