Error: "logging-operator-fluentd: wrong resource state: InProgress - Ready: 0/8" during Fleet Lifecycle upgrade

Products

• VCF Operations

Issue/Introduction

Platform health check error VCFMS-HEALTH-002 occurs during the VMware Cloud Foundation (VCF) Fleet Lifecycle upgrade when the logging-operator-fluentd service fails to reach the Log Management endpoint due to DNS resolution failures. This issue persists even after attempting the resolution in KB 449078.

Environment

• VMware Cloud Foundation (VCF) 9.1.x
• Log Management

Cause

• Fluentd buffer files are backlogged due to a communication failure with the Log Management endpoint. The Log Management endpoint is unreachable from the Control Plane Virtual Machine (VM) because the Fully Qualified Domain Name (FQDN) resolves to an incorrect IP address.

Resolution

1. SSH into the VCF Management cluster's control plane VM:

   • Log in to the VCF Operations interface at the management console.
   • Navigate to Build > Lifecycle > VCF Management.
   • On the Components tab, select the VCF services runtime.
   • Identify a control plane node and note the IP address
   • Login using vmware-system-user.
   • Switch to root using below command: su -
2. Retrieve the Fluentd pod names: kubectl get pods -A | grep fluentd
3. Connect to the logging-operator-fluentd-0 pod: kubectl exec -it logging-operator-fluentd-0 -n vmsp-platform -- /bin/bash
4. Check the disk space utilization: df -h
   • 
5. Verify if the /buffers partition usage exceeds 70%. If usage is above 70%, Fluentd pods will not operate properly.
6. Update DNS records to ensure the Log Management FQDN resolves to the correct IP address.
7. Verify reachability to the Log Management FQDN from the Control Plane VM.
8. Allow time for the Fluentd buffer to clear automatically and for storage usage to return to standard levels once reachability is re-established.
9. Confirm the Fluentd pods return to a normal Ready state: kubectl get pods -A | grep fluentd
   • 
10. Confirm the Fleet Lifecycle upgrade completes successfully.

Additional Information

Error: VCFMS-HEALTH-002 - Management components upgrade fails with an Error "logging-operator-fluentd is in a wrong resource state"
(Refer to this article if the issue is not resolved by correcting DNS resolution).