VCF Automation Upgrade Prechecks Fail with error [VCFMS-HEALTH-002] in VCF 9.1

Products

• VCF Operations/Automation (formerly VMware Aria Suite)

Issue/Introduction

• VCF Automation upgrade prechecks fail with a Platform Health Check Error [VCFMS-HEALTH-002] in VCF 9.1. 
  Platform Health Check Error [platform-statefulsets-core : 1 of 10 resources are not ok: logging-operator-fluentd: wrong resource state: InProgress - Ready: 0/1;] [VCFMS-HEALTH-002]
  
  
  
  
• The logging-operator-fluentd pod within the vmsp-platform of the VCF runtime services appears to be in a ready state. This can be verified by running the following command in the control plane node with vmware-system-user credentials and then elevating to root (sudo -i): k get pods -n vmsp-platform  | grep -i logging-operator-fluentd
  
  
• However, logging-operator-fluentd pod within the vmsp-platform of the VCF Automation appliance has only 1 of the 2 containers in running status. 
  root@<VCFAutomationApplaince> [ / ]# k get pods -n vmsp-platform  | grep -i logging-operator-fluentd
logging-operator-fluentd-0  1/2     Running 
logging-operator-fluentd-configcheck-18d5dd13  0/1     Completed  

Environment

• VMware Cloud Foundation (VCF) 9.1
• VCF Automation 9.1

Cause

When the log buffer files within the /buffers directory of the logging-operator-fluentd pod (located in the vmsp-platform namespace of the VCF Automation Appliance) exceed the default limit of 10,000 files, the readiness probe fails and triggers a health precheck failure. 

Resolution

• To resolve the logging-operator-fluentd pod health within the VCF Automation Appliance, follow the resolution steps of the KB: VCF Automation patching fails during precheck with error LCMVMSP10035
• Retry the upgrade once the pod is in a ready state.