VCF Automation patching fails during precheck with error LCMVMSP10035

Products

• VCF Automation

Issue/Introduction

• VCF Automation patching from version 9.0.1 to 9.0.2 fails during the validation phase of synthetic check with the following error code: LCMVMSP10035

• Error Code : 'LCMVMSP10035', Retry : 'true', Causing Properties : '{ CAUSE :: skipSyntheticCheck ===  }'
  com.vmware.vrealize.lcm.vmsp.common.exception.ValidateSyntheticCheckerException: Synthetic check failed. Please refer to Broadcom Knowledge Base Article https://knowledge.broadcom.com/external/article/389510 for remediation details.,"platform-statefulsets : 1 of 7 resources are not ok: logging-operator-fluentd: wrong resource state: InProgress - Ready: 0/6;" 

• The logging-operator-fluentd pods remain in a 1/2 Running state, failing the readiness probe running the following command:
  
  kubectl get pods -n kube-system | grep fluentd

vmsp-platform   logging-operator-fluentd-0                                                1/2     Running     0                33s
vmsp-platform   logging-operator-fluentd-1                                                1/2     Running     0                5m7s
vmsp-platform   logging-operator-fluentd-2                                                1/2     Running     0                3m22s
vmsp-platform   logging-operator-fluentd-3                                                1/2     Running     0                2m9s
vmsp-platform   logging-operator-fluentd-4                                                1/2     Running     0                83s
vmsp-platform   logging-operator-fluentd-5                                                1/2     Running     0                58s


 

• The VCF Automation 9.1 patch process gets stuck on "Stage VCF Automation precheck binaries". It eventually times out and fails with an error : Step stage_vsp_cluster_plugin failed. Reference code: 4B310AE8

Environment

VCF Automation 9.x

Cause

Fluentd pods fail readiness probes because log buffer accumulation exceeds defined thresholds. Slow log ingestion at the VMware Aria Operations for Logs destination prevents timely flushes, causing the logging-operator-fluentd statefulset to remain in an "InProgress" state during the upgrade's synthetic health check.

Resolution

Note: This KB bypasses the problem and does not fix it.  This issue may repeat due to the slow log ingestion to Aria Operations for logs; log forwarding issues after upgrade should be addressed.

To bypass this issue, following workaround could be used.

Workaround 1: Increase readiness probe of fluentd to allow the pods to transition to a ready state:

1. SSH to the VCF Automation node.
2. Check the accumulated buffers for each non-ready Fluentd pod by running the following commands:
   kubectl exec -it -n vmsp-platform logging-operator-fluentd-0 -- bash
ls -l buffers/ | wc -l
   Note: If the above command returns value above 20000, update the readiness probe to 50000.
3. Get name of the release template.
   kubectl get rt -n vmsp-platform | grep logging-operator
4. Edit release template. Replace X with version from last command.
   kubectl edit rt logging-operator-X -n vmsp-platform
5. Change the value of bufferFileNumberMax to 50000 from 10000, save and exit editor with wq!
6. Delete fluentd pods to reinitialize
   kubectl delete pod -n vmsp-platform logging-operator-fluentd-0
7. Re-initiate the 9.0.2 patch via VCF Operations Fleet Manager after fluentd pods are healthy, synthetic check will pass.

Workaround 2: Clear the accumulated buffers to allow the pods to transition to a ready state:
Note: Clearing accumulated buffer, old logs will be lost.  If acceptable, proceed with this workaround.

1. SSH to the VCF Automation node.
2. Running following commands will delete buffered(old) logs.
   kubectl exec -it -n vmsp-platform logging-operator-fluentd-0 -- bash -c "find /buffers -type f -delete"
kubectl delete pod -n vmsp-platform logging-operator-fluentd-0
3. Re-initiate the 9.0.2 patch via VCF Operations Fleet Manager after fluentd pods are healthy, synthetic check will pass.

Note: Repeat the above commands for all non-ready Fluentd pods.

For example, if logging-operator-fluentd-1 and logging-operator-fluentd-2 are also non-ready, run the checks on each of them.