Log Management deployment fails during worker node resizing due to storage latency in VCF Operations

Products

• VCF Operations

Issue/Introduction

When attempting to deploy Log Management in VCF Operations, the deployment task remains stuck "In Progress" for several hours (updating timestamps frequently) before eventually failing.

You will observe the following symptoms:

• The Log Management FQDN is unreachable.
• The deployment task fails with a timeout error (typically after ~6.5 hours) similar to the following:
  
  Error: exit status 143
Task timed out after 6h30m2s.
component-worker-node-resize: task errored: the server was unable to return a response in the time allotted (get configmaps component-resize-semaphore)
  
  
• Checking the Kubernetes cluster node status reveals multiple vcf-runtime nodes transitioning to a NodeNotReady state simultaneously:
  
  Node vcf-runtime-<uuid> status is now: NodeNotReady
  Node vcf-runtime-<uuid> status is now: NodeNotReady
  Node vcf-runtime-<uuid> status is now: NodeNotReady
  
  
  
• Etcd experienced severe latency when syncing data to disk.
  
  etcd-vcf-runtime-<identifier_string>/etcd/console-log-<Timestamp>.log
stderr F {"level":"warn","ts":"<Timestamp>","caller":"wal/wal.go:845","msg":"slow fdatasync","took":"1m44.012420228s","expected-duration":"1s"}

Environment

VCF Operations 9.1.0
 

Cause

This issue is caused by excessive storage latency in the underlying vSAN environment. High latency disrupts node heartbeats, causing multiple vcf-runtime nodes to transition into a NodeNotReady state. This instability disrupts communication within the Kubernetes control plane, preventing the VMSP platform from successfully scaling the worker nodes from large to xlarge and causing the deployment task to time out.

Resolution

To resolve this issue, the underlying environmental constraints must be corrected before resuming the deployment.

1. Resolve Storage Latency: Investigate the underlying vSAN infrastructure and resolve the performance issues. Ensure that storage latency is stabilized and returns to sub-millisecond levels.
2. Verify Node Health: Confirm that all VCF runtime nodes have successfully reconnected to the control plane and are showing a stable Ready status.
3. Resume Deployment: Once the infrastructure is stable and all nodes are Ready, log into the VCF Operations UI, locate the failed Log Management deployment task, and select the Retry option to resume the deployment.