CaaS node pool stuck in 'customizing' or 'provisioning' state due to caas-spoke Kafka consumer hang

Products

• VMware Telco Cloud Automation
• VMware Telco Cloud Platform

Issue/Introduction

• CaaS Node Pool is stuck in Provisioning or Customizing state indefinitely.
• Underlying VMs in vCenter appear healthy and are in a 'Normal' stage, but the TCA UI does not reflect this status.
• The TCA UI may report errors such as vmconfig CR ... does not exist or nodepolicy CR ... does not exist despite the nodes being functional.
• Patch Tool for TCA 3.2.0.1 is applied but the issue is still observed.
• caas-spoke logs show no activity for an extended period (e.g., 8+ hours) despite ongoing cluster operations.
• Node policy are stuck in provisioning or deleting state.

Environment

TCA: 3.2.0.1

TCP: 5.0

Cause

The caas-spoke Kafka consumer (located in image/caas-common/pubsub/kafka/consumer.go) can enter a silent hang state. This occurs after the consumer encounters a librdkafka ErrUnknownTopicOrPart error, typically following an edge-kafka broker outage or transient connectivity issue.

Instead of recovering or logging the failure, the consumer enters an infinite loop. Since caas-spoke is responsible for syncing NodePolicy Custom Resources (CRs) to the management cluster and reporting status back to the database/UI, the synchronization path is broken, leaving the node pool stuck in its current state.

Resolution

To Resolve this, restart caas-spoke Deployment

Restarting the deployment will force the Kafka consumer to re-initialize and clear the hang.

kubectl -n tca-cp-cn rollout restart deployment/caas-spoke

Note: Ensure you are in the correct context for the TCA-CP management cluster before running rollout commands.