Node pool Deployment stuck in processing state with Reason K8S121001

Products

• VMware Telco Cloud Automation

Issue/Introduction

When adding a new node pool using Telco Cloud Automation, we see that the task never finishes and gets stuck in processing state. In the Node Pools section of the CaaS Infrastructure we see this error and the reason for the Nodepolicy Ready is showing K8S121001:

[VcSessionEstablishmentFailed] vmconfig status is Failed, nodeconfig status is Normal.vmconfig failed: plugin placementPlugin check failed: VSphereVM.infrastructure.cluster.x-k8s.io "Node-Pool-Name-XXXX" not found
. Node-Policy stage failed. Error: node Node-Pool-Name-XXXX err: vmconfig status is Failed, nodeconfig status is Normal.vmconfig failed: plugin placementPlugin check failed: VSphereVM.infrastructure.cluster.x-k8s.io "Node-Pool-Name-XXXX" not found
.. Reason: Normal/Configuring/Failed/Total: 0/0/1/1.nodes [Node-Pool-Name-XXXX] in Failed state.NodePolicyMachineStatus - VSphere-Machine-Name-XXXX CR stage: Failed. Error: vmconfig status is Failed, nodeconfig status is Normal.vmconfig failed: plugin placementPlugin check failed: VSphereVM.infrastructure.cluster.x-k8s.io "VSphere-Machine-Name-XXXX" not found

 

In both the capv and vmconfig pod logs we can see messages similar to 'VSphereVM.infrastructure.cluster.x-k8s.io "VSphere-Machine-Name-XXXX" not found'.

Environment

2.3

Resolution

Restart the vmconfig opearator pod on the TKG Mgmt Cluster

kubectl rollout restart deploy/vmconfig-operator -n tca-system