All pods are in Pending State and unable to locate the workload/VKC cluster worker nodes.
search cancel

All pods are in Pending State and unable to locate the workload/VKC cluster worker nodes.

book

Article ID: 389359

calendar_today

Updated On:

Products

VMware vSphere Kubernetes Service

Issue/Introduction

  • All pods are in a pending state or unable to view pods status when running command: kubectl get pods -n namespace , showing a production down situation.
  • All VKC worker nodes have disappeared from the VCenter UI and are unable to view the worker node status from SSH of the Supervisor and VKC. 
  • vCenter's /var/log/vmware/vmon/vmon.log shows DRS related error messages as below. The key error message here is "Target datastore must be specified in order to deploy the OVF template to the vSphere DRS disabled cluster General-Cluster"
2025-02-01T02:51:44.366894597Z stderr F E0201 02:51:44.366840       1 controller.go:317] controller/virtualmachine "msg"="Reconciler error" "error"="deploy from content library failed for image \"ob-22748270-tkgs-ova-photon-3-v1.26.10---vmware.1-fips.1-tkg.1\": deploy error: Target datastore must be specified in order to deploy the OVF template to the vSphere DRS disabled cluster General-Cluster." "name"="worker-nodepool-pnj8j-6749c78bffx6l9hl-vxxxxxx" "namespace"="general" "reconciler group"="vmoperator.vmware.com" "reconciler kind"="VirtualMachine"

2025-02-01T02:51:55.081104912Z stderr F E0201 02:51:55.080978       1 virtualmachine_controller.go:748] VirtualMachine "msg"="Provider failed to create VirtualMachine" "error"="deploy from content library failed for image \"ob-22748270-tkgs-ova-photon-3-v1.26.10---vmware.1-fips.1-tkg.1\": deploy error: Target datastore must be specified in order to deploy the OVF template to the vSphere DRS disabled cluster General-Cluster." "name"="general/worker-nodepool-pnj8j-6749c78bffx6l9hl-cxxxxxx"

2025-02-01T02:51:55.081146104Z stderr F E0201 02:51:55.081035       1 virtualmachine_controller.go:263] VirtualMachine "msg"="Failed to reconcile VirtualMachine" "error"="deploy from content library failed for image \"ob-22748270-tkgs-ova-photon-3-v1.26.10---vmware.1-fips.1-tkg.1\": deploy error: Target datastore must be specified in order to deploy the OVF template to the vSphere DRS disabled cluster General-Cluster." "name"="general/worker-nodepool-pnj8j-6749c78bffx6l9hl-cxxxxxx"

Environment

vSphere with Tanzu

 

Cause

Fully automated DRS is a pre-requisite for vSphere with Tanzu environments.

In this case, however, the DRS was set to manual mode. The DRS mode set to manual, partially automated, or disabled will cause this error situation.

Resolution

  • Enabling DRS and setting it to fully automatic mode will resolve the issue.
  • Following this change, all worker nodes and pods will gradually transition to a ready state.
  • This adjustment will restore the VKC to a healthy and fully operational condition."

Additional Information

Powered by Wolken Software