Security Services Platform node get stuck with error "failed to reconcile VM: unable to get datastore"

Products

• VMware vDefend Firewall with Advanced Threat Prevention
• VMware vDefend Firewall

Issue/Introduction

When attempting to upgrade the Security Services Platform (SSP) or restart control plane, worker nodes, one or more nodes may become stuck in the Provisioning state. The following error message is observed in the vspherevm controller logs on the SSP Installer (SSPI):

1. Login to SSP Installer as sysadmin/root.

2. Check the machine status.

#kubectl get machines -A

3- Verify the logs of the machines stuck in the provisioning state and check for similar error messages.

# kubectl logs <machine name> -n <namespace>

 

"Reconciler error" err="failed to reconcile VM: unable to get datastore /<Datacenter>/datastore/<Datastore_Name>": datastore '/<Datacenter>/datastore/<Datastore_Name>' not found

Environment

SSP 5.0

Cause

In SSP version 5.0, the system tracks datastores using their absolute inventory path rather than a unique identifier (UUID).

If a datastore is moved to a different folder in the vCenter inventory (e.g., from a root datacenter folder into a sub-folder like /VSAN/), the Cluster API provider for vSphere (CAPV) is unable to locate the datastore at the previously stored path. This results in a reconciliation failure, preventing the provisioning or recovery of Control Plane VMs.

Resolution

To recover the environment and proceed with the upgrade, follow these steps:

1. Revert Datastore Location: Move the datastore back to its original inventory path as recorded in the error message.
   
   
2. Verify Connectivity: Once moved, the SSPI reconciler should automatically find the datastore and complete the node provisioning.
   
   
3. Upgrade SSP: After the environment returns to a Running state, upgrade to SSP 5.1 or later.
   
   
4. Post-Upgrade Move: Once on version 5.1+, the system will migrate to ID-based tracking. You may then move the datastore into any folder within the same datacenter without impacting connectivity.