Storage configuration mismatch between VCF management components and vCenter causes VCF to fail upgrade or patching

Products

VCF Automation VMware Cloud Foundation

Issue/Introduction

Storage migration (SvMotion) of management domain VMs from new datastore and removed old datastore from vCenter.

Performed as regular maintenance to move from legacy storage to newer storage.
When patching VCF 9.1 management components to later build receiving below error(s).

Message: “Platform Health Check Error [platform-vmsp-platform-validate-resources-vsphere-resources : unable to validate vSphere objects: unable to resolve datastore identifier Datastore:datastore-15: ServerFaultCode: The object ‘vim.Datastore:datastore-15’ has already been deleted or has not been completely created] [VCFMS-HEALTH-002]”

Message: “Platform Health Check Error [platform-package-core : 1 of 1 resources are not ok: vmsp-platform wrong resource state: InProgress - package deployment failed.] [VCFMS-HEALTH-002]”

Error: "Source and target systems use different datastores" - Upgrade of VCF Components from 9.0 to 9.1 fails during Import Validation Precheck

The VIDB or VCF-A upgrade from 9.0 to 9.1 fails at the Component Import Validation Precheck.
The precheck error message reports that the source (9.0) and target (9.1) clusters are using different datastores.
The error message includes the expected target datastore URL.
Retrying the upgrade without intervention continues to fail at the same precheck.
The upgrade prechecks shows the following error message for both of these components:

Source and target systems use different datastores: Source=ds:///vmfs/volumes/###############/, Target=ds:///vmfs/volumes/###############/. Please configure and retry the operation or contact support if the issue persists. [VCFMS-PRECHECK-IMPORT-###]

For this upgrade to proceed, the source and target datastores must be the same. Make necessary adjustments and retry. If the issue persists, please contact support.
Import Configuration Validation Precheck Failed
Import precheck completed with errors. [VCFMS-PRECHECK-IMPORT-###]

Environment

VCF Automation 9.0.x
VCF Automation 9.1
VCF Identity Broker 9.0.x
VCF Identity Broker 9.1
VCF 9.1.x

Cause

For storage migration task affecting VSP (vSphere Stack Platform):

A discrepancy exists between the physical location of the VCF Automation VMs and the internal metadata stored in the VCF services runtime. Manual Storage vMotion tasks are not dynamically tracked by the VSP (vSphere Stack Platform) controller, leading to a mismatch during the "Import Configuration" precheck.
Currently there is no supported clean method to perform this maintenance task from the vSphere client/VCF Ops UI to do this clean without post cleanup steps.

For Upgrade of VCF Components from 9.0 to 9.1:

VCF 9.1 uses a blue-green upgrade model for VIDB and VCF Automation. In this model, a new VCF services runtime cluster is provisioned and the component's persistent disks (PVCs) are re-attached from the 9.0 cluster to the 9.1 cluster. This re-attachment requires both the source and destination clusters to reside on the same datastore.
If the VIDB or VCF-A 9.0 cluster was originally deployed on a non-default datastore (for example iSCSI, FCoE, or local storage) and the 9.1 target runtime was provisioned on a different datastore (for example vSAN), the precheck detects the mismatch and blocks the upgrade to prevent data loss or an unrecoverable state.
The only way to resolve this is to migrate the 9.0 component nodes to the same datastore that the 9.1 runtime will use, then update the Kubernetes storage configuration to reflect the new location.

Resolution

The script attached to this KB can be used to resolve issues with VC and the VCF management components storage configuration after becoming out of sync. Follow the steps below for your scenario.

Follow the steps below based on the component (VIDB / VCFA):

VIDB (VCF Identity Broker)

If failure is observed while upgrading VIDB 9.0 follow the remediation steps described in Transition a VCF Identity Broker 9.0.x Instance to the VCF Services Runtime Environment.

VCFA (VCF Automation)

Migrate the VCFA 9.0 component to use the same datastore that the 9.1 VCF services runtime will use, and then update its storage configuration to reflect the new datastore. All the following steps need to be performed in the 9.0.x cluster.

(Optional Step) Collect Support Bundle
1. Before making any changes, collect a support bundle to preserve the current state for diagnostic purposes in case any step requires further troubleshooting. To collect the bundle, follow the relevant KB article: 'Generate VCF Automation Log Bundle'.

Identify the Target Datastore from the Precheck Error

The precheck error message contains the target datastore URL that the 9.1 cluster expects, as highlighted in screenshot below. Locate this message in the upgrade workflow output or the VCF Lifecycle Management UI and note down (or copy) the Target datastore URL.

Target Datastore URL	The precheck error message — format: `ds:///vmfs/volumes/<uuid>/`
Datastore path	The vCenter inventory — format: `/<DatacenterName>/datastore/<DatastoreName>`
Datastore MoID	This can be seen from the vCenter URL for the datastore — format: `Datastore:datastore-##` Note: Select the datastore from vSphere Client Datastore Browser and the Browser URL will show the Datastore MoID.

Migrate the VCF-A 9.0 Nodes to the New Datastore in vCenter

Use the vCenter Client to perform a Storage vMotion of the VCF-A 9.0 VMs to the target datastore before updating any Kubernetes configuration.
1. In the vSphere Client, locate VCF-A 9.0 node VMs in the folder view.
2. Right-click the VMs and select Migrate.
3. Choose Change storage only.
4. Select the same datastore that the 9.1 cluster uses.
5. Complete the migration for all VCF-A 9.0 nodes.
  
  Note: All nodes in the 9.0 cluster must be on the same datastore as the 9.1 target before proceeding.
Update the datastore configuration on the 9.0 cluster by running the attached script following the steps below (Script details).

VSP (vSphere Stack Platform) - Patching VCF 9.1.x

Gather the new Datastore ID and Location from the vCenter vSphere client UI.
1. ID is found in the URL after selecting the datastore.
Location is found in the "Summary" section of the datastore.
Update the datastore configuration by running the attached script following the steps below (Script details).

===============================================================

Script details

Execute the attached script to change the datastore information within the cluster. For this operation, connect to the control plane node and execute the script there.

SSH to the Cluster Control Plane VM and login using vmware-system-user account.
Copy the attached script to /tmp/ on the control plane.
Execute the sudo command:

sudo -i
Change the permission on the script.

chmod +x /tmp/datastore-remediation.sh
Execute the script, it can be executed using any of the below methods:
1. Validate at vCenter by providing vCenter credentials, in which case it is sufficient to just provide the target datastore URL:
```
bash /tmp/datastore-remediation.sh --ds-url "ds:///vmfs/volumes/#######-####-####-####-############/" --vc-host "vcenter.example.com"
```
2. No validation of the datastore details at vCenter, in which case all 3 parameters must be passed:
```
bash /tmp/datastore-remediation.sh --ds-url "ds:///vmfs/volumes/#######-####-####-####-############/" --ds "/<DatacenterName>/datastore/<DatastoreName>" --ds-id "Datastore:datastore-<MoID>"
```
  Note: Enter the --ds-url value with the target datastore noted down in Step 2.(i). For the "--ds" parameter, enter the datastore path noted down in Step 2.(i) and --ds-id with the datastore MoID from Step 2.(i).
The script does the following two operations:
1. Updates the platform's datastore configuration — the platform was still pointing to the old datastore. The script updates 3 settings (the datastore URL, the datastore path and the datastore ID) to match where the data actually lives now.
2. Refreshes the storage class — the cluster has a storage class that tells it which datastore to use for new volumes. After the configuration is updated, the script deletes the stale storage class so the platform automatically re-creates it with the correct datastore. It takes a backup of the storage class before deleting it, and waits to confirm the new one is in place before it exits.

A successful run prints the following output to the console:

root [ /tmp ]# bash datastore-remediation.sh --ds-url "ds:///vmfs/volumes/vsan:###########-###########/" --vc-host "vcenter.example.com"

[INFO] Configuration:
  Kubeconfig     : <kubeconfig path>
  Datastore path : (to be discovered from URL via vCenter)
  Datastore URL  : ds:///vmfs/volumes/vsan:###########-###########/
  vCenter host   : vcenter.example.com
  vCenter user   : [email protected]

[INFO] Step 0 — vCenter datastore validation
vCenter password for [email protected]: 

[INFO] Testing vCenter connection at https://vcenter.example.com...
[INFO] vCenter connection established.
[INFO] Searching all vCenter datastores for URL 'ds:///vmfs/volumes/vsan:###########-###########/'...
[INFO] Auto-discovered --ds from vCenter: /<DatacenterName>/datastore/<DatastoreName>

[INFO] Querying vCenter for datastore '/<DatacenterName>/datastore/<DatastoreName>'...
[INFO] vCenter returned:
  Datastore URL : ds:///vmfs/volumes/vsan:###########-###########/
  Datastore ID  : Datastore:datastore-##

[INFO] --ds-url matches vCenter.
[INFO] Auto-discovered --ds-id from vCenter: Datastore:datastore-##

[INFO] Validated values:
  Datastore URL  : ds:///vmfs/volumes/vsan:###########-###########/
  Datastore ID   : Datastore:datastore-##

[INFO] Starting datastore mismatch remediation...

[INFO] Step 1/6 — Backing up and deleting StorageClass vmsp-default...
[INFO]   StorageClass backed up to /backuplocation/storageclass-vmsp-default-YYYYMMDD_HHMMSS.yaml
[INFO]   Checking for delete-protection label...
[INFO]   Removing delete-protection label...
storageclass.storage.k8s.io/vmsp-default patched
storageclass.storage.k8s.io "vmsp-default" deleted
[INFO]   StorageClass deleted.

[INFO] Step 2/6 — Patching PackageDeployment: globalConfig.provider.vsphere.datastoreURL...
packagedeployment.releases.vmsp.vmware.com/vmsp-platform patched
[INFO]   datastoreURL = ds:///vmfs/volumes/vsan:###########-###########/

[INFO] Step 3/6 — Patching PackageDeployment: provider.vsphere datastoreURL and datastore path...
packagedeployment.releases.vmsp.vmware.com/vmsp-platform patched
[INFO]   datastoreURL = ds:///vmfs/volumes/vsan:###########-###########/
[INFO]   datastore    = /<DatacenterName>/datastore/<DatastoreName>

[INFO] Step 4/6 — Patching PackageDeployment: provider.vsphere.datastoreId (9.1)...
packagedeployment.releases.vmsp.vmware.com/vmsp-platform patched
[INFO]   datastoreId = Datastore:datastore-##

[INFO] Step 5/6 — Waiting for StorageClass to be re-created (timeout: 120 s)...
[INFO]   StorageClass re-created:
NAME                     PROVISIONER              RECLAIMPOLICY   VOLUMEBINDINGMODE      ALLOWVOLUMEEXPANSION   AGE
vmsp-default (default)   csi.vsphere.vmware.com   Delete          WaitForFirstConsumer   true                   2s

[INFO] Step 6/6 — Waiting for PackageDeployment to be Successful (timeout: 30 minutes)...

packagedeployment.releases.vmsp.vmware.com/vmsp-platform condition met
[INFO]   PackageDeployment is Successful!

========================================================================
[INFO] === REMEDIATION COMPLETED SUCCESSFULLY ===
[INFO] 
[INFO] IMPORTANT: Before retrying the upgrade operation:
[INFO]   1. Verify that all nodes are in a Ready state
[INFO]   2. Ensure the cluster is fully stable
[INFO] 
[INFO] Return to the VCF Lifecycle Management UI and retry the upgrade precheck.
========================================================================

Retry the Failed Upgrade or VCF 9.1 patch from the VCF Lifecycle Management UI.
1. The datastore mismatch precheck should now pass and proceed normally.

Additional Information

SHA256 Checksum of the attached script

datastore-remediation.sh - 59956F2DDC5E47B80DC9353248236DFADE0DB612AEC65AF50BA2FF60745B2551

Attachments

datastore-remediation.sh get_app