Error: "Source and target systems use different datastores" - Upgrade of VCF Components from 9.0 to 9.1 fails during Import Validation Precheck

Products

VCF Automation VMware Cloud Foundation

Issue/Introduction

The VIDB or VCF-A upgrade from 9.0 to 9.1 fails at the Component Import Validation Precheck.
The precheck error message reports that the source (9.0) and target (9.1) clusters are using different datastores.
The error message includes the expected target datastore URL.
Retrying the upgrade without intervention continues to fail at the same precheck.
The upgrade prechecks shows the following error message for both of these components:

Source and target systems use different datastores: Source=ds:///vmfs/volumes/###############/, Target=ds:///vmfs/volumes/###############/. Please configure and retry the operation or contact support if the issue persists. [VCFMS-PRECHECK-IMPORT-###]

For this upgrade to proceed, the source and target datastores must be the same. Make necessary adjustments and retry. If the issue persists, please contact support.
Import Configuration Validation Precheck Failed
Import precheck completed with errors. [VCFMS-PRECHECK-IMPORT-###]

Environment

VCF Automation 9.0.x
VCF Automation 9.1
VCF Identity Broker 9.0.x
VCF Identity Broker 9.1

Cause

VCF 9.1 uses a blue-green upgrade model for VIDB and VCF Automation. In this model, a new VCF services runtime cluster is provisioned and the component's persistent disks (PVCs) are re-attached from the 9.0 cluster to the 9.1 cluster. This re-attachment requires both the source and destination clusters to reside on the same datastore.
If the VIDB or VCF-A 9.0 cluster was originally deployed on a non-default datastore (for example iSCSI, FCoE, or local storage) and the 9.1 target runtime was provisioned on a different datastore (for example vSAN), the precheck detects the mismatch and blocks the upgrade to prevent data loss or an unrecoverable state.
The only way to resolve this is to migrate the 9.0 component nodes to the same datastore that the 9.1 runtime will use, then update the Kubernetes storage configuration to reflect the new location.

Resolution

Migrate the 9.0 component to use the same datastore that the 9.1 VCF services runtime will use, and then update its storage configuration to reflect the new datastore. All the following steps need to be performed in the 9.0.x cluster.

(Optional Step) Collect Support Bundle
1. Before making any changes, collect a support bundle to preserve the current state for diagnostic purposes in case any step requires further troubleshooting. To collect the bundle, follow the relevant KB article: 'Generate VCF Automation Log Bundle' or 'How to Manually Generate and Collect a vIDB Cluster Log Bundle'.

Identify the Target Datastore from the Precheck Error

The precheck error message contains the target datastore URL that the 9.1 cluster expects, as highlighted in screenshot below. Locate this message in the upgrade workflow output or the VCF Lifecycle Management UI and note down (or copy) the Target datastore URL.

Target Datastore URL	The precheck error message — format: `ds:///vmfs/volumes/<uuid>/`
Datastore path	The vCenter inventory — format: `/<DatacenterName>/datastore/<DatastoreName>`
Datastore MoID	This can be seen from the vCenter URL for the datastore — format: `Datastore:datastore-##` Note: Select the datastore from vSphere Client Datastore Browser and the Browser URL will show the Datastore MoID.

Migrate the VIDB / VCF-A 9.0 Nodes to the New Datastore in vCenter

Use the vCenter Client to perform a Storage vMotion of the VIDB / VCF-A 9.0 VMs to the target datastore before updating any Kubernetes configuration.
1. In the vSphere Client, locate VIDB / VCF-A 9.0 node VMs in the folder view.
2. Right-click the VMs and select Migrate.
  
  Note: In the above screenshot, VIDB nodes are shown as example. The same is applicable for VCF-A 9.0 nodes as well.
3. Choose Change storage only.
4. Select the same datastore that the 9.1 cluster uses.
5. Complete the migration for all VIDB / VCF-A 9.0 nodes.
  
  Note: All nodes in the 9.0 cluster must be on the same datastore as the 9.1 target before proceeding.

Update the datastore configuration on the 9.0 cluster

Execute the attached script to change the datastore information within the cluster. For this operation, connect to the control plane node and execute the script there.

SSH to the Cluster Control Plane VM and login using vmware-system-user account.
Copy the attached script to /tmp/ on the control plane.
Execute the sudo command:

sudo -i
Change the permission on the script.

chmod +x /tmp/datastore-remediation.sh
Execute the script, it can be executed using any of the below methods:
1. Validate at vCenter by providing vCenter credentials, in which case it is sufficient to just provide the target datastore URL:
```
bash /tmp/datastore-remediation.sh --ds-url "ds:///vmfs/volumes/#######-####-####-####-############/" --vc-host "vcenter.example.com"
```
2. No validation of the datastore details at vCenter, in which case all 3 parameters must be passed:
```
bash /tmp/datastore-remediation.sh --ds-url "ds:///vmfs/volumes/#######-####-####-####-############/" --ds "/<DatacenterName>/datastore/<DatastoreName>" --ds-id "Datastore:datastore-<MoID>"
```
  Note: Enter the --ds-url value with the target datastore noted down in Step 2.(i). For the "--ds" parameter, enter the datastore path noted down in Step 2.(i) and --ds-id with the datastore MoID from Step 2.(i).
The script does the following two operations:
1. Updates the platform's datastore configuration — the platform was still pointing to the old datastore. The script updates 3 settings (the datastore URL, the datastore path and the datastore ID) to match where the data actually lives now.
2. Refreshes the storage class — the cluster has a storage class that tells it which datastore to use for new volumes. After the configuration is updated, the script deletes the stale storage class so the platform automatically re-creates it with the correct datastore. It takes a backup of the storage class before deleting it, and waits to confirm the new one is in place before it exits.

A successful run prints the following output to the console:

root [ /tmp ]# bash datastore-remediation.sh --ds-url "ds:///vmfs/volumes/vsan:###########-###########/" --vc-host "vcenter.example.com"

[INFO] Configuration:
  Kubeconfig     : <kubeconfig path>
  Datastore path : (to be discovered from URL via vCenter)
  Datastore URL  : ds:///vmfs/volumes/vsan:###########-###########/
  vCenter host   : vcenter.example.com
  vCenter user   : [email protected]

[INFO] Step 0 — vCenter datastore validation
vCenter password for [email protected]: 

[INFO] Testing vCenter connection at https://vcenter.example.com...
[INFO] vCenter connection established.
[INFO] Searching all vCenter datastores for URL 'ds:///vmfs/volumes/vsan:###########-###########/'...
[INFO] Auto-discovered --ds from vCenter: /<DatacenterName>/datastore/<DatastoreName>

[INFO] Querying vCenter for datastore '/<DatacenterName>/datastore/<DatastoreName>'...
[INFO] vCenter returned:
  Datastore URL : ds:///vmfs/volumes/vsan:###########-###########/
  Datastore ID  : Datastore:datastore-##

[INFO] --ds-url matches vCenter.
[INFO] Auto-discovered --ds-id from vCenter: Datastore:datastore-##

[INFO] Validated values:
  Datastore URL  : ds:///vmfs/volumes/vsan:###########-###########/
  Datastore ID   : Datastore:datastore-##

[INFO] Starting datastore mismatch remediation...

[INFO] Step 1/6 — Backing up and deleting StorageClass vmsp-default...
[INFO]   StorageClass backed up to /backuplocation/storageclass-vmsp-default-YYYYMMDD_HHMMSS.yaml
[INFO]   Checking for delete-protection label...
[INFO]   Removing delete-protection label...
storageclass.storage.k8s.io/vmsp-default patched
storageclass.storage.k8s.io "vmsp-default" deleted
[INFO]   StorageClass deleted.

[INFO] Step 2/6 — Patching PackageDeployment: globalConfig.provider.vsphere.datastoreURL...
packagedeployment.releases.vmsp.vmware.com/vmsp-platform patched
[INFO]   datastoreURL = ds:///vmfs/volumes/vsan:###########-###########/

[INFO] Step 3/6 — Patching PackageDeployment: provider.vsphere datastoreURL and datastore path...
packagedeployment.releases.vmsp.vmware.com/vmsp-platform patched
[INFO]   datastoreURL = ds:///vmfs/volumes/vsan:###########-###########/
[INFO]   datastore    = /<DatacenterName>/datastore/<DatastoreName>

[INFO] Step 4/6 — Patching PackageDeployment: provider.vsphere.datastoreId (9.1)...
packagedeployment.releases.vmsp.vmware.com/vmsp-platform patched
[INFO]   datastoreId = Datastore:datastore-##

[INFO] Step 5/6 — Waiting for StorageClass to be re-created (timeout: 120 s)...
[INFO]   StorageClass re-created:
NAME                     PROVISIONER              RECLAIMPOLICY   VOLUMEBINDINGMODE      ALLOWVOLUMEEXPANSION   AGE
vmsp-default (default)   csi.vsphere.vmware.com   Delete          WaitForFirstConsumer   true                   2s

[INFO] Step 6/6 — Waiting for PackageDeployment to be Successful (timeout: 30 minutes)...

packagedeployment.releases.vmsp.vmware.com/vmsp-platform condition met
[INFO]   PackageDeployment is Successful!

========================================================================
[INFO] === REMEDIATION COMPLETED SUCCESSFULLY ===
[INFO] 
[INFO] IMPORTANT: Before retrying the upgrade operation:
[INFO]   1. Verify that all nodes are in a Ready state
[INFO]   2. Ensure the cluster is fully stable
[INFO] 
[INFO] Return to the VCF Lifecycle Management UI and retry the upgrade precheck.
========================================================================

Retry the Failed Upgrade
1. Return to the VCF Lifecycle Management UI and retry the VIDB or VCF-A upgrade from 9.0 to 9.1. The datastore mismatch precheck should now pass and the upgrade should proceed normally.

Additional Information

SHA256 Checksum of the attached script

datastore-remediation.sh - 59956F2DDC5E47B80DC9353248236DFADE0DB612AEC65AF50BA2FF60745B2551

Attachments

datastore-remediation.sh get_app