VCF Upgrade Precheck Failures Due to Supervisor VM Storage Policy Non-Compliance

Products

• VMware vSphere Kubernetes Service

Issue/Introduction

Symptoms

• VCF upgrade prechecks (e.g., version 9.0.2.0 to 9.1) fail.
• Supervisor VMs are stored on local datastores that are inaccessible to the upgrade process.
• vSphere with Tanzu/Supervisor storage policy error: "Datastore is not mounted on any availability Zone."
• Supervisor services stored on local datastores (e.g., cci, argocd, envoy/contour) appear non-compliant in the storage policy view.

Environment

VCF 9.x 

Cause

Cause This error occurs when a storage policy is configured with a Zonal storage topology type, but the local datastores or host clusters have not been mapped to vSphere Zones within the Supervisor configuration. Because the Consumption Domain is set to Zonal, only shared datastores mapped to these zones satisfied the compliance requirement. Supervisor workloads residing on unmapped local datastores are marked non-compliant, preventing successful upgrade prechecks.

Resolution

To resolve this issue, you must migrate the Supervisor workloads to a shared datastore using a temporary storage policy. Follow these steps:

1. Configure a temporary non-zonal storage policy: Create a new non-zonal storage topology policy (e.g., [Temporary_Storage_Policy_Name]) and tag both the local and shared datastores.
2. Update Supervisor Storage Policy: In Supervisor Management, assign the new [Temporary_Storage_Policy_Name] to the Supervisor storage policy.
3. Bypass Permissions: Create a temporary user to bypass container permissions as outlined in KB 319388: Bypassing vSphere with Tanzu managed virtual machine permissions for troubleshooting purposes. Please engage Broadcom support for resolution.
4. Initiate Migration: Log in as the temporary user and follow the procedures in KB 319402: Migrating VMs/Volumes across Datastores in vSphere with Tanzu / vSphere Kubernetes Supervisor.
5. Update VM Storage: Edit the VM storage policies for each Supervisor VM to use  [Temporary_Storage_Policy_Name]. They should now show as compliant.
6. Perform Storage vMotion: Complete a storage vMotion of each Supervisor VM to the shared datastore.
7. Revert Policy: In Supervisor Management, assign the original Zonal storage policy (e.g., [Original_Storage_Policy_Name]) back to the Supervisor storage policy.
8. Verify Services: As per KB 319402, Log in to the Supervisor. If any Supervisor services show as non-compliant, scale them down and back up to ensure they are re-run on the shared datastore.

After completing these steps, the VCF upgrade prechecks should complete without Supervisor-related errors.

Additional Information

Please engage Broadcom support for resolution to fully implement KBs 319402  and 319388