VKS upgrade pre-check fails with "key vmware-system.compatibilityoffering not found in cvmi.Status.VMwareSystemProperties"

Products

• VMware vSphere Kubernetes Service

Issue/Introduction

During a (vSphere Kubernetes Service (VKS) upgrade pre-check the compatibility pre-check fails with an error similar to:

Failed to run compatibility check for Supervisor Service tkg.vsphere.vmware.com on the supervisor (<supervisor-id>):
error when querying svc-tkg-.../upgrade-compatibility-serv ice:
"key vmware-system.compatibilityoffering not found in cvmi.Status.VMwareSystemProperties on cvmi <cvmi-name>"

The upgrade is blocked even though VKS and its associated content libraries appear to be configured correctly.

Environment

VMware vSphere Kubernetes Service 3.7

Cause

The UCS pre-check was enumerating all ClusterVirtualMachineImage (CVMI) objects present on the Supervisor that included images from Non-VKS content libraries which requires every CVMI to carry the vmware-system.compatibilityoffering OVF metadata property in status.vmwareSystemProperties. Non-VKS images (for example, images from a partner or customer-managed content library) do not have this property, causing the pre-check to fail.

A second trigger for the same error is when a VKS content library has been associated with the Supervisor but has not yet finished syncing its items. In that state, CVMI objects are created but their OVF metadata (including vmware-system.compatibilityoffering) is not yet populated, so the UCS pre-check reports the same missing-key error.

Resolution

This issue is fixed in the release containing commit e5b05609 (GCM-17374). The fix changes the UCS pre-check to:

Only consider CVMIs that carry a VKS library service-type label (type.services.vmware.com/tkg, services.supervisor.vmware.com/kubernetes.vmware.com, or services.supervisor.vmware.com/any). CVMIs from non-VKS libraries are silently skipped.
Skip — rather than fail — any VKS-labelled CVMI that does not yet have the vmware-system.compatibilityoffering property populated (for example, a CVMI still being synced), instead of blocking the entire pre-check.
Upgrade to the release that includes this fix to permanently resolve the issue.

Workaround:

If using an affected build and cannot upgrade immediately, two workarounds are available depending on the root cause:

Root Cause 1: If a Non-VKS content library is present unlink (ie. dissociate) the non-VKS content library from the Supervisor's image settings before running the upgrade pre-check, then re-associate it after the upgrade completes.

How to Unlink and Relink Non-VKS Content Libraries from the Supervisor

Step 1: Connect to vCenter Server via SSH

Step 2: Log in to DCLI

dcli +i +username [e-mail] +password <vc-password>

Replace <vc-password> with the vCenter [e-mail] password.

Step 3: List the Supervisors on the vCenter Server

dcli> namespacemanagement supervisors summary list

Example output:

items:
   - supervisor: 700e0eb3-8394-43a0-b244-############
     info:
        stats:
           cpu_used: 3684
           storage_capacity: 162184
           memory_used: 47174
           cpu_capacity: 49761
           memory_capacity: 99014
           storage_used: 58200
        APIEndpoint: 172.16.0.200
        kubernetes_status: READY
        name: supervisor
        messages:
        config_status: RUNNING

Note the supervisor ID for the Supervisor you plan to upgrade — you'll need it in the next steps.

Step 4: List the Content Libraries Associated with the Supervisor. Use the <supervisor-id> from above output. 

dcli> namespacemanagement supervisors workloads images settings get --supervisor <supervisor-id>

Example output:

dcli> namespacemanagement supervisors workloads images settings get --supervisor 700e0eb3-8394-43a0-b244-############
registry:
   hostname:
   port: 443
content_libraries:
   - supervisor_services:
      - kubernetes.vmware.com
     resource_naming_strategy:
     content_library: 3ed16d53-833f-44bf-a93d-############
   - supervisor_services:
     resource_naming_strategy:
     content_library: 17cc807c-19e2-4768-9775-############

The content library with kubernetes.vmware.com listed under supervisor_services is the VKS content library — keep this one.

Any other library listed (with no supervisor_services, or a different value) is a non-VKS content library.

⚠️ Important: Record the IDs of all non-VKS content libraries currently linked. You will need these IDs to re-link them after the VKS upgrade completes.

Step 5: Unlink Non-VKS Content Libraries

Update the Supervisor's content library list to retain only the VKS content library (the one tagged with kubernetes.vmware.com):

dcli> namespacemanagement supervisors workloads images settings update --supervisor <supervisor-id> --content-libraries '[
{"content_library": "<vks-content-library-id>", "supervisor_services": ["kubernetes.vmware.com"]}]'

Example:

dcli> namespacemanagement supervisors workloads images settings update --supervisor 700e0eb3-8394-43a0-b244-75c254a1fde4 --content-libraries '[
{"content_library": "3ed16d53-833f-44bf-a93d-############", "supervisor_services": ["kubernetes.vmware.com"]}]'

Step 6: Verify the Update

Re-run the get command to confirm only the VKS content library remains:

dcli> namespacemanagement supervisors workloads images settings get --supervisor <supervisor-id>

Expected output:

dcli> namespacemanagement supervisors workloads images settings get --supervisor 700e0eb3-8394-43a0-b244-############
registry:
   hostname:
   port: 443
content_libraries:
   - supervisor_services:
      - kubernetes.vmware.com
     resource_naming_strategy:
     content_library: 3ed16d53-833f-44bf-a93d-############

Step 7: Confirm Cleanup on the Supervisor

Allow a few minutes for the Supervisor to clean up the corresponding Kubernetes resources. Then SSH into the Supervisor control plane and run:

k get clustercontentlibrary -A

Confirm that the non-VKS content libraries no longer appear in the output.

Step 8: Proceed with the VKS Upgrade

Once cleanup is confirmed, proceed with the VKS upgrade.

Note: This issue is fixed in VKS 3.6.3 and VKS 3.7+. Upgrading to one of these versions (or later) prevents this issue from recurring on future upgrades.

Step 9: Re-link the Non-VKS Content Libraries

After the VKS upgrade completes, re-link the non-VKS content libraries you recorded in Step 4, along with the VKS content library:

dcli> namespacemanagement supervisors workloads images settings update --supervisor <supervisor-id> --content-libraries '[
{"content_library": "<vks-content-library-id>", "supervisor_services": ["kubernetes.vmware.com"]},
{"content_library": "<non-vks-content-library-id>"}]'

Example:

dcli> namespacemanagement supervisors workloads images settings update --supervisor 700e0eb3-8394-43a0-b244-############ --content-libraries '[
{"content_library": "3ed16d53-833f-44bf-a93d-############", "supervisor_services": ["kubernetes.vmware.com"]},
{"content_library": "17cc807c-19e2-4768-9775-############"}]'

Note: The example above shows only one non-VKS content library. If multiple non-VKS content libraries were originally linked, include all of them in the update request (both when unlinking in Step 5 and re-linking here).

Root Cause 2: If a VKS content library sync is in progress wait for the content library synchronization to complete. Once all content library items have fully synced and their OVF metadata is populated in the corresponding CVMI objects the pre-check will succeed. You can verify sync completion by checking that all ClusterVirtualMachineItem objects in the VKS content library are in a Ready state using the below command then re-run the upgrade pre-check after confirming sync is complete.
 
kubectl get clustercontentlibraryitem -A 

 

To find and remove the non-VKS Content Library associated with the Supervisor:

Step 1: Connect to vCenter Server via SSH  
  
Step 2: Log in to DCLI  
------------------------------------------  
dcli +i +username [e-mail] +password <vc-password>  

Step 3: List the Supervisors on the vCenter Server  
------------------------------------------  
dcli> namespacemanagement supervisors summary list  
Example output:  
items:  
- supervisor: XXX
   ...

Step 4: List the Content Libraries Associated with the Supervisor  
------------------------------------------  
dcli> namespacemanagement supervisors workloads images settings get --supervisor <supervisor-id>  
Example output:  
registry:
   hostname:
   port: 443
content_libraries:
   - supervisor_services:
     resource_naming_strategy: PREFER_ITEM_SOURCE_ID
     content_library: AAAA
   - supervisor_services:
      - kubernetes.vmware.com
     resource_naming_strategy:
     content_library: BBBB
repository:
  
  
Step 5: Unlink Non-VKS Content Libraries  
------------------------------------------  
Update the Supervisor's content library list to retain ONLY the VKS  content library (the one tagged with kubernetes.vmware.com):  
dcli> nnamespacemanagement supervisors workloads images settings update --supervisor <supervisor-id> --content-libraries '[
 {"content_library": "BBBB", "supervisor_services": ["kubernetes.vmware.com"]},
 ...
]'

  
Step 6: Verify the Update  
------------------------------------------  
Re-run the get command to confirm only the VKS content library remains:  
  
dcli> namespacemanagement supervisors workloads images settings get --supervisor <supervisor-id>  

Expected output:  
registry:  
hostname:  
port: 443  
content_libraries:  
- supervisor_services:  
- kubernetes.vmware.com  
resource_naming_strategy:  
content_library: BBBB

  
  
Step 7: Confirm Cleanup on the Supervisor  
------------------------------------------  
Allow a few minutes for the Supervisor to clean up the corresponding Kubernetes resources. Then SSH into the Supervisor control plane and run:  

k get clustercontentlibrary -A  

Confirm that the non-VKS content libraries no longer appear in the output.