Gateway API PackageInstall fails reconciliation: referencegrants CRD storage version error

Products

• VMware vSphere Kubernetes Service

Issue/Introduction

• During a workload cluster upgrade, the PackageInstall (PKGi) for the Gateway API fails to reconcile, which causes the package to report a ReconcileFailed status. This state prevents the upgrade from reaching successful completion.

• When describing the cluster or the clusterbootstrap resource, a detailed error message similar to the following is observed:

message: |-
    kapp: Error: update customresourcedefinition/referencegrants.gateway.networking.k8s.io (apiextensions.k8s.io/v1) cluster:
      Updating resource customresourcedefinition/referencegrants.gateway.networking.k8s.io (apiextensions.k8s.io/v1) cluster:
        API server says:
          CustomResourceDefinition.apiextensions.k8s.io "referencegrants.gateway.networking.k8s.io" is invalid: status.storedVersions[0]:
            Invalid value: "v1alpha2": missing from spec.versions; v1alpha2 was previously a storage version, and must remain in spec.versions until a storage migration ensures no data remains persisted in v1alpha2 and removes v1alpha2 from status.storedVersions (reason: Invalid)

• Direct inspection via kubectl get crd referencegrants.gateway.networking.k8s.io confirms that the resource is present in both the v1alpha2 and v1beta1 stored versions lists

Cause

The failure results from a storage version deadlock. Gateway API v1.2.0 (shipped with v1.33) deprecates the v1alpha2 storage version.

Because the API server does not automatically prune status.storedVersions, the legacy v1alpha2 entry persists. This creates a validation conflict: the CRD specification cannot exclude a version that the status still identifies as a persistence format in etcd. This mismatch triggers a validation error that halts PKGi reconciliation.

Resolution

To resolve the reconciliation failure, manually patch the CRD subresource to remove the legacy storage version reference. This clears the validation conflict and allows the package manager to apply the updated specification.

1. Apply a merge patch to the status subresource to retain only the v1beta1 version in the storedVersions list:

kubectl patch crd referencegrants.gateway.networking.k8s.io --subresource='status' --type='merge' -p '{"status":{"storedVersions":["v1beta1"]}}'

2. Confirm that the legacy v1alpha2 version has been removed from the status:

kubectl get crd referencegrants.gateway.networking.k8s.io -o jsonpath='{.status.storedVersions}'

Following the manual update (Step 1), the PackageInstall (PKGi) for the Gateway API will automatically resume reconciliation. Once the package transitions to a "Reconcile succeeded" state, the cluster upgrade will proceed.