Problems reconciling gateway-api kapp package after VKS guest cluster upgrade to 1.33.6

Products

• VMware vSphere Kubernetes Service

Issue/Introduction

After upgrade of Guest cluster to Kubernetes Release 1.33.6, gateway-api pkgi is failing to reconcile. 

On a describe of the cluster/clusterbootstrap the below error is seen. 

 

Cluster describe

 

    message: |-
            Addon Gateway-Api is not ready: kapp: Error: update customresourcedefinition/grpcroutes.gateway.networking.k8s.io (apiextensions.k8s.io/v1) cluster:
              Updating resource customresourcedefinition/grpcroutes.gateway.networking.k8s.io (apiextensions.k8s.io/v1) cluster:
                API server says:
                  CustomResourceDefinition.apiextensions.k8s.io "grpcroutes.gateway.networking.k8s.io" is invalid: status.storedVersions[0]:
                    Invalid value: "v1alpha2": missing from spec.versions; v1alpha2 was previously a storage version, and must remain in spec.versions until a storage migration ensures no data remains persisted in v1alpha2 and removes v1alpha2 from status.storedVersions (reason: Invalid)
                    
Clusterbootstrap
    refName: gateway-api.tanzu.vmware.com.1.2.1+vmware.6-tkg.1
    ...
    message: |-
      kapp: Error: update customresourcedefinition/grpcroutes.gateway.networking.k8s.io (apiextensions.k8s.io/v1) cluster:
        Updating resource customresourcedefinition/grpcroutes.gateway.networking.k8s.io (apiextensions.k8s.io/v1) cluster:
          API server says:
            CustomResourceDefinition.apiextensions.k8s.io "grpcroutes.gateway.networking.k8s.io" is invalid: status.storedVersions[0]:
              Invalid value: "v1alpha2": missing from spec.versions; v1alpha2 was previously a storage version, and must remain in spec.versions until a storage migration ensures no data remains persisted in v1alpha2 and removes v1alpha2 from status.storedVersions (reason: Invalid)
    status: "True"
    type: Gateway-Api-ReconcileFailed


   


Environment

VMware vSphere Kubernetes Service
vSphere Supervisor

Cause

The v1alpha2 storage version is removed in gateway api v1.2.0, Kr 1.33 contains this version  As this field is not automatically pruned, manual intervention is needed. To check the
value, you can run the following commands:

kubectl get crd grpcroutes.gateway.networking.k8s.io -ojsonpath="{.status.storedVersions}"
storedVersions:
- v1alpha2
- v

 

 

Resolution

In order to resolve patch the GRPCRoute CRD to remove reference to v1alpha2

kubectl patch customresourcedefinitions grpcroutes.gateway.networking.k8s.io --subresource='status' --type='merge' -p '{"status":{"storedVersions":["v1"]}}'

Additional Information

https://techdocs.broadcom.com/us/en/vmware-cis/vcf/vsphere-supervisor-services-and-standalone-components/latest/release-notes/vmware-vkr-release-notes.html#GUID-4075ed17-85c9-4865-b0cd-12c22eb06e60-en_id-dc2262af-df07-4456-9525-c7af0549b460

https://github.com/kubernetes-sigs/gateway-api/releases/tag/v1.2.0