Antrea Interworking pod crashing after upgrade of Tanzu TKGm from 2.5 to 2.5.4

Products

• VMware Tanzu Kubernetes Grid Management
• VMware Container Networking with Antrea

Issue/Introduction

• Antrea Interworking pod is crashing.
  
  kubectl get pod -A | grep internworking
vmware-system-antrea                  interworking-[ID]                                    3/4     CrashLoopBackOff   608 (39s ago)   2d3h
  
  
• A describe on the Interworking pods shows ccp-adapter container is crashing.
  
  kubectl describe po -n vmware-system-antrea interworking-[ID]
...
Events:
  Type     Reason   Age                     From     Message
  ----     ------   ----                    ----     -------
  Normal   Created  11m (x607 over 2d3h)    kubelet  Created container: ccp-adapter
  Normal   Pulled   106s (x609 over 2d3h)   kubelet  Container image "/tkg/antrea/interworking-ubuntu:0.9.1" already present on machine
  Warning  BackOff  68s (x14574 over 2d3h)  kubelet  Back-off restarting failed container ccp-adapter in pod interworking-[ID]_vmware-system-antrea(UUID)
  
  
• CCP-Adapter logs shows Antrea unable to locate a number of CRDs.
  
  kubectl logs -n vmware-system-antrea interworking-[ID] -c ccp-adapter | grep -i "failed to initialize"
F0217 14:23:26.617540      17 crdconverter.go:234] Failed to initialize CRD Converter CR store ClusterNetworkPolicy: the server could not find the requested resource (get clusternetworkpolicies.crd.antrea.io)
F0217 14:23:26.617595      17 crdconverter.go:234] Failed to initialize CRD Converter CR store ClusterGroup: the server could not find the requested resource (get clustergroups.crd.antrea.io)
  
  
• CRDs are present.
  
  kubectl get crd | grep -i clusternetworkpolicies.crd.antrea.io
...
clustergroups.crd.antrea.io                                                     2022-06-11T15:53:50Z 
clusternetworkpolicies.crd.antrea.io                                            2022-06-11T15:53:47Z
  ...
  
  
•  Antrea version is 2.3
  
  kubectl get pkgi -n tkg-system
NAME                   PACKAGE NAME                            PACKAGE VERSION             DESCRIPTION           AGE      PAUSED
antrea                 antrea.tanzu.vmware.com                 2.3.0+vmware.1-tkg.1        Reconcile succeeded   3y251d 
  
  
  
• Antrea Interworking version is 0.91
  
  kubectl describe pod -n vmware-system-antrea interworking-[ID] | grep -i "image:"
Image:         ODCCBSNPHBR.rbi1.rbi.org.in/tkg/antrea/interworking-ubuntu:0.9.1

Environment

VMware NSX
VMware Container Networking with Antrea
VMware Tanzu Kubernetes Grid Management

Cause

Antrea Interworking version is not compatible with Antrea.

Resolution

Upgrade Antrea Interworking to a compatible version.

Additional Information

Antrea Interworking compatibility.

antrea 2.5 - antrea-interworking 1.5
antrea 2.4 - antrea-interworking 1.4
antrea 2.3 - antrea-interworking 1.3
antrea 2.2 - antrea-interworking 1.2