Error running mp-adapter: failed to determine NSX CA settings:

Products

• VMware Container Networking with Antrea
• VMware vSphere Kubernetes Service

Issue/Introduction

• vCenter and Supervisor are version 9.0.1 or above.
• Antrea Interworking pod goes into crashloopbackoff state.
• From the guest cluster, the logs show the mp-adapter container is unable to read the NSX cert.
  
  kubectl logs pod/interworking-############-#######-n vmware-system-antrea -c mp-adapter
F0617 08:35:31.423494      17 main.go:103] Error running mp-adapter: failed to determine NSX CA settings: unable to read NSX CA file /etc/antrea/nsx-cert/ca.crt: open /etc/antrea/nsx-cert/ca.crt: no such file or directory
  
  
• From the Superivsor CLI, the NCP configmap does not contain NSX certificate. This can be checked with following command:
  
  kubectl get configmap nsx-ncp-config -n nsx-system -o yaml
  
  
• One of the following actions may have been completed before issue occurred. These actions are known to trigger the issue (non-exhaustive list).
  
  
  • Supervisor upgrade or restore
  • Adding LB to Superivsor
  • Scalling Supervisor for HA
  • Changing Supervisor CPU/memory 
  • Changing vCenter Primary Network Identifier (PNI) and/or certs
  • Repointing vCenter SSO
  • vCenter certificate changes

Environment

VMware vSphere Kubernetes Service
VMware Container Networking with Antrea

Resolution

This is a known issue impacting vSphere Kubernetes Service.

Workaround:

1. From the Supervisor, run:
   
   touch /var/lib/node.cfg
   
   
2. Confirm that cert is reinjected into NCP configmap:
   
   kubectl get configmap nsx-ncp-config -n nsx-system -o yaml
   
   
3. Confirm Interworking pod returns to a running state