Openshift application pods stuck in CrashLoopBackOff post upgrading to version 4.20

Products

• VMware NSX

Issue/Introduction

• Openshift pods can get stuck in CrashLoopBackOff post upgrading to Openshift version 4.20
• Below error can be observed while describing the pod stuck in crash state indicating that the pod is unable to reach the Kube API service IP, which is essential for the pod to go into a Running state:
  
  Error: error configuring catalog operator: Get "https://<Kube-service-IP>:443/apis/operators.coreos.com/v1/operatorgroups": dial tcp <Kube-service-IP>:443: i/o timeout

• Tarceflow on NSX for this communications shows traffic getting blocked by DFW rules.

Environment

VMware NSX

NCP

Cause

• Below field is configured in NCP configmap file:
  label_filtering_regex_list = .*

NOTE: This is a manual configuration and is not present by default in the NCP config map.


• In NCP code, label_filtering_regex_list is a FILTER-OUT (DENYLIST) regex list. Setting it to .* causes NCP to match and drop ALL Pod labels, reducing pod.labels to {}.
• This results in network policy rule translation to DFW without required tags due to which, the allow rules do not get pushed to the Pod nic filter.

Resolution

Set the label_filtering_regex_list value to blank in nsx-ncp-operator-config configmap as below, which pushes the same to NCP configmap followed by a restart of NCP:

label_filtering_regex_list = ""