NCP00050 "Failed to update virtual server <virtual_server_name>" with Field level validation errors: {required property rules[27].match_conditions[0].sni is missing}

Products

• VMware NSX

Issue/Introduction

• You have recently upgrade to OpenShift 4.14 or higher
• In NSX inventory, the cluster shows as down, with error NCP00050 "Failed to update virtual server <virtual_server_name>"
• You have verified that the load balancer secret is present and correct if using route TLS. 
• In the NCP logs, you may see similar to

2024-09-04T10:21:33.650Z <server_name> NSX 8 - [nsx@6876 comp="nsx-container-ncp" subcomp="ncp" level="ERROR" errorCode="NCP00050"] nsx_ujo.ncp.nsx.policy.nsxapi wrapper_f failed, cause: Unexpected error from backend manager (['###.###.###.###']) for PATCH policy/api/v1/infra/lb-virtual-servers/<virtual_server_name>: Field level validation errors: {required property rules[27].match_conditions[0].sni is missing} details: Field level validation errors: {required property rules[27].match_conditions[0].sni is missing}, args: ('<virtual_server_name>',), kwargs: {'rules': [{'phase': 'TRANSPORT', 'match_strategy': 'ALL', 'match_conditions': [[truncated]...

Environment

OpenShift 4.14

VMware NSX

 

Cause

Previously, the Ingress Operator could not successfully update the canary route because the Operator did not have permission to update spec.host or spec.subdomain on an existing route.

With this release of OpenShift 4.14, the required permission is added to the cluster role for the Operator's service account and the Ingress Operator can update the canary route. (OCPBUGS-36467)

Resolution

This is a known issue impacting VMware NSX

 

Workaround:

1) Create a service of type LB for the canary deployment.
   Configure LB for port 443 mapping to port 8888 and port 80 mapping to port 8080
2) Create a second L7 Virtual Server in NSX, ensuring the IP used for it is allocated in the external IP pool.
   Copy all of the settings from the existing Virtual Server to this new one. 
   No rule needs to be created if the pool for port 8080 created in the step above was used as default pool for the LB
3) Update upstream DNS config to point canary-openshift-canary specific DNS, to the Virtual Server created in step 2
4) Reconfigure OpenShift DNS operator to change upstream resolvers as 

 upstreamResolvers:
    policy: Sequential
    protocolStrategy: ""
    transportConfig: {}
    upstreams:
    - address: 
      port: 53
      type: Network

5) Trigger a rollout deployment of openshift-dns coredns pods to ensure all DNS caches were cleared
6) Delete/Recreate openshift-ingress-operator pod as the operator was also caching the previous DNS entry for canary-openshift-canary route.