NSX-T NCP fails to provide IP address for an ingress created in container environment
search cancel

NSX-T NCP fails to provide IP address for an ingress created in container environment

book

Article ID: 336825

calendar_today

Updated On:

Products

VMware NSX

Issue/Introduction

Symptoms:

  • You have a yaml file configured to create an ingress that will be pushed to NSX-T from NCP.
  • When you apply the yaml file 'kubectl apply -f <name-of-ingress>.yaml', the ingress is created, but does not have an IP address.
  • Running the command 'kubectl get ing' will retrun the ingress's and here you will notice the ADDRESS column is blank.
  • In ncp logs 'ncp.stdout.log' for the master node, you see error like the following:
2022-12-05T12:01:09.305Z c170103f-####-####-####-########140 NSX 19235 - [nsx@6876 comp="nsx-container-ncp" subcomp="ncp" level="WARNING"] nsx_ujo.ncp.k8s.ingress_lb_controller Ingress update workflow failed for ingress ('new-test', 'test-ingress'): 'http'
2022-12-05T12:01:09.322Z c170103f-####-####-####-########140 NSX 19235 - [nsx@6876 comp="nsx-container-ncp" subcomp="ncp" level="WARNING"] nsx_ujo.common.controller IngressLbController worker 1 failed to sync ('new-test', 'test-ingress') due to unexpected exception
Traceback (most recent call last):
  File "/usr/local/lib/python3.6/dist-packages/nsx_ujo/common/controller.py", line 463, in worker
    self.sync_handler(key)
  File "/usr/local/lib/python3.6/dist-packages/nsx_ujo/ncp/k8s/ingress_lb_controller.py", line 63, in sync_handler
    self.l7_lb_ingress_modify(ing_obj)
  File "/usr/local/lib/python3.6/dist-packages/nsx_ujo/ncp/k8s/kubernetes.py", line 1870, in wrapper_f
    func(*args, **kwargs)
  File "/usr/local/lib/python3.6/dist-packages/nsx_ujo/ncp/k8s/ingress_lb_controller.py", line 345, in l7_lb_ingress_modify
    if not self._validate_ingress(ing_obj):
  File "/usr/local/lib/python3.6/dist-packages/nsx_ujo/ncp/k8s/ingress_lb_controller.py", line 438, in _validate_ingress
    if not self._validate_host_paths(ing_obj, rule_param):
  File "/usr/local/lib/python3.6/dist-packages/nsx_ujo/ncp/k8s/ingress_lb_controller.py", line 554, in _validate_host_paths
    if (rule_param in stored_ing.host_paths and
  File "/usr/local/lib/python3.6/dist-packages/nsx_ujo/ncp/k8s/store.py", line 719, in host_paths
    for path_spec in rule['http'].get('paths', []):
KeyError: 'http'
  • Reviewing the ncp logs further, you may also notice this error is generated for all ingresses.
Here is a simple way to reproduce/verify the issue is occurring
 
# 1. create an invalid ingress:

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
name: bad-ingress
annotations:
kubernetes.io/ingress.class: non-nsx
spec:
rules:
- host: bad1.example.com
- host: bad2.example.com
http:
paths:
- backend:
service:
name: bad-svc
port:
number: 80
path: /bad
pathType: Exact

# 2. create a valid ingress in the same namespace:

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
name: tea-ingress
spec:
rules:
- host: tea.example.com
http:
paths:
- backend:
service:
name: tea-svc
port:
number: 80
path: /tea
pathType: Exact
 



Environment

VMware NSX-T Data Center 3.x
VMware NSX-T Data Center

Cause

When NCP process's an ingress, it goes over all ingresses in its store and validate them.

The error message seen above indicates there was a validation error, that an ingress in the store is improperly formatted and, in this case, points to the field: KeyError: 'http' which causes a sync failure

To review all the ingresses you have and find which one has the incorrect format, you can use either of the two commands:

kubectl get ingress -A -o yaml

or

kubectl get ingress -A -o json | jq '.items[]|select(.spec.rules[]|select(has("http")|not))'

The second command will provide much more detailed information than the first

Resolution

Fixed in NCP 4.10
Fixed in TKGI 1.16

Workaround:
Edit the ingress which has the missing or incorrectly formatted yaml file and apply it again

Powered by Wolken Software