Symptoms:
2022-12-05T12:01:09.305Z c170103f-####-####-####-########140 NSX 19235 - [nsx@6876 comp="nsx-container-ncp" subcomp="ncp" level="WARNING"] nsx_ujo.ncp.k8s.ingress_lb_controller Ingress update workflow failed for ingress ('new-test', 'test-ingress'): 'http'
2022-12-05T12:01:09.322Z c170103f-####-####-####-########140 NSX 19235 - [nsx@6876 comp="nsx-container-ncp" subcomp="ncp" level="WARNING"] nsx_ujo.common.controller IngressLbController worker 1 failed to sync ('new-test', 'test-ingress') due to unexpected exception
Traceback (most recent call last):
File "/usr/local/lib/python3.6/dist-packages/nsx_ujo/common/controller.py", line 463, in worker
self.sync_handler(key)
File "/usr/local/lib/python3.6/dist-packages/nsx_ujo/ncp/k8s/ingress_lb_controller.py", line 63, in sync_handler
self.l7_lb_ingress_modify(ing_obj)
File "/usr/local/lib/python3.6/dist-packages/nsx_ujo/ncp/k8s/kubernetes.py", line 1870, in wrapper_f
func(*args, **kwargs)
File "/usr/local/lib/python3.6/dist-packages/nsx_ujo/ncp/k8s/ingress_lb_controller.py", line 345, in l7_lb_ingress_modify
if not self._validate_ingress(ing_obj):
File "/usr/local/lib/python3.6/dist-packages/nsx_ujo/ncp/k8s/ingress_lb_controller.py", line 438, in _validate_ingress
if not self._validate_host_paths(ing_obj, rule_param):
File "/usr/local/lib/python3.6/dist-packages/nsx_ujo/ncp/k8s/ingress_lb_controller.py", line 554, in _validate_host_paths
if (rule_param in stored_ing.host_paths and
File "/usr/local/lib/python3.6/dist-packages/nsx_ujo/ncp/k8s/store.py", line 719, in host_paths
for path_spec in rule['http'].get('paths', []):
KeyError: 'http'
When NCP process's an ingress, it goes over all ingresses in its store and validate them.
The error message seen above indicates there was a validation error, that an ingress in the store is improperly formatted and, in this case, points to the field: KeyError: 'http' which causes a sync failure
To review all the ingresses you have and find which one has the incorrect format, you can use either of the two commands:
kubectl get ingress -A -o yaml
or
kubectl get ingress -A -o json | jq '.items[]|select(.spec.rules[]|select(has("http")|not))'
The second command will provide much more detailed information than the first
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
name: bad-ingress
annotations:
kubernetes.io/ingress.class: non-nsx
spec:
rules:
- host: bad1.example.com
- host: bad2.example.com
http:
paths:
- backend:
service:
name: bad-svc
port:
number: 80
path: /bad
pathType: Exact
# 2. create a valid ingress in the same namespace:
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
name: tea-ingress
spec:
rules:
- host: tea.example.com
http:
paths:
- backend:
service:
name: tea-svc
port:
number: 80
path: /tea
pathType: Exact