nginx still showing old version after patching

Products

• VMware Integrated OpenStack

Issue/Introduction

• VMware Integrated Openstack deployment patched to version 7.2
• Ingress pod, vio-ingress-cntl-nginx-ingress-controller-xxxxx, from default namespace still showing old version 1.15.x
• Ingress pod, ingress-xxxx-xxxx, from openstack namespace still showing old version 1.15.x
• New deploys of version 7.2 will show the updated version

Environment

7.x

Cause

The running ingress pod, vio-ingress-cntl-nginx-ingress-controller-xxxxx, is still using old docker image at local, even though new image has been updated in system image repository after patching.

Resolution

Workaround:

From VIO manager

1. List local docker image

docker images | grep nginx-ingress

# docker images | grep nginx-ingress
docker-registry.default.svc.cluster.vio:5000/vmware/kubernetes-ingress-controller/nginx-ingress-controller   v0.24.1                   cc46e42aa763        23 months ago       1.01GB

2. Remove local docker image

docker rmi <docker image from step above> --force

root@oms [ ~ ]# docker rmi docker-registry.default.svc.cluster.vio:5000/vmware/kubernetes-ingress-controller/nginx-ingress-controller:v0.24.1 --force
Untagged: docker-registry.default.svc.cluster.vio:5000/vmware/kubernetes-ingress-controller/nginx-ingress-controller:v0.24.1
Untagged: docker-registry.default.svc.cluster.vio:5000/vmware/kubernetes-ingress-controller/nginx-ingress-controller@sha256:####

2. Make sure the local image is deleted successfully

docker images | grep nginx-ingress 

3. Get the ingress pod name.

kubectl get pod |grep ingress

# kubectl get pod | grep ingress
vio-ingress-cntl-nginx-ingress-controller-####-####        1/1     Running   2          314d

4. Delete the ingress pod

kubectl delete pod <pod from above step>

5. After nginx-ingress pod is successfully restarted, please check nginx version inside it.

kubectl exec -it <new ingress pod name> -- nginx -v

# kubectl exec -it vio-ingress-cntl-nginx-ingress-controller-####-####-#### -- nginx -v
nginx version: nginx/1.20.1
 

​​​​​From the controller nodes

1. Get controller nodes

viocli get controllers

2. ssh into controller node

viossh <controller name from above step>

3. Get local ingress image

sudo docker images | grep ingress

$ sudo docker images | grep ingress
docker-registry.default.svc.cluster.vio:5000/vmware/kubernetes-ingress-controller/nginx-ingress-controller    v0.24.1                      cc46e42aa763        23 months ago       1.01GB

4. Delete local ingress image (if it exists)

sudo docker rmi <docker image from above step> --force

$ sudo docker rmi docker-registry.default.svc.cluster.vio:5000/vmware/kubernetes-ingress-controller/nginx-ingress-controller:v0.24.1 --force
Untagged: docker-registry.default.svc.cluster.vio:5000/vmware/kubernetes-ingress-controller/nginx-ingress-controller:v0.24.1
Untagged: docker-registry.default.svc.cluster.vio:5000/vmware/kubernetes-ingress-controller/nginx-ingress-controller@sha256:####

5. Repeat the above steps for each controller node

From VIO Manager

1. Get ingress pods (pods named like ingress-xxxxxxxxx-xxxxx )

osctl get pods | grep ingress

# osctl get pods | grep ingress
ingress-xxxxxxxxx-xxxxx                                          3/3     Running     0          25m
ingress-xxxxxxxxx-xxxxx                                          3/3     Running     0          30m

2. Delete ingress pods

osctl delete pod <ingress-xxxxx-xxxx pods from above step>

# osctl delete pod ingress-xxxxxxxxx-xxxxx
pod "ingress-xxxxxxxxx-xxxxx" deleted

3. After ingress pods are successfully restarted, please check nginx version inside it

osctl exec -it <new ingress pod name> -- nginx -v

# osctl exec -it ingress-xxxxxxxxx-xxxxx -- nginx -v
Defaulting container name to ingress.
Use 'kubectl describe pod/ingress-xxxxxxxxx-xxxxx -n openstack' to see all of the containers in this pod.
nginx version: nginx/1.20.1

Additional Information

Note:  The attached bash script kb_321774.txt contains a script that can be used in place of these manual steps.