vSphere Kubernetes Cluster Pods Running but Application Unavailable using Ingress Control through Contour Envoy PKGI due to Envoy Daemonset Unable to Start Envoy Pods

Products

VMware vSphere with Tanzu VMware Tanzu Mission Control

Issue/Introduction

Application is not working despite pod(s) in healthy running state in a vSphere Kubernetes cluster. This application uses ingress control through contour and envoy installed as a package (pkgi).

While connected to the vSphere Kubernetes cluster where the affected application is running, the following symptoms are present:

The pod(s) responsible for running the application are in Running state and their logs do not have any errors:
- ```
kubectl get pods -n <pod namespace>
```
- ```
kubectl logs <pod name> -n <pod namespace>
```
The contour installed package (pkgi) is in ReconcileFailed or Reconciling state:
- ```
kubectl get pkgi -A | grep contour
```
There are no envoy pods present in the cluster:
- ```
kubectl get pods -A | grep envoy
```
There is an envoy daemonset present in the cluster with 0 Ready:
- ```
kubectl get ds -A | grep envoy
```

Describing the envoy daemonset shows an error message similar to the following error message:

kubectl describe ds envoy -n <envoy namespace>

pods \"envoy\" is forbidden: violates PodSecurity \"restricted:latest\": allowPrivilegeEscalation != false

Environment

VMware vSphere 8.0 with Tanzu

VMware vSphere 7.0 with Tanzu

This issue can occur on vSphere Kubernetes cluster regardless of whether or not it is managed by Tanzu Mission Control (TMC)

This KB article only applies if the issue is regarding applications using contour-envoy ingress control that is installed in a vSphere Kubernetes Cluster as a package (pkgi)

Cause

Ingress control through contour with envoy will not work if envoy pods are not running in the environment.

The envoy daemonset is unable to create pods due to Pod Security Policy (PSP) or Pod Security Admission (PSA) defined in the vSphere Kubernetes cluster.

This issue can occur on first-time setup or after the envoy pods have been restarted. A restart may have been initiated by a change made to the pkgi, manual restart or outage.

Resolution

The envoy daemonset is responsible for creating the envoy pods on each worker node in the vSphere Kubernetes cluster. In this scenario, the daemonset is unable to create the envoy pods because of the PodSecurity defined in the cluster.

This KB article only applies if the issue is regarding applications using contour-envoy ingress control that is installed in a vSphere Kubernetes Cluster as a package (pkgi)

NOTE: This KB article provides a workaround to bring up the envoy pods by bypassing the PodSecurity rules. It is best practice to define the PodSecurity appropriately to allow contour, envoy and cert-manager (if applicable) pods to deploy according to your security needs.

Perform a describe on the envoy daemonset to confirm on the error message:

kubectl describe ds envoy -n <envoy namespace>

pods \"envoy\" is forbidden: violates PodSecurity \"restricted:latest\": allowPrivilegeEscalation != false

Confirm if the installed contour package (pkgi) is managed by Tanzu Mission Control (TMC):

```
kubectl get pkgi -A | grep contour
```

kubectl describe pkgi <contour pkgi> -n <contour namespace>

If the package is not managed by Tanzu Mission Control (TMC):

Add a label to the envoy namespace to bypass the PodSecurity rule:

kubectl label --overwrite ns <envoy namespace> pod-security.kubernetes.io/enforce-privileged

Confirm that the envoy daemonset no longer shows 0 Ready:
- ```
kubectl get ds envoy -n <envoy namespace>
```
Confirm that envoy pods are starting up and reaching Running state:
- ```
kubectl get pods -n <envoy namespace> -o wide
```
Confirm that the envoy service shows expected IP addresses including External:
- ```
kubectl get services -A | grep envoy
```
If cert-manager does not show ReconcileSucceeded state, check if the cert-manager daemonset is encountering the same PodSecurity error:
- cert-manager is a prerequisite for many packages, including contour.
- ```
kubectl get ds cert-manager -n <cert-manager namespace>

kubectl describe ds cert-manager -n <cert-manager namespace>
```

Add a label to the cert-manager namespace to bypass the PodSecurity rule:

kubectl label --overwrite ns <cert-manager namespace> pod-security.kubernetes.io/enforce-privilege

Confirm that the cert-manager daemonset no longer shows 0 Ready:
- ```
kubectl get ds cert-manager -n <cert-manager namespace>
```
Confirm that the contour pkgi shows ReconcileSucceeded state:
- ```
kubectl get pkgi -A
```

If contour does not show ReconcileSucceeded state, describe the contour replicaset and pkgi for more information:

kubectl get rs contour -n <contour namespace>

kubectl describe rs contour -n <contour namespace>

kubectl get pkgi -A | grep contour

kubectl describe pkgi <contour pkgi> -n <contour namespace>

As this KB only provides the above workaround to bypass the PodSecurity rules, PodSecurity rules will need to be adjusted properly to allow contour, envoy and cert-manager (if applicable) pods to create and redeploy.

If the package is managed by Tanzu Mission Control (TMC):

In the TMC web UI, check on the contour package under installed packages within the Catalog
Check that the configuration for the contour package shows pspNames: null for envoy
- ```
envoy
pspNames: null
```
Edit the configuration to change the envoy's pspNames to pspNames: "vmware-system-restricted"
- ```
envoy
pspNames: "vmware-system-restricted"
```
Confirm that the configuration shows the appropriate pspNames for contour as well, if applicable:
- ```
contour
pspNames: "vmware-system-restricted"
```
Save the configuration changes.
Confirm that the envoy daemonset no longer shows 0 Ready:
- ```
kubectl get ds envoy -n <envoy namespace>
```
Confirm that envoy pods are starting up and reaching Running state:
- ```
kubectl get pods -n <envoy namespace> -o wide
```
Confirm that the envoy service shows expected IP addresses including External:
- ```
kubectl get services -A | grep envoy
```
Confirm that the contour pkgi shows ReconcileSucceeded state:
- ```
kubectl get pkgi -A
```
If contour does not show ReconcileSucceeded state, the contour namespace may need to be labeled to bypass PodSecurity rules:
- Confirm that the contour replicaset shows the same PodSecurity "restricted:latest" error:
- ```
kubectl get rs contour -n <contour namespace>

kubectl describe rs contour -n <contour namespace>
```
- Add a label to the contour namespace to bypass the PodSecurity rule:
- ```
kubectl label --overwrite ns <contour namespace> pod-security.kubernetes.io/enforce-privilege
```
If cert-manager does not show ReconcileSucceeded state, the cert-manager namespace may need to be labeled to bypass the PodSecurity rules:
- cert-manager is a prerequisite for many packages, including contour.
- Confirm that the cert-manager daemonset shows the same PodSecurity "restricted:latest" error:
- ```
kubectl get ds cert-manager -n <cert-manager namespace>

kubectl describe ds cert-manager -n <cert-manager namespace>
```
- Add a label to the cert-manager namespace to bypass the PodSecurity rule:
- ```
kubectl label --overwrite ns <cert-manager namespace> pod-security.kubernetes.io/enforce-privilege
```

If contour does not show ReconcileSucceeded state, describe the contour replicaset and pkgi for more information:

kubectl get rs contour -n <contour namespace>

kubectl describe rs contour -n <contour namespace>

kubectl get pkgi -A | grep contour

kubectl describe pkgi <contour pkgi> -n <contour namespace>

As this KB only provides the above workarounds to bypass the PodSecurity rules, PodSecurity rules will need to be adjusted properly to allow contour, envoy and cert-manager (if applicable) pods to create and redeploy.

Additional Information

Packages managed by Tanzu Mission Control will automatically periodically revert the enforce-privilege label used to bypass the PodSecurity rules set up in the vSphere Kubernetes cluster.