VKS Pods not Creating - creating pods is forbidden due to PodSecurity - pod or container k8tz must set SecurityContext.runAsNonRoot=true

Products

VMware vSphere Kubernetes Service

Issue/Introduction

In a vSphere Kubernetes Service (VKS) cluster, one or more pods are not creating.

If it is a VKS system pod that is failing to create, it will be flagged as an Addons Reconciliation failure on the VKS cluster.

While connected to the VKS cluster's context, the following symptoms are observed:

VKS system pods will have the corresponding installed package (PKGI) in Reconcile Failed state:
```
kubectl get pkgi -A
```
The deployment created successfully but shows 0 Ready and 0 Available.
- NOTE: This KB will assume that a deployment is the owning object for the affected pod(s). Other owning objects can be daemonsets, statefulsets or jobs.
If applicable, the corresponding replicaset is created successfully but shows 0 Current and 0 Ready.
Describing the deployment shows that 0 replicasets were created successfully:
```
kubectl describe deployment -n <deployment namespace> <deployment>
```
Performing a describe on the replicaset returns an error message similar to the following:
```
kubectl describe replicaset -n <replicaset namespace> <replicaset>

Warning FailedCreate - Error creating: pods "<pod>" is forbidden: violates PodSecurity: <podSecurity setting>
```
Where <podSecurity setting> is one of the podSecurities defined in Kubernetes such as "restricted:latest".

k8tz pods and webhookconfiguration are present within the VKS cluster:

kubectl get pods,validatingwebhookconfiguration,mutatingwebhookconfiguration -A | grep -i k8tz

k8tz  pod/k8tz-<id>

k8tz  mutatingwebhookconfiguration/admission-controller.k8tz.io

Environment

vSphere Supervisor

vSphere Kubernetes Service (VKS)

Cause

k8tz is a third party application that can manage podSecurity within a Kubernetes cluster.

In this scenario, pod creation is being blocked by k8tz.

This can result in customer application pods and VKS system pods failing to create.

Failed VKS system pods will prevent upgrades to the vSphere Kubernetes Release (VKR) of a VKS cluster.

Resolution

Because k8tz is not a VMware by Broadcom product, we cannot advise on its proper configuration.

Our only workaround would be to temporarily backup and take down k8tz.

Connect into the VKS cluster context

Take a backup of the k8tz mutatingwebhookconfiguration:

kubectl get mutatingwebhookconfiguration | grep k8tz

kubectl get mutatingwebhookconfiguration <k8tz mutatingwebhookconfiguration> -o yaml > k8tz-mwc.yaml

Ensure that the backup is saved in an appropriate location.

Delete the k8tz mutatingwebhookconfiguration:

kubectl delete mutatingwebhookconfiguration <k8tz mutatingwebhookconfiguration>

Scale down the k8tz deployment to 0:

kubectl scale deployment -n k8tz k8tz --replicas=0

Confirm that the k8tz pods and mutatingwebhookconfiguration are now gone:
```
kubectl get pods,mutatingwebhookconfiguration -A | grep -i k8tz
```
Locate the deployment for the pod(s) failing to create:
NOTE: This KB will assume that a deployment is the owning object for the affected pod(s). Other owning objects can be daemonsets, statefulsets or jobs.
```
kubectl get deployment -n <pod namespace>
```
Restart the deployment to trigger the creation of a new replicaset for the failing pod:
```
kubectl rollout restart deployment -n <pod namespace> <deployment>
```
Confirm that the deployment and replicaset show Available and Ready respectively, and that the pod(s) create successfully:
```
kubectl get deployment,replicaset,pod -n <pod namespace>
```
Check if there is an application object related to the failing pod(s):
```
kubectl get app -A | grep -i <pod>
```
The application object can be triggered to perform immediate reconciliation with the below command where syncPeriod is set to a different value than its current value:
```
kubectl patch app <app name> -n <app namespace> --type='merge' -p '{"spec":{"syncPeriod":"9m"}}'
```
If applicable, confirm that the corresponding PKGI object now shows ReconcileSucceeded:
```
kubectl get pkgi -A
```

Additional Information

Future Considerations

The k8tz application should be configured to not block VKS system pod creation.
- NOTE: VMware by Broadcom is not responsible for and does not provide support for third party applications.
  Any issues with security pods and webhooks installed by a third party application should be escalated to the third party application owner.
You can restore the backed up k8tz mutatingwebhookconfiguration with the following command on the back-up file that was created in Step 2 of the Resolution:
```
kubectl apply -f k8tz-mwc.yaml
```