"is forbidden: violates PodSecurity" when creating DaemonSet to Change the vmware-system-user password expiry

Products

• Tanzu Kubernetes Runtime

Issue/Introduction

Scenario:

Your vmware-system-user account expired on vSphere Supervisor Workload Cluster nodes.

 

You show similar symptoms as mentioned in KB 319375

• Users are unable to connect via SSH directly to the workload cluster nodes using vmware-system-user account.
  
  
• TKR 1.23.8 and higher vmware-system-user password is set to expire in 60 days as part of STIG Hardening.
  
  
• While this is implemented as part of Security Hardening this impacts the ssh login to the nodes once the password has expired.
  
  

You attempt to Change the vmware-system-user password expiry on Existing Clusters using the cluster-admin DaemonSet yaml provided in KB 319375.

The DaemonSet fails to deploy.

You see the following error:

Warning  FailedCreate  3s (x5 over 41s)  daemonset-controller  (combined from similar events): Error creating: pods "cluster-admin-#####" is forbidden: violates PodSecurity "restricted:latest": allowPrivilegeEscalation != false (containers "init", "sleep" must set securityContext.allowPrivilegeEscalation=false), unrestricted capabilities (containers "init", "sleep" must set securityContext.capabilities.drop=["ALL"]), restricted volume types (volume "hostfs" uses restricted volume type "hostPath"), runAsNonRoot != true (pod or containers "init", "sleep" must set securityContext.runAsNonRoot=true), seccompProfile (pod or containers "init", "dynatrace-operator", "sleep" must set securityContext.seccompProfile.type to "RuntimeDefault" or "Localhost")

 

You note that you are utilizing Dynatrace in your environment.

You might also be using Kyverno or other PSA policies as well.

Environment

Tanzu Kubernetes Service 

Cause

• When utilizing 3rd-party or any other manual PodSecurityAdmission (PSA) Policies, some implemented policies may restrict operations such as creating the cluster-admin DaemonSet required to refresh the expired account password.
  
  
• This can be expected, depending on the Operator's choice and requirements for Kubernetes security. 
  
  
• However, such security requirements can require additional operations to be implemented in order to complete necessary workflow operations such as this one.
  
  
• By default, the DaemonSet deploys in the default namespace.

 

NOTE: 

Regarding the message "restricted:latest" reported in the above error scenario, the Operator was also using Kyverno with a policy on all of their clusters to prevent users from using the latest tag on their images.  However, we just needed to ensure having the correct labels on the new namespace to deploy the cluster-admin DaemonSet.

Resolution

Resolution summary:

For this specific issue, you would create a new K8s namespace in the cluster and ensure that the appropriate labels were added to that new namespace.

 

Resolution Procedure:

• Create a new namespace in the cluster and ensure that the following labels were attached to that namespace.

Example namespace name: temp-namespace

Include the following namespace labels:

              pod-security.kubernetes.io/audit=restricted
              pod-security.kubernetes.io/enforce=privileged
              pod-security.kubernetes.io/warn=baseline

• Add the new namespace name (instead of using the default) to the required cluster-admin DaemonSet manifest yaml mentioned in KB 319375

For example:

namespace: temp-namespace

• Then deploy the required DaemonSet mentioned in KB 319375 in that new namespace.