Cert-Manager Package Reconciliation Failed, labelling cert-manager package is overwritten

Products

Tanzu Kubernetes Runtime vSphere with Tanzu VMware vSphere Kubernetes Service

Issue/Introduction

Cert-manager package in reconciled failed state in v1.25 or later.

The deployment might fail to create any pods, and the replica set is reported as having 0/1 available replicas. Additionally, the kubectl describe command for the cert-manager deployment and replica set throws forbidden errors related to PodSecurity policies.

Events:
  Type     Reason        Age                From                   Message
  ----     ------        ----               ----                   -------
  Warning  FailedCreate  22m                replicaset-controller  Error creating: pods "cert-manager-xxxxx-xxxx" is forbidden: violates PodSecurity "restricted:latest": allowPrivilegeEscalation != false (container "cert-manager" must set securityContext.allowPrivilegeEscalation=false), unrestricted capabilities (container "cert-manager" must set securityContext.capabilities.drop=["ALL"]), seccompProfile (pod or container "cert-manager" must set securityContext.seccompProfile.type to "RuntimeDefault" or "Localhost")
  Warning  FailedCreate  22m                replicaset-controller  Error creating: pods "cert-manager--xxxxx-xxxx" is forbidden: violates PodSecurity "restricted:latest": allowPrivilegeEscalation != false (container "cert-manager" must set securityContext.allowPrivilegeEscalation=false), unrestricted capabilities (container "cert-manager" must set securityContext.capabilities.drop=["ALL"]), seccompProfile (pod or container "cert-manager" must set securityContext.seccompProfile.type to "RuntimeDefault" or "Localhost")

These errors are caused by a violation of the PodSecurity "restricted" policy, which is enforced in TKR 1.25 and above.

Environment

vSphere with Tanzu 8.0

Cause

The PodSecurity policy defined in Kubernetes 1.25 and later restricts containers from having privilege escalation. These security settings are not configured in the cert-manager deployment, leading to the forbidden errors during pod creation.

Specifically, the following constraints are violated:

allowPrivilegeEscalation != false
securityContext.capabilities.drop=["ALL"]
seccompProfile.type not set to RuntimeDefault or Localhost.

Since PodSecurity Admission (PSA) is enabled in TKR 1.25 and above, these violations cause the pod creation to fail. As a result, the cert-manager package cannot reconcile, and the deployment remains in a failed state.

Resolution

To resolve this issue, an overlay file can be created and applied to ensure that the required security settings are configured for the cert-manager deployment. This file overrides the namespace label to ensure compliance with the PodSecurity policy. Additionally, the cert-manager package can be updated with the overlay file to apply the security settings.

Steps:

1. Create an Overlay File
Create a file named patched-ns.yaml with the following content to override the PodSecurity settings:

#@ load("@ytt:overlay", "overlay")
#@overlay/match by=overlay.subset({"kind": "Namespace"})
---
#@overlay/match-child-defaults missing_ok=True
metadata:
  labels:
    pod-security.kubernetes.io/enforce: privileged

2. Update the Cert-Manager Package

tanzu package installed update cert-manager -n tkg-package  -p cert-manager.tanzu.vmware.com -v 1.7.2+vmware.3-tkg.3 --ytt-overlay-file patched-ns.yaml

3. Validate the Deployment

kubectl get deployments -n <namespace>
kubectl get pods -n <namespace>