Deploying a pod fails with: Error from server (InternalError): error when creating "test-pod.yaml". Internal error occurred: failed calling webhook

Products

• VMware Container Networking with Antrea

Issue/Introduction

• A guest cluster is deployed within VKS with Antrea CNI in use.
• A web hook is used to validate pod deployments.
• The following policy is applied.
  

  
  apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: deny-all
spec:
  podSelector: {}
  policyTypes:
  - Egress
  - Ingress
---
apiVersion: crd.antrea.io/v1beta1
kind: NetworkPolicy
metadata:
  name: allow-ingress-kube-api
spec:
  appliedTo:
  - podSelector: {}
  ingress:
  - action: Allow
    from:
    - namespaceSelector:
        matchLabels:
          kubernetes.io/metadata.name: kube-system
      podSelector:
        matchLabels:
          component: kube-apiserver
          tier: control-plane
    name: allow-ingress-kube-api
  priority: 5
  tier: platform

  
  
  
  
• Pod deployment fails with the following error:
  
  kubectl -n test create -f test-pod.yaml 
Error from server (InternalError): error when creating "test.yaml": Internal error occurred: failed calling webhook "validate-resources.example.com": failed to call webhook: Post "https://webhook-service.webhook-system.svc:443/validate?timeout=2s": context deadline exceeded
  
  
• Changing to the following policy allows connectivity to the web hook. However, this policy can interrupt VKS upgrade.
  

  
  apiVersion: crd.antrea.io/v1beta1
kind: NetworkPolicy
metadata:
  name: allow-control-plane-nodes
spec:
  appliedTo:
  - podSelector: {}
  ingress:
  - action: Allow
    from:
    - nodeSelector:
        matchExpressions:
        - key: node-role.kubernetes.io/control-plane
          operator: Exists
    name: allow-control-plane-nodes
  priority: 5
  tier: platform

   

 

Environment

VMware Container Networking with Antrea
VMware Kubernetes Service

Cause

Antrea allows traffic from nodes labeled node-role.kubernetes.io/control-plane. However, a new node does not receive this label until the end of its boot process. Because it is unlabeled during boot, the Antrea Cluster Network Policy (ANCP) blocks API requests to the webhook.

Resolution

For Antrea, there is no way to manipulate the labels to allow this traffic.

Similarly, in VKS, there is no way around the delay in node labelling which was introduced to resolve Bounding Self-Labeling Kubelets issue.

To workaround this issue, for web hooks that support the change, you can reduce the timeout to 2 or 3 seconds. Example:

webhooks:
- name: resources-webhook-service
  timeoutSeconds: 2
  failurePolicy: Ignore

Additional Information

This particular behaviour can also cause issue during VKS upgrades. The following VKS KB outlines issues and VKS general support for web hooks.