Tanzu Kubernetes Grid Integrated Edition: TKGI: After applying ValidatinWebhookConfiguration every request goes timeout.

Products

• VMware Tanzu Kubernetes Grid Integrated Edition

Issue/Introduction

Problem description: Trying to use a ValidatinWebhookConfiguration. The configuration and all looks fine, but when they apply the config every request goes into a timeout and doesn't reach the destination. 

Tested it from the master directly with a normal curl request or in a pod and from there it works fine. From kube-api logs which says why the webhook goes into a timeout is not clear the reason

Here the test's which were done to test if the config webhook endpoint work:

Deployed Regular Pod (curlimages/curl:latest)
nc -vz webhook.test.com 443
Succeeded
curl -X POST https://webhook.test.com/k8s/admission/1234/4321
Succeeded

Deployed Privileged Pod in Host Network
nc -vz webhook.test.com 443
Succeeded
curl -X POST https://webhook.test.com/k8s/admission/1234/4321
Succeeded

Master Node (SSH Access)
nc -vz webhook.test.com 443
Succeeded
curl -X POST https://webhook.test.com/k8s/admission/1234/4321
Succeeded

ValidatingWebhookConfiguration
POST Webhook https://webhook.test.com/k8s/admission/1234/4321
Timeout/TLS Handshake Error

Environment

TKGI

Resolution

When ValidatinWebhookConfiguration is in place the service  kube-apiserver on the master before sending the command for execution sends it to the configured validation webhook endpoint. 

All tests showed successful  connectivity  to the validation endpoint. 

To confirm the endpoint connection from the master node run 
netstat -putan | grep <validation webhook endpoint IP>
tcp        0      1 x.x.x.x:48764      <validation webhook endpoint IP>:443       SYN_SENT    10644/kube-apiserve

If you do not see the <validation webhook endpoint IP> in the list and considering the fact that connectivity  is not the problem as we can connect via curl from a pod and the master, it seems the problem is in the configuration of the kube-apiserver 

under the /var/vcap/jobs/kube-apiserver/config/bpm.yml
The following can be found: 
 pre_start: /var/vcap/jobs/kube-apiserver/bin/generate-config-files
  env:

    NO_PROXY: .internal,.svc,.svc.cluster.local,.svc.cluster,api.cluster.local,x.x.x.x/24,x.x.x.x/16,127.0.0.1,localhost,*.example.com
    no_proxy: .internal,.svc,.svc.cluster.local,.svc.cluster,api..cluster.local,x.x.x.x/24x.x.x.x/16,127.0.0.1,localhost,*.example.com


    HTTPS_PROXY: http://admin:[email protected]
    https_proxy: http://admin:[email protected]


    HTTP_PROXY: http://admin:[email protected]
    http_proxy: http://admin:[email protected]

As the target validation webhook endpoint <domain/IP> is not in the list of no_proxy, they by default are sent to the proxy server which fails as these are direct messages. 

To allow the request to reach the routable network the FQDN webhook.test.com from this example have to be added to the no proxy list.