Error "no route to host" observed during management cluster upgrade failure

Products

• VMware Tanzu Kubernetes Grid Management

Issue/Introduction

Management cluster upgrade fails or hangs indefinitely.

We see following error

Creating Issuer="capi-selfsigned-issuer" Namespace="capi-system"
Retrying with backoff cause="failed to create provider object cert-manager.io/v1, Kind=Issuer, capi-system/capi-selfsigned-issuer: Internal error occurred: failed calling webhook \"webhook.cert-manager.io\": failed to call webhook: Post \"https://cert-manager-webhook.cert-manager.svc:443/mutate?timeout=10s\": dial tcp 10.x.x.x:443: connect: no route to host"

Environment

TKG 2.5.x

Cause

The error indicates a network connectivity issue between the Kubernetes API server (or the CAPI controller) and the cert-manager-webhook service.
The "no route to host" message means the request to validate your new Issuer object cannot reach the cert-manager pod. 

Resolution

Restart the cert-manager deployments to refresh the certificates and network hooks:

 

• kubectl rollout restart deployment -n cert-manager cert-manager-webhook
• kubectl rollout restart deployment -n cert-manager cert-manager