TKGI cluster upgrade failed as the canary master instance's kube-apiserver failed to start up
Article ID: 449448
Updated On:
Products
Issue/Introduction
The upgrade of a TKGI cluster failed on the updating of the canary master instance. The bosh task logs show:
Task 999 | 04:51:48 | L starting jobs: master/9999-999-999-9999 (0) (canary) (00:15:36)
L Error: 'master/9999-999-999-9999 (0)' is not running after update. Review logs for failed jobs: kube-apiserver, kube-scheduler
Task 999 | 05:06:45 | Error: 'master/9999-999-999-9999 (0)' is not running after update. Review logs for failed jobs: kube-apiserver, kube-schedulerThe kube-apiserver logs show:
E0717 20:15:45.464024 6 controller.go:195] "Failed to update lease" err="leases.coordination.k8s.io \"apiserver-xxxx\" is forbidden: ValidatingAdmissionPolicy 'gatekeeper-policy' denied request: failed to configure policy: failed to find resource referenced by paramKind: 'constraints.gatekeeper.sh/v1beta1, Kind=K8sPSPSeccomp'"
The above indicates that the cluster is using native ValidatingAdmissionPolicies feature, and that there is a policy active named "gatekeeper-policy". This policy relies on an external parameter type (paramKind) pointing to a Gatekeeper CRD "constraints.gatekeeper.sh/v1beta1, Kind=K8sPSPSeccomp". Because the API server is in a cold-boot sequence and cannot cleanly map its types yet the policy engine throws a compilation error: "failed to find resource referenced by paramKind".
There could be a lot of other errors found, particularly those with the error message "No agent available":
E0717 20:15:45.149304 6 dispatcher.go:214] "Unhandled Error" err="failed calling webhook \"mutation.gatekeeper.sh\": failed to call webhook: Post \"https://gatekeeper-webhook-service.gatekeeper-system.svc:443/v1/mutate?timeout=1s\": No agent available" logger="UnhandledError"
E0717 20:15:45.532328 6 remote_available_controller.go:448] "Unhandled Error" err="v1alpha1.stats.antrea.io failed with: failing or missing response from https://10.x.x.x:10349/apis/stats.antrea.io/v1alpha1: Get \"https://10.x.x.x:10349/apis/stats.antrea.io/v1alpha1\": No agent available" logger="UnhandledError"
E0717 20:15:45.532500 6 remote_available_controller.go:448] "Unhandled Error" err="v1beta1.metrics.k8s.io failed with: failing or missing response from https://10.x.x.x:4443/apis/metrics.k8s.io/v1beta1: Get \"https://10.x.x.x:4443/apis/metrics.k8s.io/v1beta1\": No agent available"
I0717 20:15:49.365082 6 healthz.go:280] poststarthook/start-service-ip-repair-controllers,poststarthook/rbac/bootstrap-roles check failed: readyz
[-]poststarthook/start-service-ip-repair-controllers failed: not finished
[-]poststarthook/rbac/bootstrap-roles failed: not finishedEnvironment
VMware Tanzu Kubernetes Grid Integrated
OPA Gatekeeper
Cause
When a master node undergoes a "cold boot" (like during a BOSH deployment or cluster upgrade), its internal discovery cache is empty. The kube-apiserver attempts to write its background system Lease to announce it is alive. However, the VAP (ValidatingAdmissionPolicy) engine intercepts this request. Because the cache is empty, the API server cannot resolve the Gatekeeper paramKind (K8sPSPSeccomp), fails to evaluate the policy, and defaults to "Fail Closed." This completely deadlocks the API server. The core issue is that the Gatekeeper constraint is scoped too broadly, causing the generated VAP to evaluate all resources, presumably including critical control-plane Lease updates in the kube-system namespace.
See Gatekeeper issue #4530 for more details.
Resolution
The Gatekeeper constraint, if confirmed to be scoped too broadly, should be re-scoped and adjusted accordingly by the operator who manages the Gatekeeper policies.
As a temporary workaround, the corresponding Gatekeeper ConstraintTemplate can be deleted to get past the deadlock issue so that the kube-apiserver can complete its initialization.
Feedback
