VKS Control Plane Node Fails to Start kube-apiserver after ExtraAuthentication Configuration Update
Article ID: 449344
Updated On:
Products
Issue/Introduction
When a VKS cluster is configured with ExtraAuthentication (OIDC) and the authentication configuration is later updated (e.g., changing the issuer URL), a rolling update is triggered on the control plane. During this update, new control plane nodes may fail to start the kube-apiserver and remain stuck in a NotReady state.
The KubeadmControlPlane will fail to complete the rolling update. In the kube-apiserver logs on the affected control plane node, the following error is observed:
failed to load authentication configuration from file "/etc/kubernetes/api-server/authentication-config-<hash>.yaml": open /etc/kubernetes/api-server/authentication-config-<hash>.yaml: no such file or directory
The <hash> value in the error will differ from the hash of the authentication configuration file present on disk at /etc/kubernetes/api-server/, confirming a path mismatch between the kube-apiserver manifest and the written file.
Environment
VMware vSphere Kubernetes Service 3.7.0
Cause
The --authentication-config flag passed to the kube-apiserver and the authentication configuration file written to the node were derived from separate hash computations. As YAML serialization is not deterministic across two separate invocations, the two hashes can differ. This causes the flag to reference a file path that was never written to disk, preventing the kube-apiserver from starting.
Resolution
A fix is included in an upcoming release of VKS.
Note: The configuration change is not supported on VKS 3.7.0 GA and must not be updated until an upgrade to the fixed release is completed.
If the configuration change has already been applied, the following workaround can be utilized for recovery:
Option 1 — Revert the ExtraAuthentication change
Revert the ExtraAuthentication mutation on the VKS cluster object to its previous value (or remove it entirely if it was newly added).
Once reverted, the VKS controller will detect the change and abandon the failed rolling upgrade. The stuck new control plane node will be deleted automatically, and the cluster will recover to the previous working control plane node with the original authentication configuration.
Monitor recovery using the following command:
kubectl get machines -n <namespace> -w
Note: The cluster is considered recovered when only the original machines remain and are in the Running phase. After recovery, the cluster will operate with the previous ExtraAuthentication configuration. The updated configuration must not be re-applied until an upgrade to the fixed release is complete.
Option 2 — Unrecoverable cluster operations
If the cluster does not recover after reverting the change, or if the cluster object cannot be modified, one of the following actions must be taken:
Upgrade the cluster to the next patch version which contains the fix, then re-apply the ExtraAuthentication configuration.
Create a new cluster on the fixed release.
Additional Information
The software defect stems from non-deterministic YAML serialization producing different hashes across two separate calls within the same webhook invocation. In patched versions, the hash is computed once and reused for both the --authentication-config flag and the written file path, guaranteeing identical values.
Feedback
