VKS Cluster Management Service fails with 405 Proxy Error

Products

• VMware vSphere Kubernetes Service

Issue/Introduction

After deploying VCF 9.0.1 or applying the workaround from KB VKS cluster management service unhealthy after installing VCF 9.0.1 with error "tls: failed to verify certificate: x509: certificate, the VKS Cluster Management service on the Supervisor remains unhealthy.

The package svc-auto-attach.vksm.broadcom.com fails to reconcile. To confirm this specific issue, check the logs of the kubectl-plugin-vsphere pod on the Supervisor:

Find the pod: 

kubectl -n kube-system get pods | grep kubectl-plugin-vsphere-

View the logs:

kubectl -n kube-system logs <kubectl-plugin-vsphere-POD-ID>

 

Cause

VCFA Service Manager manages proxy configurations based on region lifecycle:

• Region creation: Proxy configuration is created
• Region removal: Proxy configuration is removed

The Container Registry in the Supervisor was inadvertently removed due to one of the following scenarios:

1. The fix was applied **before** the region was created
2. The region was removed and recreated

When following the KB article, the ConfigMap is updated and its owner reference is changed. This prevents the Service Manager from detecting that it needs to update the configuration again, which means the Container Registry is never recreated automatically.

Resolution

To resolve this issue, you must manually recreate the Service Manager Container Registry in vCenter using credentials retrieved from the VCFA.

1: Retrieve the Registry Secret from VCFA k8sc Cluster:

kubectl -n prelude get secret | grep vcfa-service-manager-registry

Note: There is one secret per Supervisor. Identify the correct secret by matching the Supervisor name in the suffix.

2: Extract and Decode Credentials Get the secret details (replace <secret-name> with the name found in Step 1):

kubectl -n prelude get secret <secret-name> -o yaml

Locate the serviceAccount and refreshToken values in the output. These are base64 encoded. You must decode them to get the actual credentials.

Example:

echo "<encoded_serviceAccount_string>" | base64 -d
echo "<encoded_refreshToken_string>" | base64 -d

Copy these decoded values for the next step.

3: Create the Container Registry in vSphere

• Log in to the vSphere Client.
• Navigate to Inventory > Supervisor > Configure > Container Registry.
• Click New and enter the following details exactly:

Name: mgmt-image-proxy.kube-system.svc.cluster.local
URL: https://mgmt-image-proxy.kube-system.svc.cluster.local
Username: (Paste the decoded serviceAccount value from Step 2)
Password: (Paste the decoded refreshToken value from Step 2)
TLS Certificate: None (Leave blank or unchecked)

• Click Save.

The registry connection should now be established, and the 405 error in the Supervisor logs will resolve.