VKS Core Supervisor Service Stuck in "Configuring" State Due to Missing tkgs-plugin-server Secret

Products

• VMware vSphere Kubernetes Service

Issue/Introduction

After upgrading vSphere Supervisor to 8.0 U3 or post-8.0 U3 for the first time, you may observe that VKS (vSphere Kubernetes Service), configured as a Core Supervisor Service, remains stuck in the "Configuring" state.

Environment

vSphere Supervisor 8.0.0.3 or later

Cause

One potential cause is a failure of the AppPlatform operator's user account to authenticate with vCenter (VC), which prevents the required TLS secret from being created.

Resolution

If the AppPlatform operator's credentials are invalid, follow this KB article to reset them: User wcp-appplatform-user account password unlock and reset procedure

Additional Information

This article provides steps to validate if this is the cause.

1. Check the packageInstall status Verify whether the issue originates from the deployment of the tkgs-plugin-server in the VKS service namespace:

kubectl get packageinstalls.packaging.carvel.dev -n vmware-system-supervisor-services



Note the status.usefulErrorMessage field

 

2. Inspect the plugin server pod Check the tkgs-plugin-server pod in the VKS service namespace for missing secrets:

kubectl describe pod tkgs-plugin-server-XXX-n svc-tkg-XXX

Events:
  Type     Reason       Age                    From     Message
  ----     ------       ----                   ----     -------
  Warning  FailedMount  13m (x51 over 126m)    kubelet  Unable to attach or mount volumes: unmounted volumes=[tlssecret], unattached volumes=[], failed to process volumes=[]: timed out waiting for the condition
  Warning  FailedMount  2m11s (x70 over 128m)  kubelet  MountVolume.SetUp failed for volume "tlssecret" : secret "tkgs-plugin-tls-secret" not found

3. Check AppPlatform operator logs Inspect the AppPlatform operator logs for errors related to secret creation or authentication:

kubectl logs -n vmware-system-appplatform-operator-system vmware-system-appplatform-operator-mgr-0 manager

Look for errors related to vcuiplugin creating a secret.

4. Validate the operator's vCenter credentials

1. Extract credentials from the running operator pod:

kubectl exec -it -n vmware-system-appplatform-operator-system vmware-system-appplatform-operator-mgr-0 -- /bin/bash
root [ / ]# ls /etc/vc/certs
root [ / ]# cat /etc/vc/certs/password
root [ / ]# cat /etc/vc/certs/username

2. Test login to vCenter:

Use the credentials obtained to attempt a manual login to vCenter via browser or API. If login fails, the credentials are invalid.