Using VM service to deploy VMs on a VKS namespace fails with default.mutating.virtualmachine.config.vcf.vmware.com -> connect: connection refused

Products

• VMware vSphere Kubernetes Service
• VCF Automation

Issue/Introduction

Attempting to use VM service on VCFA or Local Consumption Interface on a VKS cluster failing with error

 Internal error occurred: failed calling webhook "default.mutating.virtualmachine.config.vcf.vmware.com": failed to call webhook: Post "https://configuration-service-webhook-service.svc-configuration-###.svc:443/mutate-virtual-machine?timeout=30s": dial tcp ###.###.###:443: connect: connection refused 
default.mutating.virtualmachine.config.vcf.vmware.com -> connect: connection refused

 

Checking the service state on vCenter shows the configuration-service in a reconciling state. From the supervisor, describe of the package shows 

 usefulErrorMessage: 'kapp: Error: Timed out waiting after 15m0s for resources: [deployment/configuration-service-controller-manager
    (apps/v1) namespace: svc-configuration-###]'

 

Checking pod in the svc-configuration namespace shows pod in CrashLoopBackOff

svc-configuration-#### configuration-service-controller-manager-###         0/1     CrashLoopBackOff   269 (2m6s ago)   29h

Checking the failed pod logs
 

kubectl logs -n svc-configuration-### configuration-service-controller-manager-######-###

"logger":"setup","msg":"Configuration failure annotation key loaded","key":"condition.vmoperator.vmware.com.protected"}
{"level":"error","ts":,"logger":"setup","msg":"unable to create index for ConfigState spec.stage","error":"failed to get server groups: Get \"https://###.###.##.#.1:443/api\": dial tcp /###.###.##.#:443: i/o timeout","stacktrace":"main.main\n\t./main.go:257\nruntime.main\n\truntime/proc.go:285"}

 

Environment

VCF 9.1

Cause

Pod in failed state, possible connection issue. Check NSX Manager to verify if the NSX fabric is healthy

Resolution

Restart of svc-configuration pod to in attempt to resolve the issue. 

kubectl delete pod -n svc-configuration-### configuration-service-controller-manager-####-####

Steps to connect to Supervisor context to restart can be found here.

 

Additional Information

Should the issue persist,  please open a support case with the VMware vSphere Kubernetes Service (VKS) team to check the status of pods in the supervisor cluster.

Provide output of pod logs and describe of the failed pod, from the supervisor context. 

kubectl logs -n svc-configuration-### configuration-service-controller-manager-######-###

kubectl describe pod -n svc-configuration-### configuration-service-controller-manager-######-###