VCF services stuck in a busy state upon configuring any services from VCF Automation Services Management

Products

• VCF Automation

Issue/Introduction

Upon configuring  any services under VCF services it gets stuck in installing loop. Reviewing the services in detail, you may find that the supervisor service is in Busy state and never completes or timed out.

Environment

VCF Automation 9.1

vSphere Kubernetes Service 9.1

Cause

This issue can occur if there is a time zone mismatch between vCenter & VCF Automation appliance. The finish time comes back from the healthcheck and vcfa-services-manager assumes that it's an old request hence re-triggers a fresh healthcheck every 30 seconds.

There are no related error reported in the VCF automation logs however supervisor logs in the vCenter shows below error snippet and indicates that it could be a time zone mismatch issue.

WCP Logs:
-----------
YYYY-MM-DD HH:MM:SEC,| DEBUG | proxy-client-508518 | c.v.v.c.p.c.A.proxyingFailures | Error handler output is {"status":500,"entity":{"apiVersion":"v1","code":500,"kind":"Status","message":"VCFA: Internal Error in Namespace Proxy sub-system, contact your system administrator.","metadata":{},"reason":"InternalError","status":"Failure"},"metadata":{"Content-Type":[{"type":"application","subtype":"json","parameters":{},"wildcardType":false,"wildcardSubtype":false}]},"stringHeaders":

{"Content-Type":["application/json"]}
,"statusInfo":

{"family":"SERVER_ERROR","reasonPhrase":"Internal Server Error","statusCode":500}
,"actualEntity":{"apiVersion":"v1","code":500,"kind":"Status","message":"VCFA: Internal Error in Namespace Proxy sub-system, contact your system administrator.","metadata":{},"reason":"InternalError","status":"Failure"},"links":[],"length":-1,"allowedMethods":[],"cookies":{},"mediaType":{"type":"application","subtype":"json","parameters":

Resolution

Verify that the time zone set for vCenter and VCF Automation is UTC, if not set the time zone to UTC in both the appliance.

Restart the WCP service and it should fix the issue.