VCF Automation upgrade from 9.0 to 9.1 is failing after 6 hours of trying with the error ' vmsp_upgrade_vcfa fails. Reference Code: D567DF60

Products

• VCF Automation

Issue/Introduction

Attempt to upgrade the VCF Automation from 9.0.1 to 9.1 fails with Task timed out after 6h0m19s.

Install Component Service on 9.1 Cluster. Status: Failed Step vmsp_upgrade_vcfa failed. Reference Code: D567DF60. Check logs for details.

Environment

VCF Automation 9.x

Cause

Upon upgrading VCF Automation from 9.0.2.0.25145732 to 9.1.0.0.25370929, between these version a new backend service ' css-avi-eas ' has been added to VCFA applications.(1)

Prior to the upgrade steps from the following KB: https://knowledge.broadcom.com/external/article?articleNumber=433874 has been applied and that creates a custom settings for the `proxy-service` in `prelude` namespace. (2)

 

When both conditions (1) and (2) are met, the ' vmsp-prelude-deployer ' chart attempts to list a custom ConfigMap for every application in the `prelude` namespace. Since ' ccs-avi-eas ' is a new service, it lacks a pre-existent ConfigMap, leading to a failure in the ' vmsp-prelude-deployer ' chart:

entrypoint.sh:147]: echo 'Picking locally converged profiles with tag txxxxxx'
Picking locally converged profiles with tag txxxxxx
[entrypoint.sh:150]: /opt/scripts/load-external-profile-overrides.sh --overrides-tag txxxxxx--svc-list-file /dev/fd/6x
[[entrypoint.sh:150]: sort -u /etc/vmware-prelude/flux-services.list /etc/vmware-prelude/frontend-services.list /etc/vmware-prelude/infra-services.list /etc/vmware-prelude/services.list
xxxx-0x-0x xx:xx:xx,xxx [INFO] Creating profiles for services in /dev/fd/xx
xxxx-0x-0x xx:xx:xx,xxx [DEBUG] Initializing k8s config...
xxxx-0x-0x xx:xx:xx,xxx [DEBUG] Creating coreV1Api client...
xxxx-0x-0x xx:xx:xx,xxx [DEBUG] Fetching contents of config map abx-service-conf-txxxxxx in prelude namespace 
...
xxxx-0x-0x xx:xx:xx,xxx [DEBUG] Fetching contents of config map ccs-avi-eas-conf-txxxxxx in prelude namespace
xxxx-0x-0x xx:14:xx,xxx [DEBUG] response body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"configmaps \"ccs-avi-eas-conf-txxxxxx\" not found","reason":"NotFound","details":{"name":"ccs-avi-eas-conf-txxxxxx","kind":"configmaps"},"code":404}
...
    raise ApiException(http_resp=r)
kubernetes.client.exceptions.ApiException: (404)
Reason: Not Found
HTTP response headers: HTTPHeaderDict({'Audit-Id': 'xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx', 'Cache-Control': 'no-cache, private', 'Content-Type': 'application/json', 'X-Kubernetes-Pf-Flowschema-Uid': 'xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx', 'X-Kubernetes-Pf-Prioritylevel-Uid': 'xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx', 'Date': 'Tue, 09 Jun xxxx xx:xx:xx GMT', 'Content-Length': 'xxx'})
HTTP response body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"configmaps \"ccs-avi-eas-conf-txxxxxx\" not found","reason":"NotFound","details":{"name":"ccs-avi-eas-conf-txxxxxx","kind":"configmaps"},"code":404}

Resolution

The steps to resolve the issue requires support supervision . Please Contact Support - Broadcom Software