Unable to Patch VCF Automation to 9.0.2 due to "Synthetic checker" error on the VCF Operations Fleet Manager

Products

• VCF Automation

Issue/Introduction

• When attempting to upgrade VCFA to 9.0.2, from VCF Ops > Fleet Manager > Lifecycle > Overview > Automation > Upgrade > 9.0.2 : Fails with error "Error Code: LCMVMSP10035 Synthetic checker on the application platform failed." 
  
  
• Detailed error message examples:
  
  • Synthetic check failed. Please refer to Broadcom Knowledge Base Article https://knowledge.broadcom.com/external/article/389510 for remediation details.,"vcfa-services-prelude-package : 1 of 1 resources are not ok: vcfa-bundle: wrong resource state: InProgress - package deployment is in progress;","vksm-services-prelude-deployments : 10 of 19 resources are not ok: resource-manager-server: wrong resource state: Failed - Progress deadline exceeded;"

  • Synthetic check failed. Please refer to Broadcom Knowledge Base Article https://knowledge.broadcom.com/external/article/389510 for remediation details.,"vcfa-services-prelude-health-reporter : Unexpected status code: 500 expected: 200, response body: ""","vksm-services-prelude-deployments : 3 of 20 resources are not ok: resource-manager-server: wrong resource state: InProgress - Available: 0/3;"


• The logfile vmware_vrlcm.log shows errors such as the following - I/O was unable to get through on cluster.local addresses:
  
  • Could not load chart: GET http://source-controller.vmsp-platform.svc.cluster.local./helmchart/vmsp-platform/vmsp-platform-ndc/ndc-1.1.9.tgz giving up after 10 attempt(s): Get "http://source-controller.vmsp-platform.svc.cluster.local./helmchart/vmsp-platform/vmsp-platform-ndc/ndc-1.1.9 .tgz": dial tcp: lookup source-controller.vmsp-platform.svc.cluster.local. on ###.#.#.#:53: read udp ###.##.##.##:###->###.#.#.#:53: i/o timeout

Environment

VCF Automaton 9.x

VCF Operations Fleet Manager 9.x

Cause

• A previous partial update failure may result in the default cluster DNS being misconfigured on the VCFA cluster.
• As a result, the Strimzi-cluster-operator picks up a malformed DNS entry when it queries the cluster to identify where the services need to be initialized, thus causing a roadblock during post-upgrade initialization.

Resolution

1. SSH to the VCFA node using vmware-system-user credentials. 
2. Switch over to root using : sudo su - 
3. Load the kubeconfig:
   export KUBECONFIG=/etc/kubernetes/admin.conf
4. Set the override environment variable KUBERNETES_SERVICE_DNS_DOMAIN=cluster.local on the strimzi-cluster-operator deployment.
   The command to be run:
     kubectl -n prelude set env deployment/strimzi-cluster-operator KUBERNETES_SERVICE_DNS_DOMAIN=cluster.local
5. The Strimzi pods will pick up the environment variable after a restart. This can be triggered with:
     kubectl -n prelude rollout restart deployment/strimzi-cluster-operator
6. Reattempt to patch VCF Automation from VCF Ops.