VCF 9.0.2 Automation deployment fails at Step 6 with LCMVSPHERECONFIG1000095 "error: failed to apply the PackageDeployment: context deadline exceeded"

Products

• VCF Automation

Issue/Introduction

When deploying a VMware Cloud Foundation (VCF) 9.0.2 Automation cluster, the deployment fails at Step 6. The Fleet Manager bootstrap logs will report a timeout:

error: failed to apply the PackageDeployment: context deadline exceeded

Checking the pod status on the new Automation node with command "kubectl get pods -n vmsp-platform" reveals that the cluster-api-installer pod has failed with state: ImagePullBackOff.

Environment

VMware Cloud Foundation (VCF) Automation 9.0.2.

Cause

This issue is caused by a network communication failure. Ports 30000-30006 are restricted between the Automation cluster and the Fleet Management machine, preventing the installer from pulling the required deployment packages.

This can be verified by running the below command from the Automation node:

• curl -v telnet://<FLEET-MANAGEMENT-FQDN-or-IP>:30000

This will likely be stuck at "Trying..." indicating that access is not open between Automation and Fleet Management on port 30000.

Running the exact same command on the Fleet Management machine directly should return "Connected to <FLEET-MANAGEMENT-FQDN-or-IP> port 30000" showing that port 30000 is listening on that machine.

Resolution

To resolve the deployment failure, update your network configuration to permit traffic on the following ports:

1. Identify the IP addresses for your VCF Automation (VCFA) nodes and the Fleet Management machine.
2. Open ports 30000 through 30006 (TCP) for traffic originating from the VCFA cluster to the Fleet Management machine.
3. Retry the deployment Step 6 in the VCF interface.

Additional Information

VMware Ports and Protocols
https://ports.broadcom.com/home/VMware-Cloud-Foundation-Automation

(Select Product: "VMware Cloud Foundation Automation", NOT "VMware Aria Automation")