Error: "Failed to upload the template to vCenter" when upgrading VCF services runtime

Products

• VCF Operations

Issue/Introduction

• In VCF Operations, the upgrade fails at the subtask "Stage VCF services runtime package in VCF services runtime" with the following errors:
  
Internal error occurred. Failed to upload the template to vCenter. Please retry the operation or contact support if the issue persists. [VCFMS-PUSH-TEMPLATE-001]

Step vmsp_stage_package failed. Reference Code: ########. Check logs for details.


• In vCenter, the task "Deploy OVF template" fails for the vcf-services-runtime-template with the following error:
  
  Error caused by file vmsp.vmdk

Environment

• VCF Operations 9.1.x
• ESX 9.x
• vCenter 9.x

Cause

The VCF services runtime Control Plane node cannot connect to an ESX host to deploy the template. A firewall rule is likely preventing connectivity from the Control Plane node to the host over port 443.

To verify connectivity and identify the failure point:

1. Navigate to Build > Lifecycle > VCF Management > Components > select VCF services runtime in VCF Operations.

2. Scroll down to Nodes and note the IP address of the Control Plane node.

3. Log into the Control Plane node via SSH using the vmware-system-user account.

4. Run the following command to ensure the Control Plane node can connect to all hosts in the cluster where VCF services runtime nodes are running: curl -k -v --connect-timeout 5 https://<ESX_FQDN>

If the connection times out, the Control Plane node does not have connectivity to the ESX host on port 443.

Resolution

Configure firewall rules to allow HTTPS (port 443) traffic from the Control Plane node to all ESX hosts in the cluster where VCF services runtime nodes are running.