Error "Unexpected server error has occurred" during upgrade from 5.2.2.0 to 9.1.0.0100 VCF Management Services Platform

Products

• VCF Operations

Issue/Introduction

• VCF Management Service Platform (VMSP) deployment fails during the task "Import VCF Operations in Fleet Lifecycle" with this error:

The Fleet lifecycle 'Component import' task with ID #### failed. Internal errors: Unexpected server error has occurred.

• The task is stuck at 46% for some time before failing.
• The SDDC Manager log /var/log/vmware/vcf/domainmanager/domainmanager.log contains these errors:

YYYY-MM-DDThh:mm:ss.sss+0000 INFO [vcf_dm,####,####] [c.v.v.v.s.fleetlcm.FleetLcmService,dm-exec-xxx] Fleet LCM Task ID: ####-####-####-####-####, Status: FAILED
YYYY-MM-DDThh:mm:ss.sss+0000 ERROR [vcf_dm,####,####] [c.v.v.v.s.fleetlcm.FleetLcmService,dm-exec-xxx] Fleet LCM task ####-####-####-####-#### failed with status: FAILED
YYYY-MM-DDThh:mm:ss.sss+0000 ERROR [vcf_dm,####,####] [c.v.v.v.s.f.FleetLcmTaskExecutor,dm-exec-xxx] Fleet LCM task for Component import failed, task ID: ####-####-####-####-####, status: FAILED
YYYY-MM-DDThh:mm:ss.sss+0000 ERROR [vcf_dm,xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx,xxxx] [c.v.e.s.o.model.error.ErrorFactory,dm-exec-xxx] [IULIM3] PUBLIC_FLEET_LCM_TASK_FAILED The Fleet lifecycle {0} task with ID ####-####-####-####-#### failed. Internal errors: Unexpected server error has occurred., Unexpected server error has occurred., Unexpected server error has occurred

• The following can be seen in the logs : /VCFMgmtComponents/vcf-management-services/services-logs/vcf-sddc-lcm/####-####/vcf-sddc-build-service-sddcbuild/sddcbuild/console-log-####.log:

YYYY-MM-DDThh:mm:ss.sssssssssZ stdout F YYYY-MM-DDThh:mm:ss.sssZ ERROR sddc-build-service 52 [lcm@xxxx scPath="ops" thread="system-task-worker-x" logger="c.b.v.l.s.b.p.common.util.SslUtil" method="getCertificateChain:120"] [####-####-####-####-####, taskId=####-####-####-####-####] - Error occurred while getting certificate chain for 'vcf_operations_node_fqdn:443'.
YYYY-MM-DDThh:mm:ss.sssssssssZ stdout F java.net.ConnectException: Connection timed out

Environment

VCF 9.1.0.0100

Cause

TCP connectivity issue between the VMSP (SDDC Build Service, SBS) pods and the VCF Operations instance.

The VMSP (specifically the SBS pods) requires HTTPS connectivity to the VCF Operations node to fetch certificate chains and validate node readiness.

Resolution

1. Verify TCP connectivity from the SBS pod to the VCF Operations instance on port 443 using curl -v

   1. • Open an SSH session to the VMSP control-plane node.

      • To identify the control-plane node IP address: open the VCF Operations UI > Build > Components > click on the VCF Runtime component > scroll down to the Nodes table and copy the IP of a node with the type control plane.

   • • • # SSH to the control-plane VM

ssh vmware-system-user@<control-plane-address>

• • • • # Verify the kubeconfig exists and is not empty

cat /etc/kubernetes/admin.conf

• • • • # Export the kubeconfig environment variable

export KUBECONFIG=/etc/kubernetes/admin.conf

• • • Identify the SBS pod name and verify the connectivity
      • # Get the exact name for sddc build service with:

kubectl get pods -n vcf-sddc-lcm

• • • • # Verify the connectivity to VCF Operations

kubectl exec -n vcf-sddc-lcm vcf-sddc-build-service-sddcbuild-####-#### -- curl -v -k --connect-timeout 5 https://vcf_operations_node_fqdn 2>&1 | head -20

1. • • • A connection timeout confirms there is a connectivity issue
2. Update firewall rules to allow communication between the VCF Management Services pods and the VCF Operations instance.
3. Refer to VMware required Ports and Protocols for specific sub-network requirements.
4. Verify TCP connectivity from the SBS pod to the VCF Operations instance on port 443 using curl -v again, per step 1
5. Retry the deployment workflow in SDDC Manager.
6. The deployment workflow is expected to succeed.