Fleet deployment fails with "The Fleet lifecycle 'OVA component installation' task with ID <Task_ID> failed. Internal errors: Failed to deploy VCF Operations node. Check vCenter tasks for errors."

Products

• VMware SDDC Manager / VCF Installer

Issue/Introduction

In a Greenfield environment, new installation the Fleet deployment Task, fails consistently at the the following stage
"Install VCF Operations and VCF Cloud Proxy using Fleet Lifecycle"

The error message in the VCF Installer is:
"The Fleet lifecycle 'OVA component installation' task with ID <Task_ID> failed. Internal errors: Failed to deploy VCF Operations node. Check vCenter tasks for errors.
Reference tokens we've seen across attempts: <Token_ID> <Token_ID>"

Scenario occurs when Hosts are in a different subnet/ network, than VC, SDDC, VCF Ops, VCF runtime service.

Environment

9.1

Cause

Hosts are unable to communicate with SDDC lifecycle pod aka SBS - SDDC Build Service 

Resolution

1. Although there is no VCF Ops UI, there are a number of VMs deployed, under VMs and Templates > in folder vcf-management-services >  recognizable via naming convention *vsp-<uniq-ID>



2. To test the connection SSH on the vsp-<uniq-ID> Vms
ssh with vmware-system-user
elevate to root 
# sudo -i

3. Identify which one is the Control Plane by identifying which one has the admin.conf
# ls -althr /etc/kubernetes
total 56K
drwxr-xr-x   2 root root 4.0K Apr 18 19:32 patches
drwxr-xr-x   3 root root 4.0K Apr 29 18:33 pki
-rw-------   1 root root 5.5K Apr 29 18:33 admin.conf
-rw-------   1 root root 5.5K Apr 29 18:33 super-admin.conf
-rw-------   1 root root 5.5K Apr 29 18:33 controller-manager.conf
-rw-------   1 root root 5.5K Apr 29 18:33 scheduler.conf
drwxr-xr-x   5 root root 4.0K Apr 29 18:33 .
drwxr-xr-x   2 root root 4.0K Apr 29 18:33 manifests
-rw-------   1 root root 1.9K Apr 29 18:33 kubelet.conf
drwxr-xr-x. 61 root root 4.0K Jun 15 12:57 ..

4. Export Kube configuration 
# export KUBECONFIG=/etc/kubernetes/admin.conf

5. Switch to SBS - SDDC Build Service
SBS_POD=$(kubectl get pods -n vcf-sddc-lcm -l app=sddcbuild -o jsonpath='{.items[0].metadata.name}')
kubectl exec -it "$SBS_POD" -n vcf-sddc-lcm -- bash

6. Path and hostname will change 

7. Test DNS and reverse DNS [replace with your hostnames and IPs] - none should fail

Forward
# for fqdn in \
<your EXS1> \
<your EXS2> \
<your EXS3> \
<your EXS TBD> ; do
echo -n "  $fqdn -> "
getent hosts "$fqdn" || echo "FAILED"
done

Reverse
# for ip in IP_of_ESX1 IP_of_ESX2 IP_of_ESX3 IP_of_ESX4 ; do echo -n " $ip -> " getent hosts "$ip" || echo "FAILED" done

8. Test Connection ICPM and ports [while ping might fail - check the other ports as well, see below ]

root [ /opt/vmware/vcf/sddcbuild/lib ]# ping esx01.vcf.lab
Ping esx01.vcf.lab (192.168.1.101): 56(84) bytes.
64 bytes from 192.168.1.101: icmp_seq=1 ttl=63 time=2 ms
64 bytes from 192.168.1.101: icmp_seq=2 ttl=63 time=1 ms


root [ /opt/vmware/vcf/sddcbuild/lib ]# curl -v telnet://<your ESX1>:443
* Host <your ESX1>:443 was resolved.
* IPv6: (none)
* IPv4: 192.168.1.101
*   Trying 192.168.1.101:443...
* Established connection to <your ESX1> (192.168.1.101 port 443) from 198.18.1.22 port 53878
^C
root [ /opt/vmware/vcf/sddcbuild/lib ]# curl -v telnet://<your ESX1>:80
* Host <your ESX1>:80 was resolved.
* IPv6: (none)
* IPv4: 192.168.1.101
*   Trying 192.168.1.101:80...
* Established connection to <your ESX1> (192.168.1.101 port 80) from 198.18.1.22 port 49038
^C
root [ /opt/vmware/vcf/sddcbuild/lib ]# curl -v telnet://<your ESX1>:902
* Host <your ESX1>:902 was resolved.
* IPv6: (none)
* IPv4: 192.168.1.101
*   Trying 192.168.1.101:902...
* Established connection to <your ESX1> (192.168.1.101 port 902) from 198.18.1.22 port 50286
220 VMware Authentication Daemon Version 1.10: SSL Required, ServerDaemonProtocol:SOAP, MKSDisplayProtocol:VNC , VMXARGS supported, NFCSSL supported/t, SHA256 supported
^C
root [ /opt/vmware/vcf/sddcbuild/lib ]# curl -v telnet://<your ESX1>:443