Intermittent Workflow Failures During Management Domain Bootstrap Deployment in VMware Cloud Foundation

Products

• VMware SDDC Manager / VCF Installer
• VMware Cloud Foundation

Issue/Introduction

• VMware Cloud Foundation (VCF) management domain bootstrap and deployment workflows fail intermittently. Specific tasks, such as updating the NSX firewall exclusion list or generating VCF Operations deployment input data, fail with java.net.UnknownHostException.

• During task execution for updating the firewall exclusion, the following is observed in the SDDC Manager logs:

YYYY-MM-DDT18:12:43 ERROR [vcf_dm,################################,####] [c.v.e.s.o.model.error.ErrorFactory,dm-exec-1557]  [E5UIBM] UPDATE_NSX_FIREWALL_EXCLUSION_LIST_FAILED Caught an exception in updating Firewall Exclusion list.
com.vmware.evo.sddc.orchestrator.exceptions.OrchTaskException: Caught an exception in updating Firewall Exclusion list.
... 
Caused by: java.net.UnknownHostException: <Host FQDN>

• During the preparation of input data for deployment, the following is observed:

YYYY-MM-DDT05:17:10 ERROR [vcf_dm,################################,####] [c.v.e.s.o.model.error.ErrorFactory,dm-exec-1696] [KNF16V] FLEET_CREATE_ENVIRONMENT_FAILED Failed to prepare data for management component deployment com.vmware.evo.sddc.orchestrator.exceptions.OrchTaskException: Failed to prepare data for management component deployment ... Caused by: java.net.UnknownHostException: <Host FQDN> 

• Manual validation utilizing nslookup and ping from the appliance typically succeeds, indicating the issue is transient rather than a persistent outage.

Environment

VMware Cloud Foundation 9.x

Cause

The underlying cause is intermittent instability, packet loss, or rate-limiting originating from the configured external DNS server infrastructure. FQDNs are successfully resolved initially but fail to resolve within a 30 to 45-second window during workflow execution.

• Log evidence demonstrates successful resolution followed immediately by a failure:

YYYY-MM-DDT10:55:03 INFO  [vcf_dm,################################,####] [c.v.v.v.v.VirtualMachineSpecValidator,dm-exec-33]  DNS name resolves to IPs [IPAddress(fqdn=<vCenter FQDN>, ipAddress=##.##.###.###, ipAddressVersion=IPv4)]
...
YYYY-MM-DDT10:55:36 ERROR [vcf_dm,################################,####] [c.v.e.s.common.util.NetworkService,dm-exec-26]  Host <vCenter FQDN> cannot be resolved <vCenter FQDN>: Temporary failure in name resolution
...
YYYY-MM-DDT10:56:17 DEBUG [vcf_dm,################################,####] [c.v.v.d.common.session.VlsiClient,vcf-ds-18]  Received a response from 'https://<vCenter FQDN>:443/sdk'

Resolution

To resolve and mitigate this issue:

• Retry Workflows: Retry the failed SDDC Manager workflows from the UI. Intermittent DNS responses will eventually permit the deployment tasks to complete successfully upon retry.
• Quantify DNS Drops: Execute the following continuous resolution monitoring script from the SDDC Manager or VCF Installer appliance. This will capture the exact failure rate. Replace <Host FQDN> with the failing infrastructure hostname:

• • for i in $(seq 1 50); do if getent ahosts <Host FQDN> >/dev/null; then R="OK"; else R="FAIL"; fi; printf "[$(date '+%T')] Request #%-2d: %s\n" "$i" "$R"; sleep 0.5; done | awk '{print; count[$NF]++} END {print "--------------------"; for (status in count) print count[status], status}'

• Engage Infrastructure Teams: Provide the output of the monitoring script to the local network and DNS administrators to investigate the upstream DNS infrastructure for rate limiting, UDP packet drops, or temporary service instability.