VMSP deployment fails at "Bootstrap VCF Services Platform" step during VCF upgrade

Products

• VMware Cloud Foundation

Issue/Introduction

When upgrading VMware Cloud Foundation (VCF) from 9.0 to 9.1, the VCF Services Management Platform (VMSP) deployment fails during the "Bootstrap VCF Services Platform" step in VCF Operations.

• In vCenter Server, the bootstrap VM powers off and is deleted shortly after creation. This cycle repeats three times before the deployment task permanently fails.
• The DNS server resides in a different subnet from the VCF Services Management network.
• Running "nslookup <vCenter Server FQDN> <DNS server ip>" times out during the deployment.
• If verbose logging is enabled for bootstrap_task.sh(See Additional Information), bootstrap_task_setup.log ends with below lines indicating failure to retrieve the vCenter Server SSL fingerprint:
  
  +++ openssl s_client -connect <vCenter Server FQDN>:443
+++ openssl x509 -sha1 -fingerprint -noout
+++ awk -F= '{ print $2 }'
++ vc_fingerprint=
+++ sleep 5
+++ openssl s_client -connect <vCenter Server FQDN>:443
+++ openssl x509 -sha1 -fingerprint -noout
+++ awk -F= '{ print $2 }'
++ vc_fingerprint=
  
  
• A network trace captured on the uplink port of the ESXi host running the bootstrap VM shows that the return traffic from the DNS server is sent to a different MAC address from the one assigned to the bootstrap VM.

Environment

VMware Cloud Foundation 9.1

Cause

As the DNS server resides in a different subnet from the VCF Services Management network, the name resolution traffic must pass through a physical router.

In this particular scenario, the physical router's ARP table contains a stale MAC address entry for the bootstrap VM's IP, causing DNS reply traffic to be discarded by the virtual distributed switch which in turn causes the bootstrap VM's DNS queries to fail.

Resolution

Workaround:

1. Delete the existing failed bootstrap VM and clear the VMSP deployment task in SDDC Manager if required following: "Deploy VCF Management Components" task cannot be re-initiated after failure.
2. Clear the stale ARP entry for the bootstrap VM's IP address on physical router, or wait for the ARP entry to age out naturally.
3. Re-initiate the VMSP deployment after the ARP entry is cleared.

Permanent fix:

Engineering is aware of this issue and plans to resolve it in a future release by forcing the bootstrap VM to send a Gratuitous ARP (GARP) request during its initial boot.

Additional Information

To enable verbose logging for the bootstrap script:

1. Login to bootstrap VM and open /data/vmsp/scripts/bootstrap_task.sh in a text editor.
2. Locate the following line:
   
   set -euo pipefail
   
   
3. Change it to:
   
   set -exuo pipefail