VCF 9.1 installer fails at "VCF Management Services Deployment Task" due to unreachable NTP server

Products

• VMware SDDC Manager / VCF Installer

Issue/Introduction

• When deploying VMware Cloud Foundation (VCF) 9.1, the Management Services deployment task fails with VCF installer UI reporting the failure at "Install VCF Operations and VCF Cloud proxy using Fleet Lifecycle" even though the VCF Operations node(s) get deployed with an error similar to below:
  
  The Fleet lifecycle 'OVA component installation' task with ID <TASK_ID> failed. Internal errors: Failed to deploy VCF Operations node. Check vCenter tasks for errors., Status polling timed out. Review the logs to check operation progress. Increase the timeout value if the operation requires more time., Failed to deploy VCF Operations node. Check vCenter tasks for errors. REFERENCE TOKEN: <TOKEN>
  
  
  
  
• Reviewing the domainmanager.log on the VCF Installer or SDDC Manager shows a generic PUBLIC_FLEET_LCM_TASK_FAILED entry due to time out:
  
  /var/log/vmware/vcf/domainmanager/domainmanager.log

YYYY-MM-DDThh:mm:ss ERROR [vcf_dm,########,####] [c.v.e.s.o.model.error.ErrorFactory,dm-exec-386]  [TOKEN] PUBLIC_FLEET_LCM_TASK_FAILED The Fleet lifecycle {0} task with ID <SDDC_BUILD_TASK_ID> failed. Internal errors: Failed to deploy VCF Operations node. Check vCenter tasks for errors., Status polling timed out. Review the logs to check operation progress. Increase the timeout value if the operation requires more time. Status polling timed out.
       at com.vmware.vcf.vsp.services.fleetlcm.FleetLcmTaskExecutor.executeWithRetry(FleetLcmTaskExecutor.java:102)
        at com.vmware.vcf.vsp.services.fleetlcm.actions.InstallOvaComponentFleetLcmAction.execute(InstallOvaComponentFleetLcmAction.java:58)
...
YYYY-MM-DDThh:mm:ss DEBUG [vcf_dm,########,####] [c.v.e.s.o.c.ProcessingTaskSubscriber,dm-exec-386]  Collected the following errors for task with name InstallOvaComponentFleetLcmAction and ID ####-####-####-####: [ExecutionError [errorCode=null, errorResponse=LocalizableErrorResponse(messageBundle=com.vmware.evo.sddc.common.services.sddcmanagermessages)]]
  
  
  
•  Upon reviewing the SDDC-build-service logs to review the Management services task , NTP validation failure is observed:
  
  /vcf-sddc-lcm/####-####/vcf-sddc-build-service-sddcbuild-#########-#####/sddcbuild/console-log-####-####.log

YYYY-MM-DDThh:mm:ss stdout F ERROR sddc-build-service 53 [lcm@4413 scPath="ops" thread="system-task-worker-14" logger="c.b.v.o.casa.service.OpsCasaService" method="configureCluster:220"] [<SDDC_BUILD_TASK_ID>, taskId=####-####-####-####] - Failed to create VCF Operations cluster
YYYY-MM-DDThh:mm:ss stdout F org.springframework.web.client.HttpServerErrorException$InternalServerError: 500 500 on POST request for "https://<VCF_OPS_NODE_FQDN>:443/casa/cluster":
"{"error_message_key":"ntp.chain.validation.failed","error_arguments":["<PROBLEMATIC_NTP_SERVER_IP>"],"error_message":"Specified NTP server is specified incorrect."}"

Environment

VCF 9.1

Cause

This issue is being encountered as the VCF Operations node(s) are unable to reach the configured NTP server.

Resolution

To resolve the issue, fix the NTP server connectivity from the VCF Ops node(s) environment by following the below steps:

1. Query the NTP servers from the VCF Operations node(s) using below command:
   ntpdate -q <NTP_IP> 
   
   
2. Review the firewall config in the environment to ensure there is no blockage of UDP traffic over port 123 between NTP server and VCF Ops node(s) .

Additional Information

To retrieve the depot-service logs follow either of the below methods:

1. Offline method: Generate Support bundle for VCF Management Services
   
   
   1. Login to SDDC Manager/VCF Installer using vcf user over SSH, elevate to root using "su" and generate support bundle for VCF Management services using SOS command "/opt/vmware/sddc-support/sos --vcf-mgmt-svcs-logs --zip"
      
      
   2. Extract the bundle and navigate to /var/log/vmware/vcf/sddc-support/sos-####/VCFMgmtComponents/vcf-management-services/####-####/services-logs/vcf-sddc-lcm/####-####/vcf-sddc-build-service-sddcbuild-#####-#####/sddcbuild/console-log-####.log
      
      
2. Online method: Review the logs for sddcbuild container inside sddc-build-service pod:
   
   
   1. Login to one of the Mgmt services node using vmware-system-user


   2. Elevate to root:
      
      sudo -i
      
      
   3. Find the Control plane node:
      
      if [ -f /etc/kubernetes/manifests/kube-apiserver.yaml ]; then echo -e "\n This is a CONTROL PLANE node"; else echo -e "\n This is a WORKER node"; echo " Control Plane Endpoint: $(sudo grep 'server:' /etc/kubernetes/kubelet.conf 2>/dev/null | awk -F'/' '{print $3}')"; fi
      
      
   4. SSH to the control plane node using same user credential and set kubeconfig:
      
      export KUBECONFIG=/etc/kubernetes/admin.conf
      
      
   5. Check Fleet & SDDC pods and the tasks running to obtain the the sddc-build-service pod name under vcf-sddc-lcm namespace :
      
      echo "--- FLEET PODS ---"; kubectl get pods -A | grep -E '(fleet|sddc)'; echo -e "\n--- COMPONENT TASKS (vmsp-platform) ---"; kubectl get tasks -n vmsp-platform --no-headers | grep -E '^(install|stage|configure)-component'
      
      
   6. From the above output, use the sddc-build-service pod name and check the logs for sddcbuild container inside it:
      
      kubectl logs -n vcf-sddc-lcm vcf-sddc-build-service-sddcbuild-####-#### -c sddcbuild |less