VCF Operations Fleet Management Deployment Fails at "Import NTP server settings" with Error LCM_SETTINGS_API

Products

VCF Operations VMware Cloud Foundation VMware SDDC Manager

Issue/Introduction

When attempting to deploy VCF Operations fleet management via SDDC Manager, the deployment workflow fails at the following task
```
Import NTP server settings in VCF Operations fleet management that can be reused for environment creation
```

Error in /var/log/vmware/vcf/domainmanager/domainmanager.log in VCF Installer

ERROR [vcf_dm,69a5..............] [c.v.e.s.s.GetVcfOpsMgmtDeploymentStatusAction,dm-exec-7]  VCF Operations Management deployment workflow failed
ERROR [vcf_dm,69a5..............] [c.v.e.s.o.model.error.ErrorFactory,dm-exec-7]  [#####] FAILED_TO_DEPLOY_VCF_OPS_MGMT Failed to deploy VCF Operations fleet management on SDDC Manager
com.vmware.evo.sddc.orchestrator.exceptions.OrchTaskException: Failed to deploy VCF Operations fleet management on SDDC Manager

"Import NTP server settings in VCF Operations fleet management that can be reused for environment creation.","status":"FAILED","creationTimestamp":"yyyy-mm-ddThh-mm-ss.971Z","completionTimestamp":"yyyy-mm-ddThh-mm-ss.662Z","errors":[{"errorCode":"VCF_ERROR_INTERNAL_SERVER_ERROR","errorType":"ERROR","arguments":[],"message":"A problem has occurred on the server. Please retry or contact the service provider and provide the reference token.","causes":[{"type":"org.springframework.web.client.HttpClientErrorException$BadRequest","message":"400 : \"{\"status\":\"\",\"message\":\"Please check if the provided IP/FQDN is of an NTP server and is reachable.\",\"errorCode\":\"LCM_SETTINGS_API_ERROR0001\",\"errorLabel\":\"Settings request payload is invalid.\",\"recommendations\":[\"Check the settings data provided.\"]}\""}],"referenceToken":"#####"}]}

Error in SDDC Manager UI

Cause: 400 : "{"status":"","message":"Please check if the provided IP/FQDN is of an NTP server and is reachable.","errorCode":"LCM_SETTINGS_API_ERROR0001","errorLabel":"Settings request payload is invalid.","recommendations":["Check the settings data provided."]}"

Environment

VMware Cloud Foundation (VCF)
VMware SDDC Manager
VCF Operations (formerly vRealize/Aria Operations) Fleet Management

Cause

This issue occurs if an incorrect, non-existent, or unreachable NTP server name/IP was provided in the VCF Installer (Cloud Builder) during the initial bring-up phase.

During the fleet management deployment, the underlying Lifecycle Management (LCM) API attempts to validate the reachability of the NTP server inherited from the SDDC Manager's configuration (found in /etc/ntp.conf). Because the NTP server contains a typo or is otherwise unresolvable/unreachable, the LCM settings validation fails with a 400 Bad Request and halts the deployment workflow.

Resolution

To resolve this issue, you must correct the invalid NTP server address within the failed workflow specification and retry the task.

Follow the below steps:

Log in to the SDDC Manager appliance via SSH using the vcf user, and su to root.
Verify the incorrect NTP configuration by checking the local NTP configuration file:
```
cat /etc/ntp.conf
```
Follow the procedure outlined in KB Retry an existing workflow by modifying the workflow spec file to extract the JSON specification for the failed workflow.
Open the extracted JSON workflow spec in a text editor.
Locate the payload section containing the invalid NTP server FQDN/IP and correct the typo so that it points to a valid, reachable NTP server.
Save the modified workflow specification and update the SDDC Manager using the API as detailed in Step # 8 of KB Retry an existing workflow by modifying the workflow spec file
Return to the VCF Installer UI and click Retry on the failed VCF Operations Fleet Management deployment task.

NOTE: This issue may also occur if the NTP servers used are in the subnet 172.17.0.0/16, as this subnet is used for Docker internal networking on the Fleet Management appliance. If this is the case, use the steps from the following KB to change the default subnet used by Docker: How to change the default Docker 172.17.0.0 range in VMware Cloud Foundation 9.0 Ops Fleet Manager to avoid network conflicts with customer infrastructure