Workload domain import in VCF fails with error "Failed to update the known hosts on SDDC Manager localhost" due to incorrect format
search cancel

Workload domain import in VCF fails with error "Failed to update the known hosts on SDDC Manager localhost" due to incorrect format

book

Article ID: 440318

calendar_today

Updated On:

Products

VMware SDDC Manager / VCF Installer

Issue/Introduction

  • Attempting to perform brownfield import of a workload domain in SDDC Manager encounters the error "Failed to update the known hosts on SDDC Manager localhost" .

  • The /var/log/vmware/vcf/domainmanager/domainmanager.log on the SDDC Manager reports Failed to update known host config :

    YYYY-MM-DDThh:mm:ss INFO  [vcf_dm,<task_id>,<subtask_id>] [c.v.v.v.s.b.util.SshKnownHostsUtil,http-nio-127.0.0.1-7200-exec-9]  Updating the known hosts file on the VM localhost
    YYYY-MM-DDThh:mm:ss INFO  [vcf_dm,<task_id>,<subtask_id>] [c.v.e.s.c.s.s.SshConfigurationRestClient,http-nio-127.0.0.1-7200-exec-9]  Setting SDDC Manager(localhost) known hosts configuration
    ...
    YYYY-MM-DDThh:mm:ss INFO  [vcf_dm,<task_id>,<subtask_id>] [c.v.e.s.c.s.s.SshConfigurationRestClient,http-nio-127.0.0.1-7200-exec-9]  Getting known host config
    ...
    YYYY-MM-DDThh:mm:ss ERROR [vcf_dm,<task_id>,<subtask_id>] [c.v.e.s.c.s.s.SshConfigurationRestClient,http-nio-127.0.0.1-7200-exec-9]  Failed to update known host config
    at org.springframework.web.client.HttpClientErrorException$BadRequest: 400 : "{"errorCode":"REST_INVALID_API_INPUT","arguments":[],"message":"Invalid input","remediationMessage":"Enter correct API input","nestedErrors":[{"errorCode":"ANNOTATIONS_MISMATCH","arguments":["appliance.CS_APPLIANCE_SSH_HOST_INVALID_FORMAT,appliance.CS_APPLIANCE_SSH_HOST_INVALID_FORMAT"],"message":"Following conditions do not match - appliance.CS_APPLIANCE_SSH_HOST_INVALID_FORMAT,appliance.CS_APPLIANCE_SSH_HOST_INVALID_FORMAT"}],"referenceToken":"####"}"
    ...
    Caused by: org.springframework.web.client.HttpClientErrorException$BadRequest: 400 : "{"errorCode":"REST_INVALID_API_INPUT","arguments":[],"message":"Invalid input","remediationMessage":"Enter correct API input","nestedErrors":[{"errorCode":"ANNOTATIONS_MISMATCH","arguments":["appliance.CS_APPLIANCE_SSH_HOST_INVALID_FORMAT,appliance.CS_APPLIANCE_SSH_HOST_INVALID_FORMAT"],"message":"Following conditions do not match - appliance.CS_APPLIANCE_SSH_HOST_INVALID_FORMAT,appliance.CS_APPLIANCE_SSH_HOST_INVALID_FORMAT"}],"referenceToken":"####"}"
    ...
    YYYY-MM-DDThh:mm:ss ERROR [vcf_dm,<task_id>,<subtask_id>] [c.v.v.v.c.v.BrownfieldImportController,http-nio-127.0.0.1-7200-exec-9]  Failed to update the known hosts on SDDC Manager localhost
    com.vmware.evo.sddc.orchestrator.exceptions.OrchTaskException: Failed to update the known hosts on SDDC Manager localhost 

  • The /var/log/vmware/vcf/commonsvcs/commonsvcs.log on the SDDC Manager also reports one or more incorrect host entries as "rejected value [<INCORRECT_ENTRY>]" at "knownHosts[<line_no>]" in the known_hosts file which are not in the format of FQDN or IP address:

    YYYY-MM-DDThh:mm:ss INFO  [common,<task_id>,<subtask_id>] [c.v.v.l.a.a.ActivityLoggingInterceptor,http-nio-127.0.0.1-7100-exec-2] {"username":null,"timestamp":"YYYY-MM-DDThh:mm:ss","clientIP":"127.0.0.1","userAgent":"Apache-HttpClient/5.3.1 (Java/17.0.12)","api":"/appliancemanager/ssh/knownHosts","httpMethod":"GET","httpStatus":200,"operation":"Get the current SSH known hosts configuration","remoteIP":"127.0.0.1","duration":8}
    ...
    YYYY-MM-DDThh:mm:ss ERROR [common,<task_id>,<subtask_id>] [c.v.e.s.e.h.MethodArgumentNotValidExceptionHandler,http-nio-127.0.0.1-7100-exec-3] Invalid API input: Error fields: [knownHosts[<line_no>].host, knownHosts[<line_no>].host] Details: org.springframework.web.bind.MethodArgumentNotValidException: Validation failed for argument [0] in public void com.vmware.evo.sddc.appliance.utilities.api.rest.SshController.setSshKnownHostsConfiguration(com.vmware.evo.sddc.appliance.rest.api.model.SshKnownHostsConfigurationSpec) with <no_of_errors> errors: [Field error in object 'sshKnownHostsConfigurationSpec' on field 'knownHosts[<line_no>].host': rejected value [<INCORRECT_ENTRY>]; codes [Pattern.sshKnownHostsConfigurationSpec.knownHosts[<line_no>].host,Pattern.sshKnownHostsConfigurationSpec.knownHosts.host,Pattern.knownHosts[<line_no>].host,Pattern.knownHosts.host,Pattern.host,Pattern.java.lang.String,Pattern]; arguments [org.springframework.context.support.DefaultMessageSourceResolvable: codes [sshKnownHostsConfigurationSpec.knownHosts[<line_no>].host,knownHosts[<line_no>].host]; arguments []; default message [knownHosts[<line_no>].host],[Ljakarta.validation.constraints.Pattern$Flag;@33375648,^(([a-zA-Z0-9]|[a-zA-Z0-9][a-zA-Z0-9\\-]*[a-zA-Z0-9])\\.)*([A-Za-z0-9]|[A-Za-z0-9][A-Za-z0-9\\-]*[A-Za-z0-9])$|(\\d{1,3})\\.(\\d{1,3})\\.(\\d{1,3})\\.(\\d{1,3})]; default message [{appliance.CS_APPLIANCE_SSH_HOST_INVALID_FORMAT}]] [Field error in object 'sshKnownHostsConfigurationSpec' on field 'knownHosts[<line_no>].host': rejected value [<INCORRECT_ENTRY>]; codes [Pattern.sshKnownHostsConfigurationSpec.knownHosts[<line_no>].host,Pattern.sshKnownHostsConfigurationSpec.knownHosts.host,Pattern.knownHosts[<line_no>].host,Pattern.knownHosts.host,Pattern.host,Pattern.java.lang.String,Pattern]; arguments [org.springframework.context.support.DefaultMessageSourceResolvable: codes [sshKnownHostsConfigurationSpec.knownHosts[<line_no>].host,knownHosts[<line_no>].host]; arguments []; default message [knownHosts[<line_no>].host],[Ljakarta.validation.constraints.Pattern$Flag;@33375648,^(([a-zA-Z0-9]|[a-zA-Z0-9][a-zA-Z0-9\\-]*[a-zA-Z0-9])\\.)*([A-Za-z0-9]|[A-Za-z0-9][A-Za-z0-9\\-]*[A-Za-z0-9])$|(\\d{1,3})\\.(\\d{1,3})\\.(\\d{1,3})\\.(\\d{1,3})]; default message [{appliance.CS_APPLIANCE_SSH_HOST_INVALID_FORMAT}]]
    YYYY-MM-DDThh:mm:ss ERROR [common,<task_id>,<subtask_id>] [c.v.e.s.c.v.i.LocalizableAnnotationValidationUtil,http-nio-127.0.0.1-7100-exec-3] Spec violation CS_APPLIANCE_SSH_HOST_INVALID_FORMAT

Environment

VCF 5.x

VCF 9.x

Cause

During the brownfield import process, the SDDC Manager attempts to update the known_hosts files. This operation fails because one or more files contain hostname entries that are not in a valid Fully Qualified Domain Name (FQDN) or IP address format as reported on the commonsvcs.log. This also be verified by executing the below command:

curl -X GET http://localhost:7100/appliancemanager/ssh/knownHosts | jq '.' | grep host

Below are some examples of incorrect hostname entries that are not FQDN or IP:

Resolution

Follow the below steps to identify and remove the invalid entries from the SDDC Manager configuration files to allow the import to proceed:

  1. Take an offline snapshot of the SDDC Manager appliance before you begin.

  2. Connect to the SDDC Manager via SSH as the vcf user and switch to root.

  3. Review the the current SSH known hosts configuration for the incorrect entries reported by commonsvcs.log (rejected value [<INCORRECT_ENTRY>]) :

    curl -X GET http://localhost:7100/appliancemanager/ssh/knownHosts | jq '.'

  4. Remove the incorrect entries from the following four known_hosts files on the SDDC Manager :

    sed -i "/<INCORRECT_ENTRY>/d" /root/.ssh/known_hosts 2>/dev/null
    sed -i "/<INCORRECT_ENTRY>/d" /etc/vmware/vcf/commonsvcs/known_hosts 2>/dev/null
    sed -i "/<INCORRECT_ENTRY>/d" /home/vcf/.ssh/known_hosts 2>/dev/null
    sed -i "/<INCORRECT_ENTRY>/d" /opt/vmware/vcf/commonsvcs/defaults/hosts/known_hosts 2>/dev/null

  5. Verify the cleanup by running the curl command from step 3 again

  6. Retry the vCenter import workflow from the SDDC Manager UI.

Powered by Wolken Software