SDDC Backup Configuration Fails with "Failed to update known hosts configuration"
search cancel

SDDC Backup Configuration Fails with "Failed to update known hosts configuration"

book

Article ID: 427682

calendar_today

Updated On:

Products

VMware SDDC Manager

Issue/Introduction

  • When configuring backups in VMware Cloud Foundation (VCF), the workflow fails during the task validation phase. The VCF Operations Manager UI displays the following error:

    Message: Failed to update known hosts configuration.

    Reference Token:     7###BH

    Cause: Cause: Unable to update SSH known host configuration. Failed to post data to uri /appliancemanager/ssh/knownHosts, got status code 400 and response
    ("errorCode":"REST_INVALID_API_INPUT","arguments":[],"message":"Invalid input","remediationMessage":"Enter correct API input","nestedErrors":[("errorCode":"ANNOTATIONS_MISMATCH","arguments":["appliance.CS_APPLIANCE_SSH_HOST_INVALID_FORMAT,appliance.CS_APPLIANCE_SSH_HOST_INVALID_FORMAT"],"message":"Followingconditions do not match appliance.CS_APPLIANCE_SSH_HOST_INVALID_FORMAT,appliance.CS_APPLIANCE_SSH_HOST_INVALID_FORMAT")],"referenceToken":"7###BH"

  • The /var/log/vmware/vcf/operationmanager/operationmanager.log confirms a 400 Bad Request when the system attempts to sync SSH keys

    YYYY-MM-DDT0HH:MM:SS DEBUG [vcf_om,696a####0d1caef####0fe363c1d###,a86f] [c.v.v.secure.http.HttpClientService,pool-2-thread-2] Starting POST request from host: 127.0.0.1, port: 80, isSecure: false, path: /appliancemanager/ssh/knownHosts, queryParamMap: null, headers: {Accept=application/json,text/plain, Content-Type=application/json}
    YYYY-MM-DDT0HH:MM:SS DEBUG [vcf_om,696a####0d1caef####0fe363c1d###,a86f] [c.v.v.secure.http.HttpClientService,pool-2-thread-2] Making request: POST http://127.0.0.1:80/appliancemanager/ssh/knownHosts
    YYYY-MM-DDT0HH:MM:SS DEBUG [vcf_om,696a####0d1caef####0fe363c1d###,a86f] [c.v.v.secure.http.HttpClientService,pool-2-thread-2] Received response with status: 400 reason: POST ... for  request with host: 127.0.0.1, port: 80, isSecure: false, path: /appliancemanager/ssh/knownHosts, queryParamMap: null, headers: {Accept=application/json,text/plain, Content-Type=application/json}
    YYYY-MM-DDT0HH:MM:SS INFO  [vcf_om,696a####0d1caef####0fe363c1d###,a86f] [c.v.v.secure.http.HttpClientService,pool-2-thread-2] Response Body: {"errorCode":"REST_INVALID_API_INPUT","arguments":[],"message":"Invalid input","remediationMessage":"Enter correct API input","nestedErrors":[{"errorCode":"ANNOTATIONS_MISMATCH","arguments":["appliance.CS_APPLIANCE_SSH_HOST_INVALID_FORMAT,appliance.CS_APPLIANCE_SSH_HOST_INVALID_FORMAT"],"message":"Following conditions do not match - appliance.CS_APPLIANCE_SSH_HOST_INVALID_FORMAT,appliance.CS_APPLIANCE_SSH_HOST_INVALID_FORMAT"}],"referenceToken":"7###BH"}
    YYYY-MM-DDT0HH:MM:SS ERROR [vcf_om,696a####0d1caef####0fe363c1d###,a86f] [c.v.e.s.c.s.a.a.ApplianceAdapterImpl,pool-2-thread-2] Failed to post data to uri /appliancemanager/ssh/knownHosts, got status code 400 and response {"errorCode":"REST_INVALID_API_INPUT","arguments":[],"message":"Invalid input","remediationMessage":"Enter correct API input","nestedErrors":[{"errorCode":"ANNOTATIONS_MISMATCH","arguments":["appliance.CS_APPLIANCE_SSH_HOST_INVALID_FORMAT,appliance.CS_APPLIANCE_SSH_HOST_INVALID_FORMAT"],"message":"Following conditions do not match - appliance.CS_APPLIANCE_SSH_HOST_INVALID_FORMAT,appliance.CS_APPLIANCE_SSH_HOST_INVALID_FORMAT"}],"referenceToken":"7###BH"}
    YYYY-MM-DDT0HH:MM:SS ERROR [vcf_om,696a####0d1caef####0fe363c1d###,a86f] [c.v.e.s.c.s.a.a.ApplianceAdapterImpl,pool-2-thread-2] Failed to update known host config
    org.springframework.web.client.RestClientException: Failed to post data to uri /appliancemanager/ssh/knownHosts, got status code 400 and response {"errorCode":"REST_INVALID_API_INPUT","arguments":[],"message":"Invalid input","remediationMessage":"Enter correct API input","nestedErrors":[{"errorCode":"ANNOTATIONS_MISMATCH","arguments":["appliance.CS_APPLIANCE_SSH_HOST_INVALID_FORMAT,appliance.CS_APPLIANCE_SSH_HOST_INVALID_
    FORMAT"],"message":"Following conditions do not match - appliance.CS_APPLIANCE_SSH_HOST_INVALID_FORMAT,appliance.CS_APPLIANCE_SSH_HOST_INVALID_FORMAT"}],"referenceToken":"7###BH"}
            at com.vmware.evo.sddc.common.services.adapters.appliance.ApplianceAdapterImpl.postForResponseMessage(ApplianceAdapterImpl.java:162)
    YYYY-MM-DDT0HH:MM:SS ERROR [vcf_om,696a####0d1caef####0fe363c1d###,a86f] [c.v.v.c.f.p.b.r.a.UpdateKnownHostsAction,pool-2-thread-2] Failed to update known hosts configuration
    com.vmware.evo.sddc.common.services.error.SddcManagerServicesIsException: Unable to update SSH known host configuration.
            at com.vmware.evo.sddc.common.services.adapters.appliance.ApplianceAdapterImpl.updateKnownHostConfig(ApplianceAdapterImpl.java:71)
            at java.base/java.util.ArrayList.forEach(ArrayList.java:1511)
            ... 19 common frames omitted
    YYYY-MM-DDT0HH:MM:SS ERROR [vcf_om,696a####0d1caef####0fe363c1d###,a86f] [c.v.e.s.o.model.error.ErrorFactory,pool-2-thread-2] [7###BH] UPDATE_KNOWN_HOSTS_CONFIGURATION_FAILED Failed to update known hosts configuration.
    com.vmware.evo.sddc.orchestrator.exceptions.OrchTaskException: Failed to update known hosts configuration.
            at com.vmware.vcf.common.fsm.plugins.backup.restore.action.UpdateKnownHostsAction.execute(UpdateKnownHostsAction.java:58)
            at com.vmware.vcf.common.fsm.plugins.backup.restore.action.UpdateKnownHostsAction.execute(UpdateKnownHostsAction.java:27
    Caused by: com.vmware.evo.sddc.common.services.error.SddcManagerServicesIsException: Unable to update SSH known host configuration.
            at com.vmware.evo.sddc.common.services.adapters.appliance.ApplianceAdapterImpl.updateKnownHostConfig(ApplianceAdapterImpl.java:71)
            at com.vmware.evo.sddc.common.services.adapters.appliance.ApplianceAdapterImpl.updateKnownHostsConfigurationEntry(ApplianceAdapterImpl.java:108)

  • Further inspection of /var/log/vmware/vcf/commonsvcs.log identifies the specific rejected entries:

    YYYY-MM-DDT0HH:MM:SS ERROR [common,696####eb8####3d17238###d7d##,4d88] [c.v.e.s.e.h.MethodArgumentNotValidExceptionHandler,http-nio-127.0.0.1-7100-exec-1] Invalid API input: Error fields: [knownHosts[39].host, knownHosts[38].host] Details: org.springframework.web.bind.MethodArgumentNotValidException: Validation failed for argument [0] in public void com.vmware.evo.sddc.appliance.utilities.api.rest.SshController.setSshKnownHostsConfiguration(com.vmware.evo.sddc.appliance.rest.api.model.SshKnownHostsConfigurationSpec) with 2 errors: [Field error in object 'sshKnownHostsConfigurationSpec' on field 'knownHosts[39].host': rejected value [####.####.####.####.###]; codes [Pattern.sshKnownHostsConfigurationSpec.knownHosts[39].host,Pattern.sshKnownHostsConfigurationSpec.knownHosts.host,Pattern.knownHosts[39].host,Pattern.knownHosts.host,Pattern.host,Pattern.java.lang.String,Pattern]; arguments [org.springframework.context.support.DefaultMessageSourceResolvable: codes [sshKnownHostsConfigurationSpec.knownHosts[39].host,knownHosts[39].host]; arguments []; default message [knownHosts[39].host],[Ljakarta.validation.constraints.Pattern$Flag;@a0c01da,^(([a-zA-Z0-9]|[a-zA-Z0-9][a-zA-Z0-9\\-]*[a-zA-Z0-9])\\.)*([A-Za-z0-9]|[A-Za-z0-9][A-Za-z0-9\\-]*[A-Za-z0-9])$|(\\d{1,3})\\.(\\d{1,3})\\.(\\d{1,3})\\.(\\d{1,3})]; default message [{appliance.CS_APPLIANCE_SSH_HOST_INVALID_FORMAT}]] [Field error in object 'sshKnownHostsConfigurationSpec' on field 'knownHosts[38].host': rejected value XXXX.XXXX.XXX.XX.XX]; codes [Pattern.sshKnownHostsConfigurationSpec.knownHosts[38].host,Pattern.sshKnownHostsConfigurationSpec.knownHosts.host,Pattern.knownHosts[38].host,Pattern.knownHosts.host,Pattern.host,Pattern.java.lang.String,Pattern]; arguments [org.springframework.context.support.DefaultMessageSourceResolvable: codes [sshKnownHostsConfigurationSpec.knownHosts[38].host,knownHosts[38].host]; arguments []; default message [knownHosts[38].host],[Ljakarta.validation.constraints.Pattern$Flag;@a0c01da,^(([a-zA-Z0-9]|[a-zA-Z0-9][a-zA-Z0-9\\-]*[a-zA-Z0-9])\\.)*([A-Za-z0-9]|[A-Za-z0-9][A-Za-z0-9\\-]*[A-Za-z0-9])$|(\\d{1,3})\\.(\\d{1,3})\\.(\\d{1,3})\\.(\\d{1,3})]; default message [{appliance.CS_APPLIANCE_SSH_HOST_INVALID_FORMAT}]]
    YYYY-MM-DDT0HH:MM:SS ERROR [common,696####eb8####3d17238###d7d##,4d88] [c.v.e.s.c.v.i.LocalizableAnnotationValidationUtil,http-nio-127.0.0.1-7100-exec-1] Spec violation CS_APPLIANCE_SSH_HOST_INVALID_FORMAT
    YYYY-MM-DDT0HH:MM:SS ERROR [common,696####eb8####3d17238###d7d##,4d88] [c.v.e.s.c.v.i.LocalizableAnnotationValidationUtil,http-nio-127.0.0.1-7100-exec-1] Prefix appliance not found
    YYYY-MM-DDT0HH:MM:SS ERROR [common,696####eb8####3d17238###d7d##,4d88] [c.v.e.s.c.v.i.LocalizableAnnotationValidationUtil,http-nio-127.0.0.1-7100-exec-1] Unknown prefix appliance, Error appliance.CS_APPLIANCE_SSH_HOST_INVALID_FORMAT cannot be localized
    YYYY-MM-DDT0HH:MM:SS ERROR [common,696####eb8####3d17238###d7d##,4d88] [c.v.e.s.e.h.LocalizableRuntimeExceptionHandler,http-nio-127.0.0.1-7100-exec-1] [7###BH] REST_INVALID_API_INPUT Invalid input
    com.vmware.evo.sddc.common.core.error.CompositeInvalidInputException: Invalid input
            at com.vmware.evo.sddc.exception.handler.MethodArgumentNotValidExceptionHandler.handleMethodArgumentNotValidException(MethodArgumentNotValidExceptionHandler.java:71)
            at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)

 

 

 

Cause

The issue is caused by invalid formatting or unrecognized symbols within the known_hosts entries in the SDDC Manager local files.

Common formatting errors include an FQDN ending with a period (e.g., vcenter.example.com.)

When VCF attempts to push these entries to the appliancemanager via a REST API call, the input validation fails because the format does not match the strict CS_APPLIANCE_SSH_HOST_INVALID_FORMAT requirement.

 

Resolution

To resolve this issue, the malformed entries must be identified and removed from the various known_hosts files on the SDDC Manager.

Step 1: Identify the Malformed Entry

  1. Log in to the SDDC Manager via SSH as the vcf user and switch to root.

  2. Run the following curl command to view the current known hosts as seen by the API:

    curl -X GET http://localhost:7100/appliancemanager/ssh/knownHosts | jq '.'

  3. Examine the output for hostnames with trailing dots or leading spaces.

    • Example of a bad entry: "host": "example.com."

      vcf@<FQDN_OF_SDDC> [ ~ ]$ curl -X GET http://localhost:7100/appliancemanager/ssh/knownHosts | jq '.'
        % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                       Dload  Upload   Total   Spent    Left  Speed
      100 17916    0 17916    0     0  1218k      0 --:--:-- --:--:-- --:--:-- 1249k
      {
        "knownHosts": [
          {
            "host": " ##.##.##.##",
            "keyType": "ecdsa-sha2-nistp256",
            "key": "AAAA#####XNoYTIt#####HAyNTY####Ibmlzd#####TYA#####LbYzvSCrE/DgWL#####L89VbDpq####jbM6uVs####vC5tz/D1yjWldg#####f8a4aAhOHKq#####SdfySUz0/v6w8="
          },
          {
            "host": "example.com.",
            "keyType": "ecdsa-sha2-nistp384",
            "key": ""AAAA#####XNoYTIt#####HAyNTY####Ibmlzd#####TYA#####LbYzvSCrE/DgWL#####L89VbDpq####jbM6uVs####vC5tz/D1yjWldg#####f8a4aAhOHKq#####SdfySUz0/v6w8="
          },
          {
            "host": "example.com.",
            "keyType": "ssh-rsa",
            "key": "AAAA#####XNoYTIt#####HAyNTY####Ibmlzd#####TYA#####LbYzvSCrE/DgWL#####L89VbDpq####jbM6uVs####vC5tz/D1yjWldg#####f8a4aAhOHKq#####SdfySUz0/v6w8=/z/D1yjWldg#####f8a4aAhOHKq#####SdfySUz0/v6w8=/HAyNTY####Ibmlzd#####TYA#####LbYzvSCrE"
          },


Step 2: Clean Up Configuration Files

Once the malformed entries (such as hostnames with trailing dots) are identified, they must be removed from the local configuration files using the vi text editor.

Perform the following steps for each of the files listed below:

  1. Open the file: Use the vi command followed by the file path.

    vi /etc/vmware/vcf/commonsvcs/known_hosts
  2. Locate the malformed line: Use the arrow keys to navigate to the line containing the incorrect entry and remove ".'

  3. Save and Exit:

    • Press the Esc key to ensure you are in Command Mode.

    • Type :wq! and press Enter

  4. Repeat these steps for the remaining locations:

     vi /root/.ssh/known_hosts

     vi /home/vcf/.ssh/known_hosts

     vi /opt/vmware/vcf/commonsvcs/defaults/hosts/known_hosts

Step 3: Verify and Retry

  1. After removing the incorrect symbols or spaces, run the curl command from Step 1 again to ensure the "knownHosts" list now shows clean host/IP entries.

  2. Restart the VCF Backup Configuration workflow from the SDDC Manager UI.

Powered by Wolken Software