vRSLCM Inventory sync for vRLI fails with error: Failed to insert vRLI

Products

• VMware SDDC Manager

Issue/Introduction

vRSLCM Inventory sync for vRLI fails with error: Failed to register xxxxxx-xxxxx-xxxx-xxxx-xxxxxx:vrli in the SDDC Manager

/var/log/vmware/vcf/domainmanager/domainmanager.log:

YYYY-MM-DDTHH:MM:SS DEBUG [vcf_dm,735343b692ebb0cd,12c3] [c.v.v.s.c.s.SecurityConfigurationServiceImpl,http-nio-127.0.0.1-7200-exec-10]  Security config retrieved {"certificateValidationEnabled":true,"fipsMode":false}
YYYY-MM-DDTHH:MM:SS DEBUG [vcf_dm,735343b692ebb0cd,12c3] [c.v.evo.sddc.common.util.SshUtil,http-nio-127.0.0.1-7200-exec-10]  The command [ ssh-keyscan -t rsa <node-name-vrli> ] executed on <hostname>. Status: 1, Timed out: false
Output:
Error: <node-name-vrli>: Connection closed by remote host
YYYY-MM-DDTHH:MM:SS INFO  [vcf_dm,735343b692ebb0cd,12c3] [c.v.e.s.v.VrealizeKnownHostsService,http-nio-127.0.0.1-7200-exec-10]  Succesfully got host spec for host <node-name-vrli>
YYYY-MM-DDTHH:MM:SS ERROR [vcf_dm,735343b692ebb0cd,12c3] [c.v.e.s.v.VrealizeKnownHostsService,http-nio-127.0.0.1-7200-exec-10]  Cannot obtain ssh key
YYYY-MM-DDTHH:MM:SS ERROR [vcf_dm,735343b692ebb0cd,12c3] [c.v.v.v.c.impl.VrliManagerImpl,http-nio-127.0.0.1-7200-exec-10]  Failed to insert vRLI
com.vmware.evo.sddc.orchestrator.exceptions.OrchTaskException: A problem has occurred on the server. Please retry or contact the service provider and provide the reference token.
        at com.vmware.evo.sddc.vrealize.VrealizeKnownHostsService.toSshKnownHostSpec(VrealizeKnownHostsService.java:109)

Environment

vRLI : Version 8.14
LCM : 8.18.0.0 
SDDC : 4.5.2.0

Cause

SSHD service issue on the vRLI node.

ssh-keyscan -t rsa will fail for vRLI node.
Output:
Error: <node-name-vrli>: Connection closed by remote host

Resolution

• Take SSH to vRLI nodes.
• Run ssh-keyscan -t rsa <vRLI-node-hostname> on all the nodes and check if it is running fine.
• You may get the error on one of the node which is causing sync failure.
• Restart the SSHD service : systemctl restart sshd
• Reboot the node in case service restart doesn't fix the issue.