SSH delays from Aria Suite Lifecycle Manager due to DNS misconfiguration causing Inventory Sync Failures 'LCMVIDMIMPORT0023'

Products

• VCF Operations/Automation (formerly VMware Aria Suite)

Issue/Introduction

• SSH connections initiated from Aria Suite Lifecycle Manager to other VMware appliances (Aria components and vCenter) experienced significant delays.
• Due to this behavior, multiple product inventory synchronization tasks from Aria Suite Lifecycle Manager failed.
• One example observed was the vIDM inventory sync failure with error LCMVIDMIMPORT0023 – Cannot execute ssh commands.
• Running the SSH in verbose (ssh -vvv root@fqdn) confirms that the connection was stuck at resolving the FQDN of the node on port 22

• The issue was verified by manually initiating SSH connections from Aria Suite Lifecycle Manager. SSH access using FQDNs consistently took a long time to establish, while SSH connections using IP addresses worked immediately. Similar delays were also observed with ICMP (ping) requests when using FQDNs, indicating a DNS resolution issue rather than a network connectivity problem.
• DNS-related delays were observable via journalctl -u systemd-resolved.

Environment

Aria Suite Lifecycle Manager 8.18.x

Cause

The issue was caused by outdated DNS server entries configured in the vApp Options of the Aria Suite Lifecycle Manager appliance, which were inconsistent with the DNS configuration at the OS level.

Resolution

The DNS configuration needs to updated to ensure consistency between the OS-level settings and the vApp Options of the Aria Suite Lifecycle Manager appliance. 

Steps:

1. Update DNS servers at the OS level using the command:
   

   /opt/vmware/share/vami/vami_set_dns <DNS_Server_1> <DNS_Server_2>

2. Power off the Aria Suite Lifecycle Manager virtual appliance.
3. Update the vApp Options property
   

   vami.DNS.VMware_vRealize_Suite_Life_Cycle_Manager_Appliance with the correct DNS servers.

4. Power on the appliance.
5. Verify DNS configuration using resolvectl status and confirm successful DNS resolution and SSH connectivity.
   

   resolvectl status

Additional Information

Broadcom Documentation – Change in DNS Server