Connectivity issue due to missing entries in iptables during VMware Identity Manager Installation/Scale-out using Aria Lifecycle Manager

Products

• VMware Aria Suite

Issue/Introduction

This article addresses the connectivity issue caused by missing entries in iptables during VMware Identity Manager installation or scale-out using Aria Lifecycle Manager. During the operation, SSH commands fail, preventing successful communication between Aria Lifecycle Manager and the VMware Identity Manager node.

The following error codes are commonly observed during these failures:

• Error Code: LCMUPGRADEVIDM71212
  "Cannot execute SSH commands, Timeout socket is not established."

• Error Code: LCMVIDM74010
  "Unable to configure slave node due to command execution error" with exception message "Unable to perform pg_basebackup via command."

• Error Code: LCMVIDM74011
  "Unable to configure slave, due to unknown exception on the host" with exception message "Cannot execute SSH commands. Exception encountered: java.net.ConnectException: Connection timed out (Connection timed out)."

• Error Code: LCMVIDM74012
  "Unable to prepare slave node due to command execution." with exception message "Cannot execute SSH commands. Exception encountered: java.net.ConnectException: Connection timed out (Connection timed out)."

During these failures, SSH connections between Aria Lifecycle Manager and VMware Identity Manager are not established, causing the operation to halt.

Environment

VMware Identity Manager 3.3.x
VMware Aria Suite Lifecycle Manager 8.x

Cause

The cause of this issue is a race condition that occurs during the update of iptables rules on the VMware Identity Manager node. During the installation or scale-out process using Aria Lifecycle Manager, the system is expected to configure iptables entries that allow secure SSH and ICMP communication between nodes. However, due to timing conflicts in the execution of these commands, some iptables entries are either not created or improperly updated. This results in missing rules, which blocks incoming and outgoing SSH and ICMP traffic. Consequently, Aria Lifecycle Manager is unable to establish a connection with the affected VMware Identity Manager node, leading to timeouts and the inability to execute necessary commands.

Resolution

This is a known issue, and the following workaround can be applied to resolve it.

Workaround Steps:
Log in to the problematic node:

Note:

• SSH as root user and login to VMware Identity Manager node where the failure occurred. If SSH works, proceed with the below steps.
• Login in as root user to VMware Identity Manager node where the failure occurred using Web Console/Remote Console option in vCenter.

Verify iptables has proper entries for SSH/ICMP ports:

1. To list the iptables INPUT chain entries, run the below command:

iptables -L INPUT

2. From the list verify below entry is present:

For ICMP input port:

ACCEPT icmp -- anywhere anywhere icmp echo-request

Note: If the entry is not present, run the command:

iptables -I INPUT -p icmp -m icmp --icmp-type 0 -j ACCEPT -w 5

For SSH input port:

ACCEPT tcp -- anywhere anywhere tcp dpt:ssh ctstate ESTABLISHED
     or
ACCEPT tcp -- anywhere anywhere tcp dpt:ssh state NEW,ESTABLISHED

Note: If both the entries are not present, run the command:

iptables -I INPUT -p tcp --dport 22 -m state --state NEW,ESTABLISHED -j ACCEPT -w 5

3. To list the iptables OUTPUT chain entries, run the command:

iptables -L OUTPUT

4. From the list verify below entry is present:

For ICMP output port:

ACCEPT icmp -- anywhere anywhere icmp echo-reply

Note: If the entry is not present, run the command:
 

iptables -I OUTPUT -p icmp -m icmp --icmp-type 8 -j ACCEPT -w 5

For SSH output port:

ACCEPT tcp -- anywhere anywhere tcp spt:ssh ctstate ESTABLISHED
     or
ACCEPT tcp -- anywhere anywhere tcp spt:ssh state NEW,ESTABLISHED

Note: If the both the entries are not found in the list, run the command:

iptables -I OUTPUT -p tcp --sport 22 -m state --state NEW,ESTABLISHED -j ACCEPT -w 5