Upgrade of VMware Aria Automation from version 8.16.0 to 8.18.1 fails during cluster rebuild phase due to firewall gaps and DNS resolution timeout

Products

• VCF Automation

Issue/Introduction

When performing a multi-node cluster upgrade of VMware Aria Automation from version 8.16.0 to version 8.18.1, the upgrade process fails during the cluster-rebuild/join phase. The cluster structure becomes fragmented or destroyed, leaving the primary cluster node isolated while secondary cluster nodes fail to join.

The following symptoms are observed in this state:

• The etcd service fails to initialize or remains stopped on secondary nodes (Node 2 and Node 3), while remaining running only on Node 1.

• The vracli cluster join operation fails with an etcd member add timeout error (Exit code: 144).

• Manual port connectivity verification tests targeting ports 2379 and 2380 between cluster members time out:

  timeout 5 bash -c 'echo > /dev/tcp/<NODE_2_IP>/2379' && echo "REACHABLE" || echo "BLOCKED/TIMEOUT

• Hostname query utilities (e.g., getent hosts <FQDN>) introduce a massive processing latency of approximately 15 seconds per execution lookup.

Environment

 

• VMware Aria Automation 8.16.0

• VMware Aria Automation 8.18.1

 

Cause

This failure is caused by two compounding operational root causes:

1. Firewall/Network Layer Omission: The cluster-nodes ipset allowlist configuration on the primary node (Node 1) was missing the secondary nodes' IP addresses. This firewall restriction silently dropped all inter-node cluster database tracking traffic, causing etcd communication to time out.

2. DNS Nameserver Failure: One of the two upstream DNS nameservers configured in the appliance properties was completely non-responsive. Because cluster membership pairing operations utilize hostnames, the resulting 15-second name resolution latency caused internal RPC cluster handshakes to exceed their strict execution deadlines.

Resolution

To recover the deployment state and complete the upgrade successfully, you must fix the underlying firewall mapping and clean up the environment's DNS latency:

1. Revert all cluster nodes to a healthy standalone pre-join snapshot state (such as a snapshot taken before manual cluster-join loops were initiated) to clean out inconsistent configuration files, bad PKI directories, and damaged local databases.

2. Open an SSH connection to the primary node (Node 1) as root and manually update the cluster-nodes ipset rules to add the missing secondary peer addresses:

   ipset del cluster-nodes <NODE_2_IP> 2>/dev/null; ipset add cluster-nodes <NODE_2_IP> timeout 0
   ipset del cluster-nodes <NODE_3_IP> 2>/dev/null; ipset add cluster-nodes <NODE_3_IP> timeout 0
   

3. Verify that both secondary nodes populate correctly within the firewall rule matching list:

   ipset list cluster-nodes
   

4. Do not manually restart the etcd daemon services on the secondary nodes. The internal state machine handler must manage this service transition natively during the join steps.

5. Open the /etc/resolv.conf file on all cluster nodes and delete the non-functional or dead nameserver entries to eliminate query resolution latency.

6. To ensure that your DNS modifications survive container lifecycle operations and are not overwritten by legacy runtime definitions, persist the change globally across the local deployment matrix by running:

   vracli network dns set --local --servers <WORKING_NAMESERVER_IP>
   

7. Resume the automated upgrade state machine engine:

   vracli upgrade exec --resume
   

8. Once the unattended upgrade process completes successfully and all nodes stabilize on version 8.18.1, execute an Inventory Sync from within VMware Aria Suite Lifecycle to align the tracking records.