vRealize Network Insight 5.1.0 cluster upgrade hangs on "Foundation DB upgrade..."

Products

• VMware Aria Operations for Networks

Issue/Introduction

Symptoms:

• vRealize Network Insight is a clustered deployment in a dark, or white listed site.
• The vRealize Network Insight 5.1 upgrade hands on "Foundation DB upgrade...".
• The /home/ubuntu/logs/centralised_upgrade.log file on Platform 1 shows errors similar to:

23-01-2020:08:39:50,30 ERROR    [configure_fdb_cluster.py:32] Fail to configure fdb cluster Traceback (most recent call last):  File "/home/ubuntu/build-target/infra-base/ha_cluster/configure_fdb_cluster.py", line 30, in <module>
    fdb_cluster_helper.configure_fdb(json.loads(args.connection), json.loads(args.nodes), None)
  File "/home/ubuntu/build-target/infra-base/ha_cluster/fdb_cluster_helper.py", line 79, in configure_fdb
    _create_fdb_cluster(connection, all_nodes, update_state_in_db)
  File "/home/ubuntu/build-target/infra-base/ha_cluster/fdb_cluster_helper.py", line 29, in _create_fdb_cluster
    utils.exec_cmd("sudo /usr/lib/foundationdb/make_public.py")
  File "/home/ubuntu/build-target/infra-base/ha_cluster/utils.py", line 37, in exec_cmd
    return exec_ignorable(cmd, False, num_retries)
  File "/home/ubuntu/build-target/infra-base/ha_cluster/utils.py", line 93, in exec_ignorable
    raise Exception("Failed executing command errcode[%d] command:%s" % (code, cmd))
Exception: Failed executing command errcode[256] command:sudo /usr/lib/foundationdb/make_public.py
.
23-01-2020:08:39:50,41 INFO     [fdbpoststep.py:23] RESULT 1
23-01-2020:08:39:50,42 INFO     [fdbpoststep.py:24] active
ERROR: Could not determine an address
ERROR: Could not determine an address

Environment

VMware vRealize Network Insight 5.x
VMware vRealize Network Insight 5.1.x

Cause

This issue is caused when an upgrade script is not able to resolve www.foundationdb.org on cluster setups.

This issue can also affect single node setups when they're being converted to clusters.

Resolution

To resolve this issue, add an entry in /etc/hosts to make www.foundationdb.org resolve to localhost.

1. Log into the Platform 1 VM as the support user via SSH or Console.
2. Run the following command, replacing the variable text noted, to add the www.foundationdb.org entry to /etc/hosts:

sudo bash -c 'echo "Platform1_IP    www.foundationdb.org" >> /etc/hosts'

Note: Replace Platform1_IP with the actual IP address of your Platform 1 VM.

Example: sudo bash -c 'echo "192.168.50.10    www.foundationdb.org" >> /etc/hosts'

3. Run the following command to verify that the changes made in step 2 are visible:

cat /etc/hosts

Example output:
192.168.50.20    platform2
192.168.50.30    platform3
192.168.50.10    platform1
192.168.50.10    www.foundationdb.org

The upgrade will proceed automatically.

Note:
1. The entry added in step 2 can be removed after the upgrade is complete.

 2.  The entry for www.foundationdb.org in /etc/hosts should be present after the entries for platform1, platform2 and platform3.
During cluster creation from a single node this can be achieved by executing the Step 2 after cluster has crossed the “Configure Firewalls” step as shown in the figure below