CAPM Vertica cluster expansion from 1 to 3 nodes failing with "Error: setHost called with empty host list"

Products

• CA Performance Management - Usage and Administration
• DX NetOps

Issue/Introduction

When attempting to expand the Vertica cluster to add extra nodes, the update_vertica command fails as follows:

# /opt/vertica/sbin/update_vertica --add-hosts HOST1,HOST2 -u dradmin -l /home/dradmin -L /app/CA/IMDataRepository_vertica9/resources/vlicense.dat --rpm /app/CA/IMDataRepository_vertica9/resources/vertica-9.1.1-5.x86_64.RHEL6.rpm -T -S default --failure-threshold FAIL

>> Validating options...

Mapping hostnames in --add-hosts (-A) to addresses...

        LSMCAPMPAPP7                   => 10.65.8.214

        LSMCAPMPAPP8                   => 10.65.8.215

 

>> Starting installation tasks.

>> Getting system information for cluster (this may take a while)...

 

Default shell on nodes:

10.65.8.213 /bin/bash

10.65.8.214 /bin/bash

10.65.8.215 /bin/bash

 

>> Validating software versions (rpm or deb)...

>> Beginning new cluster creation...

successfully backed up admintools.conf on 10.65.8.213

>> Installing software (rpm or deb)...

 

Installing rpm on 2 hosts....

installing node....  10.65.8.214

Error: setHost called with empty host list

Traceback (most recent call last):

  File "/opt/vertica/oss/python/lib/python2.7/site-packages/vertica/install/__init__.py", line 1467, in run_install

    _installMain(options)

  File "/opt/vertica/oss/python/lib/python2.7/site-packages/vertica/install/__init__.py", line 1123, in _installMain

    install_RPMs(hostsToUpgradeRPM, options, installerSSH, running_as, get_os_shortname())

  File "/opt/vertica/oss/python/lib/python2.7/site-packages/vertica/install/__init__.py", line 766, in install_RPMs

    install_error = SSH.installNode( installerSSH, host, options.rpm_file_name, running_as )

  File "/opt/vertica/oss/python/lib/python2.7/site-packages/vertica/network/SSH.py", line 428, in installNode

    timeout=rpm_timeout)

  File "/opt/vertica/oss/python/lib/python2.7/site-packages/vertica/network/adapterpool.py", line 555, in copyLocalFileToHosts

    with apool.setHosts(dest_hosts):

  File "/opt/vertica/oss/python/lib/python2.7/site-packages/vertica/network/adapterpool.py", line 207, in setHosts

    raise ValueError('setHost called with empty host list')

ValueError: setHost called with empty host list

Installation FAILED with errors.

Environment

DX NetOps CAPM Release : 21.2
Component : Vertica

Resolution

To understand what is causing the issue, check in the vertica install.log located under:

/opt/vertica/log

For example, the following error was found in case with this issue:

2021-11-08 20:07:58.254 installer/222560:0x7f4072cb0740 [Scp.run] <INFO> Scp Command: scp -q /app/CA/IMDataRepository_vertica9/resources/vertica-9.1.1-5.x86_64.RHEL6.rpm [email protected]:/tmp/vstage-ff6736e5-8502-4c6b-9eba-fc1a4de6b1dc/file

2021-11-08 20:23:27.702 installer/222560:0x7f4072cb0740 [StatusInstallSoftware.printError] <ERROR> Error: setHost called with empty host list

When the error is found, look at the activity directly preceding it. In the above case, the error is raised after an attempt to copy, via scp, the Vertica RPM package from the current, active DR host to the new one. By running the same command (but with -v [verbose] set instead of -q), we can see what  is happening directly on the command line. 

The above case clearly shows that the network is throttling large copies between the servers. When attempting to copy the vertica.rpm package from one server to the others, it starts off reasonably, but then gets throttled by the network very rapidly until it stalls.

Consequently the Vertica update script fails it due to timeout. So in this case, it is a network issue that must be fixed to prevent data throttling (possibly due to MTU setting) between the servers.