When attempting to expand the Vertica cluster to add extra nodes, the update_vertica command fails as follows:
# /opt/vertica/sbin/update_vertica --add-hosts HOST1,HOST2 -u dradmin -l /home/dradmin -L /app/CA/IMDataRepository_vertica9/resources/vlicense.dat --rpm /app/CA/IMDataRepository_vertica9/resources/vertica-9.1.1-5.x86_64.RHEL6.rpm -T -S default --failure-threshold FAIL
>> Validating options...
Mapping hostnames in --add-hosts (-A) to addresses...
LSMCAPMPAPP7 => 10.65.8.214
LSMCAPMPAPP8 => 10.65.8.215
>> Starting installation tasks.
>> Getting system information for cluster (this may take a while)...
Default shell on nodes:
>> Validating software versions (rpm or deb)...
>> Beginning new cluster creation...
successfully backed up admintools.conf on 10.65.8.213
>> Installing software (rpm or deb)...
Installing rpm on 2 hosts....
installing node.... 10.65.8.214
Error: setHost called with empty host list
Traceback (most recent call last):
File "/opt/vertica/oss/python/lib/python2.7/site-packages/vertica/install/__init__.py", line 1467, in run_install
File "/opt/vertica/oss/python/lib/python2.7/site-packages/vertica/install/__init__.py", line 1123, in _installMain
install_RPMs(hostsToUpgradeRPM, options, installerSSH, running_as, get_os_shortname())
File "/opt/vertica/oss/python/lib/python2.7/site-packages/vertica/install/__init__.py", line 766, in install_RPMs
install_error = SSH.installNode( installerSSH, host, options.rpm_file_name, running_as )
File "/opt/vertica/oss/python/lib/python2.7/site-packages/vertica/network/SSH.py", line 428, in installNode
File "/opt/vertica/oss/python/lib/python2.7/site-packages/vertica/network/adapterpool.py", line 555, in copyLocalFileToHosts
File "/opt/vertica/oss/python/lib/python2.7/site-packages/vertica/network/adapterpool.py", line 207, in setHosts
raise ValueError('setHost called with empty host list')
ValueError: setHost called with empty host list
Installation FAILED with errors.
DX NetOps CAPM Release : 21.2
Component : Vertica
To understand what is causing the issue, check in the vertica install.log located under:
For example, the following error was found in case with this issue:
2021-11-08 20:07:58.254 installer/222560:0x7f4072cb0740 [Scp.run] <INFO> Scp Command: scp -q /app/CA/IMDataRepository_vertica9/resources/vertica-9.1.1-5.x86_64.RHEL6.rpm [email protected]:/tmp/vstage-ff6736e5-8502-4c6b-9eba-fc1a4de6b1dc/file
2021-11-08 20:23:27.702 installer/222560:0x7f4072cb0740 [StatusInstallSoftware.printError] <ERROR> Error: setHost called with empty host list
When the error is found, look at the activity directly preceding it. In the above case, the error is raised after an attempt to copy, via scp, the Vertica RPM package from the current, active DR host to the new one. By running the same command (but with -v [verbose] set instead of -q), we can see what is happening directly on the command line.
The above case clearly shows that the network is throttling large copies between the servers. When attempting to copy the vertica.rpm package from one server to the others, it starts off reasonably, but then gets throttled by the network very rapidly until it stalls.