After patching, Vertica is unable to start

Products

• Network Observability
• CA Performance Management

Issue/Introduction

After patching the kernel from 4.18.0-553.42.1.e18_10.x86_64 to 4.18.0-553.45.1.e18_10.x86_64, we can no longer start the Data Repository Vertica database

Messages seen when attempting to start the database:

STATUS: vertica.engine.api.compute_vdatabase is still running on X hosts <ip address>, <ip address as of <date> <time>. See /opt/vertica/log/admintools.log for full details

Could not find acknowledgement of remote call

...........

From adminTools.log

 

2025-04-30 04:52:58.241 admintools/5973:0x7fed08eaa700 [SSHConnection._send_command] <ERROR> Tried to find an acknowledgement prompt for command [/opt/vertica/oss/python3/bin/python3 -m vertica.engine.api.at_runner --module=vertica.engine.api.vertica_process_is_up --semver=1.0.0  --debug-hosts= --debug-module=], but instead received: EOF

………………..

 

 File "/opt/vertica/oss/python3/lib/python3.11/site-packages/vertica/network/adapters/pexpect_adapter.py", line 960, in _send_command

    raise ATPexpectError("Could not find acknowledgement of"

vertica.engine.api.errors.ATPexpectError: Could not find acknowledgement of remote call.

Could not make directory on host <IP>. Exception was: <ATResult> status=Error host=<IP> version=Exception/100.0.0 error_type=<class 'vertica.engine.api.errors.ATPexpectError'> error_message=Could not find acknowledgement of remote call.

 

        Stack Trace from Initiator follows

Traceback (most recent call last):

  File "/opt/vertica/oss/python3/lib/python3.11/site-packages/vertica/network/adapters/pexpect_adapter.py", line 860, in exec_module

    result = self._exec_module_impl(command, timeout)

             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

  File "/opt/vertica/oss/python3/lib/python3.11/site-packages/vertica/network/adapters/pexpect_adapter.py", line 885, in _exec_module_impl

    pexpect_result = self._send_command(command.to_python_invocation(),

                    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

  File "/opt/vertica/oss/python3/lib/python3.11/site-packages/vertica/network/adapters/pexpect_adapter.py", line 960, in _send_command

    raise ATPexpectError("Could not find acknowledgement of"

vertica.engine.api.errors.ATPexpectError: Could not find acknowledgement of remote call.

 

Environment

DX NetOps 23.3.13 
Vertica 23.4.0-9.

3 node cluster

Physical nodes
RHEL 8.10

Cause

Functionality within 4.18.0-553.45.1.e18_10.x86_64 is inhibiting the capability to communicate to/from the cluster members

Resolution

In this situation, it was chosen to revert back to the old kernel. 

To do so the Vertica nodes were stopped and the old kernel, which was still available to the systems, was chosen to load from.