bpvirtdb_srvr shutdown and restart repeatedly

Products

CA Service Management - Service Desk Manager

Issue/Introduction

After upgrade from Service Desk major version or roll up patch, CA Service Desk Manager starts up very slowly. It takes longer time than normal before all processes are connected to domsrvr. Once started up, the bpvirtdb_srvr process is shutdown (it may die due to a crash) and the following error is written to the stdlogs:

bpvirtdb_srvr 10472 ERROR api_msg.c 867 Attempt to get address while not in callback

bpvirtdb_srvr 10472 SEVERE_ERROR miscos.c 222 Signal SIGSEGV received - Exiting!

After the shutting down of the process, it is re-started automatically. Sometimes it is possible to log on through the web interface and to open an existing ticket but more often, once logged on, the Delayed Server Response message is received when you try to perform an action. If you then look in the stdlog the above error is shown following the bpvirtdb_srvr processes shutting down.

Note: The issue may occur for a new installation of CA Service Desk Manager, not just an upgrade.

Environment

CA Service Desk Manager 14.1 and higher

Cause

The system has multiple NIC's and the environment variable NX_PICK_FIRST_IPV4 is either not defined, not set to Yes, or is not in effect due to CA SDM not yet recycled after setting it.

To confirm that the issue is related to multiple NIC, you could follow these steps:

1. Stop the CA Service Desk Manager service completely, ensuring all CA SDM processes are not running. On Windows, you could stop it from Windows Services and then use Task Manager to double-check that all processes (such as sslump.exe, pdm_d_mgr, etc.) are not listed.
2. Start trace for the file named tcp_port.c by running:

pdm_logstat -f tcp_port.c TRACE

3. Start the CA Service Desk Manager service.

4. Check the stdlog for messages in this format, where the value of <ip_address_or_host_name> is not the same:
<date> <time> <server_name> bpvirtdb_srvr <####> TRACE tcp_port.c <pid> Target ip is <ip_address_or_host_name> and addrInfo->ai_family is 2

For example:

07/13 11:58:17.16 [EXAMPLE SDM SERVER] bpvirtdb_srvr 5676 TRACE tcp_port.c 564 Target ip is 192.0.2.1, and addrInfo->ai_family is 2 
07/13 11:58:17.17 [EXAMPLE SDM SERVER] bpvirtdb_srvr 5676 TRACE tcp_port.c 564 Target ip is 192.0.2.2, and addrInfo->ai_family is 2

Note that the IP addresses in this example are random, just for the purpose of demonstration, and to show merely that they are different in the two messages.

5. To turn off the trace, run:

pdm_logstat -f tcp_port.c

Resolution

To resolve the issue in versions 14.1 and 17.1, follow these steps:

Take a backup of the NX.env file
Add PICK_FIRST_IPV4 variable in NX.env file using pdm_options_mgr command to set: @NX_PICK_FIRST_IPV4 = Yes
- pdm_options_mgr -c -s PICK_FIRST_IPV4 -v Yes -a pdm_option.inst
Take a backup of the NX.env_nt.tpl file.
Add PICK_FIRST_IPV4 variable in NX.env_nt.tpl file using pdm_options_mgr command to set: @NX_PICK_FIRST_IPV4 = Yes
- pdm_options_mgr -c -s PICK_FIRST_IPV4 -v Yes -a pdm_option.inst -t
Recycle CA Service Desk Manager services.

To resolve the issue in version in version 17.2 and above, it should only be necessary to install the Options Manager option named NX_PICK_FIRST_IPV4. However, after installing the option, confirm that the statement added to NX.env is: @NX_PICK_FIRST_IPV4 = Yes . If the statement in NX.env is "@NX_NX_PICK_FIRST_IPV4 = Yes", then change it to "@NX_PICK_FIRST_IPV4 = Yes".

Important Notes:

ONLY use this solution after validating that the issue occurs and is due to a multiple NIC environment.

Other Notes:

The NX.env file is located in the base CA Service Desk Manager installation directory (referred to as $NX_ROOT, reached by running the command: nxcd).
The NX.env_nt.tpl file is located in $NX_ROOT\pdmconf

Additional Information

Please see also:

Implementing NX environment variable options in an Advanced Availability (AA) environment.
https://knowledge.broadcom.com/external/article?articleId=29844