Processd not starting processes after reboot on Linux when using Centrify AD Client

Products

• Spectrum
• Network Observability

Issue/Introduction

  Pocessd doesn't seem to start properly after reboot. Upon start up, processd was running, but it didn't seem to start everything up properly. Subsequently, when trying
     to start OneClick, spectroSERVER or ArchMgr, the processes were running via ps output, but when trying to launch the application, they were showing as down/failed.

   To work around this we had to kill all running spectrum processes, kill processd, restart processd, start up the spectrum processes (OneClick, spectroSERVER, ArchMgr).

   The reboots are being executed generally by the "reboot" or "shutdown -r now" commands. 

Environment

Release : 21.2

Component : Spectrum Core / SpectroSERVER

Cause

The spectrum install owner is not a local user but an ldap account. When systemd starts processd after reboot it was starting
    before the ldap client and therefore the Spectrum install owner was not yet valid.


processd_log
-----------------------------------------------------------------------------
Apr 21 19:56:48 Can't start new ticket: NAMINGSERVICE, because the user name is invalid: #####
Apr 21 19:56:48 Can't start new ticket: MYSQL, because the user name is invalid: #####
Apr 21 19:56:48 Can't start new ticket: TELNETD, because the user name is invalid: #####
Apr 21 19:56:48 Can't start new ticket: LOCSERV, because the user name is invalid: #####
Apr 21 19:56:48 Can't start new ticket: NCMSERV, because the user name is invalid: #####

Resolution

 

Spectrum had recently updated the processd_init.sh script which starts processd to check
   for the nss-pam-ldapd client and delay 30secs however in this case the Centrify client is being
   used.

# systemctl list-units --type=service |grep adcl

  centrifydc.service                                        loaded active running CentrifyDC adclient daemon for AD services

To adjust for the Centrify LDAP client, modify the LDAP client check in $SPECROOT/lib/SDPM/processd.init.sh

Change:
   if systemctl list-units --type=service | grep -q 'nslcd'

To:
  if systemctl list-units --type=service | grep -q 'adcl'

Then run:
  systemctl daemon-reload

 

Additional Information

Engineering has agreed to add the check for 'adcl' in a future update. Defect DE534588 has been
raised for this so it can be added to the release backlog