Customer discovered systemd-sysv-generator helps systemd run sysV init scripts. Knowing this, he found these errors:
<hostName>:~ # dmesg | grep generator
[ 1.529901] systemd-sysv-generator[551]: stat() failed on /etc/rc.d/init.d/nfa_mysqlCSE: No such file or directory
[ 1.530548] systemd-sysv-generator[551]: stat() failed on /etc/rc.d/init.d/nfa_harvester: No such file or directory
[ 1.534633] systemd-sysv-generator[551]: stat() failed on /etc/rc.d/init.d/nfa_dataretention: No such file or directory
[ 1.534662] systemd-sysv-generator[551]: stat() failed on /etc/rc.d/init.d/nfa_poller: No such file or directory
[ 1.535557] systemd-sysv-generator[551]: stat() failed on /etc/rc.d/init.d/nfa_collpollws: No such file or directory
[ 1.535779] systemd-sysv-generator[551]: stat() failed on /etc/rc.d/init.d/nfa_proxies: No such file or directory
[ 1.535808] systemd-sysv-generator[551]: stat() failed on /etc/rc.d/init.d/nfa_filewebservice: No such file or directory
[ 1.535931] systemd-sysv-generator[551]: stat() failed on /etc/rc.d/init.d/nfa_reaper: No such file or directory
Note the init scripts are symlinks:
<hostName>:~ # ls -l /etc/rc.d/init.d
total 48
-rw-r--r-- 1 root root 1160 Nov 29 2016 README
-rw-r--r-- 1 root root 15131 Sep 12 2016 functions
-rwxrwxr-x 1 root root 10576 Mar 7 17:41 mysql
-rwxr-xr-x 1 root root 2989 Sep 12 2016 netconsole
-rwxr-xr-x 1 root root 6643 Sep 12 2016 network
lrwxrwxrwx 1 root root 41 Mar 7 17:44 nfa_collpollws -> /opt/CA/NFA/Netflow/bin/config_collpollws
lrwxrwxrwx 1 root root 44 Mar 7 17:44 nfa_dataretention -> /opt/CA/NFA/Netflow/bin/config_dataretention
lrwxrwxrwx 1 root root 45 Mar 7 17:44 nfa_filewebservice -> /opt/CA/NFA/Netflow/bin/config_filewebservice
lrwxrwxrwx 1 root root 40 Mar 7 17:44 nfa_harvester -> /opt/CA/NFA/Netflow/bin/config_harvester
lrwxrwxrwx 1 root root 39 Mar 7 17:42 nfa_mysqlCSE -> /opt/CA/NFA/Netflow/bin/config_mysqlCSE
lrwxrwxrwx 1 root root 37 Mar 7 17:44 nfa_poller -> /opt/CA/NFA/Netflow/bin/config_poller
lrwxrwxrwx 1 root root 38 Mar 7 17:44 nfa_proxies -> /opt/CA/NFA/Netflow/bin/config_proxies
lrwxrwxrwx 1 root root 40 Mar 7 17:44 nfa_reaper -> /opt/CA/NFA/Netflow/Reaper/config_reaper
-rwxr-xr-x 1 root root 2437 Jun 26 2015 rhnsd
The errors in dmesg suggested the symlink destination didn't exist -- which of course could happen during boot before the destination filesystem is mounted.
This seems to be by design per https://access.redhat.com/errata/RHBA-2016:0199: "Previously, systemd ignored symbolic links in the /etc/rc.d/init.d/ directory.
As a consequence, init scripts with symbolic links included in /etc/rc.d/init.d/ could not be started using systemd.
With this update, systemd follows symbolic links included in /etc/rc.d/init.d/ to init scripts.
Note that the real location of the init script must be on the partition that is mounted in the initial ramdisk (initrd). (BZ#1288005)"
Tried replacing the symlinks with copies and this worked:
cd /etc/rc.d/init.d
rm nfa_mysqlCSE
rm nfa_collpollws
rm nfa_dataretention
rm nfa_filewebservice
rm nfa_harvester
rm nfa_poller
rm nfa_proxies
rm nfa_reaper
cp -p /opt/CA/NFA/Netflow/bin/config_collpollws ./nfa_collpollws
cp -p /opt/CA/NFA/Netflow/bin/config_dataretention ./nfa_dataretention
cp -p /opt/CA/NFA/Netflow/bin/config_filewebservice ./nfa_filewebservice
cp -p /opt/CA/NFA/Netflow/bin/config_harvester ./nfa_harvester
cp -p /opt/CA/NFA/Netflow/bin/config_poller ./nfa_poller
cp -p /opt/CA/NFA/Netflow/bin/config_proxies ./nfa_proxies
cp -p /opt/CA/NFA/Netflow/Reaper/config_reaper ./nfa_reaper
If you are on older versions of NFA wherre MySQL did not have a wrapper run:
cp -p /opt/CA/NFA/Netflow/bin/config_mysqlCSE ./nfa_mysqlCSE
If you are on newer versions of NFA wherre MySQL did not have a wrapper run:
cp -p /opt/CA/NFA/Netflow/bin/config_mysqlCSEwrapper ./nfa_mysqlCSE
/sbin/chkconfig --add nfa_mysqlCSE
/sbin/chkconfig --add nfa_collpollws
/sbin/chkconfig --add nfa_dataretention
/sbin/chkconfig --add nfa_filewebservice
/sbin/chkconfig --add nfa_harvester
/sbin/chkconfig --add nfa_poller
/sbin/chkconfig --add nfa_proxies
/sbin/chkconfig --add nfa_reaper
After a reboot, the services had started.
Whether you stick with this backward compatibility feature or implement actual systemd unit files,
you will probably see this problem unless the person installing happens to pick a filesystem that is mounted in initrd.