Symptoms:
root@nsx-controller [ / ]# df -h
Filesystem Size Used Avail Use% Mounted on
devtmpfs 2.0G 0 2.0G 0% /dev
tmpfs 2.0G 0 2.0G 0% /dev/shm
tmpfs 2.0G 15M 2.0G 1% /run
tmpfs 2.0G 0 2.0G 0% /sys/fs/cgroup
/dev/sda2 3.9G 2.2G 1.5G 61% /
/dev/sda1 976M 41M 868M 5% /boot
tmpfs 2.0G 1.4M 2.0G 1% /tmp
/dev/sda3 3.9G 8.0M 3.6G 1% /os_bak
/dev/sda5 4.8G 4.8G 0 100% /var/log
/dev/sda6 2.0G 3.1M 1.8G 1% /config
/dev/sda4 3.9G 130M 3.5G 4% /var/cloudnet/data
/dev/sda7 4.8G 10M 4.6G 1% /image
tmpfs 396M 0 396M 0% /run/user/998
tmpfs 396M 0 396M 0% /run/user/0
root@nsx-controller [ ~ ]# /usr/bin/systemctl restart rsyslog.service
Failed to restart rsyslog.service: Activation of org.freedesktop.systemd1 timed out
root@nsx-controller [ /var/log ]# service rsyslog status
Failed to get properties: Activation of org.freedesktop.systemd1 timed out
Sep 27 03:31:01 nsx-controller run-parts[28848][28904]: (/etc/cron.hourly) finished 0anacron
Sep 27 03:31:01 nsx-controller run-parts[28850][28947]: (/etc/cron.minutes) finished logrotate
Sep 27 03:31:02 nsx-controller systemd[1]: nvp-cli-merge-logs.service: Failed to fork: Cannot allocate memory
Sep 27 03:31:02 nsx-controller systemd[1]: Assertion 'pid >= 1' failed at src/core/unit.c:2026, function unit_watch_pid(). Aborting.
Sep 27 03:31:02 nsx-controller systemd[1]: Caught <ABRT>, cannot fork for core dump: Cannot allocate memory.
Sep 27 03:31:02 nsx-controller systemd[1]: Freezing execution.
This issue occurs when systemd is in a hung state. When systemd fails and hangs, none of the services can be restarted as seen in /var/log/syslog.1:
Sep 27 03:32:26 nsx-controller CROND[29247]: (root) CMDOUT (Failed to kill unit rsyslog.service: Connection timed out) Sep 27 03:32:26 nsx-controller dbus[1385]: [system] Failed to activate service 'org.freedesktop.systemd1': timed out
Sep 27 03:32:26 nsx-controller systemd-logind[1416]: Failed to start session scope session-c124197.scope: Activation of org.freedesktop.systemd1 timed out
Sep 27 03:32:26 nsx-controller CROND[29247]: (root) CMDOUT (error: error running non-shared postrotate script for /var/log/syslog of '/var/log/syslog) Sep 27 03:32:26 nsx-controller CROND[29247]: (root) CMDOUT (')
This issue is resolved in NSX 6.4.10.
Workaround:
A reboot of the controller can be used as a temporary workaround.