Patching from "vCenter Server 7.0 Update 3g" to "vCenter Server 7.0 Update 3r" fails with Error "A dependency job for rsyslog.service failed"

Products

VMware vCenter Server

Issue/Introduction

Installation Progress stuck at 0%

PatchRunner.log

2024-07-30 06:13:17,618.618Z rsyslog:Patch ERROR rsyslog Failed to restart Rsyslog after applying the patches. rc = 1, stdout: , stderr: A dependency job for rsyslog.service failed. See 'journalctl -xe' for details.
2024-07-30 06:13:17,618.618Z rsyslog:Patch ERROR vmware_b2b.patching.executor.hook_executor Patch hook 'rsyslog:Patch' failed.
Traceback (most recent call last):
File "/storage/seat/software-updaterg4al6__/stage/scripts/patches/py/vmware_b2b/patching/executor/hook_executor.py", line 74, in executeHook
executionResult = systemExtension(args)
File "/storage/seat/software-updaterg4al6__/stage/scripts/patches/libs/sdk/extensions.py", line 106, in __call__
result = self.extension(*args)
File "/storage/seat/software-updaterg4al6__/stage/scripts/patches/libs/sdk/extensions.py", line 123, in _func
return func(*args)
File "/storage/seat/software-updaterg4al6__/stage/scripts/patches/payload/components-script/rsyslog/__init__.py", line 154, in patch
restartRsyslogService()
File "/storage/seat/software-updaterg4al6__/stage/scripts/patches/payload/components-script/rsyslog/__init__.py", line 99, in restartRsyslogService
raise ValueError("Failed during Rsyslog patching.")
ValueError: Failed during Rsyslog patching.
2024-07-30T06:13:17.634Z ERROR vmware_b2b.patching.phases.patcher Patch hook Patch got ComponentWrapperError.
Traceback (most recent call last):
File "/storage/seat/software-updaterg4al6__/stage/scripts/patches/py/vmware_b2b/patching/phases/patcher.py", line 203, in patch
_patchComponents(ctx, userData, statusAggregator.reportingQueue)
File "/storage/seat/software-updaterg4al6__/stage/scripts/patches/py/vmware_b2b/patching/phases/patcher.py", line 85, in _patchComponents
executeComponentHook(Hook.Patch, ctx, c, userData, reportingQueue)
File "/storage/seat/software-updaterg4al6__/stage/scripts/patches/py/vmware_b2b/patching/executor/execution_facade.py", line 98, in executeComponentHook
reportQueue, identifier, expectedResultType)
File "/storage/seat/software-updaterg4al6__/stage/scripts/patches/py/vmware_b2b/patching/executor/execution_facade.py", line 53, in executeHook
result = executor.executeHook(scriptFile, hook, args, reportQueue, reportIdentifier)
File "/storage/seat/software-updaterg4al6__/stage/scripts/patches/py/vmware_b2b/patching/executor/hook_executor_process.py", line 119, in executeHook
raise ex
patch_errors.ComponentError
2024-07-30T06:13:18.638Z WARNING root stopping status aggregation...
2024-07-30T06:13:18.640Z ERROR __main__ Patch vCSA failed

The rsyslogd-syslog.log will either reflect empty or report write errors

rsyslogd-syslog.log

2024-07-30T09:20:01.905944+00:00 err rsyslogd file '/var/log/vmware/rsyslogd/rsyslogd-syslog.log'[14] write error - see https://www.rsyslog.com/solving-rsyslog-write-errors/ for help OS error: No space left on device [v8.2306.0 try https://www.rsyslog.com/e/2027 ]
2024-07-30T09:20:01.905952+00:00 err rsyslogd action 'action-4-builtin:omfile' (module 'builtin:omfile') message lost, could not be processed. Check for additional error messages before this one. [v8.2306.0 try https://www.rsyslog.com/e/2027 ]
2024-07-30T09:20:01.905957+00:00 err rsyslogd file '/var/log/vmware/rsyslogd/rsyslogd-syslog.log'[14] write error - see https://www.rsyslog.com/solving-rsyslog-write-errors/ for help OS error: No space left on device [v8.2306.0 try https://www.rsyslog.com/e/2027 ]
2024-07-30T09:20:01.905960+00:00 err rsyslogd action 'action-4-builtin:omfile' (module 'builtin:omfile') message lost, could not be processed. Check for additional error messages before this one. [v8.2306.0 try https://www.rsyslog.com/e/2027 ]
2024-07-30T09:20:01.905963+00:00 err rsyslogd file '/var/log/vmware/rsyslogd/rsyslogd-syslog.log'[14] write error - see https://www.rsyslog.com/solving-rsyslog-write-errors/ for help OS error: No space left on device [v8.2306.0 try https://www.rsyslog.com/e/2027 ]

rsyslogd.pid file missing and there is no Service ID

Run command in Issue vCenter----> less /var/run/rsyslogd.pid
- rsyslogd.pid file is missing.

Environment

vCenter 7.x

Cause

rsyslogd.pid file missing or does not reflect the Service ID

Resolution

Fix the rsyslog Issue by Running the Script- syslogConfigFix.sh script

Note: Take vCenter VM Snapshot prior to making any changes

1. Download the Script:
  - Access the KB : https://knowledge.broadcom.com/external/article/317886/rsyslog-stops-logging-to-vmdird-and-mess.html
  - Download the syslogConfigFix.sh script (note: the exact name might vary, so ensure you use the correct filename).
2. Transfer the Script to vCenter:
  - Use WinSCP or a similar tool to transfer the script to your vCenter server.
3. Make the Script Executable:
  - Connect to the vCenter server via SSH or another terminal.
  - Run the following command to make the script executable:
    - chmod +x /path/to/syslogConfigFix.sh
4. Execute the Script:
  - Run the script using
    - ./path/to/syslogConfigFix.sh
  - Ensure you provide the correct path to the script if it was saved in a different location.
5. Restart the rsyslog Service:
  - systemctl restart rsyslog
6. Verify the rsyslogd.pid File:
  - Check that the rsyslogd.pid file is created and contains the correct details by running:
    - less /var/run/rsyslogd.pid
    - Verify that the PID file reflects the service ID correctly (Example below)
7. Proceed with Upgrade/Patch:
  - - Once the rsyslogd.pid file contains valid details, follow your standard procedures for upgrading or patching the vCenter.