vCenter Server: Logrotate Fails with “/var/lib/logrotate.status is already locked”

Products

VMware vCenter Server VMware vCenter Server 8.0

Issue/Introduction

In VMware vCenter Server 7.x and 8.x, you may observe alerting in internal or third-party monitoring tools (such as Splunk) with the following error message:

UX-LOG(error: state file /var/lib/logrotate.status is already locked)
Additionally, the logrotate service status may show as failed when checked via the vCenter shell:# systemctl status logrotate

logrotate.service - Rotate log files

Loaded: loaded (/usr/lib/systemd/system/logrotate.service; static)

Active: failed (Result: exit-code)

TriggeredBy: ● logrotate.timer

...

logrotate[xxxxx]: error: state file /var/lib/logrotate.status is already locked

logrotate[xxxxx]: logrotate does not support parallel execution on the same set of logfiles.
Log file review may also show repeated failures:zcat /var/log/messages.1.gz | grep -i logrotate

logrotate[xxxxx]: error: state file /var/lib/logrotate.status is already locked

logrotate[xxxxx]: logrotate does not support parallel execution on the same set of logfiles.

This issue typically affects log rotation for the vsphere-ui service, including files like threadmonitor.log, catalina,log, vsphere_client_virgo.log, and Tomcat access logs - localhost_access)log.txt etc

Eg:

root@[vcsa_appliance] [ /var/log/vmware ]# du -h --max-depth=1 | sort -rh
23G .
20G ./vsphere-ui
733M ./vpxd
288M ./wcp
250M ./eam
198M ./sso
163M ./vapi
118M ./vpxd-svcs
108M ./vmware-updatemgr

root@[vcsa_appliance] [ /var/log/vmware/vsphere-ui/logs ]# du -h * | sort -rh
20G catalina.log
605M access
69M localhost.log
41M apigw.log
33M vsphere_client_virgo.log
32M opId.log
21M changelog.log
17M threadmonitor.log
14M vspheremessaging.log
3.0M opId_0.log.zip

Environment

VMware vCenter Server 7.x

VMware vCenter Server 8.x

Cause

This issue is caused by misconfigured directory paths in the /usr/lib/vmware-vsr/vcsa_service_registry.yaml file. Specifically, the incorrect paths prevent logrotate from accessing the target log files, causing the process to fail and report the file lock error.

Affected YAML Sections:

Incorrect: /var/log/vmware/vsphere-ui/log

Correct: /var/log/vmware/vsphere-ui/logs

When logrotate attempts to rotate logs that cannot be found due to path mismatches, the rotation process hangs or fails, and a lock file is left behind.

Sample:

- directory: /var/log/vmware/vsphere-ui/log/access     ===> Incorrect
    files:
    - documentation: The vsphere-ui tomcat access log
      pattern:
      - localhost_access_log.txt
      rotation_config:
        compress: enabled
        create: 600 vsphere-ui users
        dateext: true
        dateformat: -%Y-%m-%d_%H:%M:%S
        missingok: true
        notifempty: true
        rotate: 50
        size: 10MB
      rotation_config_path: /etc/logrotate.d/vmware-vsphere-ui.lr
      rotation_mechanism: log_rotate
      usage: logging
.
.
.
  - directory: /var/log/vmware/vsphere-ui/log    ===> Incorrect
    files:
    - documentation: The main vsphere ui log
      pattern:
      - vsphere_client_virgo.log
      rotation_config:
        compress: enabled
        rotate: 10
        size: 50MB
      rotation_mechanism: internal
      usage: logging

Resolution

NOTE: Always take a snapshot of the vCenter Server Appliance before making changes.

Truncate the files that has grown bigger in size under /var/log/vmware/vsphere-ui/logs and /var/log/vmware/vsphere-ui/logs/access to prevent space or size-related issues::

dd if=/dev/zero bs=1G seek=1 count=0 of=<file_name>
Backup the existing YAML configuration:cp /usr/lib/vmware-vsr/vcsa_service_registry.yaml /var/tmp/
Edit the YAML file using a text editor like vi:

vi /usr/lib/vmware-vsr/vcsa_service_registry.yamlUpdate the following lines:

Change path from - directory: /var/log/vmware/vsphere-ui/log/access to - directory: /var/log/vmware/vsphere-ui/logs/accessChange path from - directory: /var/log/vmware/vsphere-ui/log to /var/log/vmware/vsphere-ui/logs
Restart logrotate service:

systemctl restart logrotate.service
Restart vCenter services:

service-control --stop --allservice-control --start --all

Additional Information

Correcting the log directory paths ensures that logrotate can properly access and manage the defined log files. After restarting the services, log rotation resumes normal behavior and the locking issue is resolved.

Post-resolution, the logrotate service will return to an active status, and logs like threadmonitor.log will rotate as expected.