File '/storage/log/vmware-updatemgr/updatemgr-vmon.log.stderr' is very large causing vCenter services to not start

Products

• VMware vCenter Server

Issue/Introduction

When looking at the vCenter services,

service-control --status --all

services like "vmware-vpxd" is not running due to /storage/log being above 95% full.

 

The /storage/log directory is full or nearly full when running the command

df -h

Similar content may be in the file:

Starting service process with pid: ####.
INFO:ComponentScanner:Effective components: ['Intel-NVMe-Vol-Mgmt-Dev-Plugin_2.7.2173-2vmw.803.0.0.24022510', 'VMware-SDHCI-Driver_1.0.3-3vmw.803.0.0.24022510', 'Broadcom-bcm-mpi3_8.8.1.0.0.0-1vmw.803.0.0.24022510', 'Pensando-ionic-en_20.0.0-56vmw.803.0.0.24022510', 'ESXi_8.0.3-0.0.24022510', 'Mellanox-nmlx5_4.23.6.2-7vmw.803.0.0.24022510', 'Micron-mtip32xx-native_3.9.8-1vmw.803.0.0.24022510', 'MRVL-E3-Ethernet-iSCSI-FCoE_3.0.230.0-1OEM.700.1.0.15843807', 'Broadcom-lsi-mr3_7.728.02.00-1vmw.803.0.0.24022510', 'VMware-NVMeoF-RDMA_1.0.3.9-1vmw.803.0.0.24022510', 'VMware-Host-Client_2.18.0-23593406', 'V
Mware-iser_1.1.0.2-1vmw.803.0.0.24022510', 'Intel-icen_1.12.5.0-1OEM.800.1.0.20613240', 'VMware-nvme-pcie-plugin_1.0.0-1vmw.803.0.0.24022510', 'VMware-vmkusb_0.1-22vmw.803.0.0.24022510', 'dell-osname-idrac-component_8.0.0-A02', 'dell-fac-dcui-component_8.0.0-A05', 'Broadcom-lsi-msgpt2_20.00.06.00-4vmw.803.0.0.24022510', 'VMware-oem-dell-plugin_1.1.0-2vmw.803.0.0.24022510', 'Intel-irdman_1.4.4.0-1OEM.800.1.0.20143090', 'VMware-dwi2c_0.1-7vmw.803.0.0.24022510', 'VMware-cndi-drivers_1.2.10.0-1vmw.803.0.0.24022510', 'Broadcom-lsi-msgpt3_17.00.13.00-3vmw.803.0.0.24022510', 'VMware-nvmxnet3_2.0.0.31-12vmw
.803.0.0.24022510', 'nipmi_1.0-1vmw.803.0.0.24022510', 'VMware-ahci_2.0.17-1vmw.803.0.0.24022510', 'Intel-ne1000_0.9.2-1vmw.803.0.0.24022510', 'Broadcom-ELX-brcmfcoe_12.0.1500.3-4vmw.803.0.0.24022510', 'VMware-intelgpio_0.1-1vmw.803.0.0.24022510', 'Intel-ixgben_1.15.1.0-1OEM.800.1.0.20613240', 'MRVL-Atlantic-Driver-Bundle_1.0.3.0-13vmw.803.0.0.24022510', 'Microchip-smartpqi_80.4700.0.5000-2vmw.803.0.0.24022510', 'Broadcom-elxnet_12.0.1250.0-8vmw.803.0.0.24022510', 'dell-fist-util-Bundle_5.2.3-2023.03.20', 'Broadcom-lpnic_11.4.62.0-1vmw.803.0.0.24022510', 'Solarflare-NIC_2.4.0.2010-18vmw.803.0.0.2402
2510', 'Intel-SCU-rste_2.0.2.0088-7vmw.803.0.0.24022510', 'VMware-oem-lenovo-plugin_1.0.0-2vmw.803.0.0.24022510', 'VMware-HBR-UW_8.0.3-0.0.24022510', 'MRVL-E4-CNA-Driver-Bundle_6.0.382.0-1OEM.800.1.0.20613240', 'VMware-NVMe-PCIe_1.2.4.15-1vmw.803.0.0.24022510', 'VMware-nvmxnet3-ens_2.0.0.23-6vmw.803.0.0.24022510', 'VMware-pvscsi_0.1-7vmw.803.0.0.24022510', 'Broadcom-elxiscsi_12.0.1200.0-11vmw.803.0.0.24022510', 'Intel-ixgben-ens_1.9.2.0-1OEM.800.1.0.20613240', 'VMware-VM-Tools_12.4.0.23259341-24022510', 'Broadcom-lsi-msgpt35_29.00.00.00-1vmw.803.0.0.24022510', 'Intel-igbn_1.11.2.0-1OEM.800.1.0.20613
240', 'Broadcom-lsiv2-drivers-plugin_1.0.3-1vmw.803.0.0.24022510', 'dell-configuration-component_8.0.0-A04', 'esx-update_8.0.3-0.0.24022510', 'Broadcom-bnxt-Net-RoCE_229.0.246.0-1OEM.800.1.0.20613240', 'Cisco-nenic_1.0.35.0-7vmw.803.0.0.24022510', 'MRVL-QLogic-FC_5.4.82.0-1OEM.800.1.0.20613240', 'VMware-rdmahl_1.0.0-1vmw.803.0.0.24022510', 'Intel-i40en_2.5.11.0-1OEM.700.1.0.15843807', 'Broadcom-ntg3_4.1.14.0-4vmw.803.0.0.24022510', 'VMware-NVMeoF-TCP_1.0.1.28-1vmw.803.0.0.24022510', 'Pensando-ionic-cloud_20.0.0-48vmw.803.0.0.24022510', 'Broadcom-ELX-IMA-plugin_12.0.1200.0-6vmw.803.0.0.24022510', 'Ci
sco-nfnic_5.0.0.42-1vmw.803.0.0.24022510', 'Intel-Volume-Mgmt-Device_3.0.0.1010-11vmw.803.0.0.24022510', 'VMware-vmkata_0.1-1vmw.803.0.0.24022510', 'VMware-nvme-plugin_1.2.0.56-1vmw.803.0.0.24022510', 'Microchip-smartpqiv2-plugin_1.0.0-11vmw.803.0.0.24022510', 'Broadcom-ELX-lpfc_14.4.0.39-35vmw.803.0.0.24022510', 'MRVL-E3-Ethernet_1.1.0.11-2vmw.803.0.0.24022510', 'HPE-nhpsa_70.0051.0.100-5vmw.803.0.0.24022510', 'HPE-hpv2-hpsa-plugin_1.0.0-4vmw.803.0.0.24022510']
INFO:ComponentScanner:Effective components: ['VMware-mnet_0.1-1vmw.803.0.0.24022510', 'VMware-SDHCI-Driver_1.0.3-3vmw.803.0.0.24022510', 'Pensando-ionic-en_20.0.0-56vmw.803.0.0.24022510', 'ESXi_8.0.3-0.0.24022510', 'Mellanox-nmlx5_4.23.6.2-7vmw.803.0.0.24022510', 'VMware-vmkusb_0.1-22vmw.803.0.0.24022510', 'VMware-Host-Client_2.18.0-23593406', 'VMware-nvmxnet3_2.0.0.31-12vmw.803.0.0.24022510', 'VMware-mlnx-bfbootctl_0.1-6vmw.803.0.0.24022510', 'Mellanox-nmlxbf-gige_2.2-1vmw.803.0.0.24022510', 'VMware-bfedac_0.1-1vmw.803.0.0.24022510', 'VMware-pengpio_0.1-1vmw.803.0.0.24022510', 'VMware-rd1173_0.1-1v
mw.803.0.0.24022510', 'VMware-NVMe-PCIe_1.2.4.15-1vmw.803.0.0.24022510', 'VMware-pvscsi_0.1-7vmw.803.0.0.24022510', 'VMware-spidev_0.1-1vmw.803.0.0.24022510', 'VMware-nmlxbf-pmc_0.1-6vmw.803.0.0.24022510', 'VMware-penspi_0.1-1vmw.803.0.0.24022510', 'VMware-NVMeoF-TCP_1.0.1.28-1vmw.803.0.0.24022510', 'VMware-nvme-plugin_1.2.0.56-1vmw.803.0.0.24022510', 'VMware-penedac_0.1-1vmw.803.0.0.24022510', 'VMware-nvmxnet3-ens_2.0.0.23-6vmw.803.0.0.24022510', 'VMware-dwi2c_0.1-7vmw.803.0.0.24022510', 'esxio-update_8.0.3-0.0.24022510']
INFO:vmware.esximage.ImageManager.SoftwareSpecMgr:Image validation result: {'info': [{'id': 'com.vmware.vcIntegrity.lifecycle.Validate.Success', 'message': {'id': 'com.vmware.vcIntegrity.lifecycle.Validate.Success', 'default_message': 'The image is valid.', 'args': []}, 'resolution': None, 'time': '2024-08-12T18:41:37.888Z'}], 'warnings': [], 'errors': []}
DEBUG:ImageServiceLogger:Heart beat at 2024-08-12 18:41:37.888909 for command: --taskid 521a1228-561a-1e6d-abc9-42a8fd7b712c --threadid 703139 software --validate Log Count 3 MP count 80139 Gap 0:00:10.847077

Resolution

To workaround this issue edit the following file

/etc/vmware/vmware-vmon/svcCfgfiles/updatemgr.json

Remove line

"StreamRedirectFile" : "%VMWARE_LOG_DIR%/vmware/vmware-updatemgr/updatemgr-vmon.log",

Restart the update manager service

service-control --restart vmware-updatemgr