Compliance Scans Hang or Fail After VCF 9.1 Completes on ESXi stage, Stalling Upgrade Due to Stale Syslog Configuration

Products

• VCF Operations/Automation (formerly VMware Aria Suite)

Issue/Introduction

During an upgrade to VMware Cloud Foundation (VCF) 9.1, you encounter issues where compliance checks during the ESXi upgrade process hang at 80% or fail.

You may observe the following errors in vCenter:

• com.vmware.vapi.std.errors.not_allowed_in_current_state
• com.vmware.esx.task.contention <Another task is in progress. Please retry later.>
• Remediation cannot begin because another process is updating the host.

Environment

VCF 9.1

Cause

The vmsyslogd service repeatedly crashes, triggering host-wide task throttling. This is caused by a DNS resolution failure for a legacy log management URL that was replaced in the VCF 9.1 architecture.

This logging failure causes active lifecycle remediation tasks to time out, which leaves behind an orphaned lock and state file. These orphaned files prevent any subsequent compliance scans from executing correctly.

Verification can be found in /var/log/vmsyslogd.log on the impacted host, which shows repeated failure messages similar to:

ERROR vmsyslogd #### [esx@#### threadName="OutputThread"] Connect: Failed to establish connection to syslog collector NetworkLogger::UDP://####:514. getaddrinfo failed with error: Name or service not known

Resolution

To resolve this issue and allow the upgrade to proceed, you must clean up the stale logging configuration and clear the stuck task state:

1.  Remove stale syslog log server entries in the host's advanced settings - the entry that is failing to properly resolve a name when trying DNS lookup from the host.

2.  Clear the hung task:
    *   Log in to the affected ESXi host.
    *   Locate and delete the orphaned `/var/run/.esximg.pid` file that is causing the task loop.
    *   restart the settingsd service: /etc/init.d/settingsd restart

3.  Retry compliance check: Once the unresolvable loghost is removed and the PID file is deleted, retry the compliance check from vCenter.

Removing the unresolvable loghost restores stability to the vmsyslogd service, while deleting the orphaned `.esximg.pid` file clears the stuck `com.vmware.esx.task.contention` state, allowing the 9.1 upgrade to complete.

Additional Information

Note: There is a permanent fix targeted for release 9.1.1. Once that release is officially GA, please see Download Broadcom products and software for steps to download and apply the fix.