ESX 9.1 Image Compliance status unknown with "com.vmware.vapi.std.errors.not_allowed_in_current_state" post upgrade due to unreachable Syslog server
search cancel

ESX 9.1 Image Compliance status unknown with "com.vmware.vapi.std.errors.not_allowed_in_current_state" post upgrade due to unreachable Syslog server

book

Article ID: 447360

calendar_today

Updated On:

Products

VMware vCenter Server VMware vSphere ESXi

Issue/Introduction

  • Post upgrade attempt of an ESX host to 9.1 using cluster image, the host successfully boots to ESX 9.1 however vSphere Lifecycle Manager fails to recognize the updated version with the compliance status for the host as "unknown" on the Updates tab of the concerned Host with an error message similar to one of the below:

    com.vmware.vapi.std.errors.not_allowed_in_current_state
    com.vmware.esx.task.contention <Another task is in progress. Please retry later.>
    Remediation cannot begin because another process is updating the host.






  • On the vCenter update manager logs, ClusterScanTask errors similar to below are observed for affected ESX host:

    /var/log/vmware/vmware-updatemgr/vum-server/vmware-vum-server.log

    YYYY-MM-DDThh:mm:ss WARNING vmware-vum-server ####[vc@####sub="com.vmware.vcIntegrity.lifecycle.DesiredScanClusterTask" opID="####"] [ClusterScanTask, ####] Task:com.vmware.vcIntegrity.lifecycle.DesiredScanClusterTask ID:<task_id>. Status is still pending for host : <ESX_FQDN> (host-<host_id>) ..Waiting.
    ...
    YYYY-MM-DDThh:mm:ss ERROR vmware-vum-server #### [vc@####sub="com.vmware.vcIntegrity.lifecycle.DesiredScanClusterTask" opID="####"] [ClusterScanTask, ####] Task:com.vmware.vcIntegrity.lifecycle.DesiredScanClusterTask ID:<task_id>. Calling host-<host_id> host Image Scan request failed. Retriable error (= true) - Exceeded max retry count = 10 - Error:Error:
    -->   com.vmware.vapi.std.errors.error
    --> Messages:
    -->   com.vmware.vcIntegrity.lifecycle.HostScan.TaskApiError<A failure occurred when starting a host compliance check operation on host '<ESX_FQDN>' > : Error:
    -->   com.vmware.vapi.std.errors.not_allowed_in_current_state
    --> Messages:
    -->   com.vmware.esx.task.contention<Another task is in progress. Please retry later.>


  • Additionally, in case the upgrade was attempted from VCF Operations 9.1, an error is encountered on VCF Operations UI similar to below:

    After host '<ESX_FQDN>' remediation completed, compliance check reported host as 'non-compliant'. The image on the host does not match the image set for the cluster. Retry the cluster remediation operation


  • On the syslog service log for the affected ESX host, connectivity failures similar to below are observed:

    /var/log/vmsyslogd.log

    YYYY-MM-DDThh:mm:ss ERROR vmsyslogd #### [esx@#### threadName="OutputThread"] Connect: Failed to establish connection to syslog collector NetworkLogger::TCP://<UNREACHABLE_SYSLOG_SERVER_FQDN>:514. getaddrinfo failed with error: Name or service not known


  • Additionally, the ESX host might be slow to respond to commands via CLI or SSH and is not writing to one or more log files including /var/run/log/hostd.log, /var/run/log/settingsd.log, and so on.

Environment

vCenter 9.1

VMware ESX 9.1

Cause

The vmsyslogd service repeatedly crashes as it is unable to reach the configured Syslog server URL, triggering host-wide performance throttling due to task contention. This logging failure causes active lifecycle remediation tasks to time out on vCenter, and post successful ESX upgrade to 9.1, these tasks get orphaned on the ESX host causing vCenter to incorrectly detect that another task is in progress during subsequent compliance scans.

Resolution

The issue regarding the vmsyslogd service crash will be resolved in a future release. To resolve this issue on the affected host and restore compliance status, follow the steps below:

  1. Clear the syslog configuration on the affected ESXi host:
    esxcli system syslog config set --reset=loghost

  2. Reload the Syslog daemon:
    esxcli system syslog reload

  3. Place the affected ESX host into maintenance mode.

  4. Remove the orphaned lifecycle task and related files:
    rm /var/run/.esximg.pid
    rm /var/vmware/lifecycle/lifecycle.task
    rm /var/vmware/lifecycle/task-status.json.db

  5. Reboot the ESXi host.

  6. Retry the compliance check from vCenter.

Removing the unresolvable loghost restores stability to the vmsyslogd service, while deleting the orphaned .esximg.pid file clears the com.vmware.esx.task.contention state, allowing the vCenter to perform the compliance scan and detect the upgraded version of the ESX host.

For subsequent ESX hosts, prior to the upgrade, either remove the loghost config or updating it to a valid reachable endpoint or an explicit IP address. This will ensure that post-upgrade the vmsyslogd service does not crash and the host does not encounter this issue.

Additional Information

Commands to update the syslog server on an ESXi Host can be found below:
esxcli system syslog config set --loghost=tcp://<SYSLOG_IP_OR_FQDN>:514?formatter=RFC_5424&framing=octet_counting
esxcli system syslog reload

Powered by Wolken Software