Scanning ESXi hosts in Lifecycle Manager fails with the error "not_allowed_in_current_state"

Products

• VMware vSphere ESXi

Issue/Introduction

Symptoms:

• Attempting to scan clusters in lifecycle manager fails with:

Error: com.vmware.vapi.std.errors.not_allowed_in_current_state
Messages: com.vmware.esx.lifecycle.task.contention<Another task possibly in progress. Try later

• On the ESXi host, /var/run/log/settingsd.log shows messages similar to the following:

2021-06-08T12:38:53Z settingsd[2101990]: [WriteToFile] Unable to open file: /var/vmware/lifecycle/lifecycle.task
2021-06-08T12:38:53Z settingsd[2101990]: [_CheckTaskContention]: Failed to write to the pid file in task 524db25e-d532-8389-d52f-6e166d606553: std::exception

• This issue might also cause the HA configuration to fail on recently upgraded hosts.

Environment

VMware vSphere ESXi 7.0.x

Cause

This issue occurs when the scratch location is located on /tmp which is non-persistent. A routine task cleans up /tmp and removes the /var/vmware/lifecycle contents, leading to the errors.

In general, the scratch location should be configured to a persistent location.

Resolution

This issue is resolved in ESXi 7.0U3c, where the routine task will no longer remove the files in /var/vmware/lifecycle.

Workaround:
To work around this issue, configure a persistent scratch location and restart the settingsd service

/etc/init.d/settingsd restart

Note: Restarting the settingsd services should recreate the files in /var/vmware/lifecycle, however, they are subject to removal again until /scratch is persistent or the host is upgraded to ESXi 7.0 U3+.