Esxi upgrade using lifeCycle manager fails with error "Task failed with Unknown Error: Fault cause: vim.fault.MethodDisabled"

Products

VMware SDDC Manager VMware Cloud Foundation VMware vCenter Server VMware vSphere ESXi

Issue/Introduction

Esxi upgrade from 8.0.3 to 9.0.1 fails using vLCM with error cause: vim.fault.MethodDisabled
Upgrade attempted using SDDC manager or vCenter single image configuration fails with same error.
/var/log/vmware/vcf/lcm/lcm-debug.log reports below :

YYYY-MM-DDTHH:MM:SS.468+0000 ERROR [vcf_lcm,0000000000000000,0000,upgradeId=3a30b5e9-b284-44d9-8024-305bdb2b4046,resourceType=ESX_CLUSTER,resourceId=c530d823-82ad-480c-a597-1bfe6ded738d,bundleElementId=d8d0567f-0751-41bd-bc22-3078affe6f1e] [c.v.e.s.l.p.impl.cluster.VlcmUtils,Upgrade-3] Notification error for Id: com.vmware.vcIntegrity.lifecycle.TaskError.UnknownError Message: Task failed with Unknown Error: Fault cause: vim.fault.MethodDisabled
YYYY-MM-DDTHH:MM:SS.470+0000 DEBUG [vcf_lcm,0000000000000000,0000,upgradeId=3a30b5e9-b284-44d9-8024-305bdb2b4046,resourceType=ESX_CLUSTER,resourceId=c530d823-82ad-480c-a597-1bfe6ded738d,bundleElementId=d8d0567f-0751-41bd-bc22-3078affe6f1e] [c.v.e.s.l.p.i.c.VlcmPrimitiveStatusImpl,Upgrade-3] Notification id: com.vmware.vcIntegrity.lifecycle.TaskError.UnknownError, Notification message: {"defaultMessage":"Task failed with Unknown Error: Fault cause: vim.fault.MethodDisabled\n, Failed Hosts are: host-XXXX Skipped Hosts are: host-XXXX,host-XXXX,host-XXXX"}, Notification resolution: {"defaultMessage":""}
In vCenter /var/log/vmware/vsphere-updatemgr/vum-server/vmware-vum-server.log reports reboot method disabled when attempted image remediation:
YYYY-MM-DDTHH:MM:SS.959Z info vmware-vum-server[54074] [Originator@6876 sub=vmomi.soapStub[0] opID=a46aece2-72c5-4801-a379-6ca3476612cb] SOAP request returned HTTP failure; <<io_obj t:N7Vmacore6System19TCPSocketObjectAsioE, h:21, <TCP '127.0.0.1 : 56642'>, <TCP '127.0.0.1 : 1080'>>, /extension-login/sdk>, method: reboot; code: 500(Internal Server Error); fault: (vim.fault.MethodDisabled) {
--> faultCause = (vmodl.MethodFault) null,
--> faultMessage = <unset>,
--> source = "rbac"
--> msg = "Received SOAP response fault from [<<io_obj t:N7Vmacore6System19TCPSocketObjectAsioE, h:21, <TCP '127.0.0.1 : 56642'>, <TCP '127.0.0.1 : 1080'>>, /extension-login/sdk>]: reboot
--> The method is disabled by 'rbac'"
--> }

Environment

VMware Cloud Foundation 9.0
vCenter Server 9.0

Cause

The vSphere Lifecycle Manager (vLCM) remediation task fails because the essential API methods required to orchestrate the host lifecycle, specifically EnterMaintenanceMode and Reboot are disabled for the target ESXi host within the vCenter Server Managed Object Reference (MoRef) database.

During a standard remediation workflow, vLCM must successfully transition the host into maintenance mode and subsequently issue a reboot command to stage and apply the desired image payload. If these operational methods are restricted or disabled at the vCenter Server level (typically resulting from conflicting background tasks, stale state locks, or third-party solution interventions), the vCenter API actively rejects the vLCM state change requests. Consequently, the vLCM orchestration pipeline is unable to satisfy the prerequisite host conditions and immediately terminates the remediation task.

Resolution

Identify which methods are disabled in vCenter mob for esxi host using mob url

https://<vcenter_fqdn>/mob/?moid=<host_moref_id>

Here, host_moref_id will be host-XXXX id of failing host.
The Managed Object Browser (MOB) is deactivated by default in VMware Cloud Foundation (VCF) 9.0, To enable MOB, follow steps using kb "The MOB is disabled", unable to access the vCenter MOB in VCF 9.0
Login using administrator@sso_domain.local
Go to disabledMethod section and check which methods are disabled. Below methods will be disabled:

vim.HostSystem.exitStandbyMode
vim.HostSystem.enterMaintenanceMode
vim.HostSystem.reconnect
vim.HostSystem.reconfigureDAS
vim.HostSystem.reboot
vim.HostSystem.shutdown
To enable Methods, login using administrator@sso_domain.local to below URL:
https://<vc-ip>/mob/?moid=AuthorizationManager&method=enableMethods
Add below in entity type
<entity type="ManagedEntity" xsi:type="ManagedObjectReference">host-XXXX</entity>
Add below in methods to enable:
<method>vim.HostSystem.exitStandbyMode</method>
<method>vim.HostSystem.enterMaintenanceMode</method>
<method>vim.HostSystem.reconnect</method>
<method>vim.HostSystem.reconfigureDAS</method>
<method>vim.HostSystem.reboot</method>
<method>vim.HostSystem.shutdown</method>
Click Invoke Method.
Repeat Step 4,5,6 for all the hosts in single image cluster.
Perform image remediation to upgrade esxi hosts.