Unable to place Host in Maintenance Mode using vLCM remediate 'Reached maximum allowed retry attempts'
search cancel

Unable to place Host in Maintenance Mode using vLCM remediate 'Reached maximum allowed retry attempts'

book

Article ID: 387946

calendar_today

Updated On:

Products

VMware vCenter Server 8.0 VMware vSphere ESX 8.x

Issue/Introduction

  • Unable to place Host in Maintenance Mode using vLCM remediate with error  "A general system error occurred: Failed to create DRS maintenance mode requests. This is a prerequisite for outing hosts into maintenance. Reason: 'Reached maximum allowed retry attempts"
  • /var/log/vmware/vmware-updatemgr/vum-server/vmware-vum-server.log

YYYY-MM-DDTHH:MM:SS info vmware-vum-server[107363] [Originator@6876 sub=EHP opID=b8e0b933-a507-4dc6-877e-a1497ffe432d] Creating vAPI session
YYYY-MM-DDTHH:MM:SS info vmware-vum-server[107363] [Originator@6876 sub=EHP opID=b8e0b933-a507-4dc6-877e-a1497ffe432d] Scheduled: health check query for [domain-c962745], perspective: [BEFORE_START_APPLY]
YYYY-MM-DDTHH:MM:SS info vmware-vum-server[107363] [Originator@6876 sub=EHP opID=b8e0b933-a507-4dc6-877e-a1497ffe432d] [domain-cXXXXXX] A provider [drs-plugin] has finished (0 remaining).
YYYY-MM-DDTHH:MM:SS info vmware-vum-server[107363] [Originator@6876 sub=EHP opID=b8e0b933-a507-4dc6-877e-a1497ffe432d] [domain-cXXXXXX] All providers have finished. Elapsed time (sec): 0
YYYY-MM-DDTHH:MM:SS info vmware-vum-server[107363] [Originator@6876 sub=EHP opID=b8e0b933-a507-4dc6-877e-a1497ffe432d] [domain-cXXXXXX] [vSphere HA] [com.vmware.vpxd.healthPerspectives.ready_for_apply.ha] returned status: OK
YYYY-MM-DDTHH:MM:SS info vmware-vum-server[107363] [Originator@6876 sub=EHP opID=b8e0b933-a507-4dc6-877e-a1497ffe432d] Entity [domain-cXXXXXX] health status for perspective [BEFORE_START_APPLY] is: OK
YYYY-MM-DDTHH:MM:SS verbose vmware-vum-server[1109189] [Originator@6876 sub=vmomi.soapStub[102]] Initial service state request failed, disabling pings; /lookupservice/sdk, <last binding: <<TCP '127.0.0.1 : 56198'>, <TCP '127.0.0.1 : 443'>> >, HTTP Status:404 'Not Found'

YYYY-MM-DDTHH:MM:SS info vmware-vum-server[107381] [Originator@6876 sub=EHP opID=b8e0b933-a507-4dc6-877e-a1497ffe432d] Filtering health checks in provider [vcsa]...
YYYY-MM-DDTHH:MM:SS info vmware-vum-server[107381] [Originator@6876 sub=EHP opID=b8e0b933-a507-4dc6-877e-a1497ffe432d] Health check 'com.vmware.vcIntegrity.lifecycle.health.vc.no_host_in_mm' not executable.
YYYY-MM-DDTHH:MM:SS info vmware-vum-server[107381] [Originator@6876 sub=EHP opID=b8e0b933-a507-4dc6-877e-a1497ffe432d] [com.vmware.vcIntegrity.lifecycle.health.vc.no_host_in_pmm]: Live update: false
YYYY-MM-DDTHH:MM:SS info vmware-vum-server[107381] [Originator@6876 sub=EHP opID=b8e0b933-a507-4dc6-877e-a1497ffe432d] [com.vmware.vcIntegrity.lifecycle.health.vc.query_vmotion_compat_ex]: Health check not executable - not on VMC.
YYYY-MM-DDTHH:MM:SS info vmware-vum-server[107381] [Originator@6876 sub=EHP opID=b8e0b933-a507-4dc6-877e-a1497ffe432d] Health check 'com.vmware.vcIntegrity.lifecycle.health.vc.query_vmotion_compat_ex' not executable.
YYYY-MM-DDTHH:MM:SS info vmware-vum-server[107381] [Originator@6876 sub=EHP opID=b8e0b933-a507-4dc6-877e-a1497ffe432d] CheckContext: {entityMoId: "domain-cXXXXXX", vapiSession: "session id", env: {"HA enabled": true, "Host part of VMC": false, "vLCM-VMC integration, Pod service enabled": false, }}, ClusterCheckContext: {spec: {{ com.vmware.esx.health.clusters
.check_spec : { exclude_checks : [ ] , hosts : Optional< [ host-XXXXXX, ] >, maintenance_mode_type : Optional< >, memory_reservation : Optional< >, perspective : BEFORE_ENTER_MAINTENANCE, upgrade_actions : Optional< >, } }} }

YYYY-MM-DDTHH:MM:SS error vmware-vum-server[107367] [Originator@6876 sub=SamlAsyncApiProvider opID=b8e0b933-a507-4dc6-877e-a1497ffe432d] [SamlAsyncApiProvider 77] Failed to add Saml auth provider error:SSL Exception: Verification parameters:
--> PeerThumbprint: xx:xx:xx:xx:xx:xx:xx:xx:xx:xx:xx:xx:xx:xx:xx:xx:xx:xx:xx:xx
--> ExpectedThumbprint:
--> ExpectedPeerName: vcenter.sample.domain
--> The remote host certificate has these problems:
-->
--> * unable to get issuer certificate
YYYY-MM-DDTHH:MM:SS warning vmware-vum-server[107367] [Originator@6876 sub=ClusterOps opID=b8e0b933-a507-4dc6-877e-a1497ffe432d] [ClusterOps 874] Failed to create maintenance requests. Reason: Error: Error:
-->    com.vmware.vapi.std.errors.unauthenticated
--> No messages!
--> . Retry after 10 seconds

YYYY-MM-DDTHH:MM:SS error vmware-vum-server[107367] [Originator@6876 sub=ClusterOps opID=b8e0b933-a507-4dc6-877e-a1497ffe432d] [ClusterOps 882] Failed to invoke requests. Reason: Reached maximum allowed retry attempts. Giving Up
YYYY-MM-DDTHH:MM:SS error vmware-vum-server[107367] [Originator@6876 sub=RemediateClusterTask opID=b8e0b933-a507-4dc6-877e-a1497ffe432d] Failed to create maintenance requests. Reached maximum allowed retry attempts. Giving Up

 

 

 

Environment

VMware vCenter Server

Cause

This is due to a missing Custom Root certificate (enterprise or third-party CA) from the vCenter Server TRUSTED_ROOT store.

Resolution

Note: Ensure that there is a valid backup of the vCenter Server(s) or a powered off snapshot taken before proceeding.

  •  Publish Root CA example: USERTrust RSA Certification Authority to TRUSTED_ROOT store using the below methods :
    •  CLI : /usr/lib/vmware-vmafd/bin/dir-cli trusted cert publish --cert <certifcatepath> 
    •  UI :  Log into the vSphere Client -> Administration -> Certificates -> Certificate Management-->Trusted Root-->Add Trusted Root Certificate.
  • Restart all services on the vCenter using the below command.
    • service-control --stop --all  &&  service-control --start --all
  • Retry Remediation Host's using vSphere LifecycleManagement (vLCM).

 

Additional Information

The above steps can be followed in case of Intermediate certificate (enterprise or third-party CA) missing from the vCenter Server TRUSTED_ROOT store.

Powered by Wolken Software