NSX Upgrade Fails with Failed to create DRS maintenance mode requests. This is a prerequisite for putting hosts into maintenance. Reason: Reached maximum allowed retry attempts

Products

• VMware vSphere ESXi

Issue/Introduction

When attempting to perform an upgrade of VMware NSX (specifically upgrading the ESXi host transport nodes), the upgrade fails. The process halts when attempting to place hosts into Maintenance Mode.

Log snippet shows below details:

/var/log/vmware/vmware-updatemgr/vum-server/vmware-vum-server.log



Failed to get pooled connection; <cs p:00007f576c57d1c0, SsoCustomConnectionSpec:www.example.com:443>, 
SSL(<io_obj p:0x00007f566c0d6180, h:24, <TCP '127.0.0.1 : 37654'>, <TCP '127.0.0.1 : 443'>>), 
duration: 2msec, N7Vmacore3Ssl18SSLVerifyExceptionE(SSL Exception: Verification parameters:
--> PeerThumbprint: xx:xx:xx:xx:xx:xx:xx:xx
--> ExpectedThumbprint:
--> ExpectedPeerName: www.example.com
--> The remote host certificate has these problems:
-->
--> * certificate has expired)

 

Environment

VCenter 7.x
NSX 4.x

Cause

The issue is caused by an expired Trusted Root certificate present in the vCenter Server's certificate store (specifically the TRUSTED_ROOTS store).

The NSX Upgrade Coordinator requests vCenter (via DRS) to place hosts into maintenance mode. This API call relies on secure SSL communication. When the system validates the certificate chain, it encounters the expired root certificate, resulting in an SSLVerifyException.

Consequently, the DRS maintenance mode request fails to initialize. The system retries the connection until the retry limit is reached, triggering the "Reached maximum allowed retry attempts" error.

Resolution

Steps to resolve NSX upgrade issue:

1. We need to remove expired trusted certificate using KB 2146011
2. Adjust DRS Settings
   1. Log in to the vSphere Client.
   2. Select the Cluster where the upgrade is failing.
   3. Navigate to Configure > vSphere DRS.
   4. Click Edit.
   5. Change the Automation Level from Fully Automated to Partially Automated.
   6. Click OK.