Transport Node Certificate Renewal Failure on NSX Edges When Using CARR Script

Products

• VMware NSX

Issue/Introduction

• An alarm is raised in the NSX Manager GUI indicating upcoming certificate expiration for transport nodes.
  
  Alarm: Transport Node Certificate Expiration Approaching
  
  
• During certificate renewal using the CARR script, the process partially succeeds.
  
  
• The CARR script reports the following error during execution:
  
  

  carr.recovery.base_recovery_task - MainThread - ERROR - base_recovery_task.py:86 - 
  Error in recover task: <class 'carr.recovery.edge_node_cert_rotation_task.EdgeNodeCertRotationTask'> 
  cert_name: EDGE : error: Error in replacing cert for the client : ###-###-##-##-####<clientid>: 
  409 Client Error: Conflict for url: https://10.###.###.###:443/api/v1/trust-management/certificates/action/replace-host-certificate/###-###-##-##-####<clientid>

  
  This indicates that while the certificate rotation task works for some Edge nodes, it can fail with a 409 Conflict error for others.
  
  
• From the CARR logs, the following error is observed:
  
  

   - DEBUG - connectionpool.py:546 - https://10.XXX.XX.XX:443 "POST /api/v1/trust-management/certificates/action/replace-host-certificate/XXXXX-XXXX-XXXX-XXXX-XXXXXXXXX HTTP/11" 409 None
  2025-07-29 13:25:43,423 - carr.interface.rest.base_api - MainThread - ERROR - base_api.py:145 - Response : {
    "httpStatus" : "CONFLICT",
    "error_code" : 223,
    "module_name" : "common-services",
    "error_message" : "Update already in progress."
  }

  
  
  
• Additionally, no GET API request for the affected Edge node UUID is logged, indicating that the CARR script could not fetch the required certificate ownership details.
  
  

  25-07-29 13:25:43,424 - carr.recovery.base_recovery_task - MainThread - ERROR - base_recovery_task.py:86 - Error in recover task: <class 'carr.recovery.edge_node_cert_rotation_task.EdgeNodeCertRotationTask'> cert_name: EDGE : error: Error in replacing cert for the client : XXXXXX-XXXX-XXXX-XXXXX-XXXXXX: 409 Client Error: Conflict for url: https://10.XXX.XX.XX:443/api/v1/trust-management/certificates/action/replace-host-certificate/XXXXX-XXX-XXX-XXX-XXXXXXX

  
  
• From the Proton logs (/var/log/proton/nsxapi.log), when the certificate replacement is attempted, the system throws an InvalidOwnerException:
  
  

  2025-08-11T15:18:48.235Z ERROR http-nio-127.0.0.1-7440-exec-1 OwnershipValidatorImpl 5026 SYSTEM [nsx@6876 comp="nsx-manager" errorCode="MP289" level="ERROR" reqId="XXXXX-XXX-XXX-XXX" subcomp="manager" username="admin"] Principal 'admin' with role '[enterprise_admin]' attempts to delete or modify an object of type nsx$Client it doesn't own. (createUser=nsx_policy, allowOverwrite=null)
  2025-08-11T15:18:48.235Z ERROR http-nio-127.0.0.1-7440-exec-1 TxnContext 5026 TX Abort merge: nsx$Client
  com.vmware.nsx.management.container.exceptions.InvalidOwnerException: null
    at com.vmware.nsx.management.protection.OwnershipValidatorImpl.checkCallerIsOwner(OwnershipValidatorImpl.java:62) ~[?:?]
    at com.vmware.nsx.persistence.UfoTxn.checkOwnership(UfoTxn.java:885) ~[?:?]
    at com.vmware.nsx.persistence.UfoTxn$MergeCallbackImpl.doMerge(UfoTxn.java:641) ~[?:?]
    at org.corfudb.runtime.collections.TxnContext.merge(TxnContext.java:273) ~[?:?]

 

Environment

NSX version: 4.2.x

Cause

The certificate replacement process can fail for multiple reasons. However, if the failure is specifically associated with the error InvalidOwnershipException, the outlined workaround is applicable in this case.

The issue is likely due to a user role assignment change between the time the Transport Nodes were created and when the certificates are being replaced.

 

Resolution

As a workaround, the certificate for the affected Edge nodes can be manually replaced following "Transport Node certificate has expired and TN is in a disconnected state in NSX:" section in Resolution field of Alarm For Transport Node Certificate Expiration Approaching KB.