Edge upgrade fails with error: "certificate update failed"

Products

• VMware NSX

Issue/Introduction

Symptoms:

• NSX Edge upgrade fails.
• You see the error:
  
  Certificate update failed
   
• Upgrade node summary shows mix of edge node versions. 
• On a NSX-T 2.5 setup, edge nodes with version earlier than NSX-T 2.5 are found.
• An Edge node has been deleted and is no longer present on the Fabric > Nodes > Edge Transport Nodes UI page.
• The API does not list the Edge node GET https://<NSXMGR_IP>/api/v1/transport-nodes but can still be found using the UI search 
• On a NSX-T 2.5 setup, from Version for Edge upgrade will be populated to less than NSX-T 2.5.
• In the Edge support bundle, you may see entries similar to:
  
  "Exception when uploading certificate b'{\\n  \"details\" : \"org.springframework.web.client.HttpClientErrorException$BadRequest: 400 null\",\\n  \"httpStatus\" : \"INTERNAL_SERVER_ERROR\",\\n  \"error_code\" : 30014,\\n  \"module_name\" : \"upgrade-coordinator\",\\n  \"error_message\" : \"[UC] Error in rest call. url= /nsxapi/api/v1/messaging/clients/xxxxxxxx-06e5-4534-b588-xxxxxxxxxxxx , method= PUT , response= {\\\\n  \\\\\"details\\\\\" : \\\\\"Field level validation errors: {required property account_name is missing}\\\\\",\\\\n  \\\\\"httpStatus\\\\\" : \\\\\"BAD_REQUEST\\\\\",\\\\n  \\\\\"error_code\\\\\" : 255,\\\\n  \\\\\"module_name\\\\\" : \\\\\"common-services\\\\\",\\\\n  \\\\\"error_message\\\\\" : \\\\\"Field level validation errors: {required property account_name is missing}\\\\\"\\\\n} , error= 400 null .\"\\n}'\n"

Environment

VMware NSX-T Data Center 2.5.x
VMware NSX-T Data Center

Cause

This issue occurs due to an incomplete deletion of edge node having version <  NSX-T 2.5. The node is removed as a Transport Node but is not removed as a Fabric Node.

Note: Subsequent upgrades to version NSX-T 2.5 or above will be affected by these orphan edge nodes.

Resolution

This issue is resolved in VMware NSX-T Data Center 3.0, available at  Broadcom Downloads.