[HCX] Unable to extend VDS networks with HCX - Failed NSX-T request, statusCode: 500, Invalid sort key

Products

• VMware Cloud on AWS
• VMware HCX

Issue/Introduction

To explain the cause of the failed stretch operation and provide information regarding a fix.

Symptoms:
HCX Network extension operation with Cloud Provider as destination fails.
HCX Manager app.log entries for the NetworkStretchService show status code 500 and Invalid sort key messages.
These can be found in the /common/logs/admin directory of the HCX log bundle:

[NetworkStretchService_SvcThread-106, j: 4a5fbc5a, s: 4d1ca64c, , TxId: ff6ff262-fff2-fffd-b02d-000000000026] ERROR c.v.v.h.n.i.AbstractJobInt- NetworkStretchJobs workflow BridgeNetworksJobInt failed. Error: Failed NSX-T request : { "url": "https://nsxManager.sddc.vmwarevmc.com/api/v1/switching-profiles", "method": "GET" } Got response : { "status": "failure", "statusCode": 500, "details": "", "result": { "details": "Invalid sort key, key: create_time", "httpStatus": "INTERNAL_SERVER_ERROR", "error_code": 100, "module_name": "common-services", "error_message": "General error has occurred." } }

Cause

This is due to a race condition on one of the NSX Manager nodes which cause the API call from HCX to NSX fail.

Resolution

The fix will be available in an upcoming release.

Workaround:
Please open a SR with the managed service provider (VMC on AWS, AVS, Google, etc) and have them fail over the NSX Manager to another within the cluster.  This should clear the race condition and allow the segment to be extended.  If this does not further investigation will need to be performed.  Please ensure logs from the NSX manager, HCX manager and vCenter are collected.  

Additional Information

Creating and managing Broadcom support cases
Extending Networks with VMware HCX

Impact/Risks:
Unable to extend the network segment.