HCX Site Pairing is stuck in Pending due to large RemotingOutbox table

Products

• VMware HCX

Issue/Introduction

Symptoms

• HCX Site Pairing status shows as Pending on the Connector side.
  
• The Cloud side Site Pairing status shows as Connected.
• Unable to perform management tasks or initiate new workload migrations.
• Management logs (web.log or app.log) contain org.apache.http.conn.ConnectTimeoutException or java.net.SocketTimeoutException: Read timed out.

Environment

VMware HCX 4.11.#

Cause

• The issue is caused by network layer instability (such as MTU mismatch or intermittent packet loss) that leads to a deterioration of the site pairing.
• This instability causes a significant buildup of records in the remotingOutbox database table.
• While HCX version 4.11.1 and higher include improved handling for this table, a massive accumulation (e.g., >100,000 records) can still cause the management plane to appear hung or stuck in a pending state.

Resolution

This issue is resolved in HCX 4.11.1 and higher. See Download Broadcom products, patches and software for steps to download this release.

If already running version 4.11.1 or higher and encounter this state, follow these steps to recover:

1. Stabilize the Network: Verify end-to-end connectivity. Ensure the MTU is set correctly (e.g., 1432) to prevent packet fragmentation or loss across the path.
2. Wait for records to clear from RemotingOutbox. 
   
   • Log in via SSH to both the HCX Connector and the HCX Cloud manager using the admin credentials.
   • Access the HCX database by running the following command: psql -h localhost -U postgres hybridity
   • Run the select query: select count(val) from "RemotingOutbox";
     
     Example:
     

     hybridity=# select count(val) from "RemotingOutbox";
     count
     ---------
     1166523
     (1 row)

Additional Information

• HCX Site Pairing is down due to a large "RemotingOutBox" table