vpxd crashes with duplicate key value in "vpx_nw_assignment" when using HCX-IX for migrations
search cancel

vpxd crashes with duplicate key value in "vpx_nw_assignment" when using HCX-IX for migrations

book

Article ID: 323283

calendar_today

Updated On:

Products

VMware HCX VMware vCenter Server

Issue/Introduction

  • When any of the below actions are performed on an HCX IX appliance, the vCenter appliance VPXD service fails:
    • Redeployment
    • Performs any HCX Cold, vMotion, and Replication Assisted vMotion (RAV) migrations - which utilize the Mobility Agent. 

  • service-control --start vpxd fails with below : 
    Service-control failed. Error: Failed to start services in profile ALL. RC=2, stderr=Failed to start vpxd services. Error: Service crashed while starting

  • vCenter vpxd service crashed during a HCX based migration.

  • Below logging will be noticed on vCenter logs : 

    /var/log/vmware/vpxd/vpxd.log

    2024-##-##-##-##.225Z error vpxd[55220] [Originator@6876 sub=Default opID=TxId: ##############] [VdbStatement] SQL execution failed: INSERT INTO VPX_NW_ASSIGNMENT (NETWORK_ID,ENTITY_ID) VALUES (?,?)
2024-##-##-##-##.225Z error vpxd[55220] [Originator@6876 sub=Default opID=TxId: ##############] [VdbStatement] Execution elapsed time: 5 ms
2024-##-##-##-##.225Z error vpxd[55220] [Originator@6876 sub=Default opID=TxId: ##############] [VdbStatement] Statement diagnostic data from driver is 23505:0:1:ERROR: duplicate key value violates unique constraint "pk_vpx_nw_assignment"
--> DETAIL: Key (network_id, entity_id)=(###, ###) already exists.;
--> Error while executing the query
2024-##-##-##-##.225Z error vpxd[55220] [Originator@6876 sub=Default opID=TxId: ##############] [VdbStatement] Bind parameters:
2024-##-##-##-##.225Z error vpxd[55220] [Originator@6876 sub=Default opID=TxId: ##############] [VdbStatement] [0]datatype: 3, size: 8, arraySize: 0
2024-##-##-##-##.225Z error vpxd[55220] [Originator@6876 sub=Default opID=TxId: ##############] [VdbStatement] value = ###
2024-##-##-##-##.225Z error vpxd[55220] [Originator@6876 sub=Default opID=TxId: ##############] [VdbStatement] [1]datatype: 3, size: 8, arraySize: 0
2024-##-##-##-##.225Z error vpxd[55220] [Originator@6876 sub=Default opID=TxId: ##############] [VdbStatement] value = ###
2024-##-##-##-##.225Z error vpxd[55220] [Originator@6876 sub=Default opID=TxId: ##############] [Vdb::IsRecoverableErrorCode] Unable to recover from 23505:1
2024-##-##-##-##.225Z error vpxd[55220] [Originator@6876 sub=Default opID=TxId: ##############] [VdbStatement] SQLError was thrown: "ODBC error: (23505) - ERROR: duplicate key value violates unique constraint "pk_vpx_nw_assignment"
--> DETAIL: Key (network_id, entity_id)=(###, ###) already exists.;
--> Error while executing the query" is returned when executing SQL statement "INSERT INTO VPX_NW_ASSIGNMENT (NETWORK_ID,ENTITY_ID) VALUES (?,?)"
2024-##-##-##-##.229Z panic vpxd[55220] [Originator@6876 sub=vpxCommon opID=TxId: ##############] Unrecoverable VmRootError: 00007f14c0147b50 Backtrace:
--> Backtrace:

2024-##-##-##-##.229Z error vpxd[55220] [Originator@6876 sub=Default opID=TxId: ##############] An unrecoverable problem has occurred, stopping the VMware VirtualCenter service. Error: Error[VdbODBCError] (-1) "ODBC error: (23505) - ERROR: duplicate key value violates unique constraint "pk_vpx_nw_assignment"
--> DETAIL: Key (network_id, entity_id)=(###, ###) already exists.;
--> Error while executing the query" is returned when executing SQL statement "INSERT INTO VPX_NW_ASSIGNMENT (NETWORK_ID,ENTITY_ID) VALUES (?,?)"
2024-##-##-##-##.237Z panic vpxd[55220] [Originator@6876 sub=Default opID=TxId: ##############]
-->
--> Panic: Unrecoverable VmRootError. Panic!
--> Backtrace:

     

  • Below logging will be noticed on HCX logs :

    /common/logs/admin/app.log

    2024-##-##-##-##0.615 UTC [VmotionService_SvcThread-19765, Ent: HybridityAdmin, , TxId: TxId: ##-##-##-##########] INFO  c.v.h.s.vmotion.jobs.VmotionWorkflow- [migId=##-##-##-##########] Job 'c6458141-f02e-40d8-a80a-8ab603e906f4' VmotionWorkflow executing state : 'RECONFIG_SOURCE_MA'
2024-##-##-##-##7.481 UTC [VmotionService_SvcThread-19782, Ent: HybridityAdmin, , TxId: TxId: ##-##-##-##########] INFO  c.v.h.s.v.j.ReconfigureSourceMobilityAgentWorkflow- [migId=##-##-##-##########] Virtual machine Dvpgs to opaqueNewmappings  got is {TESTING-Test-Net-2_##.##.##.0/24={"opaqueNetworkId":"##-##-##-#######",
"opaqueNetworkName":"TESTING-Test-Net-2_##.##.##.0\/24","opaqueNetworkType":"nsx.LogicalSwitch"}, TESTING-Test-Net-2_##.##.##.0%2f24={"opaqueNetworkId":"##-##-##-#######",
"opaqueNetworkName":"TESTING-Test-Net-2_##.##.##.0%2f24","opaqueNetworkType":"nsx.LogicalSwitch"}}

     

Environment

vCenter Server
HCX

Cause

vCenter vpxd service crashed due to duplicate key value being inserted to table "vpx_nw_assignment". This happens when HCX migrates VM's to network port group which has any special characters.

EXAMPLE:
HCX got 2 different opaqueNetworkName for same network ID.

"opaqueNetworkName":"TESTING-Test-Net-2_##.##.##.0\/24"
"opaqueNetworkName":"TESTING-Test-Net-2_##.##.##.0%2f24"      <<<<<<<<<<<<<< ISSUE IS HERE 

Notice the character "/" got converted to "%2f" which is the actual root cause of the issue.

Resolution

To avoid this issue, please update/change the network name to not have any special characters like "/".


Please upgrade vCenter Server to 8.0 U3 and later versions to avoid such vpxd crash , available at Broadcom downloads.
A fix for vCenter 7.0 U3# version line is also in plan and will be available in a future release.
If you are having difficulty finding and downloading software, please review the Download Broadcom products and software KB.

If you believe you have encountered this issue, please open a support case with Broadcom Support and refer to this KB article. vCenter Support team can help you to get the vpxd state up.
For more information, see Creating and managing Broadcom support cases


Once the vpxd service issue is fixed, please update/change the network name to not have any special characters like "/". Then redeploy HCX-IX via HCX UI/hybridity page , so that the Mobility Agent host is added back to vCenter.

Additional Information

Once the vpxd service issue is fixed, please rename the network name to not have any special characters. Then redeploy IX via HCX UI so that the Mobility Agent(MA) host is added back to vCenter.

Go to HCX hybridity page or HCX Plugin via vCenter.
Click Interconnect >> Service Mesh >> Appliances >> Click on the IX appliance >> Click "REDEPLOY"

Powered by Wolken Software