Post checks after an ESXi Upgrade throws the following error on the NSX GUI : "An existing request to execute upgrade checks is in progress. Please wait and retry."
search cancel

Post checks after an ESXi Upgrade throws the following error on the NSX GUI : "An existing request to execute upgrade checks is in progress. Please wait and retry."

book

Article ID: 418774

calendar_today

Updated On:

Products

VMware NSX

Issue/Introduction

  • While running the Post-Checks after an ESXi Upgrade, the following error is observed on the NSX GUI : "An existing request to execute upgrade checks is in progress. Please wait and retry."
  • In the /var/log/upgrade-coordinator/localhost_access_log.txt, the following entries are observed :

    2025-09-29T08:04:29.500Z 127.0.0.1 - - "POST /upgrade-coordinator/api/v1/upgrade/HOST?action=execute_post_upgrade_checks HTTP/1.1" 202 - 
    2025-09-29T09:00:33.452Z 127.0.0.1 - - "POST /upgrade-coordinator/api/v1/upgrade/HOST?action=execute_post_upgrade_checks HTTP/1.1" 202 - 
    2025-09-29T09:02:08.837Z 127.0.0.1 - - "POST /upgrade-coordinator/api/v1/upgrade/HOST?action=execute_post_upgrade_checks HTTP/1.1" 202 - 
    2025-09-29T09:36:20.000Z 127.0.0.1 - - "POST /upgrade-coordinator/api/v1/upgrade/HOST?action=execute_post_upgrade_checks HTTP/1.1" 503 226 
    2025-09-29T09:39:19.992Z 127.0.0.1 - - "POST /upgrade-coordinator/api/v1/upgrade/HOST?action=execute_post_upgrade_checks HTTP/1.1" 503 226 
  • Along with the above, the following entries are observed under /var/log/upgrade-coordinator/upgrade-coordinator.log :

    2025-09-29T09:02:08.817Z  INFO http-nio-127.0.0.1-7442-exec-5 UpgradeCoordinatorFacadeImpl 1416817 SYSTEM [nsx@6876 comp="nsx-manager" level="INFO" subcomp="upgrade-coordinator"] Invoke execution of post-upgrade checks for component HOST 
    <ends> 
    2025-09-29T09:02:14.425Z  INFO pool-61-thread-9 UpgradeServiceImpl 1416817 SYSTEM [nsx@6876 comp="nsx-manager" level="INFO" subcomp="upgrade-coordinator"] Finish execution of post-upgrade checks for component HOST 
    <updating tables> 
    2025-09-29T09:02:14.425Z  INFO pool-135-thread-1 NsxUpgradePlugin 1416817 SYSTEM [nsx@6876 comp="nsx-manager" level="INFO" subcomp="upgrade-coordinator"] 
    Persisting internal data for component HOST 
    2025-09-29T09:02:14.426Z  INFO pool-135-thread-1 UpgradePersistenceServiceImpl 1416817 SYSTEM [nsx@6876 comp="nsx-manager" level="INFO" subcomp="upgrade-c 
    oordinator"] [UPS] Updating record in Table - UpgradeInternal. Key - 4.2.2.1.0.24765084-HOST-internalData 
    2025-09-29T09:02:14.427Z  INFO pool-61-thread-14 NsxUpgradePlugin 1416817 SYSTEM [nsx@6876 comp="nsx-manager" level="INFO" subcomp="upgrade-coordinator"] 
    Persisting internal data for component HOST 
    2025-09-29T09:02:14.428Z  INFO pool-61-thread-14 UpgradePersistenceServiceImpl 1416817 SYSTEM [nsx@6876 comp="nsx-manager" level="INFO" subcomp="upgrade-c 
    oordinator"] [UPS] Updating record in Table - UpgradeInternal. Key - 4.2.2.1.0.24765084-HOST-internalData 
    2025-09-29T09:02:14.434Z  INFO pool-135-thread-1 UpgradePersistenceServiceImpl 1416817 SYSTEM [nsx@6876 comp="nsx-manager" level="INFO" subcomp="upgrade-c 
    oordinator"] [UPS] Updating record in Table - UpgradeInternal. Key - HOSTa596####-####-####-####-######fe3b6c-internalData 
    2025-09-29T09:02:14.442Z  INFO pool-61-thread-14 UpgradePersistenceServiceImpl 1416817 SYSTEM [nsx@6876 comp="nsx-manager" level=" 
    <seeing abortions> 
    2025-09-29T09:02:14.451Z  WARN pool-61-thread-14 ObjectsView 1416817 TXEnd[TX[6c3a]] Aborted Exception 
    org.corfudb.runtime.exceptions.TransactionAbortedException: TX ABORT  | Snapshot Time = Token(epoch=164, sequence=5699725734) | Failed Transaction ID = 67 
    01####-####-####-####-######d16c3a | Offending Address = 5699725735 | Conflict Key = EED3B6FBBA30918B | Conflict Stream = nsx$UpgradeInternal | Cause = CO 
    NFLICT | Time = 9 ms 
            at org.corfudb.runtime.view.StreamsView.append(StreamsView.java:180) ~[runtime-4.2.20250408232351.3032.1.jar:?] 
            at org.corfudb.runtime.view.StreamsView.append(StreamsView.java:233) ~[runtime-4.2.20250408232351.3032.1.jar:?] 
            at org.corfudb.runtime.view.StreamsView.append(StreamsView.java:244) ~[runtime-4.2.20250408232351.3032.1.jar:?] 
            at org.corfudb.runtime.object.transactions.OptimisticTransactionalContext.getConflictSetAndCommit(OptimisticTransactionalContext.java:223) ~[runti 
    me-4.2.20250408232351.3032.1.jar:?] 
    ntime-4.2.20250408232351.3032.1.jar:?] 
            at org.corfudb.runtime.view.ObjectsV 
    2025-09-29T09:02:14.451Z  WARN pool-61-thread-14 UpgradeServiceImpl 1416817 SYSTEM [nsx@6876 comp="nsx-manager" level="WARNING" subcomp="upgrade-coordinator"] [PUC] Error while persisting post upgrade checks data to DB for component HOST due to org.corfudb.runtime.exceptions.TransactionAbortedException: TX ABORT  | Snapshot Time = Token(epoch=164, sequence=5699725734) | Failed Transaction ID = 6701bb50-9139-c02d-46f2-bded22d16c3a | Offending Address = 5699725735 | Conflict Key = EED3B6FBBA30918B | Conflict Stream = nsx$UpgradeInternal | Cause = CONFLICT | Time = 9 ms 
            at org.corfudb.runtime.view.StreamsView.append(StreamsView.java:180) 

Environment

VMware NSX 4.x

Cause

This issue occurs due to a race condition. During the post-check execution, two distinct operations can sometimes try to write to the same database entry concurrently. When this rare timing occurs, it leaves the database in an inconsistent state.

Resolution

  • As a workaround, the above issue needs an Internal Corfu Edit to clear the inconsistent state.
  • Collect the below information and open a support request with Broadcom :
    • Support Bundles of all the three NSX Manager Nodes.
    • Output of the following command from any one of the NSX Manager nodes ( via the root user in CLI ) :
    • /opt/vmware/bin/corfu_tool_runner.py -n nsx -o showTable -t UpgradeInternal
Powered by Wolken Software