"Site offboarding in not completed" error when onboarding Local Manager to Global Manager

Products

• VMware NSX

Issue/Introduction

• Attempting to add the Production NSX Local Manager to the Global Manager is failing with the error: "Site offboarding is not completed." Error Code: 530028.
• The onboarding status shows the following, indicating as "REALIZATION_INPROGRESS" or "CLEANUP_INPROGRESS"  

    # curl -k -u admin -X GET https://localhost/policy/api/v1/infra/site/offboarding-status

Enter host password for user 'admin':
{
  "status" : "REALIZATION_INPROGRESS",
  "message" : "Site offboarding cleanup resources realization still in progress.",
  "resource_type" : "SiteOffBoardingState",
  "id" : "site-offboarding-state",
  "display_name" : "site-offboarding-state",
  "path" : "/infra/site-offboarding-state",
  "relative_path" : "site-offboarding-state",
"unique_id" : "#################################",
"realization_id" : "#################################",
"owner_id" : "#################################",
  "marked_for_delete" : false,
  "overridden" : false,
  "_system_owned" : false,
  "_protection" : "NOT_PROTECTED",
  "_create_time" : 1687470241938,
  "_create_user" : "'globalmanageridentity'",
  "_last_modified_time" : 1723362363539,
  "_last_modified_user" : "system",
  "_revision" : 4384601

Enter host password for user 'admin':
{
"status": "CLEANUP_INPROGRESS",
    "message": "Site offboarding cleanup started.",
    "previousErrorMessage": "Site offboarding cleanup failed. Cleanup will be retried. Error: [The object path=[/global-infra/tier-0s/Federation_Tier0/locale-services/SL_Prod/bgp] cannot be deleted as either it is overridden or has overridden children.  The found overridden paths=[/global-infra/tier-0s/Federation_Tier0/locale-services/SL_Prod/bgp]]",
 "resource_type" : "SiteOffBoardingState",
  "id" : "site-offboarding-state",
  "display_name" : "site-offboarding-state",
  "path" : "/infra/site-offboarding-state",
  "relative_path" : "site-offboarding-state",
  "unique_id" : "#################################",
  "realization_id" : "#################################",
  "owner_id" : "#################################",
  "marked_for_delete" : false,
  "overridden" : false,
  "_system_owned" : false,
  "_protection" : "NOT_PROTECTED",
  "_create_time" : 1687470241938,
  "_create_user" : "'globalmanageridentity'",
  "_last_modified_time" : 1723362363539,
  "_last_modified_user" : "system",
  "_revision" : 4384601

• On the NSX Manager on  /var/log/proton/nsxapi.log  we find that an object needs cleanup

 

YYYY-MM-DDT06:36:30.462Z  INFO SiteOffboardingTaskManager-1-1 RealizationState 75412 POLICY [nsx@6876 comp="nsx-manager" level="INFO" subcomp="manager"] Site offboarding cleanup resources realization still in progress.
YYYY-MM-DDT06:36:31.061Z  INFO NodeStatusPropertiesMsgListener-1-1 NodeStatusPropertiesMsgListener 75412 MONITORING [nsx@6876 comp="nsx-manager" level="INFO" subcomp="manager"] NodeStatusPropertiesMsgListener onNext: corfuStreamEntries o
rg.corfudb.runtime.collections.CorfuStreamEntries@7dd0d24d.
YYYY-MM-DDT06:36:34.832Z  INFO http-nio-127.0.0.1-7440-exec-1518 PreAuthenticationFilter 75412 PreAuthenticationFilter setting username to appliance-management from x-nsx-username header.
YYYY-MM-DDT06:36:34.835Z  INFO http-nio-127.0.0.1-7440-exec-1552 PreAuthenticationFilter 75412 PreAuthenticationFilter setting username to appliance-management from x-nsx-username header.
YYYY-MM-DDT06:36:34.838Z  INFO http-nio-127.0.0.1-7440-exec-1508 PreAuthenticationFilter 75412 PreAuthenticationFilter setting username to node-mgmt from x-nsx-username header.
YYYY-MM-DDT06:36:34.843Z  INFO http-nio-127.0.0.1-7440-exec-1546 PreAuthenticationFilter 75412 PreAuthenticationFilter setting username to node-mgmt from x-nsx-username header.
YYYY-MM-DDT06:36:35.463Z  INFO SiteOffboardingTaskManager-1-1 SiteOffboardingTaskManager 75412 POLICY [nsx@6876 comp="nsx-manager" level="INFO" subcomp="manager"] started site offboard task
YYYY-MM-DDT06:36:35.464Z  INFO SiteOffboardingTaskManager-1-1 StateMachine 75412 POLICY [nsx@6876 comp="nsx-manager" level="INFO" subcomp="manager"] starting offboarding state machine from state REALIZATION
YYYY-MM-DDT06:36:35.464Z  INFO SiteOffboardingTaskManager-1-1 RealizationState 75412 POLICY [nsx@6876 comp="nsx-manager" level="INFO" subcomp="manager"] processing realization state
YYYY-MM-DDT06:36:35.517Z  INFO SiteOffboardingTaskManager-1-1 SiteOffboardingServiceImpl 75412 POLICY [nsx@6876 comp="nsx-manager" level="INFO" subcomp="manager"] intent DfwFirewallConfiguration [ idfwEnabled=false, idfwEventLogScraperEn
abled=false, idfwLoginisghtEnabled=falsedefaultSecurityPosture=BLACKLIST, enableDefaultRuleLogging=false, enableFirewall=true, disableAutoDrafts=true, overrideFlag=false, globalAddrsetModeEnabled=true, globalMacsetOptimizationModeEnabled
false] still exist for realized resource GenericPolicyRealizedResource{path=/global-infra/realized-state/enforcement-points/default/settings/firewall/security/idfw-enabled, realizationObjectId=idfw/master-switch-setting, realizationState
=REALIZED, intentVersion=0, realizedVersionOnEnforcement=null, realizationAPI=null, entityType=RealizedFirewallConfigurationIdfwEnabled, readBeforeWriteRequired=false, extendedAttributes={}, intentPaths=[/global-infra/settings/firewall/s
ecurity]}
YYYY-MM-DDT06:36:35.528Z  INFO SiteOffboardingTaskManager-1-1 SiteOffboardingServiceImpl 75412 POLICY [nsx@6876 comp="nsx-manager"  level="INFO" subcomp="manager"] for enforcement point default, some realized resources are still not cleaned [/global-infra/realized-state/enforcement-points/default/settings/firewall/security/idfw-enabled]

 

Environment

VMware NSX-T Data Center
VMware NSX 

Resolution

If you believe you have encountered this issue, please open a support case with Broadcom Support and refer to this KB article.

For more information, see Creating and managing Broadcom support cases.