Onboarding Local manager to global manager fails with error - Site offboarding in not completed

Products

• VMware NSX

Issue/Introduction

  

• Attempting to add the Production NSX Local Manager to the Global Manager is failing with the error: "Site offboarding is not completed." Error Code: 530028. 
• The onboarding status shows the following, indicating that the site cleanup is still in progress.

 

# curl -k -u admin -X GET https://localhost/policy/api/v1/infra/site/offboarding-status

Enter host password for user 'admin':
{
  "status" : "REALIZATION_INPROGRESS",
  "message" : "Site offboarding cleanup resources realization still in progress.",
  "resource_type" : "SiteOffBoardingState",
  "id" : "site-offboarding-state",
  "display_name" : "site-offboarding-state",
  "path" : "/infra/site-offboarding-state",
  "relative_path" : "site-offboarding-state",
  "unique_id" : "#################################",
  "realization_id" : "#################################",
  "owner_id" : "#################################",
  "marked_for_delete" : false,
  "overridden" : false,
  "_system_owned" : false,
  "_protection" : "NOT_PROTECTED",
  "_create_time" : 1687470241938,
  "_create_user" : "'globalmanageridentity'",
  "_last_modified_time" : 1723362363539,
  "_last_modified_user" : "system",
  "_revision" : 4384601

• In  /var/log/proton/nsxapi.log  we find that an object needs cleanup

2024-09-11T06:36:30.462Z  INFO SiteOffboardingTaskManager-1-1 RealizationState 75412 POLICY [nsx@6876 comp="nsx-manager" level="INFO" subcomp="manager"] Site offboarding cleanup resources realization still in progress.
2024-09-11T06:36:31.061Z  INFO NodeStatusPropertiesMsgListener-1-1 NodeStatusPropertiesMsgListener 75412 MONITORING [nsx@6876 comp="nsx-manager" level="INFO" subcomp="manager"] NodeStatusPropertiesMsgListener onNext: corfuStreamEntries o
rg.corfudb.runtime.collections.CorfuStreamEntries@7dd0d24d.
2024-09-11T06:36:34.832Z  INFO http-nio-127.0.0.1-7440-exec-1518 PreAuthenticationFilter 75412 PreAuthenticationFilter setting username to appliance-management from x-nsx-username header.
2024-09-11T06:36:34.835Z  INFO http-nio-127.0.0.1-7440-exec-1552 PreAuthenticationFilter 75412 PreAuthenticationFilter setting username to appliance-management from x-nsx-username header.
2024-09-11T06:36:34.838Z  INFO http-nio-127.0.0.1-7440-exec-1508 PreAuthenticationFilter 75412 PreAuthenticationFilter setting username to node-mgmt from x-nsx-username header.
2024-09-11T06:36:34.843Z  INFO http-nio-127.0.0.1-7440-exec-1546 PreAuthenticationFilter 75412 PreAuthenticationFilter setting username to node-mgmt from x-nsx-username header.
2024-09-11T06:36:35.463Z  INFO SiteOffboardingTaskManager-1-1 SiteOffboardingTaskManager 75412 POLICY [nsx@6876 comp="nsx-manager" level="INFO" subcomp="manager"] started site offboard task
2024-09-11T06:36:35.464Z  INFO SiteOffboardingTaskManager-1-1 StateMachine 75412 POLICY [nsx@6876 comp="nsx-manager" level="INFO" subcomp="manager"] starting offboarding state machine from state REALIZATION
2024-09-11T06:36:35.464Z  INFO SiteOffboardingTaskManager-1-1 RealizationState 75412 POLICY [nsx@6876 comp="nsx-manager" level="INFO" subcomp="manager"] processing realization state
2024-09-11T06:36:35.517Z  INFO SiteOffboardingTaskManager-1-1 SiteOffboardingServiceImpl 75412 POLICY [nsx@6876 comp="nsx-manager" level="INFO" subcomp="manager"] intent DfwFirewallConfiguration [ idfwEnabled=false, idfwEventLogScraperEn
abled=false, idfwLoginisghtEnabled=falsedefaultSecurityPosture=BLACKLIST, enableDefaultRuleLogging=false, enableFirewall=true, disableAutoDrafts=true, overrideFlag=false, globalAddrsetModeEnabled=true, globalMacsetOptimizationModeEnabled
false] still exist for realized resource GenericPolicyRealizedResource{path=/global-infra/realized-state/enforcement-points/default/settings/firewall/security/idfw-enabled, realizationObjectId=idfw/master-switch-setting, realizationState
=REALIZED, intentVersion=0, realizedVersionOnEnforcement=null, realizationAPI=null, entityType=RealizedFirewallConfigurationIdfwEnabled, readBeforeWriteRequired=false, extendedAttributes={}, intentPaths=[/global-infra/settings/firewall/s
ecurity]}
2024-09-11T06:36:35.528Z  INFO SiteOffboardingTaskManager-1-1 SiteOffboardingServiceImpl 75412 POLICY [nsx@6876 comp="nsx-manager" level="INFO" subcomp="manager"] for enforcement point default, some realized resources are still not clean
ed [/global-infra/realized-state/enforcement-points/default/settings/firewall/security/idfw-enabled]

 

• We are not sure if local manager was onboarded to another global manager in the past. 

 

 

Environment

VMware NSX 

VMware NSX-T datacenter

Cause

In this scenario we got stale object as "/global-infra/realized-state/enforcement-points/default/settings/firewall/security/idfw-enabled" from the  /var/log/proton/nsxapi.log 

Note: The object can be different in other cases

Resolution

Manually cleanup the object by running the API.

POST https://<LM_IP>/policy/api/v1/troubleshooting/infra/tree/realization?action=cleanup
{
"paths":["//global-infra/realized-state/enforcement-points/default/settings/firewall/security/idfw-enabled"]
}

 

The command should be executed with response 200 OK

Check the onboarding status again to ensure it is successful

# curl -k -u admin -X GET https://localhost/policy/api/v1/infra/site/offboarding-status
Enter host password for user 'admin':
{
  "status" : "SUCCESSFUL",
  "message" : "Site offboarding completed successfully.",
  "resource_type" : "SiteOffBoardingState",
  "id" : "site-offboarding-state",
  "display_name" : "site-offboarding-state",
  "path" : "/infra/site-offboarding-state",
  "relative_path" : "site-offboarding-state",
  "unique_id" : "XXXX-XXXX-XXXX-XXXX-",
  "realization_id" : "XXXX-XXXX-XXXX-XXXX-",
  "owner_id" : "XXXX-XXXX-XXXX-XXXX-",
  "marked_for_delete" : false,
  "overridden" : false,
  "_system_owned" : false,
  "_protection" : "NOT_PROTECTED",
  "_create_time" : 1687470241938,
  "_create_user" : "'globalmanageridentity'",
  "_last_modified_time" : 1726039503577,
  "_last_modified_user" : "system",
  "_revision" : 4384604

 

The LM onboarding should be successful after following above steps.