The SDDC Manager backup fails with the error: “Backup operation failed for SDDC Manager System Configuration.”

Products

VMware SDDC Manager VCF Operations

Issue/Introduction

The SDDC backup fails with the error "Backup operation failed for SDDC Manager System Configuration"
The following entries are seen in /var/log/vmware/vcf/sddc-support/backup-/sos.log:

[YYYY-MM-DDTHH:MM:SS] INFO [vcf sos] [sddcunifiedlogicalinventory.py: :get domain id for domain: : 143: :backup-db#####-#####-####-####-##########35d] Attempting to fetch Domainid for name <Domain-Name>

[YYYY-MM-DDTHH:MM:SS] DEBUG [vcf_sos] [sddcunifiedlogicalinventory.py: :get_domain_inventory: : 173: :backup-db#####-#####-####-####-##########35d] Attempting to fetch domain details for id 0d#####-######-####-########101f

[YYYY-MM-DDTHH:MM:SS] ERROR [vcf_sos] [util.py: : log_deployment_details: : 2875: :backup-db#####-#####-####-####-#########35d] Invalid value for 'management_ip_address', must not be 'None

ValueError: Invalid value for 'management_ip_address', must not be 'None'

Environment

SDDC 5.2.x

VMware Cloud Foundation 9.x

Cause

The SDDC Manager backup fails during the BackupSDDCManagerSystemConfiguration sub-task because it is unable to retrieve the Management Domain details using the inventory-service Python SDK. This occurs when the NSX-T Edge Cluster configuration in the SDDC Manager is missing the management IP address.

Resolution

This is a known issue in SDDC Manager 9.x, with a fix to be included in an upcoming 9.x release.

Workaround:

Manually update the NSX-T Edge Cluster management IP details within the SDDC Manager configuration.

Take a snapshot of the SDDC Manager VM from vCenter before making any modifications.
SSH into the SDDC Manager using the vcf user account.
Switch to the root user by running:
- su root
Obtain the management IP addresses of all NSX-T Edge nodes using the nslookup command.
Retrieve the Edge Cluster ID by following the below steps :
- Option 1: Logging into the SDDC Manager UI → navigating to Domain > Edge Cluster → copying the ID from the URL after /edge-clusters, (Please refer to the below screenshot).

- Option 2: Run the following commands from SDDC CLI to retrieve NSX-T Edge cluster ID:
  - psql --host=localhost -U postgres -d platform
  - select * from nsxt_edge_cluster;
    
    The following output is reported. The NSX-T Edge Cluster ID (highlighted in the id field).
    - id | 0d########-####-####-####-#########371
      creation_time | ###########67
      modification_time | ###########12
      status | ACTIVE
      name | <NSX-EDGE-Cluster-Name>
      nsxt_edge_nodes | [{"vmManagementIpAddress":"#.#.#.#","vmHostname":"<Edge_Node>","sourceId":"b8########-####-####-####-#########3a62","id":"ba########-####-####-####-#########78d"},{"vmManagementIpAddress":"#.#.#.#","vmHostname":"<Edge_Node>","sourceId":"a4########-####-####-####-#########b6d","id":"62########-####-####-####-#########101"}]
      source_id | ec########-####-####-####-#########aa8
      is_tier0managed_by_system | t
      skip_tep_routability_check | f

6. Export the Edge Cluster details and save them to a JSON file by running the below command:

- curl localhost/inventory/nsxt-edgeclusters?id=<edge-cluster-id> | jq > edge_cluster.json

Example:

- curl localhost/inventory/nsxt-edgeclusters?id=f19#####-####-####-####-##########50f | jq > edge_cluster.json

Review the JSON output using the command:
cat edge_cluster.json. You should see entries similar to the below,

[
  {
    "clusterIds": [
      "8f########-####-####-####-#########ee8"
    ],
    "nsxtClusterId": "0d########-####-####-####-#########371",
    "id": "f1########-####-####-####-#########50f",
    "edgeClusterNsxtId": "ed########-####-####-####-#########fcc",
    "name": "<nsx-cluster-name>",
    "status": "ACTIVE",
    "nsxtEdgeNodes": [
      {
        "id": "97########-####-####-####-#########cb5",
        "hostName": "<nsxt-edge-node>",
        "edgeNodeNsxtId": "bb########-####-####-####-##########6f1"
      },
      {
        "id": "f2########-####-####-####-#########a30",
        "hostName": "<nsxt-edge-node>",
        "edgeNodeNsxtId": "0c########-####-####-####-#########f38"
      }
    ],
    "isTier0ManagedBySystem": true,
    "skipTepRoutabilityCheck": false,
    "originType": "CLASSIC",
    "applianceType": "EDGE_NODE"
  }
]

7. Edit the JSON file (edge_cluster.json) to include the management IP addresses for each Edge Node by using vi to edit the file:

Example:

   {     "clusterIds": [       "8f########-####-####-####-#########ee8"     ],     "nsxtClusterId": "0d########-####-####-####-#########371",     "id": "f1########-####-####-####-#########50f",     "edgeClusterNsxtId": "ed########-####-####-####-#########fcc",     "name": "<nsx-cluster-name>",     "status": "ACTIVE",     "nsxtEdgeNodes": [       {         "id": "97########-####-####-####-#########cb5",         "hostName": "<nsxt-edge-node>",
        "managementIpAddress": "#.#.#.#",
      "edgeNodeNsxtId": "bb########-####-####-####-##########6f1"       },       {         "id": "f2########-####-####-####-#########a30",         "hostName": "<nsxt-edge-node>",
        "managementIpAddress": "#.#.#.#",
"edgeNodeNsxtId": "0c########-####-####-####-#########f38"
      }
    ],
    "isTier0ManagedBySystem": true,
    "skipTepRoutabilityCheck": false,
    "originType": "CLASSIC",
    "applianceType": "EDGE_NODE"
  }

8. Save the file using the following command in vi:

- :wq!

9. Update the NSX-T Edge Cluster in the SDDC Manager inventory by running the below PUT call:

- curl -X PUT -H "Content-Type: application/json" --data @edge_cluster.json localhost/inventory/nsxt-edgeclusters/{edge-cluster-id} | jq

Example :

- curl -X PUT -H "Content-Type: application/json" --data @edge_cluster.json localhost/inventory/nsxt-edgeclusters/0d########-####-####-####-#########371 | jq