NVDS to VDS migration precheck fails for APPLY_TOPOLOGY_FAILED

Products

• VMware NSX

Issue/Introduction

• NVDS to VDS migration precheck fails with a state APPLY_TOPOLOGY_FAILED.
  nsxm > vds-migrate precheck
VDSUpgradePrecheck: {"httpStatus": "BAD_REQUEST", "error_code": 250012, "module_name": "NvdsUpgradeUrt Service", "error_message": "Precheck is in progress with state APPLY_TOPOLOGY_FAILED."}
• It is logged that DVS not found during precheck.
  /var/log/proton/nsxapi.log
  <TimeStamp>  INFO NvdsUpgradeTaskExecutor1 NvdsUpgradeReadinessCheckByClusterServiceImpl 4500 FABRIC [nsx@6876 comp="nsx-manager" level="INFO" subcomp="manager"] NVDS-UPGRADE vds <DVS UUID> not found, sleeping for 30 sec ..
  ...
  <TimeStamp> ERROR NvdsUpgradeTaskExecutor1 NvdsUpgradeReadinessCheckByClusterServiceImpl 4500 FABRIC [nsx@6876 comp="nsx-manager" errorCode="MP250011" level="ERROR" subcomp="manager"] VDS <DVS UUID> not found in NSX inventory
  <TimeStamp>  WARN NvdsUpgradeTaskExecutor1 NvdsUpgradeReadinessCheckByClusterServiceImpl 4500 FABRIC [nsx@6876 comp="nsx-manager" level="WARNING" subcomp="manager"] NVDS-UPGRADE Failed to fetch VDS, APPLY_TOPOLOGY_FAILED
• The DVS has been created in VC inventory.
• Inventory sync has a trouble.
  /var/log/cm-inventory/cm-inventory.log
  <TimeStamp>  INFO task-executor-0 MpRestService 10877 SYSTEM [nsx@6876 comp="nsx-manager" level="INFO" subcomp="cm-inventory"] Getting cluster certificate using REST API nsxapi/api/v1/trust-management/certificates?type=cluster_api_certificate
  <TimeStamp>  INFO task-executor-0 LoggingRestTemplate 10877 SYSTEM [nsx@6876 comp="nsx-manager" level="INFO" subcomp="cm-inventory"] Request::URI:http://localhost:7440/nsxapi/api/v1/trust-management/certificates?type=cluster_api_certificate method:GET
  <TimeStamp>  WARN task-executor-0 CmDataSenderImpl 10877 SYSTEM [nsx@6876 comp="nsx-manager" level="WARNING" subcomp="cm-inventory"] Issue processing cm <UUID> message: CmInvMsg{cmId='<UUID>', msgType='FULL_SYNC', <snip>
  java.lang.ArrayIndexOutOfBoundsException: null
• The command in engineering mode on NSX Manager returns no certificate.
  curl -k -i -X GET -H 'X-NSX-Username:admin' -H "Content-Type: application/json" http://localhost:7441/api/v1/cluster/api-virtual-ip

Environment

VMware NSX-T Datacenter

Cause

For some reason, for example NSX Manager cluster has been restored from backup, the VIP certificate is not correctly configured.
Identify the VIP certificate in the result of GET /api/v1/trust-management/certificates/ .

 {
    "pem_encoded" : "<Encoded certificate>",
    "has_private_key" : true,
    "used_by" : [ {
      "node_id" : "<Invalid node ID>",
      "service_types" : [ "MGMT_CLUSTER" ]
    } ],
    <snip>
    "_revision" : 2
  }

Valid node_id for MGMT_CLUSTER should be either site ID or a node ID of any existing Manager node.
The site ID can be found in /config/site-manager/siteId .

If the node_id is invalid, for example the ID of Manager node before restore, it delays inventory sync from VC to NSX Manager and CVDS is not recognized during precheck.

 

 

Resolution

1. Configure the VIP certificate of the Manager cluster again.
   POST /api/v1/trust-management/certificates/<cert-id>?action=apply_certificate&service_type=MGMT_CLUSTER
2. Clean up prechecks.
   POST /api/v1/nvds-urt?action=cleanup
3. Invoke precheck again.

Additional Information

https://bugzilla.eng.vmware.com/show_bug.cgi?id=3314285