Upgrading to VMware Cloud Foundation on Dell EMC VxRail 4.x fails

Products

VMware Cloud Foundation

Issue/Introduction

Symptoms:
When upgrding an AVN VxRail environment with a single vSphere Distributed Switch, you experience this symptom:

The upgrade fails during config drift migration.

Cause

This issue occurs because the AVN specific VLAN port groups are missing from the SDDC Manager inventory.

Notes:

You can find the error message on the LCM User Interface (UI) or in the SDDC Manager VM under the directory /var/log/vmware/vcf/lcm/thirdparty/upgrades/<UPGRADE_ID>/sddcmager-migration-app/logs.
Each upgrade will have different file location.

Section A :-

Sample error contains error similar to:

"ADD_EDGE_TEAMING_STANDBY_UPLINK_UPDATE_FAILED Failed to add standby uplink to edge cluster sfo-m01-ec01 portgroup sfo-m01-cl01-vds01-pg-uplink01".

"Failed to find portgroup sfo-m01-cl01-vds01-pg-uplink01 in DVS SDDC-Dswitch-Private VCF inventory".

Section B:-

Additional Log snippets below

2022-01-21 17:15:22.894 ERROR [,,,] 42502 --- [pool-4-thread-8] v.v.m.a.AddPortgroupTeamingStandbyUplink : Found more than one eligible standby uplink: [uplink1, uplink3, uplink4]
2022-01-21 17:15:22.898 WARN [,,,] 42502 --- [pool-4-thread-8] CompilerBase$ConnectionMonitorThreadBase : Shutting down the connection monitor.
2022-01-21 17:15:22.898 WARN [,,,] 42502 --- [ction-monitor-1] CompilerBase$ConnectionMonitorThreadBase : Interrupted, no more connection pool cleanups will be performed.
2022-01-21 17:15:22.919 INFO [,,,] 42502 --- [pool-4-thread-8] com.vmware.vipclient.i18n.VIPCfg : Formatting cache created.
2022-01-21 17:15:22.920 INFO [,,,] 42502 --- [pool-4-thread-8] com.vmware.vipclient.i18n.VIPCfg : Translation Cache created.
2022-01-21 17:15:22.929 ERROR [,,,] 42502 --- [pool-4-thread-8] c.v.e.s.o.model.error.ErrorFactory : [9FACM4] ADD_EDGE_TEAMING_STANDBY_UPLINK_UPDATE_FAILED Failed to add standby uplink to edge cluster <xyz> portgroup VCF-edge

Resolution

Section A :-

This is a known issue affecting upgrades of VMware Cloud Foundation 4.1 on Dell EMC VxRail (VCF on VxRail 4.1) to VMware Cloud Foundation 4.2 on Dell EMC VxRail (VCF on VxRail 4.2).
This issue is resolved in VMware Cloud Foundation 4.3 version

Section B:-

This is a known issue with VCF 4.x and will be fixed in the upcoming release.

Workaround:
To work around this issue:

Section A :-

Log in to the SDDC Manager and retrieve the default Cluster Name from the Management Domain. Note down this cluster name.
Open an SSH session to the SDDC Manager vm as the vcf user.
Perform an HTTP GET request on the /v1/cluster API to find out the Cluster ID associated with the Cluster Name from step 1.

a. Retrieve the token first for authorization:

vcf@sddc-manager [ ~ ]$ curl -k -X POST --data '{ "username": "[email protected]", "password": "******" }' -H "Content-Type: application/json" http://localhost/v1/tokens -s | json_pp
{
"accessToken" : "<token>",
"refreshToken" : {
"id" : "cfd0f9c4-####-####-####-8e3cdddf6912"
}
}

b. Perform an HTTP GET call on the /v1/clusters. Please replace <token> with the actual token received from previous step (3.a)

curl -H "Authorization: Bearer <token>" -H "Content-Type:application/json" 127.0.0.1/v1/clusters | json_pp
{ "elements" : [
     {
        "primaryDatastoreType" : "VSAN",
        "hosts" : [
           {
              "id" : "274a1311-####-####-####-ff8fc172119e"
           },
           {
              "id" : "b17cd3b5-####-####-####-ebbeaa3bf3d7"
           },
           {
              "id" : "a9b84398-####-####-####-5bbb9773894b"
           },
           {
              "id" : "16f35dfa-####-####-####-d335b10e13ce"
           }
        ],
        "name" : "VxRail-Virtual-SAN-Cluster",
        "primaryDatastoreName" : "VxRail-Virtual-SAN-Datastore",
        "isStretched" : false,
        "id" : "b0140cef-####-####-####-ccabbd426ab7"
    }
  ]
}
Sample command for HTTP GET call:
curl -H "Authorization: Bearer cfd0f9c4-####-####-####-8e3cdddf6912" -H "Content-Type:application/json" 127.0.0.1/v1/clusters | json_pp
c. Note down "id" (cluster id) matching cluster name (from step 1). Eg. b0140cef-####-####-####-ccabbd426ab7 in this example.
Log in to the vCenter Server and find out VDS name from the Management Domain.

Note: You can navigate to the vCenter Server from the SDDC Manager User Interface (UI) as well.

a. Navigate to Inventory > Workload Domains.
b. Click the Management Domain.
c. On the Services tab, click the link to open the Management Domain vCenter Server.

d. On the vCenter Server User Interface (UI), Navigate to the Networking tab to look for the VDS information. (The VDS information may be different in your environment). Note down VDS name. Eg: SDDC-Dswitch-Private in this example:
SSH to SDDC Manager VM through using the vcf user and then do su to log in as root.
Run this CURL command to retrieve the existing VDS information.

root@sddc-manager [ /home/vcf ]# curl 127.0.0.1/inventory/vds | json_pp > vds.json

a. Identify the VDS record associated with the Cluster ID from Step 3.c and the VDS name from step 4.d.

For example,

{
"version" : "7.0.0",
"isUsedByNsxt" : false,
"mtu" : 0,
"portGroups" : [
{
"mtu" : 0,
"transportType" : "VSAN",
"vlanId" : 0,
"type" : "EARLY_BINDING",
"activeUplinks" : [
"uplink1",
"uplink2"
],
"name" : "Virtual SAN",
"standbyUplinks" : []
},
{
"standbyUplinks" : [],
"name" : "vSphere vMotion",
"activeUplinks" : [
"uplink1",
"uplink2"
],
"type" : "EARLY_BINDING",
"vlanId" : 0,
"mtu" : 0,
"transportType" : "VMOTION"
},
{
"transportType" : "MANAGEMENT",
"mtu" : 0,
"name" : "Management Network",
"standbyUplinks" : [],
"vlanId" : 0,
"type" : "EPHEMERAL",
"activeUplinks" : [
"uplink1",
"uplink2"
]
}
],
"name" : "SDDC-Dswitch-Private",
"id" : "4dd10d99-####-####-####-a7e324e514c0",
"niocs" : [
{
"network" : "Management Traffic Type",
"level" : "Custom",
"value" : 0
},
{
"level" : "Custom",
"value" : 0,
"network" : "Fault Tolerance (FT) Traffic Type"
},
{
"network" : "vMotion Traffic Type",
"value" : 0,
"level" : "Custom"
},
{
"value" : 0,
"level" : "Custom",
"network" : "Virtual Machine Traffic Type"
},
{
"network" : "iSCSI Traffic Type",
"level" : "Custom",
"value" : 0
},
{
"network" : "NFS Traffic Type",
"value" : 0,
"level" : "Custom"
},
{
"network" : "vSphere Replication (VR) Traffic Type",
"value" : 0,
"level" : "Custom"
},
{
"network" : "vSAN Traffic Type",
"level" : "Custom",
"value" : 0
},
{
"value" : 0,
"level" : "Custom",
"network" : "vSphere Data Protection Backup Traffic Type"
},
{
"network" : "vSphere Backup NFC Traffic Type",
"value" : 0,
"level" : "Custom"
}
],
"clusterIds" : [
"b0140cef-####-####-####-ccabbd426ab7"
]
}
Copy the filtered JSON from the vds.json to the vds-updated.json. Ensure that the final vds-updated.json contains only one VDS entry with a matching vCenter VDS name.

Important Note: Ensure that the JSON data in the vds-updated.json is enclosed using { } instead of [ ]. Refer to the output shown in Step 6.
Use command cat file.json | json_pp This should return valid json file. If json is file is not proper, it should throw error showing the line number
Navigate to the vCenter Server MGMT Domain and look for the port groups associated with the Edge VMs (Deployed for AVNs).

You can verify these port group names from topology (in vCenter Server) as shown below:

Prepare the information for the AVN-specific VLAN port groups as show below. Use port group names from step 8 (use other information as shown below).

{
    "name" : "uplink01",
    "activeUplinks" : [
       "uplink1"
    ],
    "type" : "EPHEMERAL",
    "mtu" : 0,
    "standbyUplinks" : [
       "uplink2"
    ],
    "transportType" : "EXTERNAL",
    "vlanId" : 0
},
{
    "standbyUplinks" : [
       "uplink1"
    ],
    "vlanId" : 0,
    "transportType" : "EXTERNAL",
    "mtu" : 0,
    "type" : "EPHEMERAL",
    "name" : "uplink02",
    "activeUplinks" : [
       "uplink2"
    ]
  }
Add the AVN-specific VLAN port group information to the vds-updated.json file saved in Step 7. In this case, after adding the AVN-specific VLAN port group information, the vds-updated.json should look like the one shown below.

{
"isUsedByNsxt" : false,
"clusterIds" : [
"b0140cef-####-####-####-ccabbd426ab7"
],
"portGroups" : [
{
"type" : "EARLY_BINDING",
"name" : "Virtual SAN",
"activeUplinks" : [
"uplink1",
"uplink2"
],
"standbyUplinks" : [],
"transportType" : "VSAN",
"vlanId" : 0,
"mtu" : 0
},
{
"standbyUplinks" : [],
"transportType" : "VMOTION",
"vlanId" : 0,
"mtu" : 0,
"type" : "EARLY_BINDING",
"activeUplinks" : [
"uplink1",
"uplink2"
],
"name" : "vSphere vMotion"
},
{
"type" : "EPHEMERAL",
"activeUplinks" : [
"uplink1",
"uplink2"
],
"name" : "Management Network",
"standbyUplinks" : [],
"vlanId" : 0,
"transportType" : "MANAGEMENT",
"mtu" : 0
},
{
"name" : "uplink01",
"activeUplinks" : [
"uplink1"
],
"type" : "EPHEMERAL",
"mtu" : 0,
"standbyUplinks" : [
"uplink2"
],
"transportType" : "EXTERNAL",
"vlanId" : 0
},
{
"standbyUplinks" : [
"uplink1"
],
"vlanId" : 0,
"transportType" : "EXTERNAL",
"mtu" : 0,
"type" : "EPHEMERAL",
"name" : "uplink02",
"activeUplinks" : [
"uplink2"
]
}
],
"name" : "SDDC-Dswitch-Private",
"id" : "4dd10d99-####-####-####-a7e324e514c0",
"niocs" : [
{
"network" : "Management Traffic Type",
"level" : "Custom",
"value" : 0
},
{
"value" : 0,
"level" : "Custom",
"network" : "Fault Tolerance (FT) Traffic Type"
},
{
"level" : "Custom",
"value" : 0,
"network" : "vMotion Traffic Type"
},
{
"level" : "Custom",
"value" : 0,
"network" : "Virtual Machine Traffic Type"
},
{
"value" : 0,
"level" : "Custom",
"network" : "iSCSI Traffic Type"
},
{
"value" : 0,
"level" : "Custom",
"network" : "NFS Traffic Type"
},
{
"network" : "vSphere Replication (VR) Traffic Type",
"level" : "Custom",
"value" : 0
},
{
"level" : "Custom",
"value" : 0,
"network" : "vSAN Traffic Type"
},
{
"value" : 0,
"level" : "Custom",
"network" : "vSphere Data Protection Backup Traffic Type"
},
{
"value" : 0,
"level" : "Custom",
"network" : "vSphere Backup NFC Traffic Type"
}
],
"mtu" : 0,
"version" : "7.0.0"
}
Retrieve the VDS ID from the vds-updated.json file. In this case, it will be 4dd10d99-4e91-4efa-93f6-a7e324e514c0.
Run this CURL command to populate the SDDC Manager Inventory with the AVN-specific VLAN port groups.

curl -X PUT -H "Content-Type:application/json" --data @vds-updated.json 127.0.0.1/inventory/vds/{VDS_ID} | json_pp

In this case, it will be as shown below, with the VDS_ID = 4dd10d99-4e91-4efa-93f6-a7e324e514c0.

curl -X PUT -H "Content-Type:application/json" --data @vds-updated.json 127.0.0.1/inventory/vds/4dd10d99-4e91-4efa-93f6-a7e324e514c0 | json_pp

{
   "version" : "7.0.0",
   "isUsedByNsxt" : false,
   "niocs" : [
      {
         "network" : "Management Traffic Type",
         "value" : 0,
         "level" : "Custom"
      },
      {
         "network" : "Fault Tolerance (FT) Traffic Type",
         "level" : "Custom",
         "value" : 0
      },
      {
         "network" : "vMotion Traffic Type",
         "level" : "Custom",
         "value" : 0
      },
      {
         "network" : "Virtual Machine Traffic Type",
         "value" : 0,
         "level" : "Custom"
      },
      {
         "network" : "iSCSI Traffic Type",
         "value" : 0,
         "level" : "Custom"
      },
      {
         "network" : "NFS Traffic Type",
         "level" : "Custom",
         "value" : 0
      },
      {
         "value" : 0,
         "level" : "Custom",
         "network" : "vSphere Replication (VR) Traffic Type"
      },
      {
         "value" : 0,
         "level" : "Custom",
         "network" : "vSAN Traffic Type"
      },
      {
         "network" : "vSphere Data Protection Backup Traffic Type",
         "value" : 0,
         "level" : "Custom"
      },
      {
         "value" : 0,
         "level" : "Custom",
         "network" : "vSphere Backup NFC Traffic Type"
      }
   ],
   "mtu" : 0,
   "clusterIds" : [
      "b0140cef-####-####-####-ccabbd426ab7"
   ],
   "name" : "SDDC-Dswitch-Private",
   "id" : "4dd10d99-####-####-####-a7e324e514c0",
   "portGroups" : [
      {
         "activeUplinks" : [
            "uplink1",
            "uplink2"
         ],
         "transportType" : "VSAN",
         "mtu" : 0,
         "standbyUplinks" : [],
         "vlanId" : 0,
         "type" : "EARLY_BINDING",
         "name" : "Virtual SAN"
      },
      {
         "activeUplinks" : [
            "uplink1",
            "uplink2"
         ],
         "transportType" : "VMOTION",
         "name" : "vSphere vMotion",
         "type" : "EARLY_BINDING",
         "standbyUplinks" : [],
         "vlanId" : 0,
         "mtu" : 0
      },
      {
         "mtu" : 0,
         "vlanId" : 0,
         "standbyUplinks" : [],
         "type" : "EPHEMERAL",
         "name" : "Management Network",
         "activeUplinks" : [
            "uplink1",
            "uplink2"
         ],
         "transportType" : "MANAGEMENT"
      },
      {
         "name" : "uplink01",
         "type" : "EPHEMERAL",
         "vlanId" : 0,
         "standbyUplinks" : [
            "uplink2"
         ],
         "mtu" : 0,
         "activeUplinks" : [
            "uplink1"
         ],
         "transportType" : "EXTERNAL"
      },
      {
         "transportType" : "EXTERNAL",
         "activeUplinks" : [
            "uplink2"
         ],
         "type" : "EPHEMERAL",
         "name" : "uplink02",
         "mtu" : 0,
         "vlanId" : 0,
         "standbyUplinks" : [
            "uplink1"
         ]
      }
   ]
}
Finally, execute an HTTP GET command to retrieve the VDS list from the SDDC Manager Inventory and verify that the added port group information is present.

curl 127.0.0.1/inventory/vds | json_pp

Note: If config drift update on VCF 4.2.0.0 is already in a failed state, retry the update after completing steps in this article.

Section B :-

Change the Edge Cluster uplinks in such a way that each have only one active uplink and one stand by uplink this should workaround the Section B issue.