vMotion connectivity test incorrectly using vmnic0 instead of the intended vmnic during Workload Domain deployment

Products

• VMware Cloud Foundation

Issue/Introduction

Symptoms:

• A vmnic other than vmnic0 is used for vMotion during the deployment of a Workload Domain.
• vMotion connectivity tests fail during the following workflows:
  • Deployment of a new Workload Domain
  • Adding a new cluster to an existing Workload Domain
  • Adding hosts to an existing cluster in a Workload Domain
• /var/log/vmware/vcf/domainmanager/domainmanager.log on SDDC Manager contains errors such as the following:

YYYY-MM-DDTHH:mm:ss WARN  [vcf_dm,aa98963e57e3e99c,8a5b] [c.v.e.s.c.u.c.SshCommandExecuter,dm-exec-5]  Error occurred while executing command vmkping <vMotion IP Address> -c 4 -d -s 1472 -I vmk1 -S vmotion : exit code 2
YYYY-MM-DDTHH:mm:ss INFO  [vcf_dm,aa98963e57e3e99c,8a5b] [c.v.e.s.c.util.HostValidationUtil,dm-exec-5]  Failed to confirm connectivity, Retrying vmkping in 5 seconds
YYYY-MM-DDTHH:mm:ss DEBUG [vcf_dm,aa98963e57e3e99c,30c9] [c.v.evo.sddc.common.util.SshUtil,dm-exec-9]  End of execution of command [vmkping <vMotion IP Address> -c 4 -d -s 1472 -I vmk1 -S vmotion ], Status: 2
Output: PING <vMotion IP Address> (<vMotion IP Address>): 1472 data bytes

Error output: sendto() failed (No route to host)

Command timed out: false

Environment

VMware Cloud Foundation 4.x

Cause

This is a known issue affecting VMware Cloud Foundation 4.x.

Resolution

Apply the attached hot patch by following the below steps:

1. Ensure that there are no running workflows via the SDDC Manager UI.
2. Take a snapshot of the SDDC Manager VM. Do not skip this step.
3. Download the correct file for your VCF version from the attachments section, for example, vcf-domain-manager-4.0.1-16990943.noarch.rpm and upload to /tmp on the SDDC Manager VM using WinSCP.
   If you are currently using VMware Cloud Foundation 4.0.1, please download vcf-domain-manager-4.0.1-16990943.noarch.rpm
   If you are currently using VMware Cloud Foundation 4.1.0.1, please download vcf-domain-manager-4.1.0-17591696.noarch.rpm
   If you are currently using VMware Cloud Foundation 4.1.0.0, please download vcf-domain-manager-4.1.0-18249021.noarch.rpm
4. Using SSH, connect to the SDDC Manager VM as the vcf user, then switch to the root user using the following command:
   su -
5. Execute following command, depending on version:
   rpm -Uvh /tmp/vcf-domain-manager-4.0.1-16990943.noarch.rpm
   Or
   rpm -Uvh /tmp/vcf-domain-manager-4.1.0-17591696.noarch.rpm
   Or
   rpm -Uvh /tmp/vcf-domain-manager-4.1.0-18249021.noarch.rpm
6. Perform a new Workload Domain deployment.

After installing the above hot patch, you need to configure version aliases:

1. SSH to the SDDC Manager as vcf user
2. Obtain an authorization bearer token:
   curl -k https://<SDDC Manager FQDN>/v1/tokens -H 'Content-Type: application/json' -i -k -d '{"username": "<username>", "password": "<password>"}'
3. Using the "accessToken" value from the API response, alias the Domain Manager service using one of the following commands:
   VMware Cloud Foundation 4.0.1: curl -k 'https://<SDDC MANAGER FQDN>/v1/system/settings/version-aliases/DOMAIN_MANAGER/4.0.1-vcf4010RELEASE-16428513' -k -X PUT -H 'Content-Type: application/json' -H 'Accept: application/json' -d '{"aliases" : [ "4.0.1-vcf4010RELEASE-16990943" ], "forceUpdate" : true}' -H "Authorization: Bearer <token from Step 2>"
   VMware Cloud Foundation 4.1.0.1: curl 'https://<SDDC MANAGER FQDN>/v1/system/settings/version-aliases/DOMAIN_MANAGER/4.1.0-vcf4101RELEASE-17206881' -k -X PUT -H 'Content-Type: application/json' -H 'Accept: application/json' -d '{"aliases" : [ "4.1.0-vcf4101RELEASE-17591696" ], "forceUpdate" : true}' -H "Authorization: Bearer <token from Step 2>"
   VMware Cloud Foundation 4.1.0.0: curl 'https://<SDDC MANAGER FQDN>/v1/system/settings/version-aliases/DOMAIN_MANAGER/4.1.0-vcf4100RELEASE-16961769' -k -X PUT -H 'Content-Type: application/json' -H 'Accept: application/json' -d '{"aliases" : [ "4.1.0-vcf4100RELEASE-18249021" ], "forceUpdate" : true}' -H "Authorization: Bearer <token from Step 2>"