Operations against an existing NSX-T workload domain fails after upgrading VMware Cloud Foundation to version 3.5.1

Products

• VMware Cloud Foundation

Issue/Introduction

Symptoms:

• Operations against an existing NSX-T workload domain fails on the "Gather input to add host to NSX-T Fabric and migrate networking from dvs to nvds" task.  These operations can include creating a new cluster, expanding a cluster or removing a host from a cluster.
• VMware Cloud Foundation was upgraded from version 3.5 to version 3.5.1 after the NSX-T workload domain was created.
• The default NSX-T certificate has not been replaced.
• You see messages similar to the following in the /var/log/vmware/vcf/domainmanager/domainmanager.log file on the SDDC Manager VM.

2018-12-19 12:10:56.492 [vcf_dm,7ef3e7f82436c1bc,878d3675a0613279] [-thread-15] ERROR [ c.v.v.c.f.p.nsxt.action.NsxtAddClusterHeader] Error occurred while generating input for add hosts in cluster in nsxt environment
java.lang.RuntimeException: Cannot execute request
 at com.vmware.vapi.internal.protocol.client.rest.DefaultRequestExecutorFactory$DefaultHttpResponseHandler.onError(DefaultRequestExecutorFactory.java:94)

Caused by: java.lang.RuntimeException: javax.net.ssl.SSLHandshakeException: sun.security.validator.ValidatorException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target

Caused by: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target

2019-03-14T20:36:59.867Z 127.0.0.1 - "GET /ui/api/domainmanager/nsxt/controlcluster HTTP/1.0" 500 2 - 1485.791 ms - "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_14_4) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/72.0.3626.121 Safari/537.36"
2019-03-14T21:09:09.563Z 127.0.0.1 - "GET /ui/api/domainmanager/nsxt/controlcluster HTTP/1.0" 500 2 - 505.748 ms - "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_14_4) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/72.0.3626.121 Safari/537.36"

2019-03-14 20:36:58.358 [vcf_dm,58af6bd16418dac4,58af6bd16418dac4] [200-exec-8] INFO [ c.v.vcf.domainmanager.rest.DomainManagerAbout] Getting domainmanager service info
2019-03-14 20:36:58.972 [vcf_dm,f1a52c3513e5d811,f1a52c3513e5d811] [200-exec-9] DEBUG [ c.v.e.s.c.s.a.c.CredentialServiceAdapterImpl] Getting API credential for entity 5c3b893d-1014-4332-b3b9-0220d0c4ef82 of entity type NSXT_MANAGER
2019-03-14 20:36:59.856 [vcf_dm,f1a52c3513e5d811,f1a52c3513e5d811] [200-exec-9] ERROR [ c.v.v.v.controller.ViManagerNsxtController] Received an exception while getting NsxtControlCluster for domainId: null
2019-03-14 21:09:08.961 [vcf_dm,d4413bd718eb2bb0,d4413bd718eb2bb0] [200-exec-3] INFO [ c.v.vcf.domainmanager.rest.DomainManagerAbout] Getting domainmanager service info
2019-03-14 21:09:09.393 [vcf_dm,23e582533587b6f2,23e582533587b6f2] [200-exec-1] DEBUG [ c.v.e.s.c.s.a.c.CredentialServiceAdapterImpl] Getting API credential for entity 5c3b893d-1014-4332-b3b9-0220d0c4ef82 of entity type NSXT_MANAGER
2019-03-14 21:09:09.558 [vcf_dm,23e582533587b6f2,23e582533587b6f2] [200-exec-1] ERROR [ c.v.v.v.controller.ViManagerNsxtController] Received an exception while getting NsxtControlCluster for domainId: null

2019-03-14 22:54:24.976 [vcf_om,633f3689af5c123a,633f3689af5c123a] [om-exec-13] ERROR [ c.v.v.r.s.u.vsphere.cache.ClusterMetricsCache] Unable to retrieve vLAN Id from NSX-T manager
java.lang.RuntimeException: Cannot execute request

Note: The preceding log excerpts are only examples. Date, time, and environmental variables may vary depending on your environment.

Environment

VMware Cloud Foundation 3.5.x

Resolution

This is a know issue affecting VMware Cloud Foundation 3.5.1. There is currently no resolution.

Workaround:
Use the following steps to workaround this issue:

Note: If you have already replaced the default NSX-T certificate with a CA certificate, do not complete these steps as they will overwrite the CA certificate.

1. Download the attached 67030_cert-gen.zip file.
2. Use a file transfer utility to copy the 67030_cert-gen.zip file to the /home/vcf folder on the SDDC Manager VM.
3. ssh to the SDDC Manager VM as the vcf user.
4. Issue the following command to extract the contents of the /home/vcf/67030_cert-gen.zip file:

unzip 67030_cert-gen.zip

Note: You will see output similar to the following:

Archive:  /tmp/67030_cert-gen.zip
  inflating: /tmp/ec-ec.1-upgrade.py
  inflating: /tmp/getcertificate.py
  inflating: /tmp/setup_logger.py
  inflating: /tmp/certificate_util.py

5. Issue the following command to execute the ec-ec.1-upgrade.py script:

python ec-ec.1-upgrade.py

Note: You will see output similar to the following:

***Starts Execution***
***Ends Execution***

6. In the SDDC Manager UI, restart the failed task.

Additional Information

To be alerted when this article is updated, click the Subscribe to Article link in the Actions box.