VCF 4.2 deployment fails with error "Failed to populate AVN inventory on SDDC Manager during bringup"

Products

• VMware Cloud Foundation

Issue/Introduction

Symptoms:

• In the platform DB, vcf_security_config table is empty.
• In the operations manager service logs (/var/log/vmware/vcf/operationsmanager/operationsmanager.log) FIPS related errors are observed:

2021-07-14T20:35:13.244+0000 ERROR [vcf_om,lfd06919e1a3c7ce,8236] (c.v.v.r.s.utils.VcDataCollector,ra-exec-3] Could not get vCenter vcenter.vmware.com metrics. An exception has occurred.
Lom.vmware.vcf.secure.config.SecurityConfiaxception: Fail to retrieve security configuraticr
at com.vmware.vcf.secure.config.sddc.SecurityConfigurationServiceImpl.isFipsModeEnabled(SecurityConfigurationServiceImpl.java:47) at com.vmare.vcf.secure.config.sddc.SecurityConfigurationServiceimpl.getCipherSuites(SecurityConfigurationServicempl.java:66) at com.vmuare.vcf.secure.config.VcfSslConfiguration.getSslCipherSultes(VcfSslConfiguration. Java: 50) at com.vmmare.vcf.secure.https.ssl.SSLSocketFactoryFactory.createSslConnectionSocketFactory(SSLSocketFactoryFactory.java:48) at com.vmware.evo.sddc.common.client.vmware.AbstractVimManagerFactory.getSSLConnectionSocketFactory(AbstractVimManagerFactory.java:22) at com.vmware.evo.sddc.common.client.vmware.vsphere.VcManagerFactory.createVcManager(VcManagerfactory.java:54) at com.vmware.vcf.ra.service.utils.VcDataCollector.createVcHanager(VcDataCollector.java:261) at com.vmware.vcf.ra.service.utils.VcDataCollector.getVcMetrics(VcDataCollector.java:69) at com.vmware.vcf.ra.service.utils.vsphere.cache.VcMetricsCache.getRealtimeVcMetrics(VcMetricsCache.java:112) at com.vmware.vcf.ra.service.utils.vsphere.cache.VcMetricsCache.loadValue(VdietricsCache.java:80) at com.vmware.evo.sddc.common.cache.Abstractfache$1.1oad(AbstractCache.java:120) at com.vmware.evo.sddc.common.cache.AbstractfacheS1$1.call(AbstractCache.java:136) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at org.springframework.cloud.sleuth.instrument.async.TraceRunnable.run(TraceRunnable.java:67) at java.util.concurrent.ThreadRoolExecutor.runliorker(ThreadRoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor8Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) Caused by: com.vmware.cloud.foundation.rest.commonsvcs.runtime.ApiException: at com.vmware.cloud.foundation.rest.commonsvcs.runtime.ApiClient.handleResponse(ApiClient.java:926) at com.vmare.cloud.foundation.rest.commonsycs.runtime.ApiClient.execute(ApiClient.java:842) at cos.varware.cloud.foundation.rest.commonsycs.service.SecuritySettingsApi.getSecurityConfigliithHttpInfo(SecuritySettingsApi.java:134) at com.vmare.cloud.foundation.rest.commonsvcs.service.SecuritySettingsApi.getSecurityConfig(SecuritySettingsApi.java:121) at com.vmware.vcf.secure.config.sddc.SecurityConfigurationServiceImpl.isFipsModeEnabled(SecurityConfigurationServicelmpl.java:43) ... 16 common frames omitted

 

Environment

VMware Cloud Foundation 4.2

Cause

Race condition during SDDC Manager services startup which prevents proper auto-population of the DB.

Resolution

This issue is fixed in VCF 4.3.

Workaround:
To workaround this issue, please update VCF database by populating vcf_security_config table similar to the below example.

Guidance:

• certificate_verification should be true for new (greenfield) environments and false for upgraded (brownfield) environments.
• Put correct values from the environment in creation_time and modification_time

platform=# select * from vcf_security_config;
                  id | certificate_verification | creation_time | modification_time
--------------------------------------+--------------------------+---------------+-------------------
 ########-####-####-####-########35d3 | t       | 1625770668054 | 1625770668054