[vcf] Unable to deploy new Management Domain with FIPS enabled

Products

• VMware SDDC Manager

Issue/Introduction

Symptoms 

When we perform a management domain deployment with FIPS enabled in the spec file. 

The first attempt of the bringup might fail at this stage, as per the screenshot below. 
If you click the retry after a minute, the check will succeed. 

 

 

/var/log/vmware/vcf/bringup/vcf-bringup.log

YYYY-MM-DDTHH:26:19.048+0000 [bringup,68ad6192d5162e7f587b9ddb5981b7a9,2b88] INFO  [c.v.v.b.c.v1.BringupPublicController,http-nio-0.0.0.0-9080-exec-2] Create SDDC with the following spec: { "....

, "fipsEnabled": true, "esxLicense": "*****", "skipEsxThumbprintValidation": true, "deployWithoutLicenseKeys": false }

YYYY-MM-DDTHH:48:11.326+0000 [bringup,xxxxxxxxxxx5625dac4dd47faf1ca91cff,322d] INFO  [c.v.v.vapi.vsphere.VcApplianceClient,pool-2-thread-13] Found FIPS value false for vCenter_FQDN
YYYY-MM-DDTHH:48:11.374+0000 [bringup,xxxxxxxxxxx5625dac4dd47faf1ca91cff,322d] INFO  [c.v.v.vapi.vsphere.VcApplianceClient,pool-2-thread-13] Updated FIPS to true for vCenter_FQDN
YYYY-MM-DDTHH:48:11.375+0000 [bringup,xxxxxxxxxxx5625dac4dd47faf1ca91cff,322d] INFO  [c.v.e.sddc.common.util.SleepService,pool-2-thread-13] Sleeping for 30 seconds

After 30 sec we get this error.
YYYY-MM-DDTHH:48:41.383+0000 [bringup,xxxxxxxxxxx5625dac4dd47faf1ca91cff,322d] ERROR [c.v.v.s.https.vapi.VapiClientFactory,pool-2-thread-13] Exception occurred during vAPI invocation
java.util.concurrent.ExecutionException: com.vmware.vapi.client.exception.ConnectionException: https://vCenter_FQDN:443/api invocation failed with "org.apache.http.conn.HttpHostConnectException: Connect to vCenter_FQDN:443 [vCenter_FQDN/vCenter_IP] failed: Connection refused"
        at java.base/java.util.concurrent.CompletableFuture.reportGet(CompletableFuture.java:396)
        at java.base/java.util.concurrent.CompletableFuture.get(CompletableFuture.java:2073)
        at com.vmware.vapi.bindings.CompletionStageFuture.get(CompletionStageFuture.java:45)

YYYY-MM-DDTHH:48:41.385+0000 [bringup,xxxxxxxxxxx5625dac4dd47faf1ca91cff,322d] WARN  [c.v.evo.sddc.common.util.TimerUtil,pool-2-thread-13] Operation failed, retrying... 1
java.lang.RuntimeException: Exception occurred during vAPI invocation: java.util.concurrent.ExecutionException: com.vmware.vapi.client.exception.ConnectionException: https://vCenter_FQDN:443/api invocation failed with "org.apache.http.conn.HttpHostConnectException: Connect to vCenter_FQDN:443 [vCenter_FQDN/vCenter_IP] failed: Connection refused"

After 3 re-tries, and when the VC comes up, we see the FIPS enabled for the VC, but by this time, the bringup in SDDC failed.
YYYY-MM-DDTHH:50:15.054+0000 [bringup,xxxxxxxxxxx5625dac4dd47faf1ca91cff,322d] INFO  [c.v.v.vapi.vsphere.VcApplianceClient,pool-2-thread-13] Found FIPS value true for vCenter_FQDN

Hence, when we click the retry operation in bringup, the task completes immediately. 

Environment

VMware SDDC Manager 5.2.x.x

Cause

• During bringup, the workflow fails at the phase of enabling FIPS on vCenter.

• The failure occurs due to an UNAUTHENTICATED error returned by the vpxd service.

• This is expected behavior since the client session was created before the VC restart.

• After the restart, the session is deleted from vpxd memory and becomes invalid.

• When the VcApplianceClient instance closes, it invokes the JsonRpcApiClient close method.

• JsonRpcApiClient Makes two calls to VC just for logging.

• The first call fails  UNAUTHENTICATED because it relies on the invalid pre-restart session.

Resolution

The engineering team is aware of the issue. 

Workaround

Retry the operation in the cloud builder post failure; this will succeed.

Fix 

The issue will be fixed in VCF 9.1.