LCMVMSP10002 error when deploying VCF Automation 9.0

Products

VCF Operations

Issue/Introduction

Symptoms:

Initial setup of VCF Automation fails during the deployment phase. The installer becomes unresponsive during the final stages of the process while waiting for the automation components to finalize their configuration.

The deployment logs show the below failure (the text has been truncated to only show the relevant error):

Request create environment failed with error cause [{"messageId":"LCMVMSP10002","message":"Failed to deploy product on the application platform."," 
...
failed to create resource: HTTPRoute.gateway.networking.k8s.io \"vcfa-common-default-route\" is invalid: [spec.hostnames[0]: Invalid value: \"###\": spec.hostnames[0] in body should match '^(\\*\\.)?[a-z0-9]([-a-z0-9]*[a-z0-9])?(\\.[a-z0-9]([-a-z0-9]*[a-z0-9])?)*$',

Deploying VCF Automation 9.0 from VCF Operations Fleet Management fails with error code LCMVMSP10002. In the request flow chart under VCF Operations > Fleet Management > Lifecycle > VCF Management > Tasks > (vcfa deployment task), we see the below error message:
```
Task failure on Stage 8 shows issues similar to:
Request Failed
LCMVMSP10002
Error Code: LCMVMSP10002
Failed to deploy product on the application platform.
Product deployment failed.
```

In the Fleet management appliance, vmware_vrlcm.log shows log messages similar to:

INFO vrlcm[####] [pool-3-thread-63] [c.v.v.l.p.a.s.Task]  -- Injecting task failure event. Error Code : 'LCMVMSP10002', Retry : 'true', Causing Properties : '{ CAUSE ::  }'
com.vmware.vrealize.lcm.vmsp.common.exception.DeploymentFailedException: No failed pods found.
and
"localizedMessageId" : "LCM_VMSP_PKG_PUSH_ERROR_0001",
and
error: context deadline exceeded

Environment

VCF Automation 9.0.x
VCF Operations 9.0.x

Cause

This can be caused by capital letters in the defined FQDN during the deployment wizard.

The environment file in the failed VCFA log bundle shows:

Helm upgrade failed for release prelude/vcfa-common with chart [email protected]: failed to create resource: HTTPRoute.gateway.networking.k8s.io "vcfa-common-default-route" is invalid: [spec.hostnames[0]: Invalid value: "CAPITALHOSTNAME.example.com" spec.hostnames[0] in body should match...

The vmware_vrlcm.log file can be searched for easy identification:

Search for DEPLOYMENT_FAILED in the logs and in the section labeled "productId" : "vra". Look for "fqdn" and see if there are CAPITAL letters in the FQDN.

{
  "productId" : "vra",
  "vcfInstances" : [ ],
  "componentDTO" : {
    "vmid" : "########-####-####-####-###########",
    "tenant" : "<tenantname>",
    "componentVmid" : "########-####-####-####-###########",
    "componentType" : "vra",
    "currentVersion" : "9.0.0.0",
    "environmentId" : "########-####-####-####-###########",
    "fqdn" : "CAPITALHOSTNAME.example.com",
    "componentStatus" : "FAILED",
    "engineRequestId" : "########-####-####-####-###########",
    "fleet" : true,
    "properties" : { },
    "references" : [ ],
    "status" : "DEPLOYMENT_FAILED",
    "action" : "VIEW_DETAILS",
    "compatibleVersions" : [ ]

Resolution

The provided FQDNs must be specified in lowercase during the Automation deployment wizard in VCF Operations > Fleet Management > Lifecycle.

Additional Information

If the provided FQDN is already in lowercase, then retry the failed request from VCF Operations > Fleet Management > Lifecycle > VCF Management > Tasks.

Gather the failed VCFA log bundle using: Generating a log bundle from a failed VCF Automation or VCF Identity Broker component deployment.

Note: The deployment of the SDDC appliance will fail if the DNS record for the SDDC appliance contains uppercase characters.

VMware Cloud Foundation deployment fails if the SDDC Manager FQDN includes uppercase letters