Deploying Aria Suite LCM from SDDC manager fails at 'Request and Configure VMware Aria Suite Lifecycle SSL Certificate' with InvalidArgument and CSR value being blank

Products

VMware SDDC Manager VCF Operations/Automation (formerly VMware Aria Suite)

Issue/Introduction

Deployment of VMware Aria Lifecycle Configuration Manager through SDDC Manager fails during the "Request and Configure VMware Aria Suite Lifecycle SSL Certificate" step. The error occurs due to an SSH timeout when attempting to retrieve the contents of the server.cfg file from the vRSLCM instance, resulting in a certificate error.
- YYYY-MM-DDTHH:MM:SS.349+0000 DEBUG [vcf_dm,UUID,697e] [c.v.evo.sddc.common.util.SshUtil,dm-exec-17] The command [ cat /tmp/ssl_cert_vrslcm/server.cfg ] executed on <vrslcm_fqdn>. Status: -1, Timed out: true
  Output:
  Error:
  YYYY-MM-DDTHH:MM:SS.349+0000 ERROR [vcf_dm,UUID,697e] [c.v.evo.sddc.common.util.SshUtil,dm-exec-17] SSH Execution of command cat /tmp/ssl_cert_vrslcm/server.cfg : Failed (Exit Code -1 ) :
  YYYY-MM-DDTHH:MM:SS.350+0000 DEBUG [vcf_dm,UUID,697e] [c.v.v.s.c.s.SecurityConfigurationServiceImpl,dm-exec-17] Security config retrieved {"fipsMode":false}
  YYYY-MM-DDTHH:MM:SS:56.619+0000 ERROR [vcf_dm,UUID,697e] [c.v.v.vapi.vsphere.VcenterVapiHelper,dm-exec-17] Exception occurred during VC vAPI invocation
  java.util.concurrent.ExecutionException: com.vmware.vapi.std.errors.InvalidArgument: InvalidArgument (com.vmware.vapi.std.errors.invalid_argument) => {
  messages = [LocalizableMessage (com.vmware.vapi.std.localizable_message) => {
  id = com.vmware.certificateauthority.invalidargument,
  defaultMessage = The CSR input specified is invalid (Certificate Signing Request value is blank),
  args = [Certificate Signing Request value is blank],
  params = <null>,
  localized = <null>
  }],
  data = <null>,
  errorType = INVALID_ARGUMENT
The deployment of VMware Aria Lifecycle 8.18.0.24029603 for VCF 5.2 environment via SDDC manager , The details were filled as per the VMware document but while deploying it stayed at 38% completion and later failed with below error.
- Description : Request and Configure VMware Aria Suite Lifecycle SSL Certificate
  Progress Messages : Replacing VMware Aria Suite Lifecycle certificates failed.
  Error Message: Replacing VMware Aria Suite Lifecycle certificates failed.
  Remediation Message: Check if the Jumbo frames between SDDC Manager network and the VMware Aria Suite Lifecycle network are enabled and if the required ports listed at https://ports.esp.vmware.com are open.
  Reference Token: M09N4V
  Cause: InvalidArgument (com.vmware.vapi.std.errors.invalid_argument) => { messages = [LocalizableMessage (com.vmware.vapi.std.localizable_message) => { id = com.vmware.certificateauthority.invalidargument, defaultMessage = The CSR input specified is invalid (Certificate Signing Request value is blank), args = [Certificate Signing Request value is blank], params = <null>, localized = <null> }], data = <null>, errorType = INVALID_ARGUMENT }

Environment

VMware Cloud Foundation 4.x and later
VMware Aria Lifecycle Manager 8.12 and later

Cause

The Tunnel endpoint in NSX and the underlying Infrastructure (Network Switch) had values lower than 9000 Bytes

Resolution

1. Verify Network Settings:

- Ensure that jumbo frames are enabled on all layer 3 gateways involved in the communication between SDDC Manager and Aria Suite Lifecycle.

2. Verify MTU on ESXi hosts.

# esxcfg-nics -l

# esxcfg-vmknic -l

- Reference Link: Testing VMkernel network connectivity with the vmkping command
- Example Output:
  - [root@vcfnode2:~] esxcfg-nics -l
    Name PCI Driver Link Speed Duplex MAC Address MTU Description
    vmnic0 0000:02:00.0 ntg3 Down 0Mbps Half XX:XX:XX:XX:XX:XX 1500 Broadcom Corporation NetXtreme BCM5720 Gigabit Ethernet
    vmnic1 0000:02:00.1 ntg3 Down 0Mbps Half XX:XX:XX:XX:XX:XX 1500 Broadcom Corporation NetXtreme BCM5720 Gigabit Ethernet
    vmnic2 0000:5a:00.0 bnxtnet Up 25000Mbps Full XX:XX:XX:XX:XX:XX 9000 Broadcom NetXtreme E-Series Quad-port 25Gb OCP 3.0 Ethernet Adapter
    vmnic3 0000:5a:00.1 bnxtnet Up 25000Mbps Full XX:XX:XX:XX:XX:XX 9000 Broadcom NetXtreme E-Series Quad-port 25Gb OCP 3.0 Ethernet Adapter
    vmnic4 0000:5a:00.2 bnxtnet Up 25000Mbps Full XX:XX:XX:XX:XX:XX 9000 Broadcom NetXtreme E-Series Quad-port 25Gb OCP 3.0 Ethernet Adapter
    vmnic5 0000:5a:00.3 bnxtnet Up 25000Mbps Full XX:XX:XX:XX:XX:XX 9000 Broadcom NetXtreme E-Series Quad-port 25Gb OCP 3.0 Ethernet Adapter
  - [root@vcfnode2:~] esxcfg-vmknic -l
    Interface Port Group/DVPort/Opaque Network IP Family IP Address Netmask Broadcast MAC Address MTU TSO MSS Enabled Type NetStack
    vmk10 UUID IPv4 XX.XX.XX.XX XX.XX.XX.XX XX.XX.XX.XX XX:XX:XX:XX:XX:XX 9000 65535 true STATIC vxlan
    vmk11 UUID IPv4 XX.XX.XX.XX XX.XX.XX.XX XX.XX.XX.XX XX:XX:XX:XX:XX:XX 9000 65535 true STATIC vxlan
- If the Above output are set to Standard MTU (1500 Bytes) follow the below Broadcom Article to set the MTU to 9000 Bytes.
  - Reference Link: iSCSI and Jumbo Frames configuration on VMware ESXi/ESX

3. Increase MTU Size:

- Increase the Management network MTU size to support jumbo frames (typically set to 9000) for both host TEP and Edge TEP interfaces under Global Fabric Settings in NSX.

4. Check Network Connectivity:

- Confirm that network connectivity is maintained between SDDC Manager and Aria Suite Lifecycle after adjusting the MTU settings.

# ping -M do -s 8700 [destination IP]

- If the ping fails with "message too long," involve the network administrator/vendor to adjust MTU for the underlying infrastructure.

5. Validate SSH Command Execution:

- After making these changes, execute the SSH command to retrieve the server.cfg file manually from the vRSLCM instance:

# ssh [username]@[vrslcm-host] "cat /tmp/ssl_cert_vrslcm/server.cfg"

6. Deploy Aria LCM:

- Once the SSH command succeeds without a timeout, proceed with redeploying Aria LCM through SDDC Manager.