Error while preparing SDDC Manager payload for product registration - LCMVCF19113 or Day 2 operations failing from Aria Suite Lifecycle with 'unknown source' error

Products

• VMware Aria Suite

Issue/Introduction

• Product is up and running and is accessible via SSH.
• Unable to trigger day 2 operations for products from Aria Suite Lifecycle. 
• Log traces of failed tasks with error messages similar to :
  Note: Below log trace pertains to an example instance where day 2 tasks fail on of Aria Automation in a VCF aware Aria Suite Lifecycle.

2024-08-01 12:07:34.740 INFO  [http-xxx-8080-exec-9] c.v.v.l.a.g.s.EngineServiceImpl -  -- Error Cause ::
{
  "messageId" : "LCMVCF19113",
  "message" : "Error while preparing SDDC Manager payload for product registration. Check logs at /var/log/vrlcm/vmware_vrlcm.log and try again.",
  "eventId" : "2exxxxce-4xxc-4xx7-bxxd-0dxxxxxxxx4c4",
  "retry" : true,
  "exceptionMessage" : "Error occurred when getting certificate details of  'vra'",
  "exceptionStackTrace" : "com.vmware.vrealize.lcm.vcf.plugin.exception.VCFCertificateException: VMware Aria Automation Certificate chain is empty.\n\tat com.vmware.vrealize.lcm.vcf.plugin.tasks.PrepareVCFProductPayloadForRegistrationTask.fillDetailsForvRA800(PrepareVCFProductPayloadForRegistrationTask.java:257)\n\tat com.vmware.vrealize.lcm.vcf.plugin.tasks.PrepareVCFProductPayloadForRegistrationTask.fillProductDetails(PrepareVCFProductPayloadForRegistrationTask.java:194)\n\tat com.vmware.vrealize.lcm.vcf.plugin.tasks.PrepareVCFProductPayloadForRegistrationTask.execute(PrepareVCFProductPayloadForRegistrationTask.java:138)\n\tat com.vmware.vrealize.lcm.vcf.plugin.tasks.PrepareVCFProductPayloadForRegistrationTask.retry(PrepareVCFProductPayloadForRegistrationTask.java:582)\n\tat com.vmware.vrealize.lcm.automata.core.TaskThread.run(TaskThread.java:60)\n\tat java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)\n\tat java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)\n\tat java.base/java.lang.Thread.run(Unknown Source)\n",
  "localizedMessageId" : "VCF_VCF_PAYLOAD_PREPARATION_ERROR_00006",
  "parameters" : [ "vra" ],
  "properties" : { }

Observe that in the above error snippet the 'properties' parameter is blank as it isn't able to fetch any properties for the product or node due to lost reference. Also observe that the error indicates that the function calls are being run on an 'Unknown Source'.

 

• Additionally, when viewing the environment, We do not see any Product Node references or details being available:
  

Environment

Aria Suite Lifecycle 8.x. 

Cause

Though the error talks about being unable to fetch certificate, it is due to the product reference being corrupted and shown as null. Thus being unable to read certificates from a Null reference, it errors out. 
The product reference may be corrupted for various reasons such as connectivity issues, Manual modifications to the product nodes outside of Aria Suite Lifecycle's awareness. 

Resolution

• Currently, there is no way to fix or restore the existing reference which is corrupted.
• To Workaround this condition, Delete the product instance from Aria Suite Lifecycle without deleting associated VMs and re-import the product back into Aria Suite Lifecycle environment.
  1. Steps to be followed when deleting the corrupted Product instance from the env:
     
     1. Take proper snapshots of Aria Suite Lifecycle and of the VMs associated to the product under consideration from vCenter. Take Back up of the VMs if required. For details on taking a snapshot from vCenter, Refer here.
     2. From the Environment page, select a product instance and right-click on the vertical ellipses.
     3. Click Delete Product.
     4. To ensure you do not delete the associated VMs from vCenter for the selected product, make sure that the Delete associated VMs check box is unchecked. For more details, Refer here.
     5. Submit the Delete Request and wait for the task to complete. Confirm that the VMs are not deleted from vCenter.
  2. Steps to Re-import the Product into:
     
     1. Click on the vertical ellipses on the Environment and select Add Product.
     2. Select Import Radio Button under the Product of being imported and select the associated check box. 
     3. Click next and fill in the required details.
     4. Submit the Import request and wait for the request to be completed. 
     5. Confirm that the product has been imported back in the environment and all node details are visible.