VCF 9.x deployment fails at 'Retrieve the status of the VCF Operations collector node addition to the existing VCF Operations' with error Request scaleout failed with error messageId "LCMVROPSYSTEM25127"

Products

• VMware SDDC Manager
• VMware Cloud Foundation

Issue/Introduction

• VCF 9.x deployment fails at Join the existing operations appliance stage.
  
  
  
  
• Under VCF 9 deployment fails 'Retrieve the status of the VCF Operations collector node addition to the existing VCF Operations'. Throws an error message 'Request scaleout failed with error cause [{"messageId":"LCMVROPSYSTEM25127","message":"Error while configuring the adapter"
  
  
  
  
• Under SDDC Manager's /var/log/vmware/vcf/domainmanager/domainmanager.log below log snips are found:
  
  YYYY-MM-DDTHH:MM:SS ERROR [vcf_dm,#####################,####] [c.v.e.s.v.s.GetRequestStatusWithRetryFromVcfOpsMgmtAction,dm-exec-#]  The request createenvironment failed, even after retry attempt with error cause [{"messageId":"LCMVROPSYSTEM25127","message":"Error while configuring the adapter.","eventId":"#####################","retry":true,"exceptionMessage":"Some SDDC Manager hosts are not configured. Failed SDDC Manager hosts are: [FQDN]","exceptionStackTrace":"com.vmware.vrealize.lcm.plugin.common.vrops.exceptions.AdapterConfigurationException: Some SDDC Manager hosts are not configured. Failed SDDC Manager hosts are: [<FQDN>]\n\tat com.vmware.vrealize.lcm.plugin.core.vrops.tasks.VcfAdapterConfigurationTask.execute(VcfAdapterConfigurationTask.java:162)\n\tat com.vmware.vrealize.lcm.plugin.core.vrops.tasks.VcfAdapterConfigurationTask.retry(VcfAdapterConfigurationTask.java:210)\n\tat com.vmware.vrealize.lcm.automata.core.TaskThread.run(TaskThread.java:60)\n\tat java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)\n\tat java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)\n\tat java.base/java.lang.Thread.run(Unknown Source)\n","localizedMessageId":null,"parameters":null,"properties":{"skipTask":"false"}}]
  
  
  
• Under Fleet Manager's /var/log/vrlcm/vmware_vrlcm.log, below log snips are found:
  
  YYYY-MM-DDTHH:MM:SS  INFO vrlcm[####] [pool-3-thread-##] [c.v.v.l.d.v.VropsEndpoint]  -- Response received. Status code: 400 Response data: {"type":"Error","message":"Invalid input format.","moreInformation":[{"name":"errorMessage"," value":"Failed to create VCF cloud account. vCenter instance id is missing for domain: ####"}],"httpStatusCode":400,"apiErrorCode":400}
YYYY-MM-DDTHH:MM:SS  ERROR vrlcm[####] [pool-3-thread-##] [c.v.v.l.p.c.v.t.VcfAdapterConfigurationTask]  -- Failed :: VMware Aria Operations Adapter Configuration
YYYY-MM-DDTHH:MM:SS  ERROR vrlcm[####] [pool-3-thread-91] [c.v.v.l.p.c.v.t.VcfAdapterConfigurationTask]  -- response data: {"type":"Error","message":"Invalid input format.","moreInformation":[{"name":"errorMessage","value":"Failed to create VCF cloud account. vCenter instance id is missing for domain: ####"}],"httpStatusCode":400,"apiErrorCode":400}
YYYY-MM-DDTHH:MM:SS  ERROR vrlcm[####] [pool-3-thread-91] [c.v.v.l.p.c.v.t.VcfAdapterConfigurationTask]  -- response message: 400
YYYY-MM-DDTHH:MM:SS  INFO vrlcm[####] [pool-3-thread-91] [c.v.v.l.p.a.s.Task]  -- Injecting task failure event. Error Code : 'LCMVROPSYSTEM25127', Retry : 'true', Causing Properties : '{ CAUSE :: skipTask ===  }' com.vmware.vrealize.lcm.plugin.common.vrops.exceptions.AdapterConfigurationException: Some SDDC Manager hosts are not configured. Failed SDDC Manager hosts are: [<FQDN>]
        at com.vmware.vrealize.lcm.plugin.core.vrops.tasks.VcfAdapterConfigurationTask.execute(VcfAdapterConfigurationTask.java:162) [vmlcm-vropsplugin-core-9.x-SNAPSHOT.jar!/:?]
        at com.vmware.vrealize.lcm.plugin.core.vrops.tasks.VcfAdapterConfigurationTask.retry(VcfAdapterConfigurationTask.java:210) [vmlcm-vropsplugin-core-9.x-SNAPSHOT.jar!/:?]
        at com.vmware.vrealize.lcm.automata.core.TaskThread.run(TaskThread.java:60) [vmlcm-engineservice-core-9.x-SNAPSHOT.jar!/:?]
  
  

Environment

• VMware Cloud Foundation 9.x
• SDDC 9.x

Cause

The state of the Platform Services Controller (PSC), vCenter Server, or ESX host is not set to ACTIVE in the SDDC manager PostgreSQL database.

Resolution

To resolve this issue, follow below steps:

1. Take a snapshot of the SDDC Manager VM.
2. Connect to the SDDC manager appliance via SSH using the vcf user and switch to the root user.
3. Connect to the SDDC Manager Platform database, using below command:
   
   psql -h localhost -U postgres -d platform
   
   
4. Check the status for PSC, vCenter and ESX, using below commands:
   
   SELECT id, vm_hostname, status FROM vcenter;
SELECT id, hostname, status FROM host;
SELECT id, vm_hostname, status FROM psc;
   
   Example O/p:
   

   platform=# SELECT id, vm_hostname, status FROM vcenter;
                  id                  |          vm_hostname           | status
--------------------------------------+--------------------------------+--------
 ########-####-####-####-############ |          vcenter-fqdn          | ERROR
(1 row)

   platform=# SELECT id, hostname, status FROM host;
                  id                  |         hostname          | status
--------------------------------------+---------------------------+--------
 ########-####-####-####-############ |         esxi-fqdn         | ERROR
(1 rows)

   platform=# SELECT id, vm_hostname, status FROM psc;
                  id                  |          vm_hostname           | status
--------------------------------------+--------------------------------+--------
 ########-####-####-####-############ |            psc-fqdn            | ERROR
(1 row)

5. Update the status of the components in ERROR status to ACTIVE, using the below commands:
   
   UPDATE vcenter SET state = 'ACTIVE' WHERE id = '<ID>';
UPDATE host SET state = 'ACTIVE' WHERE id = '<ID>';
UPDATE psc SET state = 'ACTIVE' WHERE id = '<ID>';
   
   Example O/p:
   
   platform=# UPDATE vcenter SET status = 'ACTIVE' WHERE id = '########-####-####-####-############';
UPDATE 1
platform=# UPDATE host SET status = 'ACTIVE' WHERE id = '########-####-####-####-############';
UPDATE 1
platform=# UPDATE psc SET status = 'ACTIVE' WHERE id = '########-####-####-####-############';
UPDATE 1
   
   
   
6. Exit the database, using below command:
   
   \q
   
   
7. Restart the SDDC manager services, using the below command:
   
   /opt/vmware/vcf/operationsmanager/scripts/cli/sddcmanager_restart_services.sh


8. Retry VCF deployment task from the VCF Installer UI.