Error Code: LCMVROPSSYSTEM29002 Message: “Operations collector group operation failure – Wait timeout, Cloud Proxy node didn’t join the Operations cluster
Article ID: 430069
Updated On:
Products
Issue/Introduction
- In the VCF Operations UI, Fleet Management >> Lifecycle >> Tasks displays the error:
LCMVROPSSYSTEM29002 - The Fleet management logs show
Filename:/var/log/vrlcm/vmware-vrlcm.log
INFO vrlcm[1243] [http-nio-8080-exec-10] [c.v.v.l.a.g.s.EngineServiceImpl] – Error Cause ::{"messageId" : "LCMVROPSSYSTEM29002","message" : "Operations collector group operation failure","eventId" : "########-####-####-####-##########","retry" : true,"exceptionMessage" : "Wait timeout, Cloud Proxy node didn't join the Operations cluster, retry after sometime.","exceptionStackTrace" : "com.vmware.vrealize.lcm.plugin.common.vrops.exceptions.CollectorGroupOperationsException: Wait timeout, Cloud Proxy node didn't join the Operations cluster, retry after sometime.\n\tat com.vmware.vrealize.lcm.plugin.core.vrops.tasks.VropsOnPremCpNodeWaitForStartTask.execute(VropsOnPremCpNodeWaitForStartTask.java:67)\n\tat com.vmware.vrealize.lcm.plugin.core.vrops.tasks.VropsOnPremCpNodeWaitForStartTask.retry(VropsOnPremCpNodeWaitForStartTask.java:122)\n\tat com.vmware.vrealize.lcm.automata.core.TaskThread.run(TaskThread.java:60)\n\tat java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)\n\tat java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)\n\tat java.base/java.lang.Thread.run(Unknown Source)\n","localizedMessageId" : "VROPS_ON_PREM_CP_NODE_WAIT_FOR_START_TASK_00002","parameters" : null,"properties" :{ "deleteVm" : "false" }}
- VCF Installer displays 75% completed, Retrieve the status of the VCF Operations collector node addition to existing VCF Operations Failed
- VCF Operations Cloud Proxy logs
Filename: /var/log/haproxy-admin.log
YYYY-MM-DD:HH:MM:Sec:Msec+00:00 localhost haproxy[9519]: backend PrxyRC_CRUSH_FTP_DEV_BE has no server available!YYYY-MM-DD:HH:MM:Sec:Msec:00 localhost haproxy[9519]: Server PrxyRC_CRUSH_FTP_STAGE_BE/CRUSH_FTP_STG_0 is DOWN, reason: Layer4 connection problem, info: "Connection timed out", check duration: 130425ms. 0 active and 0 backup servers left. 0 sessions active, 0 requeued, 0 remaining in queue.YYYY-MM-DD:HH:MM:Sec:Msec00 localhost haproxy[9519]: backend PrxyRC_CRUSH_FTP_STAGE_BE has no server available!YYYY-MM-DD:HH:MM:Sec:Msec:00 localhost haproxy[9519]: Server PrxyRC_CRUSH_FTP_PROD_BE/CRUSH_FTP_PROD_0 is DOWN, reason: Layer4 connection problem, info: "Connection timed out", check duration: 130141ms. 0 active and 0 backup servers left. 0 sessions active, 0 requeued, 0 remaining in queue.YYYY-MM-DD:HH:MM:Sec:Msec:00 localhost haproxy[9519]: backend PrxyRC_CRUSH_FTP_PROD_BE has no server available!YYYY-MM-DD:HH:MM:Sec:Msec:00 localhost haproxy[9519]: Server PrxyRC_CRUSH_FTP_DEV_BE/CRUSH_FTP_DEV_0 was DOWN and now enters maintenance (DNS NX status).
Filename: /var/log/haproxy-traffic.log
YYYY-MM-DD:HH:MM:Sec:Msec:00 localhost haproxy[9519]: 127.0.0.1:37348 [DD/MM/YYYY:HH:MM:Sec.026] PrxyRC_FE~ PrxyRC_BE/VROPS_1 0/0/13/1026/1039 401 733 - - ---- 1/1/0/0/0 0/0 "POST /suite-api/api/auth/token/acquire?_no_links=true HTTP/1.1"YYYY-MM-DD:HH:MM:Sec:Msec:00 localhost haproxy[9519]: 127.0.0.1:37358 [DD/MM/YYYY:HH:MM:Sec.847] PrxyRC_FE~ PrxyRC_BE/VROPS_0 0/0/13/3030/3043 401 733 - - ---- 2/2/0/0/0 0/0 "POST /suite-api/api/auth/token/acquire HTTP/1.1"YYYY-MM-DD:HH:MM:Sec:Msec:00 localhost haproxy[9519]: 127.0.0.1:52742 [DD/MM/YYYY:HH:MM:Sec.073] PrxyRC_FE~ PrxyRC_BE/VROPS_2 0/0/12/5026/5038 401 733 - - ---- 1/1/0/0/0 0/0 "POST /suite-api/api/auth/token/acquire?_no_links=true HTTP/1.1"YYYY-MM-DD:HH:MM:Sec:Msec:00 localhost haproxy[9519]: 127.0.0.1:40002 [DD/MM/YYYY:HH:MM:Sec.117] PrxyRC_FE~ PrxyRC_BE/VROPS_1 0/0/14/5027/5041 401 733 - - ---- 1/1/0/0/0 0/0 "POST /suite-api/api/auth/token/acquire?_no_links=true HTTP/1.1"YYYY-MM-DD:HH:MM:Sec:Msec:00 localhost haproxy[9519]: 127.0.0.1:60406 [DD/MM/YYYY:HH:MM:Sec.166] PrxyRC_FE~ PrxyRC_BE/VROPS_0 0/0/13/3030/3043 401 733 - - ---- 1/1/0/0/0 0/0 "POST /suite-api/api/auth/token/acquire?_no_links=true HTTP/1.1"YYYY-MM-DD:HH:MM:Sec:Msec:00 localhost haproxy[9519]: 127.0.0.1:60418 [DD/MM/YYYY:HH:MM:Sec.238] PrxyRC_FE~ PrxyRC_BE/VROPS_2 0/0/13/22/35 401 946 - - ---- 2/2/0/0/0 0/0 "POST /casa/authorize HTTP/1.1"
Filename: /storage/log/vcops/log/collector.log
YYYY-MM-DD:HH:MM:Sec:MsecYYYY-MM-DD:HH:MM:Sec:Msec
Environment
VMware Cloud Foundation Operations 9.x
VMware Cloud Foundation Installer 9.x
Resolution
Take a snapshot of the VCF Operations instance (referencing the guide:Snapshot Creation in VMware Aria Operations)
In the VCF Operations UI, from the left menu, navigate to Administration >> Cloud Proxies. Click the Add button and generate a new OTK key.
In vCenter, select the target Cloud Proxy and powered it off.
Navigate to Configure >> vApp Options and select the OTK property. Scroll up, click Set Value, delete the old OTK, and enter the new key generated in VCF Operations (step 2).
Power on the Cloud proxy. The Cloud Proxy will automatically appear as going online VCF Operations.
SSH into the new Cloud Proxy using root credentials.
Edit the/etc/hostsfile and comment out the line<VM GATEWAY_IP> eapi.broadcom.com(by adding a#at the beginning of the line).Re-initiate the Cloud Proxy deployment request from the VCF installer. Then, in the VCF Operations UI, Fleet Management, retry the failed Cloud Proxy deployment task.
Both tasks should finish successfully, and the Cloud Proxy will now show up in the VCF Operations integrations under the VMware Cloud Foundation adapter instances.
Feedback
