SDDC Precheck failing for vRealize Suite products (vROps, vRA and vRLI) even though their nodes are healthy/green

Products

VMware Cloud Foundation

Issue/Introduction

Precheck tasks for vRealize Suite products (vROps, vRA and vRLI) is failing in SDDC Manager but vRSLCM health check request is successful
You see similar error in the file:
```
/var/log/vmware/vcf/lcm/lcm-debug.log
```

YYYY-MM-DDTHH:MM:SS ERROR [vcf_lcm,0000000000000000,0000,precheckId=9c5089db-####-####-####-########605,resourceType=VROPS,resourceId=badc6717-####-####-####-########706:vrops] [c.v.e.s.l.p.i.v.VropsP
recheckRunner,pool-4-thread-23] Unexpected exception occurred during vROps health check
org.springframework.web.client.HttpServerErrorException$InternalServerError: 500 : [no body]
        at org.springframework.web.client.HttpServerErrorException.create(HttpServerErrorException.java:100)
        at org.springframework.web.client.DefaultResponseErrorHandler.handleError(DefaultResponseErrorHandler.java:186)
        at org.springframework.web.client.DefaultResponseErrorHandler.handleError(DefaultResponseErrorHandler.java:125)
        at org.springframework.web.client.ResponseErrorHandler.handleError(ResponseErrorHandler.java:63)
        at org.springframework.web.client.RestTemplate.handleResponse(RestTemplate.java:780)
        at org.springframework.web.client.RestTemplate.doExecute(RestTemplate.java:738)
        at org.springframework.web.client.RestTemplate.execute(RestTemplate.java:672)
        at org.springframework.web.client.RestTemplate.postForEntity(RestTemplate.java:447)
        at com.vmware.evo.sddc.common.vrealize.vrslcm.VrslcmService.checkProductHealth(VrslcmService.java:1260)
        at com.vmware.evo.sddc.lcm.primitive.impl.vrops.VropsPrecheckRunner.performHealthCheckOnVropsVrslcm8x(VropsPrecheckRunner.java:136)
        at com.vmware.evo.sddc.lcm.primitive.impl.vrops.VropsPrecheckRunner.performHealthCheckOnVrops(VropsPrecheckRunner.java:120)
        at com.vmware.evo.sddc.lcm.primitive.impl.vrops.VropsPrecheckRunner.performHealthCheckOnVrops(VropsPrecheckRunner.java:184)
        at com.vmware.evo.sddc.lcm.primitive.impl.vrops.VropsPrecheckRunner.precheckResource(VropsPrecheckRunner.java:64)
        at com.vmware.evo.sddc.lcm.primitive.impl.vrealize.VrslcmPrecheckUpgradeService.processPrecheckTasks(VrslcmPrecheckUpgradeService.java:69)
        at com.vmware.evo.sddc.lcm.primitive.impl.vrealize.VrealizeVrslcmPrimitiveImpl.precheck(VrealizeVrslcmPrimitiveImpl.java:167)
        at com.vmware.evo.sddc.lcm.services.impl.UpgradePrecheckAsyncHandler$1.run(UpgradePrecheckAsyncHandler.java:174)
        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
        at java.util.concurrent.FutureTask.run(FutureTask.java:266)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:748)

You see similar error in the file:

 /var/log/vrlcm/vmware-vrlscm.log

YYYY-MM-DDTHH:MM:SS ERROR [http-nio-8080-exec-8] c.v.v.l.l.c.DeploymentsController -  -- Exception while checking product health: com.vmware.vrealize.lcm.lcops.common.exception.Operatio
nForbiddenException: Another operation: [b3226de5-####-####-####-########260] is already in progress for the environment : badc6717-####-####-####-########706.

Cause

The issue is observed when multiple products are deployed inside the same environment in vRealize Suite Lifecycle Manager. When triggering the Prechecks from SDDC Manager they are triggered in parallel and the health check request for some of the products may fail.

Resolution

The recommendation is to deploy vRA, vROps and vRLI in a separate product environments inside vRealize Suite Lifecycle Manager.

Workaround:

To work around this issue, wait for about 2-3 mins before triggering the precheck again and if the problem still occurs retry the failed precheck only and ignoring the error status in the UI if the health check is completed successfully inside vRSLCM.