SDDC Precheck failing for vRealize Suite products (vROps, vRA and vRLI) even though their nodes are healthy/green

Products

• VMware Cloud Foundation

Issue/Introduction

• Precheck tasks for vRealize Suite products (vROps, vRA and vRLI) is failing in SDDC Manager but vRSLCM health check request is successful

• In the /var/log/vmware/vcf/lcm/lcm-debug.log file, you see similar to:

2021-06-24T12:56:04.847+0000 ERROR [vcf_lcm,0000000000000000,0000,precheckId=9c5089db-7174-4af2-aa8e-4bbb16ec6605,resourceType=VROPS,resourceId=badc6717-320d-4e06-8933-bc5fc30d8706:vrops] [c.v.e.s.l.p.i.v.VropsP
recheckRunner,pool-4-thread-23] Unexpected exception occurred during vROps health check
org.springframework.web.client.HttpServerErrorException$InternalServerError: 500 : [no body]
        at org.springframework.web.client.HttpServerErrorException.create(HttpServerErrorException.java:100)
        at org.springframework.web.client.DefaultResponseErrorHandler.handleError(DefaultResponseErrorHandler.java:186)
        at org.springframework.web.client.DefaultResponseErrorHandler.handleError(DefaultResponseErrorHandler.java:125)
        at org.springframework.web.client.ResponseErrorHandler.handleError(ResponseErrorHandler.java:63)
        at org.springframework.web.client.RestTemplate.handleResponse(RestTemplate.java:780)
        at org.springframework.web.client.RestTemplate.doExecute(RestTemplate.java:738)
        at org.springframework.web.client.RestTemplate.execute(RestTemplate.java:672)
        at org.springframework.web.client.RestTemplate.postForEntity(RestTemplate.java:447)
        at com.vmware.evo.sddc.common.vrealize.vrslcm.VrslcmService.checkProductHealth(VrslcmService.java:1260)
        at com.vmware.evo.sddc.lcm.primitive.impl.vrops.VropsPrecheckRunner.performHealthCheckOnVropsVrslcm8x(VropsPrecheckRunner.java:136)
        at com.vmware.evo.sddc.lcm.primitive.impl.vrops.VropsPrecheckRunner.performHealthCheckOnVrops(VropsPrecheckRunner.java:120)
        at com.vmware.evo.sddc.lcm.primitive.impl.vrops.VropsPrecheckRunner.performHealthCheckOnVrops(VropsPrecheckRunner.java:184)
        at com.vmware.evo.sddc.lcm.primitive.impl.vrops.VropsPrecheckRunner.precheckResource(VropsPrecheckRunner.java:64)
        at com.vmware.evo.sddc.lcm.primitive.impl.vrealize.VrslcmPrecheckUpgradeService.processPrecheckTasks(VrslcmPrecheckUpgradeService.java:69)
        at com.vmware.evo.sddc.lcm.primitive.impl.vrealize.VrealizeVrslcmPrimitiveImpl.precheck(VrealizeVrslcmPrimitiveImpl.java:167)
        at com.vmware.evo.sddc.lcm.services.impl.UpgradePrecheckAsyncHandler$1.run(UpgradePrecheckAsyncHandler.java:174)
        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
        at java.util.concurrent.FutureTask.run(FutureTask.java:266)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:748)

• In the /var/log/vrlcm/vmware-vrlscm.log file, you see similar to:

2021-06-24 13:17:39.482 ERROR [http-nio-8080-exec-8] c.v.v.l.l.c.DeploymentsController -  -- Exception while checking product health: com.vmware.vrealize.lcm.lcops.common.exception.Operatio
nForbiddenException: Another operation: [b3226de5-dd0b-4ae5-82d0-6fc08d161260] is already in progress for the environment : badc6717-320d-4e06-8933-bc5fc30d8706.

Cause

The issue is observed when multiple products are deployed inside the same environment in vRealize Suite Lifecycle Manager. When triggering the Prechecks from SDDC Manager they are triggered in parallel and the health check request for some of the products may fail.

Resolution

The recommendation is to deploy vRA, vROps and vRLI in a separate product environments inside vRealize Suite Lifecycle Manager.

Workaround:
To workaround this issue wait for about 2-3 mins before triggering the precheck again and if the problem still occurs retry the failed precheck only and ignoring the error status in the UI if the health check is completed successfully inside vRSLCM.