Error 'I/O error on POST request for "http://idem-service.prelude.svc.cluster.local:4015/graphql": Read timed out' when Aria Automation is updating the vips on Avi Advanced Load Balancer

Products

• VCF Operations/Automation (formerly VMware Aria Suite)

Issue/Introduction

• Error 'I/O error on POST request for "http://idem-service.prelude.svc.cluster.local:4015/graphql": Read timed out'
• You see the error in Automation when trying to create new OCP cluster with AVI ALB
• You also see the error when trying to update a deployment with OCP Virtual Machines. 
• In the idem-service-task-manager.log you see the below error: 
   ERROR idem-service-task-manager [host=<host> thread=<thread>] com.vmware.ensemble.idem.manager.service.RedisOperationServiceImpl -
   Error occurred while running handleQueueForExpiredTasks. Reason: Text '""' could not be parsed at index 0

Environment

Aria Automation 8.18.1

Cause

 One or more of the idem service instances is experiencing recurring StackOverflowError crashes. This is due to the node handling significantly more traffic than the other nodes under high concurrent load. 

Resolution

Workaround: 

A restart of Aria Automation resolves this temporarily but it is likely to recur. 

This issue is resolved in Aria Automation 8.18.1 Patch 4