Aria Automation fails to list catalog items in service broker

Products

• VCF Operations/Automation (formerly VMware Aria Suite)

Issue/Introduction

• When attempting to view catalog items from Aria Automation UI > Service Broker > Catalog > fails to load any catalogs items.
• The catalog items are published from Assembler and were previously listed. 
• When viewing the content source, the number of items tab loads 0 items as below:
  
  
  
• Attempting to sync / import content fails with authentication token error.
• Catalog-service-app.log contains information similar to:
  ERROR catalog-service-app [host='catalog-service-app-<pod-id>' thread='boundedElastic-<thread-id>' user='' org='' trace='<trace_id>'] c.v.t.d.service.ResourceSyncServiceImpl - Failed to sync resource with id: <resource_id> of type: '<type_of_resource>' and origin: 'DISCOVERED' with error: finishConnect(..) failed: Connection refused: identity-service.prelude.svc.cluster.local/<pod_internal_ip>:8000
or
Suppressed: java.lang.Exception: #block terminated with an error
Caused by: io.netty.channel.AbstractChannel$AnnotatedConnectException: finishConnect(..) failed: Connection refused: identity-service.prelude.svc.cluster.local/
Caused by: java.net.ConnectException: finishConnect(..) failed: Connection refused

Environment

Aria Automation 8.x 

Cause

The Identity-service-pod may have lost connection to the vIDM and thus failing to fetch required authentication token used for inter-pod communications to fetch required details. 

Resolution

• Ensure vIDM cluster is up and running healthily and the UI is accessible as expected. 
• Validate Cluster health on vIDM Administrator portal > Dashboards > System diagnostics Dashboard : Validate all checks are green.
• Validate output of vracli vidm command run on an SSH for the Aria Automation node.
  • Confirm the idm host, user, certificate and other configuration fields have values as expected
• If there have been no configuration level changes on vIDM, SSH into any node of vIDM and run /opt/scripts/deploy.sh. 
  • If the /deploy.sh script fails, head over to Aria Suite Lifecycle > Lifecycle Operations > Environments > Aria Automation env > Vertical ellipses > Re-trust with vIDM (would recreate the pods as a part of the re-trust)
• Once ready, Navigate to Aria Automation UI > Service broker > Content & Policies > Re-sync the Content sources.
• Validate the catalog items are listed as expected.