TMC-SM Cluster Agent Pulling Images from invalid location

Products

• VMware vSphere Kubernetes Service
• VMware Tanzu Mission Control - SM

Issue/Introduction

Pods on newly attached or reattached workload clusters experience image pullback errors after joining to Tanzu Mission Control - Self Managed and do not complete deployment. 
When describing the pods, the wrong image repository is indicated in the output.

Running the following check against the TMC-SM postgres database shows the correct registry for all values:

PGURL=$(kubectl get secret cluster-agent-postgres-creds -n tmc-local -o yaml | yq '.data.PGURL' | base64 -d)
kubectl exec -it postgres-postgresql-0 -n tmc-local -- psql "$PGURL"
SELECT image_registry FROM extension_definition;

However, the postgres pod logs show the maximum number of connections have been made:

==== START logs for container postgresql of pod ###-#####/########-##########-# ====
####-##-## ##:##:##.### GMT [301106] FATAL:  remaining connection slots are reserved for ###-########### superuser connections
####-##-## ##:##:##.### GMT [301105] FATAL:  remaining connection slots are reserved for ###-########### superuser connections
####-##-## ##:##:##.### GMT [301107] FATAL:  remaining connection slots are reserved for ###-########### superuser connections
####-##-## ##:##:##.### GMT [301109] FATAL:  remaining connection slots are reserved for ###-########### superuser connections
####-##-## ##:##:##.### GMT [301110] FATAL:  remaining connection slots are reserved for ###-########### superuser connections

Other processes and jobs within TMC may also be hanging or experiencing slowness. 

Environment

Tanzu Mission Control - Self Managed 1.x

Cause

The TMC-SM deployment was configured with too low of a max session count for the requirements of the TMC-SM deployment
See: Tanzu Mission Control Self-Managed 1.4 - Configure Dependent Services

Resolution

The following database queries can be used to determine what is causing the sessions to be maxed out:

SELECT count(*) FROM pg_stat_activity;
SELECT datname, count(*) FROM pg_stat_activity GROUP BY datname;

Increase the max session count for the TMC-SM postgres by updating the values.yaml 
Use the following KB for guidance to update the max session by updating the configuration in values.yaml - Update TMC Self Managed Registry Post Deployment