The Postgres pod in TCA Manager fail to start due to an expired certificate

Products

• VMware Telco Cloud Automation

Issue/Introduction

TCA pods are unable to start because the Postgres pod is down due to expired certificates.

Environment

3.x

Cause

TCA Pods cannot connect to postgres instance as the certificates have expired.

kubectl get pods -A | grep -i postgres

postgres-operator-system postgres-operator-57XXXXXXXX-XXXXX 1/1 Running 0 21m
tca-mgr postgres-0         4/5 Running 6 (4m33s ago) 21m
tca-mgr postgres-monitor-0 3/4 CrashLoopBackOff 8 (3m23s ago) 21m

 

2024-10-16T07:46:23.682Z INFO postgresinstance failed to connect to `host=postgres-monitor-0.postgres-agent.tca-mgr.svc.cluster.local user=autoctl_node database=pg_auto_failover`: dial error (dial tcp 100.xxx.xxx.xxx:5432: connect: connection refused)

2024-10-16T07:46:28.690Z INFO postgresinstance failed to connect to `host=postgres-monitor-0.postgres-agent.tca-mgr.svc.cluster.local user=autoctl_node database=pg_auto_failover`: tls error (x509: certificate has expired or is not yet valid: current time 2024-10-16T07:46:28Z is after 2024-10-09T14:31:50Z)

Resolution

Below are the steps to renew postgres certificates :

kubectl delete secret -n postgres-operator-system postgres-operator-webhook-server-cert

[For TCA-CP]

kubectl delete secret -n tca-cp-cn postgres-internal-ssl-secret
kubectl delete secret -n tca-cp-cn postgres-metrics-tls-secret

[For TCA-M]

kubectl delete secret -n tca-mgr postgres-internal-ssl-secret
kubectl delete secret -n tca-mgr postgres-metrics-tls-secret

Post deleting the secret, please reboot the appliances.

Additional Information

To verify the certificates, use the following commands.

kubectl get secret -n postgres-internal-ssl-secret tca-cp-cn -o jsonpath='{.data.tls\.crt}' | base64 -d | openssl x509 -noout -text

kubectl get secret -n postgres-metrics-tls-secret tca-cp-cn -o jsonpath='{.data.tls\.crt}' | base64 -d | openssl x509 -noout -text