TMC Self-Managed pods of gateway and policy-insights are in CrashLoopBackOff status

Products

• VMware Tanzu Mission Control - SM

Issue/Introduction

The pods of gateway and policy-insights are in CrashLoopBackOff status.

kubectl -n tmc-local get pods | grep -vE 'Running|Completed'

NAME                                                READY   STATUS             RESTARTS        AGE
agent-gateway-server-67cc64f7fd-lm4db               0/1     CrashLoopBackOff   6 (2m25s ago)   8m22s
agent-gateway-server-67cc64f7fd-x2k6s               0/1     CrashLoopBackOff   6 (2m24s ago)   8m22s
api-gateway-server-5bcb9fcbf6-bg7q9                 0/1     CrashLoopBackOff   6 (2m23s ago)   8m21s
api-gateway-server-5bcb9fcbf6-h92gh                 0/1     CrashLoopBackOff   6 (2m17s ago)   8m22s
policy-insights-server-cccb875dc-4lxt2              0/1     CrashLoopBackOff   6 (2m12s ago)   8m26s
policy-insights-server-cccb875dc-trthk              0/1     CrashLoopBackOff   6 (2m3s ago)    8m25s

The pod log messages like:

time="YYYY-MM-DDThh:mm:ssZ" level=warning msg="[core] [Channel #21 SubChannel #22] grpc: addrConn.createTransport failed to connect to {Addr: \"###.tmc.###.###.###.###:443\", ServerName : \"###.tmc.###.###.###.###:443\", }. Err: connection error: desc = \"error reading server preface: remote error: tls: bad certificate\"" subcomponent=grpc-runtime

{"error":"rpc error: code = Unavailable desc = last connection error: connection error: desc = \"error reading server preface: remote error: tls: bad certificate\"","level":"error","msg":"failed to list org information","time":"YYYY-MM-DDThh:mm:ssZZ"}
{"error":"rpc error: code = Unavailable desc = last connection error: connection error: desc = \"error reading server preface: remote error: tls: bad certificate\"","level":"error","msg":"could not load org cache.","time":"YYYY-MM-DDThh:mm:ssZ"}
{"level":"fatal","msg":"Failed to initialize server with err : rpc error: code = Unavailable desc = last connection error: connection error: desc = \"error reading server preface: remote error: tls: bad certificate\"","time":"YYYY-MM-DDThh:mm:ssZ"}

Environment

Tanzu Mission Control - Self Managed

Cause

Using a TLS certificate created with just serverAuth does not allow GTS transport to utilize the custom certificate for internal client-based authentication from service to service.

Prepare Your Cluster to Host Tanzu Mission Control Self-Managed - Importing certificates

Resolution

Regenerate the certificate for stack-tls and landing-service-tls either leave server configuration x509_extensions.extendedKeyUsage to default value or include "serverAuth, clientAuth".

1. Update the server.conf

subjectAltName = @alternate_names
basicConstraints = critical, CA:false
extendedKeyUsage = serverAuth, clientAuth # <<<<<<<<<< ADD
authorityKeyIdentifier=keyid,issuer
keyUsage = digitalSignature, nonRepudiation, keyEncipherment,dataEncipherment

[alternate_names]
DNS.1=###.###.###

2. Regenerate the cert

openssl x509 -req -in server.csr -CA ca.pem -CAkey ca.key  -CAcreateserial -out server.crt -days 365 -sha256 -extfile server.conf