Tanzu Mission Control Self-Managed (TMC-SM) x509 unknown authority due to malformed trustedCAs

Products

• VMware Tanzu Mission Control

Issue/Introduction

Tanzu Mission Control Self-Managed (TMC-SM) deployment fails during reconciliation. Core application pods, including auth-manager-server, s3-access-operator, and api-gateway-server, report the following exact error string in their logs:

tls: failed to verify certificate: x509: certificate signed by unknown authority

The s3-access-operator pod is unable to authenticate to the backend storage, resulting in a failure to generate the required audit-s3-creds secret. Consequently, dependent pods remain in a Error state.

Diagnostic logs from the tmc-local namespace confirm the TLS handshake failures:

level=error msg="Unable to retrieve metadata: Get \"https://pinniped-supervisor.<REDACTED_HOSTNAMES>/provider/pinniped/.well-known/openid-configuration\": tls: failed to verify certificate: x509: certificate signed by unknown authority" idp=oidc-pinniped error in retrieving credentials value: RequestError: send request failed\ncaused by: Post \"https://s3.<REDACTED_HOSTNAMES>/\": tls: failed to verify certificate: x509: certificate signed by unknown authority

Cause

The trustedCAs block in the deployment's values.yaml file is missing the YAML literal block scalar operator (|- or |). This structural formatting fault causes the YAML parser to fold the multi-line PEM certificate into a single continuous string. Consequently, the required carriage returns are replaced by spaces when the tls-ca-bundles ConfigMap is generated. The resulting malformed trust store prevents the application pods from parsing the custom Root CA, inducing the TLS verification failures.

Visual Representation of the Generated tls-ca-bundles ConfigMap:

# INCORRECT (Malformed ConfigMap resulting from missing |- in values.yaml):
# The certificate is improperly folded into a single line separated by spaces.

    -----BEGIN CERTIFICATE----- MIIDzTCCArWgBwIBAgIQC3p... <base64 payload with spaces> ...yO8K1a -----END CERTIFICATE-----

# CORRECT (Proper ConfigMap resulting from using |- in values.yaml):
# The certificate maintains strict X.509 carriage returns.

    -----BEGIN CERTIFICATE-----
    MIIDzTCCBrWgAwIBAgIQC3p...
    <base64 payload with explicit newlines>
    ...yO8K1a
    -----END CERTIFICATE-----

Resolution

1. Access the values.yaml file utilized for the Tanzu Mission Control Self-Managed deployment.
2. Locate the trustedCAs configuration block.
3. Inject the literal block scalar operator (|- or |) immediately after the descriptive key name for the custom CA.
4. Format the PEM certificate payload to ensure explicit line breaks separate the header, base64 body, and footer. Example:

   trustedCAs:
     "custom-ca": |-
       -----BEGIN CERTIFICATE-----
       <REDACTED_SECRETS>
       -----END CERTIFICATE-----

5. Apply the corrected configuration to the cluster using the Tanzu CLI: tanzu package installed update tanzu-mission-control -p tmc.tanzu.vmware.com --version <VERSION_REQUIRED> --values-file <path-to-values.yaml> --namespace tmc-local
   
   Note: This command may be different if installing with the tmc-sm CLI: Install Using the tmc-sm CLI
   
   
6. Verify the tls-ca-bundles ConfigMap in the tmc-local namespace regenerates with explicit newlines.
7. Confirm the s3-access-operator pod successfully authenticates and generates the audit-s3-creds secret, allowing dependent pods to initialize.

Additional Information

References:

• Preparing your cluster to host Tanzu Mission Control Self-Managed
• Install Using the tmc-sm CLI
• Install Using the Tanzu CLI