TAP scan job failed due to certificate verification failure

Products

• VMware Tanzu Application Platform

Issue/Introduction

The TAP Scan job might fail with certificate verification error as shown by the following example.

$ kubectl logs scan-daily-mgm-saga-####-pod -n ### -c step-metadata-store-plugin
Flag --cyclonedxtype has been deprecated, will be removed in version "v2.0.0". Use "input-format" instead.

Warning: You are using an older version of the SCST - Store. Please upgrade to v1.5.0 and above.

?  Error: Post "https://metadata-store-app.metadata-store.svc.cluster.local:8443/api/sourceReport?": tls: failed to verify certificate: x509: certificate has expired or is not yet valid: current time 2025-09-12T01:31:29Z is after 2025-09-11T07:01:02Z

 

Environment

Tanzu Application Platform

Cause

 

When TAP deploys metadata-store package, it will create two certificate resources app-tls-ca-cert/app-tls-cert to be consumed by the metadata-store endpoint (https://metadata-store-app.metadata-store.svc.cluster.local:8443).
The certificate resources app-tls-ca-cert/app-tls-cert are associated with TLS type secret respectively.

$ kubectl -n metadata-store get certificate,secret | grep app-tls
certificate.cert-manager.io/app-tls-ca-cert   True    app-tls-ca-cert   4d21h
certificate.cert-manager.io/app-tls-cert      True    app-tls-cert      5d20h
secret/app-tls-ca-cert                     kubernetes.io/tls                     3      4d21h
secret/app-tls-cert                        kubernetes.io/tls                     3      5d20h

Secret/app-tls-ca-cert has same value for "ca.crt" and "tls.crt" fields, which is the CA certificate to sign the leaf certificate stored in secret/app-tls-cert.

And "ca.crt" field in secret/app-tls-cert is equivalent to that of secret/app-tls-ca-cert, while the "tls.crt" filed is the leaf certificate which is signed by "ca.crt" of secret/app-tls-cert. That means "ca.crt" of secret/app-tls-cert will be used to verify the certificate ("tls.crt" of secret/app-tls-cert) returned from endpoint metadata-store-app.metadata-store.svc.cluster.local:8443.

$ kubectl -n metadata-store get secret/app-tls-ca-cert -o yaml
apiVersion: v1
data:
  ca.crt: ####
  tls.crt: ####
  tls.key: ####
kind: Secret
metadata:
  annotations:
    cert-manager.io/alt-names: ""
    cert-manager.io/certificate-name: app-tls-ca-cert
    cert-manager.io/common-name: ca-cert
    cert-manager.io/ip-sans: ""
    cert-manager.io/issuer-group: ""
    cert-manager.io/issuer-kind: ""
    cert-manager.io/issuer-name: ca-issuer
    cert-manager.io/uri-sans: ""
  labels:
    controller.cert-manager.io/fao: "true"
  name: app-tls-ca-cert
  namespace: metadata-store
type: kubernetes.io/tls

$ kubectl -n metadata-store get secret/app-tls-cert -o yaml
apiVersion: v1
data:
  ca.crt: ####
  tls.crt: ####
  tls.key: ####
kind: Secret
metadata:
  annotations:
    cert-manager.io/alt-names: metadata-store-app,metadata-store-app.metadata-store.svc.cluster.local
    cert-manager.io/certificate-name: app-tls-cert
    cert-manager.io/common-name: ""
    cert-manager.io/ip-sans: ""
    cert-manager.io/issuer-group: ""
    cert-manager.io/issuer-kind: ""
    cert-manager.io/issuer-name: cert-issuer
    cert-manager.io/uri-sans: ""
  labels:
    controller.cert-manager.io/fao: "true"
  name: app-tls-cert
  namespace: metadata-store
type: kubernetes.io/tls

The certificate resources app-tls-ca-cert/app-tls-cert are managed by cert-manager, which will automatically renew these certificates by default prior to expiration date. Refer to TAP document for more details. 

According to design of cert-manager, "ca.crt" of secret/app-tls-cert will not be automatically updated with new CA value of secret/app-tls-ca-cert after it's renewed. So when the "old" "ca.crt" of secret/app-tls-cert becomes expired, the aforementioned verification failure will happen for accessing endpoint like metadata-store-app.metadata-store.svc.cluster.local:8443.

Resolution

Manual intervention is required to rectify the situation. 

You can delete resource secret/app-tls-cert, which will be recreated automatically by cert-manager with latest CA value in secret/app-tls-ca-cert. 

$ kubectl -n metadata-store delete secret/app-tls-cert 

Or you can also try cmctl renew as mentioned in cert-manager document.