ImageScan failed with TLS certificate error

Products

• VMware Tanzu Application Service for VMs

Issue/Introduction

Workload deployment failed on image-scanner resource. 

$ tanzu apps workload get tanzu-java-web-app
......
📦 Supply Chain
   name:   source-test-scan-to-url
   RESOURCE           READY   HEALTHY   TIME   OUTPUT
   source-provider    True    True      40h    GitRepository/tanzu-java-web-app
   source-tester      True    True      40h    Runnable/tanzu-java-web-app
   image-provider     True    True      40h    Image/tanzu-java-web-app
   image-scanner      False   False     14s    ImageScan/tanzu-java-web-app
   config-provider    False   Unknown   40h    not found
   app-config         False   Unknown   40h    not found
   service-bindings   False   Unknown   40h    not found
   api-descriptors    False   Unknown   40h    not found
   config-writer      False   Unknown   40h    not found
   deliverable        True    True      40h    ConfigMap/tanzu-java-web-app-deliverable

🚚 Delivery
   Delivery resources not found.

💬 Messages
Messages
   Workload [HealthyConditionRule]:   condition status: False, message: Scan job failed: container step-metadata-store-plugin. error: scan job failed. logs:
   ... -format api-json --path /path/to/file.json 1>out.txt 2>error.txt (* Output will be redirected to out.txt and error will be redirected to error.txt .)

In the logs for failed scan pod, it is likely to see certificate errors similar to below:

Error: failed to connect to `host=metadata-store-db user=metadata-store-user database=metadata-store`: failed to write startup message (write failed: tls: failed to verify certificate: x509: certificate has expired or is not yet valid: current time 2023-11-28T05:34:29Z is after 2023-11-06T02:13:07Z)

Environment

Product Version: 1.6

Resolution

Root cause: 

• cert-manager rotates the certificates, but the metadata-store-app and the PostgreSQL db are unaware of the change, and are using the old certificates.

Temporary workaround:

• Delete the metadata-store-app and the metadata-store-db pod one by one.

$ kubectl delete pod metadata-store-app-xxxx -n metadata-store

$ kubectl delete pod metadata-store-db-0 -n metadata-store

Permanent fix:

• Currently, Tanzu product team does have a plan to introduce the permanent fix in TAP (Tanzu Application Platform) 1.8.0. Stay tuned and check the TAP release notes if necessary. Please contact VMware Tanzu support team if you need any further information.