Trivy image scan fails with "image scan error" due to /tmp "no space left on device"

Products

• VMware vSphere Kubernetes Service

Issue/Introduction

YYYY-MM-DDTHH:MM:SS    FATAL    Fatal error    image scan error: scan error: scan failed: failed analysis: analyze error: pipeline error: failed to analyze layer (sha256:deb7d8874f38d4ec281d990aac2c7badbfcd5b97d602a388056e3f918a3f8cc7): post analysis error: post analysis error: Unable to initialize the Java DB: Java DB update failed: OCI artifact error: failed to download Java DB: failed to download artifact from ghcr.io/aquasecurity/trivy-java-db:1: oci download error: copy error: write /tmp/trivy3011653552/javadb.tar.gz: no space left on device
: general response handler: unexpected status code: 500

Cause

Trivy uses the /tmp directory during image scanning. If the image is large or /tmp has insufficient space, the scan may fail. To address this, you can set the TMPDIR environment variable to redirect Trivy's temporary file usage to a directory with adequate storage.

Resolution

1. Set Harbor as Read-Only mode on UI

• • To make the registry read-only, access the Harbor UI as an Administrator.
  • Navigate to Administration > Configuration > System Settings (or General Settings depending on your version).
  • Enable the "Read Only" option and Save the settings.

Impact: Image uploads and deletion would fail, but docker push is expected to work.

2. Create add-trivy-tmpdir-overlay.yaml on the supervisor control plane node 

 

#@ load("@ytt:overlay", "overlay")

#@overlay/match by=overlay.and_op(overlay.subset({"kind": "StatefulSet"}), overlay.subset({"metadata": {"name": "harbor-trivy"}}))

---

spec:

  template:

    spec:

      containers:

        #@overlay/match by="name"

        - name: trivy

          env:

            #@overlay/append

            - name: "TMPDIR"

              value: /home/scanner/.cache  # Corrected path

 

3. Create a generic secret with the overlay 

 

• • kubectl -n vmware-system-supervisor-services create secret generic add-trivy-tmpdir-overlay -o yaml --dry-run=client --from-file=add-trivy-tmpdir-overlay.yaml | kubectl apply -f -

 

4. Patch the Harbor package with the secret (Adjust Namespace and Package Name)

 

• • kubectl -n vmware-system-supervisor-services annotate packageinstalls svc-harbor.tanzu.vmware.com ext.packaging.carvel.dev/ytt-paths-from-secret-name.0=add-trivy-tmpdir-overlay

5. Check if the reconciliation succeeded using the command: 

• • watch kubectl get pkgi -n vmware-system-supervisor-services 

6. Login to the pod and check if the TMPDIR variable is successfully changed: 

• • kubectl exec -it harbor-trivy-0 -n svc-harbor-domain-c8 -- /bin/bash
  • echo $TMPDIR

Expected output: 

 

scanner [ / ]$ echo $TMPDIR

/home/scanner/.cache

 

7. Revert READONLY mode on Harbor UI 

 

Additional Information

https://trivy.dev/v0.42/docs/references/troubleshooting/#running-out-of-space-during-image-scans