Trivy Image Scan Failure Due to Insufficient Space in /tmp Directory

Products

• VMware vSphere Kubernetes Service

Issue/Introduction

When using Trivy for image scanning, the scan may fail if the image is large or if there is insufficient space in the /tmp directory. Trivy uses the /tmp directory to store temporary files during the image scan. If the available space in the /tmp directory is insufficient, the scan will fail with the following error:

image scan failed:
failed to copy the image:
write /tmp/fanal-#########: no space left on device

Environment

VMware vSphere with Tanzu
VMware Tanzu Kubernetes Grid Multicloud
All Harbor Package versions lower than  v2.12.2_vmware.1-tkg.2

Cause

Trivy uses the /tmp directory during the image scan. If the image is large or if /tmp has insufficient space, the scan will fail due to lack of space.

Resolution

For Harbor Version >= v2.12.2_vmware.1-tkg.2

You can configure the TMPDIR environment variable directly through the values.yaml file. Follow these steps:

trivy:
  # Enable the flag to enable Trivy scanner
  enabled: true
  # If /tmp doesn't work for you, you can change it by setting this environment variable
  # This is an optional setting; it will use the default /tmp dir when not specified
  # https://aquasecurity.github.io/trivy/v0.54/docs/references/troubleshooting/
  tmpdir: /path/to/large_storage_dir

 

For Harbor Version < v2.12.2_vmware.1-tkg.2

If you are using a Harbor version older than v2.12.2_vmware.1-tkg.2, follow these steps to set the TMPDIR environment variable.

Step 1: Set Harbor to READONLY mode on the UI.

Step 2: Create a file named add-trivy-tmpdir-overlay.yaml with the following content:

#@ load("@ytt:overlay", "overlay")

#@overlay/match by=overlay.and_op(overlay.subset({"kind": "StatefulSet"}), overlay.subset({"metadata": {"name": "harbor-trivy"}}))
---
spec:
  template:
    spec:
      containers:
        #@overlay/match by="name"
        - name: trivy
          env:
            #@overlay/append
            - name: "TMPDIR"
              value: /home/scanner/.cache

 

Step 3: Create a secret using the following command:

kubectl -n vmware-system-supervisor-services create secret generic add-trivy-tmpdir-overlay -o yaml --dry-run=client --from-file=add-trivy-tmpdir-overlay.yaml | kubectl apply -f -

 

Step 4: Patch the Harbor package using the following command:

kubectl -n vmware-system-supervisor-services annotate packageinstalls svc-harbor.tanzu.vmware.com ext.packaging.carvel.dev/ytt-paths-from-secret-name.0=add-trivy-tmpdir-overlay

 

Step 5: Wait for the Harbor package to reconcile successfully, and the Trivy pod to come to running state. 

Step 6: Once the pod is up, exec into the pod and check the environment variable:

kubectl exec -it harbor-trivy-0 -n tanzu-system-registry -- /bin/bash

scanner [ / ]$ echo $TMPDIR
/home/scanner/.cache

 

Step 7: After confirming that the TMPDIR is set to /home/scanner/.cache, revert the Harbor package from READONLY mode on the UI.

Step 8: Test by scanning a random artifact, and the scan should now be successful.

 

 

Additional Information

https://trivy.dev/v0.43/docs/references/troubleshooting/