Harbor Image Scanning Failure: “No Space Left on Device” for all harbor images

Products

• VMware vSphere Kubernetes Service

Issue/Introduction

Image scanning in Harbor may fail with an error similar to the following:

scan job: fetch scan report: running Trivy: exit status 1:
'--vuln-type' is deprecated. Use '--pkg-types' instead.
Vulnerability scanning is enabled.
Fatal error: image scan failed: failed analysis: pipeline error:
failed to analyze layer: failed to process file: failed to copy file to temp:
write /tmp/analyzer-fs-*/file-* : no space left on device
unexpected status code: 500, expected: 200

Environment

VMware vCenter Server 8.0.x 

vSphere with Tanzu 8.0.x

Cause

This failure occurs because Trivy requires temporary storage space to extract and analyze image layers. When the underlying system—specifically the /tmp directory or the Trivy cache location—runs out of available disk space, the scanning workflow cannot complete, resulting in the error above.

Resolution

To resolve this issue in Harbor deployed as a Supervisor Service, the Trivy scanner must be redirected to use a different temporary directory backed by a Persistent Volume Claim (PVC). 

Prerequisites

These steps must be performed after SSH’ing into the Supervisor Control Plane Node as root, because non-root or authenticated vSphere SSO users do not have permission to create secrets in the vmware-system-supervisor-services namespace.

Step 1:Create a file named add-trivy-tmpdir-overlay.yaml on the Supervisor node and add the following contents:

#@ load("@ytt:overlay", "overlay")

#@overlay/match by=overlay.and_op(overlay.subset({"kind": "StatefulSet"}), overlay.subset({"metadata": {"name": "harbor-trivy"}}))
---
spec:
  template:
    spec:
      containers:
        #@overlay/match by="name"
        - name: trivy
          env:
            #@overlay/append
            - name: "TMPDIR"
              value: /home/scanner/.cache

Step 2: Create the Secret in the Correct Namespace

kubectl -n vmware-system-supervisor-services create secret generic add-trivy-tmpdir-overlay --from-file=add-trivy-tmpdir-overlay.yaml -o yaml --dry-run=client | kubectl apply -f -

Step 3: Patch the Harbor Supervisor Service Package

kubectl -n vmware-system-supervisor-services annotate packageinstalls svc-harbor.tanzu.vmware.com ext.packaging.carvel.dev/ytt-paths-from-secret-name.0=add-trivy-tmpdir-overlay --overwrite

Step 4: Verify the TMPDIR Setting in the Trivy Container and the expected output is /home/scanner/.cache

kubectl exec -it harbor-trivy-0 -n tanzu-system-registry -- /bin/bash

echo $TMPDIR