Edge Certificate update failure due to No space left on device

Products

• VMware VeloCloud SD-WAN

Issue/Introduction

Edge Certificate update failure due to No space left on device.

This happens if there is no space left in the /root directory.

The below message (confidential strings are replaced in the sample output) is seen in the mgd log.

 

2025-01-13T01:43:53.379 DEBUG   [heartbeat (8135:Heartbeat:8253)] Heartbeat reply: Configurations={}, Actions={"action": "renewCertificate", "data": {"renew": true}}, {"action": "setCrl", "data": {"id": 1259, "created": "2025-01-10T04:17:31.000Z", "crlUrl": null, "issuer": "CN = xxxx, OU = OPS, O = VeloCloud", "authorityKeyId": "xxxxxxxxxxxxxxxxxxxxxxxxxx", "crlNumber": "xxxxxxxxxxx", "crl": "-----BEGIN X509 CRL-----< CRL String snipped>-----END X509 CRL-----\n", "timestamp": "2025-01-10T04:17:31.000Z"}}, {"action": "enterLiveMode", "id": 626785, "data": {"liveToken": "***", "url": "/livepush/liveData", "liveHeartbeatPeriodSeconds": 2, "extensionSeconds": 900}}, {"action": "generateDiagnosticBundle", "id": 628150, "data": {"options": {"type": "diagnosticDump", "maxCores": 0}, "requestId": "3b7907a0-9217-4e46-9cbf-339f76b79ae7", "maxBundleSize": 180000000}}
2025-01-13T01:43:53.380 ERROR   [heartbeat (8135:Heartbeat:8253)] Heartbeat failure: File I/O error encountered while saving CRL
OSError: [Errno 28] No space left on deviceDuring handling of the above exception, another exception occurred:Traceback (most recent call last):
  File "/opt/vc/lib/python/pyutil/pki.py", line 552, in save_vco_ca_crl
    fd.write(signedcacrl)
OSError: [Errno 28] No space left on device

 

This might also cause the diagbundle generation failure.

Other issue related to diagbundle generation failure is mentioned in KB - Article 369911

Environment

Applicable to all Velocloud SDWAN edge versions.

Cause

Check the root directory for any unwanted files.

cd /root

df -h

Example:

edge:b2-edge1:~# df -h
Filesystem                Size      Used Available Use% Mounted on
/dev/root               958.7M    645.6M    245.9M  72% /
tmpfs                     3.8G     35.0M      3.8G   1% /tmp
tmpfs                   512.0K         0    512.0K   0% /dev
/dev/sda2                63.9M      2.4M     61.5M   4% /boot
/dev/sda6                 4.9G    849.2M      4.1G  17% /velocloud
tmpfs                     3.8G     35.0M      3.8G   1% /tmp/run/netns
edge:b2-edge1:~#

If /dev/root is 100% utilized then new certificates can not be stored.

Resolution

Find the unwanted files which is holding the space. Sometimes there are some residual files while manually copying any files from an edge to another.