Logs in vSAN File Services VMs are filling up space and not being cleared automatically.

Products

• VMware vSAN

Issue/Introduction

• You may see the symptoms in ARIA operations as "Guest: File System::/log|Partition Utilization (GB)" being full on vSAN File Service nodes.
• You may see alerts of "One or more virtual machine guest file systems are running out of disk space" as the example below:

Environment

VMware vSAN 8.0.x.

Cause

This is a known issue identified where the logs in /log/samba_logs of the FSVMs are filling up the space and not being deleted automatically.

Resolution

The fix for this issue is being worked on to be available with VCF 9.0.1 release. Please keep monitoring this article for updates.

 

Workaround:

 

1st - Delete the logs from older containers.

 

1. Access the vCenter Web UI.

2. In the VC UI, navigate to the FSVM:
   Click "vSAN File Service Node (...)" on the left panel, then select "LAUNCH WEB CONSOLE" button on the right.

3. In the FSVM console, run "docker ps" to list active containers, showing output like:

 

root@photon-2f102caa946b [ ~ ]# docker ps
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
29e872b7cc4d vsfs "/entrypoint.py" 2 minutes ago Up 2 minutes h10-###-#8-120

4. Run "ls /log/samba_logs/" to list log directories, showing output like this:

root@photon-2f102caa946b [ ~ ]# ls /log/samba_logs/
h10-###-#8-120 h10-###-#2-245 h10-###-#6-81 h10-###-#0-187 h10-###-#1-118 h10-###-#1-62

5. Remove the folders that do not match any "NAMES" in Step 3, e.g.:

rm -rf /log/samba_logs/h10-###-#0-187/

 

6. Restart the FSVM for the usage information to be refreshed.

 

Note: In most cases, those steps will be enough to see results, but that is not always the case. If needed, proceed with the second activity proposed below.

 

2nd - If not enough space is cleared after deleting old container logs, you can run a cron job:

 

1. (After cleanup of inactive containers) Log in to the FSVM console and navigate to the samba logs directory:

cd /log/samba_logs
du -sh *

This will display the sizes of the active container log folders. For example: 

root@photon-### [ /log/samba_logs ]# du -sh *
4.0K h10-###-#36-5
3G h10-###-#38-64

 

2. For folders exceeding 2G in size (e.g., h10-###-#38-64), perform the following steps to manually rotate the logs:

cd  h10-###-#38-64/

 

3. From the directory /log/samba_logs/h10-161-238-64/, execute the following script:

 
for file in ./*.log; do
    gzip "$file"
    gz_file="${file}.gz"
    basename=$(basename "$file" .log)
    timestamp=$(date -r "$gz_file" +"%Y%m%d-%s")
    mv "$gz_file" "./${basename}.log-${timestamp}.gz"
done

 

4. After running the script, the current log files will be compressed and renamed as shown below:
 

root@photon-### [ /log/samba_logs/h10-161-238-64 ]# ls
0.0.0.0.log-20241108-1731081897.gz  log.wb-H10-###-#38-64  nmbd.log-20241108-1731093219.gz
cores                               log.wb-VSANFS-SH       smbd.log-20241108-1731092698.gz
log.wb-BUILTIN                      log.winbindd-idmap     winbindd.log-20241108-1731093219.gz

* This should not impact the server. Samba will shortly create new log files and continue writing to them as usual.

 

5. Restart the FSVM for the usage information to be refreshed.