NSX Network Detection and Response - Manager backup stops working without any errors

Products

• VMware

Issue/Introduction

The purpose of the KB article is to understand how to troubleshoot the issue with the Lastline backup

Symptoms:

• The Lastline System backup is not working
• Clicking on backup now manually from the GUI does not work
• Recreating the backup configuration does not resolve the issue
• You see messages similar to the following in the /var/log/previct_backup/previct-backup.log file on the manager node:

2022-12-14 05:36:30,955: output: StorageErrorOperationFailed: Operation failed: Some command exited with an error code: Command "/usr/bin/ssh -oBatchMode=yes -oConnectTimeout=30 -oPasswordAuthentication=no -oStrictHostKeyChecking=no -p 22 -i /usr/share/previct_backup/ssh_key/backup-32d21467-d8de-4018-a8f7-6ca9cfb12fc8 -o TCPKeepAlive=yes -o ServerAliveInterval=60 -o ServerAliveCountMax=3 [email protected] mkdir -p /backup/lastline/lastline-manager.lastline.local/backup-20221207_035703-full_backup-full && chown lastline /backup/lastline/lastline-manager.lastline.local/backup-20221207_035703-full_backup-full && chmod 750 /backup/lastline/lastline-manager.lastline.local/backup-20221207_035703-full_backup-full" return error code: 1 (mkdir: cannot create directory '/backup/lastline/lastline-manager.lastline.local/backup-20221207_035703-full_backup-full': No space left on device

Cause

This issue occurs if the backend storage used for the backup is out of disk space

Resolution

From the error seen in the previct-backup.log file, it's clear that the backend storage used for the lastline backup is out of disk space.

You can use the following steps to further validate the issue with the configured storage. 

1. List the storage configured for backup and note down the ssh_id by running the following command: previct-backup.py storage list 
2. Check the details of the backup job by running the following command: previct-backup.py job list
3. Check the status of the last backup job by running the following command: previct-backup.py job lastatus
4. Note down the ssh key_path of the ssh_id(from step 1) used for the backup by running the following command: previct-backup.py ssh-id list
5. From the manager node, use the ssh key/path obtained in steps 1 and 4 and ssh into the backend storage to  check the disk space utilization by running a command similar to the following: ssh -i /usr/share/previct_backup/ssh_key/backup-<key-id> user@<storage-ip> df -h

To resolve the issue with the failing backup, you must resolve the backend storage out-of-disk space issue.

Additional Information

This article is applicable to the standalone NSX Network Detection and Response product (formerly Lastline) and is not intended to be applied to the NSX NDR feature of NSX-T.

Impact/Risks:

Automated and manual backup stops working