NSX-T backups are failing with error "backup operation unknown error"

Products

• VMware NSX

Issue/Introduction

Symptoms:

• NSX-T backups are failing with error "backup operation unknown error" as shown in the below screenshot:

 

• NSX Manager logs (nsxapi.log) display messages similar to the messages below:

#cat nsxapi.log | grep -A 150 "Error happened when backup system" | grep -E 'Error happened when backup system|Caused by' | head -n 2
2019-06-18T17:22:22.605Z ERROR http-nio-127.0.0.1-7440-exec-2 MaintenanceServiceImpl - - [nsx@6876 comp="nsx-manager" errorCode="MP4520" subcomp="manager"] Error happened when backup system.
com.vmware.nsx.management.common.exceptions.BaseException: Exception encountered backing up regions.
Caused by: org.corfudb.runtime.exceptions.TransactionAbortedException: TX ABORT | Snapshot Time = Token(epoch=7, sequence=698919) | Failed Transaction ID = 45c53022-080b-43d1-a68d-b8757f2cXXXX | Offending Address = -1 | Conflict Key = | Conflict Stream = 3caa7629-42c8-3172-bfc0-819c3cfdXXXX | Cause = TRIM | Time = 2741771 ms | Message = null
 

• NSX Manager logs (nsxapi.log or policy.log) indicates the one of the table took a long time to be processed:

#grep "Backup region" /var/log/proton/nsxapi.log
#grep "Backup region" /var/log/policy/policy.log
** suppressed output for clarity **
2019-06-26T15:52:37.604Z  INFO http-nio-127.0.0.1-7440-exec-16 CorfuBackupMaintenanceDaoImpl - - [nsx@6876 comp="nsx-manager" subcomp="manager"] Backup region: nsx-manager LogicalPortAttachers c47e
2019-06-26T15:52:37.616Z  INFO http-nio-127.0.0.1-7440-exec-16 CorfuBackupMaintenanceDaoImpl - - [nsx@6876 comp="nsx-manager" subcomp="manager"] Backup region: nsx-manager DirectoryGroupMember 7a4
2019-06-26T21:02:46.664Z  INFO http-nio-127.0.0.1-7440-exec-16 CorfuBackupMaintenanceDaoImpl - - [nsx@6876 comp="nsx-manager" subcomp="manager"] Backup region: nsx-manager PermanentAllocationMarker 10b1
** suppressed output for clarity **

Environment

VMware NSX-T
VMware NSX-T Data Center
VMware NSX-T Data Center 2.5.x
VMware NSX-T Data Center 2.x

Cause

The issue is due to NSX Manager table(s) exceeding their size limit (refer to https://configmax.vmware.com/ for maximum configurations).

Resolution

Currently, there is no resolution. Refer to the Workaround section.

Workaround:
To work around this issue, file a support request with Broadcom Support and quote this Knowledge Base article ID (326350)  in the problem description.