VAMI backup on vCenter Server behind NAT fails with error "Backup encountered and exception"

Products

• VMware vCenter Server

Issue/Introduction

• vCenter Server is behind the NAT and VAMI backup fails with below error, even though partial data transfer to the backup target was successful and as a result few files can be seen been as created on the backup target location:

BackupManager encountered an exception. See logs for details. https://<vcenter_fqdn>/appliance/support-bundle

• On the backup logs entries similar to below are encountered which show that the Backup Manager script is encountering error as the vCenter FQDN is resolving to an IP address which is not the expected internal IP:

/var/log/vmware/applmgmt/backup.log

YYYY-MM-DDTHH:MM:SS.MS [yyyymmdd-hhmmss-22####22] [MainProcess:PID-##] [Net::GetPNIDAddrInfo:Net.py:639] INFO: PNID '<vCenter_fqdn>' is resolved with addresses: [##.##.##.##] 
YYYY-MM-DDTHH:MM:SS.MS [yyyymmdd-hhmmss-22####22] [MainProcess:PID-##] [BackupManager::GenerateBackupMetadata:BackupManager.py:202] ERROR: PNID is not resolvable to any IP on theexisting system
YYYY-MM-DDTHH:MM:SS.MS [yyyymmdd-hhmmss-22####22] [MainProcess:PID-##] [BackupManager::main:BackupManager.py:592] ERROR: BackupManager encountered an exception: name 'MetadataException' is not defined
YYYY-MM-DDTHH:MM:SS.MS [yyyymmdd-hhmmss-22####22] [MainProcess:PID-##] [BackupManager::HandleBackupCleanup:BackupManager.py:466] INFO: Cleaning up the backup job.

...

YYYY-MM-DDTHH:MM:SS.MS [yyyymmdd-hhmmss-22####22] [MainProcess:PID-##] [BackupManager::HandleBackupCleanup:BackupManager.py:489] INFO: Cleaned up the backup job.
YYYY-MM-DDTHH:MM:SS.MS [yyyymmdd-hhmmss-22####22] [MainProcess:PID-##] [vpxdevent_lib::dispatch_events:vpxdevent_lib.py:274] INFO: Event com.vmware.applmgmt.backup.job.failed.event successfully posted to http://localhost:8085/sdk
YYYY-MM-DDTHH:MM:SS.MS [yyyymmdd-hhmmss-22####22] [MainProcess:PID-##] [BackupManager::main:BackupManager.py:672] INFO: Backup job failed.

Environment

vCenter Server 7.x, 8.x

Cause

Since the vCenter Server is behind NAT, the PNID/FQDN of the vCenter might be resolving to a different IP like the external NAT IP than the expected internal IP address, thus causing the Backup Manager script, which is responsible for the backups, to fail.

Resolution

Please note NAT between vCenter Server and ESXi host is not a supported configuration as mentioned on Using NAT between the vCenter Server system and ESXi hosts will disconnect the host from the vCenter

Kindly contact Broadcom Support if you are facing this issue on a vCenter wherein the vCenter appliance along with the ESXi hosts have been configured to be behind NAT as this requires modifying the Backup Manager script that needs to be done with the assistance of Broadcom Support Engineer

Additional Information