Backup of vCenter Server fails at 95% with error "Backup job failed"

Products

• VMware vCenter Server

Issue/Introduction

VAMI based backup of vCenter Server Appliance (VCSA) fails with the error "Backup job failed."
The issue is seen during a scheduled or manual backup of VCSA.

/var/log/vmware/applmgmt/backup.log:

YYYY-MM-DDThh:mm:ss [YYYYMMDD-hhmmss-####] [MainProcess:PID-####] [Net::GetPNIDAddrInfo:Net.py:651] WARNING: Failure in name resolution(v6) for the PNID '<vCenter_fqdn>' RC: 9 Err:
YYYY-MM-DDThh:mm:ss [YYYYMMDD-hhmmss-####] [MainProcess:PID-####] [Net::GetPNIDAddrInfo:Net.py:657] INFO: PNID '<vcenter_fqdn>' is resolved with addresses: [';; communications error to <dns-server-ip>#53: connection refused', ';; communications error to <dns-server-ip>#53: timed out',]

YYYY-MM-DDThh:mm:ss [YYYYMMDD-hhmmss-####] [MainProcess:PID-####] [BackupManager::GenerateBackupMetadata:BackupManager.py:220] ERROR: PNID is not resolvable to any IP on the existing system

YYYY-MM-DDThh:mm:ss [YYYYMMDD-hhmmss-####] [MainProcess:PID-####] [Net::GetPNIDAddrInfo:Net.py:657] INFO: PNID '<vcenter_fqdn>' is resolved with addresses: ['communications error to <dns-server-ip>#53: connection refused', ';;  connection refused', '<dns-server-ip>: timed out']

YYYY-MM-DDThh:mm:ss [YYYYMMDD-hhmmss-####] [MainProcess:PID-####] [BackupManager::Cleanup:BackupManager.py:446] INFO: Cleaning up failed backup files.
YYYY-MM-DDThh:mm:ss [YYYYMMDD-hhmmss-####] [MainProcess:PID-####] [BackupManager::HandleBackupCleanup:BackupManager.py:489] INFO: Cleaned up the backup job.
YYYY-MM-DDThh:mm:ss [YYYYMMDD-hhmmss-####] [MainProcess:PID-####] [vpxdevent_lib::dispatch_events:vpxdevent_lib.py:274] INFO: Event com.vmware.applmgmt.backup.job.failed.event successfully posted to http://localhost:8085/sdk
YYYY-MM-DDThh:mm:ss [YYYYMMDD-hhmmss-####] [MainProcess:PID-####] [BackupManager::main:BackupManager.py:672] INFO: Backup job failed.

Environment

• VMware vCenter Server 7.0.x
• VMware vCenter Server 8.0.x

Cause

DNS name resolution is failing in the vCenter

Resolution

• Ping the DNS server from the vCenter and check.
• Run nslookup from vCenter to any other server and check.

          example: 

         ping <dns-server-ip>

         nslookup <any_esxi_host_ip>

         nslookup <any_esxi_host_fqdn>

• The ping or nslookup will fail; the DNS server needs to be investigated and fixed.