SDDC Manager DNS Health Check Failure: Forward DNS Lookup Configuration Error

Products

• VMware SDDC Manager / VCF Installer

Issue/Introduction

• SDDC Manager web interface (GUI) inaccessible with a "502 Bad Gateway" error.
• Failure to replace or import SDDC Manager certificates without a clear error in the GUI.
• PostgreSQL database service failing to start with "Temporary failure in name resolution" 
• DNS health check on SDDC fails with error.
• Running /opt/vmware/sddc-support/sos --dns-health on SDDC manager shows DNS in red:
  DNS lookup Status : RED
• vcf-sos.log shows error logging similar to: /var/log/vmware/vcf/sddc-support/vcf-sos.log

INFO [vcf_sos] [progressreporter.py::update_task_status::_parallel_check_nsx_dns_configuredThread0] Overall status for DNS-CHECK : COMPLETED_WITH_FAILURE
INFO [vcf_sos] [progressreporter.py::update_task_status::_parallel_check_nsx_dns_configuredThread0] Current Status: Task-Name: DNS-CHECK, Entity-Name: <edge-fqdn>: Forward DNS lookup check, Task-Status: COMPLETED
INFO [vcf_sos] [progressreporter.py::determine_progress::_parallel_check_nsx_dns_configuredThread0] All Tasks : {'VCF-SUMMARY': 'NOT_COMPLETED', 'DNS-CHECK:<nsx-fqdn>: Reverse DNS lookup check': 'COMPLETED', 'DNS-CHECK:<vcenter-fqdn>: Reverse DNS lookup check': 'COMPLETED', 'DNS-CHECK:<sddc-manager-fqdn>: Reverse DNS lookup check': 'COMPLETED', 'DNS-CHECK:<nsx-fqdn>: Forward DNS lookup check': 'COMPLETED', 'DNS-CHECK:<esxi-host-fqdn-03>: Reverse DNS lookup check': 'COMPLETED', 'DNS-CHECK:<esxi-host-fqdn-02>: Reverse DNS lookup check': 'COMPLETED', 'DNS-CHECK:<esxi-host-fqdn-04>: Reverse DNS lookup check': 'COMPLETED', 'DNS-CHECK:<esxi-host-fqdn-01>: Reverse DNS lookup check': 'COMPLETED', 'DNS-CHECK:<sddc-manager-fqdn>: Forward DNS lookup check': 'FAILED',

• The loopback address resolution in the "/etc/hosts" file is commented out or the hosts file is blank.

  # Begin /etc/hosts (network card version)
  # End /etc/hosts (network card version)
  # VAMI EDIT BEGIN
  # Generated by Studio VAMI service. Do not modify manually.
  # 127.0.0.1 <sddc_manager_fqdn> <sddc_manager_hostname> localhost
  # :: 1 <sddc_manager_fqdn> <sddc_manager_hostname> localhost ipv6-localhost ipv6-loopback
  # VAMI EDIT END

Environment

• SDDC Manager 5.x
• VCF 5.x

Cause

• Issue is caused due to improperly formatted hosts file in the SDDC Manager.
• Loopback entries may be inadvertently commented out during DNS update workflows via VAMI regeneration scripts. 

Resolution

• Correct the /etc/hosts file configuration;
  • SSH to the SDDC Manager appliance as the vcf user.
  • Switch to root user:
    su -
  • Create a backup of the current hosts file:
    cp /etc/hosts /etc/hosts.backup
  • Check the current /etc/hosts configuration:
    cat /etc/hosts
• Edit the hosts file and uncomment the lines as below using vi editor:
  vi /etc/hosts

# Begin /etc/hosts (network card version)
# End /etc/hosts (network card version)
# VAMI EDIT BEGIN
# Generated by Studio VAMI service. Do not modify manually.
127.0.0.1 <sddc_manager_fqdn> <sddc_manager_hostname> localhost
:: 1 <sddc_manager_fqdn> <sddc_manager_hostname> localhost ipv6-localhost ipv6-loopback
# VAMI EDIT END

• Re-run the command /opt/vmware/sddc-support/sos --dns-health and if it still shows RED, apply the below steps;
  • Edit the hosts file to match the contents/format exactly as below; 

# Begin /etc/hosts (network card version)
127.0.0.1 localhost.localdomain
127.0.0.1 localhost
127.0.0.1 photon
#End /etc/hosts (network card version)

• • Save and exit the file: Press ESC and type wq! and press Enter to exit vi editor.
  • Verify the changes:
    cat /etc/hosts
  • Restart SDDC Manager Appliance.