SDDC Manager 5.2.2 Upgrade fails at "Refresh Custom Certificates"

Products

• VMware SDDC Manager

Issue/Introduction

When upgrading VMware Cloud Foundation (VCF) from 5.2.1.2 to 5.2.2.0, the upgrade is stuck at "Refresh Custom Certificates" and eventually the SDDC UI goes inaccessible with "502 Bad Gateway" message.

The domain manager and operations manager services do not successfully start:

systemctl status domainmanager
* domainmanager.service - VMware Cloud Foundation Domain Manager
     Loaded: loaded (/etc/systemd/system/domainmanager.service; enabled; vendor preset: enabled)
     Active: inactive (dead)

systemctl status operationsmanager
* operationsmanager.service - VMware Cloud Foundation Operations Manager
     Loaded: loaded (/etc/systemd/system/operationsmanager.service; enabled; vendor preset: enabled)
     Active: inactive (dead)

The following errors are observed in the SDDC lcm logs.

/var/log/vmware/vcf/lcm/lcm-debug.log

LOG: could not bind IPv4 address "##.###.##.##": cannot assign requested address
WARNING: could not create listen socket for "local host"
FATAL: could not create any TCP/IP sockets
LOG:datbase system is shutdown
 pg ctl: could not start server
 Examine the log output.
 postgres. service: Control process exited, code=exited, status=1/FAILURE
-- Subject: Unit process exited
-- Defined-By: systemd
-- Support: https://lists. freedesktop.org/mailman/listinfo/systemd-devel
--
-- An ExecStart= process belonging to unit postgres. service has exited.
--
-- The process' exit code is 'exited' and its exit status is 1.
############ : postgres. service: Failed with result 'exit-code'.
-- Subject: Unit failed
-- Defined-By: systemd
Support: https://lists.freedesktop.org/mailman/listinfo/systemd-devel

The unit postgres. service has entered the 'failed' state with result 'exit-code'.
###########: Failed to start Postgres.
Subject: A start job for unit postgres. service has failed
-- Defined-By: systemd
-- Support: https://lists. freedesktop.org/mailman/listinfo/systemd-devel

-- A start job for unit postgres.service has finished with a failure.

-- The job identifier is 941 and the job result is failed.

 

 

Environment

VMware Cloud Foundation 5.2.2

Cause

The /etc/hosts file on the SDDC Manager appliance contains commented-out localhost entries. Virtual Appliance Management Infrastructure (VAMI) modified these entries. The Java-based migration application requires localhost hostname resolution to function properly.

The commented entries appear as:

#127.0.0.1  hostname.domain hostname localhost
#::1        hostname.domain hostname localhost ipv6-localhost ipv6-loopback

Resolution

To resolve this issue, correct the /etc/hosts file configuration:

1. SSH to the SDDC Manager appliance as the vcf user.
   
   
2. Switch to root user:
   

   su -

3. Create a backup of the current hosts file:

   cp -p /etc/hosts /etc/hosts.backup

4. Check the current /etc/hosts configuration:

   cat /etc/hosts

5. Edit the /etc/hosts file:

   vi /etc/hosts

6. Edit the hosts file so that it is exactly like the section below.
   
   

   # Begin /etc/hosts (network card version)
   
   127.0.0.1 localhost.localdomain
   127.0.0.1 localhost
   127.0.0.1 photon# End /etc/hosts (network card version)

7. Save and exit the file (press ESC, type :wq, press Enter).
8. Verify the changes:

   cat /etc/hosts

9. Return to the SDDC Manager UI.
10. Retry the upgrade to VCF 5.2.2.0.

The upgrade proceed successfully.