VCF upgrade fails at "SDDC Manager Deployment Drift"

Products

VMware SDDC Manager

Issue/Introduction

SDDC Manager Deployment Drift failure - VCF upgrade to 5.2.2.0 fails with localhost resolution error in SDDC Manager
When upgrading VMware Cloud Foundation (VCF) to version 5.2.2.0, the upgrade fails after approximately 15 minutes. The error message states: "Failed to resolve 'localhost'" in the SDDC Manager migration logs. The upgrade halts during the SDDC Manager migration phase.

The following error can be found in this log file:

/var/log/vmware/vcf/lcm/thirdparty/upgrades/########-####-####-####-############/sddcmanager-migration-app/logs/sddcmanager_migration_app_upgrade.log shows

YYYY-MM-DDTHH:MM:SS+0000 DEBUG [vcf_lcm,################,####] [c.v.v.s.c.s.SecurityConfigurationServiceImpl,main] Security config retrieved {"fipsMode":false}
YYYY-MM-DDTHH:MM:SS+0000 ERROR [vcf_lcm,################,####] [c.v.e.s.m.s.s.ExportDataService,main] Error while fetching inventory data org.springframework.web.reactive.function.client.WebClientRequestException: Failed to resolve 'localhost' [A(1)]
at org.springframework.web.reactive.function.client.ExchangeFunctions$DefaultExchangeFunction.lambda$wrapException$9(ExchangeFunctions.java:136)
Suppressed: reacto.core.publisher.FluxOnAssembly$OnAssemblyException: Error has been observed at the following site(s):
*__checkpoint ⇢ Request to GET http://localhost/inventory/export [DefaultWebClient]
Original Stack Trace:
at org.springframework.web.reactive.function.client.ExchangeFunctions$DefaultExchangeFunction.lambda$wrapException$9(ExchangeFunctions.java:136)
at reactor.core.publisher.MonoErrorSupplied.subscribe(MonoErrorSupplied.java:55)

This error blocks the ability to complete the upgrade to VCF 5.2.2.0.
Also, when checking the SDDC manager services (ie systemctl status commonsvcs) will show the service as "Active: inactive (dead)".

Environment

VCF 5.2.2

Cause

The /etc/hosts file on the SDDC Manager appliance contains commented-out localhost entries. Virtual Appliance Management Infrastructure (VAMI) modified these entries. The Java-based migration application requires localhost hostname resolution to function properly.

The commented entries appear as:

#127.0.0.1 hostname.domain hostname localhost

#::1 hostname.domain hostname localhost ipv6-localhost ipv6-loopback

Resolution

To resolve this issue, correct the /etc/hosts file configuration:

SSH to the SDDC Manager appliance as the vcf user.
Switch to root user:

su -
Create a backup of the current hosts file:

cp -p /etc/hosts /etc/hosts.backup
Check the current /etc/hosts configuration:cat /etc/hosts
Edit the /etc/hosts file:vi /etc/hosts
Edit the hosts file so that it is exactly like the section below.

# Begin /etc/hosts (network card version)

127.0.0.1 localhost.localdomain
127.0.0.1 localhost
127.0.0.1 photon# End /etc/hosts (network card version)
Save and exit the file (press ESC, type :wq, press Enter).
Verify the changes:

cat /etc/hosts
Start Postgres with the cmd below:

systemctl start postgres
Restart the services on the SDDC manager:

/opt/vmware/vcf/operationsmanager/scripts/cli/sddcmanager_restart_services.sh
Return to the SDDC Manager UI.
Retry the upgrade to VCF 5.2.2.0.

The upgrade proceed successfully.

Note: A reboot of the SDDC manager might be needed after making the changes to the SDDC manager hosts file.

Additional Information

Pre-upgrade Verification

Before starting any upgrade to VCF 5.2.2.0, verify localhost entries exist:

grep -E "^127.0.0.1.*localhost" /etc/hosts

If no results appear, apply the resolution steps before upgrading.

Best Practice

Always create a snapshot of the SDDC Manager VM before making configuration changes. This provides a rollback option if issues occur.