SDDC daily 'Health-Check operation for SDDC' fails on 'Password-check' subtask - Password-check: Perform Password expiry status checks on SDDC components Failed

Products

• VMware SDDC Manager

Issue/Introduction

UI Error:

• Error: "Password-check: Perform Password expiry status checks on SDDC components       Failed"

• The SDDC runs a daily health check to assess the health of the environment.
• One of these checks is designed to check the password expiry status of all the managed components.
• If the SDDC cannot communicate with a component, this password expiry check will fail.
• If the health check encounters a failure, the check is repeated multiple times in a day, which can cause the tasks pane in the SDDC to become spammed with Health-Check failures.
• To identify what component or components are failing the check run a manual password health check from the SDDC command line:

/opt/vmware/sddc-support/sos --password-health

 

• This will show a status of "Failed to get details" against the failing components:

 

 

• /var/log/vmware/vcf/sddc-support/vcf_sos.log suggests that this is a hostkey mismatch issue:

 

2025-04-09T08:06:48.314+0000 DEBUG [vcf_sos] [commandutils.py::run_cmds_over_ssh:440]:get_password_expirationThread4] server: EXAMPLE.COM --- stderr: Host key for server 'EXAMPLE.COM' does not match: got '######################################################', expected '###############################################################'

2025-04-09T08:06:48.314+0000 INFO [vcf_sos] [commandutils.py::run_cmds_over_ssh:443]:get_password_expirationThread4] rc is not 0, bRet: -1

2025-04-09T08:06:48.315+0000 ERROR [vcf_sos] [util.py::log_password_check:2239]:get_password_expirationThread4] Failed to get password information

2025-04-09T08:06:48.315+0000 INFO [vcf_sos] [util.py::log_password_check:2252]:get_password_expirationThread4] Failed to get password expiration information for user root on : EXAMPLE.COM

Environment

VCF 5.x

Cause

The SDDC will not allow a connection to a component that presents a hostkey or keys that the SDDC is unfamiliar with.

This is by design.

 

Resolution

• Update the known_hosts files on the SDDC for each of the components failing the password-expiry check using fix_known_hosts KB.
• NOTE: Use the python script attached to the KB primarily.
• Run /opt/vmware/sddc-support/sos --password-health  again and confirm the "Failed to get details" status(es) have changed to "GREEN"
• The next run of the automatic health check should now succeed.

Additional Information

NOTE: running ssh -o StrictHostKeyChecking=yes <user>@<EXAMPLE.COM> may actually succeed in connecting to the component, which suggests the component hostkeys are ok. It is essential to check the sos log AS WELL.