SDDC Manager reports that multiple component service accounts are either disconnected or nearing expiration

Products

• VMware SDDC Manager
• VMware Cloud Foundation

Issue/Introduction

• Status of service accounts of multiple component under SSDC or VCF Operations 9.x UI, display either Disconnected or Expiring in # days.
   
  
  
  
• When attempting to resolve the issue using the Remediate or Rotate password functions in SDDC Manager, the tasks show as successful, but the account status remains Disconnected or the Expiring status does not disappear.
• Or the Task will fail with below error messages:
  
  Description:
Password remediate for resource : vCenter-fqdn.sso.domain, user : [email protected] and credential type : SSOProgress
Messages: Unable to establish connection with resource.ErrorMessage: Unable to establish connection with resource.Remediation
Message: Please verify if the account credentials can be used to login to the resource. If the password of the account has expired, manually reset the password in the product and then perform a REMEDIATE operation in the SDDC Manager, to update its stored copy of the password.Reference Token: TOKEN#
Cause: Unable to obtain Security Token Service from SSO 'vCenter-fqdn.sso.domain' as provided credentials are invalid
  

  
  Description:
Password remediate for resource : nsx-fqdn.sso.domain, user : svc-user and credential type : APIProgress
Messages: Failed to get NSX user details.Error
Message: Failed to get NSX user details.Remediation Message:Reference Token: TOKEN#
Cause: The credentials were incorrect or the account specified has been locked.

• Under SDDC Manager's /var/log/vmware/vcf/operationsmanager/operationsmanager.log, below log snippets are available.
  
  YYYY-MM-DDTHH:MM:SS DEBUG [vcf_om,##############,####] [c.v.v.p.v.u.CredentialsValidationTaskExecutor,om-exec-2] Updating expiration details for credentialId: ######-####-####-####-###### in credential expiry cache
YYYY-MM-DDTHH:MM:SS DEBUG [vcf_om,##############,####] [c.v.v.p.s.PasswordExpirationService,om-exec-#] Validation checks size = 1
YYYY-MM-DDTHH:MM:SS DEBUG [vcf_om,##############,####] [c.v.v.p.s.PasswordExpirationService,om-exec-#] expiry cache entity is available : true
YYYY-MM-DDTHH:MM:SS DEBUG [vcf_om,##############,####] [c.v.v.p.s.PasswordExpirationService,om-exec-#] Expiry retrieval status : SUCCEEDED ,  Diagnostic message : null
YYYY-MM-DDTHH:MM:SS WARN  [##############,####] [o.h.e.jdbc.spi.SqlExceptionHelper,om-exec-2] SQL Error: 0, SQLState: #####
YYYY-MM-DDTHH:MM:SS ERROR [vcf_om,##############,####] [o.h.e.jdbc.spi.SqlExceptionHelper,om-exec-2] ERROR: timestamp out of range: "#######-##-## HH:MM:SS"
  Where: unnamed portal parameter $4 = '...'
YYYY-MM-DDTHH:MM:SS ERROR [vcf_om,##############,####] [c.v.v.p.s.PasswordExpirationService,om-exec-2] could not execute statement [ERROR: timestamp out of range: "#######-##-## HH:MM:SS"
  
  
• Additionally, querying the passwordmanager.credential_expiry table in the SDDC Manager PostgreSQL database shows that the fetch_time for the affected credential is stuck on an older date and is no longer updating.

Environment

VMware Cloud Foundation Operations 9.x
SDDC Manager 5.x / 9.x

Cause

• This issue occurs when the vCenter Server SSO Password Policy is configured with an expiration value that calculates to an extremely far-future date (such as 0 for "never expires,"  or a very large number like 999999, which vCenter may translate to a year like 2739933).

• When SDDC Manager attempts to fetch this password expiry date to update its internal PostgreSQL database, the value exceeds the database's maximum supported timestamp limit. This throws a timestamp out of range exception, causing the backend sync to fail and leaving the account permanently in a Disconnected state, regardless of password remediation success.

• SDDC manager database stores and processes dates in the format YYYY-MM-DD. 
  
  
• Any expiration date after 9999-12-31 will cause the password expiration check to fail and the component will show as disconnected. 

Resolution

To resolve this issue, lower the vCenter SSO password expiration policy to a supported, finite value, and then remediate the password in SDDC Manager.

Step 1: Retrieve the service accounts credentials from SDDC Manager for the Disconnected/Nearing to expire Service Account, by following KB: Retrieve the service accounts credentials from SDDC Manager

Step 2: Remediate Password in SDDC Manager UI

1. 1. Log in to the SDDC Manager UI.

   2. Navigate to Administration > Security > Password Management 

   3. Select the affected service account.

   4. Click Remediate

   5. And update the Password, that was retrieved in Step 1 and click confirm REMEDIATE PASSWORD
      
      
   6. Repeat above steps for all the affected Components.

Additional Information

• Command to check the  Password Manager expiration Table, using below command in the SDDC SSH:
  
  psql -h localhost -U postgres -d operationsmanager -c "SELECT credential_id, resource_fqdn, username, expiry_date, fetch_time, fetch_status FROM passwordmanager.credential_expiry WHERE resource_type = 'VCENTER' ORDER BY fetch_time DESC;"

  
  Example O/p:
  
  
  
  
• For setting the Service Account to never expire in the vCenter, follow below TechDoc:
  https://techdocs.broadcom.com/us/en/vmware-cis/vsphere/vsphere/8-0/vsphere-authentication-8-0/vsphere-authentication-with-vcenter-single-sign-on-authentication/vmware-single-sign-on-polices-authentication.html