SDDC Manager UI shows red banner for certificate expiration: "Certificates have expired. Visit Domain Management page to take action"

Products

• VMware Cloud Foundation
• VMware Cloud Foundation 5.x

Issue/Introduction

• SDDC Manager UI displays a red banner indicating "Certificates have expired" or "Installation Failed" despite certificates being successfully updated on the component (e.g., vCenter Server). This often occurs after an out-of-band certificate renewal or a failed retrust operation with NSX.
  
  
  
  
• SDDC Manager UI shows a red banner: "Certificates have expired. Visit Domain Management page to take action" or "Installation Failed".
• SDDC Workload Domains show the error message: "[Domain Name] has expired certificate".
• Certificate replacement workflow fails during the retrust phase with error: 400 Bad Request: "FQDNs are not configured for nodes IPs".
• Logs in /var/log/vmware/vcf/operationsmanager/operationsmanager.log show: ERROR: relation "certificate_chain_expiry_seq" does not exist.
• SDDC manager logs contains errors similar to the excerpt below in /var/log/vmware/vcf/operationsmanager/operationsmanager.log : 
  • WARN  [vcf_om,########,####] [o.h.e.jdbc.spi.SqlExceptionHelper,om-exec-11] SQL Error: 0, SQLState: 42P01
    ERROR [vcf_om,########,####] [o.h.e.jdbc.spi.SqlExceptionHelper,om-exec-11] ERROR: relation "certificate_chain_expiry_seq" does not exist
      Position: 16
    YYYY-MM-DD ERROR [vcf_om,6809b360b1a4e#####3d567a1f4640,1b36] [c.v.v.c.s.e.UpdateCertificateExpiryService,om-exec-18] Failed to update the certificate expiry cache could not extract ResultSet [ERROR: relation "certificate_chain_expiry_seq" does not exist
      Position: 16] [select nextval('certificate_chain_expiry_seq')]; SQL [select nextval('certificate_chain_expiry_seq')]
    org.springframework.dao.InvalidDataAccessResourceUsageException: could not extract ResultSet [ERROR: relation "certificate_chain_expiry_seq" does not exist
      Position: 16] [select nextval('certificate_chain_expiry_seq')]; SQL [select nextval('certificate_chain_expiry_seq')]
            at org.springframework.orm.jpa.vendor.HibernateJpaDialect.convertHibernateAccessException(HibernateJpaDialect.java:256)
            at org.springframework.orm.jpa.vendor.HibernateJpaDialect.translateExceptionIfPossible(HibernateJpaDialect.java:229)
            at org.springframework.orm.jpa.AbstractEntityManagerFactoryBean.translateExceptionIfPossible(AbstractEntityManagerFactoryBean.java:550)
            at org.springframework.dao.support.ChainedPersistenceExceptionTranslator.translateExceptionIfPossible(ChainedPersistenceExceptionTranslator.java:61)
            at org.springframework.dao.support.DataAccessUtils.translateIfNecessary(DataAccessUtils.java:243)
            at org.springframework.dao.support.PersistenceExceptionTranslationInterceptor.invoke(PersistenceExceptionTranslationInterceptor.java:152)
            at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:184)
            at org.springframework.data.jpa.repository.support.CrudMethodMetadataPostProcessor$CrudMethodMetadataPopulatingMethodInterceptor.invoke(CrudMethodMetadataPostProcessor.java:164)
            at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:184)
            at org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:97)
            at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:184)
            at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:244)
            at jdk.proxy2/jdk.proxy2.$Proxy208.save(Unknown Source)
    [...]
    Caused by: org.hibernate.exception.SQLGrammarException: could not extract ResultSet [ERROR: relation "certificate_chain_expiry_seq" does not exist
      Position: 16] [select nextval('certificate_chain_expiry_seq')]

Environment

• VMware Cloud Foundation (VCF) 4.x, 5.x
• VCF on VxRail
• SDDC Manager

Cause

This issue occurs when the SDDC Manager operationsmanager database is not updated with the new certificate expiration dates or task status. This can happen if certificates are renewed directly on the component (out-of-band) or if the automated retrust workflow fails due to DNS/FQDN mismatches, leaving stale "FAILED" or expired entries in the database.

Resolution

See Download Broadcom products, patches and software to download the latest release.

For environments where an upgrade is not immediately possible, use the following steps to manually reconcile the database.

Warning: Take a snapshot of the SDDC Manager appliance before performing database modifications.

1. Log in to the SDDC Manager via SSH as vcf and switch to root.
2. Identify expired certificate entries in the database: psql -h localhost -U postgres -d operationsmanager -X -c "select * from certificatemanagement.certificate_chain_expiry where expiry_date < now();"
3. Remove the identified stale entries: psql -h localhost -U postgres -d operationsmanager -X -c "delete from certificatemanagement.certificate_chain_expiry where id in ('<ID1>','<ID2>');"
4. If the UI shows "Installation Failed", check the task status table: psql -U postgres -h localhost -d operationsmanager -c "select replacement_status_id, replacement_status, resource_fqdn from certificatemanagement.replacement_status where replacement_status='FAILED'"
5. Update the status to SUCCESSFUL for the identified IDs: psql -U postgres -h localhost -d operationsmanager -c "update certificatemanagement.replacement_status set replacement_status='SUCCESSFUL' where replacement_status_id='<ID>'"
6. Refresh the SDDC Manager UI to verify the banner is cleared.

If the issue persists, contact Broadcom Support. See Contact Broadcom Support for assistance.