Replacement of vCenter Server Certificates using SDDC Manager fails with "Failed to notify APPLMGMT"

Products

VMware SDDC Manager / VCF Installer

Issue/Introduction

While replacing the certificates on vCenter Server (VCSA) with CA-signed certificates using the certificate management of SDDC Manager, the certificate replacement is completed successfully; however, an error stating "Failed to notify APPLMGMT" is reported.
This issue may be encountered while replacing the certificates on the VCSA using vSphere client.
The certificate details are updated on the SDDC Manager, however, the task shows as failed.
Errors reported in /var/log/vmware/vcf/operationsmanager/operationsmanager.log :

YYYY-MM-DDTHH:MM:SS:###Z DEBUG [vcf_om,################################,####] [c.v.v.c.s.f.i.CertificateOperationsFacadeImpl,http-nio-127.0.0.1-7300-exec-8] Get: REPLACE_CERTIFICATE operation details from store
YYYY-MM-DDTHH:MM:SS:###Z DEBUG [vcf_om,################################,####] [c.v.v.c.s.f.i.CertificateOperationsFacadeImpl,http-nio-127.0.0.1-7300-exec-8] DomainCertificateOperation: {"workflowId":"########-####-####-############-####","domainName":"Management","operationType":"REPLACE_CERTIFICATE","operationStatus":"*****","resourceCertificateOperations":[{"resource":{"hostName":"<vCenter_Server_FQDN>","resourceType":"vcenter","master":false},"result":{"status":"FAILED","message":"{\"code\":\"CERTIFICATE_REPLACEMENT_FAILED_WITH_ERROR\",\"args\":[\"*****\",\"500 Internal Server Error: \\\"{\\\"type\\\":\\\"com.vmware.vapi.std.errors.error\\\",\\\"value\\\":{\\\"error_type\\\":\\\"ERROR\\\",\\\"messages\\\":[{\\\"args\\\":[\\\"Failed to notify APPLMGMT on http://localhost:1080/api/appliance/certificates/notification, on all retries.\\\"],\\\"default_message\\\":\\\"Exception found (Failed to notify APPLMGMT on http://localhost:1080/api/appliance/certificates/notification, on all retries.)\\\",\\\"id\\\":\\\"com.vmware.certificatemanagement.error\\\"}]}}\\\"\"]}"},"creationTimestamp":#############,"updateTimestamp":#############}],"retryOperation":false}

Errors reported in /var/log/vmware/applmgmt/applmgmt.log :

YYYY-MM-DDTHH:MM:SS:###Z [######]ERROR:root:Unable to authorize request with authz client: SoapException:faultcode: ns0:FailedAuthenticationfaultstring: Password of the user logging on is expired. :: Password of the user logging on is expired. :: User account expired: {Name: vmware-applmgmtservice-########-####-####-####-############, Domain: vsphere.local}faultxml: <?xml version='1.0' encoding='UTF-8'?><S:Envelope xmlns:S="http://schemas.xmlsoap.org/soap/envelope/"><S:Body><S:Fault xmlns:ns4="http://www.w3.org/2003/05/soap-envelope"><faultcode xmlns:ns0="http://docs.oasis-open.org/ws-sx/ws-trust/200512">ns0:FailedAuthentication</faultcode><faultstring>Password of the user logging on is expired. :: Password of the user logging on is expired. :: User account expired: {Name: vmware-applmgmtservice-########-####-####-####-############, Domain: vsphere.local}</faultstring></S:Fault></S:Body></S:Envelope>

Environment

VMware Cloud Foundation 4.x

VMware Cloud Foundation 5.x

Cause

This issue is caused due to the applmgmt (Appliance management) solution user credential not being updated during the certificate replacement

Resolution

On the vCenter Server :

SSH to the VCSA using the root account.
Restart the Appliance management service (applmgmt) and Lighttpd web server service service using the below commands :

service-control --restart applmgmt

For vCenter Server 8.0 U3 and later :

systemctl restart cap-lighttpd

For versions prior to vCenter Server 8.0 U3 :

systemctl restart vami-lighttp
Validate the status of both the services after restart, using the following commands :
service-control --status applmgmt

For vCenter Server 8.0 U3 and later :

systemctl status cap-lighttpd

For versions prior to vCenter Server 8.0 U3 :

systemctl status vami-lighttp

On the SDDC Manager :

Proceed with the replacement of the certificate on vCenter Server after generating a new CSR on SDDC Manager. Refer to : Install Third-Party CA-Signed Certificates Using Server Certificate and Certificate Authority Files