Replacement of vCenter Certificates using SDDC Manager fails with "Failed to notify APPLMGMT"
Article ID: 413611
Updated On:
Products
Issue/Introduction
- While replacing the certificates on vCenter Server with CA-signed certificates using the certificate management of SDDC Manager, the certificate replacement is completed successfully; however, an error stating "Failed to notify APPLMGMT" is reported.
- The certificate details are updated on the SDDC Manager, however, the task shows as failed.
- Errors reported in /var/log/vmware/vcf/operationsmanager/operationsmanager.log :
YYYY-MM-DDTHH:MM:SS:###Z DEBUG [vcf_om,################################,####] [c.v.v.c.s.f.i.CertificateOperationsFacadeImpl,http-nio-127.0.0.1-7300-exec-8] Get: REPLACE_CERTIFICATE operation details from storeYYYY-MM-DDTHH:MM:SS:###Z DEBUG [vcf_om,################################,####] [c.v.v.c.s.f.i.CertificateOperationsFacadeImpl,http-nio-127.0.0.1-7300-exec-8] DomainCertificateOperation: {"workflowId":"########-####-####-############-####","domainName":"Management","operationType":"REPLACE_CERTIFICATE","operationStatus":"*****","resourceCertificateOperations":[{"resource":{"hostName":"<vCenter_Server_FQDN>","resourceType":"vcenter","master":false},"result":{"status":"FAILED","message":"{\"code\":\"CERTIFICATE_REPLACEMENT_FAILED_WITH_ERROR\",\"args\":[\"*****\",\"500 Internal Server Error: \\\"{\\\"type\\\":\\\"com.vmware.vapi.std.errors.error\\\",\\\"value\\\":{\\\"error_type\\\":\\\"ERROR\\\",\\\"messages\\\":[{\\\"args\\\":[\\\"Failed to notify APPLMGMT on http://localhost:1080/api/appliance/certificates/notification, on all retries.\\\"],\\\"default_message\\\":\\\"Exception found (Failed to notify APPLMGMT on http://localhost:1080/api/appliance/certificates/notification, on all retries.)\\\",\\\"id\\\":\\\"com.vmware.certificatemanagement.error\\\"}]}}\\\"\"]}"},"creationTimestamp":#############,"updateTimestamp":#############}],"retryOperation":false}
- Errors reported in /var/log/vmware/applmgmt/applmgmt.log :
YYYY-MM-DDTHH:MM:SS:###Z [######]ERROR:root:Unable to authorize request with authz client: SoapException:faultcode: ns0:FailedAuthenticationfaultstring: Password of the user logging on is expired. :: Password of the user logging on is expired. :: User account expired: {Name: vmware-applmgmtservice-########-####-####-####-############, Domain: vsphere.local}faultxml: <?xml version='1.0' encoding='UTF-8'?><S:Envelope xmlns:S="http://schemas.xmlsoap.org/soap/envelope/"><S:Body><S:Fault xmlns:ns4="http://www.w3.org/2003/05/soap-envelope"><faultcode xmlns:ns0="http://docs.oasis-open.org/ws-sx/ws-trust/200512">ns0:FailedAuthentication</faultcode><faultstring>Password of the user logging on is expired. :: Password of the user logging on is expired. :: User account expired: {Name: vmware-applmgmtservice-########-####-####-####-############, Domain: vsphere.local}</faultstring></S:Fault></S:Body></S:Envelope>
Environment
VMware Cloud Foundation 4.x
VMware Cloud Foundation 5.x
Cause
This issue is caused due to the applmgmt solution user credential not being updated during the certificate replacement
Resolution
On the vCenter Server :
- SSH to the vCenter Server using the root account.
- Restart the Appliance management service (applmgmt) and Lighttpd web server service service using the below commands :
service-control --restart applmgmt
For vCenter Server 8.0 U3 and later :
systemctl restart cap-lighttpd
For versions prior to vCenter Server 8.0 U3 :
systemctl restart vami-lighttp - Validate the status of both the services after restart, using the following commands :
service-control --status applmgmt
For vCenter Server 8.0 U3 and later :
systemctl status cap-lighttpd
For versions prior to vCenter Server 8.0 U3 :
systemctl status vami-lighttp
On the SDDC Manager :
- Proceed with the replacement of the certificate on vCenter Server after generating a new CSR on SDDC Manager. Refer to : Install Third-Party CA-Signed Certificates Using Server Certificate and Certificate Authority Files
Feedback
