Replacement of vCenter Server Certificates Using the API Reports "Failed to notify APPLMGMT"

Products

• VMware vCenter Server

Issue/Introduction

• While replacing the standard self-signed certificate on vCenter with CA signed certificates using an API replaces the certificate but reports an error "Failed to notify APPLMGMT"
• Using vSphere API endpoint https://{api_host}/api/vcenter/certificate-management/vcenter/tls a HTTP code 500 is returned with a JSON response

"json": {
    "error_type": "ERROR",
    "messages": [
      {
        "args": [
          "Failed to notify APPLMGMT on http://localhost:1080/api/appliance/certificates/notification, on all retries."
        ],
        "default_message": "Exception found (Failed to notify APPLMGMT on http://localhost:1080/api/appliance/certificates/notification, on all retries.)",
        "id": "com.vmware.certificatemanagement.error"
      }
    ]
  }

• Replacing the certificate from the vSphere UI under certificate management page fails with the error:

• Using PowerCLI Invoke-vSphereApiClient cmdlet the response is as follows

Invoke-vSphereApiClient: C:\Program Files\PowerShell\Modules\VMware.Sdk.vSphere.vCenter.CertManagement\8.0.2099.24145081\Api\TlsApi.ps1:1116:33
Line |
1116 | $invokeResult = Invoke-vSphereApiClient @invokeParams
     | ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
     | [ERROR] Exception found (Failed to notify APPLMGMT on http://localhost:1080/api/appliance/certificates/notification, on all retries.) 

• Errors seen in /var/log/vmware/vapi/endpoint/endpoint-access.log

2025-04-02T06:40:00.710Z | jetty-default-30261       | <session_id> | Invoking com.vmware.appliance.certificates.notification:notify
2025-04-02T06:40:00.750Z | vAPI-I/O dispatcher-0     | <session_id> | 127.0.0.1 - - [02/Apr/2025:06:40:00 +0000] "POST /api/appliance/certificates/notification HTTP/1.1" 403 142 "-" "Java/1.8.0_412" 40

• Errors seen in /var/log/vmware/applmgmt/applmgmt.log

2025-04-02T06:40:00 AM UTC [8512]DEBUG:vmware.vapi.security.jwt.jwt_authentication_handler:Authenticated user with username - machine-<UUID>
2025-04-02T06:40:00 AM UTC [8512]INFO:vmware.appliance.vapi.auth:Authorization request for service_id: com.vmware.appliance.certificates.notification, operation_id
: notify
2025-04-02T06:40:00 AM UTC [8512]ERROR:root:Unable to authorize request with authz client: SoapException:
faultcode: ns0:FailedAuthentication
faultstring: Password of the user logging on is expired. :: Password of the user logging on is expired. :: User account expired: {Name: vmware-applmgmtservice-<UUID>, Domain: ########.#######.#######.###}
faultxml: ns0:FailedAuthenticationPassword of the user log
ging on is expired. :: Password of the user logging on is expired. :: User account expired: {Name: vmware-applmgmtservice-<UUID>, Domain:  ########.#######.#######.###}

Environment

VMware vCenter Server

Cause

This is a known issue that affects certificate replacement through the API.

The errors show that the solution user is expired though it should renew with certificate replacement, leaving the system in a misconfigured state.

Resolution

To workaround this issue manually restart `applmgmt` service and `lighttpd.service`, then retry the certificate renewal.

• This can be labeled either "vami-lighttpd" or "cap-lighttpd" depending on version.
• To confirm the service name on the system in question, run the following command on the VCSA command line as a user with system administrative permission:
  • systemctl list-unit-files | grep lighttpd
• Once the service name is confirmed, you can restart the service (the example below is using "cap-lighttpd" as the service name but replace it with the service name listed from the command above, removing the ".service" suffix when running in this command).
  • service-control --restart applmgmt
  • systemctl restart cap-lighttpd
• To ensure both services are running as expected after restart, use the following commands (the example below is using "cap-lighttpd" as the service name but replace it with the service name listed from the command above, removing the ".service" suffix when running in this command). 
  • service-control --status applmgmt
  • systemctl status cap-lighttpd

• After restarting those services, retry the certificate renewal.