Replacement of vCenter Server Certificates Using the API Reports "Failed to notify APPLMGMT"
search cancel

Replacement of vCenter Server Certificates Using the API Reports "Failed to notify APPLMGMT"

book

Article ID: 392844

calendar_today

Updated On:

Products

VMware vCenter Server VMware vCenter Server 8.0

Issue/Introduction

  • While replacing the standard self-signed certificate on vCenter with CA signed certificates using an API replaces the certificate but reports an error "Failed to notify APPLMGMT".

  • Using vSphere API endpoint https://{api_host}/api/vcenter/certificate-management/vcenter/tls a HTTP code 500 is returned with a JSON response

    "json": {
    "error_type": "ERROR",
    "messages": [
      {
        "args": [
          "Failed to notify APPLMGMT on http://localhost:1080/api/appliance/certificates/notification, on all retries."
        ],
        "default_message": "Exception found (Failed to notify APPLMGMT on http://localhost:1080/api/appliance/certificates/notification, on all retries.)",
        "id": "com.vmware.certificatemanagement.error"
      }
    ]
  }
  • Replacing the certificate from the vSphere UI under certificate management page fails with following error:

    [CERTIFICATE] Replace cert Failed: Exception found (Failed to notify APPLMGMT on http://localhost:1080/api/appliance/certificates/notification, on all
    retries.)

         

  • Using PowerCLI Invoke-vSphereApiClient cmdlet the response is as follows:

    Invoke-vSphereApiClient: C:\Program Files\PowerShell\Modules\VMware.Sdk.vSphere.vCenter.CertManagement\8.0.2099.24145081\Api\TlsApi.ps1:1116:33
Line |
1116 | $invokeResult = Invoke-vSphereApiClient @invokeParams

     | ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~|
     
    [ERROR] Exception found (Failed to notify APPLMGMT http://localhost:1080/api/appliance/certificates/notification, on all retries.)
  • Following log snippets are observed at vCenter: 

    /var/log/vmware/vapi/endpoint/endpoint-access.log
     
    YYYY-MM-DDTHH:MM:SS.710Z | jetty-default-30261       | <session_id> | Invoking                  com.vmware.appliance.certificates.notification:notify
YYYY-MM-DDTHH:MM:SS.750Z | vAPI-I/O dispatcher-0     | <session_id> |- - [DD/MM/YY-MM-DDTHH:MM:SS+0000] "POST / api/appliance/certificates/notification HTTP/1.1" 403 142 "-" "Java/1.8.0_412" 40

    /var/log/vmware/applmgmt/applmgmt.log
     
    YYYY-MM-DDTHH:MM:SS AM UTC [8512]DEBUG:vmware.vapi.security.jwt.jwt_authentication_handler:Authenticated user with username -  machine-<UUID>
YYYY-MM-DDTHH:MM:SS AM UTC [8512]INFO:vmware.appliance.vapi.auth:Authorization request for service_id:
com.vmware.appliance.certificates.notification, operation_id : notify
YYYY-MM-DDTHH:MM:SS AM UTC [8512]ERROR:root:Unable to authorize request with authz client: SoapException:
faultcode: ns0:FailedAuthentication
faultstring: Password of the user logging on is expired. :: Password of the user logging on is expired. :: User account expired: {Name: vmware-applmgmtservice-<UUID>, Domain: ########.#######.#######.###}
faultxml: ns0:FailedAuthenticationPassword of the user logging on is expired. :: Password of the user logging on is expired. :: User account expired: {Name: vmware-applmgmtservice-<UUID>, Domain:  ########.#######.#######.###}

    /var/log/vmware/vsphere-ui/logs/vsphere_client_virgo.log

    [YYYY-MM-DDThh:mm:ss] [INFO ] vc-service-async-pool-288304  c.v.v.p.e.propertycollector.SolutionInstallPropertyCollector      Scheduling re-subscription with delay of 5000 milliseconds.
[YYYY-MM-DDThh:mm:ss] [ERROR] nio-127.0.0.1-5090-exec-3950  com.vmware.vise.mvc.exception.GlobalExceptionHandler              Exception handled while processing request for /ui/certificate-ui/ctrl/certificates/renew-machine-cert:  com.vmware.vapi.std.errors.Error: Error (com.vmware.vapi.std.errors.error) => {
    messages = [LocalizableMessage (com.vmware.vapi.std.localizable_message) => {
    id = com.vmware.certificatemanagement.error,
    defaultMessage = Exception found (Failed to notify APPLMGMT on http://localhost:1080/api/appliance/certificates/notification, on all retries.),
    args = [Failed to notify APPLMGMT on http://localhost:1080/api/appliance/certificates/notification, on all retries.],
    params = <null>,
    localized = <null>
}],
    data = <null>,
    errorType = ERROR
}
        at java.lang.Thread.getStackTrace(Thread.java:1564)
        at com.vmware.vapi.bindings.client.AsyncCallbackSyncAdapter.get_aroundBody1$advice(AsyncCallbackSyncAdapter.java:49)
        at com.vmware.vapi.bindings.client.AsyncCallbackSyncAdapter.get(AsyncCallbackSyncAdapter.java:1)
        at com.vmware.vapi.internal.bindings.Stub.invokeMethod(Stub.java:145)
        at com.vmware.vcenter.certificate_management.vcenter.TlsStub.renew(TlsStub.java:123)
Caused by: com.vmware.vapi.std.errors.Error: Error (com.vmware.vapi.std.errors.error) => {
    messages = [LocalizableMessage (com.vmware.vapi.std.localizable_message) => {
    id = com.vmware.certificatemanagement.error,
    defaultMessage = Exception found (Failed to notify APPLMGMT on http://localhost:1080/api/appliance/certificates/notification, on all retries.),
    args = [Failed to notify APPLMGMT on http://localhost:1080/api/appliance/certificates/notification, on all retries.],
    params = <null>,
    localized = <null>
}],
    data = <null>,
    errorType = ERROR
}
        at com.vmware.vapi.std.errors.Error._newInstance2(Error.java:671)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at com.vmware.vapi.internal.bindings.convert.impl.JavaClassStructConverter.createStructBinding(JavaClassStructConverter.java:275)
        at com.vmware.vapi.internal.bindings.convert.impl.JavaClassStructConverter.fromValue(JavaClassStructConverter.java:79)
        at com.vmware.vapi.internal.bindings.convert.impl.JavaClassStructConverter.fromValue(JavaClassStructConverter.java:33)
        at com.vmware.vapi.internal.bindings.TypeConverterImpl$ValueToJavaVisitor.visit(TypeConverterImpl.java:332)
        at com.vmware.vapi.bindings.type.ErrorType.accept(ErrorType.java:31)
        ... 1 common frames omitted




      

Cause

  • This is a known issue that affects certificate replacement through the API or vCenter UI.
  • The errors show that the solution user is expired though it should renew with certificate replacement, leaving the system in a misconfigured state.

Resolution

This issue is resolved in vCenter Server 8.0 Update 3g (Build 24853646). 

Workaround: 

If an immediate upgrade is not possible, follow the steps below to restart the necessary management services and retry the certificate replacement operation.

  1. Log in to the vCenter Server Appliance via SSH as root.

  2. Identify the correct service name for lighttpd. Depending on the vCenter version, this service is labeled as either "vami-lighttpd" or "cap-lighttpd". Run the following command to confirm which one is present:
     systemctl list-unit-files | grep lighttpd

     3. Restart the Appliance Management service and the specific lighttpd service identified in the step 2.(for example: cap-lighttpd or vami-lighttpd).
          service-control --restart applmgmt
     systemctl restart <service_name>

     5. Verify that both services are active and running:
          service-control --status applmgmt
     systemctl status <service_name>

Once the services are confirmed running, retry the certificate replacement operation.

Powered by Wolken Software