vSphere functions in Aria Orchestrator stop working: ephemeral certificate lifetime is too short

Products

• VCF Operations/Automation (formerly VMware Aria Suite)

Issue/Introduction

Possible symptoms:

• Scheduled workflows in Orchestrator stop working after one week with errors such as the below seen in the logs (vco-server-app.log):
  • ERROR vco [host='vco-app-########-####' thread='tokenLifetimeMonitorScheduler-1' user='' org='' trace=''] {} com.vmware.o11n.security.session.ManagedTokenRegistryImpl - Unable to convert token with id ################################
com.vmware.vcac.authentication.http.SamlAuthenticationException: Token expiration date: <DATE> is in the past.
  • ERROR vco [host='vco-app-########-####' thread='tokenLifetimeMonitorScheduler-1' user='' org='' trace=''] {} com.vmware.vim.sso.client.impl.SoapBindingImpl - SOAP fault
com.sun.xml.ws.fault.ServerSOAPFaultException: Client received SOAP Fault from server: Unable to renew non-renewable token Please see the server log to find more detail regarding exact cause of the failure.
• Workflows may fail about every 90 minutes since the token is renewed without extending the expiry time:
  • INFO vco [host='vco-app-########-####' thread='http-nio-8280-exec-##' user='<username>' org='-' trace='####-##-##-##-######'] {} com.vmware.o11n.security.sso.support.SamlTokenLifetimeService - Token renewed successfully: id _####-##-##-##-######, new expiration date: <DATE1>
  • ERROR vco [host='vco-app-########-####' thread='http-nio-8280-exec-##' user='-' org='-' trace='-'] {} com.vmware.vim.sso.client.impl.SoapBindingImpl - SOAP fault
com.sun.xml.ws.fault.ServerSOAPFaultException: Client received SOAP Fault from server: EndTime: <DATE1> is not after startTime: <DATE2> Please see the server log to find more detail regarding exact cause of the failure.
• vCenters may be listed as "unusable" in Orchestrator -> Administration -> Inventory -> vCenter plugin.
• If vCenter SSO is used for authentication, logging in to Orchestrator may show the  "${message}  ${backToLoginLabel}" error screen:
  • 

 

 

Environment

Aria Automation/Orchestrator 8.18.1

VCF Automation/Orchestrator 9.0.1

Cause

Aria Orchestrator fails to renew its access token to vCenter appropriately due to ephemeral certificate expiry.

Resolution

Follow the steps below to resolve the issue:

    1. Install Orchestrator 8.18.1 Patch 3 on the affected environment.
    2. Add the custom property com.vmware.o11n.sso.svcaccount.ephemeral-cert-lifetime-ms to the affected Orchestrator node as described in KB 408782.
    3. Configure the custom property com.vmware.o11n.authentication.sts.SamlTokenService.retryRenew and set its value to true using the following command:
            vracli vro properties set -k "com.vmware.o11n.authentication.sts.SamlTokenService.retryRenew" -v "true"
            
    4. Remove the existing schedule associated with the affected workflow.
    5. Recreate or reschedule the workflow to ensure the updated configuration takes effect.

Note: Although the ephemeral-cert-lifetime-ms property was introduced in Patch 4 of standalone Aria orchestrator and Patch 5 of Embedded Orchestrator ( Aria automation), the SamlTokenService.retryRenew property is planned for inclusion only in a future release. The manual addition of the property described in Step 3 is still required to fully resolve the issue.