Scheduled workflows does not get triggered with the status shows as pending or error

Products

• VCF Operations/Automation (formerly VMware Aria Suite)

Issue/Introduction

• Scheduled workflows (tasks) are seen to expire and not run after 5 days has passed since scheduling them
• When recreated it runs for few days and stops again
• These errors are seen in the vco-app logs:   ( /services-logs/prelude/vco-server-app/file-logs/vco-server-app.log )

####-##-#####:##:##.###Z ERROR vco [host='###-###-######-####' thread='#############################-#' user='' org='' trace=''] {} com.vmware.vim.sso.client.impl.SoapBindingImpl - SOAP fault
com.sun.xml.ws.fault.ServerSOAPFaultException: Client received SOAP Fault from server: Unable to renew ###-######### token Please see the server log to find more detail regarding exact cause of the failure.
        at com.sun.xml.ws.fault.SOAP11Fault.getProtocolException(SOAP11Fault.java:163) ~[#####-##-#.3.5.jar:2.3.5]
        at com.sun.xml.ws.fault.SOAPFaultBuilder.createException(SOAPFaultBuilder.java:95) ~[#####-##-#.3.5.jar:2.3.5]
        at com.sun.xml.ws.client.dispatch.DispatchImpl.doInvoke(DispatchImpl.java:244) ~[#####-##-#.3.5.jar:2.3.5]
        at com.sun.xml.ws.client.dispatch.DispatchImpl.invoke(DispatchImpl.java:274) ~[#####-##-#.3.5.jar:2.3.5]

Environment

 Aria Automation Orchestrator 8.18.1

Cause

Authentication tokens are not renewed

Resolution

Broadcom is aware of this issue, follow the below workaround 

1.Take snapshot all the nodes in the cluster without memory (Do not skip this step)

2.SSH into the VA

3.Run the following command to remove property forcing database configuration:

base64 -d <<< "a3ViZWN0bCBleGVjIC1uIHByZWx1ZGUgIiQodnJhY2xpIHN0YXR1cyB8IGpxIC1yICcuZGF0YWJhc2VOb2Rlc1tdIHwgc2VsZWN0KC5Sb2xlPT0icHJpbWFyeSIpLiJOb2RlIG5hbWUiJyB8IGN1dCAtZCAnLicgLWYgMSkiIC0tIGNocHN0IC11IHBvc3RncmVzIHBzcWwgXAogICAgLWQgInZjby1kYiIgXAogICAgLWMgIgogICAgICBERUxFVEUgRlJPTSB2bW9fY29uZmlnaXRlbQogICAgICBXSEVSRSBuYW1lPSdjb20udm13YXJlLm8xMW4uZm9yY2UtZGF0YWJhc2UtY29uZmlndXJhdGlvbic7CiAgICAgICIgJiYgXAp2cmFjbGkgY2x1c3RlciBleGVjIC0tIHNlZCAtaSAnL2NvbS52bXdhcmUubzExbi5mb3JjZS1kYXRhYmFzZS1jb25maWd1cmF0aW9uL2QnIC9kYXRhL3Zjby91c3IvbGliL3Zjby9hcHAtc2VydmVyL2NvbmYvdm1vLnByb3BlcnRpZXM=" | bash -

4.Use the legacy command to re-register the authentication with the following steps.

• • Login into the Orchestrator server container
    
          kubectl exec -itn prelude $(kubectl get pod -n prelude -l app=vco-app -o jsonpath="{.items[0].metadata.name}") -c vco-server-app -- bash

  • Install the legacy configuration tool
    
         rpm -i --nodeps vco-cfg-cli.rpm && cp /usr/lib/vco/app-server/deploy/vco/WEB-INF/lib/* /usr/lib/vco-cli/lib

  • Re-configure the authentication. Replace VC_URL, VC_USERNAME, VC_PASSWORD, VC_ADMIN_GROUP, VC_ADMIN_GROUP_DOMAIN, VC_TENANT with the corresponding vCenter URL, Administrator Username, Administrator Password, Orchestrator Admin Group, Orchestrator Admin Group Domain and  vCenter Default Tenant (everything except the password can be found by running vracli vro authentication outside the container)
    
         /usr/lib/vco-cli/bin/vro-configure-inner.sh authentication-vsphere --register --lsUrl https://<FQDN> --username [email protected] --password "${VC_PASSWORD}" --adminGroup "${VC_ADMIN_GROUP}" --adminGroupDomain ${VC_ADMIN_GROUP_DOMAIN} --tenant ${VC_TENANT}

  • Restart the orchestrator server and exit from the container
    
        kill 1
    
    ***For clustered deployments, please perform the following 2 additional steps to ensure the database will sync between all the nodes.***
    • Force database sync on next pod restart:
      vracli cluster exec -- rm -f /data/vco/usr/lib/vco/app-server/conf/configured
    • Restart all pods:
      kubectl -n prelude delete pods -lapp=vco-app

5.Wait for orchestrator to start

   kubectl -n prelude get pods -w

6.Recreate the scheduled workflows.