Deployments fail due to timeout in VCF Automation 9.1.0.0 after upgrade
search cancel

Deployments fail due to timeout in VCF Automation 9.1.0.0 after upgrade

book

Article ID: 441703

calendar_today

Updated On:

Products

VCF Automation

Issue/Introduction

  • The issue specifically affects environments that have been upgraded to VCF Automation 9.1.0.0, where a blocking Orchestrator subscription was originally created in a previous version (VCF Automation 9.0.x or VMware Aria Automation 8.x).

  • VM deployments via VCF Automation 9.1.0.0 hang after triggering the first subscribed orchestrator workflow, and the deployment fails due to a timeout. The following error is observed in the UI :

    Extensibility triggered task failed. Event ID: <UUID>. Failure: Timeout: due was 'YYYY-MM-DDTHH:MM:SS.###Z' but expired

  • Additionally, the ebs service logs contain the following error :

    ERROR event-broker [host='ebs-app-#####-#####' thread='main-pool-45' user='' org='' trace='<TRACE_ID>' request-trace=''] c.v.a.e.b.s.s.impl.EventServiceImpl.lambda$processReplyEvent$25:333 - Error handling ReplyEvent[id=<EVENT_ID>, timeStamp=null, sourceType=extensibility.vro, sourceIdentity=extensibility.vro, originEventId=<EVENT_ID>, eventTraceEntryId=<EVENT_ID>]
    com.vmware.automation.spring.webflux.platform.server.service.exception.ValidationServiceException: 22027-Current user 'service-account-project-serviceaccount' is not authorized to reply on event trace entry '<UUID>' for event '<EVENT_ID>'.

Environment

VCF Automation 9.1.0.0

Cause

During the upgrade to VCF Automation 9.1.0.0, the system service account is modified. However, the ownership of Orchestrator subscriptions created in previous versions is not automatically updated to the new service account service-account-project-serviceaccount , leaving them owned by service-account-vro-gateway-serviceaccount. This causes authorization failures when the EBS (Event Broker Service) attempts to process reply events.

Resolution

This issue is targeted to be fixed in a future release. Once the fix is officially released, this article will be updated with the specific version and download instructions.

To resolve the issue in the interim, use one of the following two workarounds.

Option 1: Re-enabling Orchestrator Workflow Subscriptions

  1. Log in to the VCF Automation 9.1.0.0 UI.

  2. Navigate to Extensibility > Subscriptions.

  3. Identify, disable, and re-enable all Orchestrator subscriptions.

Option 2: Manual Database Update

This method involves modifying the subscriber_id directly in the ebs database.

  1. (Mandatory) Take an On-Demand Backup of the VCF Automation 9.1.0 from VCF Operations.

  2. Log in to VCF Automation 9.1.0.0 over SSH using the vmware-system-user account.

  3. Open a root shell by running:

    sudo su -

  4. Apply the patch by running the following command:

    base64 -d <<< "IyEvYmluL2Jhc2gKCiMgQ29weXJpZ2h0IChjKSAyMDI2IEJyb2FkY29tLiBBbGwgUmlnaHRzIFJlc2VydmVkLgojIEJyb2FkY29tIENvbmZpZGVudGlhbC4gVGhlIHRlcm0gIkJyb2FkY29tIiByZWZlcnMgdG8gQnJvYWRjb20gSW5jLgojIGFuZC9vciBpdHMgc3Vic2lkaWFyaWVzLgoKSVNTVUU9IlZDRkNPTi01NjM0MSIKTkFNRVNQQUNFPSJwcmVsdWRlIgpPTERfU1VCU0NSSUJFUj0ic2VydmljZS1hY2NvdW50LXZyby1nYXRld2F5LXNlcnZpY2VhY2NvdW50IgpORVdfU1VCU0NSSUJFUj0ic2VydmljZS1hY2NvdW50LXByb2plY3Qtc2VydmljZWFjY291bnQiCkRCPSJlYnNfZGIiCgp0aW1lc3RhbXA9JChkYXRlICcrJVktJW0tJWQtJUgtJU0tJVMnKQpXT1JLX0RJUj0icGF0Y2gtJElTU1VFLSR0aW1lc3RhbXAiCm1rZGlyIC1wICIkV09SS19ESVIiCkxPR19GSUxFPSIkV09SS19ESVIvcGF0Y2gubG9nIgoKZXhlYyAyPiA+KHRlZSAtYSAiJExPR19GSUxFIikKZXhlYyA+JjIKCmg9JChob3N0bmFtZSkKbG9nKCkgewogICAgZWNobyAiWyQxXVskKGRhdGUgIislWS0lbS0lZC0lSC0lTS0lUyIpXVskaF0gJDIiID4mMgp9Cgpsb2cgSU5GTyAiQXBwbHlpbmcgcGF0Y2ggZm9yICRJU1NVRSAobG9nOiAkTE9HX0ZJTEUpIgoKaWYgW1sgLXogIiRLVUJFQ09ORklHIiBdXTsgdGhlbgogICAgZXhwb3J0IEtVQkVDT05GSUc9L2V0Yy9rdWJlcm5ldGVzL2FkbWluLmNvbmYKICAgIGxvZyBJTkZPICJLVUJFQ09ORklHIHNldCB0byAvZXRjL2t1YmVybmV0ZXMvYWRtaW4uY29uZiIKZmkKCmlmICEga3ViZWN0bCBjbHVzdGVyLWluZm8gJj4vZGV2L251bGw7IHRoZW4KICAgIGxvZyBFUlJPUiAia3ViZWN0bCBpcyBub3Qgd29ya2luZy4gRW5zdXJlIEtVQkVDT05GSUcgaXMgY29ycmVjdGx5IHNldCBhbmQgdGhlIGNsdXN0ZXIgaXMgYWNjZXNzaWJsZS4iCiAgICBleGl0IDEKZmkKCmxvZyBJTkZPICJGaW5kaW5nIFBvc3RncmVzIGxlYWRlciBwb2QuLi4iCkxFQURFUl9QT0Q9JChrdWJlY3RsIGV4ZWMgLW4gIiROQU1FU1BBQ0UiIHZjZmFwb3N0Z3Jlcy0wIC0tIHBhdHJvbmljdGwgbGlzdCAyPi9kZXYvbnVsbCB8IGF3ayAnL0xlYWRlci8ge3ByaW50ICQyfScpCmlmIFtbIC16ICIkTEVBREVSX1BPRCIgXV07IHRoZW4KICAgIGxvZyBFUlJPUiAiQ291bGQgbm90IGRldGVybWluZSB0aGUgUG9zdGdyZXMgbGVhZGVyIHBvZC4gQ2hlY2sgdGhhdCB0aGUgcG9zdGdyZXMgcG9kcyBhcmUgaGVhbHRoeS4iCiAgICBleGl0IDEKZmkKbG9nIElORk8gIlBvc3RncmVzIGxlYWRlciBwb2Q6ICRMRUFERVJfUE9EIgoKbG9nIElORk8gIkNoZWNraW5nIGZvciBhZmZlY3RlZCBzdWJzY3JpcHRpb25zIGluICREQi4uLiIKUk9XX0NPVU5UPSQoa3ViZWN0bCAtbiAiJE5BTUVTUEFDRSIgZXhlYyAiJExFQURFUl9QT0QiIC0tIHBzcWwgLVUgcG9zdGdyZXMgLWQgIiREQiIgLXRBYyBcCiAgICAiU0VMRUNUIENPVU5UKCopIEZST00gZWJzX3N1YnNjcmlwdGlvbiBXSEVSRSBzdWJzY3JpYmVyX2lkID0gJyRPTERfU1VCU0NSSUJFUic7IiAyPi9kZXYvbnVsbCB8IHRyIC1kICdbOnNwYWNlOl0nKQppZiBbWyAteiAiJFJPV19DT1VOVCIgXV07IHRoZW4KICAgIGxvZyBFUlJPUiAiRmFpbGVkIHRvIHF1ZXJ5IHRoZSBkYXRhYmFzZS4gQ2hlY2sgdGhhdCB0aGUgUG9zdGdyZXMgcG9kIGlzIGFjY2Vzc2libGUuIgogICAgZXhpdCAxCmZpCgpsb2cgSU5GTyAiRm91bmQgJFJPV19DT1VOVCBzdWJzY3JpcHRpb24ocykgd2l0aCBzdWJzY3JpYmVyX2lkPSckT0xEX1NVQlNDUklCRVInLiIKaWYgW1sgIiRST1dfQ09VTlQiIC1lcSAwIF1dOyB0aGVuCiAgICBsb2cgSU5GTyAiTm8gYWZmZWN0ZWQgc3Vic2NyaXB0aW9ucyBmb3VuZC4gVGhlIHBhdGNoIG1heSBhbHJlYWR5IGhhdmUgYmVlbiBhcHBsaWVkIG9yIHRoZSBlbnZpcm9ubWVudCBpcyBub3QgYWZmZWN0ZWQuIgogICAgZXhpdCAwCmZpCgpsb2cgSU5GTyAiVXBkYXRpbmcgc3Vic2NyaWJlcl9pZCBmcm9tICckT0xEX1NVQlNDUklCRVInIHRvICckTkVXX1NVQlNDUklCRVInLi4uIgprdWJlY3RsIC1uICIkTkFNRVNQQUNFIiBleGVjICIkTEVBREVSX1BPRCIgLS0gcHNxbCAtVSBwb3N0Z3JlcyAtZCAiJERCIiAtYyBcCiAgICAiVVBEQVRFIGVic19zdWJzY3JpcHRpb24gU0VUIHN1YnNjcmliZXJfaWQgPSAnJE5FV19TVUJTQ1JJQkVSJyBXSEVSRSBzdWJzY3JpYmVyX2lkID0gJyRPTERfU1VCU0NSSUJFUic7IgoKUkM9JD8KaWYgW1sgJFJDIC1uZSAwIF1dOyB0aGVuCiAgICBsb2cgRVJST1IgIkRhdGFiYXNlIHVwZGF0ZSBmYWlsZWQgd2l0aCBleGl0IGNvZGUgJFJDLiIKICAgIGV4aXQgMQpmaQoKbG9nIElORk8gIlZlcmlmeWluZyB1cGRhdGUuLi4iClJFTUFJTklORz0kKGt1YmVjdGwgLW4gIiROQU1FU1BBQ0UiIGV4ZWMgIiRMRUFERVJfUE9EIiAtLSBwc3FsIC1VIHBvc3RncmVzIC1kICIkREIiIC10QWMgXAogICAgIlNFTEVDVCBDT1VOVCgqKSBGUk9NIGVic19zdWJzY3JpcHRpb24gV0hFUkUgc3Vic2NyaWJlcl9pZCA9ICckT0xEX1NVQlNDUklCRVInOyIgMj4vZGV2L251bGwgfCB0ciAtZCAnWzpzcGFjZTpdJykKaWYgW1sgIiRSRU1BSU5JTkciIC1lcSAwIF1dOyB0aGVuCiAgICBsb2cgSU5GTyAiVmVyaWZpY2F0aW9uIHN1Y2Nlc3NmdWwuICRST1dfQ09VTlQgc3Vic2NyaXB0aW9uKHMpIHVwZGF0ZWQuIgogICAgbG9nIElORk8gIlBhdGNoICRJU1NVRSBhcHBsaWVkIHN1Y2Nlc3NmdWxseS4iCmVsc2UKICAgIGxvZyBFUlJPUiAiVmVyaWZpY2F0aW9uIGZhaWxlZC4gJFJFTUFJTklORyByb3cocykgd2l0aCBzdWJzY3JpYmVyX2lkPSckT0xEX1NVQlNDUklCRVInIHN0aWxsIHJlbWFpbi4iCiAgICBleGl0IDEKZmkK" | bash

  5. Confirm the update was applied:

    kubectl -n prelude exec -it vcfapostgres-0 -- psql -U postgres -d ebs_db -c "SELECT subscriber_id, COUNT(*) FROM ebs_subscription GROUP BY subscriber_id;"

    Verify that no rows remain for the subscriber_id service-account-vro-gateway-serviceaccount.

Additional Information

  • Logs for the patch used in Option 2 is written to /var/log/vmware/prelude/patch-VCFCON-56341-<timestamp>.log

  • The patch is idempotent: running it more than once is safe. If no affected rows are found, it exits without making changes.

Powered by Wolken Software