Error: ALLOCATE_IN_PROGRESS - Deployments fail due to timeout in VCF Automation 9.1.0.0 after upgrade
search cancel

Error: ALLOCATE_IN_PROGRESS - Deployments fail due to timeout in VCF Automation 9.1.0.0 after upgrade

book

Article ID: 441703

calendar_today

Updated On:

Products

VCF Automation

Issue/Introduction

VM deployments in VCF Automation 9.1.0 hang and eventually time out during the allocation phase. This occurs specifically after an upgrade from version 9.0.x 

Symptoms:

  • Deployments fail with: 
    Extensibility triggered task failed. Event ID: ####. Failure: Timeout: due was 'YYYY-MM-DD...' but expired.

      Or

Extensibility triggered task failed. Event ID: AAAAAAAA-AAAA-AAAA-AAAA-AAAAAAAAAAAA. Failure: Extensibility error received for topic compute.removal.post, eventId = 'AAAAAAAA-AAAA-AAAA-AAAA-AAAAAAAAAAAA': [10030] SubscriberID: service-account-vro-gateway-serviceaccount, RunnableID: CCCCCCCC-CCCC-CCCC-CCCC-CCCCCCCCCCCC and SubscriptionID: sub_1655987244762 failed with the following error: No reply from blocking subscription AD Remove server groups(id=sub_1655987244762) for event AAAAAAAA-AAAA-AAAA-AAAA-AAAAAAAAAAAA. Expire date 2026-06-24 11:38:50.682.. Original Task Error: 'Extensibility triggered task failed. Event ID: BBBBBBBB-BBBB-BBBB-BBBB-BBBBBBBBBBBB. Failure: Extensibility error received for topic compute.provision.pre, eventId = 'BBBBBBBB-BBBB-BBBB-BBBB-BBBBBBBBBBBB': [10030] SubscriberID: service-account-vro-gateway-serviceaccount, RunnableID: DDDDDDDD-DDDD-DDDD-DDDD-DDDDDDDDDDDD and SubscriptionID: sub_1655987133107 failed with the following error: No reply from blocking subscription AD Create server groups(id=sub_1655987133107) for event BBBBBBBB-BBBB-BBBB-BBBB-BBBBBBBBBBBB. Expire date 2026-06-24 11:33:17.878

  • EBS logs show: 
    ERROR event-broker [host='ebs-app-#####-#####' thread='main-pool-45' user='' org='' trace='<TRACE_ID>' request-trace=''] c.v.a.e.b.s.s.impl.EventServiceImpl.lambda$processReplyEvent$25:333 - Error handling ReplyEvent[id=<EVENT_ID>, timeStamp=null, sourceType=extensibility.vro, sourceIdentity=extensibility.vro, originEventId=<EVENT_ID>, eventTraceEntryId=<EVENT_ID>]
    com.vmware.automation.spring.webflux.platform.server.service.exception.ValidationServiceException: 22027-Current user 'service-account-project-serviceaccount' is not authorized to reply on event trace entry '<UUID>' for event '<EVENT_ID>'.

Environment

VCF Automation 9.1.0.0
Upgraded from 9.0.x

Cause

During upgrade, the system service account is modified, but ownership of existing Orchestrator subscriptions is not updated from service-account-vro-gateway-serviceaccount to service-account-project-serviceaccount. This causes authorization failures when the EBS (Event Broker Service) attempts to process reply events.

Resolution

This issue is targeted to be fixed in a future release. Once the fix is officially released, this article will be updated with the specific version and download instructions.

To resolve the issue in the interim, use one of the following two workarounds.

Option 1: Re-enabling Orchestrator Workflow Subscriptions

  1. Log in to the VCF Automation 9.1.0.0 UI.

  2. Navigate to Extensibility > Subscriptions.

  3. Identify, disable, and re-enable all Orchestrator subscriptions.

Option 2: Manual Database Update

This method involves modifying the subscriber_id directly in the ebs database.

  1. (Mandatory) Take an On-Demand Backup of the VCF Automation 9.1.0 from VCF Operations.

  2. Log in to VCF Automation 9.1.0.0 over SSH using the vmware-system-user account.

  3. Open a root shell by running:

    sudo su -

  4. Apply the patch by running the following command:

    base64 -d <<< "IyEvYmluL2Jhc2gKCiMgQ29weXJpZ2h0IChjKSAyMDI2IEJyb2FkY29tLiBBbGwgUmlnaHRzIFJlc2VydmVkLgojIEJyb2FkY29tIENvbmZpZGVudGlhbC4gVGhlIHRlcm0gIkJyb2FkY29tIiByZWZlcnMgdG8gQnJvYWRjb20gSW5jLgojIGFuZC9vciBpdHMgc3Vic2lkaWFyaWVzLgoKSVNTVUU9IlZDRkNPTi01NjM0MSIKTkFNRVNQQUNFPSJwcmVsdWRlIgpPTERfU1VCU0NSSUJFUj0ic2VydmljZS1hY2NvdW50LXZyby1nYXRld2F5LXNlcnZpY2VhY2NvdW50IgpORVdfU1VCU0NSSUJFUj0ic2VydmljZS1hY2NvdW50LXByb2plY3Qtc2VydmljZWFjY291bnQiCkRCPSJlYnNfZGIiCgp0aW1lc3RhbXA9JChkYXRlICcrJVktJW0tJWQtJUgtJU0tJVMnKQpXT1JLX0RJUj0icGF0Y2gtJElTU1VFLSR0aW1lc3RhbXAiCm1rZGlyIC1wICIkV09SS19ESVIiCkxPR19GSUxFPSIkV09SS19ESVIvcGF0Y2gubG9nIgoKZXhlYyAyPiA+KHRlZSAtYSAiJExPR19GSUxFIikKZXhlYyA+JjIKCmg9JChob3N0bmFtZSkKbG9nKCkgewogICAgZWNobyAiWyQxXVskKGRhdGUgIislWS0lbS0lZC0lSC0lTS0lUyIpXVskaF0gJDIiID4mMgp9Cgpsb2cgSU5GTyAiQXBwbHlpbmcgcGF0Y2ggZm9yICRJU1NVRSAobG9nOiAkTE9HX0ZJTEUpIgoKaWYgW1sgLXogIiRLVUJFQ09ORklHIiBdXTsgdGhlbgogICAgZXhwb3J0IEtVQkVDT05GSUc9L2V0Yy9rdWJlcm5ldGVzL2FkbWluLmNvbmYKICAgIGxvZyBJTkZPICJLVUJFQ09ORklHIHNldCB0byAvZXRjL2t1YmVybmV0ZXMvYWRtaW4uY29uZiIKZmkKCmlmICEga3ViZWN0bCBjbHVzdGVyLWluZm8gJj4vZGV2L251bGw7IHRoZW4KICAgIGxvZyBFUlJPUiAia3ViZWN0bCBpcyBub3Qgd29ya2luZy4gRW5zdXJlIEtVQkVDT05GSUcgaXMgY29ycmVjdGx5IHNldCBhbmQgdGhlIGNsdXN0ZXIgaXMgYWNjZXNzaWJsZS4iCiAgICBleGl0IDEKZmkKCmxvZyBJTkZPICJGaW5kaW5nIFBvc3RncmVzIGxlYWRlciBwb2QuLi4iCkxFQURFUl9QT0Q9JChrdWJlY3RsIGV4ZWMgLW4gIiROQU1FU1BBQ0UiIHZjZmFwb3N0Z3Jlcy0wIC0tIHBhdHJvbmljdGwgbGlzdCAyPi9kZXYvbnVsbCB8IGF3ayAnL0xlYWRlci8ge3ByaW50ICQyfScpCmlmIFtbIC16ICIkTEVBREVSX1BPRCIgXV07IHRoZW4KICAgIGxvZyBFUlJPUiAiQ291bGQgbm90IGRldGVybWluZSB0aGUgUG9zdGdyZXMgbGVhZGVyIHBvZC4gQ2hlY2sgdGhhdCB0aGUgcG9zdGdyZXMgcG9kcyBhcmUgaGVhbHRoeS4iCiAgICBleGl0IDEKZmkKbG9nIElORk8gIlBvc3RncmVzIGxlYWRlciBwb2Q6ICRMRUFERVJfUE9EIgoKbG9nIElORk8gIkNoZWNraW5nIGZvciBhZmZlY3RlZCBzdWJzY3JpcHRpb25zIGluICREQi4uLiIKUk9XX0NPVU5UPSQoa3ViZWN0bCAtbiAiJE5BTUVTUEFDRSIgZXhlYyAiJExFQURFUl9QT0QiIC0tIHBzcWwgLVUgcG9zdGdyZXMgLWQgIiREQiIgLXRBYyBcCiAgICAiU0VMRUNUIENPVU5UKCopIEZST00gZWJzX3N1YnNjcmlwdGlvbiBXSEVSRSBzdWJzY3JpYmVyX2lkID0gJyRPTERfU1VCU0NSSUJFUic7IiAyPi9kZXYvbnVsbCB8IHRyIC1kICdbOnNwYWNlOl0nKQppZiBbWyAteiAiJFJPV19DT1VOVCIgXV07IHRoZW4KICAgIGxvZyBFUlJPUiAiRmFpbGVkIHRvIHF1ZXJ5IHRoZSBkYXRhYmFzZS4gQ2hlY2sgdGhhdCB0aGUgUG9zdGdyZXMgcG9kIGlzIGFjY2Vzc2libGUuIgogICAgZXhpdCAxCmZpCgpsb2cgSU5GTyAiRm91bmQgJFJPV19DT1VOVCBzdWJzY3JpcHRpb24ocykgd2l0aCBzdWJzY3JpYmVyX2lkPSckT0xEX1NVQlNDUklCRVInLiIKaWYgW1sgIiRST1dfQ09VTlQiIC1lcSAwIF1dOyB0aGVuCiAgICBsb2cgSU5GTyAiTm8gYWZmZWN0ZWQgc3Vic2NyaXB0aW9ucyBmb3VuZC4gVGhlIHBhdGNoIG1heSBhbHJlYWR5IGhhdmUgYmVlbiBhcHBsaWVkIG9yIHRoZSBlbnZpcm9ubWVudCBpcyBub3QgYWZmZWN0ZWQuIgogICAgZXhpdCAwCmZpCgpsb2cgSU5GTyAiVXBkYXRpbmcgc3Vic2NyaWJlcl9pZCBmcm9tICckT0xEX1NVQlNDUklCRVInIHRvICckTkVXX1NVQlNDUklCRVInLi4uIgprdWJlY3RsIC1uICIkTkFNRVNQQUNFIiBleGVjICIkTEVBREVSX1BPRCIgLS0gcHNxbCAtVSBwb3N0Z3JlcyAtZCAiJERCIiAtYyBcCiAgICAiVVBEQVRFIGVic19zdWJzY3JpcHRpb24gU0VUIHN1YnNjcmliZXJfaWQgPSAnJE5FV19TVUJTQ1JJQkVSJyBXSEVSRSBzdWJzY3JpYmVyX2lkID0gJyRPTERfU1VCU0NSSUJFUic7IgoKUkM9JD8KaWYgW1sgJFJDIC1uZSAwIF1dOyB0aGVuCiAgICBsb2cgRVJST1IgIkRhdGFiYXNlIHVwZGF0ZSBmYWlsZWQgd2l0aCBleGl0IGNvZGUgJFJDLiIKICAgIGV4aXQgMQpmaQoKbG9nIElORk8gIlZlcmlmeWluZyB1cGRhdGUuLi4iClJFTUFJTklORz0kKGt1YmVjdGwgLW4gIiROQU1FU1BBQ0UiIGV4ZWMgIiRMRUFERVJfUE9EIiAtLSBwc3FsIC1VIHBvc3RncmVzIC1kICIkREIiIC10QWMgXAogICAgIlNFTEVDVCBDT1VOVCgqKSBGUk9NIGVic19zdWJzY3JpcHRpb24gV0hFUkUgc3Vic2NyaWJlcl9pZCA9ICckT0xEX1NVQlNDUklCRVInOyIgMj4vZGV2L251bGwgfCB0ciAtZCAnWzpzcGFjZTpdJykKaWYgW1sgIiRSRU1BSU5JTkciIC1lcSAwIF1dOyB0aGVuCiAgICBsb2cgSU5GTyAiVmVyaWZpY2F0aW9uIHN1Y2Nlc3NmdWwuICRST1dfQ09VTlQgc3Vic2NyaXB0aW9uKHMpIHVwZGF0ZWQuIgogICAgbG9nIElORk8gIlBhdGNoICRJU1NVRSBhcHBsaWVkIHN1Y2Nlc3NmdWxseS4iCmVsc2UKICAgIGxvZyBFUlJPUiAiVmVyaWZpY2F0aW9uIGZhaWxlZC4gJFJFTUFJTklORyByb3cocykgd2l0aCBzdWJzY3JpYmVyX2lkPSckT0xEX1NVQlNDUklCRVInIHN0aWxsIHJlbWFpbi4iCiAgICBleGl0IDEKZmkK" | bash

  5. Confirm the update was applied:

    kubectl -n prelude exec -it vcfapostgres-0 -- psql -U postgres -d ebs_db -c "SELECT subscriber_id, COUNT(*) FROM ebs_subscription GROUP BY subscriber_id;"

    Verify that no rows remain for the subscriber_id service-account-vro-gateway-serviceaccount.

Additional Information

  • Logs for the patch used in Option 2 is written to /var/log/vmware/prelude/patch-VCFCON-56341-<timestamp>.log

  • The patch is idempotent: running it more than once is safe. If no affected rows are found, it exits without making changes.

Powered by Wolken Software