Embedded Orchestrator Integration Shows Incorrect Endpoint After Upgrade from Aria Automation 8.18.x to VCFA 9.x (VM Apps Tenancy)

Products

• VCF Automation

Issue/Introduction

After upgrading from Aria Automation 8.18.x to VCFA 9.x, one or more classic tenants observe the following:

The Embedded Orchestrator integration in the VCFA UI is not displayed as Embedded.

Orchestrator-based catalog items, extensibility subscriptions, or workflows fail or behave unexpectedly.

Inspecting the endpoint_state table in the provisioning database reveals that the affected vro endpoint record has hostName pointing to external VRO cluster node FQDNs (e.g., https://<external-vro-fqdn>:443) instead of the expected internal placeholder https://embedded.orchestrator:443.

Environment

VCF Automation 9.x

Cause

During the upgrade from Aria Automation 8.18.x to VCFA 9.x, the migration process may fail to correctly remap existing classic tenant vRO endpoint records to the Embedded Orchestrator placeholder URL (https://embedded.orchestrator:443).
As a result, the tenant's endpoint_state record retains the pre-upgrade external cluster node hostnames and is missing fields required by the VCFA 9.x Embedded Orchestrator integration.

Resolution

Engineering has identified this as known issue and working on fix for upcoming release.  

Workaround:

Important: This procedure is only applicable when:

The affected tenant uses Classic tenancy.  The tenant has no external Orchestrator integrations and only the Embedded one.
The SQL UPDATE identifies the incorrect record using a negation (<>). If external endpoints exist, they would also be incorrectly updated unless they are explicitly excluded with additional WHERE clause.

Step 0: Back up the environment

Take an on-demand backup of VCF Automation 9.x from VCF Operations before making any changes.

Step 1: SSH into the VCFA platform node and open a root shell

ssh vmware-system-user@<VCFA-Platform-FQDN>
sudo su -

Step 2: Identify the Patroni Leader pod

The Postgres cluster in VCFA 9.x is managed by Patroni (Spilo). Write operations must be executed on the Leader node.

LEADER_POD=$(kubectl exec -n prelude vcfapostgres-0 -- patronictl list 2>/dev/null \
| awk '/Leader/ {print $2}')
echo "Leader pod: $LEADER_POD"

Verify the output is a non-empty pod name (e.g., vcfapostgres-0 or vcfapostgres-1).

You can confirm the node is the primary by running:

kubectl -n prelude exec -it "$LEADER_POD" -c postgres -- \
psql -U postgres -t -A -c "SELECT pg_is_in_recovery();"

Expected output: f (false = primary). If the result is t, do not proceed — re-identify the leader.

Step 3: Open a psql session to the provisioning database

kubectl -n prelude exec -it "$LEADER_POD" -c postgres -- \
psql -U postgres -d provisioning_db

Step 4: Inspect the current state (optional but recommended)

Before modifying data, verify the affected records to ensure the condition holds (no external endpoints):

SELECT id,
custom_properties ->> 'hostName' AS cp_hostname,
endpoint_properties ->> 'hostName' AS ep_hostname,
endpoint_type
FROM endpoint_state
WHERE endpoint_type = 'vro'
ORDER BY custom_properties ->> 'hostName';

Expected output: at least one vro row having https://embedded.orchestrator:443 (correct) and one or more rows with an incorrect hostname.

Step 5: Apply the fix by copying the SQL below into a plain-text file before pasting into the psql session to avoid smart-quote substitution by terminal emulators or rich-text tools (which would cause a syntax error).

UPDATE endpoint_state AS dest
SET
custom_properties = (dest.custom_properties || jsonb_build_object(
'hostName', src.custom_properties -> 'hostName',
'privateKeyId', src.custom_properties -> 'privateKeyId',
'__nodeNames', src.custom_properties -> '__nodeNames',
'apiEndpoint', src.custom_properties -> 'apiEndpoint',
'certificate', src.custom_properties -> 'certificate',
'__originalDefaultEndpointName', src.custom_properties -> '__originalDefaultEndpointName'
)) - 'acceptSelfSignedCertificate',
endpoint_properties = (dest.endpoint_properties || jsonb_build_object(
'dcId', src.endpoint_properties -> 'dcId',
'hostName', src.endpoint_properties -> 'hostName',
'privateKey', src.endpoint_properties -> 'privateKey',
'apiEndpoint', src.endpoint_properties -> 'apiEndpoint',
'privateKeyId', src.endpoint_properties -> 'privateKeyId'
)) - 'acceptSelfSignedCertificate'
FROM endpoint_state AS src
WHERE src.endpoint_type = 'vro'
AND src.custom_properties ->> 'hostName' = 'https://embedded.orchestrator:443'
AND dest.endpoint_type = 'vro'
AND dest.custom_properties ->> 'hostName' <> 'https://embedded.orchestrator:443';

Expected output:
The number of rows being updated. For example:

UPDATE 1

If the output is UPDATE 0, no matching incorrect record was found, verify the inspection query from Step 4.

Step 6: Exit psql

\q

Verification: 
Log in to the VCFA UI.
Navigate to Infrastructure → Integrations → Orchestrator.
Confirm that the Orchestrator integration is displayed as Embedded and its status is healthy.
Optionally trigger a simple vRO workflow or check an extensibility subscription to confirm end-to-end connectivity.

Additional Information

Notes:

VM Apps tenancy only. In All Apps tenancy, the Embedded Orchestrator endpoint is exclusively owned by the Provider Consumption Organization and is provisioned differently. This procedure does not apply.
No external endpoints. The <> filter in the WHERE clause is safe only when no external VRO endpoints exist. Always run the inspection query in Step 4 first.
Avoid rich-text copy-paste. Pasting SQL from Slack, email, or Word may silently replace ASCII single quotes (') with typographic curly quotes ('/'), causing a PostgreSQL syntax error. Always paste via a plain-text editor or use a .sql file.