Custom Resource Actions Disappear in VCFA 9.1

Products

• VCF Automation

Issue/Introduction

In VMware Cloud Foundation Automation (VCFA) 9.1 environments, custom resource actions exhibit the following behavior:

• Custom resource actions for a deployment or deployment resource like Cloud.vSphere.Machine disappear from the UI action list
  
  
• Calling the deployment actions REST API omits the custom action from the JSON response array while standard built-in actions remain present.
  
  
• Navigating to Design > Resource Actions and re-saving the action temporarily restores visibility in the UI, but the action vanishes again later anywhere from about 30 minutes up to a few hours, depending on timing.

Environment

• VCF Automation (VMware Cloud Foundation Automation) 9.1.0
• VCF Automation (VMware Cloud Foundation Automation) 9.1.1

Cause

This issue is caused by a defect in the ProviderRegistrationService component, which is invoked by the ProviderRegistrationScheduler task running inside the catalog-service-app pod.

 

Every 30 minutes, this ProviderRegistrationScheduler task checks whether a provider's declared catalog has changed since the last run. When a change is detected, the ProviderRegistrationService rewrites the provider's resource types and actions.

 

However, due to a bug in this rewrite step, the process can incorrectly delete all actions attached to the affected resource types, including custom actions belonging to unrelated providers. Only the resource types and actions belonging to the provider being re-registered (its own, built-in ones) are automatically restored immediately afterward — custom actions belonging to other providers are not, and remain missing until manually resaved.

 

Additionally, an unrelated defect in how the catalog data is internally ordered when generating this comparison causes the scheduler to frequently detect a "change" even when the catalog hasn't meaningfully changed, so this deletion can be triggered far more often than an actual catalog update would warrant.

Resolution

A permanent software fix will be included in VCF Automation Version 9.1.1.0200

 

Workaround

Until the permanent fix is available, deploy the automated resave script (attached to the KB) and increase the provider registration scheduler interval. Once both steps are complete, no further manual action is required — the script runs unattended on its own schedule and automatically restores any custom Day-2 action removed by this bug. You do not need to repeat these steps.

 

Requirements: The script is written in Python therefore a Python 3 runtime environment needs to be installed on the machine which periodically will execute the script.

 

 

Step 1 — Deploy the automated resave script

The script periodically re-saves all custom Day-2 actions via the API.

1. Copy resave-script.zip attached to this KB to an appropriate location on the system where the script will be executed (a jump server, or the VCFA host itself).

2. Extract the archive.

   • Linux/Photon OS:

     python3 -m zipfile -e resave-script.zip .

   • Windows: 

     python -m zipfile -e resave-script.zip .

3. Go to the vcfa-resave directory.

4. Download the VCFA certificate. The certificate should be saved in vcfa-resave directory.

   • Linux/Photon OS:

     openssl s_client -connect <VCFA_HOST>:443 -showcerts </dev/null 2>/dev/null | sed -n '/-----BEGIN CERTIFICATE-----/,/-----END CERTIFICATE-----/p' > vcfa-cert.pem

   • Windows:

     • Navigate to https://<VCFA_HOST> in your browser.
     • Click the padlock (view site information) icon to the left of the URL.
     • In the popup that appears, select "Connection is secure" (or similar), then click "Certificate is valid" to open the certificate viewer.
     • In the Certification Path tab, select the topmost (root) certificate in the chain, click "View Certificate" → Details tab → "Copy to File...", and in the export wizard choose "Base-64 encoded X.509 (.CER)". Save it (e.g. vcfa-cert.cer).

     ⓘ The click path above (padlock → "Connection is secure" → "Certificate is valid") matches Chrome at the time of writing. Other browsers, or a future browser update, may use slightly different wording or icons for the same flow. In general: after clicking the padlock/site-information icon, look for a certificate or "connection is secure" option in the popup — this leads to the certificate viewer, after which the Certification Path → root certificate → Copy to File steps above still apply the same way.

5. Get an API token for the VCFA user.

   • Log in to VCFA.
   • Click the user name in the top-right corner → My Account → API Tokens tab → New.
   • Enter a client name (e.g. resave-custom-action) → Create. Copy the token.
   • Create a file named credentials in the same directory, containing exactly one line: 

     VCFA_REFRESH_TOKEN=<the token you copied>

   • On Linux/Photon OS, restrict its permissions: 

     chmod 600 credentials

   ⓘ Note: this API token is valid for three months from creation. Before it expires, generate a new token and replace the value in the credentials file — if the token is allowed to expire first, the script will start failing until it's replaced. No restart of the scheduled task/timer is required after updating the file; the next scheduled run will automatically pick up the new token.

6. Get the VCFA org name and org id.

   • Log in to VCFA.
   • Click the user name in the top-right corner and note the organization id and organization name (the org name, not the display name).

7. Perform a dry run to verify the script can authenticate to VCFA and correctly identify all the custom Day-2 actions it would resave, without making any actual changes.

   • Linux/Photon OS:

     python3 resave_custom_actions.py https://<VCFA_HOST> <org_name> <org_id> --creds-file ./credentials --cacert ./vcfa-cert.pem --dry-run

   • Windows:

     python resave_custom_actions.py https://<VCFA_HOST> <org_name> <org_id> --creds-file .\credentials --cacert .\vcfa-cert.cer --dry-run

   
   Messages are logged to resave_actions.log in the same directory. A successful dry run looks like:

   2026-08-13T12:37:31+05:30 [DRY RUN] Would resave 'Deployment.custom.customaction1'.
   2026-08-13T12:37:32+05:30 [DRY RUN] Would resave 'Cloud.vSphere.Machine.custom.customaction2'.
   2026-08-13T12:37:32+05:30 [DRY RUN] Resave complete: 2 succeeded, 0 missing, 0 failed.

   A successful actual run (without --dry-run) looks like:

   2026-08-13T12:37:52+05:30 Resaved 'Deployment.custom.customaction1' successfully.
   2026-08-13T12:37:54+05:30 Resaved 'Cloud.vSphere.Machine.custom.customaction2' successfully.
   2026-08-13T12:37:54+05:30 Resave complete: 2 succeeded, 0 missing, 0 failed.

8. Schedule the script to run every 20 minutes.

   • Linux/Photon OS (systemd):

     ⓘ Creating files under /etc/systemd/system/ needs root — if a plain editor without sudo fails with "can't open file for writing," use sudo <editor>.

     1. Create /etc/systemd/system/vcfa-resave-custom-actions.service: 

        [Unit]
        Description=Resave VCFA custom resource actions
        
        [Service]
        Type=oneshot
        User=<name of the logged in user>
        ExecStart=/usr/bin/python3 <absolute path resave_custom_actions.py> https://<host> <org-name> <org-id> --creds-file <absolute path credentials file> --cacert <absolute path vcfa-cert.pem>

     2. Create /etc/systemd/system/vcfa-resave-custom-actions.timer: 

        [Unit]
        Description=Run resave_custom_actions.py
        
        [Timer]
        OnBootSec=5min
        OnUnitActiveSec=20min
        
        [Install]
        WantedBy=timers.target

     3. Then, run the following commands to enable the service: 

        sudo systemctl daemon-reload
        sudo systemctl enable --now vcfa-resave-custom-actions.timer

   • Windows (Task Scheduler):

     1. Create a batch wrapper run-vcfa.bat in the same directory: 

        @echo off
        cd /d "<\path\to>\vcfa-resave"
        <\path\to>\python.exe resave_custom_actions.py https://<host> <org-name> <org-id> --creds-file credentials --cacert vcfa-cert.cer

     2. Open Task Scheduler → Create Task...

        • General tab: name it vcfa-resave-custom-actions; select "Run whether user is logged on or not."
        • Triggers tab → New... → On a schedule → One time, set a start date/time → expand Advanced settings → check "Repeat task every: 20 minutes" for a duration of "Indefinitely."
        • Actions tab → New... → Action: "Start a program" → Program/script: C:\path\to\vcfa-resave\run-vcfa.bat.
        • Click OK (enter your account password if prompted).

 

Step 2 — Increase the provider registration scheduler interval

By default, the background job that can trigger this bug runs every 30 minutes. This step slows it down to once every 24 hours, so a custom action you've resaved is far less likely to disappear again before the next scheduled resave.

Impact: This also slows down how quickly any change to a provider's registered resource types or actions takes effect. For example, if a feature toggle changes which resource types or actions a provider exposes, that change will now take up to 24 hours to appear in (or disappear from) the catalog, instead of up to 30 minutes.

1. Log in to a VCFA node as vmware-system-user.

2. Switch to root and export configuration:

   sudo -i
   export KUBECONFIG=/etc/kubernetes/admin.conf

3. Check whether JAVA_TOOL_OPTIONS is already set on the catalog-service-app deployment in the prelude namespace:

   kubectl get deployment catalog-service-app -n prelude -o jsonpath='{.spec.template.spec.containers[*].env[?(@.name=="JAVA_TOOL_OPTIONS")].value}'

4. If the above command returns content, include it alongside the two new flags so existing settings aren't overwritten:

   kubectl set env deployment/catalog-service-app -n prelude JAVA_TOOL_OPTIONS="<existing value from step 3> -Dprovider.register.fixedDelayMillis=86400000 -Dprovider.register.initialDelayMillis=86400000"

   If it returns nothing, run:

   kubectl set env deployment/catalog-service-app -n prelude JAVA_TOOL_OPTIONS="-Dprovider.register.fixedDelayMillis=86400000 -Dprovider.register.initialDelayMillis=86400000"

   ⚠ Note: applying this automatically restarts the pod(s).

5. Verify it took effect:

   kubectl get pods -n prelude -l app=catalog-service-app          # confirm pods recreated (new AGE)
   kubectl exec -n prelude <new-pod-name> -- env | grep JAVA_TOOL_OPTIONS

ⓘ Even with this workaround in place, a custom action can still disappear for a brief period — this cannot be eliminated entirely until the permanent fix (9.1.1.0200) is installed. This can happen in two circumstances: a restart of the catalog-service-app pod (for example, during planned maintenance), or the scheduler's own periodic tick, which — now that Step 2 has slowed it to once every 24 hours — still has one guaranteed opportunity per day to fire due to the underlying ordering defect. Pod restarts are irregular and environment-specific.

In either case, the action will be missing for no longer than your configured resave-script interval — the script's next scheduled run will find and restore it automatically, with no action required from you. If you need it back sooner, you can restore it immediately at any time by either running resave_custom_actions.py manually (without --dry-run), or by re-saving the action from Design > Resource Actions in the UI.