Async patch tool fails with error "Running the following command(s) as su user has failed"

Products

• VMware SDDC Manager

Issue/Introduction

Attempts to install patches using Async patch tool command ./vcf-async-patch-tool -i /path/to/inputspec --du broadcom_support_email --sddcSSOUser SSOuser --sddcSSHUser vcf --it ONLINE --proxyServer fails with error  "Running the following command(s) as su user has failed"

/var/log/vmware/vcf/lcm/tools/asyncpatchtool/async_patch_tool.log

YYYY-MM-DDTHH:MM:SS.032+0000 [main] DEBUG [com.vmware.evo.sddc.lcm.tools.common.BundleUtil]
                      Fetching vcf sddc manager bundle elements from Manifest file: /nfs/vmware/vcf/nfs-mount/apToolBundles/sddc-bundle-files/bundle-######/sddc-manager-upgrade/Manifest.json
YYYY-MM-DDTHH:MM:SS.040+0000 [main] DEBUG [com.vmware.evo.sddc.lcm.tools.common.SddcManagerServiceUtil]
                      Completed updating SDDC service config fields
YYYY-MM-DDTHH:MM:SS.040+0000 [main] DEBUG [com.vmware.evo.sddc.lcm.tools.common.BundleUtil]
                      Modifying file permissions to vcf:vcf
YYYY-MM-DDTHH:MM:SS.051+0000 [main] DEBUG [com.vmware.evo.sddc.lcm.tools.bundletransfer.common.FilePermissionUtil]
                      Modified file permissions to 755 on the file: /nfs/vmware/vcf/nfs-mount/apToolBundles/sddc-bundle-files
YYYY-MM-DDTHH:MM:SS.302+0000 [main] ERROR [com.vmware.evo.sddc.lcm.tools.asyncpatch.AsyncPatchToolMain]
                      Error while initiating required functionality: ENABLE_ASYNC_PATCH
YYYY-MM-DDTHH:MM:SS.314+0000 [main] ERROR [com.vmware.evo.sddc.lcm.tools.asyncpatch.AsyncPatchToolMain]
                      SERVICE_UPGRADE_SCRIPT_FAILED
-- Error Message: VCF service upgrade or config property update script failed with error Running the following command(s) as su user has failed 

/var/log/vmware/vcf/lcm/lcm.log

YYYY-MM-DDTHH:MM:SS.028+0000 INFO  [vcf_lcm,] [c.v.v.c.c.i.v.r.client.VvsApiClient,Scheduled-2] vvs uri with query params: https://vvs.esp-staging.vmware-aws.com/v1/products/bundles/type/
vcf-lcm-bundle?format=json
YYYY-MM-DDTHH:MM:SS.612+0000 ERROR [vcf_lcm,] [c.v.v.c.c.i.v.r.client.VvsApiClient,Scheduled-2] response from vvs wasn't successful with statusCode: 403
YYYY-MM-DDTHH:MM:SS.612+0000 ERROR [vcf_lcm,] [c.v.v.c.c.i.v.r.client.VvsApiClient,Scheduled-2] Invalid null response from VMware Verification Service v1/products/bundles/type/vcf-lcm-bun
dle api.
YYYY-MM-DDTHH:MM:SS.613+0000 ERROR [vcf_lcm,] [c.v.e.s.l.c.s.VmwareCompatibilityDataDownloadScheduler,Scheduled-2] VVS Compatibility Data update failed
com.vmware.vcf.compatibility.controllers.internal.vvs.exception.VvsException: The VVS api call to v1/products/bundles/type/vcf-lcm-bundle failed.



Test connectivity to "vvs.esp" with the command in an SSH session to the SDDC Manager:

curl --location 'https://vvs.esp.vmware.com/v1/products/bundles/type/vcf-lcm-bundle?format=json'   --header 'X-Vmw-Esp-ClientId: vcf-lcm' > VmwareCompatibilityData.json -vvv  --proxy proxyFQDN:port

Failed connection example output



* Issue another request to this URL: 'https://storage.googleapis.com/vmwesp-prod-66-poqoo-vvs-esp-cf-bucket

 

* Hostname proxyFQDN was found in DNS cache
*   Trying ###.###.##.##:####...
* Connected to proxy (###.###.##.##) port #### (#1

< HTTP/1.1 403 Forbidden
< Cache-Control: no-cache
< X-XSS-Protection: 1
< Connection: Keep-Alive
< Content-Type: text/html; charset=utf-8
< Content-Length: 1932
< Pragma: no-cache
<
* CONNECT tunnel failed, response 403
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0
* Closing connection 1
curl: (56) CONNECT tunnel failed, response 403





Environment

VCF 5.1
VCF 5.2

Cause

SDDC Manager is unable to connect to the public URLs in order to download the VMwareCompatibilityData.json file due to response from vvs wasn't successful with statusCode: 403



Resolution

Ensure SDDC public URLs are added to the proxy whitelist to allow download of VMwareCompatibilityData.json file

Apart from above resolution, for VCF 5.1, review Update compatibility endpoint for VCF 5.1 deployments