When trying to create Auto Deploy rule, task fails with: "Error occurred while performing the task. Something went wrong while converting items to pxe profile:"

Products

• VMware vCenter Server

Issue/Introduction

To workaround the issue: "Could not find file: HTTP Error 403: Forbidden" so that we can create Auto Deploy rules.

Symptoms:
When attempting to create new auto deploy rule, the task fails with: "An error occurred while performing the task. Something went wrong while converting items to pxe profile: Could not find file: HTTP Error 403: Forbidden".

vCenter is using a proxy.

/var/log/vmware/rbd/rbd-server.log has errors that are similar to below snippet:

2022-02-28T15:37:37.727 [59857]INFO:dbsetup:installed DB connection <sqlite3.Connection object at 0x7f1678652730> for thread <Thread(Crea
teRuleWithTransform-task-7617368, started daemon 139734361806592)>
2022-02-28T15:37:37.727 [59857]INFO:cache_inject:inject stream -- vib, VMW_bootbank_pvscsi_0.1-4vmw.703.0.20.19193900, <socket.socket fd=
31, family=AddressFamily.AF_UNIX, type=SocketKind.SOCK_STREAM, proto=0, laddr=b'\x00/var/run/rbd/vib-6bd470e44172c9bfdae6c7dd7c55d4d0'>
2022-02-28T15:37:37.727 [59857]INFO:vibutil:starting download of vib -- VMW_bootbank_pvscsi_0.1-4vmw.703.0.20.19193900 -> /tmp/VMW_bootba
nk_pvscsi_0.1-4vmw.703.0.20.191939006p4g54hi
2022-02-28T15:37:37.758 [59857]ERROR:pxe_profile_util:Something went wrong while converting items to pxe profile:Could not find file: HTT
P Error 403: Forbidden
Traceback (most recent call last):
  File "bora/install/vmvisor/autodeploy/site-packages/vmware/rbd/vmodl_impl/pxe_profile_util.py", line 245, in downloadVibs
  File "bora/install/vmvisor/autodeploy/site-packages/vmware/rbd/utils/vibutil.py", line 131, in downloadVibInternal
  File "/usr/lib/python3.7/urllib/request.py", line 222, in urlopen
    return opener.open(url, data, timeout)
  File "/usr/lib/python3.7/urllib/request.py", line 531, in open
    response = meth(req, response)
  File "/usr/lib/python3.7/urllib/request.py", line 641, in http_response
    'http', request, response, code, msg, hdrs)
  File "/usr/lib/python3.7/urllib/request.py", line 569, in error
    return self._call_chain(*args)
  File "/usr/lib/python3.7/urllib/request.py", line 503, in _call_chain
    result = func(*args)
  File "/usr/lib/python3.7/urllib/request.py", line 649, in http_error_default
    raise HTTPError(req.full_url, code, msg, hdrs, fp)
urllib.error.HTTPError: HTTP Error 403: Forbidden

Environment

VMware vCenter Server 7.0.x
VMware vCenter Server 6.7.x

Cause

The autodeploy service queries imagebuilder for file URLs by using the API and then downloads them. Both the API calls and the URL GETs happen through the proxy on :443.

There is a known issue with no_proxy variable where no_proxy variable values with leading dot not properly handled.

Example: 

no_proxy = '127.0.0.1, localhost, .vmware.com'

Resolution

• Take a normal snapshot on the vCenter and offline snapshots if the vCenter is configured using ELM.
• Include the vCenter FQDN in no_proxy variable in the /etc/sysconfig/proxy file:
  • Example: no_proxy = '127.0.0.1, localhost, vcenter.vmware.com'
• Restart vCenter services. [service-control --stop --all && service-control --start --all]
• Retry creating the AutoDeploy rule.

Additional Information

Impact/Risks:

• Restart of vCenter services required.