Aria Automation 8.18.1 Deployments fail with "Infoblox IPAM is misconfigured: infoblox_username and infoblox

Products

VMware Aria Suite

Issue/Introduction

The following symptoms might be observed on Aria Automation 8.18.1 HA setups:
Deployments or data collection occasionally fail with "Error allocating in network or range: Infoblox IPAM is misconfigured: infoblox_username and infoblox_password are incorrect." or similar authorization issues, even though the username and password are valid.

Environment

Aria Automation 8.18.1 HA

Cause

An internal platform component upgrade in Aria Automation 8.18.1 caused a regression in the IPAM SDK authentication flow used by the ABX engine.
In clustered environments, the IP address from which subsequent requests are made can differ from the one expected by the Infoblox server.
This leads to the requests being denied.

Resolution

Recommended Workaround

VMware Aria Automation 8.18.1 Cumulative Update - Patch 1 addresses this issue, please refer to to the following KB article: https://knowledge.broadcom.com/external/article/385294

Alternative Workaround
Scaling the proxy-service deployment to 1 replica resolves the issue. This is a temporary measure until an official patch is provided.

The following steps should be executed:
1. Take simultaneous VM snapshots (without memory, with quiesced option selected) of all 3 VAs. It can be taken from vRSLCM or manually from vCenter.
2. Execute the following command on only one of the nodes in order to apply and persist the patch :

vracli cluster exec -- bash -c 'base64 -d <<< "IyEvYmluL2Jhc2gKCmxvZ0Rpcj0iL3Zhci9sb2cvdm13YXJlL3ByZWx1ZGUiCnRpbWVzdGFtcD0kKGRhdGUgJyslWS0lbS0lZC0lSC0lTS0lUycpCnByPSJWQ0ZDT04tMjYzNiIKQkFDS1VQX0RJUj0iL2RhdGEvcGF0Y2gtYmFja3VwLyRwci0kdGltZXN0YW1wIgoKbWtkaXIgLXAgIiRsb2dEaXIiCmV4ZWMgMj4gPih0ZWUgLWEgIiRsb2dEaXIvcHJveHktcGF0Y2gtcHIkcHItJHRpbWVzdGFtcC5sb2ciKQpleGVjID4mMgoKaD0kKGhvc3RuYW1lKQpsb2coKSB7CiAgICBlY2hvICJbJDFdWyQoZGF0ZSAiKyVZLSVtLSVkLSVILSVNLSVTIildWyRoXSAkMiIgPiYyCn0KCmlmIFsgLWYgL2V0Yy92bXdhcmUtcHJlbHVkZS9wcm9maWxlcy9wcm94eS1zZXJ2aWNlL2NoZWNrIF07IHRoZW4KICBsb2cgSU5GTyAiUGF0Y2ggZm9yIFBSICRwciBhbHJlYWR5IGFwcGxpZWQuIE5vdGhpbmcgdG8gZG8uIgogIGV4aXQgMApmaQoKbG9nIElORk8gIkFwcGx5aW5nIHBhdGNoIGZvciBQUiAkcHIiCgp2ZXI9JCh2cmFjbGkgdmVyc2lvbiB8IGhlYWQgLW4gMSB8IGF3ayAne3ByaW50ICQzfScgfCBhd2sgLUYgLiAne3ByaW50ICQxJDIkM30nKQppZiBbICIkdmVyIiAhPSAiODE4MSIgXTsgdGhlbgogIGxvZyBJTkZPICJUaGlzIEtCIGlzIG9ubHkgYXBwbGljYWJsZSBmb3IgQXJpYSBBdXRvbWF0aW9uIHZlcnNpb24gOC4xOC4xIgogIGV4aXQgMApmaQoKbWtkaXIgLXAgL2V0Yy92bXdhcmUtcHJlbHVkZS9wcm9maWxlcy9wcm94eS1zZXJ2aWNlL2hlbG0vcHJlbHVkZV9wcm94eS1zZXJ2aWNlCmNhdCA+IC9ldGMvdm13YXJlLXByZWx1ZGUvcHJvZmlsZXMvcHJveHktc2VydmljZS9jaGVjayA8PEVPRgojIS9iaW4vc2gKdHJ1ZQpFT0YKY2htb2QgYSt4IC9ldGMvdm13YXJlLXByZWx1ZGUvcHJvZmlsZXMvcHJveHktc2VydmljZS9jaGVjawpjYXQgPiAvZXRjL3Ztd2FyZS1wcmVsdWRlL3Byb2ZpbGVzL3Byb3h5LXNlcnZpY2UvaGVsbS9wcmVsdWRlX3Byb3h5LXNlcnZpY2UvNTAtcmVzb3VyY2VzLnlhbWwgPDxFT0YKY2x1c3RlcjoKICByZXBsaWNhQ291bnRPdmVycmlkZTogMQpFT0YKCm1rZGlyIC1wICRCQUNLVVBfRElSCmNwIC9vcHQvY2hhcnRzL3Byb3h5LXNlcnZpY2UvdGVtcGxhdGVzL2RlcGxveW1lbnQueWFtbCAkQkFDS1VQX0RJUi9kZXBsb3ltZW50LnlhbWwKCmNkIC8KcGF0Y2ggLS1uby1iYWNrdXAtaWYtbWlzbWF0Y2ggLXMgLWYgLS1yZWplY3QtZmlsZT0tIC1wMSA8PCAnRU9GJwotLS0gL29wdC9jaGFydHMvcHJveHktc2VydmljZS90ZW1wbGF0ZXMvZGVwbG95bWVudC55YW1sCisrKyAvb3B0L2NoYXJ0cy9wcm94eS1zZXJ2aWNlL3RlbXBsYXRlcy9kZXBsb3ltZW50LnlhbWwKQEAgLTUsNyArNSwxMSBAQAogICBsYWJlbHM6CiAgICAgc2VydmljZS1tb25pdG9yOiB7eyAuUmVsZWFzZS5OYW1lIHwgcXVvdGUgfX0KIHNwZWM6CisgIHt7LSBpZiAuVmFsdWVzLmNsdXN0ZXIucmVwbGljYUNvdW50T3ZlcnJpZGUgfX0KKyAgcmVwbGljYXM6IHt7IC5WYWx1ZXMuY2x1c3Rlci5yZXBsaWNhQ291bnRPdmVycmlkZSB9fQorICB7ey0gZWxzZSB9fQogICByZXBsaWNhczoge3sgLlZhbHVlcy5jbHVzdGVyLnJlcGxpY2FDb3VudCB9fQorICB7ey0gZW5kIH19CiAgIHNlbGVjdG9yOgogICAgIG1hdGNoTGFiZWxzOgogICAgICAgYXBwOiBwcm94eS1zZXJ2aWNlCkVPRgoKcmVzdWx0PSQ/CmlmIFtbICRyZXN1bHQgPT0gMSBdXTsgdGhlbgogIGxvZyBJTkZPICJQYXRjaCBhbHJlYWR5IGFwcGxpZWQuIgpmaQoKbG9nIElORk8gInByb3h5LXNlcnZpY2UgcHJvZmlsZSBjcmVhdGVkLiBTY2FsaW5nIGRvd24gZGVwbG95bWVudCByZXBsaWNhcyB0byAxLi4uIgprdWJlY3RsIC1uIHByZWx1ZGUgc2NhbGUgZGVwbG95bWVudCBwcm94eS1zZXJ2aWNlIC0tcmVwbGljYXM9MQo=" | bash -'

Additional Information

Notes:

The impact of scaling down the proxy-service pod to 1 should be minimal. In case of a node failure event on the node where pod was running, there might be a temporary downtime in provisioning operations until the proxy-service pod is started on one of the other nodes. This usually takes less than 1 minute.