Execution of ABX actions fail on Cloud Extensibility Proxy (CEXP) appliances after agent restart

Products

• VMware
• VMware Aria Suite

Issue/Introduction

VMware Cloud Extensibility Proxy 8.3.0 appliances, build number Version - 8.3.0.15139 Build 17792628, released on 27th of March 2021 introduced a problem that prevents Kubernetes from starting properly due to an incorrect network configuration provided to the Kubernetes client.

Symptoms:

• Users are not able to create remote ABX endpoints or execute ABX actions.
• Cloud Assembly > Infrastructure > Integrations > + Add Integration > Extensibility Actions on Prem functionality displays an error similar to:

Failed to validate credentials. Error: Error in action run provider: Failed to notify session [http://<IP>:8000/le-mans/v1/sessions/<UUID>] for direct command [<UUID>] with no session

• Existing and newly create ABX action executions display an error similar to:

[Provisioning Service] No suitable cloud accounts for providers: 'aws, azure, on-prem' project: '<UUID>'! Reason: Selected endpoint [your-extensibility-on-prem-endpoint] is not healthy. Reason [Failed to notify session [http://IPAddress:8000/le-mans/v1/sessions/<UUID>]

• CEXP appliance ABX agent logs located in /data1/abx-agent-logs/ display an error similar to:

[266][I][2021-04-01T11:12:20.975Z][1][K8SUtil][initK8Cluster][Initialized K8s cluster with address 'https://<IP>:6443' and auth type 'PublicKey'...]
[267][I][2021-04-01T11:12:20.979Z][1][K8SUtil][processClusterCertificate][Start processing cluster certificate]
[268][I][2021-04-01T11:12:21.062Z][1][CertificateUtil][lambda$getCertificateInfo$1][Importing SSL certificate for url: https://<IP>:6443]
[269][I][2021-04-01T11:12:22.693Z][26][NettyChannelPool][logGroupStatus][Maintenance on xn-cnx-tag-default:data.next.symphony-dev.com:443, pending: 0, available channels: 0]

 

Resolution

Upgrade to the next released version of the CEXP appliance.

Workaround:

1. Take a Virtual Machine snapshot without memory of the CEXP appliance.
2. Run the follow commands from an SSH / PuTTy session on the appliance:
   1. echo "IyEvYmluL2Jhc2gKL3Vzci9zYmluL2lwIGFkZHIgc2hvdyAkezF9IHwgZ3JlcCBpbmV0IHwgaGVhZCAtbiAxIHwgeGFyZ3MgfCBjdXQgLWQgJyAnIC1mIDIgfCBjdXQgLWQgJy8nIC1mIDEK" | base64 -d > /usr/bin/iface-ip; chmod a+x /usr/bin/iface-ip
   2. /opt/scripts/renew_k8s_certificates.sh
   3. docker restart cloudassembly-abx-agent