Error: "Cloud authentication has failed" when upgrading Usage Meter to 9.x

Products

• VMware Usage Meter

Issue/Introduction

Symptom 1:

• An upgrade of Usage Meter from version 4.8 to 9.0 fails during the authentication phase, presenting the following error message:

cloud authentication has failed. Please confirm that the correct access token is being used and that there are not network connectivity issues impeding the authentication process.
Connection reset by peer in connection to eapi.broadcom.com:443
curl: (35) Recv failure: Connection reset by peer

• This occurs even when the provided access token is confirmed to be valid.

 

Symptom 2:

• An upgrade of Usage Meter from 4.8 to 9.x version fails with the following error message:

      Cloud authentication has failed. Please confirm that the correct access token is being used and that there are no network connectivity issues impeding the authentication process.
      Note: As the upgrade procedure has not been initiated, reverting to a snapshot is not necessary at this time.

Environment

• VCF Usage Meter 9.0.x

Cause

Symptom 1:

• The primary cause of this authentication failure is the inability of the Usage Meter appliance to reach necessary Broadcom cloud services. Specifically, the required URL eapi.broadcom.com is not whitelisted in the network firewall rules governing the Usage Meter appliance's outbound connectivity.

 

Symptom 2:

• The cause of the "Cloud authentication has failed" error is a Man-in-the-Middle (MITM) SSL/TLS interception being performed by a Fortinet firewall or security appliance in the network path.

• This interception prevents the Usage Meter appliance from establishing a trusted, direct SSL connection with Broadcom's eapi.broadcom.com endpoint.

Resolution

Symptom 1:

To successfully complete the upgrade, the Usage Meter appliance must be able to establish communication with Broadcom's cloud authentication endpoint.

Please follow the steps:

1. Whitelist eapi.broadcom.com: Add the URL eapi.broadcom.com to your network firewall's whitelist or create a rule allowing outbound HTTPS (port 443) traffic from the Usage Meter appliance to this domain.

2. If the environment uses a network device, firewall, load balancer or proxy configured with SSL certificates, import the certificate of respective component to Usage Meter appliance keystore by following the Import a Certificate to the VCF Usage Meter Appliance Keystore section.

3. Re-attempt upgrade once the firewall rule is in place. The cloud authentication step should now succeed, allowing the upgrade to proceed to completion.

 

Symptom 2:

1. Use the following log file and commands to verify the issuer of the eapi.broadcom.com certificate:

   • Log file : setup_cloud_auth.log in /opt/vmware/cloudusagemetering

   • openssl s_client -connect eapi.broadcom.com:443 -servername eapi.broadcom.com

   • openssl s_client -connect eapi.broadcom.com:443

   • curl -vk --http1.1 --noproxy "*" --location https://eapi.broadcom.com:443

   • time echo | openssl s_client -connect eapi.broadcom.com:443 -servername eapi.broadcom.com -alpn "h2,http/1.1" -tls1_3

2. If the certificate presented by eapi.broadcom.com is issued by Firewall, for e.g. issuer: C=Country; ST=State; L=City; O=<Firewall-Name>; OU=Certificate Authority; CN=<Firewall-ID>; emailAddress=<Firewall-Email>, it would indicate that the firewall is intercepting and re-signing the SSL certificate. The Usage Meter appliance does not trust this intermediary Fortinet Certificate Authority, breaking the secure cryptographic chain required for cloud authentication. This prevents the UM from securely exchanging credentials or obtaining a valid access token, leading to the authentication failure.

3. Configure the firewall to bypass SSL/TLS inspection for all outbound traffic originating from the Usage Meter appliance destined for Broadcom cloud endpoints, specifically eapi.broadcom.com on port 443.