Error: "HTTP failure response for <API Call>: 0 Unknown Error" when interacting with Virtual Machines / vApps or NAT rules in Edge gateways.

Products

• VMware Cloud Director
• VMware Cloud Director Availability - Disaster Recovery 4.x

Issue/Introduction

• When listing Virtual Machines (VM) or vApps, the UI fails to load and prompts error in Cloud Director (VCD) similar to:

HTTP failure response for <API Call>: 0 Unknown Error

• When filtering Virtual Machines or vApps, the UI fails to load and prompts error similar to:

HTTP failure response for <API Call>: 0 Unknown Error

• When attempting to add or view NAT rules under Edge gateways, and prompts error similar to:

HTTP failure response for <API Call>: 0 Unknown Error

• When attempting to establish a Remote Console to a particular VM from Cloud Director (VCD), and prompts error similar to:

HTTP failure response for <API Call>: 0 Unknown Error

• A large number of API calls (data-engine) are sent to Cloud Director Availability (VCDA) prior to the UI failing to load objects.
  
  
• VMware Cloud Director UI randomly generates errors as below : 
  
  "0 Unknown error" from the following request "request": {
    "method": "GET",
    "url": "https://VCD-URL/api/query?type=vm&format=records&type=vm&page=1&pageSize=15&filterEncoded=true&filter=(((isExpired==false));((isVAppTemplate==false;vdc==xxxxxxxx-xxxx-xxxx-xxxxxxxxxxxx)))&sortDesc=dateCreated&fields=metadata%40SYSTEM%3Avmware.cloud.director.availability.protected%2Cmetadata%40GENERAL%3Avmware.cloud.director.ui.badges.blue%2Cmetadata%40GENERAL%3Avmware.cloud.director.ui.badges.red%2Cmetadata%40GENERAL%3Avmware.cloud.director.ui.badges.orange%2Cmetadata%40GENERAL%3Avmware.cloud.director.ui.badges.green%2Cmetadata%40GENERAL%3Avmware.cloud.director.ui.badges.magenta&links=true"
}

Environment

VMware Cloud Director 10.x

Cause

This issue is a browser specific issue which can occur when the Client Browser has issues sending and receiving API calls to/from Cloud Director.

Factors known to contribute to this would be

1. Client Memory Resources
2. Network Connectivity Issues

Resolution

There is a known issue in Cloud Director 10.4.x which leverages the 0.0.3 version of the NgHyperPlugin.

• This plugin is responsible for the functionality of all other subsequently installed plugins.
• An issue with subscriptions to the Cloud Director Availability (VCDA) plugins causes an incremental increase in overall API calls being sent which can result in Client Memory Resource issues being observed.

To verify the version of the NgHyperPlugin being used, navigate to the Provider UI > More  > Customize Portal.

Note: If the NgHyperPlugin version is 0.0.3 or lower then follow below steps.

1. Download the attached file 'hyper_plugin.zip'.
2. Upload the Plugin to Cloud Director for reference : Upload a Plug-in to Your VMware Cloud Director
3. Scope needs to be Service Providers and Tenants
4. Disable the 0.0.3 Version of the NgHyperPlugin, for reference: Activate or Deactivate a Plug-in Using Your VMware Cloud Director
5. Reproduce the issue.
6. If there are issues with the new HyperPlugin, follow the rollback section below.

Rollback

1. Enable the 0.0.3 Version of the NgHyperPlugin.
2. Disable the 0.0.4 Version of the NgHyperPlugin.
3. Delete the 0.0.4 Version of the NgHyperPlugin, for reference: Delete a Plug-in From Your VMware Cloud Director

Additional Information

If using the 0.0.4 version of the NgHyperPlugin and still observing issues, it is most likely encountered due to environmental factors which can lead to the observation of the 'Unknown Error'.

• Opening the Dev Tools of that browser and navigating to the Network tab will help with isolation.
  • Locate the Status column and note the error observed at the time of issue.

Client machine has sufficient memory resources.

• This may be more prevalent on client machines with low memory resources and are also running a number of applications and/or a large number of browser tabs.
• The Status column would denote a ERR_INSUFFICIENT_RESOURCES error if the browser is running out of memory.

Network Connectivity Issues

• Any connectivity issue which would result in the Client machine losing connectivity to the Cloud Director Cells/Load Balancer would result in the error.
• Instances which have been observed to cause the issue would be
  1. VPN Disconnection or Re-authentication
  2. Machine NIC disconnection
  3. Issues with local routing to internet