503 Error when accessing Horizon UI

Products

• VMware Integrated OpenStack

Issue/Introduction

• Intermittent issues when accessing Horizon may see the following:
  503 service temporarily unavailable | nginx
• Log observations: Both ingress and Horizon logs show a significant number of `/project/images` requests, with many requests returning a 503 status code in ingress logs.
• Error scope: Only Horizon being unavailable can cause 503 error.
  If OpenStack services like Glance or Nova Compute were the problem, the error code would be 500 shown in Horizon UI.
  If ingress were malfunctioning, the UI would be completely inaccessible instead of returning a 503.
• Logs from other critical services, such as Glance and Nova Compute, show no errors related to this issue.
• Sufficient resources: The nodes have ample resources, with no signs of excessive consumption.

Environment

7.x

Cause

Cause of this issue is that the UI client is sending too many '/project/images requests', which overwhelms Horizon and prevents it from responding.

Resolution

Scale Horizon replicas to 3, ensure there is a Horizon pod running on each controller node.  

1. Log in to the Integrated OpenStack Manager web interface as the admin user.
2. In OpenStack Deployment, click the name of your deployment and open the Manage tab.
3. On the Services tab, find the desired OpenStack component (Horizon) and select Actions > Scale out.
   For each service in the component, enter the desired number of pods (3)
   
   Note: The maximum number of replicas for each service cannot be greater than the number 
   of worker nodes (Default is 3)
4. Clean up failed image uploads: Remove any failed images and ensure all images are in the `saved` state.