SDDC Manager UI Inaccessible with 502 Bad Gateway

Products

• VMware SDDC Manager / VCF Installer

Issue/Introduction

• When attempting to access the VMware Cloud Foundation (VCF) SDDC Manager UI, the web browser displays the following error: "502 Bad Gateway"
  
  
  
  
  
• Connecting to the SDDC Manager appliance via SSH and checking the disk space utilizing the df -h command reveals that the root partition (/)  is at 100% capacity.
  
  
  
  

Environment

VCF 9.x

Cause

• This issue occurs because the root (/) filesystem has reached maximum capacity.
  
  
• When the root partition is full, critical SDDC Manager microservices (such as vcf-commonsvcs and operationsmanager), internal databases, and underlying logging frameworks lose the ability to write temporary files, PID files, or runtime data. This storage exhaustion blocks these essential orchestration services from completing their startup sequences, leading to service crashes and locking the UI into a perpetual initialization loop (resulting in the 502 Bad Gateway error).
  
  
• This exhaustion most frequently occurs when the SDDC Manager appliance is configured to act as the SFTP target for NSX backups via a local user account. 

Resolution

To resolve this issue, follow the steps below:

1: Prerequisite

Take an offline snapshot of the SDDC Manager virtual machine via the vSphere Client before proceeding.

2: SSH and Elevate Privileges

1. Connect to the SDDC Manager appliance via SSH using the vcf user account.

2. Elevate to the root user:

   su

3: Identify Large Files Consuming Space

1. Download the Max_File_size.sh file from the Attachments section SDDC Manager UI becomes inaccessible due to root (/dev/sda4) partition filling up

2. Upload the Max_File_size.sh script to the SDDC Manager using Winscp or other file transfer protocol

3. Make the script executable
   
   chmod +x Max_File_size.sh

4. Run the script to pinpoint the largest directories:

   ./Max_File_size

4: Reclaim Disk Space

1. Navigate to the local SFTP backup directory (or the directory identified in Step 3):

   cd /var/srv/sftpuser

2. Identify and delete old, historical NSX backup files or directories that are no longer required.
   
   rm -rf <old_backup_filename_or_directory>

3. Verify that the disk space has been successfully reclaimed and the root partition is no longer at 100%:

   df -h

5: Restart SDDC Manager Services

1. Once sufficient disk space is available, run the global service restart script to bring the UI and backend microservices back online:
   
   /opt/vmware/vcf/operationsmanager/scripts/cli/sddcmanager_restart_services.sh
   
   
2. Wait a few minutes for all microservices to initialize, then attempt to access the SDDC Manager UI again.