SDDC manager UI is loading with 502 Bad default gateway error
search cancel

SDDC manager UI is loading with 502 Bad default gateway error

book

Article ID: 316024

calendar_today

Updated On:

Products

VMware Cloud Foundation

Issue/Introduction

This article explains the steps required to fix permission issues in SDDC Manager log directories


Symptoms:

  • SDDC manager UI will not load - 502 Bad default gateway
  • commonsvc, domainmanager, operationsmanager, sddc-mamage-ui-app services are in activating state
  • Error in /var/log/vmware/vcf/sddc-manager-ui/sddcManagerServer.log

ERROR [37b8fc04114f44c5] [services/util.js, http-get-upgrade-status, attemptAsyncFunctionWithRetry:482] StatusCodeError: Maximum number of unsuccessful attempts(3) reached for function: requestWrapper - 404 - "<html>\r\n<head><title>404 Not Found</title></head>\r\n<body>\r\n<center><h1>404 Not Found</h1></center>\r\n<hr><center>nginx</center>\r\n</body>\r\n</html>\r\n" {"args":"http://127.0.0.1/solutions/features/list"],"upn":"[email protected]","id":"379a34d7-####-####-####-##########2f"}

2023-08-29T18:29:12.788+0000 WARN [37b8fc04114f44c5] [services/featureToggle.js, http-get-upgrade-status, logBackendServiceError:35]
700.164: VError: Encountered exception getting feature info for service ""
    at Object.encounterExceptionGettingFeatureError (/opt/vmware/vcf/sddc-manager-ui-app/server/src/errors/VCFError.js:100:5)
    at logBackendServiceError (/opt/vmware/vcf/sddc-manager-ui-app/server/src/services/featureToggle.js:29:35)
    at /opt/vmware/vcf/sddc-manager-ui-app/server/src/services/featureToggle.js:65:13
    at arrayMap (/opt/vmware/vcf/sddc-manager-ui-app/server/node_modules/lodash/lodash.js:574:23)
    at Function.map (/opt/vmware/vcf/sddc-manager-ui-app/server/node_modules/lodash/lodash.js:8851:14)
    at getFeatureInfo (/opt/vmware/vcf/sddc-manager-ui-app/server/src/services/featureToggle.js:64:27)
    at runMicrotasks (<anonymous>)
    at processTicksAndRejections (node:internal/process/task_queues:96:5)
Error Info: {"service":"","status":403,"errorModule":700,"errorCode":164}
All Errors Info:
 {"service":"","status":403,"upn":"[email protected]","id":"379a34d7-####-####-####-##########2f"}
2023-08-29T18:29:12.789+0000 WARN [37b8fc04114f44c5] [services/featureToggle.js, http-get-upgrade-status, getFeatureInfo:76]
700.165: VError: Not all backend services responded successfully with feature info. "$s" services responded with errors
    at Object.notAllBackEndServicesRespondedCorrectlyError (/opt/vmware/vcf/sddc-manager-ui-app/server/src/errors/VCFError.js:100:5)
    at getFeatureInfo (/opt/vmware/vcf/sddc-manager-ui-app/server/src/services/featureToggle.js:70:43)
    at runMicrotasks (<anonymous>)
    at processTicksAndRejections (node:internal/process/task_queues:96:5)
Error Info: {"len":1,"status":403,"errorModule":700,"errorCode":165}
All Errors Info:
 {"len":1,"status":403,"upn":"[email protected]","id":"379a34d7-####-####-####-##########2f"

 

Environment

VMware Cloud Foundation 4.x

Cause

Permissions of directories /var/log/vmware/vcf is incorrectly set

Resolution

  • Take snapshot of SDDC Manager VM
  • SSH to SDDC Manager with vcf user and su to root
  • Change the permissions for all the directories under /var/log/vmware/vcf to match as below
    • chown -R vcf_commonsvcs:vcf commonsvcs
    • chown -R vcf_operationsmanager:vcf operationsmanager
    • chown -R vcf_domainmanager:vcf domainmanager
    • chown -R vcf_lcm:vcf lcm
    • chown -R vcf_sddc_manager_ui_app:vcf sddc-manager-ui-app
  • Restart sddc manager services
    • /opt/vmware/vcf/operationsmanager/scripts/cli/sddcmanager_restart_services.sh