SDDC Manager UI is failing to launch

Products

• VMware Cloud Foundation

Issue/Introduction

 

• SDDC Manager UI is failing to launch due to middleware wrapper user vcfui failing to authenticate.
  
  
• Unable to access the SDDC Manger UI due to error Session middleware wrapper failed initially and all retry attempts.
  
  
• /var/log/vmware/vcf/sddc-manager-ui-app/sddcManagerServer.log displays:

YYYY-MM-DD ERROR [772257e2c735401d] [services/wrappers/pgSessionWrapper.js, http-get-ui, sessionMiddlewareWrap:96] Error from session middleware caught in middleware wrapper 1 {"error":{"name":"error","length":101,"severity":"FATAL","code":"28P01","file":"auth.c","line":"328","routine":"auth_failed","level":"error"}}
YYYY-MM-DD VERBOSE [772257e2c735401d] [services/wrappers/pgSessionWrapper.js, http-get-ui, sessionMiddlewareWrap:108] Start attemptPostgresInit()
YYYY-MM-DD INFO [772257e2c735401d] [../internal/process/next_tick.js, http-get-ui, _tickCallback:68] Starting sessionMiddleware function; current retry count = 2
YYYY-MM-DD WARN [772257e2c735401d] [../node_modules/express-session/index.js, http-get-ui, null:450] Session middleware produced an error while validating session. Retrying
YYYY-MM-DD ERROR [772257e2c735401d] [../internal/process/next_tick.js, http-get-ui, _tickCallback:68] undefined {"name":"error","length":101,"severity":"FATAL","code":"28P01","file":"auth.c","line":"328","routine":"auth_failed"}
YYYY-MM-DD ERROR [772257e2c735401d] [services/wrappers/pgSessionWrapper.js, http-get-ui, sessionMiddlewareWrap:96] Error from session middleware caught in middleware wrapper 2 {"error":{"name":"error","length":101,"severity":"FATAL","code":"28P01","file":"auth.c","line":"328","routine":"auth_failed","level":"error"}}
YYYY-MM-DD VERBOSE [772257e2c735401d] [services/errorHandling.js, http-get-ui, csrfErrorLogger:187] Start csrfErrorLogger()
YYYY-MM-DD ERROR [772257e2c735401d] [services/errorHandling.js, http-get-ui, productionErrorRoute:85] Sending error response: {"id":"8c15a025-d97###8-468a-####-##########","requestedPath":"/ui/","error":{"name":"error","length":101,"severity":"FATAL","code":"28P01","file":"auth.c","line":"328","routine":"auth_failed","level":"error"},"stack":"error: Session middleware wrapper failed initial and all retry attempts. Cannot continue. Throwing error from session middleware; password authentication failed for user \"vcfui\"\n at Connection.parseE (/opt/vmware/vcf/sddc-manager-ui-app/server/node_modules/pg/lib/connection.js:546:11)\n at Connection.parseMessage (/opt/vmware/vcf/sddc-manager-ui-app/server/node_modules/pg/lib/connection.js:371:19)\n at Socket.<anonymous> (/opt/vmware/vcf/sddc-manager-ui-app/server/node_modules/pg/lib/connection.js:114:22)\n at Socket.emit (events.js:189:13)\n at addChunk (_stream_readable.js:284:12)\n at readableAddChunk (_stream_readable.js:265:11)\n at Socket.Readable.push (_stream_readable.js:220:10)\n at TCP.onStreamRead [as onread] (internal/stream_base_commons.js:94:17)"}
YYYY-MM-DD VERBOSE [5f968fdfcc0b42d6] [services/util.js, http-get-ui, logRequest:262] Incoming request {"url":"/ui/","method":"GET","id":"bfe18553-#####-####-####-#######682"} 

Environment

VMware Cloud Foundation 4.4
VMware Cloud Foundation on VxRail 3.8.x
Vmware Cloud Foundation 4.5

Cause

The current issue is due to the service restart happening during a fresh install in VCF 3.8.0. The permanent fix for this database issue is in VCF 3.9.0.
This issue applies to VCF 4.4.x, and upgrades to 4.5.x.

Resolution

Workaround:

1. Log into SDDC Manager postgres database
   • ssh into SDDC Manager as vcf
   • su into root
   • Launch postgres:

psql --host=localhost -U postgres

2. Drop the affected DATABASE / USER:

DROP DATABASE IF EXISTS sddc_manager_ui; DROP ROLE IF EXISTS vcfui;

Should return output as:

DROP DATABASE
DROP ROLE

3. Quit out of the postgres DB

\q

4. Restart SDDC Manager UI services

systemctl restart sddc-manager-ui-db
systemctl restart sddc-manager-ui-app

Note : If this workaround is applied during VCF 4.5 Upgrade failure, it could probably cause CAP upgrade failure. Please refer  https://knowledge.broadcom.com/external/article/301600/sddc-manager-ui-is-failing-to-launch.html