SDDC Manager UI is failing to launch

Products

• VMware Cloud Foundation

Issue/Introduction

Symptoms:

• SDDC Manager UI is failing to launch due to middleware wrapper user vcfui failing to authenticate.
• Unable to access the SDDC Manger UI due to error Session middleware wrapper failed initially and all retry attempts.
• /var/log/vmware/vcf/sddc-manager-ui-app/sddcManagerServer.log displays:

2020-02-14T15:08:48.561Z ERROR [772257e2c735401d] [services/wrappers/pgSessionWrapper.js, http-get-ui, sessionMiddlewareWrap:96] Error from session middleware caught in middleware wrapper 1 {"error":{"name":"error","length":101,"severity":"FATAL","code":"28P01","file":"auth.c","line":"328","routine":"auth_failed","level":"error"}}
2020-02-14T15:08:48.562Z VERBOSE [772257e2c735401d] [services/wrappers/pgSessionWrapper.js, http-get-ui, sessionMiddlewareWrap:108] Start attemptPostgresInit()
2020-02-14T15:08:48.562Z INFO [772257e2c735401d] [../internal/process/next_tick.js, http-get-ui, _tickCallback:68] Starting sessionMiddleware function; current retry count = 2
2020-02-14T15:08:48.564Z WARN [772257e2c735401d] [../node_modules/express-session/index.js, http-get-ui, null:450] Session middleware produced an error while validating session. Retrying
2020-02-14T15:08:48.564Z ERROR [772257e2c735401d] [../internal/process/next_tick.js, http-get-ui, _tickCallback:68] undefined {"name":"error","length":101,"severity":"FATAL","code":"28P01","file":"auth.c","line":"328","routine":"auth_failed"}
2020-02-14T15:08:48.565Z ERROR [772257e2c735401d] [services/wrappers/pgSessionWrapper.js, http-get-ui, sessionMiddlewareWrap:96] Error from session middleware caught in middleware wrapper 2 {"error":{"name":"error","length":101,"severity":"FATAL","code":"28P01","file":"auth.c","line":"328","routine":"auth_failed","level":"error"}}
2020-02-14T15:08:48.566Z VERBOSE [772257e2c735401d] [services/errorHandling.js, http-get-ui, csrfErrorLogger:187] Start csrfErrorLogger()
2020-02-14T15:08:48.566Z ERROR [772257e2c735401d] [services/errorHandling.js, http-get-ui, productionErrorRoute:85] Sending error response: {"id":"8c15a025-d978-468a-9438-07de9eaf50c4","requestedPath":"/ui/","error":{"name":"error","length":101,"severity":"FATAL","code":"28P01","file":"auth.c","line":"328","routine":"auth_failed","level":"error"},"stack":"error: Session middleware wrapper failed initial and all retry attempts. Cannot continue. Throwing error from session middleware; password authentication failed for user \"vcfui\"\n at Connection.parseE (/opt/vmware/vcf/sddc-manager-ui-app/server/node_modules/pg/lib/connection.js:546:11)\n at Connection.parseMessage (/opt/vmware/vcf/sddc-manager-ui-app/server/node_modules/pg/lib/connection.js:371:19)\n at Socket.<anonymous> (/opt/vmware/vcf/sddc-manager-ui-app/server/node_modules/pg/lib/connection.js:114:22)\n at Socket.emit (events.js:189:13)\n at addChunk (_stream_readable.js:284:12)\n at readableAddChunk (_stream_readable.js:265:11)\n at Socket.Readable.push (_stream_readable.js:220:10)\n at TCP.onStreamRead [as onread] (internal/stream_base_commons.js:94:17)"}
2020-02-14T15:08:51.300Z VERBOSE [5f968fdfcc0b42d6] [services/util.js, http-get-ui, logRequest:262] Incoming request {"url":"/ui/","method":"GET","id":"bfe18553-dfad-4e79-a885-1f5a373cb682"}

 

Environment

VMware Cloud Foundation 4.4
VMware Cloud Foundation on VxRail 3.8.x
Vmware Cloud Foundation 4.5

Cause

The current issue is due to the service restart happening during a fresh install in VCF 3.8.0. The permanent fix for this database issue is in VCF 3.9.0.
This issue applies to VCF 4.4.x, and upgrades to 4.5.x.

Resolution

Workaround:

1. Log into SDDC Manager postgres database
   • ssh into SDDC Manager as vcf
   • su into root
   • Launch postgres:

psql --host=localhost -U postgres

2. Drop the affected DATABASE / USER:

DROP DATABASE IF EXISTS sddc_manager_ui; DROP ROLE IF EXISTS vcfui;

Should return output of:

DROP DATABASE
DROP ROLE

3. Quite out of the postgres DB

\q

4. Restart SDDC Manager UI services

systemctl restart sddc-manager-ui-db
systemctl restart sddc-manager-ui-app

Note : If this workaround is applied during VCF 4.5 Upgrade failure, it could probably cause CAP upgrade failure. Please refer   KB89977