NSX Cloud - Cloud Service Manager UI is not coming up on 4.0.1.1

Products

• VMware NSX

Issue/Introduction

Symptoms:
1. Error in CSM UI on version 4.0.1.1:
{"module_name":"common-service","error_code":"98","sub_error":"/csmapi/csm.html","error_message":"upstream connect error or disconnect/reset before headers. reset reason: connection failure"}


2. nsx-cloud-service-manager service is stopped and will not start:

root@nsxcsm1:~# /etc/init.d/nsx-cloud-service-manager status
nsx-cloud-service-manager.service - LSB: VMware NSX Cloud Service Manager API server
   Loaded: loaded (/etc/init.d/nsx-cloud-service-manager; enabled; vendor preset: enabled)
   Active: activating (auto-restart) (Result: exit-code) since Wed 2022-10-12 07:02:57 UTC; 24s ago
     Docs: man:systemd-sysv-generator(8)
  Process: 12147 ExecStopPost=/etc/init.d/nsx-cloud-service-manager poststop (code=exited, status=0/SUCCESS)
  Process: 12047 ExecStartPre=/etc/init.d/nsx-cloud-service-manager prestart (code=exited, status=1/FAILURE)

Oct 12 07:02:57 nsxcsm1 systemd[1]: Failed to start LSB: VMware NSX Cloud Service Manager API server.

root@nsxcsm1:~# /etc/init.d/nsx-cloud-service-manager start
[....] Starting nsx-cloud-service-manager (via systemctl): nsx-cloud-service-manager.serviceJob for nsx-cloud-service-manager.service failed because the control process exited with error code.
See "systemctl status nsx-cloud-service-manager.service" and "journalctl -xe" for details.
 failed!

root@nsxcsm1:~# journalctl -xe
-- Subject: Unit UNIT has finished shutting down
-- Defined-By: systemd
-- Support: http://www.ubuntu.com/support
-- 
-- Unit UNIT has finished shutting down.
Oct 12 07:04:06 nsxcsm1 systemd[12713]: Closed GnuPG cryptographic agent and passphrase cache (restricted).
-- Subject: Unit UNIT has finished shutting down
-- Defined-By: systemd
-- Support: http://www.ubuntu.com/support
-- 
-- Unit UNIT has finished shutting down.
Oct 12 07:04:06 nsxcsm1 systemd[12713]: Closed GnuPG cryptographic agent and passphrase cache (access for web browsers).
-- Subject: Unit UNIT has finished shutting down
-- Defined-By: systemd
-- Support: http://www.ubuntu.com/support
-- 
-- Unit UNIT has finished shutting down.
Oct 12 07:04:06 nsxcsm1 systemd[12713]: Closed GnuPG cryptographic agent (ssh-agent emulation).
-- Subject: Unit UNIT has finished shutting down
-- Defined-By: systemd
-- Support: http://www.ubuntu.com/support
-- 
-- Unit UNIT has finished shutting down.
Oct 12 07:04:06 nsxcsm1 systemd[12713]: Closed GnuPG cryptographic agent and passphrase cache.
-- Subject: Unit UNIT has finished shutting down
-- Defined-By: systemd
-- Support: http://www.ubuntu.com/support
-- 
-- Unit UNIT has finished shutting down.
Oct 12 07:04:06 nsxcsm1 systemd[12713]: Reached target Shutdown.
-- Subject: Unit UNIT has finished start-up
-- Defined-By: systemd
-- Support: http://www.ubuntu.com/support
-- 
-- Unit UNIT has finished starting up.
-- 
-- The start-up result is RESULT.
Oct 12 07:04:06 nsxcsm1 systemd[12713]: Starting Exit the Session...
-- Subject: Unit UNIT has begun start-up
-- Defined-By: systemd
-- Support: http://www.ubuntu.com/support
-- 
-- Unit UNIT has begun starting up.
Oct 12 07:04:06 nsxcsm1 systemd[12713]: Received SIGRTMIN+24 from PID 12747 (kill).
Oct 12 07:04:06 nsxcsm1 nsx-cloud-service-manager[12744]: CSM: performing operation: (poststop)
Oct 12 07:04:06 nsxcsm1 systemd[1]: Stopped User Manager for UID 113.
-- Subject: Unit [email protected] has finished shutting down
-- Defined-By: systemd
-- Support: http://www.ubuntu.com/support
-- 
-- Unit [email protected] has finished shutting down.
Oct 12 07:04:06 nsxcsm1 audispd[663]: node=nsxcsm1 type=SERVICE_STOP msg=audit(1665558246.829:10213): pid=1 uid=0 auid=4294967295 ses=4294967295 subj==unconfined msg='unit=user@113 comm="systemd" exe="/lib/systemd/systemd" hostname=? addr
Oct 12 07:04:06 nsxcsm1 systemd[1]: Removed slice User Slice of uproton.
-- Subject: Unit user-113.slice has finished shutting down
-- Defined-By: systemd
-- Support: http://www.ubuntu.com/support
-- 
-- Unit user-113.slice has finished shutting down.
Oct 12 07:04:06 nsxcsm1 nsx-cloud-service-manager[12744]: CSM: finished performing operation: (poststop)
Oct 12 07:04:06 nsxcsm1 systemd[1]: nsx-cloud-service-manager.service: Failed with result 'exit-code'.
Oct 12 07:04:06 nsxcsm1 systemd[1]: Failed to start LSB: VMware NSX Cloud Service Manager API server.
-- Subject: Unit nsx-cloud-service-manager.service has failed
-- Defined-By: systemd
-- Support: http://www.ubuntu.com/support
-- 
-- Unit nsx-cloud-service-manager.service has failed.
-- 
-- The result is RESULT.
Oct 12 07:04:06 nsxcsm1 audispd[663]: node=nsxcsm1 type=SERVICE_START msg=audit(1665558246.837:10214): pid=1 uid=0 auid=4294967295 ses=4294967295 subj==unconfined msg='unit=nsx-cloud-service-manager comm="systemd" exe="/lib/systemd/system
Oct 12 07:04:08 nsxcsm1 async-replicator-service[10394]: Waiting for CBM to lift the barrier file...
Oct 12 07:04:08 nsxcsm1 site-manager-service[10575]: Waiting for CBM to lift the barrier file...


3. csm logs are not available as above service is not starting

Resolution

This issue will be resolved in NSX 4.1.0

Workaround:
Run below commands as root user on CSM appliance once it is deployed:
 
Step 1:
 /opt/vmware/bin/nsx_proton_cleanup -s 

Step 2:
Create a backup of "/etc/init.d/nsx-cloud-service-manager" in a persistent path.
Example:
root@nsxcsm1:~# cp /etc/init.d/nsx-cloud-service-manager /root/nsx-cloud-service-manager

Step 3:
 sed -e '393s/$/ -s \/bin\/bash/' -e '419s/$/ -s \/bin\/bash/' -e '433s/$/ -s \/bin\/bash/' /etc/init.d/nsx-cloud-service-manager > /etc/init.d/nsx-cloud-service-manager.tmp && mv /etc/init.d/nsx-cloud-service-manager.tmp /etc/init.d/nsx-cloud-service-manager && chmod 755 /etc/init.d/nsx-cloud-service-manager 

Step 4:
Reboot