NSX Manager UI is not accessible with "HTTP Status 404" error after reboot step from KB 87099

Products

• VMware NSX

Issue/Introduction

The purpose of this article is to inform that the symptom can be seen where NSX manager UI is not accessible with reboot from log4j vulnerability KB  345893.
The issue here is not caused by log4j patch  and is instead due to corrupted ROOT.war file

• With reboot step from Information regarding CVE-2021-44228 & CVE-2021-45046 in NSX Data Center for vSphere, NSX manager UI not accessible. 
• SSH access to NSX manager works fine.
• NSX Manager UI is not accessible. "HTTP Status 404" error is seen 
• NSX plugin in vCenter shows "No NSX managers available. Verify current user has role assigned on NSX Manager."
• nsx-wrapper logs show below JM launching and connector failure : 
  
  STATUS | wrapper | 2021/12/18 14:15:48 | Launching a JVM...
INFO | jvm 1 | 2021/12/18 14:15:48 | WrapperManager: Initializing...
  
  
• You may see messages similar to the following in the manager logs:
  

  appliance_mgmt/appmgmt-wrapper.log:1072:INFO | jvm 1 | 2021/12/18 17:24:26 | 18-Dec-2021 17:24:26.283 SEVERE [localhost-startStop-1] org.apache.catalina.startup.HostConfig.deployWAR Error deploying web application archive [/usr/appmgmt-webserver/webapps/ROOT.war]

  appliance_mgmt/catalina.2021-12-18.log:3485:18-Dec-2021 12:57:38.548 SEVERE [localhost-startStop-2] org.apache.catalina.startup.HostConfig.deployWAR Error deploying web application archive [/usr/appmgmt-webserver/webapps/ROOT.war]

  appliance_mgmt/catalina.2021-12-18.log:3589:18-Dec-2021 14:15:52.003 SEVERE [localhost-startStop-1] org.apache.catalina.startup.HostConfig.deployWAR Error deploying web application archive [/usr/appmgmt-webserver/webapps/ROOT.war]

• As a part of log4j patching, backup of the ROOT.war files is taken in /home/secureall/secureall/log4j-backup/
• Verify the size of log4j-backup file with a working setup ROOT.war files 
• Also extracting the corrupted war files in backup folder will throw error. 

Note: The preceding log excerpts are only examples. Date, time, and environmental variables may vary depending on your environment

 

Cause

Root cause for the NSX manager UI is not log4j Patch, instead corrupted ROOT.war file

INFO | jvm 1 | 2021/12/18 14:15:52 | 18-Dec-2021 14:15:52.023 SEVERE [WrapperStartStopAppMain] org.apache.catalina.core.StandardService.startInternal Failed to start connector [Connector[AJP/1.3-8001]]
INFO | jvm 1 | 2021/12/18 14:15:52 | Caused by: java.lang.IllegalArgumentException: The AJP Connector is configured with secretRequired="true" but the secret attribute is either null or "". This combination is not valid.

Resolution

The script attached to Information regarding CVE-2021-44228 & CVE-2021-45046 in NSX Data Center for vSphere has been updated with a fix to detect any ROOT.war file corruption

Workaround:
Restore NSX Manager from backup with uncorrupted ROOT.war files and then apply the latest script.

Additional Information

Impact/Risks:
NSX manager UI not accessible. NSX not available from VC plugin as well for any changes.