NSX AUDIT PRECHECK FAILS WITH 'UNKNOWN EXCEPTION' ERROR

Products

VMware Cloud Foundation

Issue/Introduction

Symptoms:

When running a domain precheck prior to upgrade, the NSXT Audit precheck fails with "NSX-T Manager Audit for xxxxxx-xxxxxx failed with unknown exception"
The lcm-debug.log logs similar to below:

name=nsxt.corp.local] [c.v.v.i.p.c.r.h.AllowAllHostnameVerifier,Scheduled-9] Cannot validate the identity of nsxt.corp.local due

to the hostname-verification being disabled. You are susceptible to man-in-the-middle attacks!

2023-02-15T00:02:08.144+0000 ERROR [vcf_lcm,464354a159a3f66a,bd2b,auditId=8360c948-7d6d-472d-a3be-13f71ab35d82,resourceType=NSX_T_MANAGER,resourceId=nsxt.corp.local ,name=nsxt.corp.local] [c.v.v.i.p.c.j.JsonDirectDeserializer,Scheduled-9] Error deserializing message.

com.fasterxml.jackson.core.JsonParseException: Unexpected character ('<' (code 60)): expected a valid value (JSON String, Number, Array, Object or token 'nul

l', 'true' or 'false')

at [Source: (ByteArrayInputStream); line: 1, column: 2]

at com.fasterxml.jackson.core.JsonParser._constructError(JsonParser.java:2391)

at com.fasterxml.jackson.core.base.ParserMinimalBase._reportError(ParserMinimalBase.java:735)

at com.fasterxml.jackson.core.base.ParserMinimalBase._reportUnexpectedChar(ParserMinimalBase.java:659)

You do not necessarily see any alarms of concern in the NSXT Manager UI.
NSXT Manager logs report issues similar to below in /var/log/upgrade-coordinator log:

2021-12-29T11:25:24.964Z ERROR localhost-startStop-1 ContextLoader - Context initialization failed

org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'upgradeCoordinatorStartupActions' defined in ServletContext resource [/WEB-INF/spring/upgrade-coordinator-config.xml]: Invocation of init method failed; nested exception is com.vmware.nsx.management.upgrade.exceptions.CannotStartUpgradeCoordinatorException: Failed to get required information from management plane. Upgrade coordinator cannot be started. Please ensure that management plane is up before starting upgrade coordinator.

at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.initializeBean(AbstractAutowireCapableBeanFactory.java:1694) ~[spring-beans-5.0.12.RELEASE.jar:5.0.12.RELEASE]

at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:573) ~[spring-beans-5.0.12.RELEASE.jar:5.0.12.RELEASE]

at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:495) ~[spring-beans-5.0.12.RELEASE.jar:5.0.12.RELEASE]

NOTE: iI is also possible that the upgrade-coordinator log hasn't logged in a considerable period of time.

The upgrade-coordinator service status may show that it's up and running:

/etc/init.d/upgrade-coordinator status
* upgrade-coordinator.service - upgrade-coordinator: VMware NSX Upgrade Coordinator
Loaded: loaded (/etc/init.d/upgrade-coordinator; disabled; vendor preset: enabled)
Active: active (running) since Wed 2022-08-10 04:45:49 UTC; 6 months 6 days ago
  Docs: man:systemd-sysv-generator(8)
Process: 5287 ExecStart=/etc/init.d/upgrade-coordinator start (code=exited, status=0/SUCCESS)
Process: 5260 ExecStartPre=/etc/init.d/upgrade-coordinator prestart (code=exited, status=0/SUCCESS)
  Tasks: 50 (limit: 4915)
CGroup: /system.slice/upgrade-coordinator.service
     |-5367 /usr/tanuki/bin/./wrapper /usr/tanuki/bin/../conf/upgrade-c

The /var/log/proton logs show log entries similar to below:

APPLICATION IS GOING RESTART (ShardingService: sharding allocation changed without notification on node ClusterNodeConfigModel/d2130242-0300-c523-0731-3149b162905a) =====

2021-12-25T17:05:29.304Z INFO application-restartor restartor - - [nsx@6876 comp="nsx-manager" level="INFO" subcomp="manager"] ===== APPLICATION IS GOING RESTART (Disconnected from cluster) =====

2021-12-29T08:53:57.863Z INFO application-restartor restartor - - [nsx@6876 comp="nsx-manager" level="INFO" subcomp="manager"] ===== APPLICATION IS GOING RESTART (Disconnected from database) =====

2021-12-29T10:51:00.549Z INFO application-restartor restartor - - [nsx@6876 comp="nsx-manager" level="INFO" subcomp="manager"] ===== APPLICATION IS GOING RESTART (Disconnected from database) =====

Environment

VMware Cloud Foundation 4.x

Cause

This issue is caused by problems occuring on the upgrade-controller service running on the NSX manager.

Resolution

Restart the upgrade-coordinator service on the UC node:

/etc/init.d/upgrade-coordinator restart

Run the precheck again and confirm the failure has cleared.

Additional Information

The KBs below also address common NSXT Audit failures:

https://kb.vmware.com/s/article/90683?lang=en_US

https://kb.vmware.com/s/article/86047?lang=en_US

https://kb.vmware.com/s/article/85764?lang=en_US