NSX audit precheck fails with 'unknown exception' error

Products

VMware Cloud Foundation

Issue/Introduction

When running a domain precheck prior to upgrade, the NSX-T audit precheck fails with "NSX-T Manager Audit for xxxxxx-xxxxxx failed with unknown exception"
SDDC Manager - /var/log/vmware/vcf/lcm/lcm-debug.log

name=nsxt.example.com] [c.v.v.i.p.c.r.h.AllowAllHostnameVerifier,Scheduled-9] Cannot validate the identity of nsxt.example.local due to the hostname-verification being disabled. You are susceptible to man-in-the-middle attacks!

2023-02-15T00:02:08.144+0000 ERROR [vcf_lcm,464354a159a3f66a,bd2b,auditId=8360c948-7d6d-####-####-########d82,resourceType=NSX_T_MANAGER,resourceId=nsxt.example.local ,name=nsxt.example.com] [c.v.v.i.p.c.j.JsonDirectDeserializer,Scheduled-9] Error deserializing message.com.fasterxml.jackson.core.JsonParseException: Unexpected character ('<' (code 60)): expected a valid value (JSON String, Number, Array, Object or token 'null', 'true' or 'false') at [Source: (ByteArrayInputStream); line: 1, column: 2]

at com.fasterxml.jackson.core.JsonParser._constructError(JsonParser.java:2391)

at com.fasterxml.jackson.core.base.ParserMinimalBase._reportError(ParserMinimalBase.java:735)

at com.fasterxml.jackson.core.base.ParserMinimalBase._reportUnexpectedChar(ParserMinimalBase.java:659)

No alarms of concern appear in the NSX-T Manager UI.
NSXT Manager logs report issues similar to below in /var/log/upgrade-coordinator log:

2021-12-29T11:25:24.964Z ERROR localhost-startStop-1 ContextLoader - Context initialization failed

org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'upgradeCoordinatorStartupActions' defined in ServletContext resource [/WEB-INF/spring/upgrade-coordinator-config.xml]: Invocation of init method failed; nested exception is com.vmware.nsx.management.upgrade.exceptions.CannotStartUpgradeCoordinatorException: Failed to get required information from management plane. Upgrade coordinator cannot be started. Please ensure that management plane is up before starting upgrade coordinator.

at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.initializeBean(AbstractAutowireCapableBeanFactory.java:1694) ~[spring-beans-5.0.12.RELEASE.jar:5.0.12.RELEASE]

at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:573) ~[spring-beans-5.0.12.RELEASE.jar:5.0.12.RELEASE]

at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:495) ~[spring-beans-5.0.12.RELEASE.jar:5.0.12.RELEASE]

Note: It is also possible that the upgrade-coordinator log hasn't logged in a considerable period of time.

The upgrade-coordinator service status may show that it's up and running:

/etc/init.d/upgrade-coordinator status
* upgrade-coordinator.service - upgrade-coordinator: VMware NSX Upgrade Coordinator
Loaded: loaded (/etc/init.d/upgrade-coordinator; disabled; vendor preset: enabled)
Active: active (running) since Wed 2022-08-10 04:45:49 UTC; 6 months 6 days ago
  Docs: man:systemd-sysv-generator(8)
Process: 5287 ExecStart=/etc/init.d/upgrade-coordinator start (code=exited, status=0/SUCCESS)
Process: 5260 ExecStartPre=/etc/init.d/upgrade-coordinator prestart (code=exited, status=0/SUCCESS)
  Tasks: 50 (limit: 4915)
CGroup: /system.slice/upgrade-coordinator.service
     |-5367 /usr/tanuki/bin/./wrapper /usr/tanuki/bin/../conf/upgrade-c

The /var/log/proton logs show log entries similar to below:

APPLICATION IS GOING RESTART (ShardingService: sharding allocation changed without notification on node ClusterNodeConfigModel/d2130242-0300-c523-0731-3149b162905a) =====

2021-12-25T17:05:29.304Z INFO application-restartor restartor - - [nsx@6876 comp="nsx-manager" level="INFO" subcomp="manager"] ===== APPLICATION IS GOING RESTART (Disconnected from cluster) =====

2021-12-29T08:53:57.863Z INFO application-restartor restartor - - [nsx@6876 comp="nsx-manager" level="INFO" subcomp="manager"] ===== APPLICATION IS GOING RESTART (Disconnected from database) =====

2021-12-29T10:51:00.549Z INFO application-restartor restartor - - [nsx@6876 comp="nsx-manager" level="INFO" subcomp="manager"] ===== APPLICATION IS GOING RESTART (Disconnected from database) =====