Manager logs spammed with FILE_INTEGRITY_CHECK

Products

VMware NSX

Issue/Introduction

You are seeing VMware Aria Operations for Logs showing FILE_INTEGRITY_CHECK_FAILED messages regarding files or directories modified relating to /usr/share/corfu/conf* and /opt/vmware/cbm/* but there are no data path impacts.
There has been no intentional file changes

Below are few sample logs:

<Date>T04:01:08.165Z <manager fqdn> NSX 2357877 - [nsx@6876 comp="nsx-manager" subcomp="integrity-checker" username="root" level="WARNING"] [FILE_INTEGRITY_CHECK_FAILED] Reason : REGULAR FILE MODIFIED, Name : "/opt/vmware/cbm/etc/cbm-cloud.json" at <Date> 08:00:19.647994477 +0000
<Date>T04:01:08.166Z <manager fqdn> NSX 2357877 - [nsx@6876 comp="nsx-manager" subcomp="integrity-checker" username="root" level="WARNING"] [FILE_INTEGRITY_CHECK_FAILED] Reason : REGULAR FILE MODIFIED, Name : "/opt/vmware/cbm/etc/cbm.json" at <Date> 08:00:19.647994477 +0000
<Date>T04:01:08.168Z <manager fqdn> NSX 2357877 - [nsx@6876 comp="nsx-manager" subcomp="integrity-checker" username="root" level="WARNING"] [FILE_INTEGRITY_CHECK_FAILED] Reason : REGULAR FILE MODIFIED, Name : "/usr/share/corfu/conf/corfu-compactor-config.yml" at <Date> 08:00:26.178562775 +0000
<Date>T04:01:08.168Z <manager fqdn> NSX 2357877 - [nsx@6876 comp="nsx-manager" subcomp="integrity-checker" username="root" level="WARNING"] [FILE_INTEGRITY_CHECK_FAILED] Reason : REGULAR FILE MODIFIED, Name : "/usr/share/corfu/conf/nonconfig-corfu-compactor-config.yml" at <Date> 08:00:19.347995143 +0000
<Date>T04:01:08.168Z <manager fqdn> NSX 2357877 - [nsx@6876 comp="nsx-manager" subcomp="integrity-checker" username="root" level="WARNING"] [FILE_INTEGRITY_CHECK_FAILED] Reason : DIRECTORY MODIFIED, Name : "/usr/share/corfu/conf" at <Date> 08:00:26.178562775 +0000

<Date>T14:01:04.283Z <manager fqdn> NSX 907890 - [nsx@6876 comp="nsx-manager" subcomp="integrity-checker" username="root" level="WARNING" invalid="true"] [FILE_INTEGRITY_CHECK] Overall status : FAILED. RESULTS: TOTAL: 85846 VERIFIED: 87164 [ADDED: 1318 DELETED: 0 MODIFIED: 27]

Environment

VMware NSX
VMware NSX-T DataCenter

Cause

Few nsx services (corfu, cbm, proton, sha etc) runs commands like chown / chmod at the time of start of the service (from init.d scripts).
chown / chmod changes ctime of files / directories (even if permissions, ownership are same).
- Change timestamp (ctime) refers to the last time some metadata related to the file was changed.
Integrity-checker checks for ctime of files/dirs and it will report files as modified if it's ctime gets changed.
This will occur if you reboot the appliance or restart nsx services like corfu, cbm etc

Resolution

cbm and corfu related files are excluded from integrity-checker service in 9.1 release

To workaround this issue, a new baseline should be created and as preventive approach, feel free to follow the below workaround after every reboot of the manager or service restart.

Please run the following commands on all 3 Managers:

ssh as root user to the NSX Manager appliance
Create the new baseline (it may take some time):

# /opt/vmware/integrity-checker/bin/integrity_checker.py -f baseline
Manually run the integrity checker to confirm the logging has stopped

# /opt/vmware/integrity-checker/bin/integrity_checker.py -f check
# tail -f /var/log/vmware/integrity_checker.log

It will show 'Status : OK' (In the last line)

Note: integrity-checker does not affect any NSX functionality.

Additional Information

Below files relating to corfu and cbm are excluded from file integrity checker service in 9.1 version

REGULAR FILE MODIFIED, Name : "/usr/share/corfu/conf/corfu-compactor-config.yml"
REGULAR FILE MODIFIED, Name : "/usr/share/corfu/conf/nonconfig-corfu-compactor-config.yml"
REGULAR FILE MODIFIED, Name : "/opt/vmware/cbm/etc/cbm-cloud.json"
REGULAR FILE MODIFIED, Name : "/opt/vmware/cbm/etc/cbm.json"
REGULAR FILE MODIFIED, Name : "/opt/vmware/cbm/etc/log4j2.xml"
REGULAR FILE MODIFIED, Name : "/opt/vmware/cbm/etc/ufo-factory.properties"

Few other File Integrity check failures on below files or directories and their updates

Fix for below files is available in 9.1 version

SYMBOLIC LINK ADDED, Name : "/usr/lib/python3/dist-packages/OpenSSL/OpenSSL"
SYMBOLIC LINK ADDED, Name : "/usr/lib/python3/dist-packages/cryptography/cryptography"
REGULAR FILE MODIFIED, Name : "/opt/vmware/nsx-netopa/bin/nsx-sha"
REGULAR FILE MODIFIED, Name : "/opt/vmware/nsx-netopa/bin/sha_watchdog.sh"
DIRECTORY MODIFIED, Name : "/usr/lib/python3/dist-packages/OpenSSL"
DIRECTORY MODIFIED, Name : "/usr/lib/python3/dist-packages/cryptography"
Usage of below files got removed in 4.2.0 release (4.2 and above manager do not creates these files)

REGULAR EMPTY FILE MODIFIED, Name : "/usr/lib/saas/.large-ff"
REGULAR EMPTY FILE MODIFIED, Name : "/usr/lib/saas/.medium-ff"
There is a PR already open for below file and is being actively worked on

REGULAR FILE MODIFIED, Name : "/opt/vmware/nsx-opsagent/bin/watchdog.sh"

Note: Feel free to reach out to Broadcom support if you are seeing above log message to get an update on which version is the fix available on and refer this KB.