Disk write of NSX Edge VMs periodically spikes on the hour
Article ID: 322871
Updated On:
Products
VMware NSX Networking
Issue/Introduction
Symptoms:
- vCenter performance chart of Edge VMs shows disk write periodically spikes on the hour.
- Other VMs might be suffered from degraded storage performance if many Edge VMs reside in the same physical storage.
- Many 8MB files are generated on the hour in /var/log/journal/<machine-id> .
Environment
VMware NSX-T Data Center
VMware NSX-T Data Center 3.x
VMware NSX-T Data Center 3.x
Cause
Edge appliances run integrity checker on the hour.
It executes find / -print0 | xargs -0 to check integrity of many files in the appliance.
Since 3.1.0 auditd logs execve system calls and the logs are stored in journal log.
Integrity checker passes tremendous numbers of arguments to xargs, and all the arguments of execve logs are considered as field name by journald.
So field hash table of a journal file grows rapidly beyond the threshold, and the file is rotated immediately.
Each journal file is 8MB at minimum.
Thus 8MB journal files rotate so fast and so many journal files are generated that large amount of disk write is triggered on the hour.
Manager VMs are not affected because auditd does not log execve system calls.
It executes find / -print0 | xargs -0 to check integrity of many files in the appliance.
Since 3.1.0 auditd logs execve system calls and the logs are stored in journal log.
Integrity checker passes tremendous numbers of arguments to xargs, and all the arguments of execve logs are considered as field name by journald.
So field hash table of a journal file grows rapidly beyond the threshold, and the file is rotated immediately.
Each journal file is 8MB at minimum.
Thus 8MB journal files rotate so fast and so many journal files are generated that large amount of disk write is triggered on the hour.
Manager VMs are not affected because auditd does not log execve system calls.
Resolution
This is a known issue affecting NSX-T 3.1.0 - 3.1.2.1.
The issue is resolved in NSX-T 3.1.3 or 3.2.
Workaround:
There are 2 workarounds.
- Mask systemd-journald-audit.socket.
/bin/systemctl stop systemd-journald-audit.socket
/bin/systemctl disable systemd-journald-audit.socket
/bin/systemctl mask systemd-journald-audit.socket
It is implemented on 3.1.3 and 3.2 or later.
Then restart journald.
systemctl restart systemd-journald
- Disable integrity checker.
/opt/vmware/integrity-checker/bin/integrity_checker.py -f disable
Additional Information
Impact/Risks:
Edge VMs trigger large disk write on the hour, at the same time.
It might degrade datastore performance if many Edge VMs reside in the same physical storage.
Other VMs might be suffered from such degraded storage performance.
Edge VMs trigger large disk write on the hour, at the same time.
It might degrade datastore performance if many Edge VMs reside in the same physical storage.
Other VMs might be suffered from such degraded storage performance.
Feedback
Yes
No
Powered by
