How to handle audit_log_health.audit_log_file_update_error alarm in NSX Manager

Products

• VMware NSX

Issue/Introduction

Title: Alarm for audit_log_health.audit_log_file_update_error
Event ID: audit_log_health.audit_log_file_update_error
Added in release: 3.1.0
Alarm Description

• Purpose: The purpose of this alarm is to inform the user that at least one of the monitored log files has read-only permissions or has incorrect user/group ownership on Manager, Global Manager, Edge, Public Cloud Gateway, KVM or Linux Physical Server nodes. Or log folder is missing in Windows Physical Server nodes. Or rsyslog.log is missing on Manager, Global Manager, Edge or Public Cloud Gateway nodes.
• Impact: Expect to see log write failure and log content is missing.

Environment

VMware NSX-T Data Center

Resolution

How to view the information require on the: 

- ls -l 

 here is a Sample of what it should look like : 

good (healthy)

-rw-r-----  1 syslog         adm              5797901 Sep 17 20:13 auth.log
-rw-r-----  1 syslog         adm             50335732 Sep 17 17:08 auth.log.1

bad (root- this is related to section 2) 

-rw-r-----  1 syslog         root              0 Sep 17 20:13 auth.log
-rw-r-----  1 syslog         adm             50335732 Sep 17 17:08 auth.log.1

 

1. On Manager and Global Manager nodes, Edge and Public Cloud Gateway nodes, Ubuntu KVM Host nodes ensure the permissions for the /var/log directory is 775 and the ownership is root:syslog. One Rhel KVM and BMS Host nodes ensure the permission for the /var/log directory is 755 and the ownership is root:root.

2. On Manager and Global Manager nodes, ensure the file permissions for:

- auth.log,

- nsx-audit.log,

- nsx-audit-write.log,

- rsyslog.log

 - syslog

under /var/log is 640 and ownership is syslog:adm.

3. On Edge and Public Cloud Gateway nodes, ensure the file permissions for:

- rsyslog.log

- syslog

under /var/log is 640 and ownership is syslog:adm.

4. On Ubuntu KVM Host and Ubuntu Physical Server nodes, ensure the file permissions of auth.log and vmware/nsx-syslog under /var/log is 640 and ownership is syslog:adm.

5. On Rhel KVM Host nodes and Centos/Rhel/Sles Physical Server nodes, ensure the file permission of vmware/nsx-syslog under /var/log is 640 and ownership is root:root.

6. If any of these files have incorrect permissions or ownership, invoke the commands chmod <mode> <path> and chown <user>:<group> <path>.

7. If rsyslog.log is missing on Manager, Global Manager, Edge or Public Cloud Gateway nodes, invoke the NSX CLI command restart service syslog which restarts the logging service and regenerates /var/log/rsyslog.log.

8. On Windows Physical Server nodes, ensure the log folder: C:\ProgramData\VMware\NSX\Logs exists. If not, re-install NSX on the Windows Physical Server nodes.

• Maintenance window required for remediation? No
• Note: If the error doesn't clear after the situation has been corrected a reboot of the manager will be require